Distributed machine learning training system based on adaptive topology and auxiliary routing

Abstract

The application discloses a distributed machine learning training system based on adaptive topology and auxiliary routing, and comprises a user plane, a scheduling plane and a data plane.The user plane is used for providing a custom policy for constructing a topology; the scheduling plane comprises a network perception module, a policy making module and a policy consistency module; and the data plane comprises a policy execution module and a network measurement module.The application alleviates the communication bottleneck problem of multi-intelligent algorithm centers across wide-area interconnection, and solves the problem that the efficiency of an existing distributed machine learning system is limited in a wide-area network with heterogeneous resources and dynamic changes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent computing system technology, specifically relating to a distributed machine learning training system based on adaptive topology and assisted routing. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, large-scale language models, such as ChatGPT, have significantly improved the quality and application potential of AI-generated content, initially opening up new areas for enhancing human productivity. However, training such large-scale models with billions of parameters requires extremely large computing and storage resources, far exceeding the operational capabilities of current infrastructure. At the same time, high-performance computing capabilities are particularly scarce in data-intensive and computationally-intensive new intelligent computing businesses in the digital economy era, such as in the East-West Computing Project. Therefore, accelerating the breaking down of computing power barriers is particularly urgent.

[0003] As a result, intelligent computing has emerged. It combines cross-domain resources such as high-performance computing systems, cloud clusters, and edge infrastructure. By horizontally distributing different computing platforms and vertically unifying resources from the edge to the cloud, it achieves horizontal expansion and vertical synergy of computing power, providing efficient computing services for artificial intelligence training tasks. However, intelligent computing systems are struggling with communication bottlenecks caused by network bandwidth limitations.

[0004] Specifically, as the distribution of intelligent computing nodes continues to expand, the physical distance between different network domains is also increasing. This makes the exchange of resources such as data, parameters, and models between different regions subject to network bandwidth and latency limitations, resulting in slower cross-domain transmission speeds. This problem is particularly pronounced in dynamic heterogeneous networks; for example, the connection quality between edge devices and cloud data centers is affected by various environmental factors. Furthermore, in communication-intensive tasks such as distributed machine learning, due to the deterministic nature of the tasks, computing nodes need to perform deep iterations and parameter exchanges to achieve joint learning, which places high demands on real-time transmission and communication performance. However, when massive training samples are distributed and stored in different network domains, frequent large-scale parameter exchanges often cause network congestion, reducing overall training efficiency.

[0005] To address the pain points faced by current AI systems with dynamically heterogeneous resources and wide-area distributed training, there are three main existing technologies: setting multiple roles for servers, dynamic tree topology construction based on network detection modules, and link throughput measurement based on application traffic.

[0006] Some methods leverage the multiple roles of servers to achieve data flow management and system scalability. Through clever configuration of root, leaf, and intermediate nodes, data flows from the leaves to the root, while intermediate nodes aggregate flows from their child nodes, reducing network traffic and minimizing the amount of data received by the server. Furthermore, scalability is improved by redistributing hot traffic on the server to idle links between worker nodes. However, these methods are inherently limited by a lack of network resource information, restricting the system's accurate understanding of network conditions and further impacting its adaptability and performance. Moreover, they typically employ a hierarchical parametric server architecture based on balanced tree coverage, with some systems consisting of only two layers. This fixed structure is inefficient for fully utilizing heterogeneous and dynamic wide area networks. Therefore, these methods are unsuitable for wide-area distributed machine learning training systems with dynamically heterogeneous resources.

[0007] Some methods construct dynamic tree topologies that adapt to changing network conditions by setting up network probing modules. For example, some studies integrate DPDK-based latency probing and iPerf-based bandwidth probing modules for speed measurement to perform a comprehensive scan of cloud data center networks, obtaining the actual location and link performance characteristics of each computing node, such as latency and bandwidth data. Then, based on the collected physical network topology data, a two-layer balanced tree virtual topology model matching the actual physical network is constructed. The probing techniques used in these methods generate additional probing traffic. Long-term, large-scale probing may increase network load and potentially interfere with application traffic. In addition, probing scheduling and result reporting also consume system resources, making it impossible to achieve high-precision real-time monitoring in production environments. Therefore, these methods are not suitable for wide-area distributed machine learning training systems with dynamically heterogeneous resources.

[0008] Some methods use application traffic itself as a probe to measure link throughput. For example, application traffic is sent to the target link and the transmission and acknowledgment times are observed, and the link throughput is calculated using a stop-and-wait protocol. The obtained throughput data is used to construct a minimum spanning tree, further allowing for automatic adjustment of the tree topology to respond promptly to network changes. These methods, employing a stop-and-wait protocol for link throughput probing, limit the overlap between transmission and aggregation, reduce the parallelism of distributed training, and significantly slow down training speed. Furthermore, the need to wait for acknowledgment signals leads to inefficient use of network resources, increases communication round-trip time, and consequently increases communication overhead. Especially in high-latency network environments, this protocol further reduces data transmission efficiency and is sensitive to network latency, making it unsuitable for wide-area distributed machine learning training systems with dynamically heterogeneous resources. Summary of the Invention

[0009] To address the aforementioned shortcomings in existing technologies, the distributed machine learning training system and method provided by this invention solve the performance bottleneck problem of distributed training in existing wide-area environments.

[0010] To achieve the aforementioned objectives, the present invention employs the following technical solution: a distributed machine learning training system based on adaptive topology and assisted routing, comprising:

[0011] The user plane is used to provide custom strategies for building topologies;

[0012] The scheduling plane includes a network awareness module, a policy formulation module, and a policy consistency module. The network awareness module is used to perceive network parameters. The policy formulation module is used to construct the fastest aggregation topology of multiple root servers based on the perceived network parameters and user-provided custom policies. The policy consistency module is used to publish the constructed fastest aggregation topology of multiple root servers as a policy for model transmission paths to the data plane based on topology consistency protocol and auxiliary routing consistency protocol.

[0013] The data plane includes a policy execution module and a network measurement module. The policy execution module is used to execute distributed policies across multiple intelligent computing nodes in the network, while the network measurement module is used to acquire network status and report data in real time as the data basis for the network perception module to perceive network parameters.

[0014] Furthermore, the network awareness module senses network parameters by using lightweight, real-time measurement of link throughput and hierarchical clock synchronization techniques.

[0015] Furthermore, the topology consistency protocol is used to complete the adaptive scheduling of the fastest aggregation topology of multiple root servers, which includes the topology update request strategy and scheduling before communication, as well as the transmission and aggregation of cached model data and local topology updates.

[0016] The auxiliary routing consistency protocol is used to assist the main path in data transmission based on multipath auxiliary path technology and idle paths. It updates the routing policy according to the construction process of the fastest aggregation topology of multiple root servers and publishes the updated routing policy to the intelligent computing nodes. The message structure of the nodes on the auxiliary path recorded in the routing policy includes sender identifier, receiver identifier, auxiliary route identifier, route identifier, other identifiers and transmission data.

[0017] Furthermore, the method by which the strategy formulation module constructs the fastest aggregation topology for multiple root servers is as follows:

[0018] S1. Initialize the undirected graph G = (V, E), initialize the number of trees N, and initialize the data throughput s between two adjacent nodes; where V is the set of vertices of the undirected graph, E is the set of edges of the undirected graph, and the weight w of the edges is a positive integer.

[0019] S2. Perform non-overlapping path search to obtain the root set R and candidate paths P;

[0020] S3. For each root vi∈R, initialize the tree T rooted at node vi. vi ;

[0021] S4. For each non-root node vj, select the first path p←P from the candidate paths P. vi→vj .POP_FIRST() is used as the fastest aggregation path; where vj∈V,

[0022] S5. Reset vl = vi, where vr is the adjacent node of vi;

[0023] Here, vl is the root node vi, and vr is the next node of vl;

[0024] S6. Let nodes vl and vr traverse path p and find all nodes involved in path p to establish parent-child relationships vl.CHILD.ADD(vr) and vr.PARENT(vl);

[0025] S7. Obtain the fastest aggregation topology for the R root server.

[0026] Furthermore, in step S2, the method for performing non-overlapping path search is specifically as follows:

[0027] S21. Initialize the undirected graph G = (V, E), initialize the number of trees N, initialize the data throughput s between two adjacent nodes, and initialize the root set. and candidate paths

[0028] S22, Reset edge weights w vi→vj =1 / s vi→vj , vj∈V, where vi and vj are on edge e vi→vj ∈E adjacent; where s vi→vj This represents the throughput of the edge corresponding to vi→vj, w vi→vj This represents the cost of the edge corresponding to vi→vj;

[0029] S23. Based on the reset edge weights, perform R-root FAPT joint tree coverage to obtain the root set R and node path H;

[0030] S24. For any two reachable nodes vi and vj, add a candidate path P. vi→vj =P vi→vj ∪H vi→vj Remove used edges E = Ee(H) vi→vj);in,

[0031] S25. Determine if there are any unused edges such that |E|>0;

[0032] If so, return to step S22;

[0033] If not, proceed to step S26;

[0034] S26. Obtain the root set R and candidate paths P.

[0035] Furthermore, in step S23, the method for performing the R-root FAPT joint tree cover is specifically as follows:

[0036] S23-1. Initialize the undirected graph G = (V, E), initialize the positive edge weights w, initialize the root set R, and initialize the number of trees N;

[0037] S23-2. For each node vi∈V as the root, run Dijkstra's algorithm to obtain the cost w from the root vi to other nodes vj. vi→vj and node path H vi→vj ,

[0038] S23-3. Add node path to node vi

[0039] S23-4. Calculate the tree T rooted at node vi. vi Cost w(T) vi ):

[0040]

[0041] S24-5, Determining the root set Is it valid?

[0042] If so, proceed to step S24-6;

[0043] If not, proceed to step S24-7;

[0044] S24-6, Calculate the nodes in V according to their cost w(T) vi ), Sort the nodes and select the N nodes with the lowest cost to obtain the root set R;

[0045] S24-7. Based on node vi, obtain the root set R and its corresponding node path H.

[0046] Furthermore, for constructing the fastest aggregation topology for multiple root servers, a quality score is performed based on the network parameters perceived by the network awareness module, and the topology structure is adjusted to achieve load balancing among the multiple root servers.

[0047] Furthermore, when constructing the fastest aggregation topology for multiple root servers, multipath-assisted routing technology is used for routing policy scheduling. The routing policy scheduling method is as follows:

[0048] When the communication scheduler receives a data tensor while managing the message queue, the data tensor is divided into small blocks. The communication scheduler then polls the main path and auxiliary paths in turn and allocates the new blocks to their message queues.

[0049] During the polling process, the primary path has a higher priority than the secondary path, and the secondary paths are polled in order of priority, with the priority of the secondary path being inversely proportional to its cost.

[0050] When the primary path is busy, new blocks are allocated to secondary paths in a round-robin fashion. Each secondary path has a message queue, and secondary paths with idle message queues are allocated to transmit new blocks. If all secondary paths are busy, the primary path is allocated to transmit new blocks.

[0051] Furthermore, when implementing the deployment environment of the system, it is deployed in multiple intelligent computing center clusters across multiple domains, and the intelligent computing center clusters are interconnected through a wide area network;

[0052] The multiple intelligent computing center clusters include an intelligent computing scheduling center that serves as a scheduling plane and multiple intelligent computing training centers that serve as data planes.

[0053] The intelligent computing scheduling center and the intelligent computing training center communicate bidirectionally, and multiple intelligent computing training centers are connected through a wide area network infrastructure.

[0054] Furthermore, when implementing the deployment environment of the system, it is deployed in a smart computing center cluster, and the smart computing centers are interconnected via a wide area network;

[0055] The intelligent computing center contains at least two physical nodes, including a scheduling node that serves as the scheduling plane and other training nodes that serve as the data plane.

[0056] The scheduling node and the training node communicate bidirectionally, and the training nodes are interconnected within the local area network.

[0057] The beneficial effects of this invention are as follows:

[0058] This invention alleviates the communication bottleneck problem of cross-wide area interconnection among multiple intelligent computing centers and solves the performance limitation problem of existing distributed machine learning systems in wide area networks with heterogeneous and dynamically changing resources. Specifically, the system of this invention has the following beneficial effects:

[0059] (1) Improves the overall training efficiency of wide-area distributed machine learning systems by 7.5 to 9.2 times: Existing wide-area distributed machine learning training systems cannot adapt to changes in the wide area network in a timely manner, easily generating network hotspots during training, leading to communication bottlenecks and severely affecting communication efficiency. Furthermore, the systems have poor fault tolerance and are unsuitable for wide area network environments with dynamically heterogeneous resources. The system and method proposed in this invention can significantly improve the processing capabilities of interconnected intelligent computing centers via wide area networks, greatly improving the efficiency of resource sharing and collaborative computing between different regions in a wide-area distributed machine learning system.

[0060] (2) The fastest aggregation topology technology using multiple root servers improves the synchronization efficiency of distributed machine learning models by 4.2 to 6.7 times: Existing methods use fixed tree structures, leading to unbalanced communication patterns and potential bottlenecks, resulting in inefficient data synchronization of distributed machine learning models. This invention proposes a fastest aggregation topology technology using multiple root servers and designs a corresponding construction method. This technology not only supports load balancing among servers with multiple parameters, fully utilizes idle links and reduces network hotspots, but also minimizes model synchronization time.

[0061] (3) Distributed machine learning multipath-assisted routing technology improves transmission parallelism and significantly alleviates congestion on the main path, resulting in a gain of 30% to 65%. Strictly adhering to the optimal topology transmission model restricts traffic flow to links within the selected topology, making it impossible to measure throughput on external links, potentially leading to local optima. Incomplete network awareness module information and slow topology updates reduce the efficiency of distributed training. This invention introduces distributed machine learning multipath-assisted routing technology, allowing measurement of throughput on external links of the selected topology, collecting complete network information, and reducing the risk of local optima. Multiple auxiliary paths can automatically switch to backup paths when the main path fails, improving system fault tolerance. Simultaneously, multipath transmission can balance service load, reduce congestion, and accelerate distributed training.

[0062] (4) Network awareness technology using distributed machine learning native flow probes enables lightweight and accurate link throughput measurement. Enabling network awareness can reduce the average runtime by 24%. Existing methods have two major problems: DPDK probes generate additional traffic interference and cannot measure latency in real time; application traffic probing methods hinder transmission aggregation and overlap, significantly reducing the speed of distributed training. This invention introduces network awareness technology using distributed machine learning native flow probes to achieve lightweight, real-time measurement of link throughput, and ensures high accuracy through hierarchical clock synchronization.

[0063] (5) Distributed Node Topology Consistency Protocol Ensures Consistency of Topology and Routing Policies: In distributed training, once a new policy is formulated, it should be immediately communicated to the intelligent computing nodes. However, due to the uncertainty of network latency, ensuring policy consistency among all intelligent computing nodes becomes a key challenge. The coexistence of old and new policies may lead to confusion or even errors in model transmission. This invention proposes a distributed node topology consistency protocol that can adjust the topology according to network changes, ensuring consistency of the main topology and routing policies among all intelligent computing nodes.

[0064] (6) As the number of model parameters increases, the efficiency of existing distributed training systems drops sharply. Overcoming this scalability barrier becomes particularly urgent. The scalability of this invention ensures that it is applicable to training of wide-area distributed machine learning systems with different models and cluster sizes. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the distributed machine learning training system architecture based on adaptive topology and assisted routing provided by the present invention.

[0066] Figure 2 This is a schematic diagram of the network awareness technology for distributed machine learning native stream probes provided by the present invention.

[0067] Figure 3 This is a schematic diagram of the topology consistency protocol provided by the present invention.

[0068] Figure 4 A schematic diagram of the auxiliary routing consistency protocol provided by the present invention.

[0069] Figure 5 This is a schematic diagram of the multi-root server load balancing technology provided by the present invention.

[0070] Figure 6 This is a schematic diagram illustrating the scheduling process of the multipath assisted routing technology provided by the present invention.

[0071] Figure 7 This is a schematic diagram illustrating the synergistic effects of the technical points of this invention.

[0072] Figure 8 This is a schematic diagram of normalized throughput for different topology coverages provided by the present invention.

[0073] Figure 9 The normalized throughput provided by this invention as the number of root servers increases.

[0074] Figure 10 The system throughput of the multi-path auxiliary routing provided by this invention.

[0075] Figure 11 This is a schematic diagram illustrating the iteration runtime when enabling and disabling the sensing function, as provided by the present invention.

[0076] Figure 12 A schematic diagram illustrating the scalability of NETSTORM provided by this invention; (a) model size; (b) cluster size. Detailed Implementation

[0077] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0078] This invention provides a distributed machine learning training system based on adaptive topology and assisted routing, such as... Figure 1 As shown, it includes:

[0079] The user plane is used to provide custom strategies for building topologies;

[0080] The scheduling plane includes a network awareness module, a policy formulation module, and a policy consistency module. The network awareness module is used to perceive network parameters. The policy formulation module is used to construct the fastest aggregation topology of multiple root servers based on the perceived network parameters and user-provided custom policies. The policy consistency module is used to publish the constructed fastest aggregation topology of multiple root servers as a policy for model transmission paths to the data plane based on topology consistency protocol and auxiliary routing consistency protocol.

[0081] The data plane includes a policy execution module and a network measurement module. The policy execution module is used to execute distributed policies across multiple intelligent computing nodes in the network, while the network measurement module is used to acquire network status and report data in real time as the data basis for the network perception module to perceive network parameters.

[0082] In this embodiment of the invention, the user plane provides a custom strategy for customizing the initial or fixed topology, enabling users to better control the system of the invention when transmitting and aggregating model traffic.

[0083] In this embodiment of the invention, for the scheduling plane, the network awareness module senses network parameters by using lightweight, real-time measurement of link throughput and hierarchical clock synchronization techniques.

[0084] Specifically, such as Figure 2As shown, network awareness technology is implemented based on distributed machine learning native flow probes. Its technical features include lightweight design, real-time measurement of link throughput, and high accuracy ensured through the use of hierarchical clock synchronization. The lightweight network awareness technology keeps resource consumption to a minimum, monitoring network throughput by collecting and analyzing model traffic without causing additional traffic to enter the network. The real-time link throughput measurement network awareness technology ensures uninterrupted network awareness during training, using the model traffic of the running distributed machine learning application as a probe. The high-accuracy network awareness technology, using hierarchical clock synchronization, minimizes clock drift during NTP operation. The scheduler synchronizes the root server's clock, and the root server synchronizes the clocks of its child nodes. This ensures minimal latency from the intelligent computing node to the root server in FAPT.

[0085] In this embodiment of the invention, for the scheduling plane, the consistency protocol in the policy consistency module includes a topology consistency protocol and an auxiliary routing consistency protocol;

[0086] Specifically, such as Figure 3 As shown, the topology consistency protocol is used to achieve adaptive scheduling of the fastest aggregation topology across multiple root servers. It includes a pre-communication topology update request strategy and scheduling, as well as the caching of model data and the transmission and aggregation of locally updated topology data. Specifically, the pre-communication topology update request strategy and scheduling allows intelligent computing nodes to initiate topology update requests to the scheduler before each communication phase and remain blocked until their local topology is updated. The caching of model data and the transmission and aggregation of locally updated topology data allows outdated intelligent computing nodes to receive model data from an unknown intelligent computing node, which will cache this data until its local topology is updated. Once the update is complete, the intelligent computing node can perform model aggregation and transmission.

[0087] For the auxiliary routing consistency protocol, it is used to supplement the main path for data transmission based on multi-path auxiliary path technology, utilizing idle paths. It updates the routing policy according to the construction process of the fastest aggregation topology of multiple root servers and publishes the updated routing policy to the intelligent computing nodes; among them, such as Figure 4As shown, the message structure recording nodes on the secondary path in the routing strategy includes sender identifier (sender_id), receiver identifier (receiver_id), secondary route identifier (Is_Aux), route identifier (Route), other identifiers (Others), and transmission data (Data). The sender identifier field is used by the message sender to determine the receiver address; the receiver identifier field is used by the message receiver to confirm whether it is the correct receiver; the secondary route identifier field is used by the message receiver to confirm whether it is part of the secondary path and does not participate in aggregation or change the destination node; the route identifier field records information about the nodes on the secondary path, which the message receiver uses to forward the message along the nodes recorded in the route field; the other identifier fields are typically used to determine specific attributes of the message or indicate information such as the message type, priority, or processing requirements; the transmission data field carries the actual message content, data, payload, or payload, which is the main information part of the message and needs to be accurately delivered to the destination to ensure effective message delivery and processing.

[0088] In this embodiment of the invention, for the scheduling plane, the policy formulation module constructs a custom policy based on the topology issued by the user, building a multi-root server fastest aggregation topology. During topology construction, it is assumed that the physical topology and link bandwidth are known, and the goal is to construct an optimal tree topology. To quantify the quality of the generated topology, the completion time of the tree is defined as: given a tree T... vi With node vi as the root, edge e ∈ E, and edge weight w(e) as positive, then the weight w(T) of the tree is... vi The completion time of a tree is defined as the maximum cumulative weight among all possible paths P from the root vi to the leaf vj.

[0089]

[0090] Where p l ∈P is one of the paths from the root vi to a leaf node, and path p l The weight w(e) of edge e on the edge represents the time required to transmit one unit of data.

[0091] Definition Tree T vi The quality score is q i :

[0092]

[0093] FAPT is defined as applying Dijkstra's algorithm to every possible root vi∈V to minimize the cumulative edge weights of all paths.

[0094] Define G = (V, E) as an undirected graph with positive edge weights w: E → R+, and let... Let R be a set of roots. Define the root FAPT union tree cover. It is a subgraph that is the union of a set of FAPTs, i.e., T = {T v1 , ···, T vN}(N=|R|) makes And each tree T vi Each has a distinct root vi∈R, and its node set |V(T) vi | = |V|. Trees in T can share edges. Define the cost of the R-rooted FAPT joint tree cover as the maximum weight of any tree in T, i.e. The goal is to find the coverage with minimum cost J. Where J is the maximum weight of any tree in T. This problem can be expressed mathematically as follows:

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] Since the root set is known, for each root node vi, construct all possible trees T with vi as the root. vi set Therefore, a subgraph is constructed from graph G. The problem can be transformed into a problem involving sets. Search for the optimal tree T vi The questions are as follows:

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] The goal of this problem is to find a set of trees given a root R. Make the weight w(T) in T vi The largest tree T viThe weight of each tree is minimized. Considering the independence of any two trees in the set, this can be achieved by minimizing the weight of all trees in T. When the weight of each tree is minimized, the maximum weight of these trees is naturally minimized.

[0107] Therefore, the problem can be further simplified to:

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Therefore, the fastest aggregation topology for multiple root servers can be constructed by searching FAPT for each root.

[0114] Based on this, the method for constructing the fastest aggregation topology for multiple root servers in the strategy formulation module of this embodiment is as follows:

[0115] S1. Initialize the undirected graph G = (V, E), initialize the number of trees N, and initialize the data throughput s between two adjacent nodes; where V is the set of vertices of the undirected graph, E is the set of edges of the undirected graph, and the weight w of the edges is a positive integer.

[0116] S2. Perform non-overlapping path search to obtain the root set R and candidate paths P;

[0117] S3. For each root vi∈R, initialize the tree T rooted at node vi. vi ;

[0118] S4. For each non-root node vj, select the first path p←P from the candidate paths P. vi→vj .POP_FIRST() is used as the fastest aggregation path; where vj∈V,

[0119] S5. Reset vl = vi, where vr is the adjacent node of vi;

[0120] Here, vl is the root node vi, and vr is the next node of vl;

[0121] S6. Let nodes vl and vr traverse path p and find all nodes involved in path p to establish parent-child relationships vl.CHILD.ADD(vr) and vr.PARENT(vl);

[0122] In this context, the parent-child relationship is represented by vl.CHILD.ADD(vr) and vr.PARENT(vl), which means adding node vr as a child node of node vl and setting node vl as the parent node of node vr.

[0123] S7. Obtain the fastest aggregation topology for the R root server.

[0124] In step S2 of this embodiment, the method for performing non-overlapping path search is as follows:

[0125] S21. Initialize the undirected graph G = (V, E), initialize the number of trees N, initialize the data throughput s between two adjacent nodes, and initialize the root set. and candidate paths

[0126] S22, Reset edge weights w vi→vj =1 / s vi→vj , vj∈V, where vi and vj are on edge e vi→vj ∈E adjacent; where s vi→vj w represents the throughput (i.e., weight) of the edge corresponding to vi→vj. vi→vj The weight s represents the cost of the edge from vi to vj (i.e., the time consumed to transmit a unit of data), and its weight is s. vi→vj The larger the value, the longer the time w is consumed in transmitting a unit of data. vi→vj The smaller;

[0127] S23. Based on the reset edge weights, perform R-root FAPT joint tree coverage to obtain the root set R and node path H;

[0128] S24. For any two reachable nodes vi and vj, add a candidate path P. vi→vj =P vi→vj ∪H vi→vj Remove used edges E = Ee(H) vi→vj );in,

[0129] S25. Determine if there are any unused edges such that |E|>0;

[0130] If so, return to step S22;

[0131] If not, proceed to step S26;

[0132] S26. Obtain the root set R and candidate paths P;

[0133] In step S23, based on the reset edge weights, the root FAPT joint tree is covered to obtain the root set R. In step S24, candidate paths are added to any two reachable nodes vi and vj to obtain candidate paths P.

[0134] In step S23 of this embodiment, the method for performing R-root FAPT joint tree coverage is as follows:

[0135] S23-1. Initialize the undirected graph G = (V, E), initialize the positive edge weights w, initialize the root set R, and initialize the number of trees N;

[0136] S23-2. For each node vi∈V as the root, run Dijkstra's algorithm to obtain the cost w from the root vi to other nodes vj. vi→vj and node path H vi→vj ,

[0137] S23-3. Add node path to node vi

[0138] S23-4. Calculate the tree T rooted at node vi. vi Cost w(T) vi ):

[0139]

[0140] S24-5, Determining the root set Is it valid?

[0141] If so, proceed to step S24-6;

[0142] If not, proceed to step S24-7;

[0143] S24-6, Calculate the nodes in V according to their cost w(T) vi ), Sort the nodes and select the N nodes with the lowest cost to obtain the root set R;

[0144] S24-7. Based on node vi, obtain the root set R and its corresponding node path H;

[0145] The node path is obtained by adding a node path to node vi in ​​steps S23-2 to S23-3.

[0146] In this embodiment of the invention, the quality of the network parameters perceived by the network awareness module is scored for constructing the fastest aggregation topology of multiple root servers, and the topology structure is adjusted to balance the load of the multiple root servers.

[0147] Specifically, in this embodiment, such as Figure 5As shown, the constant CHUNK_SIZE is used to control the upper limit of the data block size, reducing the memory and bandwidth pressure on a single root server. Tensors smaller than CHUNK_SIZE are treated as a single block, while large tensors are divided into smaller blocks of similar size.

[0148] In this embodiment, the multi-parameter server load balancing technology based on the fastest aggregation topology of multiple roots performs a quality score on the constructed topology based on perceived network parameters, and allocates the aforementioned partitioned tensor blocks to multiple root servers to ensure low cost w(T) vi The root server vi of ) accounts for a large portion of the parameters because it has better network conditions. Among them, the quality score qi = 1 / (w(Tvi)).

[0149] In this embodiment of the invention, when constructing the fastest aggregation topology for multiple root servers, multi-path assisted routing technology is used for routing policy scheduling, such as... Figure 6 As shown, its routing policy scheduling method is as follows:

[0150] When the communication scheduler receives a data tensor while managing the message queue, the data tensor is divided into small blocks. The communication scheduler then polls the main path and auxiliary paths in turn and allocates the new blocks to their message queues.

[0151] During the polling process, the primary path has a higher priority than the secondary path, and the secondary paths are polled in order of priority, with the priority of the secondary path being inversely proportional to its cost.

[0152] When the primary path is busy, new blocks are allocated to secondary paths in a round-robin fashion. Each secondary path has a message queue, and secondary paths with idle message queues are allocated to transmit new blocks. If all secondary paths are busy, the primary path is allocated to transmit new blocks.

[0153] Specifically, in this embodiment, during the routing path scheduling process, a default rule is set to make the communication scheduler follow the message queue that allocates new blocks to the main path.

[0154] When the primary path is currently busy, i.e., when the number of data blocks in its message queue exceeds the value of PRIMARY_BUSY_BOUND, new blocks are allocated to one of the secondary paths.

[0155] Each secondary path has a finite message queue size AUXILIARY_QUEUE_LENGTH. Any secondary path with free queue space is considered free. If a free secondary path is found, it is allocated to help transmit new blocks. Otherwise, if all secondary paths are currently busy, the primary path is allocated for transmission.

[0156] In this embodiment of the invention, based on the above system structure, two system deployment methods are provided:

[0157] (1) When implementing the deployment environment of the system, it is deployed in multiple intelligent computing center clusters across domains, and the intelligent computing center clusters are interconnected through a wide area network; among the multiple intelligent computing center clusters, there is an intelligent computing scheduling center as a scheduling plane and multiple intelligent computing training centers as data planes; the intelligent computing scheduling center and the intelligent computing training centers communicate bidirectionally, and the multiple intelligent computing training centers are connected through a wide area network infrastructure.

[0158] All intelligent computing centers possess diverse network resource configurations. The intelligent computing scheduling center, responsible for the scheduling plane function, has the ability to acquire user-defined policies, while the intelligent computing training center, responsible for the data plane function, has the ability to acquire training data. The intelligent computing scheduling center is responsible for implementing the scheduling plane function of the distributed machine learning training system architecture with adaptive topology scheduling and assisted routing acceleration, and deploys a central scheduler that implements the network awareness module, policy formulation module, and policy consistency module. Multiple intelligent computing training centers are responsible for implementing the data plane function of the distributed machine learning training system architecture with adaptive topology scheduling and assisted routing acceleration. Each intelligent computing training center deploys an intelligent computing node responsible for the network measurement module and policy execution module, as well as a database or file system for storing training datasets.

[0159] (2) When implementing the deployment environment of the system, it is deployed in a smart computing center cluster and interconnected within the smart computing center through a wide area network; the smart computing center contains at least two physical nodes, including a scheduling node as a scheduling plane and other training nodes as data planes; the scheduling node and the training node communicate bidirectionally, and the training nodes are interconnected within a local area network.

[0160] In this system, all physical nodes possess diverse network resource configurations. The scheduling node, responsible for the scheduling plane function, has the ability to acquire user-defined policies, while the training nodes, responsible for the data plane function, have the ability to acquire training data. The scheduling node serves as the central scheduler of the scheduling plane in the distributed machine learning training system architecture with adaptive topology scheduling and assisted routing acceleration, and is equipped with a network awareness module, a policy formulation module, and a policy consistency module. Multiple training nodes are the intelligent computing nodes responsible for implementing the data plane of the distributed machine learning training system architecture with adaptive topology scheduling and assisted routing acceleration, and are equipped with a network measurement module, a policy execution module, and a database or file system for storing the training dataset.

[0161] In this embodiment of the invention, experimental data analysis of the above system is provided:

[0162] like Figure 7The experimental data shown indicates that the overall system architecture provided by this invention can achieve a wide-area distributed acceleration effect of 7.5 to 9.2 times that of MXNET, and is suitable for wide-area distributed machine learning systems with dynamic and heterogeneous resources.

[0163] like Figure 8 The experimental data shown indicates that the fastest aggregation topology technology for multi-root servers provided by this invention brings a system performance gain of 4.2 to 6.7 times. Figure 9 The experimental data shown demonstrates that data throughput increases linearly with the number of root servers, and having more root servers ensures a more even distribution of network traffic throughout the network. The fastest aggregation topology technology using multiple root servers in this invention can significantly accelerate model data synchronization in distributed machine learning.

[0164] like Figure 10 The experimental data shown demonstrates that distributed machine learning multipath assisted routing technology improves parallelism and distributes traffic by activating auxiliary paths in advance, thereby significantly enhancing transmission parallelism and alleviating congestion on the main path, resulting in a gain of 30% to 65%.

[0165] like Figure 11 The experimental data shown demonstrates that network awareness enables our system to dynamically adjust its topology based on network changes, ensuring that traffic path decisions are always optimal. Enabling network awareness can reduce average runtime by 24%. With network awareness enabled, the system better handles network dynamics and adapts to network fluctuations. This adaptability is crucial for robust and efficient distributed training across wide area networks.

[0166] like Figure 12 The experimental data shown demonstrates that the system and method proposed in this invention significantly slow down the growth rate of runtime, exhibiting near-linear scalability with respect to model size or computing node size. This gives the invention a performance advantage in training large models or large-scale wide-area distributed machine learning systems.

[0167] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0168] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A distributed machine learning training system based on adaptive topology and assisted routing, characterized in that, include: The user plane is used to provide custom strategies for building topologies; The scheduling plane includes a network awareness module, a policy formulation module, and a policy consistency module. The network awareness module is used to perceive network parameters. The policy formulation module is used to construct the fastest aggregation topology of multiple root servers based on the perceived network parameters and user-provided custom policies. The policy consistency module is used to publish the constructed fastest aggregation topology of multiple root servers as a policy for model transmission paths to the data plane based on topology consistency protocol and auxiliary routing consistency protocol. The data plane includes a policy execution module and a network measurement module. The policy execution module is used to execute distributed policies across multiple intelligent computing nodes in the network, while the network measurement module is used to acquire network status and report data in real time as the data basis for the network sensing module to perceive network parameters. The method by which the strategy formulation module constructs the fastest aggregation topology for multiple root servers is as follows: S1. Initialize the undirected graph Initialize the number of trees Initialize the data throughput between two adjacent nodes. ;in, V Let be the set of vertices of an undirected graph. E Let be the set of edges of an undirected graph, and let the weights of the edges be... w It is a positive integer; S2. Perform non-overlapping path search to obtain the root set R and candidate paths P; S3, For each root Initialize with node vi Tree with roots ; S4. For each non-root node Choose the first path from candidate path P. As the fastest aggregation path; among them... ; S5, Reset , for Adjacent nodes; in, vl As the root node vi, vr yes vl The next node; S6, let the node , Path traversal Find all nodes involved in path p to establish parent-child relationships. , ; S7. Obtain the fastest aggregation topology for the R root server. ; In step S2, the method for performing non-overlapping path search is as follows: S21. Initialize the undirected graph Initialize the number of trees Initialize the data throughput between two adjacent nodes. Initialize the root set and candidate paths ; S22, Reset edge weights ,in and On the side Adjacent in the middle; among them, express The throughput of the corresponding edge, express Cost of the corresponding edge; S23. Based on the reset edge weights, perform an R-root FAPT joint tree cover to obtain the root set. and node path H; S24. For any two reachable nodes and Add candidate paths Remove used edges ;in, ; S25. Determine if there are any unused edges. ; If so, return to step S22; If not, proceed to step S26; S26. Obtain the root set R and candidate paths. ; In step S23, the method for performing the R-root FAPT joint tree cover is as follows: S23-1. Initialize the undirected graph Initialize the weights of the positive edges Initialize the root set Initialize the number of trees ; S23-2, For each node As the root, run Dijkstra's algorithm, starting from the root. To other nodes Acquisition cost and node path ; S23-3, For nodes Add node path ; S23-4, Calculate the nodes Tree with roots Cost : S24-5, Determining the root set Is it valid? If so, proceed to step S24-6; If not, proceed to step S24-7; S24-6, will The nodes in the data are based on their cost. , Sort, select N Find the node with the minimum cost to obtain the root set. ; S24-7, Node-based , thus obtaining the root set R and its corresponding node path H .

2. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, The network awareness module senses network parameters by using lightweight, real-time link throughput measurement and hierarchical clock synchronization techniques.

3. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, The topology consistency protocol is used to complete the adaptive scheduling of the fastest aggregation topology of multiple root servers. It includes the topology update request strategy and scheduling before communication, as well as the transmission and aggregation of cached model data and local topology updates. The auxiliary routing consistency protocol is used to assist the main path in data transmission based on multipath auxiliary path technology and idle paths. It updates the routing policy according to the construction process of the fastest aggregation topology of multiple root servers and publishes the updated routing policy to the intelligent computing nodes. The message structure of the nodes on the auxiliary path recorded in the routing policy includes sender identifier, receiver identifier, auxiliary route identifier, route identifier, other identifiers and transmission data.

4. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, To construct the fastest aggregation topology for multiple root servers, a quality score is given based on network parameters perceived by the network awareness module, and the topology structure is adjusted to achieve load balancing among the multiple root servers.

5. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, When constructing a multi-root server fastest aggregation topology, multi-path assisted routing technology is used for routing policy scheduling. The routing policy scheduling method is as follows: When the communication scheduler receives a data tensor while managing the message queue, the data tensor is divided into small blocks. The communication scheduler then polls the main path and auxiliary paths in turn and allocates the new blocks to their message queues. During the polling process, the primary path has a higher priority than the secondary path, and the secondary paths are polled in order of priority, with the priority of the secondary path being inversely proportional to its cost. When the primary path is busy, new blocks are allocated to secondary paths in a round-robin fashion. Each secondary path has a message queue, and secondary paths with idle message queues are allocated to transmit new blocks. If all secondary paths are busy, the primary path is allocated to transmit new blocks.

6. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, When implementing the deployment environment of the system, it is deployed in multiple intelligent computing center clusters across multiple domains, and the intelligent computing center clusters are interconnected through a wide area network. The multiple intelligent computing center clusters include an intelligent computing scheduling center that serves as a scheduling plane and multiple intelligent computing training centers that serve as data planes. The intelligent computing scheduling center and the intelligent computing training center communicate bidirectionally, and multiple intelligent computing training centers are connected through a wide area network infrastructure.

7. The distributed machine learning training system based on adaptive topology and assisted routing according to claim 1, characterized in that, When implementing the deployment environment of the system, it is deployed in a smart computing center cluster, and the smart computing centers are interconnected through a wide area network; The intelligent computing center contains at least two physical nodes, including a scheduling node that serves as the scheduling plane and other training nodes that serve as the data plane. The scheduling node and the training node communicate bidirectionally, and the training nodes are interconnected within the local area network.

Images