An in-network multi-hop gradient aggregation method for distributed model training

Abstract

The application discloses an in-network multi-hop gradient aggregation method for distributed model training, which comprises the following steps: a controller calculates an in-network multi-hop gradient aggregation structure of the distributed model training according to a network state, and marks gradient fragments sent by each training node according to the calculated in-network multi-hop gradient aggregation structure; and the programmable switch can flexibly analyze data packets, so that when the gradient fragments pass through the programmable switch, the multiple programmable switches cooperate to perform multiple aggregations and forwarding of the gradient fragments in the data center network, thereby greatly reducing network traffic generated by model gradient communication, increasing the throughput of the distributed model training, and accelerating the model training.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, and specifically to an intranet multi-hop gradient aggregation method for distributed model training. Background Technology

[0002] With the development of big data technology and the surge in enterprise application demands, the training data for neural network models is increasing, and the complexity of the models is also rising. The storage and computing resources of a single machine are no longer sufficient to support the ever-growing volume of training data and the increasingly complex training models. To accelerate model training, distributed model training has emerged. Distributed model training is typically deployed in data center networks, utilizing the computing and storage resources of multiple computing nodes within the network for parallel training, thereby accelerating the model training process. The parameter server architecture is a popular distributed model training architecture. This architecture consists of training nodes and model aggregation server nodes. In the parameter server architecture, each training node trains a local model based on its local training data and sends the gradients of the trained local model to the model aggregation server node for aggregation. The model aggregation server node aggregates the model gradients from different training nodes and sends the global aggregation result of the gradients to each training node, allowing each training node to update its local model using the global aggregation result before proceeding to the next round of iterative training. However, as the amount of training data and model complexity increase in distributed model training, the amount of gradient communication required for each training iteration also increases. For example, the Megatron-Turing natural language processing model jointly developed by Microsoft and Nvidia requires each training node to exchange 530 billion gradient data points in each training iteration. Numerous existing studies have shown that the increased gradient communication volume leads to reduced efficiency in distributed model training, and network communication overhead has become a major bottleneck in model training.

[0003] In-network computing offers a novel opportunity to address network communication bottlenecks in distributed model training. By implementing user-defined packet processing and forwarding logic on programmable switches (PLS), in-network computing enables online computation during network transmission, reducing communication latency and improving overall computational efficiency. Based on this concept, recent research utilizes in-network aggregation technology to transfer the gradient aggregation process in distributed model training from the model aggregation server node to the PLS node. Specifically, each training node sends its locally trained model gradients to the network. When these gradients pass through the PLS node, the PLS node aggregates the gradients from multiple training nodes and forwards the aggregated gradients to the model aggregation server node. The model aggregation server node then performs global gradient aggregation. In this way, in-network aggregation technology reduces the communication overhead of gradient interactions during distributed model training, increases training throughput, and accelerates model training.

[0004] In recent years, scholars have proposed a series of on-network aggregation solutions for distributed model training. Representative solutions include SwitchML, ATP, Panama, and GRID. SwitchML designs a pool-based streaming aggregation protocol between the rack top switch and each training node within a single rack; ATP fixes the gradient aggregation location at the rack top switch connecting each training node and the rack top switch connecting the model aggregation server node, providing an on-network aggregation solution for multi-tenant and cross-rack scenarios; Panama customizes a NetFPGA-based accelerator to distribute the gradient aggregation task evenly across multiple aggregation trees in a round-robin manner, improving the efficiency of model gradient aggregation; GRID uses mathematical modeling and optimization methods to calculate gradient routing and aggregation strategies, improving the throughput of model training.

[0005] However, the main problem with existing on-network gradient aggregation solutions is that they do not support the flexible selection of multiple programmable switches within the data center network to perform multiple aggregations of model gradients within the network. This limits further improvements in network throughput and incurs significant network communication overhead. For example, SwitchML and ATP can only aggregate gradients through a few fixed-location rack-top switches, and cannot flexibly select higher-level switches within the data center network (such as aggregation switches and core switches in a fat-tree network topology, or spine switches in a leaf-spine network topology) for gradient aggregation. Therefore, when the packet processing capacity or available memory resources of the rack-top switches are limited, it will affect the gradient aggregation efficiency, resulting in longer model gradient communication times. Panama and GRID only allow model gradients to aggregate at most once within the data center network, so their effect on reducing gradient traffic is limited, and the network communication overhead caused by distributed model training within the data center network remains significant. Summary of the Invention

[0006] In view of this, the purpose of this invention is to overcome the shortcomings of existing on-network convergence schemes and provide an intra-network multi-hop gradient convergence method for distributed model training, which can realize the convergence of model gradients on multiple programmable switch nodes in a data center network, thereby accelerating model training and reducing network communication load overhead.

[0007] To achieve the above-mentioned objectives, this invention proposes an intra-network multi-hop gradient aggregation method for distributed model training. The distributed model training includes a controller, at least one local training node, at least one programmable switch node, and at least one model aggregation server node. The controller is responsible for calculating the intra-network multi-hop gradient aggregation structure of the model during distributed model training initialization. The training nodes are responsible for training the model locally during each training iteration and sending the trained model gradient stream to the model aggregation server node via the programmable switch node. The programmable switch node is responsible for aggregating and forwarding model gradients. The model aggregation server node is responsible for globally aggregating the model gradients from all training nodes and sending the global aggregation result stream of the model gradients to each training node.

[0008] The distributed model training specifically includes the following steps:

[0009] Step S1: The controller calculates the intra-network multi-hop gradient convergence structure based on the network status.

[0010] The controller calculates the intra-network multi-hop gradient aggregation structure corresponding to the distributed model training based on the collected network resource status, and sends the aggregation structure to the model aggregation server node and each training node. The intra-network multi-hop gradient aggregation structure includes the following two aspects:

[0011] Firstly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node, and the aggregation metadata of each node within the node set, wherein the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node, and the aggregation metadata of each node within the node set, are ordered according to the order of the nodes along the gradient flow path from that training node to the model aggregation server node; the aggregation metadata of each node within the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node includes whether the gradient flow should be aggregated at that node, the number of training nodes that should be aggregated at that node, and the outgoing port number of the gradient flow forwarded at that node;

[0012] Secondly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to each training node, and the convergence metadata of each node in the node set, wherein the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node, and the convergence metadata of each node in the node set, are ordered according to the order of the nodes along the path of the gradient global convergence result stream from the model convergence server node to that training node; the convergence metadata of each node in the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node includes whether a gradient stream converged at that node and the outgoing port number of the gradient global convergence result stream forwarded at that node;

[0013] In the intranet multi-hop gradient convergence structure in step S1, the set of programmable switch nodes along the path of the gradient flow from any training node to the model convergence server node is completely consistent with the set of programmable switch nodes along the path of the gradient global convergence result flow from the model convergence server node to the training node, but the order of the nodes in the set is reversed.

[0014] In step S1, after each training node and model aggregation server node receives the intra-network multi-hop gradient aggregation structure issued by the controller, each training node and model aggregation server node saves the intra-network multi-hop gradient aggregation structure in memory.

[0015] Step S2: Each training node marks gradient slices according to the corresponding intranet multi-hop gradient convergence structure.

[0016] Based on the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node in the multi-hop gradient aggregation structure within the network issued by the controller, as well as the aggregation metadata of each node in the node set, each training node marks the trained gradient fragments and sets the destination address of the gradient fragments to the model aggregation server node. Then, each training node sends the gradient fragments to the network.

[0017] Step S3: Programmable switch node processes gradient sharding

[0018] Step S3.1: The programmable switch node determines whether gradient sharding should converge.

[0019] When a gradient fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient fragment is used to determine whether the gradient fragment should be aggregated on the programmable switch node. If it should not be aggregated, step S3.2 is executed; otherwise, step S3.3 is executed.

[0020] Step S3.2: Programmable switch nodes forward gradient fragments that should not be aggregated.

[0021] If the gradient fragment should not be aggregated on the programmable switch node, the programmable switch node records the forwarding port number in the aggregation meta-information of the first node in the node set of the gradient fragment tag, then deletes the first node in the node set of the gradient fragment tag and its aggregation meta-information, and forwards the gradient fragment to the next-hop node in the network according to the recorded forwarding port number, and executes step S3.6.

[0022] Step S3.3: The programmable switch node should aggregate the gradient fragments that should be aggregated.

[0023] If the gradient slice should be converged on the programmable switch node, the programmable switch node allocates the gradient slice to a specific memory unit for convergence, increments the number of converged training nodes recorded in the memory unit by 1, and executes step S3.4.

[0024] Step S3.4: The programmable switch node determines whether gradient sharding has been completed and converged.

[0025] After the gradient fragment aggregation is completed, the programmable switch node compares the number of training nodes that have been aggregated in the corresponding memory unit with the number of training nodes that should be aggregated on the programmable switch node in the aggregation metadata of the first node in the node set in the gradient fragment marker. If the two are equal, it means that the programmable switch has completed the aggregation of the corresponding gradient fragments of the training nodes that should be aggregated on it, and then executes step S3.5. If the two are not equal, execute step S3.7.

[0026] Step S3.5: Programmable switch node forwarding gradient fragmentation aggregation results

[0027] The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient fragment marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient fragment marker, writes the gradient aggregation result in the corresponding memory unit into the gradient fragment, and forwards the gradient fragment carrying the gradient aggregation result to the next hop node in the network according to the recorded forwarding port number. Then, it proceeds to step S3.6.

[0028] Step S3.6: Is the next-hop node in the network a programmable switch node?

[0029] If the next-hop node is a programmable switch node, repeat step S3; if the next-hop node is a model aggregation server node, execute step S4.

[0030] Step S3.7: The programmable switch node discards gradient fragments;

[0031] Step S4: The model aggregation server node processes gradient fragments.

[0032] Step S4.1: The model aggregation server node performs gradient global aggregation and generates gradient global aggregation result fragments.

[0033] When a gradient shard reaches the model aggregation server node, the model aggregation server node performs global gradient aggregation. After all gradient shards corresponding to all training nodes have been aggregated, the model aggregation server node generates N gradient global aggregation result shards. Each gradient global aggregation result shard carries the same gradient global aggregation result, where N is equal to the number of training nodes in the distributed model training.

[0034] Step S4.2: Mark gradient global aggregation result fragments on the model aggregation server node.

[0035] The model aggregation server node sets the destination addresses of the N gradient global aggregation result fragments to the respective training nodes in sequence. Then, based on the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to each training node and the aggregation metadata of each node in the node set, stored in memory in step S1, the model aggregation server node marks the N gradient global aggregation result fragments and sends them to the network. Each gradient global aggregation result fragment marks the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to a certain training node and the aggregation metadata of each node in the node set.

[0036] Step S5: The programmable switch node processes the gradient global aggregation result.

[0037] Step S5.1: The programmable switch node determines whether any gradient sharding has converged on it.

[0038] When a gradient global aggregation result fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient global aggregation result fragment is used to determine whether a gradient fragment has been aggregated on the programmable switch node. If a gradient fragment has been aggregated on the programmable switch node, it means that the gradient aggregation result in the specific memory unit corresponding to the gradient fragment needs to be cleared so that it can be reused to aggregate gradients. Then, step S5.2 is executed; otherwise, step S5.3 is executed.

[0039] Step S5.2: Clear the gradient aggregation results of specific memory cells in the programmable switch node.

[0040] If gradient fragments converge on the programmable switch node, the programmable switch node clears the gradient convergence result of the specific memory cell and then executes step S5.3.

[0041] Step S5.3: Fragmentation of global aggregation results of programmable switch node forwarding gradient.

[0042] The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient global aggregation result fragmentation marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient global aggregation result fragmentation marker, and forwards the gradient global aggregation result fragment to the next hop node in the network according to the recorded forwarding port number, and executes step S5.4.

[0043] Step S5.4: Is the next-hop node in the network a training node?

[0044] If the next-hop node is a training node, then execute step S6; if the next-hop node is a programmable switch node, then repeat step S5.

[0045] Step S6: The training node receives the global gradient convergence result and updates the local model.

[0046] When a gradient global convergence result slice arrives at the training node, the training node uses the gradient global convergence result slice to update its local model, so that the training node can use the updated local model to perform the next round of iteration training.

[0047] The objective of this invention is achieved as follows:

[0048] This invention discloses an intra-network multi-hop gradient aggregation method for distributed model training. The method involves a controller calculating the intra-network multi-hop gradient aggregation structure for distributed model training. Leveraging the ability of programmable switches (PLS) to flexibly parse data packets, when a gradient fragment passes through a PPS, the method determines whether the fragment should be aggregated on that PPS based on the intra-network multi-hop gradient aggregation structure in the fragment's tag. If so, gradient aggregation is performed; otherwise, the PPS node directly forwards the fragment. Once the PPS has aggregated all the gradient fragments that should be aggregated on it, it forwards the gradient fragment carrying the aggregation result to the next-hop PPS node in the network. The next-hop PPS node repeats the above judgment and aggregation process until some of the aggregated gradient fragments are forwarded to the model aggregation server node, which then performs the aggregation. Global aggregation of gradient shards is achieved through collaborative aggregation and forwarding of gradient shards across programmable switches in the data center network. This forms a multi-hop tree-like aggregation structure for model gradients within the network. Each aggregation switch in this structure can perform partial aggregation of model gradients, significantly reducing network traffic generated by gradient communication. Furthermore, since the programmable switches have reduced gradient traffic considerably, the number of gradient shards ultimately reaching the model aggregation server node is decreased. This alleviates the network bandwidth bottleneck at the model aggregation server node during distributed model training, reducing the probability of network congestion. Consequently, this further promotes an increase in the gradient transmission rate of training nodes in distributed model training, thereby increasing the throughput of distributed model training and accelerating model training. Attached Figure Description

[0049] Figure 1 This is a diagram of the logical architecture proposed in this invention;

[0050] Figure 2 This is a flowchart of the intra-network multi-hop gradient convergence method for distributed model training proposed in this invention.

[0051] Figure 3 This is a specific example diagram of a distributed model training network topology proposed in this invention.

[0052] Figure 4(a) shows a specific example of a model gradient aggregation structure in the prior art that restricts the model gradient to converge at most once in the network.

[0053] Figure 4(b) shows a specific example of the model gradient convergence structure calculated by the controller proposed in this invention based on the network state.

[0054] Figure 5 This is a specific example diagram of the gradient sharding data packet structure for training nodes in the distributed model training proposed in this invention;

[0055] Figure 6 This is a specific example diagram of the fragmented data packet structure of the global aggregation result of the gradient of the distributed model training model aggregation server node proposed in this invention; Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and embodiments, so that those skilled in the art can better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0057] This invention provides an intranet multi-hop gradient aggregation method for distributed model training, named HINA. Its basic idea is to improve the model training throughput and accelerate distributed model training by having the model gradients aggregated multiple times on multiple programmable switch nodes within the data center network, while reducing network communication overhead.

[0058] Figure 1 This is a logical architecture diagram of the present invention, such as... Figure 1 As shown, the distributed model training is deployed in a data center network and includes a controller, at least one local training node, at least one programmable switch node, and at least one model aggregation server node. All training nodes form a training node pool; all model aggregation server nodes form a model aggregation server node pool; all programmable switch nodes within the data center form a programmable switch node pool. Within the programmable switch node pool, all programmable switches are divided into multiple layers according to the data center network topology, with each layer consisting of at least one programmable switch. The controller is responsible for collecting network resource status during the initialization of the distributed model training task, calculating the intra-network multi-hop gradient aggregation structure based on the collected network status, and distributing the calculated intra-network multi-hop gradient aggregation structure to each training node and model aggregation server node in the distributed model training. Training nodes are responsible for performing local model training and sending the trained gradient fragments to the model aggregation server node via multiple programmable switch nodes. Each layer of programmable switch nodes is responsible for partially aggregating and forwarding gradient fragments to the next layer of programmable switch nodes or up to the model aggregation server node. Gradient fragments undergo multiple aggregations on the multi-layered programmable switch nodes within the data center, thus forming the intra-network multi-hop gradient aggregation structure. Simultaneously, programmable switch nodes are also responsible for forwarding the global gradient aggregation result fragments to each training node. The model aggregation server node is responsible for performing global aggregation on the received partially aggregated gradient fragments and sending the global gradient aggregation result fragments to each training node via multiple programmable switch nodes.

[0059] Figure 2The flowchart illustrates the intranet multi-hop gradient convergence method for distributed model training provided by this invention.

[0060] In this embodiment, as Figure 2 As shown, the intra-network multi-hop gradient convergence method for distributed model training according to the present invention includes the following steps:

[0061] Step S1: The controller calculates the intra-network multi-hop gradient convergence structure based on the network status.

[0062] The controller calculates the intranet multi-hop gradient aggregation structure corresponding to the distributed model training based on the collected network resource status, and sends the aggregation structure to the model aggregation server node and each training node.

[0063] In this embodiment, the distributed model training network topology is as follows: Figure 3 As shown, a distributed model training task is deployed in a data center network. The data center network architecture adopts a fat-tree network topology. The distributed model training task includes 6 training nodes (w1-w6) identified by squares and a model aggregation server node identified by a circle. Figure 3 The nodes are labeled PS. Both the training node and the model aggregation server node PS reside on physical host servers within the data center network. This data center network contains seven programmable switch nodes (S1-S7) identified by rounded rectangles. Each programmable switch has a model gradient packet aggregation capacity of 6Gbps. The black connecting lines represent links in the data center network, each with a capacity of 6Gbps. The letters (L1-L7) next to the link indicate its name. For ease of reading and explanation, Figure 3 Components within the data center network that are not related to distributed model training are omitted, and only components that are relevant to it are displayed.

[0064] The existing model gradient aggregation structure, which restricts the model gradient to converge at most once in the network, is shown in Figure 4(a). The model gradient aggregation structure calculated by the controller based on the network state in this invention is shown in Figure 4(b). The descriptions of the components in Figures 4(a) and 4(b) are as follows: Figure 3 To maintain consistency, the following section addresses the parts contained in Figures 4(a) and 4(b). Figure 3For components not explicitly described, in Figures 4(a) and 4(b), training nodes w1-w6 send gradient streams composed of gradient fragments to the model aggregation server node PS via programmable switch nodes. Gray rounded rectangles represent programmable switches that perform gradient fragment aggregation and forwarding, while white rounded rectangles represent programmable switches that only perform gradient fragment forwarding without aggregation. In Figure 4(a), the 6 / 6 in the upper right corner of the aggregation switch S1 indicates that the programmable switch's model gradient packet aggregation capacity is 6Gbps. Since gradient fragment aggregation consumes 6Gbps of model gradient packet aggregation capacity (as shown in other aggregation nodes in Figures 4(a) and 4(b)...), The numbers in the upper right corner of the switch are similar. In Figure 4(a), the 4 / 6 next to the L2 link represents the L2 link capacity of 6Gbps. Since the link load consumed by gradient fragmentation forwarding is 4Gbps (the numbers next to other links in Figure 4(a) and Figure 4(b) are similar), the number 2 below each training node in Figure 4(a) represents the maximum transmission rate of the training node side that can be achieved by the prior art of 2Gbps. The number 3 below each training node in Figure 4(b) represents the maximum transmission rate of the training node side that can be achieved by the present invention of 3Gbps. In Figure 4(a) and Figure 4(b), f1-f6 respectively represent the gradient stream composed of gradient fragments sent by w1-w6. The representative is the training node w i The gradient stream f composed of the sent gradient slices i With training node w j The gradient stream f composed of the sent gradient slices j At programmable switch node S k The gradient flow formed by the partially converged gradient fragments generated by the upward convergence. (For example, in Figure 4(b)) This indicates that gradient fragment f4 sent by w4 and gradient fragment f5 sent by w5 converged on switch S3, forming a gradient fragment. The representative is the training node w i -w j The transmitted gradient fragment fi-fj is sent to the programmable switch node S. k Partial convergence gradient sharding generated by upper convergence (For example, in Figure 4(a) This indicates that gradient fragment f4 sent by w4, gradient fragment f5 sent by w5, and gradient fragment f6 sent by w6 converge on switch S6, forming a partially converged gradient fragment. All gradient flows composed of all gradient pieces are represented by a solid black line with arrows.

[0065] To illustrate the differences between this invention and other solutions, and to explain the advantages of this invention, a comparative description is presented of the two gradient convergence structures shown in Figures 4(a) and 4(b).

[0066] Figure 4(a) shows the model gradient aggregation structure generated by the existing GRID technology. As shown in Figure 4(a), GRID restricts the model gradient to be aggregated at most once within the data center network. Specifically, S1 aggregates the gradient stream f1-f3 sent by w1-w3, and then S1 aggregates the aggregated stream... The gradient packets are sent to the model aggregation server node PS, where global aggregation is performed. Since the aggregation capability of model gradient packets in S1 is 6Gbps, gradient fragments w1-w3 can be sent at a speed of 6 / 3 = 2Gbps. S6 aggregates the gradient stream f4-f6 sent by w4-w6, and then S6 merges the aggregated stream. The data is sent to the model aggregation server node PS, where global aggregation is performed. Since the model gradient packet aggregation capability of S6 is 6Gbps, w4-w6 can also send gradient fragments at a speed of 6 / 3 = 2Gbps. At this time, the remaining available bandwidth of link L1 between S5 and S1 is 6 - 2 * 2 - 2 = 0Gbps (the entire link bandwidth is used by...). (f2 and f3 are occupied). In summary, the maximum transmission rate on the training node side of GRID is 2Gbps.

[0067] Figure 4(b) shows the model gradient aggregation structure generated by HINA of the present invention. As shown in Figure 4(b), HINA allows model gradients to arbitrarily select programmable switch nodes within the data center network to perform gradient aggregation, and does not limit the number of times model gradients aggregate in the network. Among them, w4 and w5 can send f4 and f5 to S3 at a speed of 6 / 2 = 3Gbps respectively. S3 aggregates f4 and f5 to form After that, S3 will Forwarded to S6 at 3Gbps; w6 can send f6 to S6 at 3Gbps along the w6-S4-S6 path; S6 receives Together with f6, they converge to form And Forwarded at 3Gbps along path S6-S7-S5 to S5; w2 and w3 can send f2 and f3 to S2 respectively at 3Gbps, and S2 converges f2 and f3 to form After that, S2 will Forwarded to S5 at 3Gbps speed; S5 aggregates received data. and Forming convergence flow Then S5 will reach a speed of 3Gbps Forwarded to S1; w1 sends f1 to S1 at 3Gbps; S1 aggregates the received data. Together with f1, they form a converging flow. S1 will then By forwarding data at 3Gbps to the model aggregation server node PS, HINA achieves a maximum transmission rate of 3Gbps on the training node side, a 50% improvement compared to GRID. As shown in Figure 4(b) of the gradient multi-hop aggregation process, this also further reduces the ingress bandwidth on the model aggregation server node PS: the ingress bandwidth consumption of the model aggregation server node PS in Figure 4(b) is 3Gbps, a 25% reduction compared to GRID. In summary, by implementing model gradient multi-hop aggregation, HINA can further improve the maximum transmission rate of each training node, increase the throughput of distributed model training, and thus shorten gradient communication time.

[0068] The aforementioned intranet multi-hop gradient convergence structure includes the following two aspects:

[0069] Firstly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node, and the aggregation metadata of each node within the node set, wherein the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node, and the aggregation metadata of each node within the node set, are ordered according to the order of the nodes along the gradient flow path from that training node to the model aggregation server node; the aggregation metadata of each node within the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node includes whether the gradient flow should be aggregated at that node, the number of training nodes that should be aggregated at that node, and the outgoing port number of the gradient flow forwarded at that node;

[0070] As shown in Figure 4(b), it can be seen that the set of programmable switch nodes along the path of the gradient flow from each training node to the model aggregation server node in the distributed model training system, as calculated by the controller, and the aggregation metadata of each node in the node set are as follows:

[0071] For ease of explanation and understanding, the specific port numbers of the programmable switch forwarding the gradient flow are omitted in this example. Instead, the name of the link to which the programmable switch is connected is used to represent the corresponding forwarding port number along the link from which the gradient flow is forwarded. Specifically, as shown in Figure 4(b), for the training node w1, the gradient flow sent to the model aggregation server node PS is f1 and the aggregated gradient flow. The gradient flow path only passes through S1, so the intranet multi-hop gradient aggregation structure for w1 is {w1: [S1: {Agg: 1, Cnt: 6, Port: L7}]}. In the intranet multi-hop gradient aggregation structure for w1 {w1: [S1: {Agg: 1, Cnt: 6, Port: L7}]}, w1 represents that the aggregation structure is related to w1, {w1: [S1]} represents that the gradient flow passes through S1, S1: {Agg: 1, Cnt: 6, Port: L7} represents the aggregation metadata related to the S1 node, Agg: 1 means that the gradient slice sent by w1 should be aggregated on S1, Cnt: 6 means that the number of training nodes on S1 that should be aggregated is 6, and Port: L7 means that the gradient slice sent by w1 is forwarded to the model aggregation server node PS on S1 along the port connected to L7.

[0072] For training node w2, the gradient stream sent to model aggregation server node PS is f2, and The gradient flow path passes through S2, S5, and S1. Therefore, the intranet multi-hop gradient convergence structure for w2 is {w2: [S2: {Agg: 1, Cnt: 2, Port: L2}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]}, where w2 represents the convergence structure for w2, {w2: [S2, S5, S1]} represents the gradient flow sequence passing through S2, S5, and S1, and S2: {Agg: 1, Cnt: 2, Port: L2} represents the gradient slices sent by w2 that should converge on S2. The number of training nodes that should converge gradients on S2 is 2 (w2 and w3 in Figure 4(b)), and the gradient slices sent by w2 converge along S2. The gradient fragments sent by w2 are forwarded to S5 via the port connected to L2; S5: {Agg: 1, Cnt: 5, Port: L1} means that the gradient fragments sent by w2 should be converged on S5, and the number of training nodes on S5 that should converge gradients is 5 (w2, w3, w4, w5 and w6 in Figure 4(b)). The gradient fragments sent by w2 are forwarded to S1 via the port connected to L1 on S5; S1: {Agg: 1, Cnt: 6, Port: L7} means that the gradient fragments sent by w2 should be converged on S1, and the number of training nodes on S1 that should converge gradients is 6 (w1, w2, w3, w4, w5 and w6 in Figure 4(b)). The gradient fragments sent by w2 are forwarded to the model convergence server node PS via the port connected to L7 on S1.

[0073] For training node w4, the gradient stream sent to the model aggregation server node is f4. and The gradient flow path passes through S3, S6, S7, S5, and S1. Therefore, the intranet multi-hop gradient convergence structure with respect to w4 is {w4: [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]}, where w4 represents the convergence structure with respect to w4, {w4: [S3, S6, ... S7, S5, S1]} represents the gradient flow sequentially passing through S3, S6, S7, S5 and S1. S3: {Agg: 1, Cnt: 2, Port: L3} means that the gradient fragment sent by w4 should converge on S3, and the number of training nodes on S3 that should converge gradients is 2 (w4 and w5 in Figure 4(b)). The gradient fragment sent by w4 is forwarded to S6 on S3 along the port connected to L3. S6: {Agg: 1, Cnt: 3, Port: L6} means that the gradient fragment sent by w4 should converge on S6, and gradient convergence should occur on S6. The number of training nodes is 3 (w4, w5, and w6 in Figure 4(b)), and the gradient fragments sent by w4 are forwarded to S7 on S6 along the port connected to L6; S7: {Agg: 0, Cnt: 0, Port: L5} means that the gradient fragments sent by w4 should not converge on S7, and the number of training nodes on S7 where gradient convergence should occur is 0, and the gradient fragments sent by w4 are forwarded to S5 on S7 along the port connected to L5; S5: {Agg: 1, Cnt: 5, Port: L1} means that the gradient fragments sent by w4 should converge on S5, and S5 The number of training nodes where gradient convergence should occur is 5 (w2, w3, w4, w5 and w6 in Figure 4(b)), and the gradient fragment sent by w4 is forwarded to S1 on S5 along the port connected to L1; S1: {Agg: 1, Cnt: 6, Port: L7} means that the gradient fragment sent by w4 should converge on S1, and the number of training nodes where gradient convergence should occur on S1 is 6 (w1, w2, w3, w4, w5 and w6 in Figure 4(b)), and the gradient fragment sent by w4 is forwarded to the model convergence server node PS on S1 along the port connected to L7.

[0074] The gradient aggregation structure of the model gradient slices sent by w3, w5, and w6 follows a similar structure to that of w1, w2, and w4. Finally, the intra-network multi-hop gradient aggregation structure sent from each training node to the model aggregation server node PS is as follows:

[0075] {w1:[S1:{Agg:1,Cnt:6,Port:L7}]};

[0076] {w2: [S2: {Agg: 1, Cnt: 2, Port: L2}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]};

[0077] {w3: [S2: {Agg: 1, Cnt: 2, Port: L2}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]};

[0078] {w4: [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]};

[0079] {w5: [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]};

[0080] {w6: [S4: {Agg: 0, Cnt: 0, Port: L4}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]};

[0081] The controller sends the corresponding intra-network multi-hop gradient pooling structure of each training node to each training node. Each training node saves the corresponding intra-network multi-hop gradient pooling structure in memory for subsequent labeling of model gradient slices.

[0082] Secondly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to each training node, and the convergence metadata of each node in the node set, wherein the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node, and the convergence metadata of each node in the node set, are ordered according to the order of the nodes along the path of the gradient global convergence result stream from the model convergence server node to that training node; the convergence metadata of each node in the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node includes whether a gradient stream converged at that node and the outgoing port number of the gradient global convergence result stream forwarded at that node;

[0083] In the intranet multi-hop gradient convergence structure in step S1, the set of programmable switch nodes along the path of the gradient flow from any training node to the model convergence server node is completely consistent with the set of programmable switch nodes along the path of the gradient global convergence result flow from the model convergence server node to the training node, but the order of the nodes in the set is reversed.

[0084] In step S1, after each training node and model aggregation server node receives the intra-network multi-hop gradient aggregation structure issued by the controller, each training node and model aggregation server node saves the intra-network multi-hop gradient aggregation structure in memory.

[0085] As shown in Figure 4(b), it can be seen that the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to each training node in the distributed model training system, calculated by the controller, and the aggregation metadata of each node in the node set are as follows:

[0086] For ease of explanation and understanding, the specific port numbers of the programmable switch forwarding the global gradient aggregation result stream are omitted in this example. Instead, the link name or the training node name connected to the programmable switch is used to represent the forwarding port number corresponding to the global gradient aggregation result stream. Specifically, as shown in Figure 4(b), for the model aggregation server node, the global gradient aggregation result stream sent to the training node w3 is completely identical to the programmable switch node set along the path from the training node w3 to the model aggregation server node. The order of the nodes in the node set is reversed. Since the intranet multi-hop gradient aggregation structure of the model gradient shard sent from the training node w3 to the model aggregation server node is {w3: [S2: {Agg: 1, Cnt: 2, Port:} Given L2}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]}, the intra-network multi-hop gradient aggregation structure of the gradient global aggregation result shard sent from the model aggregation server node to the training node w3 is {w3: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L2}, S2: {Agg: 1, Port: w3}]}, where w3 represents the intra-network multi-hop gradient aggregation structure. Regarding w3, {w3: [S1, S5, S2]} represents the gradient global convergence result fragments that passed through S1, S5, and S2 in that order (reverse from the order of the programmable switch nodes along the path of the gradient flow from w3 to the model convergence server node); S1: {Agg: 1, Port: L1} represents the gradient global convergence result fragments sent to w3 passing through S1 and the corresponding gradient fragments being converged, and the gradient global convergence result fragments sent to w3 being forwarded to S5 along the port connected to L1 on S1; S5 {Agg: 1, Port: L2} represents the gradient global aggregation result fragment sent to w3 passing through S5 and the corresponding gradient fragments being aggregated, and the gradient global aggregation result fragment sent to w3 being forwarded to S2 on S5 along the port connected to L2; S2: {Agg: 1, Port: w3} represents the gradient global aggregation result fragment sent to w3 passing through S2 and the corresponding gradient fragments being aggregated, and the gradient global aggregation result fragment sent to w3 being forwarded to w3 on S2 along the port connected to w3.

[0087] The intra-network multi-hop gradient aggregation structure for the global aggregation results of model gradients sent from the model aggregation server node to w1, w2, w4, w5, and w6 follows a similar structure to w3. Finally, the intra-network multi-hop gradient aggregation structure for the global aggregation results of model gradients sent from the model aggregation server node to each training node is as follows:

[0088] {w1:[S1:{Agg:1,Port:w1}]};

[0089] {w2: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L2}, S2: {Agg: 1, Port: w2}]};

[0090] {w3: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L2}, S2: {Agg: 1, Port: w3}]};

[0091] {w4: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L5}, S7: {Agg: 0, Port: L6}, S6: {Agg: 1, Port: L3}, S3: {Agg: 1, Port: w4}]};

[0092] {w5: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L5}, S7: {Agg: 0, Port: L6}, S6: {Agg: 1, Port: L3}, S3: {Agg: 1, Port: w5}]};

[0093] {w6: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L5}, S7: {Agg: 0, Port: L6}, S6: {Agg: 1, Port: L4}, S4: {Agg: 0, Port: w6}]};

[0094] The controller sends the calculated intra-network multi-hop gradient convergence structure to the corresponding training nodes and model convergence server nodes, and the training nodes and model convergence server nodes store the intra-network multi-hop gradient convergence structure in memory.

[0095] Step S2: Each training node marks gradient slices according to the corresponding intranet multi-hop gradient convergence structure.

[0096] Based on the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node in the multi-hop gradient aggregation structure within the network issued by the controller, as well as the aggregation metadata of each node in the node set, each training node marks the trained gradient fragments and sets the destination address of the gradient fragments to the model aggregation server node. Then, each training node sends the gradient fragments to the network.

[0097] Figure 5 The distributed model training gradient sharding data packet structure of the present invention is as follows: Figure 5As shown, this gradient sharding includes a traditional IP field, a newly added gradient sharding intra-network multi-hop gradient aggregation structure field, and an INA field. The traditional IP field functions the same as a traditional data packet in a data center; the newly added gradient sharding intra-network multi-hop gradient aggregation structure field is used to mark the intra-network multi-hop gradient aggregation structure generated in step S1. Each training iteration follows the corresponding intra-network multi-hop gradient aggregation structure stored in memory in step S1. Figure 5 The newly added gradient sharding network multi-hop gradient aggregation structure field is filled in from left to right. This field contains multiple {Agg, Cnt, Port} field groups. Each {Agg, Cnt, Port} field group represents the aggregation metadata of a programmable switch node along the path the gradient shards sequentially traverse. Agg indicates whether the gradient aggregation occurred at that programmable switch node, Cnt represents the number of training nodes that should have aggregated at that programmable switch node, and Port represents the forwarding port number of the gradient shard at that programmable switch node. The INA field guides the model gradient aggregation. Specifically, an Ack value of 0 in the INA field indicates that the data packet is a gradient shard sent from a training node to the model aggregation server node; conversely, an Ack value indicates that the data packet is a gradient global aggregation result shard sent from the model aggregation server node to the training node. The SlotId in the INA field identifies a specific memory cell within the programmable switch node. The programmable switch node uses this field to determine which specific memory cell the gradient shard should aggregate at.

[0098] Specifically, for training node w4, the intra-network multi-hop gradient aggregation structure corresponding to the gradient fragment it sends is {w4: [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]}. This structure is marked sequentially into the newly added gradient fragment intra-network multi-hop gradient aggregation structure field in the gradient fragment data packet. After marking, w4 sets the destination IP address in the traditional IP field to the model aggregation server node, sets Ack in the INA field to 0, sets the SlotId field to the identifier of the specific memory unit of the programmable switch node where the gradient fragment is aggregated, and finally sends the gradient fragment to the network. The gradient fragment marking and processing methods for other training nodes are similar to w4.

[0099] Step S3: Programmable switch node processes gradient sharding

[0100] Step S3.1: The programmable switch node determines whether gradient sharding should converge.

[0101] When a gradient fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient fragment is used to determine whether the gradient fragment should be aggregated on the programmable switch node. If it should not be aggregated, step S3.2 is executed; otherwise, step S3.3 is executed.

[0102] Step S3.2: Programmable switch nodes forward gradient fragments that should not be aggregated.

[0103] If the gradient fragment should not be aggregated on the programmable switch node, the programmable switch node records the forwarding port number in the aggregation meta-information of the first node in the node set of the gradient fragment tag, then deletes the first node in the node set of the gradient fragment tag and its aggregation meta-information, and forwards the gradient fragment to the next-hop node in the network according to the recorded forwarding port number, and executes step S3.6.

[0104] Step S3.3: The programmable switch node should aggregate the gradient fragments that should be aggregated.

[0105] If the gradient slice should be converged on the programmable switch node, the programmable switch node allocates the gradient slice to a specific memory unit for convergence, increments the number of converged training nodes recorded in the memory unit by 1, and executes step S3.4.

[0106] Step S3.4: The programmable switch node determines whether gradient sharding has been completed and converged.

[0107] After the gradient fragment aggregation is completed, the programmable switch node compares the number of training nodes that have been aggregated in the corresponding memory unit with the number of training nodes that should be aggregated on the programmable switch node in the aggregation metadata of the first node in the node set in the gradient fragment marker. If the two are equal, it means that the programmable switch has completed the aggregation of the corresponding gradient fragments of the training nodes that should be aggregated on it, and then executes step S3.5. If the two are not equal, execute step S3.7.

[0108] Step S3.5: Programmable switch node forwarding gradient fragmentation aggregation results

[0109] The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient fragment marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient fragment marker, writes the gradient aggregation result in the corresponding memory unit into the gradient fragment, and forwards the gradient fragment carrying the gradient aggregation result to the next hop node in the network according to the recorded forwarding port number. Then, it proceeds to step S3.6.

[0110] Step S3.6: Is the next-hop node in the network a programmable switch node?

[0111] If the next-hop node is a programmable switch node, repeat step S3; if the next-hop node is a model aggregation server node, execute step S4.

[0112] Step S3.7: The programmable switch node discards gradient fragments;

[0113] Specifically, as shown in Figure 4(b), the gradient fragment sent by w4 carries gradient fragment markers [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}] to S3. S3 determines that the gradient fragment should converge on itself based on the aggregation metadata {Agg: 1, Cnt: 2, Port: L3} of the first node in the node set of the gradient fragment markers, where Agg: 1. Therefore, S3 divides the gradient fragment into... The training nodes are assigned to a specific memory unit for aggregation. At this point, the number of aggregated training nodes in that memory unit is 0 + 1 = 1 (containing only w4). This is different from the number of training nodes Cnt: 2 that should be aggregated on this programmable switch node in the aggregation metadata {Agg: 1, Cnt: 2, Port: L3} of the first node in the node set in the gradient shard marker. Therefore, S3 discards this gradient shard. Subsequently, the gradient shard sent by w5 carries the gradient shard markers [S3: {Agg: 1, Cnt: 2, Port: L3}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5}. S1: {Agg: 1, Cnt: 6, Port: L7} to S3. S3 determines that the gradient shard should converge on itself based on the convergence metadata {Agg: 1, Cnt: 2, Port: L3} of the first node in the node set of the gradient shard label. Therefore, S3 allocates the gradient shard to a specific memory unit for convergence. At this time, the number of training nodes that have been converged in the memory unit is 1 + 1 = 2 (including w4 and w5), which is the same as the number of training nodes Cnt: 2 that should converge on the programmable switch node in the convergence metadata {Agg: 1, Cnt: 2, Port: L3} of the first node in the node set of the gradient shard label. Then, S3 records the forwarding port number Port: L3 in the aggregation metadata of the first node in the node set of the gradient piece label {Agg: 1, Cnt: 2, Port: L3}. Then, the first node and its aggregation metadata [S3: {Agg: 1, Cnt: 2, Port: L3}] are deleted from the node set of the gradient piece label. (The gradient piece label after deleting the first node and its aggregation metadata is [S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]).The gradient aggregation result in the corresponding memory unit is written to the gradient fragment, and according to the recorded forwarding port number Port: L3, the gradient fragment carrying part of the gradient aggregation result is forwarded to the next-hop node S6 in the network through the L3 link.

[0114] When S6 receives the converged gradient fragment, it repeats step S3. Based on the convergence metadata {Agg: 1, Cnt: 3, Port: L6} of the first node in the node set of the gradient fragment label, it is determined that the gradient fragment should converge on S6 itself. Therefore, S6 allocates the gradient fragment to a specific memory unit for convergence. At this time, the number of training nodes that have been converged in the memory unit is 2 (including w4 and w5), which is different from the number of training nodes Cnt: 3 that should converge on the programmable switch node in the convergence metadata {Agg: 1, Cnt: 3, Port: L6} of the first node in the node set of the gradient fragment label. S6 discards the gradient fragment.

[0115] When S4 receives a gradient fragment sent by w6, S4 determines that this gradient fragment should not converge on itself based on the aggregation metadata {Agg: 0, Cnt: 0, Port: L4} of the first node in the node set of the gradient fragment markers [S4: {Agg: 0, Cnt: 0, Port: L4}, S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]. Therefore, S4 records the first node in the node set of the gradient fragment markers. The forwarding port number Port: L4 in the aggregation meta-information is used. Then, the first node in the node set in the gradient fragment label and its aggregation meta-information [S4: {Agg: 0, Cnt: 0, Port: L4}] are deleted (the gradient fragment label after deleting the first node and its aggregation meta-information is [S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]). The gradient fragment is then forwarded to the next-hop node S6 in the network through the L4 link according to the recorded forwarding port number Port: L4.

[0116] When S6 receives the gradient shard, it executes step S3. Based on the aggregation metadata {Agg: 1, Cnt: 3, Port: L6} of the first node in the node set of the gradient shard marker [S6: {Agg: 1, Cnt: 3, Port: L6}, S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}, it is determined that the gradient shard should be aggregated on S6 itself. Therefore, S6 allocates the gradient shard to a specific memory unit for aggregation. At this time, the number of training nodes aggregated in the memory unit is 2 + 1 = 3 (including w4, w5, and w6). This is consistent with the aggregation metadata {Agg: 1, Cnt: 3, Port: L6} of the first node in the node set of the gradient shard marker, indicating that aggregation should occur on the programmable switch node. If the number of training nodes in the cluster is the same (Cnt: 3), then S6 records the forwarding port number Port: L6 in the aggregation meta-information {Agg: 1, Cnt: 3, Port: L6} of the first node in the node set of the gradient shard label. Then, the first node and its aggregation meta-information [S6: {Agg: 1, Cnt: 3, Port: L6}] are deleted from the node set of the gradient shard label (the gradient shard label after deleting the first node and its aggregation meta-information is [S7: {Agg: 0, Cnt: 0, Port: L5}, S5: {Agg: 1, Cnt: 5, Port: L1}, S1: {Agg: 1, Cnt: 6, Port: L7}]). The gradient aggregation result in the corresponding memory unit is written into the gradient shard, and the gradient shard carrying the gradient aggregation result is forwarded to the next hop node S7 in the network through the L6 link according to the recorded forwarding port number Port: L6. Other switches S1, S2, S5, and S7 in the data center network perform similar steps to S3, S6, and S4.

[0117] Step S4: The model aggregation server node processes gradient fragments.

[0118] Step S4.1: The model aggregation server node performs gradient global aggregation and generates gradient global aggregation result fragments.

[0119] When a gradient shard reaches the model aggregation server node, the model aggregation server node performs global gradient aggregation. After all gradient shards corresponding to all training nodes have been aggregated, the model aggregation server node generates N gradient global aggregation result shards. Each gradient global aggregation result shard carries the same gradient global aggregation result, where N is equal to the number of training nodes in the distributed model training.

[0120] Step S4.2: Mark gradient global aggregation result fragments on the model aggregation server node.

[0121] The model aggregation server node sets the destination addresses of the N gradient global aggregation result fragments to the respective training nodes in sequence. Then, based on the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to each training node and the aggregation metadata of each node in the node set, stored in memory in step S1, the model aggregation server node marks the N gradient global aggregation result fragments and sends them to the network. Each gradient global aggregation result fragment marks the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to a certain training node and the aggregation metadata of each node in the node set.

[0122] Figure 6 The distributed model training model aggregation server node gradient global aggregation result fragmented data packet structure of the present invention is as follows: Figure 6 As shown, the gradient global aggregation result fragment includes a traditional IP field, a newly added gradient global aggregation result fragment intra-network multi-hop gradient aggregation structure field, and an INA field. The traditional IP field functions the same as a traditional data packet in a data center; the newly added gradient global aggregation result fragment intra-network multi-hop gradient aggregation structure field is used to mark the intra-network multi-hop gradient aggregation structure generated in step S1. The model aggregation server node, based on the corresponding intra-network multi-hop gradient aggregation structure stored in memory in step S1, performs the aggregation according to... Figure 6 The newly added gradient global convergence result fragmentation network multi-hop gradient convergence structure field is filled in from left to right. This field contains multiple {Agg, Port} field groups. Each {Agg, Port} field group represents the convergence metadata of a programmable switch node along the path through which the gradient global convergence result fragments pass. Agg indicates whether the gradient fragments converged at that programmable switch node, and Port represents the forwarding port number of the gradient global convergence result fragment at that programmable switch node. The INA field guides the clearing of model gradients in specific memory units. Specifically, an Ack value of 0 in the INA field indicates that the data packet is a gradient fragment sent from the training node to the model convergence server node; conversely, an Ack value indicates that the data packet is a gradient global convergence result fragment sent from the model convergence server node to the training node. The SlotId in the INA field identifies a specific memory unit within the programmable switch node. The programmable switch node uses this field to determine which specific memory unit should clear part of the gradient convergence result, facilitating the reuse of that memory unit for new model gradient fragment convergence.

[0123] When a gradient fragment arrives at the model aggregation server node, the model aggregation server node performs global gradient aggregation and generates N gradient global aggregation result fragments. The model aggregation server node sets the destination addresses of the N gradient global aggregation result fragments to the respective training nodes w1-w6 in sequence. Then, for the gradient global aggregation result fragment sent by the model aggregation server node to training node w4, the corresponding intra-network multi-hop gradient aggregation structure is {w4: [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L5}, S7: {Agg: 0, Port: L6}, S6: {Agg: 1, Port: L3}, S3: {Agg: 1, Port: w4}]}. The model aggregation server node marks this structure sequentially into the newly added gradient global aggregation result fragment intra-network multi-hop gradient aggregation structure field in the gradient global aggregation result fragment data packet. The marking method for gradient global aggregation result fragments from the model aggregation server node to the other training nodes w1, w2, w3, w5, and w6 is similar to that for w4. After the model aggregation server node marks the newly added gradient global aggregation result fragment network multi-hop gradient aggregation structure field in the gradient global aggregation result fragment, it sets Ack in the INA field of N gradient global aggregation result fragments to 1, sets the SlotId field to the identifier of the specific memory unit of the programmable switch node where the gradient fragment aggregation occurs, and finally sends the gradient global aggregation result fragment to the network.

[0124] Step S5: The programmable switch node processes the gradient global aggregation result.

[0125] Step S5.1: The programmable switch node determines whether any gradient sharding has converged on it.

[0126] When a gradient global aggregation result fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient global aggregation result fragment is used to determine whether a gradient fragment has been aggregated on the programmable switch node. If a gradient fragment has been aggregated on the programmable switch node, it means that the gradient aggregation result in the specific memory unit corresponding to the gradient fragment needs to be cleared so that it can be reused to aggregate gradients. Then, step S5.2 is executed; otherwise, step S5.3 is executed.

[0127] Step S5.2: Clear the gradient aggregation results of specific memory cells in the programmable switch node.

[0128] If gradient fragments converge on the programmable switch node, the programmable switch node clears the gradient convergence result of the specific memory cell and then executes step S5.3.

[0129] Step S5.3: Fragmentation of global aggregation results of programmable switch node forwarding gradient.

[0130] The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient global aggregation result fragmentation marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient global aggregation result fragmentation marker, and forwards the gradient global aggregation result fragment to the next hop node in the network according to the recorded forwarding port number, and executes step S5.4.

[0131] Step S5.4: Is the next-hop node in the network a training node?

[0132] If the next-hop node is a training node, then execute step S6; if the next-hop node is a programmable switch node, then repeat step S5.

[0133] Specifically, as shown in Figure 4(b), when the gradient global convergence result fragment sent by the model convergence server node to w4 passes through S1, S1 determines that a gradient fragment has converged on this programmable switch node based on the convergence metadata {Agg: 1, Port: L1} of the first node in the node set of the gradient global convergence result fragment marker [S1: {Agg: 1, Port: L1}, S5: {Agg: 1, Port: L5}, S7: {Agg: 0, Port: L6}, S6: {Agg: 1, Port: L3}, S3: {Agg: 1, Port: w4}]. At this time, S1 clears the gradient convergence result of the specific memory unit and records the gradient global convergence result. The forwarding port number Port:L1 is recorded in the aggregation metadata {Agg:1, Port:L1} of the first node in the node set of the gradient global aggregation result. Then, the first node in the node set of the gradient global aggregation result is deleted, along with its aggregation metadata [S1:{Agg:1, Port:L1}] (the resulting gradient global aggregation result is labeled as [S5:{Agg:1, Port:L5}, S7:{Agg:0, Port:L6}, S6:{Agg:1, Port:L3}, S3:{Agg:1, Port:w4}]). Based on the recorded forwarding port number Port:L1, the gradient global aggregation result fragment is forwarded to the next-hop node S5 in the network via the L1 link.

[0134] When the gradient global aggregation result fragment sent from the model aggregation server node to w6 passes through S4, S4 determines that no gradient fragment has been aggregated on this programmable switch node based on the aggregation metadata {Agg: 0, Port: w6} of the first node in the node set of the gradient global aggregation result fragment marker [S4: {Agg: 0, Port: w6}]. At this time, S4 records the forwarding port number Port: w6 in the aggregation metadata {Agg: 0, Port: w6} of the first node in the node set of the gradient global aggregation result fragment marker, and then deletes the first node in the node set of the gradient global aggregation result fragment marker and its aggregation metadata [S4: {Agg: 0, Port: w6}] (the gradient global aggregation result fragment marker after deletion is []), and forwards the gradient global aggregation result fragment to the next-hop node w6 in the network according to the recorded forwarding port number Port: w6. Other switches in the data center network perform similar steps to S1 and S4.

[0135] Step S6: The training node receives the global gradient convergence result and updates the local model.

[0136] When the gradient global convergence result slice arrives at the training node, the training node uses the gradient global convergence result carried in the gradient global convergence result slice to update the local model, so that the training node can use the updated local model to perform the next round of iteration training.

[0137] When the gradient global convergence result fragment arrives at training nodes w1-w6, w1-w6 use the gradient global convergence result carried in the gradient global convergence result fragment to update the local model and use the updated local model to perform the next round of distributed model training.

[0138] To further verify the effectiveness of the intra-network multi-hop gradient aggregation method provided by this invention, simulation evaluation was conducted on the proposed method. The evaluation results show that, compared with the current best method, this invention reduces the gradient communication time per iteration of model training by 40%-87%, accelerating model training; it also reduces network communication load by 37%-66%, reducing intra-network gradient communication traffic.

[0139] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A multi-hop gradient aggregation method within a network for distributed model training, wherein the distributed model training comprises a controller, at least one local training node, at least one programmable switch node, and at least one model aggregation server node, characterized in that, Includes the following steps: Step S1: The controller calculates the intra-network multi-hop gradient convergence structure based on the network status. The controller calculates the intra-network multi-hop gradient aggregation structure corresponding to the distributed model training based on the collected network resource status, and sends the aggregation structure to the model aggregation server node and each training node. The intra-network multi-hop gradient aggregation structure includes the following two aspects: Firstly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node, and the aggregation metadata of each node within the node set, wherein the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node, and the aggregation metadata of each node within the node set, are ordered according to the order of the nodes along the gradient flow path from that training node to the model aggregation server node; the aggregation metadata of each node within the set of programmable switch nodes along the path of the gradient flow sent from any training node to the model aggregation server node includes whether the gradient flow should be aggregated at that node, the number of training nodes that should be aggregated at that node, and the outgoing port number of the gradient flow forwarded at that node; Secondly, in the distributed model training system, the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to each training node, and the convergence metadata of each node in the node set, wherein the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node, and the convergence metadata of each node in the node set, are ordered according to the order of the nodes along the path of the gradient global convergence result stream from the model convergence server node to that training node; the convergence metadata of each node in the set of programmable switch nodes along the path of the gradient global convergence result stream sent from the model convergence server node to any training node includes whether a gradient stream converged at that node and the outgoing port number of the gradient global convergence result stream forwarded at that node; In the intranet multi-hop gradient convergence structure in step S1, the set of programmable switch nodes along the path of the gradient flow from any training node to the model convergence server node is completely consistent with the set of programmable switch nodes along the path of the gradient global convergence result flow from the model convergence server node to the training node, but the order of the nodes in the set is reversed. In step S1, after each training node and model aggregation server node receives the intra-network multi-hop gradient aggregation structure issued by the controller, each training node and model aggregation server node saves the intra-network multi-hop gradient aggregation structure in memory. Step S2: Each training node marks gradient slices according to the corresponding intranet multi-hop gradient convergence structure. Based on the set of programmable switch nodes along the path of the gradient flow sent from each training node to the model aggregation server node in the multi-hop gradient aggregation structure within the network issued by the controller, as well as the aggregation metadata of each node in the node set, each training node marks the trained gradient fragments and sets the destination address of the gradient fragments to the model aggregation server node. Then, each training node sends the gradient fragments to the network. Step S3: Programmable switch node processes gradient sharding Step S3.1: The programmable switch node determines whether gradient sharding should converge. When a gradient fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient fragment is used to determine whether the gradient fragment should be aggregated on the programmable switch node. If it should not be aggregated, step S3.2 is executed; otherwise, step S3.3 is executed. Step S3.2: Programmable switch nodes forward gradient fragments that should not be aggregated. If the gradient fragment should not be aggregated on the programmable switch node, the programmable switch node records the forwarding port number in the aggregation meta-information of the first node in the node set of the gradient fragment tag, then deletes the first node in the node set of the gradient fragment tag and its aggregation meta-information, and forwards the gradient fragment to the next-hop node in the network according to the recorded forwarding port number, and executes step S3.

6. Step S3.3: The programmable switch node should aggregate the gradient fragments that should be aggregated. If the gradient slice should be converged on the programmable switch node, the programmable switch node allocates the gradient slice to a specific memory unit for convergence, increments the number of converged training nodes recorded in the memory unit by 1, and executes step S3.

4. Step S3.4: The programmable switch node determines whether gradient sharding has been completed and converged. After the gradient fragment aggregation is completed, the programmable switch node compares the number of training nodes that have been aggregated in the corresponding memory unit with the number of training nodes that should be aggregated on the programmable switch node in the aggregation metadata of the first node in the node set in the gradient fragment marker. If the two are equal, it means that the programmable switch has completed the aggregation of the corresponding gradient fragments of the training nodes that should be aggregated on it, and then executes step S3.

5. If the two are not equal, execute step S3.

7. Step S3.5: Programmable switch node forwarding gradient fragmentation aggregation results The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient fragment marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient fragment marker, writes the gradient aggregation result in the corresponding memory unit into the gradient fragment, and forwards the gradient fragment carrying the gradient aggregation result to the next hop node in the network according to the recorded forwarding port number. Then, it proceeds to step S3.

6. Step S3.6: Is the next-hop node in the network a programmable switch node? If the next-hop node is a programmable switch node, repeat step S3; if the next-hop node is a model aggregation server node, execute step S4. Step S3.7: The programmable switch node discards gradient fragments; Step S4: The model aggregation server node processes gradient fragments. Step S4.1: The model aggregation server node performs gradient global aggregation and generates gradient global aggregation result fragments. When a gradient shard reaches the model aggregation server node, the model aggregation server node performs global gradient aggregation. After all gradient shards corresponding to all training nodes have been aggregated, the model aggregation server node generates N gradient global aggregation result shards. Each gradient global aggregation result shard carries the same gradient global aggregation result, where N is equal to the number of training nodes in the distributed model training. Step S4.2: Mark gradient global aggregation result fragments on the model aggregation server node. The model aggregation server node sets the destination addresses of the N gradient global aggregation result fragments to the respective training nodes in sequence. Then, based on the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to each training node and the aggregation metadata of each node in the node set, stored in memory in step S1, the model aggregation server node marks the N gradient global aggregation result fragments and sends them to the network. Each gradient global aggregation result fragment marks the set of programmable switch nodes along the path of the gradient global aggregation result stream from the model aggregation server node to a certain training node and the aggregation metadata of each node in the node set. Step S5: The programmable switch node processes the gradient global aggregation result. Step S5.1: The programmable switch node determines whether any gradient sharding has converged on it. When a gradient global aggregation result fragment arrives at a programmable switch node, the aggregation element information of the first node in the node set marked by the gradient global aggregation result fragment is used to determine whether a gradient fragment has been aggregated on the programmable switch node. If a gradient fragment has been aggregated on the programmable switch node, it means that the gradient aggregation result in the specific memory unit corresponding to the gradient fragment needs to be cleared so that it can be reused to aggregate gradients. Then, step S5.2 is executed; otherwise, step S5.3 is executed. Step S5.2: Clear the gradient aggregation results of specific memory cells in the programmable switch node. If gradient fragments converge on the programmable switch node, the programmable switch node clears the gradient convergence result of the specific memory cell and then executes step S5.

3. Step S5.3: Fragmentation of global aggregation results of programmable switch node forwarding gradient. The programmable switch node records the forwarding port number in the aggregation element information of the first node in the node set of the gradient global aggregation result fragmentation marker. Then, it deletes the first node and its aggregation element information in the node set of the gradient global aggregation result fragmentation marker, and forwards the gradient global aggregation result fragment to the next hop node in the network according to the recorded forwarding port number, and executes step S5.

4. Step S5.4: Is the next-hop node in the network a training node? If the next-hop node is a training node, then execute step S6; if the next-hop node is a programmable switch node, then repeat step S5. Step S6: The training node receives the global gradient convergence result and updates the local model. When a gradient global convergence result slice arrives at the training node, the training node uses the gradient global convergence result slice to update its local model, so that the training node can use the updated local model to perform the next round of iteration training.

Images