A distributed training gradient data synchronization method, device, medium and product

Abstract

The present invention discloses a gradient data synchronization method, device, medium, and product for distributed training, relating to the field of computer technology. The method comprises: obtaining a two-dimensional network topology of a distributed training system; wherein the two-dimensional network topology includes multiple nodes and directed links between adjacent nodes; generating N path trees based on the two-dimensional network topology, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology; and the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on the N path trees. The present invention improves the efficiency of gradient data synchronization during distributed training.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a gradient data synchronization method, device, medium, and product for distributed training. Background Art

[0002] As models and distributed systems scale, gradient data synchronization and communication become increasingly important throughout the training process, becoming a critical bottleneck in distributed training that needs to be addressed. Previously, gradient data synchronization was performed using the MultiTree algorithm. However, when the MultiTree algorithm is applied to a two-dimensional grid topology, the tree height increases significantly, leading to increased communication latency and reduced communication efficiency.

[0003] Therefore, how to improve the efficiency of gradient data synchronization during distributed training is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present invention is to provide a gradient data synchronization method and device, medium and product for distributed training, which improves the efficiency of gradient data synchronization during distributed training.

[0005] To achieve the above objectives, the present invention provides a gradient data synchronization method for distributed training, comprising:

[0006] Acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0007] Generating N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes included in the N path trees covers all nodes in the two-dimensional network topology structure;

[0008] The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees.

[0009] The generating of N path trees based on the two-dimensional network topology structure includes:

[0010] Taking N different nodes in the two-dimensional network topology structure as root nodes respectively, the nodes in the two-dimensional network topology structure are traversed along directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees.

[0011] Wherein, the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

[0012] The method of taking N different nodes in the two-dimensional network topology structure as root nodes and traversing the nodes in the two-dimensional network topology structure along directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees includes:

[0013] Taking nodes corresponding to N vertices in the two-dimensional mesh topology structure as root nodes respectively, the nodes in the two-dimensional mesh topology structure are traversed along directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees.

[0014] The method of taking nodes corresponding to N vertices in the two-dimensional mesh topology structure as root nodes and traversing the nodes in the two-dimensional mesh topology structure along directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees includes:

[0015] The nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0016] The method comprises taking the nodes corresponding to the three vertices in the two-dimensional mesh topology structure as root nodes, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles, and generating three path trees, including:

[0017] The nodes corresponding to the vertices at the upper left corner, upper right corner, and lower right corner of the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0018] The method uses the nodes corresponding to the vertices at the upper left corner, upper right corner, and lower right corner of the two-dimensional mesh topology structure as root nodes, traverses the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles, and generates three path trees, including:

[0019] Taking the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the column direction and then along the row direction to generate a first path tree;

[0020] Taking the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the row direction and then along the column direction to generate a second path tree;

[0021] Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traverse the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree;

[0022] Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topology structure.

[0023] The process of traversing the nodes in the two-dimensional grid topology structure along the column direction and then along the row direction, with the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, to generate a first path tree includes:

[0024] Accessing a node corresponding to a vertex in an upper left corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as a root node to a first path tree, and adding the root node to a first queue;

[0025] Determine whether there are unvisited nodes;

[0026] If so, taking the head node from the first queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topology structure along the column direction and then along the row direction, adding the visited nodes and the corresponding access paths to the first path tree, adding the visited nodes to the end of the first queue in the order of access, and re-entering the step of determining whether there are unvisited nodes;

[0027] If not, the final generated first path tree is obtained.

[0028] The process of traversing the nodes in the two-dimensional grid topology structure along the row direction and then along the column direction, with the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, to generate a second path tree includes:

[0029] Accessing the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as a root node to the second path tree, and adding the root node to the second queue;

[0030] Determine whether there are unvisited nodes;

[0031] If so, taking the head node from the second queue as the starting node, sequentially visiting the unvisited nodes in the two-dimensional grid topology structure along the row direction and then along the column direction, adding the visited nodes and the corresponding access paths to the second path tree, adding the visited nodes to the end of the second queue in the order of access, and re-entering the step of determining whether there are unvisited nodes;

[0032] If not, the final generated second path tree is obtained.

[0033] The method includes taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle, and generating a third path tree, including:

[0034] Accessing the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as a root node to the third path tree, and adding the root node to the third queue;

[0035] Determine whether there are unvisited nodes;

[0036] If so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor nodes and the directed links between the current node and the neighbor nodes to the third path tree, adding the neighbor nodes to the end of the third queue, and re-entering the step of determining whether there are any unvisited nodes;

[0037] If not, the final generated third path tree is obtained.

[0038] The data synchronization operation on the target data block based on the target path tree includes:

[0039] The leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node;

[0040] The i-th layer node of the target path tree calculates average gradient data based on the data blocks received from the child nodes and the data blocks that the node needs to synchronize based on the target path tree, and sends the average gradient data to its parent node; wherein 1<i<N, N is the number of layers included in the target path tree;

[0041] The root node of the target path tree calculates final average gradient data based on the data blocks received from the child nodes and the data blocks that the root node needs to synchronize based on the target path tree, and sends the final average gradient data to the child nodes of the root node;

[0042] The j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein 1<j≤N.

[0043] After the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, the system further includes:

[0044] The nodes in the distributed training system divide the data blocks that need to be synchronized based on each path tree into multiple communication blocks;

[0045] Accordingly, the data synchronization operation is performed on the target data block based on the target path tree, including:

[0046] The nodes of the target path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the target path tree;

[0047] The leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block as follows: calculating average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: calculating final average gradient data based on the data blocks received from the child nodes and its own data blocks that need to be synchronized based on the target path tree, and sending the final average gradient data to the child nodes of the root node;

[0048] The j-th layer node of the target path tree sequentially and continuously receives the average gradient data corresponding to the plurality of communication blocks from the parent node; wherein 1<j≤M.

[0049] To achieve the above object, the present invention provides a gradient data synchronization device for distributed training, comprising:

[0050] An acquisition module, configured to acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0051] A generating module, configured to generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the set of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure;

[0052] The data synchronization module is used to divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0053] To achieve the above object, the present invention provides an electronic device, comprising:

[0054] memory for storing computer programs;

[0055] A processor is configured to implement the steps of the above-mentioned method for synchronizing gradient data for distributed training when executing the computer program.

[0056] To achieve the above-mentioned objectives, the present invention provides a non-volatile storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the gradient data synchronization method for distributed training are implemented as described above.

[0057] To achieve the above objectives, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the gradient data synchronization method for distributed training as described above.

[0058] Through the above scheme, it can be seen that the present invention provides a distributed training gradient data synchronization method, including: obtaining a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes multiple nodes and directed links between adjacent nodes; generating N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the set of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure; the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on the N path trees.

[0059] The beneficial effects of the present invention are as follows: the gradient data synchronization method for distributed training provided by the present invention generates N path trees that do not share any directed links based on a two-dimensional network topology structure. These path trees jointly cover all nodes in the network topology. The path trees constructed in this way can ensure that the height of each path tree is moderate, effectively controlling the transmission delay. Since there are no shared directed links between the path trees, the nodes can divide the gradient data to be synchronized into N data blocks and synchronize the data blocks in parallel based on these path trees, significantly reducing congestion and delays in the data transmission process, and improving the efficiency of gradient data synchronization in the distributed training process. The present invention also discloses a gradient data synchronization device for distributed training, an electronic device, a non-volatile storage medium, and a computer program product, which can also achieve the above-mentioned technical effects.

[0060] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure, but do not constitute a limitation of the present disclosure. In the drawings:

[0062] Figure 1 4 is a flowchart of a method for synchronizing gradient data for distributed training according to an exemplary embodiment;

[0063] Figure 2 4 is a flowchart of another method for synchronizing gradient data for distributed training according to an exemplary embodiment;

[0064] Figure 3 This is a flowchart of a specific method for generating a topology-aware tree according to an exemplary embodiment;

[0065] Figure 4 This is a schematic diagram of a 3×3 two-dimensional grid topology;

[0066] Figure 5 The following is a schematic diagram of three initial trees generated using a 3×3 two-dimensional grid as an example;

[0067] Figure 6 This is a schematic diagram of the three topology-aware trees generated, with disjoint links, using a 3×3 two-dimensional grid as an example.

[0068] Figure 7 This is a flow chart showing a pipelined parallel communication block scheduling method according to an exemplary embodiment;

[0069] Figure 8 Schematic diagram showing the performance comparison between a communication block scheduling method that adopts pipeline parallelism and a communication block scheduling method that does not adopt pipeline parallelism;

[0070] Figure 9 1 is a structural diagram of a gradient data synchronization device for distributed training according to an exemplary embodiment;

[0071] Figure 10 The figure is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, in the embodiments of the present invention, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0073] A 2D-mesh topology is a common physical topology in which each node is directly connected to its neighbors, forming a regular two-dimensional grid structure. This grid is typically divided into rows and columns, with each node connected only to a fixed number of surrounding nodes. This layout simplifies communication routing and data transmission, and offers excellent scalability, allowing for easy expansion of the number of nodes. Consequently, it is widely used in large-scale parallel systems. Furthermore, links in current 2D mesh topologies are typically bidirectional.

[0074] The MultiTree algorithm is a topology-aware, efficient gradient data synchronization method that improves link utilization by constructing a set of topology-aware trees. The MultiTree algorithm constructs multiple trees based on the physical topology. Assuming there are n compute nodes in the physical topology, a tree is constructed with each compute node as the root, for a total of n trees. Each resulting tree should contain all compute nodes in the topology. The tree construction process is top-down, starting from the root. For each time_step, links are assigned using the topology graph, the remaining nodes are connected to the spanning tree, and the assigned links are removed from the graph. When there are no more available links connecting the remaining nodes to any tree, the new topology graph is used in the next time step.

[0075] When the MultiTree algorithm is applied to a two-dimensional grid topology, the tree height increases significantly, leading to increased communication latency and reduced communication efficiency. Furthermore, although the physical links used by each tree in the MultiTree algorithm are different within each time step, repeated physical links are used between different time steps. This means that during gradient synchronization communication, to avoid link congestion, only one communication block can be processed within the tree at a time, rather than using a pipelined parallel approach to schedule multiple communication blocks simultaneously within each tree. This significantly limits the potential for improving gradient synchronization communication efficiency.

[0076] Therefore, in this invention, N path trees that do not share any directed links are generated based on a two-dimensional network topology. These path trees collectively cover all nodes in the network topology. This construction of path trees ensures that each path tree has a moderate height, effectively controlling transmission delay. Because the path trees lack shared directed links, nodes can divide the gradient data to be synchronized into N data blocks and synchronize these blocks in parallel based on these path trees. This significantly reduces congestion and delay during data transmission and improves the efficiency of gradient data synchronization during distributed training.

[0077] The embodiment of the present invention discloses a gradient data synchronization method for distributed training, which improves the gradient data synchronization efficiency during the distributed training process.

[0078] See also Figure 1 , according to an exemplary embodiment, a flowchart of a gradient data synchronization method for distributed training is shown, such as Figure 1 Shown, including:

[0079] S101: Acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0080] In practice, during the initialization phase of the distributed training system, a two-dimensional network topology diagram is created, consisting of multiple nodes and directed links between them. This topology diagram details the connection relationships between the nodes and provides the necessary network architecture information for subsequent data synchronization operations.

[0081] As a feasible implementation, the two-dimensional network topology of the distributed training system is a two-dimensional mesh topology, that is, this embodiment can be applied to a two-dimensional mesh topology (2D-mesh).

[0082] S102: Generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and at least two path trees contain all nodes in the two-dimensional network topology structure;

[0083] In practice, N path trees are generated based on the acquired two-dimensional network topology. These path trees are designed to not share any directed links, ensuring full utilization of network resources and load balancing. Each path tree is independent, but its node set must cover all nodes in the entire network topology. This ensures that every node is effectively included during data synchronization.

[0084] As a feasible implementation method, the generation of N path trees based on the two-dimensional network topology structure includes: taking N different nodes in the two-dimensional network topology structure as root nodes, traversing the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles, and generating N path trees.

[0085] In practice, N different nodes are selected in a two-dimensional network topology as the root nodes of their respective path trees. Then, based on specific traversal principles, all nodes in the network are traversed along directed links to construct N independent path trees. Each path tree begins with its root node and, through traversal, connects the nodes in the network in a specific order, forming a tree-like structure. This ensures that each path tree covers unique nodes and links, avoiding link duplication and achieving optimal allocation and utilization of network resources.

[0086] If the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure, as a feasible implementation method, N different nodes in the two-dimensional network topology structure are used as root nodes, and the nodes in the two-dimensional network topology structure are traversed along the directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees, including: taking the nodes corresponding to N vertices in the two-dimensional grid topology structure as root nodes, and traversing the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate N path trees.

[0087] In a specific implementation, when the two-dimensional network topology of the distributed training system is a two-dimensional grid topology, N vertices in the two-dimensional grid topology are selected as the root nodes of their respective path trees. Then, according to their respective traversal principles, they traverse along the directed links in the grid to construct N independent path trees. Each path tree starts from its corresponding vertex node and, through the traversal process, connects the other nodes in the grid in a specific order to form a tree structure. This ensures that each path tree covers unique nodes and links, thereby optimizing the use of network resources and improving the efficiency of data synchronization.

[0088] As a feasible implementation method, the nodes corresponding to the N vertices in the two-dimensional grid topology structure are respectively used as root nodes, and the nodes in the two-dimensional grid topology structure are traversed along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate N path trees, including: taking the nodes corresponding to the three vertices in the two-dimensional grid topology structure as root nodes, and traversing the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate three path trees.

[0089] In a specific implementation, within the two-dimensional grid topology of a distributed training system, three specific nodes corresponding to vertices are selected as root nodes to generate three path trees. Specifically, three different traversal principles are used to traverse the directed links within the grid topology, sequentially accessing each node within the grid. This generates a unique path tree for each root node, and the three path trees together cover all nodes in the entire two-dimensional grid topology. Directed links within each path tree are unique, thus achieving efficient allocation of network resources and optimizing data synchronization operations.

[0090] As a feasible implementation method, the nodes corresponding to the three vertices in the two-dimensional grid topology structure are respectively used as root nodes, and the nodes in the two-dimensional grid topology structure are traversed along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate three path trees, including: taking the nodes corresponding to the vertices in the upper left corner, upper right corner, and lower right corner of the two-dimensional grid topology structure as root nodes, and traversing the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate three path trees.

[0091] In a specific implementation, in the two-dimensional grid topology of the distributed training system, the three vertices at the upper left, upper right, and lower right corners are selected as the root nodes of their respective path trees. Then, according to their respective traversal principles, traversal is performed along the directed links in the grid to construct three independent path trees. Each path tree starts from its corresponding vertex node and connects the other nodes in the grid in a specific order through the traversal process to form a tree structure, ensuring that the nodes and links covered by each path tree are unique, thereby optimizing the use of network resources and improving the efficiency of data synchronization. This method helps to achieve load balancing and reduce communication delays in distributed training. Especially in large-scale computing tasks, this strategy can significantly improve the efficiency of data synchronization and the performance of the entire training process.

[0092] S103: The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0093] In practice, each node in the distributed training system divides the gradient data to be synchronized into N data blocks, each corresponding to a path tree. The system then performs data synchronization on each of the N data blocks based on these N path trees. This approach allows for parallel synchronization, with each path tree responsible for synchronizing a single data block, thereby improving the efficiency and speed of data transmission. This parallel processing mechanism can significantly reduce the total time required for data synchronization, especially in large-scale distributed training scenarios.

[0094] As a feasible implementation method, a data synchronization operation is performed on the target data block based on the target path tree, including: the leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node; the i-th layer node of the target path tree calculates the average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sends the average gradient data to its own parent node; wherein, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree calculates the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sends the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein, 1<j≤M.

[0095] In the specific implementation, each leaf node first sends the data block it is responsible for to its parent node. At the i-th layer of the path tree (1<i<M), each node collects data blocks from its child nodes, calculates the average gradient data based on its own data blocks, and then sends this average value to the parent node of the previous layer. This process continues until the root node, which calculates the final average gradient data based on the data blocks received from all child nodes and sends this result back to the child nodes. At the j-th layer of the path tree (1<j≤M), the node receives the final average gradient data from the parent node and sends it further down. This hierarchical aggregation process ensures that in distributed training, the gradient data can be efficiently synchronized between nodes, thereby achieving global consistency of model parameters.

[0096] As another feasible implementation, a data synchronization operation is performed on the target data block based on the target path tree, including: the leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node; the i-th layer node of the target path tree sends the data block received from the child node and the data block that needs to be synchronized based on the target path tree to its own parent node; wherein, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree calculates the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sends the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein, 1<j≤M.

[0097] In a specific implementation, the leaf nodes of the path tree send the data blocks they are responsible for directly to their parent nodes. For non-leaf nodes, they collect data blocks from their child nodes, merge their own data blocks with them, and then send them to their parent nodes. This process is repeated at each layer of the path tree until the root node. The root node summarizes all the data blocks received from the child nodes and calculates the final average gradient data, and then sends this average value back to its child nodes. The nodes in the jth layer (1<j≤M) will receive the final average gradient data from the parent node. In this implementation, only the root node calculates the average gradient data, and other nodes are only responsible for the transmission of gradient data, which reduces the resource usage of other nodes.

[0098] As a preferred embodiment, after the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, the further step includes: the nodes in the distributed training system divide the data blocks that need to be synchronized based on each of the path trees into multiple communication blocks; accordingly, the target data blocks are synchronized based on the target path tree, including: the nodes of the target path tree sequentially and continuously process the multiple communication blocks that need to be synchronized based on the target path tree; wherein the leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block The process is as follows: based on the current communication block received from the child node and its own current communication block, average gradient data is calculated, and the average gradient data is sent to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: based on the data block received from the child node and its own data block that needs to be synchronized based on the target path tree, the final average gradient data is calculated, and the final average gradient data is sent to the child node of the root node; the j-th layer node of the target path tree sequentially receives the average gradient data corresponding to multiple communication blocks from the parent node; wherein, 1<j≤M.

[0099] In practice, after dividing the gradient data into N data blocks, the nodes in the distributed training system further subdivide each data block into multiple communication blocks and synchronize these communication blocks in each path tree. Specifically, each node processes the communication blocks it is responsible for in sequence. Leaf nodes directly send their communication blocks to the parent node; non-leaf nodes receive communication blocks from child nodes, combine them with their own communication blocks to calculate the average gradient data, and then send the results to their parent nodes. This process is repeated at each layer of the path tree until the root node. The root node calculates the final average gradient data and sends this data back to the child nodes. During this process, nodes at each layer (1<j≤M) continuously receive the average gradient data corresponding to the communication blocks from the parent node, ensuring layer-by-layer aggregation and synchronization of data, thereby achieving effective synchronization and updating of gradient data throughout the distributed system.

[0100] The gradient data synchronization method for distributed training provided in an embodiment of the present invention generates N path trees based on a two-dimensional network topology that do not share any directed links. These path trees collectively cover all nodes in the network topology. This constructed path tree ensures that each path tree has a moderate height, effectively controlling transmission delay. Because the path trees lack shared directed links, nodes can divide the gradient data to be synchronized into N data blocks and synchronize the data blocks in parallel based on these path trees. This significantly reduces congestion and delay during data transmission and improves the efficiency of gradient data synchronization during distributed training.

[0101] The embodiment of the present invention discloses a method for synchronizing gradient data for distributed training. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0102] See also Figure 2 , according to an exemplary embodiment, a flowchart of another method for synchronizing gradient data for distributed training is shown, such as Figure 2 Shown, including:

[0103] S201: Obtain a two-dimensional grid topology structure of a distributed training system;

[0104] S202: Taking the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, traverse the nodes in the two-dimensional grid topology structure first along the column direction and then along the row direction to generate a first path tree;

[0105] In a specific implementation, the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure is used as the root node, and the tree is first traversed along the column direction and then along the row direction to generate the first path tree.

[0106] As a feasible implementation method, the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure is used as the root node, and the nodes in the two-dimensional grid topology structure are traversed first along the column direction and then along the row direction to generate a first path tree, including: visiting the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node to the first path tree, and adding the root node to the first queue; judging whether there is an unvisited node; if so, taking the head node from the first queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topology structure in sequence along the column direction and then along the row direction, adding the visited nodes and the corresponding access paths to the first path tree, adding the visited nodes to the end of the first queue in the order of access, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the finally generated first path tree.

[0107] In the specific implementation, first, select the node corresponding to the vertex in the upper left corner of the grid as the root node and add it to the first path tree and the first queue. Then, cyclically check whether there are unvisited nodes. If so, take out the node at the front of the queue as the starting node, visit the unvisited nodes in the grid in the order of columns first and rows later, and add the visited nodes and their access paths to the first path tree. At the same time, the newly visited nodes will be added to the end of the first queue according to the access order for subsequent processing. This process will be repeated continuously until there are no unvisited nodes. At this time, the first path tree has been generated, which contains all the nodes visited from the root node in the specified traversal order, and there are no repeated directed links.

[0108] S203: Using the node corresponding to the vertex at the lower right corner of the two-dimensional grid topology structure as a root node, traverse the nodes in the two-dimensional grid topology structure first along the row direction and then along the column direction to generate a second path tree;

[0109] In a specific implementation, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology is used as the root node. The tree is first traversed along the rows and then along the columns to generate the second path tree. This traversal order helps to create a path tree that is different from the first path tree and covers all nodes.

[0110] As a feasible implementation method, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure is used as the root node, and the nodes in the two-dimensional grid topology structure are traversed first in the row direction and then in the column direction to generate a second path tree, including: visiting the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node to the second path tree, and adding the root node to the second queue; judging whether there is an unvisited node; if so, taking the head node from the second queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topology structure in sequence along the row direction and then in the column direction, adding the visited nodes and the corresponding access paths to the second path tree, adding the visited nodes to the end of the second queue in the order of access, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the final generated second path tree.

[0111] In the specific implementation, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology is used as the root node, and the root node is first added to the second path tree and the second queue. Then, it is cyclically determined whether there are unvisited nodes. If so, the node at the front of the queue is taken out as the starting node, and the unvisited nodes in the grid are visited in the order of rows first and columns later, and the visited nodes and their access paths are added to the second path tree. At the same time, the newly visited nodes will be added to the end of the second queue according to the access order for subsequent processing. This process will be repeated until there are no unvisited nodes. At this time, the second path tree has been generated, which contains all nodes visited from the root node in the specified traversal order, and there are no repeated directed links. This method ensures that the generation of the path tree is orderly and efficient, and provides another structured path for data synchronization in distributed training.

[0112] S204: Using the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as a root node, traverse the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on a breadth-first traversal principle to generate a third path tree;

[0113] In a specific implementation, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure is used as the root node, and a breadth-first search strategy is adopted to traverse along the directed links to generate the third path tree.

[0114] As a feasible implementation method, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure is used as the root node, and the nodes in the two-dimensional grid topology structure are traversed along the directed links in the two-dimensional grid topology structure based on the breadth-first traversal principle to generate a third path tree, including: visiting the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as the root node to the third path tree, and adding the root node to the third queue; judging whether there is an unvisited node; if so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor node and the directed links between the current node and the neighbor node to the third path tree, adding the neighbor node to the end of the third queue, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the final generated third path tree.

[0115] In the specific implementation, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology is used as the root node. First, the root node is added to the third path tree and the third queue. Then, the system will cyclically check whether there are unvisited nodes. If so, the node at the front of the queue is taken out as the current node, and all unvisited neighbor nodes are visited. The access order of each neighbor node is not limited here, and these neighbor nodes and the directed links between them and the current node are added to the third path tree. At the same time, these newly visited neighbor nodes will be added to the end of the third queue for subsequent processing. This process will be repeated until there are no unvisited nodes. At this time, the third path tree has been generated, which includes all nodes and corresponding directed links visited in the breadth-first traversal order starting from the root node, ensuring that the path tree covers all nodes and links are not repeated.

[0116] S205: Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topology structure.

[0117] In a specific implementation, after generating the third path tree, the target node corresponding to the lower left corner vertex in the third path tree and the path connected thereto are deleted to determine that the directed links contained in the third path tree do not overlap with the directed links contained in the first path tree and the second path tree.

[0118] S206: The node in the distributed training system divides the gradient data to be synchronized into N data blocks, and divides the data blocks that need to be synchronized based on each path tree into multiple communication blocks;

[0119] In a specific implementation, the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and further subdivide each data block into multiple communication blocks so as to synchronize based on each path tree. Such division helps to improve the efficiency and parallelism of data transmission.

[0120] S207: The nodes of the path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the path tree;

[0121] The leaf node of the path tree processes the current communication block by sending the current communication block to the parent node of the leaf node. The i-th layer node of the path tree processes the current communication block by calculating average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node. The root node of the path tree processes the current communication block by calculating final average gradient data based on the data blocks received from the child node and its own data blocks that need to be synchronized based on the path tree, and sending the final average gradient data to the child node of the root node.

[0122] In practice, each node sequentially and continuously processes multiple communication blocks to synchronize gradient data. For leaf nodes in the path tree, the process of processing communication blocks is relatively simple, primarily sending the current communication block directly to its parent node. This direct transfer is the starting point for data aggregation, ensuring consistent data flow from leaf to root. Non-leaf nodes, particularly those in the i-th layer (1 < i < M), require more complex operations. These nodes first receive communication blocks from their children and then calculate average gradient data based on the communication blocks they are responsible for. This step is the core of gradient data aggregation, allowing nodes to merge information from multiple data sources to reduce data transmission redundancy and improve synchronization efficiency. Once calculated, these average gradient data are sent to the parent node in the previous layer for further aggregation. The root node of the path tree is responsible for the final aggregation operation. Based on the data blocks received from all child nodes and the data blocks it is responsible for, it calculates the final average gradient data for the entire path tree. This final result is then sent back to the root node's children, completing the data distribution process from root to leaves.

[0123] That is, the three path trees simultaneously execute the ScatterReduce operation in a pipeline manner starting from the leaf nodes. After each node processes a communication block, it sends it to its parent node, and then immediately receives the next communication block for processing until all the data to be synchronized is processed.

[0124] S208: The j-th layer node of the path tree sequentially and continuously receives the average gradient data corresponding to the plurality of communication blocks from the parent node.

[0125] In practice, nodes at the jth level of the tree (1 < j ≤ M) receive average gradient data from their parent nodes. These nodes sequentially and continuously receive average gradient data corresponding to multiple communication blocks, a key step in the data synchronization process from the root node to the leaf nodes. This ensures that every node receives the average gradient information for the entire tree, ensuring consistent gradient data across all nodes during distributed training.

[0126] That is, the three path trees simultaneously execute the Allgather operation in a pipeline manner starting from the root node. After each node processes a communication block, it sends it to its leaf node, and then immediately receives the next communication block for processing until all the data to be synchronized is processed.

[0127] It can be seen that this embodiment adopts a pipelined parallel communication block scheduling method, which effectively improves the efficiency of gradient data synchronization and alleviates the bottleneck problem caused by gradient data synchronization in large-scale data synchronization training.

[0128] The following describes an application embodiment provided by the present invention, which generates an m×n two-dimensional grid into three non-intersecting trees, each of which is generated with a node at a corner of the two-dimensional grid as a tree root. The specific process of generating a topology-aware tree is as follows: Figure 3 As shown, the following steps are included:

[0129] Step 1: First, initialize the numbering of the m×n two-dimensional grid topology: serialize the numbering of nodes and physical links in the topology. Taking the 3×3 grid as an example, the numbered two-dimensional grid is as follows Figure 4 shown.

[0130] Step 2: Initialize the root nodes of the three trees to 1, n, and nm respectively: For an m×n grid, designate the nodes numbered 1, n, and nm (the upper left node, upper right node, and lower right node of the two-dimensional grid, respectively) as the roots of the three trees.

[0131] Step 3: Construct each tree in turn:

[0132] Generate a tree rooted at node 1. With node 1 as the root, connect all other nodes in the grid to the tree along the directed links in the 2D mesh topology. Specifically, starting with node 1, first connect all nodes along the y-axis of the 2D mesh topology, then connect all other nodes along the x-axis, until all nodes in the grid are connected to the tree.

[0133] Generate a tree rooted at node nm. With node nm as the root, connect all other nodes in the grid to the tree along each directed link in the two-dimensional grid topology. Specifically, starting with node nm, first connect each node along the x-axis of the two-dimensional grid topology, then connect all other nodes along the y-axis until all nodes in the grid are connected to the tree.

[0134] Generate a tree rooted at node n. With node n as the root, connect all other nodes in the grid to the tree along the directed links in the 2D grid topology. Specifically, starting from node n, traverse each node in the 2D grid using the breadth-first traversal principle, adding each node in the 2D grid topology to the current tree.

[0135] by Figure 4 Taking the 3×3 scale two-dimensional grid as an example, the three trees generated are as follows Figure 5 shown.

[0136] The process of generating a tree with node 1 as the root is as follows: visit node 1, add node 1 as the root node to tree 1, and add node 1 to the first queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node, that is, node 1, from the first queue as the starting node. First, visit node 4 and node 7 in the column direction, and then visit node 2 and node 3 in the row direction. Add node 4, node 7, node 2, node 3 and the corresponding access path to tree 1, and add node 4, node 7, node 2, node 3 to the end of the first queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node, that is, node 4, from the first queue. Since node 4 is the starting node in the column direction, There are no unvisited nodes, so starting from node 4, we visit node 5 and node 6 in the row direction, add node 5, node 6 and the corresponding access path to tree 1, and add node 5 and node 6 to the end of the first queue in turn. At this time, there are unvisited nodes in the two-dimensional grid, and the first node, that is, node 7, is taken out from the first queue. Since there are no unvisited nodes starting from node 7 along the column direction, starting from node 7, we visit node 8 and node 9 in the row direction, add node 8, node 9 and the corresponding access path to tree 1, and add node 8 and node 9 to the end of the first queue in turn. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 1 is obtained.

[0137] The process of generating a tree with node 9 as the root is as follows: visit node 9, add node 9 as the root node to tree 2, and add node 9 to the second queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node of the queue, that is, node 9, from the second column as the starting node. First, visit node 8 and node 7 in the row direction, and then visit node 6 and node 3 in the column direction. Add node 8, node 7, node 6, node 3 and the corresponding access path to tree 2, and add node 8, node 7, node 6, node 3 to the end of the second queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node of the queue, that is, node 8, from the second queue. Since node 8 is the starting node in the row direction, There are no unvisited nodes, so starting from node 8, we visit node 5 and node 2 in sequence along the column direction, add node 5, node 2 and the corresponding access path to tree 2, and add node 5 and node 2 to the end of the second queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid, and the first node, that is, node 7, is taken out from the second queue. Since there are no unvisited nodes starting from node 7 along the row direction, starting from node 7, we visit node 4 and node 1 in sequence along the column direction, and add node 4, node 1 and the corresponding access path to tree 2, and add node 4 and node 1 to the end of the second queue in sequence. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 2 is obtained.

[0138] 3 and 1 are added to the tree 3. The process of generating a tree with node 3 as the root is as follows: visit node 3, add node 3 as the root node to tree 3, and add node 3 to the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 3, from the third queue, visit all unvisited neighbor nodes of node 3, node 2, node 6, add node 2, node 6 and the directed link between node 3 and node 2, and the directed link between node 3 and node 6 to tree 3, and add node 2 and node 6 to the end of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 2, from the third queue, visit all unvisited neighbor nodes of node 2, node 5, node 1, add node 5, node 1 and the directed link between node 2 and node 5, and the directed link between node 2 and node 1 to tree 3, and add node 5 and node 1 to the end of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 6, from the third queue, visit all unvisited neighbor nodes of node 6, node 9, and add node 9 and Directed link 9 between node 6 and node 9 is added to tree 3, and node 9 is added to the tail of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. The head node, that is, node 5, is taken out from the third queue. All unvisited neighbor nodes of node 5, node 4 and node 8, are visited. Node 4, node 8, and the directed link between node 5 and node 4, and the directed link between node 5 and node 8 are added to tree 3. Node 4 and node 8 are added to the tail of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. The head node, that is, node 1, is taken out from the third queue. Since node 1 has no unvisited neighbor nodes, the head node, that is, node 9, is continued to be taken out from the third queue. Since node 9 has no unvisited neighbor nodes, the head node, that is, node 4, is continued to be taken out from the third queue. All unvisited neighbor nodes of node 4, node 7, are visited. Node 7 and the directed link between node 4 and node 7 are added to tree 3. Node 7 is added to the tail of the third queue. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 3 is obtained.

[0139] Step 4: Observe the three generated trees and find that the first and third trees reuse the link (16) between node 4 and node 7. Finally, in order to make the three generated trees disjoint, that is, to have no reused links, remove the node numbered n(m-1)+1 from the tree with root node n.

[0140] Taking a 3×3 2D grid as an example, remove node 7 from the tree with root node 3. The three topology-aware trees with disjoint links are finally generated as follows: Figure 6 shown.

[0141] Based on the topological perception tree with disjoint links generated above, gradient data synchronization is performed. The gradient data synchronization process is mainly the scheduling process of multiple communication blocks. A pipeline parallel communication block scheduling method is proposed. The specific implementation steps are as follows: Figure 7 As shown, the following steps are included:

[0142] Step 1: Divide the data to be synchronized into 3 equal parts, and assign each part to a tree for processing.

[0143] Step 2: On each tree, divide the data to be processed into q communication blocks (q can be set according to the actual situation). Assuming the size of the data to be synchronized is P, the size of each communication block is P / 3q.

[0144] Step 3: ScatterReduce Pipelining: The three topology-aware trees simultaneously execute ScatterReduce operations starting from the leaf nodes in a pipelined manner. That is, each node processes a communication block and sends it to its parent node, then immediately receives the next communication block for processing until all pending data is processed.

[0145] Step 4: Allgather Phase Pipeline Operation: After the ScatterReduce phase completes, the three topology-aware trees execute the Allgather operation in a pipelined manner, starting from the root node. That is, each node processes a communication block and sends it to its leaf node, which then receives the next communication block for processing until all pending data is processed.

[0146] It can be seen that this embodiment of the present invention proposes a gradient data synchronization method for two-dimensional grid topology. By adopting a new method to construct the topology tree, the height of each constructed tree is moderate, which effectively reduces the transmission delay. There are no shared physical links between the generated trees and between the time steps, which makes it possible to simultaneously schedule multiple communication blocks in each tree in a pipeline parallel manner. In addition, in the process of generating the topology-aware tree, almost all links in the two-dimensional grid topology are fully utilized, which greatly improves the physical link utilization compared with the existing methods. Next, based on the constructed topology-aware tree with non-intersecting links, we proposed a pipeline parallel communication block scheduling method. Through pipeline parallelism, the communication time is effectively reduced and the data synchronization efficiency in the two-dimensional grid topology is improved. The performance comparison diagram of the communication block scheduling method using pipeline parallelism and the communication block scheduling method not using pipeline parallelism is shown in the figure. Figure 8 As shown, it can be seen that the method of the present invention effectively improves the efficiency of gradient data synchronization and alleviates the bottleneck problem caused by gradient data synchronization in large-scale data synchronization training.

[0147] The following introduces a gradient data synchronization device for distributed training provided by an embodiment of the present invention. The gradient data synchronization device for distributed training described below and the gradient data synchronization method for distributed training described above can be referenced to each other.

[0148] See also Figure 9 , according to an exemplary embodiment, a structural diagram of a gradient data synchronization device for distributed training is shown, such as Figure 9 Shown, including:

[0149] An acquisition module 100 is configured to acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0150] A generating module 200 is configured to generate N path trees based on the two-dimensional network topology, wherein the N path trees do not contain the same directed links, and the set of nodes contained in the N path trees covers all nodes in the two-dimensional network topology;

[0151] The data synchronization module 300 is configured to divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0152] The gradient data synchronization device for distributed training provided in an embodiment of the present invention generates N path trees based on a two-dimensional network topology that do not share any directed links. These path trees collectively cover all nodes in the network topology. This constructed path tree ensures that each path tree has a moderate height, effectively controlling transmission delay. Because the path trees lack shared directed links, nodes can divide the gradient data to be synchronized into N data blocks and synchronize the data blocks in parallel based on these path trees. This significantly reduces congestion and delay during data transmission, thereby improving the efficiency of gradient data synchronization during distributed training.

[0153] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: use N different nodes in the two-dimensional network topology structure as root nodes, traverse the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles, and generate N path trees.

[0154] Based on the above embodiment, as a preferred implementation, the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

[0155] Based on the above embodiment, as a preferred implementation mode, the generation module 200 is specifically used to: use the nodes corresponding to the N vertices in the two-dimensional grid topology structure as root nodes, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles, and generate N path trees.

[0156] Based on the above embodiment, as a preferred implementation mode, the generation module 200 is specifically used to: use the nodes corresponding to the three vertices in the two-dimensional grid topology structure as root nodes, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles, and generate three path trees.

[0157] Based on the above embodiment, as a preferred implementation mode, the generation module 200 is specifically used to: use the nodes corresponding to the vertices in the upper left corner, upper right corner, and lower right corner of the two-dimensional grid topology structure as root nodes, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles, and generate three path trees.

[0158] Based on the above embodiment, as a preferred implementation manner, the generation module 200 is specifically used to: use the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, first traverse the nodes in the two-dimensional grid topology structure along the column direction and then along the row direction to generate a first path tree; use the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, first traverse the nodes in the two-dimensional grid topology structure along the row direction and then along the column direction to generate a second path tree; use the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as the root node, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on the breadth-first traversal principle to generate a third path tree; delete the target node and the path connected to the target node in the third path tree; wherein, the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topology structure.

[0159] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: access the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as a root node to the first path tree, and add the root node to the first queue; determine whether there is an unvisited node; if so, take out the head node from the first queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure in sequence along the column direction and then along the row direction, add the visited nodes and the corresponding access paths to the first path tree, add the visited nodes to the end of the first queue in the order of access, and re-enter the step of determining whether there is an unvisited node; if not, obtain the final generated first path tree.

[0160] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: visit the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node to the second path tree, and add the root node to the second queue; determine whether there is an unvisited node; if so, take out the head node from the second queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure in the row direction and then in the column direction, add the visited nodes and the corresponding access paths to the second path tree, add the visited nodes to the end of the second queue in the order of access, and re-enter the step of determining whether there is an unvisited node; if not, obtain the final generated second path tree.

[0161] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: access the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as the root node to the third path tree, and add the root node to the third queue; determine whether there is an unvisited node; if so, take out the head node from the third queue as the current node, access all unvisited neighbor nodes of the current node, add the neighbor node and the directed link between the current node and the neighbor node to the third path tree, add the neighbor node to the end of the third queue, and re-enter the step of determining whether there is an unvisited node; if not, obtain the final generated third path tree.

[0162] Based on the above embodiments, as a preferred implementation mode, the leaf node of the target path tree sends the data block that it needs to synchronize based on the target path tree to the parent node of the leaf node; the i-th layer node of the target path tree calculates the average gradient data based on the data block received from the child node and the data block that it needs to synchronize based on the target path tree, and sends the average gradient data to its own parent node; wherein, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree calculates the final average gradient data based on the data block received from the child node and the data block that it needs to synchronize based on the target path tree, and sends the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein, 1<j≤M.

[0163] Based on the above embodiment, as a preferred implementation, it further includes:

[0164] A division module, configured to divide a data block that needs to be synchronized based on each path tree into a plurality of communication blocks;

[0165] Correspondingly, the nodes of the target path tree sequentially and continuously process multiple communication blocks that they need to synchronize based on the target path tree; wherein, the leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block as follows: calculating the average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: calculating the final average gradient data based on the data blocks received from the child nodes and the data blocks that it needs to synchronize based on the target path tree, and sending the final average gradient data to the child node of the root node; the j-th layer node of the target path tree sequentially and continuously receives the average gradient data corresponding to multiple communication blocks from the parent node; wherein, 1<j≤M.

[0166] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0167] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device, Figure 10 FIG. 1 is a structural diagram of an electronic device according to an exemplary embodiment. Figure 10 As shown, the electronic equipment includes:

[0168] Communication interface 1, capable of exchanging information with other devices such as network devices;

[0169] Processor 2 is connected to communication interface 1 to implement information exchange with other devices and is used to execute the gradient data synchronization method for distributed training provided by one or more of the above technical solutions when running a computer program. The computer program is stored in memory 3.

[0170] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 10 Various buses are labeled as bus system 4.

[0171] The memory 3 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0172] It is understood that the memory 3 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk or a magnetic tape. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 3 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0173] The method disclosed in the above embodiment of the present invention can be applied to processor 2 or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the hardware integrated logic circuit in processor 2 or by instructions in software form. The above processor 2 can be a general-purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. Processor 2 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the above method in combination with its hardware.

[0174] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present invention are implemented, which will not be described here for the sake of brevity.

[0175] In an exemplary embodiment, the present invention further provides a non-volatile storage medium storing a computer program. The computer program can be executed by the processor 2 to complete the steps of the aforementioned method.

[0176] In an exemplary embodiment, the present invention further provides a computer program product, including a computer program, which is executed by the processor 2 to complete the steps of the aforementioned method.

[0177] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiments can be implemented by hardware related to computer program instructions, and the aforementioned computer program can be stored in a non-volatile storage medium. When the computer program is executed, it executes the steps of the above-mentioned method embodiments. Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, server, network device, etc.) to execute all or part of the methods described in each embodiment of the present invention.

[0178] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A gradient data synchronization method for distributed training, characterized in that: include: Obtaining a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure is a physical topology structure, the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes, and the directed links are physical links; Generating N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes included in the N path trees covers all nodes in the two-dimensional network topology structure; The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively; The generating of N path trees based on the two-dimensional network topology structure includes: Taking N different nodes in the two-dimensional network topology structure as root nodes, traversing the nodes in the two-dimensional network topology structure along directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees; The data synchronization operation on the target data block based on the target path tree includes: The leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node; The i-th layer node of the target path tree calculates average gradient data based on the data blocks received from the child nodes and the data blocks that the node needs to synchronize based on the target path tree, and sends the average gradient data to its parent node; wherein 1<i<M, M is the number of layers included in the target path tree; The root node of the target path tree calculates final average gradient data based on the data blocks received from the child nodes and the data blocks that the root node needs to synchronize based on the target path tree, and sends the final average gradient data to the child nodes of the root node; The j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein 1<j≤M.

2. The gradient data synchronization method for distributed training according to claim 1, characterized in that: The two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

3. The gradient data synchronization method for distributed training according to claim 2, characterized in that: The method of taking N different nodes in the two-dimensional network topology structure as root nodes and traversing the nodes in the two-dimensional network topology structure along directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees includes: Taking nodes corresponding to N vertices in the two-dimensional mesh topology structure as root nodes respectively, the nodes in the two-dimensional mesh topology structure are traversed along directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees.

4. The gradient data synchronization method for distributed training according to claim 3, characterized in that: The method of generating N path trees by traversing the nodes in the two-dimensional mesh topology structure along directed links in the two-dimensional mesh topology structure based on different traversal principles, respectively taking the nodes corresponding to the N vertices in the two-dimensional mesh topology structure as root nodes, includes: The nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

5. The gradient data synchronization method for distributed training according to claim 4, characterized in that: The method comprises taking nodes corresponding to three vertices in the two-dimensional mesh topology structure as root nodes, traversing the nodes in the two-dimensional mesh topology structure along directed links in the two-dimensional mesh topology structure based on different traversal principles, and generating three path trees, including: The nodes corresponding to the vertices at the upper left corner, upper right corner, and lower right corner of the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

6. The gradient data synchronization method for distributed training according to claim 5, characterized in that: The method comprises taking the nodes corresponding to the vertices at the upper left corner, the upper right corner, and the lower right corner of the two-dimensional mesh topology structure as root nodes, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles, and generating three path trees, including: Taking the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the column direction and then along the row direction to generate a first path tree; Taking the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the row direction and then along the column direction to generate a second path tree; Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traverse the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree; Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topology structure.

7. The gradient data synchronization method for distributed training according to claim 6, characterized in that: Taking the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the column direction and then along the row direction to generate a first path tree, including: Accessing a node corresponding to a vertex in an upper left corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as a root node to a first path tree, and adding the root node to a first queue; Determine whether there are unvisited nodes; If so, taking the head node from the first queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topology structure along the column direction and then along the row direction, adding the visited nodes and the corresponding access paths to the first path tree, adding the visited nodes to the end of the first queue in the order of access, and re-entering the step of determining whether there are unvisited nodes; If not, the final generated first path tree is obtained.

8. The gradient data synchronization method for distributed training according to claim 6, characterized in that: Taking the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the row direction and then along the column direction to generate a second path tree, including: Accessing the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as a root node to the second path tree, and adding the root node to the second queue; Determine whether there are unvisited nodes; If so, taking the head node from the second queue as the starting node, sequentially visiting the unvisited nodes in the two-dimensional grid topology structure along the row direction and then along the column direction, adding the visited nodes and the corresponding access paths to the second path tree, adding the visited nodes to the end of the second queue in the order of access, and re-entering the step of determining whether there are unvisited nodes; If not, the final generated second path tree is obtained.

9. The gradient data synchronization method for distributed training according to claim 6, characterized in that: Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree, including: Accessing the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as a root node to the third path tree, and adding the root node to the third queue; Determine whether there are unvisited nodes; If so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor nodes and the directed links between the current node and the neighbor nodes to the third path tree, adding the neighbor nodes to the end of the third queue, and re-entering the step of determining whether there are any unvisited nodes; If not, the final generated third path tree is obtained.

10. The gradient data synchronization method for distributed training according to claim 1, characterized in that: After the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, the system further includes: The nodes in the distributed training system divide the data blocks that need to be synchronized based on each path tree into multiple communication blocks; Accordingly, the data synchronization operation is performed on the target data block based on the target path tree, including: The nodes of the target path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the target path tree; The leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block as follows: calculating average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: calculating final average gradient data based on the data blocks received from the child nodes and its own data blocks that need to be synchronized based on the target path tree, and sending the final average gradient data to the child nodes of the root node; The j-th layer node of the target path tree sequentially and continuously receives the average gradient data corresponding to the plurality of communication blocks from the parent node; wherein 1<j≤M.

11. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the gradient data synchronization method for distributed training as claimed in any one of claims 1 to 10 when executing the computer program.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed, implements the steps of the gradient data synchronization method for distributed training according to any one of claims 1 to 10.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the gradient data synchronization method for distributed training according to any one of claims 1 to 10 when the computer program is executed.

Images