Distributed training gradient data synchronization method and device, medium and product

Abstract

The invention discloses a gradient data synchronization method and device for distributed training, a medium and a product, and relates to the technical field of computers, and the method comprises the steps: obtaining a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure comprises a plurality of nodes and directed links between adjacent nodes; n path trees are generated based on the two-dimensional network topology structure, the N path trees do not contain the same directed link, and a set of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure; a node in the distributed training system divides gradient data to be synchronized into N data blocks, and data synchronization operation is performed on the N data blocks based on N path trees. The gradient data synchronization efficiency in the distributed training process is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a gradient data synchronization method and device, medium and product for distributed training. Background Art

[0002] As the scale of models and distributed systems becomes larger and larger, the proportion of gradient data synchronization communication in the entire training process becomes larger and larger, and gradient data synchronization communication has become a bottleneck problem that needs to be solved in distributed training. In related technologies, gradient data is synchronized through the MultiTree algorithm, but when the MultiTree algorithm is applied to a two-dimensional grid topology, the height of the tree will increase significantly, resulting in increased communication delay and reduced communication efficiency.

[0003] Therefore, how to improve the efficiency of gradient data synchronization during distributed training is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a gradient data synchronization method and device, medium and product for distributed training, which improves the gradient data synchronization efficiency during distributed training.

[0005] To achieve the above object, the present invention provides a gradient data synchronization method for distributed training, comprising:

[0006] Acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0007] Generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure;

[0008] The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0009] The step of generating N path trees based on the two-dimensional network topology structure includes:

[0010] Taking N different nodes in the two-dimensional network topology structure as root nodes respectively, the nodes in the two-dimensional network topology structure are traversed along the directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees.

[0011] Among them, the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

[0012] The method of taking N different nodes in the two-dimensional network topology structure as root nodes, traversing the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles, and generating N path trees includes:

[0013] The nodes corresponding to the N vertices in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees.

[0014] The method of taking nodes corresponding to N vertices in the two-dimensional mesh topology structure as root nodes, traversing nodes in the two-dimensional mesh topology structure along directed links in the two-dimensional mesh topology structure based on different traversal principles, and generating N path trees includes:

[0015] The nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0016] The nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees, including:

[0017] The nodes corresponding to the vertices at the upper left corner, the upper right corner, and the lower right corner in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0018] The nodes corresponding to the vertices at the upper left corner, the upper right corner, and the lower right corner in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees, including:

[0019] Taking the node corresponding to the vertex at the upper left corner in the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the column direction and then along the row direction to generate a first path tree;

[0020] Taking the node corresponding to the vertex at the lower right corner in the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the row direction and then along the column direction to generate a second path tree;

[0021] Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree;

[0022] Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topological structure.

[0023] The node corresponding to the vertex at the upper left corner of the two-dimensional grid topology structure is taken as the root node, and the nodes in the two-dimensional grid topology structure are traversed along the column direction and then along the row direction to generate a first path tree, including:

[0024] Access the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as a root node to the first path tree, and add the root node to the first queue;

[0025] Determine whether there are unvisited nodes;

[0026] If so, take the head node from the first queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure along the column direction and then along the row direction, add the visited nodes and the corresponding access paths to the first path tree, add the visited nodes to the end of the first queue in the access order, and re-enter the step of determining whether there are unvisited nodes;

[0027] If not, the final generated first path tree is obtained.

[0028] The node corresponding to the vertex at the lower right corner of the two-dimensional grid topology structure is taken as the root node, and the nodes in the two-dimensional grid topology structure are traversed along the row direction and then along the column direction to generate a second path tree, including:

[0029] Access the node corresponding to the vertex at the lower right corner in the two-dimensional grid topology structure, add the node corresponding to the vertex at the lower right corner in the two-dimensional grid topology structure as a root node to the second path tree, and add the root node to the second queue;

[0030] Determine whether there are unvisited nodes;

[0031] If so, take the head node from the second queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure in the row direction and then in the column direction, add the visited nodes and the corresponding access paths to the second path tree, add the visited nodes to the end of the second queue in the access order, and re-enter the step of determining whether there are unvisited nodes;

[0032] If not, the final generated second path tree is obtained.

[0033] The method comprises taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle, and generating a third path tree, including:

[0034] Access the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as a root node to the third path tree, and add the root node to the third queue;

[0035] Determine whether there are unvisited nodes;

[0036] If so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor nodes and the directed links between the current node and the neighbor nodes to the third path tree, adding the neighbor nodes to the end of the third queue, and re-entering the step of determining whether there are unvisited nodes;

[0037] If not, the final generated third path tree is obtained.

[0038] The data synchronization operation on the target data block based on the target path tree includes:

[0039] The leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node;

[0040] The i-th layer node of the target path tree calculates average gradient data based on the data blocks received from the child nodes and the data blocks that need to be synchronized based on the target path tree, and sends the average gradient data to its parent node; wherein 1<i<N, N is the number of layers included in the target path tree;

[0041] The root node of the target path tree calculates final average gradient data based on the data blocks received from the child nodes and the data blocks that the root node needs to synchronize based on the target path tree, and sends the final average gradient data to the child nodes of the root node;

[0042] The j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein 1<j≤N.

[0043] After the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, the system further includes:

[0044] The nodes in the distributed training system divide the data blocks that need to be synchronized based on each path tree into multiple communication blocks;

[0045] Accordingly, a data synchronization operation is performed on the target data block based on the target path tree, including:

[0046] The nodes of the target path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the target path tree;

[0047] Among them, the processing process of the leaf node of the target path tree for the current communication block is: sending the current communication block to the parent node of the leaf node; the processing process of the i-th layer node of the target path tree for the current communication block is: calculating the average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the processing process of the root node of the target path tree for the current communication block is: calculating the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sending the final average gradient data to the child node of the root node;

[0048] The j-th layer node of the target path tree receives the average gradient data corresponding to the plurality of communication blocks from the parent node in sequence and continuously; wherein 1<j≤M.

[0049] To achieve the above object, the present invention provides a gradient data synchronization device for distributed training, comprising:

[0050] An acquisition module, used to acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0051] A generating module, configured to generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure;

[0052] The data synchronization module is used to divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0053] To achieve the above object, the present invention provides an electronic device, comprising:

[0054] Memory for storing computer programs;

[0055] A processor is used to implement the steps of the gradient data synchronization method for distributed training as described above when executing the computer program.

[0056] To achieve the above-mentioned object, the present invention provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the gradient data synchronization method for distributed training as described above are implemented.

[0057] To achieve the above objectives, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the gradient data synchronization method for distributed training as described above.

[0058] It can be seen from the above scheme that a gradient data synchronization method for distributed training provided by the present invention includes: obtaining a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes multiple nodes and directed links between adjacent nodes; generating N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure; the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on the N path trees respectively.

[0059] The beneficial effects of the present invention are as follows: the gradient data synchronization method for distributed training provided by the present invention generates N path trees that do not share any directed links based on a two-dimensional network topology structure, and these path trees jointly cover all nodes in the network topology. The path trees constructed in this way can ensure that the height of each path tree is moderate, and effectively control the transmission delay. Since there are no shared directed links between the path trees, the nodes can divide the gradient data to be synchronized into N data blocks, and synchronize the data blocks in parallel based on these path trees, significantly reducing congestion and delays in the data transmission process, and improving the efficiency of gradient data synchronization in the distributed training process. The present invention also discloses a gradient data synchronization device for distributed training, an electronic device, a non-volatile storage medium, and a computer program product, which can also achieve the above-mentioned technical effects.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation methods, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0062] Figure 1 It is a flowchart of a gradient data synchronization method for distributed training according to an exemplary embodiment;

[0063] Figure 2 The present invention is a flowchart of another method for synchronizing gradient data of distributed training according to an exemplary embodiment;

[0064] Figure 3 The present invention is a flowchart of a specific method for generating a topology-aware tree according to an exemplary embodiment;

[0065] Figure 4 It is a schematic diagram of the topology of a 3×3 two-dimensional grid;

[0066] Figure 5 This is a schematic diagram of three initial trees generated using a 3×3 two-dimensional grid as an example;

[0067] Figure 6 Taking a 3×3 two-dimensional grid as an example, the schematic diagram of the three topology-aware trees whose links do not intersect with each other is finally generated;

[0068] Figure 7 It is a flow chart of a pipeline parallel communication block scheduling method according to an exemplary embodiment;

[0069] Figure 8 A schematic diagram showing the performance comparison between a communication block scheduling method that adopts pipeline parallelism and a communication block scheduling method that does not adopt pipeline parallelism;

[0070] Fig. 9 is a structural diagram of a gradient data synchronization device for distributed training according to an exemplary embodiment;

[0071] Fig.10 The figure is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, in the embodiments of the present invention, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0073] 2D-mesh topology is a common physical topology, in which each node is directly connected to its adjacent nodes, forming a regular two-dimensional grid structure. The grid is usually divided into rows and columns, and each node is only connected to a fixed number of nodes around it. This layout helps to simplify communication routing and data transmission, and has good scalability, and can easily expand the number of nodes, so it is widely used in large-scale parallel systems. In addition, the links in the two-dimensional mesh topology in the current physical environment are usually bidirectional links.

[0074] The MultiTree algorithm is a topology-aware and efficient gradient data synchronization method that improves link utilization by constructing a set of topology-aware trees. The construction method of MultiTree is: construct multiple trees according to the physical topology. Assuming that there are n computing nodes in the physical topology, a tree is constructed with each computing node as the root, and a total of n trees are constructed. Each tree finally generated should contain all computing nodes in the topology. In the tree construction process, a top-down approach is adopted, that is, the tree is constructed from the root. For each time_step, links are assigned using the topology graph, the remaining nodes are connected to the spanning tree, and the assigned links are deleted from the graph. When there are no more available links to connect the remaining nodes to any tree, a new topology graph will be used in the next time step.

[0075] When the MultiTree algorithm is applied to a two-dimensional grid topology, the height of the tree will increase significantly, resulting in increased communication delay and reduced communication efficiency. On the other hand, although the physical links used by each tree in each time_step of the tree constructed by the MultiTree algorithm are different, repeated physical links are used between different time_steps. This means that when performing gradient synchronization communication, in order to avoid communication congestion on the link, only one communication block can be processed in the tree at a time, and multiple communication blocks cannot be scheduled simultaneously in each tree in a pipelined parallel manner. This greatly limits the room for improving the efficiency of gradient synchronization communication.

[0076] Therefore, in the present invention, N path trees that do not share any directed links are generated based on the two-dimensional network topology structure. These path trees jointly cover all nodes in the network topology. The path trees constructed in this way can ensure that the height of each path tree is moderate and effectively control the transmission delay. Since there are no shared directed links between the path trees, the nodes can divide the gradient data to be synchronized into N data blocks, and synchronize the data blocks in parallel based on these path trees, significantly reducing congestion and delay in the data transmission process, and improving the efficiency of gradient data synchronization in the distributed training process.

[0077] The embodiment of the present invention discloses a gradient data synchronization method for distributed training, which improves the gradient data synchronization efficiency during the distributed training process.

[0078] See also Figure 1 , a flowchart of a gradient data synchronization method for distributed training according to an exemplary embodiment is shown, such as Figure 1 As shown, including:

[0079] S101: Acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0080] In the specific implementation, during the initialization phase of the distributed training system, a two-dimensional network topology diagram containing multiple nodes and directed links between them is created. This topology diagram describes the connection relationship between the nodes in detail and provides the necessary network architecture information for subsequent data synchronization operations.

[0081] As a feasible implementation, the two-dimensional network topology of the distributed training system is a two-dimensional mesh topology, that is, this embodiment can be applicable to a two-dimensional mesh topology (2D-mesh).

[0082] S102: generating N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and at least two path trees contain all nodes in the two-dimensional network topology structure;

[0083] In the specific implementation, N path trees are generated based on the obtained two-dimensional network topology. These path trees are designed not to share any directed links with each other, ensuring full utilization of network resources and load balancing. Each path tree is independent, but their node collection must cover all nodes in the entire network topology, so that each node can be effectively included in the data synchronization process.

[0084] As a feasible implementation method, the generation of N path trees based on the two-dimensional network topology structure includes: taking N different nodes in the two-dimensional network topology structure as root nodes, traversing the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles, and generating N path trees.

[0085] In the specific implementation, N different nodes are selected as the root nodes of their respective path trees in the two-dimensional network topology, and then all nodes are traversed along the directed links in the network according to a specific traversal principle to construct N independent path trees. Each path tree starts from its root node, and the nodes in the network are connected in a specific order through the traversal process to form a tree structure, thereby ensuring that the nodes and links covered by each path tree are unique, avoiding the reuse of links, and realizing the optimal allocation and utilization of network resources.

[0086] If the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure, as a feasible implementation method, N different nodes in the two-dimensional network topology structure are respectively used as root nodes, and the nodes in the two-dimensional network topology structure are traversed along the directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees, including: taking the nodes corresponding to N vertices in the two-dimensional grid topology structure as root nodes, and traversing the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles to generate N path trees.

[0087] In a specific implementation, when the two-dimensional network topology of the distributed training system is a two-dimensional grid topology, N vertices in the two-dimensional grid topology are selected as the root nodes of each path tree, and then traverse along the directed links in the grid according to their own different traversal principles to build N independent path trees. Each path tree starts from its corresponding vertex node, and connects other nodes in the grid in a specific order through the traversal process to form a tree structure, ensuring that the nodes and links covered by each path tree are unique, thereby optimizing the use of network resources and improving the efficiency of data synchronization.

[0088] As a feasible implementation mode, the nodes corresponding to the N vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees, including: the nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0089] In a specific implementation, in the two-dimensional grid topology of the distributed training system, the nodes corresponding to three specific vertices are selected as root nodes to generate three path trees. The specific implementation method is to traverse along the directed links in the grid topology according to three different traversal principles, and visit each node in the grid in turn. In this way, each root node will generate a unique path tree, and the three path trees together cover all nodes in the entire two-dimensional grid topology, and the directed links in each path tree do not repeat each other, thereby achieving effective allocation of network resources and optimization of data synchronization operations.

[0090] As a feasible implementation mode, the nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees, including: the nodes corresponding to the vertices in the upper left corner, upper right corner, and lower right corner of the two-dimensional mesh topology structure are respectively used as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

[0091] In the specific implementation, in the two-dimensional grid topology of the distributed training system, the three vertices in the upper left corner, upper right corner and lower right corner are selected as the root nodes of their respective path trees. Then, according to their different traversal principles, traverse along the directed links in the grid to construct three independent path trees. Each path tree starts from its corresponding vertex node, and connects other nodes in the grid in a specific order through the traversal process to form a tree structure, ensuring that the nodes and links covered by each path tree are unique, thereby optimizing the use of network resources and improving the efficiency of data synchronization. This method helps to achieve load balancing and reduce communication delays in distributed training. Especially in large-scale computing tasks, this strategy can significantly improve the efficiency of data synchronization and the performance of the entire training process.

[0092] S103: The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0093] In the specific implementation, each node in the distributed training system divides the gradient data to be synchronized into N data blocks, each of which corresponds to a path tree. Then, the system performs data synchronization operations on the N data blocks according to these N path trees. This method allows parallel synchronization, with each path tree responsible for synchronizing a data block, thereby improving the efficiency and speed of data transmission. Especially in large-scale distributed training scenarios, this parallel processing mechanism can significantly reduce the total time required for data synchronization.

[0094] As a feasible implementation method, a data synchronization operation is performed on a target data block based on a target path tree, including: a leaf node of the target path tree sends a data block that needs to be synchronized based on the target path tree to a parent node of the leaf node; an i-th layer node of the target path tree calculates average gradient data based on a data block received from a child node and a data block that needs to be synchronized based on the target path tree, and sends the average gradient data to its parent node; wherein 1<i<M, M is the number of layers contained in the target path tree; a root node of the target path tree calculates final average gradient data based on a data block received from a child node and a data block that needs to be synchronized based on the target path tree, and sends the final average gradient data to a child node of the root node; a j-th layer node of the target path tree receives the final average gradient data from a parent node; wherein 1<j≤M.

[0095] In the specific implementation, each leaf node first sends the data block it is responsible for to its parent node. At the i-th layer of the path tree (1<i<M), each node collects data blocks from its child nodes, calculates the average gradient data based on its own data blocks, and then sends this average value to the parent node of the previous layer. This process continues until the root node, which calculates the final average gradient data based on the data blocks received from all child nodes and sends this result back to the child nodes. At the j-th layer of the path tree (1<j≤M), the node receives the final average gradient data from the parent node and sends it further down. Such a hierarchical aggregation process ensures that in distributed training, gradient data can be efficiently synchronized between nodes, thereby achieving global consistency of model parameters.

[0096] As another feasible implementation, a data synchronization operation is performed on the target data block based on the target path tree, including: the leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node; the i-th layer node of the target path tree sends the data block received from the child node and the data block that needs to be synchronized based on the target path tree to its parent node; wherein 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree calculates the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sends the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein 1<j≤M.

[0097] In a specific implementation, the leaf nodes of the path tree send the data blocks they are responsible for directly to their parent nodes. For non-leaf nodes, they collect data blocks from their child nodes, merge their own data blocks with them, and then send them to their parent nodes. This process is repeated at each layer of the path tree until the root node. The root node summarizes all the data blocks received from the child nodes and calculates the final average gradient data, and then sends this average back to its child nodes. The nodes at the jth layer (1<j≤M) receive the final average gradient data from the parent node. In this implementation, only the root node calculates the average gradient data, and other nodes are only responsible for the transmission of gradient data, reducing the resource usage of other nodes.

[0098] As a preferred implementation, after the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, it also includes: the nodes in the distributed training system divide the data blocks that need to be synchronized based on each of the path trees into multiple communication blocks; accordingly, the target data blocks are synchronized based on the target path tree, including: the nodes of the target path tree sequentially and continuously process the multiple communication blocks that need to be synchronized based on the target path tree; wherein the leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block The process is: based on the current communication block received from the child node and its own current communication block, average gradient data is calculated, and the average gradient data is sent to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, the final average gradient data is calculated, and the final average gradient data is sent to the child node of the root node; the j-th layer node of the target path tree receives the average gradient data corresponding to multiple communication blocks from the parent node in sequence and continuously; wherein, 1<j≤M.

[0099] In the specific implementation, after the nodes in the distributed training system divide the gradient data into N data blocks, they further subdivide each data block into multiple communication blocks and synchronize these communication blocks in each path tree. Specifically, each node will process the communication blocks it is responsible for in turn, and the leaf node will directly send its communication block to the parent node; the non-leaf node will receive the communication block from the child node, and combine it with its own communication block to calculate the average gradient data, and then send the result to their parent node. This process is repeated at each layer of the path tree until the root node. The root node calculates the final average gradient data and sends it back to the child node. In this process, the nodes of each layer (1<j≤M) will continuously receive the average gradient data corresponding to the communication block from the parent node, ensuring the layer-by-layer aggregation and synchronization of the data, thereby realizing the effective synchronization and update of the gradient data in the entire distributed system.

[0100] The gradient data synchronization method for distributed training provided by the embodiment of the present invention generates N path trees that do not share any directed links based on a two-dimensional network topology structure. These path trees jointly cover all nodes in the network topology. The path trees constructed in this way can ensure that the height of each path tree is moderate, and effectively control the transmission delay. Since there is no shared directed link between the path trees, the node can divide the gradient data to be synchronized into N data blocks, and synchronize the data blocks in parallel based on these path trees, significantly reducing congestion and delay in the data transmission process, and improving the efficiency of gradient data synchronization in the distributed training process.

[0101] The embodiment of the present invention discloses a gradient data synchronization method for distributed training. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0102] See also Figure 2 , a flowchart of another method for synchronizing gradient data for distributed training according to an exemplary embodiment is shown, such as Figure 2 As shown, including:

[0103] S201: Obtain a two-dimensional grid topology structure of a distributed training system;

[0104] S202: Taking the node corresponding to the vertex at the upper left corner of the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the column direction and then along the row direction to generate a first path tree;

[0105] In a specific implementation, the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure is taken as the root node, and the tree is first traversed along the column direction and then along the row direction, so as to generate the first path tree.

[0106] As a feasible implementation method, the node corresponding to the vertex in the upper left corner of the two-dimensional grid topological structure is used as the root node, and the nodes in the two-dimensional grid topological structure are traversed first along the column direction and then along the row direction to generate a first path tree, including: visiting the node corresponding to the vertex in the upper left corner of the two-dimensional grid topological structure, adding the node corresponding to the vertex in the upper left corner of the two-dimensional grid topological structure as the root node to the first path tree, and adding the root node to the first queue; judging whether there is an unvisited node; if so, taking the head node from the first queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topological structure in sequence along the column direction and then along the row direction, adding the visited nodes and the corresponding access paths to the first path tree, adding the visited nodes to the end of the first queue in the order of access, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the finally generated first path tree.

[0107] In the specific implementation, first, select the node corresponding to the vertex in the upper left corner of the grid as the root node, and add it to the first path tree and the first queue. Then, check whether there are unvisited nodes in a loop. If so, take out the node at the front of the queue as the starting node, visit the unvisited nodes in the grid in the order of columns first and rows later, and add the visited nodes and their access paths to the first path tree. At the same time, the newly visited nodes will be added to the end of the first queue according to the access order for subsequent processing. This process will be repeated continuously, and there will be no unvisited nodes. At this time, the first path tree has been generated, which contains all nodes visited from the root node in the specified traversal order, and there are no repeated directed links.

[0108] S203: Taking the node corresponding to the vertex at the lower right corner of the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the row direction and then along the column direction to generate a second path tree;

[0109] In a specific implementation, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology is used as the root node, and the second path tree is generated by traversing along the row direction first and then along the column direction. This traversal order helps to create a path tree that is different from the first path tree and covers all nodes.

[0110] As a feasible implementation method, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topological structure is used as the root node, and the nodes in the two-dimensional grid topological structure are traversed in the row direction and then in the column direction to generate a second path tree, including: visiting the node corresponding to the vertex in the lower right corner of the two-dimensional grid topological structure, adding the node corresponding to the vertex in the lower right corner of the two-dimensional grid topological structure as the root node to the second path tree, and adding the root node to the second queue; judging whether there is an unvisited node; if so, taking the head node from the second queue as the starting node, first visiting the unvisited nodes in the two-dimensional grid topological structure in sequence in the row direction and then in the column direction, adding the visited nodes and the corresponding access paths to the second path tree, adding the visited nodes to the end of the second queue in the order of access, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the final generated second path tree.

[0111] In the specific implementation, the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology is used as the root node, and the root node is first added to the second path tree and the second queue. Then, it is cyclically determined whether there are unvisited nodes. If so, the node at the front of the queue is taken out as the starting node, and the unvisited nodes in the grid are visited in the order of rows first and columns later, and the visited nodes and their access paths are added to the second path tree. At the same time, the newly visited nodes will be added to the end of the second queue according to the access order for subsequent processing. This process will be repeated until there are no unvisited nodes. At this time, the second path tree has been generated, which contains all nodes visited from the root node in the specified traversal order, and there are no repeated directed links. This method ensures that the generation of the path tree is orderly and efficient, and provides another structured path for data synchronization in distributed training.

[0112] S204: Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traverse the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree;

[0113] In a specific implementation, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure is used as the root node, and a breadth-first search strategy is adopted to traverse along the directed links to generate the third path tree.

[0114] As a feasible implementation method, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure is used as the root node, and the nodes in the two-dimensional grid topology structure are traversed along the directed links in the two-dimensional grid topology structure based on the breadth-first traversal principle to generate a third path tree, including: visiting the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, adding the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as the root node to the third path tree, and adding the root node to the third queue; judging whether there is an unvisited node; if so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor node and the directed links between the current node and the neighbor node to the third path tree, adding the neighbor node to the end of the third queue, and re-entering the step of judging whether there is an unvisited node; if not, obtaining the finally generated third path tree.

[0115] In the specific implementation, the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology is used as the root node. First, the root node is added to the third path tree and the third queue. Then, the system will cyclically check whether there are unvisited nodes. If so, the node at the front of the queue is taken out as the current node, and all unvisited neighbor nodes are visited. The access order of each neighbor node is not limited here, and these neighbor nodes and the directed links between the current node are added to the third path tree. At the same time, these newly visited neighbor nodes will be added to the end of the third queue for subsequent processing. This process will be repeated until there are no unvisited nodes. At this time, the third path tree has been generated, which includes all nodes and corresponding directed links visited in breadth-first traversal order starting from the root node, ensuring that the path tree covers all nodes and links are not repeated.

[0116] S205: Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topological structure.

[0117] In a specific implementation, after generating the third path tree, the target node corresponding to the lower left corner vertex in the third path tree and the path connected thereto are deleted to determine that the directed links contained in the third path tree do not overlap with the directed links contained in the first path tree and the second path tree.

[0118] S206: The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and divide the data blocks that need to be synchronized based on each path tree into multiple communication blocks;

[0119] In a specific implementation, the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and further subdivide each data block into multiple communication blocks so as to synchronize based on each path tree. Such division helps to improve the efficiency and parallelism of data transmission.

[0120] S207: The nodes of the path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the path tree;

[0121] The processing process of the leaf node of the path tree for the current communication block is: sending the current communication block to the parent node of the leaf node; the processing process of the i-th layer node of the path tree for the current communication block is: calculating the average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node; the processing process of the root node of the path tree for the current communication block is: calculating the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the path tree, and sending the final average gradient data to the child node of the root node;

[0122] In the specific implementation, each node will process multiple communication blocks in sequence and continuously to achieve the synchronization of gradient data. For the leaf nodes in the path tree, the process of processing communication blocks is relatively simple, mainly sending the current communication block directly to its parent node. This direct transmission method is the starting point of data aggregation and ensures the data flow from leaf to root. For non-leaf nodes, especially the nodes in the i-th layer (1<i<M), they need to perform more complex operations. Some nodes first receive communication blocks from their child nodes, and then calculate the average gradient data in combination with the communication blocks they are responsible for. This step is the core of gradient data aggregation, which allows nodes to merge information from multiple data sources to reduce data transmission redundancy and improve synchronization efficiency. After the calculation is completed, these average gradient data will be sent to the parent node of the previous layer for further aggregation. The root node of the path tree is responsible for the final aggregation operation. It calculates the final average gradient data of the entire path tree based on the data blocks received from all child nodes and the data blocks it is responsible for. This final result is then sent back to the child nodes of the root node, completing the data distribution process from root to leaf.

[0123] That is, the three path trees simultaneously execute the ScatterReduce operation in a pipeline manner starting from the leaf nodes. After each node processes a communication block, it sends it to its parent node, and then immediately receives the next communication block for processing until all the data to be synchronized are processed.

[0124] S208: The j-th layer node of the path tree sequentially and continuously receives the average gradient data corresponding to the plurality of communication blocks from the parent node.

[0125] In the specific implementation, the nodes in the jth layer of the path tree (1<j≤M) receive the average gradient data from the parent node. These nodes will receive the average gradient data corresponding to multiple communication blocks in sequence and continuously, which is the key link in the data synchronization process from the root node to the leaf node. In this way, each node can obtain the average gradient information of the entire path tree, thereby ensuring that the gradient data of all nodes in distributed training remain consistent.

[0126] That is, the three path trees simultaneously execute the Allgather operation in a pipeline manner starting from the root node. After each node processes a communication block, it sends it to its leaf node, and then immediately receives the next communication block for processing until all the data to be synchronized are processed.

[0127] It can be seen that this embodiment adopts a pipeline-parallel communication block scheduling method, which effectively improves the efficiency of gradient data synchronization and alleviates the bottleneck problem caused by gradient data synchronization in large-scale data synchronization training.

[0128] The following describes an application embodiment provided by the present invention, in which an m×n two-dimensional grid is generated into three non-intersecting trees, each of which is generated using a node at a corner of the two-dimensional grid as a tree root. The specific generation method of the topology-aware tree is as follows: Figure 3 As shown, the following steps are included:

[0129] Step 1: First, initialize the numbering of the m×n two-dimensional grid topology: serialize the numbering of the nodes and physical links in the topology. Taking the 3×3 grid as an example, the numbered two-dimensional grid is as follows Figure 4 shown.

[0130] Step 2: Initialize the root nodes of the three trees to 1, n, and nm respectively: For an m×n grid, designate the nodes numbered 1, n, and nm (the upper left node, upper right node, and lower right node of the two-dimensional grid, respectively) as the roots of the three trees.

[0131] Step 3: Construct each tree in turn:

[0132] Generate a tree with node 1 as the root. With node 1 as the root, connect other nodes in the grid to the tree along each directed link in the two-dimensional grid topology. Specifically, with node 1 as the starting point, first connect each node along the y-axis in the two-dimensional grid topology, then connect other nodes along the x-axis until all nodes in the grid are connected to the tree.

[0133] Generate a tree with node nm as the root, and connect other nodes in the grid to the tree along each directed link in the two-dimensional grid topology. Specifically, with node nm as the starting point, first connect each node along the x-axis in the two-dimensional grid topology, and then connect other nodes along the y-axis until all nodes in the grid are connected to the tree.

[0134] Generate a tree with node n as the root, and connect other nodes in the grid to the tree along the directed links in the two-dimensional grid topology. Specifically, starting from node n, traverse each node in the two-dimensional grid according to the principle of breadth-first traversal, and add each node in the two-dimensional grid topology to the current tree in turn.

[0135] by Figure 4 Taking the 3×3 scale two-dimensional grid as an example, the three trees generated are as follows Figure 5 shown.

[0136] The process of generating a tree with node 1 as the root is as follows: visit node 1, add node 1 as the root node to tree 1, and add node 1 to the first queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 1, from the first queue as the starting node. First, visit node 4 and node 7 in the column direction, and then visit node 2 and node 3 in the row direction. Add node 4, node 7, node 2, node 3 and the corresponding access path to tree 1, and add node 4, node 7, node 2, node 3 to the end of the first queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 4, from the first queue. Since node 4 is the starting node in the column direction, There are no unvisited nodes, so starting from node 4, node 5 and node 6 are visited in turn along the row direction, and node 5, node 6 and the corresponding access path are added to tree 1. Node 5 and node 6 are added to the end of the first queue in turn. At this time, there are unvisited nodes in the two-dimensional grid. The first node, that is, node 7, is taken out from the first queue. Since there are no unvisited nodes along the column direction starting from node 7, node 8 and node 9 are visited in turn along the row direction starting from node 7. Node 8, node 9 and the corresponding access path are added to tree 1. Node 8 and node 9 are added to the end of the first queue in turn. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 1 is obtained.

[0137] The process of generating a tree with node 9 as the root is as follows: visit node 9, add node 9 as the root node to tree 2, and add node 9 to the second queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node, that is, node 9, from the second column as the starting node. First, visit node 8 and node 7 in the row direction, and then visit node 6 and node 3 in the column direction. Add node 8, node 7, node 6, node 3 and the corresponding access path to tree 2, and add node 8, node 7, node 6, node 3 to the end of the second queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid. Take the first node, that is, node 8, from the second queue. Since node 8 is the starting node in the row direction, There are no unvisited nodes, so starting from node 8, node 5 and node 2 are visited in sequence along the column direction, and node 5, node 2 and the corresponding access path are added to tree 2. Node 5 and node 2 are added to the end of the second queue in sequence. At this time, there are unvisited nodes in the two-dimensional grid. The head node, that is, node 7, is taken out from the second queue. Since there are no unvisited nodes along the row direction starting from node 7, node 4 and node 1 are visited in sequence along the column direction starting from node 7. Node 4, node 1 and the corresponding access path are added to tree 2. Node 4 and node 1 are added to the end of the second queue in sequence. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 2 is obtained.

[0138] The process of generating a tree with node 3 as the root is as follows: visit node 3, add node 3 as the root node to tree 3, and add node 3 to the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 3, from the third queue, visit all unvisited neighbor nodes of node 3, node 2, node 6, add node 2, node 6 and the directed link between node 3 and node 2, and the directed link between node 3 and node 6 to tree 3, and add node 2 and node 6 to the end of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 2, from the third queue, visit all unvisited neighbor nodes of node 2, node 5, node 1, take node 5, node 1 and the directed link between node 2 and node 5, and the directed link between node 2 and node 1 to tree 3, and add node 5 and node 1 to the end of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. Take the head node, that is, node 6, from the third queue, visit all unvisited neighbor nodes of node 6, node 9, add node 9 and Directed link 9 between node 6 and node 9 is added to tree 3, and node 9 is added to the tail of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. The head node, that is, node 5, is taken out from the third queue, and all unvisited neighbor nodes of node 5, node 4 and node 8, are visited. Node 4, node 8 and the directed link between node 5 and node 4, and the directed link between node 5 and node 8 are added to tree 3, and nodes 4 and node 8 are added to the tail of the third queue. At this time, there are unvisited nodes in the two-dimensional grid. The head node, that is, node 1, is taken out from the third queue. Since node 1 does not have any unvisited neighbor nodes, the head node, that is, node 9, is continued to be taken out from the third queue. Since node 9 does not have any unvisited neighbor nodes, the head node, that is, node 4, is continued to be taken out from the third queue. All unvisited neighbor nodes of node 4, node 7, are visited. Node 7 and the directed link between node 4 and node 7 are added to tree 3, and node 7 is added to the tail of the third queue. At this time, there are no unvisited nodes in the two-dimensional grid, and the generated tree 3 is obtained.

[0139] Step 4: Observe the three generated trees and find that the first and third trees reuse the link (16) between node 4 and node 7. Finally, in order to make the three generated trees disjoint, that is, to have no reused links, remove the node numbered n(m-1)+1 from the tree with root node n.

[0140] Taking a 3×3 2D grid as an example, remove node 7 from the tree with root node 3. The three topology-aware trees with disjoint links are finally generated as follows: Figure 6 shown.

[0141] Based on the non-intersecting topology perception tree generated above, gradient data synchronization is performed. The gradient data synchronization process is mainly the scheduling process of multiple communication blocks. A pipeline parallel communication block scheduling method is proposed. The specific implementation steps are as follows: Figure 7 As shown, the following steps are included:

[0142] Step 1: Divide the data to be synchronized into 3 parts equally, and assign each part to a tree for processing.

[0143] Step 2: On each tree, divide the data to be processed on the tree into q communication blocks (q can be set according to the actual situation). Assuming the size of the data to be synchronized is P, the size of each communication block is P / 3q.

[0144] Step 3: Pipeline operation in the ScatterReduce phase: The three topology-aware trees simultaneously execute the ScatterReduce operation starting from the leaf nodes in a pipeline manner. That is, each node sends a communication block to its parent node after processing it, and then immediately receives the next communication block for processing until all the data to be synchronized is processed.

[0145] Step 4: Allgather (all collection) phase pipeline operation: After the ScatterReduce phase is completed, the three topology-aware trees perform the Allgather operation in a pipeline manner starting from the root node. That is, after each node processes a communication block, it sends it to its leaf node, and then immediately receives the next communication block for processing until all the data to be synchronized are processed.

[0146] It can be seen that this embodiment of the present invention proposes a gradient data synchronization method for two-dimensional grid topology. By adopting a new method to construct a topology tree, the height of each constructed tree is moderate, which effectively reduces the transmission delay. There are no shared physical links between the generated trees and between the time steps, which makes it possible to simultaneously schedule multiple communication blocks in each tree in a pipeline parallel manner. In addition, in the process of generating the topology-aware tree, almost all links in the two-dimensional grid topology are fully utilized, which greatly improves the utilization rate of physical links compared with the existing methods. Next, based on the constructed topology-aware tree with non-intersecting links, we proposed a pipeline parallel communication block scheduling method. Through pipeline parallelism, the communication time is effectively reduced and the data synchronization efficiency in the two-dimensional grid topology is improved. The performance comparison diagram of the communication block scheduling method using pipeline parallelism and the communication block scheduling method not using pipeline parallelism is shown in the figure. Figure 8 As shown, it can be seen that the method of the present invention effectively improves the efficiency of gradient data synchronization and alleviates the bottleneck problem caused by gradient data synchronization in large-scale data synchronization training.

[0147] A gradient data synchronization device for distributed training provided by an embodiment of the present invention is introduced below. The gradient data synchronization device for distributed training described below and the gradient data synchronization method for distributed training described above can be referenced to each other.

[0148] See also Fig. 9 , according to an exemplary embodiment, a structural diagram of a gradient data synchronization device for distributed training is shown, such as Fig. 9 As shown, including:

[0149] The acquisition module 100 is used to acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes;

[0150] A generating module 200, configured to generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure;

[0151] The data synchronization module 300 is used to divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

[0152] The gradient data synchronization device for distributed training provided by the embodiment of the present invention generates N path trees that do not share any directed links based on a two-dimensional network topology structure. These path trees jointly cover all nodes in the network topology. The path trees constructed in this way can ensure that the height of each path tree is moderate, and effectively control the transmission delay. Since there is no shared directed link between the path trees, the node can divide the gradient data to be synchronized into N data blocks, and synchronize the data blocks in parallel based on these path trees, significantly reducing congestion and delay in the data transmission process, and improving the efficiency of gradient data synchronization in the distributed training process.

[0153] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: use N different nodes in the two-dimensional network topology structure as root nodes, traverse the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles, and generate N path trees.

[0154] Based on the above embodiments, as a preferred implementation, the two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

[0155] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: take the nodes corresponding to the N vertices in the two-dimensional grid topology structure as root nodes, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles, and generate N path trees.

[0156] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: take the nodes corresponding to the three vertices in the two-dimensional mesh topology structure as root nodes, traverse the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles, and generate three path trees.

[0157] Based on the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: use the nodes corresponding to the vertices in the upper left corner, upper right corner, and lower right corner of the two-dimensional grid topology structure as root nodes, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on different traversal principles, and generate three path trees.

[0158] On the basis of the above embodiment, as a preferred implementation mode, the generation module 200 is specifically used to: take the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as the root node, first traverse the nodes in the two-dimensional grid topology structure along the column direction and then along the row direction to generate a first path tree; take the node corresponding to the vertex in the lower right corner of the two-dimensional grid topology structure as the root node, first traverse the nodes in the two-dimensional grid topology structure along the row direction and then along the column direction to generate a second path tree; take the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as the root node, traverse the nodes in the two-dimensional grid topology structure along the directed links in the two-dimensional grid topology structure based on the breadth-first traversal principle to generate a third path tree; delete the target node and the path connected to the target node in the third path tree; wherein, the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topology structure.

[0159] On the basis of the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: visit the node corresponding to the vertex in the upper left corner of the two-dimensional grid topological structure, add the node corresponding to the vertex in the upper left corner of the two-dimensional grid topological structure as a root node to the first path tree, and add the root node to the first queue; determine whether there are unvisited nodes; if so, take out the head node from the first queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topological structure in sequence along the column direction and then along the row direction, add the visited nodes and the corresponding access paths to the first path tree, add the visited nodes to the end of the first queue in the order of access, and re-enter the step of determining whether there are unvisited nodes; if not, obtain the finally generated first path tree.

[0160] On the basis of the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: visit the node corresponding to the vertex in the lower right corner of the two-dimensional grid topological structure, add the node corresponding to the vertex in the lower right corner of the two-dimensional grid topological structure as a root node to the second path tree, and add the root node to the second queue; determine whether there is an unvisited node; if so, take out the head node from the second queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topological structure in the row direction and then in the column direction, add the visited nodes and the corresponding access paths to the second path tree, add the visited nodes to the end of the second queue in the order of access, and re-enter the step of determining whether there is an unvisited node; if not, obtain the final generated second path tree.

[0161] On the basis of the above embodiments, as a preferred implementation mode, the generation module 200 is specifically used to: visit the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as a root node to the third path tree, and add the root node to the third queue; determine whether there is an unvisited node; if so, take out the head node from the third queue as the current node, visit all unvisited neighbor nodes of the current node, add the neighbor node and the directed link between the current node and the neighbor node to the third path tree, add the neighbor node to the end of the third queue, and re-enter the step of determining whether there is an unvisited node; if not, obtain the final generated third path tree.

[0162] On the basis of the above embodiments, as a preferred implementation mode, the leaf node of the target path tree sends the data block that it needs to synchronize based on the target path tree to the parent node of the leaf node; the i-th layer node of the target path tree calculates the average gradient data based on the data block received from the child node and the data block that it needs to synchronize based on the target path tree, and sends the average gradient data to its parent node; wherein, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree calculates the final average gradient data based on the data block received from the child node and the data block that it needs to synchronize based on the target path tree, and sends the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein, 1<j≤M.

[0163] Based on the above embodiment, as a preferred implementation, it also includes:

[0164] A division module, used for dividing the data block which needs to be synchronized based on each of the path trees into a plurality of communication blocks;

[0165] Correspondingly, the nodes of the target path tree process multiple communication blocks that they need to synchronize based on the target path tree in sequence and continuously; wherein, the leaf node of the target path tree processes the current communication block as follows: sending the current communication block to the parent node of the leaf node; the i-th layer node of the target path tree processes the current communication block as follows: calculating average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the root node of the target path tree processes the current communication block as follows: calculating the final average gradient data based on the data blocks received from the child node and the data blocks that it needs to synchronize based on the target path tree, and sending the final average gradient data to the child node of the root node; the j-th layer node of the target path tree receives the average gradient data corresponding to multiple communication blocks from the parent node in sequence and continuously; wherein, 1<j≤M.

[0166] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0167] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device, Fig.10 FIG. 1 is a structural diagram of an electronic device according to an exemplary embodiment. Fig.10 As shown, the electronic equipment includes:

[0168] Communication interface 1, capable of exchanging information with other devices such as network devices;

[0169] The processor 2 is connected to the communication interface 1 to realize information interaction with other devices, and is used to execute the gradient data synchronization method of distributed training provided by one or more technical solutions when running a computer program. The computer program is stored in the memory 3.

[0170] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 4.

[0171] The memory 3 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0172] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 3 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0173] The method disclosed in the above embodiment of the present invention can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 2 or the instruction in the form of software. The above processor 2 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and completes the steps of the above method in combination with its hardware.

[0174] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present invention are implemented, which will not be described here for the sake of brevity.

[0175] In an exemplary embodiment, the embodiment of the present invention further provides a non-volatile storage medium storing a computer program, which can be executed by the processor 2 to complete the steps of the aforementioned method.

[0176] In an exemplary embodiment, the present invention further provides a computer program product, including a computer program, which is executed by the processor 2 to complete the steps of the aforementioned method.

[0177] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to computer program instructions, and the aforementioned computer program can be stored in a non-volatile storage medium. When the computer program is executed, it executes the steps of the above method embodiments. Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a non-volatile storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention.

[0178] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A gradient data synchronization method for distributed training, characterized in that: include: Acquire a two-dimensional network topology structure of a distributed training system; wherein the two-dimensional network topology structure includes a plurality of nodes and directed links between adjacent nodes; Generate N path trees based on the two-dimensional network topology structure, wherein the N path trees do not contain the same directed links, and the collection of nodes contained in the N path trees covers all nodes in the two-dimensional network topology structure; The nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, and perform data synchronization operations on the N data blocks based on N path trees respectively.

2. The gradient data synchronization method for distributed training according to claim 1, characterized in that: The generating N path trees based on the two-dimensional network topology structure includes: Taking N different nodes in the two-dimensional network topology structure as root nodes respectively, the nodes in the two-dimensional network topology structure are traversed along the directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees.

3. The gradient data synchronization method for distributed training according to claim 2, characterized in that: The two-dimensional network topology structure of the distributed training system is a two-dimensional grid topology structure.

4. The gradient data synchronization method for distributed training according to claim 3, characterized in that: The method of taking N different nodes in the two-dimensional network topology structure as root nodes and traversing the nodes in the two-dimensional network topology structure along the directed links in the two-dimensional network topology structure based on different traversal principles to generate N path trees includes: The nodes corresponding to the N vertices in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate N path trees.

5. The method for synchronizing gradient data for distributed training according to claim 4, characterized in that: The method of taking nodes corresponding to N vertices in the two-dimensional mesh topology structure as root nodes, traversing nodes in the two-dimensional mesh topology structure along directed links in the two-dimensional mesh topology structure based on different traversal principles, and generating N path trees includes: The nodes corresponding to the three vertices in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

6. The gradient data synchronization method for distributed training according to claim 5, characterized in that: The method of taking the nodes corresponding to the three vertices in the two-dimensional mesh topology structure as root nodes and traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees includes: The nodes corresponding to the vertices at the upper left corner, the upper right corner, and the lower right corner in the two-dimensional mesh topology structure are respectively taken as root nodes, and the nodes in the two-dimensional mesh topology structure are traversed along the directed links in the two-dimensional mesh topology structure based on different traversal principles to generate three path trees.

7. The method for synchronizing gradient data for distributed training according to claim 6, characterized in that: The method uses the nodes corresponding to the vertices at the upper left corner, the upper right corner, and the lower right corner of the two-dimensional mesh topology structure as root nodes, traverses the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on different traversal principles, and generates three path trees, including: Taking the node corresponding to the vertex at the upper left corner in the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the column direction and then along the row direction to generate a first path tree; Taking the node corresponding to the vertex at the lower right corner in the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the row direction and then along the column direction to generate a second path tree; Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle to generate a third path tree; Delete the target node and the path connected to the target node in the third path tree; wherein the target node is the node corresponding to the vertex in the lower left corner of the two-dimensional grid topological structure.

8. The gradient data synchronization method for distributed training according to claim 7, characterized in that: Taking the node corresponding to the vertex at the upper left corner in the two-dimensional grid topological structure as the root node, traversing the nodes in the two-dimensional grid topological structure first along the column direction and then along the row direction to generate a first path tree, including: Access the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper left corner of the two-dimensional grid topology structure as a root node to the first path tree, and add the root node to the first queue; Determine whether there are unvisited nodes; If so, take the head node from the first queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure along the column direction and then along the row direction, add the visited nodes and the corresponding access paths to the first path tree, add the visited nodes to the end of the first queue in the access order, and re-enter the step of determining whether there are unvisited nodes; If not, the final generated first path tree is obtained.

9. The method for synchronizing gradient data for distributed training according to claim 7, characterized in that: Taking the node corresponding to the vertex at the lower right corner in the two-dimensional grid topology structure as the root node, traversing the nodes in the two-dimensional grid topology structure first along the row direction and then along the column direction to generate a second path tree, including: Access the node corresponding to the vertex at the lower right corner in the two-dimensional grid topology structure, add the node corresponding to the vertex at the lower right corner in the two-dimensional grid topology structure as a root node to the second path tree, and add the root node to the second queue; Determine whether there are unvisited nodes; If so, take the head node from the second queue as the starting node, first visit the unvisited nodes in the two-dimensional grid topology structure in the row direction and then in the column direction, add the visited nodes and the corresponding access paths to the second path tree, add the visited nodes to the end of the second queue in the access order, and re-enter the step of determining whether there are unvisited nodes; If not, the final generated second path tree is obtained.

10. The gradient data synchronization method for distributed training according to claim 7, characterized in that: Taking the node corresponding to the vertex in the upper right corner of the two-dimensional mesh topology structure as the root node, traversing the nodes in the two-dimensional mesh topology structure along the directed links in the two-dimensional mesh topology structure based on the breadth-first traversal principle, generating a third path tree, including: Access the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure, add the node corresponding to the vertex in the upper right corner of the two-dimensional grid topology structure as a root node to the third path tree, and add the root node to the third queue; Determine whether there are unvisited nodes; If so, taking the head node from the third queue as the current node, visiting all unvisited neighbor nodes of the current node, adding the neighbor nodes and the directed links between the current node and the neighbor nodes to the third path tree, adding the neighbor nodes to the end of the third queue, and re-entering the step of determining whether there are unvisited nodes; If not, the final generated third path tree is obtained.

11. The method for synchronizing gradient data for distributed training according to claim 1, characterized in that: Perform data synchronization operations on the target data block based on the target path tree, including: The leaf node of the target path tree sends the data block that needs to be synchronized based on the target path tree to the parent node of the leaf node; The i-th layer node of the target path tree calculates average gradient data based on the data blocks received from the child nodes and the data blocks that need to be synchronized based on the target path tree, and sends the average gradient data to its parent node; wherein 1<i<M, M is the number of layers included in the target path tree; The root node of the target path tree calculates final average gradient data based on the data blocks received from the child nodes and the data blocks that the root node needs to synchronize based on the target path tree, and sends the final average gradient data to the child nodes of the root node; The j-th layer node of the target path tree receives the final average gradient data from the parent node; wherein 1<j≤M.

12. The gradient data synchronization method for distributed training according to claim 1, characterized in that: After the nodes in the distributed training system divide the gradient data to be synchronized into N data blocks, the system further includes: The nodes in the distributed training system divide the data blocks that need to be synchronized based on each path tree into multiple communication blocks; Accordingly, a data synchronization operation is performed on the target data block based on the target path tree, including: The nodes of the target path tree sequentially and continuously process a plurality of communication blocks that need to be synchronized based on the target path tree; Among them, the processing process of the leaf node of the target path tree for the current communication block is: sending the current communication block to the parent node of the leaf node; the processing process of the i-th layer node of the target path tree for the current communication block is: calculating the average gradient data based on the current communication block received from the child node and its own current communication block, and sending the average gradient data to its own parent node, 1<i<M, M is the number of layers contained in the target path tree; the processing process of the root node of the target path tree for the current communication block is: calculating the final average gradient data based on the data block received from the child node and the data block that needs to be synchronized based on the target path tree, and sending the final average gradient data to the child node of the root node; The j-th layer node of the target path tree receives the average gradient data corresponding to the plurality of communication blocks from the parent node in sequence and continuously; wherein 1<j≤M.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the gradient data synchronization method for distributed training as claimed in any one of claims 1 to 12 when executing the computer program.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed, implements the steps of the gradient data synchronization method for distributed training as claimed in any one of claims 1 to 12.

15. A computer program product, characterized in that It comprises a computer program, which, when executed, implements the steps of the gradient data synchronization method for distributed training as claimed in any one of claims 1 to 12.

Images