Data synchronization method and apparatus, computer program product, and electronic device

Abstract

The present disclosure relates to a data synchronization method and device, a computer program product and an electronic device, and relates to the technical field of computers. The method comprises: obtaining a computing node, generating a node grid of the computing node, obtaining a corner node in the node grid, and determining the corner node as a root node of a tree; initializing the root node, generating a node tree based on the root node and a computing node associated with the corner node in the node grid; and synchronizing data based on nodes in the node tree. The present disclosure reduces communication delay between nodes and improves data transmission efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data synchronization method and device, a computer program product, and an electronic device. Background Technology

[0002] In training large models, data-parallel training is one of the main methods for handling large model datasets. Parallel training involves dividing the data into mini-batches and distributing them across multiple computing nodes. After each node independently computes the gradient, the gradients are synchronized using the AllReduce algorithm to ensure that the model remains consistent across all nodes. AllReduce is a commonly used operation in distributed computing for exchanging and aggregating data between different computing nodes. Its function is to summarize and synchronize the data on each computing node to achieve global data consistency across the entire cluster.

[0003] Currently, there are various methods for achieving data synchronization, such as ring-based AllReduce and attribute-structured algorithms. These algorithms can reduce communication overhead; however, when scaled to more complex network architectures, network bandwidth limitations significantly increase training time, making communication latency the main bottleneck for training speed.

[0004] Therefore, a new data synchronization method is needed.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a data synchronization method, data synchronization device, computer program product, and electronic device, thereby overcoming, at least to some extent, the problem of significantly increased training time caused by limitations and defects in related technologies.

[0007] According to one aspect of this disclosure, a data synchronization method is provided, comprising:

[0008] Obtain computing nodes, generate a node grid for the computing nodes, obtain corner nodes in the node grid, and determine the corner nodes as the root nodes of the tree;

[0009] The root node is initialized, and a node tree is generated based on the root node and the compute nodes in the node mesh associated with the corner node;

[0010] Data is synchronized based on the nodes in the node tree.

[0011] In one exemplary embodiment of this disclosure, the steps of obtaining computing nodes, generating a node mesh of the nodes, obtaining corner nodes in the node mesh, and determining the corner nodes as the root nodes of the tree include:

[0012] Determine the dimensions of the node mesh, and generate the node mesh of the computing node based on the dimensions;

[0013] Obtain the corner nodes included in the node grid, and determine the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of the tree.

[0014] In one exemplary embodiment of this disclosure, obtaining the corner nodes included in the node mesh, and determining the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of the tree, includes:

[0015] The first computation node in the node grid is determined as the first corner node;

[0016] The last computation node in the node grid is determined based on the dimension of the node grid, and the last computation node in the node grid is determined as the second corner node;

[0017] Determine any one of the remaining corner nodes as the third corner node;

[0018] The first corner node, the second corner node, and the third corner node are determined as the root node of the tree.

[0019] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0020] Using the first corner node as the root node, and based on the first direction of the first corner node in the node mesh, determine the first computing node connected to the first corner node;

[0021] Based on the second direction of the first computing node in the node grid, a second computing node connected to the first computing node is determined;

[0022] Based on the first corner node, the first computed node, and the second computed node, generate a first node tree with the first corner node as the root node.

[0023] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0024] Using the second corner node as the root node, a third computing node connected to the second corner node is determined based on the third direction of the second corner node in the node mesh;

[0025] Based on the fourth direction of the third computing node in the node grid, a fourth computing node connected to the third computing node is determined;

[0026] Based on the second corner node, the third computed node, and the fourth computed node, a second node tree is generated with the second corner node as the root node.

[0027] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0028] The fifth computation node is obtained by performing a breadth-first traversal based on the third corner node.

[0029] Based on the third corner node and the fifth computed node, a third node tree is generated with the third corner node as the root node.

[0030] In one exemplary embodiment of this disclosure, the step of performing a breadth-first traversal based on the third corner node to obtain the fifth computed node includes:

[0031] Based on the third corner node, a breadth-first traversal is performed on the node mesh to obtain the computational nodes associated with the third corner node;

[0032] The sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing node; wherein, the sixth computing node is a corner node in the node mesh other than the first corner node, the second corner node and the third corner node.

[0033] According to one aspect of this disclosure, a data synchronization apparatus is provided, comprising:

[0034] The root node determination module is used to obtain computing nodes, generate a node grid of the computing nodes, obtain corner nodes in the node grid, and determine the corner nodes as the root nodes of the tree.

[0035] The node tree generation module is used to initialize the root node and generate a node tree based on the root node and the compute nodes in the node mesh associated with the corner node;

[0036] The data synchronization module is used to synchronize data based on the nodes in the node tree.

[0037] According to one aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data synchronization method described in any of the exemplary embodiments above.

[0038] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0039] A processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data synchronization method described in any of the exemplary embodiments above by executing the executable instructions.

[0040] This disclosure provides a data synchronization method that involves acquiring computing nodes, generating a node grid of the computing nodes, acquiring corner nodes in the node grid, and determining the corner nodes as the root nodes of a tree; initializing the root nodes, and generating a node tree based on the root nodes and the computing nodes associated with the corner nodes in the node grid; and synchronizing data based on the nodes in the node tree. On one hand, acquiring corner nodes in the node grid and constructing a node tree based on them, and using this node tree for collective communication scheduling decisions, improves link bandwidth utilization, reduces communication latency between nodes, and increases data transmission efficiency. On the other hand, the node tree constructed using corner nodes is a disjoint tree, which improves the utilization rate of available connections and avoids contention during broadcast and aggregation blocks.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0043] Figure 1 A flowchart illustrating a data synchronization method according to an example embodiment of the present disclosure is shown schematically.

[0044] Figure 2 The flowchart illustrates a method for generating a node mesh according to an exemplary embodiment of the present disclosure, obtaining corner nodes in the node mesh, and determining the corner nodes as the root nodes of a tree.

[0045] Figure 3The flowchart illustrates a method for obtaining corner nodes included in a node mesh according to an exemplary embodiment of the present disclosure, and determining the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of a tree.

[0046] Figure 4 A schematic diagram of a node mesh according to an exemplary embodiment of the present disclosure is shown.

[0047] Figure 5 The flowchart illustrates a method for generating a node tree based on a root node and compute nodes associated with corner nodes in a node mesh, according to an exemplary embodiment of the present disclosure.

[0048] Figure 6 The flowchart illustrates a method for generating a node tree based on a root node and compute nodes associated with corner nodes in a node mesh, according to an exemplary embodiment of the present disclosure.

[0049] Figure 7 The flowchart illustrates a method for generating a node tree based on a root node and compute nodes associated with corner nodes in a node mesh, according to an exemplary embodiment of the present disclosure.

[0050] Figure 8 The flowchart illustrates a method for obtaining a fifth computed node by performing a breadth-first traversal based on a third corner node according to an example embodiment of the present disclosure.

[0051] Figure 9 A block diagram of a data synchronization apparatus according to an exemplary embodiment of the present disclosure is shown schematically.

[0052] Figure 10 An electronic device for implementing the data synchronization method described above is illustrated according to an example embodiment of the present disclosure. Detailed Implementation

[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0054] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0055] Data-parallel training is one of the main methods for handling large datasets when training large models. Current mainstream techniques typically use backpropagation to optimize the weights and biases of deep neural networks. In this process, the model undergoes the following steps:

[0056] Forward propagation: Input data is processed through multiple layers of neurons to generate output results.

[0057] Loss function calculation: Calculate the loss value based on the difference between the model's output and the true label.

[0058] Gradient calculation: The gradient of the loss function with respect to each parameter is calculated using the backpropagation algorithm and the chain rule.

[0059] Parameter update: The model parameters are updated based on the calculated gradient using optimization algorithms such as stochastic gradient descent (SGD).

[0060] To improve training efficiency, especially when dealing with large-scale data, data-parallel distributed training is widely used. This process typically involves dividing the data into mini-batches and distributing them across multiple computing nodes. After each node independently computes the gradient, these gradients are synchronized using the AllReduce algorithm to ensure model consistency across all nodes. AllReduce is a common operation in distributed computing used for exchanging and aggregating data between different computing nodes. Its role is to summarize and synchronize the data on each computing node to achieve global data consistency across the entire cluster. This operation is commonly used in scenarios requiring massively parallel computing, such as deep learning training.

[0061] Currently, there are various algorithms for implementing AllReduce, such as ring-based AllReduce and tree-structured algorithms. Ring-based AllReduce utilizes a ring topology, ensuring each node accesses the network only once, thus reducing communication overhead. However, these algorithms may face performance bottlenecks when scaling to more complex network architectures. Although existing distributed training schemes have improved training efficiency to some extent, some technical challenges remain:

[0062] Communication bottleneck: In large-scale distributed training, frequent data exchange between nodes makes communication latency a major bottleneck to training speed. Especially with high-dimensional data and large-scale models, network bandwidth limitations significantly increase training time.

[0063] Uneven load: In some cases, the computing power and data volume of different nodes may be uneven, causing some nodes to be overloaded while other nodes are idle, thus affecting the overall training efficiency.

[0064] Gradient synchronization delay: When using the AllReduce algorithm, gradient synchronization between nodes takes time, especially when the network topology is not ideal, which may cause parameter updates to lag during the training process.

[0065] Scalability issues: Existing AllReduce algorithms often fail to achieve the expected performance improvement when scaling to more nodes, partly because the algorithm design does not adequately consider the complexity of the network structure.

[0066] Among the aforementioned problems, the most significant technical challenge is the communication bottleneck. As model size and data volume continue to grow, effectively reducing communication latency between nodes and improving data transmission efficiency remains a key challenge in deep learning research.

[0067] To address one or more of the aforementioned issues, this exemplary embodiment first provides a data synchronization method that can run on a terminal device. Of course, those skilled in the art can also run the method of this invention on other platforms as needed, and this exemplary embodiment does not impose any special limitations on this. (Reference) Figure 1 As shown, the data synchronization method may include steps S110-S130:

[0068] Step S110. Obtain computing nodes, generate a node mesh for the computing nodes, obtain corner nodes in the node mesh, and determine the corner nodes as the root nodes of the tree;

[0069] Step S120. Initialize the root node and generate a node tree based on the root node and the compute nodes in the node mesh associated with the corner node;

[0070] Step S130. Synchronize the data based on the nodes in the node tree.

[0071] The aforementioned data synchronization method involves acquiring computing nodes, generating a node grid of the computing nodes, acquiring corner nodes within the node grid, and determining the corner nodes as the root nodes of a tree; initializing the root nodes; generating a node tree based on the root nodes and the computing nodes associated with the corner nodes in the node grid; and synchronizing data based on the nodes in the node tree. On one hand, acquiring corner nodes from the node grid and constructing a node tree based on them, and using this node tree for collective communication scheduling decisions, improves link bandwidth utilization, reduces communication latency between nodes, and increases data transmission efficiency. On the other hand, the node tree constructed using corner nodes is a disjoint tree, which improves the utilization rate of available connections and avoids contention during broadcast and aggregation blocks.

[0072] The following provides a detailed explanation and description of each step involved in the data synchronization method of the exemplary embodiments of this disclosure.

[0073] First, the application scenarios and purposes of the exemplary embodiments of this disclosure will be explained and described. Specifically, the exemplary embodiments of this disclosure can be applied to data-parallel distributed training of large models, mainly studying how to effectively reduce communication latency between nodes and improve data transmission efficiency as the model size and data volume continue to grow.

[0074] This disclosure uses computing nodes participating in data training as a foundation. A node grid is generated based on the computing nodes, corner nodes are obtained from the node grid, and the node among the corner nodes is determined as the root node of the tree. After determining the root node, it is initialized. Following initialization, the computing nodes associated with the root node in the node grid are determined. A node tree is constructed based on the root node and the computing nodes. Training data is synchronized based on the nodes in the node tree, improving link utilization, reducing communication latency between nodes, and increasing data transmission efficiency.

[0075] Secondly, steps S110 to S130 will be explained and described in detail.

[0076] In step S110, a computing node is obtained, a node mesh of the computing node is generated, a corner node in the node mesh is obtained, and the corner node is determined as the root node of the tree.

[0077] In this system, computing nodes refer to the computing devices of various participants involved in training the large model. After acquiring the computing nodes, a node grid is generated based on them. When generating the node grid, the dimensions of the grid can be set: n rows and m columns, where m and n are both greater than 1. After generating the node grid, the corner nodes can be identified, and the root node of the tree is determined based on these corner nodes. A corner node is a node located at a corner of the node grid. In a 3x3 node grid, the 1st, 3rd, 7th, and 9th computing nodes are corner nodes. However, not all corner nodes can be used as the root node of the tree.

[0078] In this example embodiment, to better utilize available connections and implement a parallel AllReduce overlapped method for multiple blocks, multiple disjoint trees can be created based on a two-dimensional grid. To avoid contention during broadcasting and aggregation blocks, the trees must be completely disjoint. However, the following should be considered when constructing disjoint trees based on a two-dimensional grid:

[0079] The maximum possible number of disjoint trees is 3: In a 2D mesh, due to scarce connections, building more than three disjoint trees is infeasible. In the mesh topology, each node can have at most 4 outgoing or incoming connections. Therefore, each node needs to be connected to all disjoint trees, making the construction of more than 4 disjoint trees impractical. For an n*n mesh, the total number of available connections is (4n...). 2 -4n). To form 4 non-intersecting trees, a total of (4n) is required. 2 -4) connections, because each tree requires (n) connections. 2 -1) connections. For any (n>1), the number of connections required to form 4 disjoint trees exceeds the number of connections available in the topology; therefore, it is impossible to form four disjoint trees in a 2D grid.

[0080] None of the nodes meet the requirements for creating three disjoint trees: to have three disjoint trees, each node should have at least three outgoing connections. However, in a 2D mesh, each corner node can only have a maximum of two outgoing connections. A potential solution is to create disjoint trees rooted at corner nodes, thus avoiding the need to connect corner nodes within their own trees, and instead using the two available connections to establish connections in the other two trees. However, even with this approach, when using three of the four corner nodes as roots to construct disjoint trees, the remaining corner nodes still need to send data to the three trees, requiring three outgoing connections, which are unavailable.

[0081] In this disclosure, to solve the above problems, (n) is used. 2-1) nodes are used for training. Specifically, during the training phase, one corner node is selected not to participate in training, while the other three corner nodes are used to form three disjoint trees. In this way, we can effectively utilize all available connections to form three trees, and since no corner node requires three connections, we can ensure that the trees do not intersect, thereby enabling parallel communication of multiple blocks.

[0082] refer to Figure 2 As shown, the node mesh of the node is generated, the corner nodes in the node mesh are obtained, and the corner nodes are determined as the root nodes of the tree, including:

[0083] Step S210. Determine the dimensions of the node mesh, and generate the node mesh of the computing node based on the mesh dimensions;

[0084] Step S220. Obtain the corner nodes included in the node mesh, and determine the first corner node, the second corner node and the third corner node among the corner nodes as the root node of the tree.

[0085] The following will further explain and illustrate steps S210 and S220. Specifically, first, the dimensions of the node mesh are determined, and then the node mesh for calculating the nodes is generated based on these dimensions. The first corner node, second corner node, and third corner node can be obtained from the node mesh, and these three nodes are determined as the root nodes of the tree.

[0086] Further reference Figure 3 As shown, the step of obtaining the corner nodes included in the node mesh, and determining the first corner node, the second corner node, and the third corner node as the root node of the tree, includes:

[0087] Step S310. Determine the first computation node in the node mesh as the first corner node;

[0088] Step S320. Determine the last computation node in the node grid according to the dimension of the node grid, and designate the last computation node in the node grid as the second corner node;

[0089] Step S330. Determine any one of the remaining corner nodes as the third corner node;

[0090] Step S340. Determine the first corner node, the second corner node, and the third corner node as the root node of the tree.

[0091] The following will further explain and illustrate steps S310-S340. Specifically, the first computation node in the node grid is determined as the first corner node. After determining the first corner node, the last computation node in the node grid is determined according to the dimensions of the node grid, and is determined as the second corner node, that is, the n*m-th computation node in the node grid is the second corner node. In the node grid, any one of the two remaining corner nodes besides the first and last corner nodes is determined as the third corner node. After determining the first, second, and third corner nodes, these three corner nodes are used as the root nodes of the tree, thereby generating three non-overlapping trees.

[0092] For example, according to Figure 4 The node grid shown is a 3x3 grid. In this grid, the compute node labeled 1 can be designated as the first corner node, the compute node labeled 9 as the second corner node, and any one of the remaining corner nodes, compute node 3 and compute node 7, can be designated as the third corner node. When compute node 3 is designated as the third corner node, compute node 7 is excluded during training; however, the connections associated with compute node 7 can still be used to form a tree.

[0093] In step S120, the root node is initialized, and a node tree is generated based on the root node and the compute nodes in the node mesh associated with the corner node.

[0094] After determining the root node of the tree, it can be initialized, and the compute nodes associated with that corner node can be determined in the node grid. A node tree is then generated based on the root node and the compute nodes associated with it.

[0095] refer to Figure 5 As shown, a node tree is generated based on the root node and the compute nodes in the node mesh associated with the corner node, including:

[0096] Step S510. Using the first corner node as the root node, determine the first computing node connected to the first corner node based on the first direction of the first corner node in the node mesh;

[0097] Step S520. Based on the second direction of the first computing node in the node mesh, determine the second computing node connected to the first computing node;

[0098] Step S530. Generate a first node tree with the first corner node as the root node based on the first corner node, the first computed node, and the second computed node.

[0099] The following will further explain and illustrate steps S510-S530. Specifically, taking the first corner node as the root node, based on the first direction of the first corner node in the node mesh, the first computation node connected to the first corner node is determined, that is, the node reachable from the first corner node is determined, where the first direction can be the y-axis direction of the coordinate axis formed with the first corner node as the origin. After obtaining the first computation node connected to the first corner node, based on the second direction of the first computation node in the node mesh, the second computation node connected to the first computation node is determined, where the second direction can be the x-axis direction of the coordinate axis formed with the first computation node as the origin. After determining the first computation node and the second computation node, a first node tree is generated based on the first corner node, the first computation node, and the second computation node.

[0100] by Figure 4 Taking the node mesh shown as an example, computation node 1 is the first corner node. The y-axis of the coordinate axis formed with computation node 1 as the origin determines computation nodes 4 and 7 as the first computation nodes. Then, computation node 1 is connected to computation node 4, and computation node 4 is connected to computation node 7. Next, the x-axis of the coordinate axis formed with computation node 4 as the origin determines computation nodes 5 and 6 as the second computation nodes, connected to computation node 4. Computation node 5 is connected to computation node 4, and computation node 6 is connected to computation node 5. Then, the x-axis of the coordinate axis formed with computation node 7 as the origin determines computation nodes 8 and 9 as the second computation nodes, connected to computation node 7, and computation node 9 is connected to computation node 8. Finally, the first node tree is generated using computation nodes 1, 4, 7, 5, 6, 8, and 9.

[0101] After generating the first node tree, generate the second node tree, refer to [reference needed]. Figure 6 As shown, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0102] Step S610. Using the second corner node as the root node, determine the third computing node connected to the second corner node based on the third direction of the second corner node in the node mesh;

[0103] Step S620. Based on the fourth direction of the third computing node in the node mesh, determine the fourth computing node connected to the third computing node;

[0104] Step S630. Generate a second node tree with the second corner node as the root node based on the second corner node, the third computing node, and the fourth computing node.

[0105] The following will further explain and illustrate steps S610-S630. Specifically, the second corner node is taken as the root node. Based on the third direction of the second corner node in the node mesh, the third computation node connected to the second corner node is determined. This third direction can be the x-axis direction of the coordinate axis formed with the second corner node as the origin. After obtaining the third computation node, based on the fourth direction of the third computation node in the node mesh, the fourth computation node connected to the third computation node is determined. This fourth direction is the y-axis direction of the coordinate axis formed with the third computation node as the origin in the node mesh. After determining the fourth computation node, a second node tree is generated based on the second corner node, the third computation node, and the fourth computation node.

[0106] by Figure 4 Taking the node mesh shown as an example, computation node 9 is the second corner node. The x-axis of the coordinate axis formed with computation node 9 as the origin determines computation nodes 8 and 7 as the third computation nodes. Therefore, computation node 8 is connected to computation node 9, and computation node 7 is connected to computation node 8. Next, the y-axis of the coordinate axis formed with computation node 8 as the origin determines computation nodes 5 and 2, which are connected to computation node 8, as the fourth computation nodes. Computation node 5 is connected to computation node 8, and computation node 2 is connected to computation node 5. Then, the y-axis of the coordinate axis formed with computation node 7 as the origin determines computation nodes 4 and 1 as the fourth computation nodes. Computation node 4 is connected to computation node 7, and computation node 1 is connected to computation node 4. Finally, a second node tree is generated using computation nodes 9, 8, 7, 5, 2, 4, and 1.

[0107] After generating the second node tree, the third node tree can be generated based on the third corner node.

[0108] refer to Figure 7 As shown, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0109] Step S710. Perform a breadth-first traversal based on the third corner node to obtain the fifth computed node;

[0110] Step S720. Generate a third node tree with the third corner node as the root node based on the third corner node and the fifth calculation node.

[0111] The following will provide further explanation and description of steps S710 and S720. Specifically, when constructing the third node tree based on the third corner node, the node grid can be regarded as an undirected graph. A breadth-first traversal is performed starting from the third corner node to obtain all fifth computation nodes reachable from the third corner node. After obtaining the fifth computation nodes, the third node tree is generated based on the fifth computation nodes and the third corner node.

[0112] Further reference Figure 8 As shown, the breadth-first traversal based on the third corner node to obtain the fifth computed node includes:

[0113] Step S810. Perform a breadth-first traversal of the node mesh based on the third corner node to obtain the computational nodes associated with the third corner node;

[0114] Step S820. Delete the sixth computing node from the computing nodes associated with the third corner node, and determine the remaining computing nodes as the fifth computing node; wherein, the sixth computing node is a corner node in the node mesh other than the first corner node, the second corner node, and the third corner node;

[0115] Step S830. When the fifth compute node includes an unselected corner node, exclude the corner node and generate the third node tree based on the third corner node and the remaining fifth compute nodes.

[0116] The following will further explain and illustrate steps S810-S830. Specifically, a breadth-first traversal is performed on the grid nodes based on the third corner node to obtain the computation nodes associated with the third corner node. After obtaining the computation nodes associated with the third corner node, since the computation nodes associated with the third corner node include excluded corner nodes, the sixth computation node should be deleted from the computation nodes associated with the third corner node, and the remaining computation nodes are determined as the fifth computation node. The deleted sixth computation node is the corner node in the node grid other than the first corner node, the second corner node, and the third corner node.

[0117] by Figure 4 Taking the node mesh as an example, when the computation node 3 is the third corner node, if the traversed computation nodes include computation node 7, then the computation node 7 is deleted, and the remaining computation nodes are determined as the fifth computation node. The third node tree is generated based on the fifth computation node and the third corner node.

[0118] In step S130, data is synchronized based on the nodes in the node tree.

[0119] After generating the node tree, data synchronization can be performed based on the computation nodes in the node tree. AllReduce consists of two stages: ReduceScatter, which aggregates gradients, and AllGather, which distributes gradients. When executing AllReduce hierarchically in a two-dimensional network, an overlapping method is used to execute ReduceScatter and AllGather in each dimension. Generally, ReduceScatter is performed in one dimension first, and after that dimension is completed, it is performed in another dimension. Related technologies use node spanning trees, which is basically the same as the corner node spanning tree used in this disclosure for ReduceScatter latency. However, thereafter, overlapping becomes beneficial, reducing the latency of each block by N / 2 instead of N-1. With large amounts of data, the overlapping method of generating disjoint trees using corner nodes is much faster, and the link utilization is significantly increased.

[0120] The data synchronization device provided in this disclosure has at least the following advantages: First, this disclosure constructs a node tree using corner nodes and performs data synchronization through the nodes in the node tree. Although reducing the number of nodes participating in training reduces the size of the small batch during training, leading to an increase in training time, the benefits of accelerating AllReduce through the node tree far outweigh the increased training time. Second, by constructing a non-overlapping tree to achieve parallel data synchronization, the performance degradation of traditional algorithms in large-scale environments is avoided, thereby optimizing resource utilization efficiency and shortening the AllReduce operation time in large-scale and deep learning training processes. Third, by introducing gradient partitioning, multiple gradient blocks participate in the scheduling and communication of AllReduce simultaneously, and multiple node trees are constructed for collective communication scheduling strategies, improving the link bandwidth utilization capability.

[0121] This disclosure also provides a data synchronization apparatus, as illustrated in the example embodiments. Figure 9 As shown, it may include: a root node determination module 910, a node tree generation module 920, and a data synchronization module 930.

[0122] in:

[0123] The root node determination module 910 is used to obtain computing nodes, generate a node grid of the computing nodes, obtain corner nodes in the node grid, and determine the corner nodes as the root nodes of the tree.

[0124] The node tree generation module 920 is used to initialize the root node and generate a node tree based on the root node and the compute nodes in the node mesh associated with the corner node.

[0125] The data synchronization module 930 is used to synchronize data based on the nodes in the node tree.

[0126] The specific details of each module in the aforementioned data synchronization device have been described in detail in the corresponding data synchronization method, so they will not be repeated here.

[0127] In one exemplary embodiment of this disclosure, the steps of obtaining computing nodes, generating a node mesh of the nodes, obtaining corner nodes in the node mesh, and determining the corner nodes as the root nodes of the tree include:

[0128] Determine the dimensions of the node mesh, and generate the node mesh of the computing node based on the dimensions;

[0129] Obtain the corner nodes included in the node grid, and determine the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of the tree.

[0130] In one exemplary embodiment of this disclosure, obtaining the corner nodes included in the node mesh, and determining the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of the tree, includes:

[0131] The first computation node in the node grid is determined as the first corner node;

[0132] The last computation node in the node grid is determined based on the dimension of the node grid, and the last computation node in the node grid is determined as the second corner node;

[0133] Determine any one of the remaining corner nodes as the third corner node;

[0134] The first corner node, the second corner node, and the third corner node are determined as the root node of the tree.

[0135] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0136] Using the first corner node as the root node, and based on the first direction of the first corner node in the node mesh, determine the first computing node connected to the first corner node;

[0137] Based on the second direction of the first computing node in the node grid, a second computing node connected to the first computing node is determined;

[0138] Based on the first corner node, the first computed node, and the second computed node, generate a first node tree with the first corner node as the root node.

[0139] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0140] Using the second corner node as the root node, a third computing node connected to the second corner node is determined based on the third direction of the second corner node in the node mesh;

[0141] Based on the fourth direction of the third computing node in the node grid, a fourth computing node connected to the third computing node is determined;

[0142] Based on the second corner node, the third computed node, and the fourth computed node, a second node tree is generated with the second corner node as the root node.

[0143] In one exemplary embodiment of this disclosure, generating a node tree based on the root node and the compute nodes in the node mesh associated with the corner node includes:

[0144] The fifth computation node is obtained by performing a breadth-first traversal based on the third corner node.

[0145] Based on the third corner node and the fifth computed node, a third node tree is generated with the third corner node as the root node.

[0146] In one exemplary embodiment of this disclosure, the step of performing a breadth-first traversal based on the third corner node to obtain the fifth computed node includes:

[0147] Based on the third corner node, a breadth-first traversal is performed on the node mesh to obtain the computational nodes associated with the third corner node;

[0148] The sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing node; wherein, the sixth computing node is a corner node in the node mesh other than the first corner node, the second corner node and the third corner node.

[0149] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0150] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0151] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0152] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0153] The following reference Figure 10 To describe an electronic device 1000 according to such an embodiment of the present disclosure. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0154] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0155] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1010 can perform actions such as... Figure 1 The steps shown are as follows: S110: Obtain computing nodes, generate a node grid of computing nodes, obtain corner nodes in the node grid, and determine the corner nodes as the root nodes of the tree; S120: Initialize the root nodes, and generate a node tree based on the root nodes and the computing nodes in the node grid associated with the corner nodes; S130: Synchronize data based on the nodes in the node tree.

[0156] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0157] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0158] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0159] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0160] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0161] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0162] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0164] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0166] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0168] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A data synchronization method, characterized in that, include: The dimensions of the node mesh are determined, and a node mesh for computing nodes is generated based on the dimensions; wherein, nodes located at the corners of the node mesh are corner nodes, and the corner nodes include at least a first corner node, a second corner node, and a third corner node; The first computation node in the node grid is determined as the first corner node; the last computation node in the node grid is determined according to the dimension of the node grid, and the last computation node in the node grid is determined as the second corner node; any remaining corner node is determined as the third corner node; the first corner node, the second corner node, and the third corner node are determined as the root node of the tree; The root node is initialized, and a node tree is generated based on the root node and the compute nodes in the node grid associated with the corner node; wherein, generating the node tree based on the root node and the compute nodes in the node grid associated with the corner node includes: determining the nodes in the node grid that can reach the root node, and generating the node tree based on the root node and the nodes that can reach the root node. Data is synchronized based on the nodes in the node tree.

2. The method according to claim 1, characterized in that, The process of generating a node tree based on the root node and the compute nodes in the node grid associated with the corner node includes: Using the first corner node as the root node, a first computation node connected to the first corner node is determined based on the first direction of the first corner node in the node mesh; wherein, the first direction is the y-axis direction of the coordinate axis formed with the first corner node as the origin; Based on the second direction of the first computing node in the node grid, a second computing node connected to the first computing node is determined; wherein, the second direction is the x-axis direction of the coordinate axis formed with the first computing node as the origin; Based on the first corner node, the first computed node, and the second computed node, generate a first node tree with the first corner node as the root node.

3. The method according to claim 1, characterized in that, The process of generating a node tree based on the root node and the compute nodes in the node grid associated with the corner node includes: Using the second corner node as the root node, a third computation node connected to the second corner node is determined based on the third direction of the second corner node in the node mesh; wherein, the third direction is the x-axis direction of the coordinate axis formed with the second corner node as the origin; Based on the fourth direction of the third computing node in the node mesh, a fourth computing node connected to the third computing node is determined; wherein, the fourth direction is the y-axis direction of the coordinate axis formed with the third computing node as the origin in the node mesh; Based on the second corner node, the third computed node, and the fourth computed node, a second node tree is generated with the second corner node as the root node.

4. The method according to claim 1, characterized in that, The process of generating a node tree based on the root node and the compute nodes in the node grid associated with the corner node includes: Based on the third corner node, a breadth-first traversal is performed on the node mesh to obtain the computing nodes associated with the third corner node; the sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing node; wherein, the sixth computing node is a corner node in the node mesh other than the first corner node, the second corner node, and the third corner node; Based on the third corner node and the fifth computed node, a third node tree is generated with the third corner node as the root node.

5. A data synchronization device, characterized in that, include: The root node determination module is used to determine the dimension of the node mesh and generate a node mesh for computing nodes based on the dimension; wherein, nodes located at the corners of the node mesh are corner nodes, and the corner nodes include at least a first corner node, a second corner node, and a third corner node; The first computation node in the node grid is determined as the first corner node; the last computation node in the node grid is determined according to the dimension of the node grid, and the last computation node in the node grid is determined as the second corner node; any remaining corner node is determined as the third corner node; the first corner node, the second corner node, and the third corner node are determined as the root node of the tree; A node tree generation module is used to initialize the root node and generate a node tree based on the root node and the compute nodes in the node grid associated with the corner node; wherein, generating a node tree based on the root node and the compute nodes in the node grid associated with the corner node includes: determining the nodes in the node grid that can reach the root node, and generating the node tree based on the root node and the nodes that can reach the root node; The data synchronization module is used to synchronize data based on the nodes in the node tree.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-4.

7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-4 by executing the executable instructions.

Images