Data synchronization method and device, computer program product and electronic equipment

Abstract

The invention relates to a data synchronization method and device, a computer program product and electronic equipment, and relates to the technical field of computers. The method comprises the steps that computing nodes are acquired, node grids of the computing nodes are generated, corner nodes in the node grids are acquired, and the corner nodes are determined as root nodes of a tree; initializing the root node, and generating a node tree based on the root node and computing nodes associated with the corner node in the node grid; and synchronizing the data based on the nodes in the node tree. According to the invention, the communication delay between the nodes is reduced, and the data transmission efficiency is improved.

Description

Background Art

[0002] When training a large model, data parallel training is one of the main methods for processing large model data sets. Parallel training involves dividing the data into small batches and distributing them to multiple computing nodes. After each node independently calculates the gradient, the gradient is synchronized through the AllReduce algorithm to ensure that the models of all nodes remain consistent. AllReduce is a common operation in distributed computing, which is used to exchange and aggregate data between different computing nodes. Its function is to aggregate and synchronize the data on each computing node to achieve global data consistency across the entire cluster.

[0003] Currently, there are many methods to achieve data synchronization, such as ring-based AllReduce, attribute structure algorithm, etc. These algorithms can reduce communication overhead, but when extended to more complex network architectures, network bandwidth limitations significantly increase training time, causing communication delay to become the main bottleneck of training speed.

[0004] Therefore, a new data synchronization method needs to be provided.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data synchronization method, a data synchronization device, a computer program product, and an electronic device, thereby overcoming, at least to a certain extent, the problem of significantly increased training time due to limitations and defects of related technologies.

[0007] According to one aspect of the present disclosure, a data synchronization method is provided, comprising:

[0008] Obtaining a computing node, generating a node grid of the computing node, obtaining a corner node in the node grid, and determining the corner node as a root node of a tree;

[0009] Initializing the root node, and generating a node tree based on the root node and the computational nodes associated with the corner nodes in the node grid;

[0010] Data is synchronized based on the nodes in the node tree.

[0011] In an exemplary embodiment of the present disclosure, the obtaining of the computing node, generating a node grid of the node, obtaining a corner node in the node grid, and determining the corner node as a root node of the tree includes:

[0012] Determine the dimension of the node grid, and generate the node grid of the computing node according to the dimension;

[0013] Corner nodes included in the node grid are obtained, and a first corner node, a second corner node, and a third corner node among the corner nodes are determined as root nodes of the tree.

[0014] In an exemplary embodiment of the present disclosure, acquiring corner nodes included in the node grid, and determining a first corner node, a second corner node, and a third corner node among the corner nodes as root nodes of the tree, includes:

[0015] determining a first computational node in the node grid as the first corner node;

[0016] Determine a last computing node in the node grid according to the dimension of the node grid, and determine the last computing node in the node grid as the second corner node;

[0017] Determine any remaining corner node among the corner nodes as the third corner node;

[0018] The first corner node, the second corner node, and the third corner node are determined as root nodes of the tree.

[0019] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0020] Taking the first corner node as the root node, and determining a first computing node connected to the first corner node based on a first direction of the first corner node in the node grid;

[0021] Determining a second computing node connected to the first computing node based on a second direction of the first computing node in the node grid;

[0022] A first node tree with the first corner node as a root node is generated according to the first corner node, the first computing node, and the second computing node.

[0023] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0024] Taking the second corner node as the root node, and determining a third computing node connected to the second corner node based on a third direction of the second corner node in the node grid;

[0025] Determining a fourth computing node connected to the third computing node based on a fourth direction of the third computing node in the node grid;

[0026] A second node tree with the second corner node as a root node is generated according to the second corner node, the third computing node, and the fourth computing node.

[0027] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0028] Perform breadth-first traversal based on the third corner node to obtain the fifth computing node;

[0029] A third node tree with the third corner node as a root node is generated according to the third corner node and the fifth computing node.

[0030] In an exemplary embodiment of the present disclosure, performing breadth-first traversal based on the third corner node to obtain the fifth computing node includes:

[0031] Performing a breadth-first traversal on the node grid based on the third corner node to obtain a computing node associated with the third corner node;

[0032] The sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing nodes; wherein the sixth computing node is a corner node in the node grid except the first corner node, the second corner node and the third corner node.

[0033] According to one aspect of the present disclosure, there is provided a data synchronization device, comprising:

[0034] A root node determination module, used to obtain a computing node, generate a node grid of the computing node, obtain a corner node in the node grid, and determine the corner node as a root node of the tree;

[0035] A node tree generation module, used to initialize the root node, and generate a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid;

[0036] The data synchronization module is used to synchronize data based on the nodes in the node tree.

[0037] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the data synchronization method described in any one of the above exemplary embodiments is implemented.

[0038] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0039] A processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data synchronization method described in any of the above exemplary embodiments by executing the executable instructions.

[0040] A data synchronization method provided by an embodiment of the present disclosure comprises the steps of obtaining computing nodes, generating a node grid of the computing nodes, obtaining corner nodes in the node grid, determining the corner nodes as the root nodes of the tree; initializing the root nodes, generating a node tree based on the root nodes and computing nodes associated with the corner nodes in the node grid; and synchronizing data based on the nodes in the node tree. On the one hand, the corner nodes in the node grid are obtained, a node tree is constructed based on the corner nodes, and collective communication scheduling decisions are made through the node tree, thereby improving the link bandwidth utilization capability, reducing the communication delay between nodes, and improving the data transmission efficiency. On the other hand, the node tree constructed using the corner nodes is a disjoint tree, which improves the utilization rate of available connections and avoids contention during broadcast and aggregation blocks.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0043] Figure 1 The following schematically shows a flow chart of a data synchronization method according to an exemplary embodiment of the present disclosure.

[0044] Figure 2 A flowchart of a method for generating a node grid of nodes, obtaining corner nodes in the node grid, and determining the corner nodes as root nodes of a tree according to an exemplary embodiment of the present disclosure is schematically shown.

[0045] Figure 3 A flowchart of a method for acquiring corner nodes included in a node grid and determining a first corner node, a second corner node, and a third corner node among the corner nodes as root nodes of a tree according to an exemplary embodiment of the present disclosure is schematically shown.

[0046] Figure 4 A schematic diagram of a node grid according to an example embodiment of the present disclosure is schematically shown.

[0047] Figure 5 A flowchart of a method for generating a node tree based on a root node and computing nodes associated with corner nodes in a node grid according to an exemplary embodiment of the present disclosure is schematically shown.

[0048] Figure 6 A flowchart of a method for generating a node tree based on a root node and computing nodes associated with corner nodes in a node grid according to an exemplary embodiment of the present disclosure is schematically shown.

[0049] Figure 7 A flowchart of a method for generating a node tree based on a root node and computing nodes associated with corner nodes in a node grid according to an exemplary embodiment of the present disclosure is schematically shown.

[0050] Figure 8 A flowchart of a method for performing breadth-first traversal based on a third corner node to obtain a fifth computing node according to an exemplary embodiment of the present disclosure is schematically shown.

[0051] Fig. 9 A block diagram schematically shows a data synchronization device according to an exemplary embodiment of the present disclosure.

[0052] Fig.10 An electronic device for implementing the above data synchronization method according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0054] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0055] When training large models, data parallel training is one of the main methods for processing large data sets. The current mainstream technical solutions usually use the backpropagation algorithm to optimize the weights and biases of deep neural networks. In this process, the model goes through the following steps:

[0056] Forward propagation: Input data is calculated through multiple layers of neurons to generate output results.

[0057] Loss function calculation: Calculate the loss value based on the difference between the model's output and the true label.

[0058] Gradient calculation: Through the back-propagation algorithm, the chain rule is used to calculate the gradient of the loss function with respect to each parameter.

[0059] Parameter update: Use optimization algorithms such as Stochastic Gradient Descent (SGD) to update model parameters based on the calculated gradients.

[0060] In order to improve training efficiency, especially when processing large-scale data, data parallel distributed training is widely used. This process usually involves dividing the data into small batches and distributing them to multiple computing nodes. After each node independently calculates the gradient, these gradients are synchronized through the AllReduce algorithm to ensure that the models of all nodes are consistent. Among them, AllReduce is a common operation in distributed computing, which is used to exchange and aggregate data between different computing nodes. Its function is to aggregate and synchronize the data on each computing node to achieve global data consistency across the entire cluster. This operation is often used in scenarios that require large-scale parallel computing, such as deep learning training.

[0061] Currently, there are many algorithms for implementing AllReduce, such as ring-based AllReduce, tree-structured algorithms, etc. Ring-based AllReduce uses a ring topology so that each node is only visited once, thereby reducing communication overhead. However, these algorithms may face performance bottlenecks when expanding to more complex network architectures. Although existing distributed training solutions have improved training efficiency to a certain extent, there are still some technical problems:

[0062] Communication bottleneck: In large-scale distributed training, data exchange between nodes is frequent, resulting in communication delays becoming the main bottleneck of training speed. Especially in the case of high-dimensional data and large-scale models, network bandwidth limitations significantly increase training time.

[0063] Uneven load: In some cases, the computing power and data volume of different nodes may be uneven, causing some nodes to be overloaded while others are idle, thus affecting the overall training efficiency.

[0064] Gradient synchronization delay: When using the AllReduce algorithm, it takes time to synchronize the gradients of each node, especially when the network topology is not ideal, which may cause parameter updates to lag during training.

[0065] Scalability issue: The performance improvement of existing AllReduce algorithms is often not as good as expected when they are expanded to more nodes, partly because the algorithm design fails to fully consider the complexity of the network structure.

[0066] Among the above problems, the most important technical problem is the communication bottleneck. As the model size and data volume continue to grow, how to effectively reduce the communication delay between nodes and improve data transmission efficiency is one of the key challenges in deep learning research.

[0067] Based on one or more of the above problems, this exemplary embodiment first provides a data synchronization method, which can be run on a terminal device; of course, those skilled in the art can also run the method of the present invention on other platforms as required, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the data synchronization method may include steps S110 to S130:

[0068] Step S110. Obtain a computing node, generate a node grid of the computing node, obtain a corner node in the node grid, and determine the corner node as a root node of the tree;

[0069] Step S120: Initialize the root node, and generate a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid;

[0070] Step S130: Synchronize data based on the nodes in the node tree.

[0071] The above data synchronization method obtains computing nodes, generates a node grid of the computing nodes, obtains corner nodes in the node grid, and determines the corner nodes as the root nodes of the tree; initializes the root nodes, generates a node tree based on the root nodes and computing nodes associated with the corner nodes in the node grid; and synchronizes data based on the nodes in the node tree. On the one hand, the corner nodes in the node grid are obtained, a node tree is constructed based on the corner nodes, and collective communication scheduling decisions are made through the node tree, which improves the link bandwidth utilization capability, reduces the communication delay between nodes, and improves the data transmission efficiency. On the other hand, the node tree constructed using the corner nodes is a disjoint tree, which improves the utilization rate of available connections and avoids contention during broadcast and aggregation blocks.

[0072] The following is a detailed explanation and description of each step involved in the data synchronization method of the exemplary embodiment of the present disclosure.

[0073] First, the application scenarios and purposes of the exemplary embodiments of the present disclosure are explained and illustrated. Specifically, the exemplary embodiments of the present disclosure can be applied to data parallel distributed training of large model training, mainly studying how to effectively reduce communication delays between nodes and improve data transmission efficiency as the model scale and data volume continue to grow.

[0074] The present disclosure is based on the computing nodes participating in data training, generates a node grid according to the computing nodes, obtains the corner nodes in the node grid, and determines the node in the corner node that is the root node of the tree. After determining the root node, the root node is initialized. After initialization, the computing node associated with the root node in the node grid is determined, and a node tree is constructed based on the root node and the computing node. The training data is synchronized based on the nodes in the node tree, thereby improving the utilization rate of the link, reducing the communication delay between nodes, and improving the data transmission efficiency.

[0075] Next, steps S110 to S130 are explained and illustrated in detail.

[0076] In step S110, a computing node is obtained, a node grid of the computing node is generated, a corner node in the node grid is obtained, and the corner node is determined as a root node of the tree.

[0077] Among them, the computing nodes are the computing devices of each participant participating in the large model training. After the computing nodes are obtained, a node grid is generated according to the computing nodes. When generating the node grid, the dimension of the node grid can be set. The number of rows of the grid is n, the number of columns is m, and both m and n are greater than 1. After the node grid is generated, the corner nodes in the node grid can be obtained, and the root node of the tree is determined according to the obtained corner nodes. Among them, the corner nodes are nodes at the corners of the node grid. When the first computing node, the third computing node, the seventh computing node and the ninth computing node in the 3*3 node grid are corner nodes. After obtaining 4 corner nodes, not all corner nodes can be used as the root node of the tree.

[0078] In this example embodiment, in order to better utilize the available connections and implement the parallel AllReduce overlapping method for multiple blocks, multiple disjoint trees can be created based on the two-dimensional grid. To avoid contention during broadcasting and aggregation of blocks, the trees must be completely disjoint. However, the following needs to be considered when constructing disjoint trees based on the two-dimensional grid:

[0079] The maximum possible number of disjoint trees is 3: In a 2D grid, it is not feasible to build more than three disjoint trees due to the scarcity of connections. In a grid topology, each node can have at most 4 outgoing or incoming connections. Therefore, each node needs to be connected to all disjoint trees, which makes it impractical to build more than 4 disjoint trees. For an n*n grid, the total number of available connections is (4n 2 -4n). If you want to form 4 disjoint trees, you will need a total of (4n 2 -4) connections, because each tree requires (n 2 -1) connections. For any (n>1), the number of connections required to form 4 disjoint trees exceeds the number of connections available in the topology, so it is impossible to form four disjoint trees in a 2D grid.

[0080] All nodes do not meet the conditions for creating 3 disjoint trees: To have 3 disjoint trees, each node should have at least three outgoing connections. However, in a 2D grid, all corner nodes can have at most two outgoing connections. A potential solution is to create disjoint trees with corner nodes as roots, so that instead of connecting the corner nodes in their own trees, they can use the available 2 connections to establish connections in the other 2 trees. However, even with this approach, when using 3 of the 4 corner nodes as roots to build disjoint trees, the remaining corner nodes still need to send data to the 3 trees, which requires three outgoing connections, which are not available.

[0081] In order to solve the above problems, the present invention uses (n 2-1) nodes to train. Specifically, during the training phase, one corner node is selected not to participate in the training, while the other three corner nodes are used to form three disjoint trees. In this way, we can effectively utilize all available connections to form three trees, and since no corner node needs to use three connections, the trees can be guaranteed to be disjoint, thus achieving parallel communication of multiple blocks.

[0082] refer to Figure 2 As shown, generating a node grid of the node, obtaining a corner node in the node grid, and determining the corner node as a root node of the tree includes:

[0083] Step S210. Determine the dimension of the node grid, and generate the node grid of the computing node according to the grid dimension;

[0084] Step S220: Acquire corner nodes included in the node grid, and determine a first corner node, a second corner node, and a third corner node among the corner nodes as root nodes of the tree.

[0085] In the following, step S210 and step S220 are further explained and illustrated. Specifically, the dimension of the node grid is first determined, and the node grid of the calculation node is generated according to the dimension of the node grid. The first corner node, the second corner node and the third corner node can be obtained in the node grid, and the obtained first corner node, the second corner node and the third corner node are determined as the root node of the tree.

[0086] For further reference, Figure 3 As shown, the acquiring of the corner nodes included in the node grid and determining the first corner node, the second corner node and the third corner node among the corner nodes as the root node of the tree includes:

[0087] Step S310. Determine the first computing node in the node grid as the first corner node;

[0088] Step S320. Determine the last computing node in the node grid according to the dimension of the node grid, and determine the last computing node in the node grid as the second corner node;

[0089] Step S330. Determine any remaining corner node among the corner nodes as the third corner node;

[0090] Step S340: Determine the first corner node, the second corner node and the third corner node as root nodes of the tree.

[0091] Below, step S310-step S340 will be further explained and illustrated. Specifically, the first computing node in the node grid is determined as the first corner node; after the first corner node is determined, the last computing node in the node grid is determined according to the dimension of the node grid, and the last computing node in the node grid is determined as the second corner node, that is, the n*mth computing node in the node grid is the second corner node. In the node grid, any one of the remaining two corner nodes other than the first corner node and the last corner node is determined as the third corner node. After the first corner node, the second corner node and the third corner node are determined, the first corner node, the second corner node and the third corner node are respectively used as the root nodes of the tree, thereby generating three non-intersecting trees.

[0092] For example, according to Figure 4 The node grid shown is a 3*3 grid, in which the computation node numbered 1 can be determined as the first corner node, the computation node numbered 9 can be determined as the second corner node, and any of the remaining corner nodes, computation node 3 and computation node 7, can be determined as the third corner node. When computation node 3 is used as the third corner node, computation node 7 is excluded during the training process, but the connection associated with computation node 7 can still be used to form a tree.

[0093] In step S120, the root node is initialized, and a node tree is generated based on the root node and the computation nodes associated with the corner nodes in the node grid.

[0094] After the root node of the tree is determined, the root node may be initialized, and a computation node associated with the corner node may be determined in the node grid. A node tree is generated based on the root node and the computation nodes associated with the root node.

[0095] refer to Figure 5 As shown, based on the root node and the computing nodes associated with the corner nodes in the node grid, a node tree is generated, including:

[0096] Step S510. Taking the first corner node as the root node, and based on a first direction of the first corner node in the node grid, determining a first computing node connected to the first corner node;

[0097] Step S520. Determine a second computing node connected to the first computing node based on a second direction of the first computing node in the node grid;

[0098] Step S530. Generate a first node tree with the first corner node as a root node based on the first corner node, the first computing node, and the second computing node.

[0099] In the following, step S510-step S530 will be further explained and illustrated. Specifically, the first corner node is taken as the root node, and based on the first direction of the first corner node in the node grid, the first computing node connected to the first corner node is determined, that is, the node that can reach the first corner node is determined, wherein the first direction can be the y-axis direction of the coordinate axis formed with the first corner node as the origin. After obtaining the first computing node connected to the first corner node, based on the second direction of the first computing node in the node grid, the second computing node connected to the first computing node is determined, wherein the second direction can be the x-axis direction of the coordinate axis formed with the first computing node as the origin. After the first computing node and the second computing node are determined, the first node tree is generated according to the first corner node, the first computing node, and the second computing node.

[0100] by Figure 4 Taking the node grid shown as an example, computation node 1 is the first corner node, and the y-axis direction of the coordinate axis formed with computation node 1 as the origin determines computation node 4 and computation node 7 as the first computation node, then computation node 1 is connected to computation node 4, and computation node 4 is connected to computation node 7. Then, the x-axis direction of the coordinate axis formed with computation node 4 as the origin determines computation node 5 and computation node 6 connected to computation node 4 as the second computation node, wherein computation node 5 is connected to computation node 4, and computation node 6 is connected to computation node 5. Next, the x-axis direction of the coordinate axis formed with computation node 7 as the origin determines computation node 8 and computation node 9 as the second computation node, wherein computation node 8 is connected to computation node 7, and computation node 9 is connected to computation node 8. Finally, the first node tree is generated through computation node 1, computation node 4, computation node 7, computation node 5, computation node 6, computation node 8, and computation node 9.

[0101] After generating the first node tree, generate the second node tree, refer to Figure 6 As shown, the generating of the node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0102] Step S610. Taking the second corner node as the root node, and based on the third direction of the second corner node in the node grid, determining a third computing node connected to the second corner node;

[0103] Step S620. Determine a fourth computing node connected to the third computing node based on a fourth direction of the third computing node in the node grid;

[0104] Step S630. Generate a second node tree with the second corner node as the root node according to the second corner node, the third computing node, and the fourth computing node.

[0105] Below, step S610-step S630 will be further explained and illustrated. Specifically, the second corner node is used as the root node, and based on the third direction of the second corner node in the node grid, the third computing node connected to the second corner node is determined. Among them, the third direction can be the x-axis direction of the coordinate axis formed with the second corner node as the origin. After obtaining the third computing node, based on the fourth direction of the third computing node in the node grid, the fourth computing node connected to the third computing node is determined. Among them, the fourth direction is the y-axis direction of the coordinate axis formed with the third computing node as the origin in the node grid. After the fourth computing node is determined, the second node tree is generated according to the second corner node, the third computing node and the fourth computing node.

[0106] by Figure 4 Taking the node grid shown as an example, computing node 9 is the second corner node, and computing node 8 and computing node 7 are determined as the third computing node in the x-axis direction of the coordinate axis formed with computing node 9 as the origin, then computing node 8 is connected to computing node 9, and computing node 7 is connected to computing node 8. Then, computing node 5 and computing node 2 connected to computing node 8 are determined as the fourth computing node in the y-axis direction of the coordinate axis formed with computing node 8 as the origin, wherein computing node 5 is connected to computing node 8, and computing node 2 is connected to computing node 5. Next, computing node 4 and computing node 1 are determined as the fourth computing node in the y-axis direction of the coordinate axis formed with computing node 7 as the origin, wherein computing node 4 is connected to computing node 7, and computing node 1 is connected to computing node 4. Finally, the second node tree is generated through computing node 9, computing node 8, computing node 7, computing node 5, computing node 2, computing node 4, and computing node 1.

[0107] After the second node tree is generated, a third node tree may be generated according to the third corner node.

[0108] refer to Figure 7 As shown, the generating of the node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0109] Step S710. Perform breadth-first traversal based on the third corner node to obtain a fifth computing node;

[0110] Step S720. Generate a third node tree with the third corner node as the root node based on the third corner node and the fifth computing node.

[0111] In the following, step S710 and step S720 will be further explained and illustrated. Specifically, when constructing the third node tree based on the third corner node, the node grid can be regarded as an undirected graph, and a breadth-first traversal is performed starting from the third corner node to obtain all fifth computing nodes reachable from the third corner node. After obtaining the fifth computing node, the third node tree is generated according to the fifth computing node and the third corner node.

[0112] For further reference, Figure 8 As shown, the breadth-first traversal is performed based on the third corner node to obtain the fifth computing node, including:

[0113] Step S810. Perform a breadth-first traversal on the node grid based on the third corner node to obtain a computing node associated with the third corner node;

[0114] Step S820. Delete the sixth computing node from the computing nodes associated with the third corner node, and determine the remaining computing nodes as the fifth computing nodes; wherein the sixth computing node is a corner node in the node grid except the first corner node, the second corner node and the third corner node;

[0115] Step S830: When the fifth computing nodes include an unselected corner node, exclude the corner node, and generate a third node tree based on the third corner node and the remaining fifth computing nodes.

[0116] In the following, step S810 to step S830 will be further explained and illustrated. Specifically, a breadth-first traversal is performed in the grid nodes based on the third corner node to obtain the computing nodes associated with the third corner node. After obtaining the computing nodes associated with the third corner node, since the computing nodes associated with the third corner node include the excluded corner node, the sixth computing node should be deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing nodes, wherein the deleted sixth computing node is a corner node in the node grid except the first corner node, the second corner node, and the third corner node.

[0117] by Figure 4 Taking the node grid as an example, when computing node 3 is the third corner node, and computing node 7 is included in the traversed computing nodes, computing node 7 is deleted, and the remaining computing nodes are determined as the fifth computing nodes, and the third node tree is generated according to the fifth computing node and the third corner node.

[0118] In step S130, data is synchronized based on the nodes in the node tree.

[0119] After the node tree is generated, data synchronization can be performed according to the computing nodes in the node tree. AllReduce consists of two stages, ReduceScatter for aggregating gradients and AllGather for distributing gradients. When AllReduce is executed hierarchically in a two-dimensional network, an overlapping method is used to execute ReduceScatter and AllGather in each dimension. Generally, ReduceScatter is performed in one dimension first, and then in another dimension after the execution of that dimension. The use of node generation trees in the related art is basically the same as the delay of using corner node generation trees to perform ReduceScatter in the present disclosure, but after that, overlapping becomes beneficial, and the delay of each block is reduced by N / 2 instead of N-1. When the amount of data is large, the overlapping method of using corner nodes to generate non-intersecting trees is much faster, and the link utilization rate is significantly increased.

[0120] The data synchronization device provided by the example embodiment of the present disclosure has at least the following advantages: First, the present disclosure constructs a node tree through corner nodes, and performs data synchronization through the nodes in the node tree. Although the reduction in the number of nodes participating in training will reduce the size of small batches during training, resulting in an increase in training time, the benefits of AllReduce acceleration through the node tree far outweigh the increased training time; second, data synchronization parallelization is achieved by constructing non-overlapping trees, which avoids the performance degradation of traditional algorithms in large-scale environments, thereby optimizing resource utilization efficiency and shortening the AllReduce operation time in large-scale and deep learning training processes; third, by introducing the gradient partitioning block method, multiple gradient blocks are simultaneously involved in the scheduling communication of AllReduce, and multiple node trees are constructed for collective communication scheduling strategies, thereby improving the link bandwidth utilization capability.

[0121] The exemplary embodiment of the present disclosure also provides a data synchronization device, referring to Fig. 9 As shown, it may include: a root node determination module 910, a node tree generation module 920, and a data synchronization module 930.

[0122] in:

[0123] A root node determination module 910 is used to obtain a computing node, generate a node grid of the computing node, obtain a corner node in the node grid, and determine the corner node as a root node of the tree;

[0124] A node tree generation module 920, configured to initialize the root node and generate a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid;

[0125] The data synchronization module 930 is used to synchronize data based on the nodes in the node tree.

[0126] The specific details of each module in the above-mentioned data synchronization device have been described in detail in the corresponding data synchronization method, so they will not be repeated here.

[0127] In an exemplary embodiment of the present disclosure, the obtaining of the computing node, generating a node grid of the node, obtaining a corner node in the node grid, and determining the corner node as a root node of the tree includes:

[0128] Determine the dimension of the node grid, and generate the node grid of the computing node according to the dimension;

[0129] Corner nodes included in the node grid are obtained, and a first corner node, a second corner node, and a third corner node among the corner nodes are determined as root nodes of the tree.

[0130] In an exemplary embodiment of the present disclosure, acquiring corner nodes included in the node grid, and determining a first corner node, a second corner node, and a third corner node among the corner nodes as root nodes of the tree, includes:

[0131] determining a first computational node in the node grid as the first corner node;

[0132] Determine a last computing node in the node grid according to the dimension of the node grid, and determine the last computing node in the node grid as the second corner node;

[0133] Determine any remaining corner node among the corner nodes as the third corner node;

[0134] The first corner node, the second corner node, and the third corner node are determined as root nodes of the tree.

[0135] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0136] Taking the first corner node as the root node, and determining a first computing node connected to the first corner node based on a first direction of the first corner node in the node grid;

[0137] Determining a second computing node connected to the first computing node based on a second direction of the first computing node in the node grid;

[0138] A first node tree with the first corner node as a root node is generated according to the first corner node, the first computing node, and the second computing node.

[0139] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0140] Taking the second corner node as the root node, and determining a third computing node connected to the second corner node based on a third direction of the second corner node in the node grid;

[0141] Determining a fourth computing node connected to the third computing node based on a fourth direction of the third computing node in the node grid;

[0142] A second node tree with the second corner node as a root node is generated according to the second corner node, the third computing node, and the fourth computing node.

[0143] In an exemplary embodiment of the present disclosure, generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid includes:

[0144] Perform breadth-first traversal based on the third corner node to obtain the fifth computing node;

[0145] A third node tree with the third corner node as a root node is generated according to the third corner node and the fifth computing node.

[0146] In an exemplary embodiment of the present disclosure, performing breadth-first traversal based on the third corner node to obtain the fifth computing node includes:

[0147] Performing a breadth-first traversal on the node grid based on the third corner node to obtain a computing node associated with the third corner node;

[0148] The sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing nodes; wherein the sixth computing node is a corner node in the node grid except the first corner node, the second corner node and the third corner node.

[0149] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0150] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0151] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0152] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0153] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present disclosure is described. Fig.10 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0154] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), and a display unit 1040.

[0155] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the following steps: Figure 1 Steps S110 shown in: obtain a computing node, generate a node grid of the computing node, obtain a corner node in the node grid, and determine the corner node as the root node of the tree; S120: initialize the root node, and generate a node tree based on the root node and the computing nodes associated with the corner node in the node grid; S130: synchronize data based on the nodes in the node tree.

[0156] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0157] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0158] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0159] The electronic device 1000 may also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0160] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0161] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0162] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0163] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0166] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data synchronization method, characterized in that: include: Obtaining a computing node, generating a node grid of the computing node, obtaining a corner node in the node grid, and determining the corner node as a root node of a tree; Initializing the root node, and generating a node tree based on the root node and the computational nodes associated with the corner nodes in the node grid; Data is synchronized based on the nodes in the node tree.

2. The method according to claim 1, characterized in that The obtaining of the computing node, generating a node grid of the node, obtaining a corner node in the node grid, and determining the corner node as a root node of the tree includes: Determine the dimension of the node grid, and generate the node grid of the computing node according to the dimension; Corner nodes included in the node grid are obtained, and a first corner node, a second corner node, and a third corner node among the corner nodes are determined as root nodes of the tree.

3. The method according to claim 2, characterized in that The acquiring of the corner nodes included in the node grid, and determining the first corner node, the second corner node, and the third corner node among the corner nodes as the root node of the tree, comprises: determining a first computational node in the node grid as the first corner node; Determine a last computing node in the node grid according to the dimension of the node grid, and determine the last computing node in the node grid as the second corner node; Determine any remaining corner node among the corner nodes as the third corner node; The first corner node, the second corner node, and the third corner node are determined as root nodes of the tree.

4. The method according to claim 2, characterized in that: The generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid comprises: Taking the first corner node as the root node, and determining a first computing node connected to the first corner node based on a first direction of the first corner node in the node grid; Determining a second computing node connected to the first computing node based on a second direction of the first computing node in the node grid; A first node tree with the first corner node as a root node is generated according to the first corner node, the first computing node, and the second computing node.

5. The method according to claim 2, characterized in that: The generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid comprises: Taking the second corner node as the root node, and determining a third computing node connected to the second corner node based on a third direction of the second corner node in the node grid; Determining a fourth computing node connected to the third computing node based on a fourth direction of the third computing node in the node grid; A second node tree with the second corner node as a root node is generated according to the second corner node, the third computing node, and the fourth computing node.

6. The method according to claim 2, characterized in that The generating a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid comprises: Perform breadth-first traversal based on the third corner node to obtain the fifth computing node; A third node tree with the third corner node as a root node is generated according to the third corner node and the fifth computing node.

7. The method according to claim 6, characterized in that The step of performing breadth-first traversal based on the third corner node to obtain a fifth computing node includes: Performing a breadth-first traversal on the node grid based on the third corner node to obtain a computing node associated with the third corner node; The sixth computing node is deleted from the computing nodes associated with the third corner node, and the remaining computing nodes are determined as the fifth computing nodes; wherein the sixth computing node is a corner node in the node grid except the first corner node, the second corner node and the third corner node.

8. A data synchronization device, characterized in that: include: A root node determination module, used to obtain a computing node, generate a node grid of the computing node, obtain a corner node in the node grid, and determine the corner node as a root node of the tree; A node tree generation module, used to initialize the root node, and generate a node tree based on the root node and the computing nodes associated with the corner nodes in the node grid; The data synchronization module is used to synchronize data based on the nodes in the node tree.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images