Asynchronous distributed training system and method

Abstract

The present invention discloses an asynchronous distributed training system and method, wherein the system comprises a group manager and machine nodes node1 to node2 connected to the group manager network. n The group manager includes a group generation unit, a group clearing unit and a node information table; each machine node node i It is configured to iteratively train the model based on the local data set; the group generation unit is configured to be based on the machine node node i The grouping request is grouped according to the preset grouping conditions. i Grouping and feedback of group information; the group clearing unit is configured to be based on the machine node node i The release request clears the machine node node i The node information table is used to record the status information of the n machine nodes node1 to noden and can be shared by the group generation unit and the group clearing unit; wherein n is an integer greater than 2, and i = 0 to n. The present invention can reduce the communication waiting time of distributed training and the overall time of distributed training, thereby improving the performance of distributed training.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network model training, and in particular to an asynchronous distributed training system and method. Background Art

[0002] Distributed training is a method that uses multiple machines to train models in parallel to speed up training. It plays an important role in deep learning and is receiving increasing attention both in academia and industry. In distributed training, there are two commonly used parallel methods: data parallelism and model parallelism. Data parallelism is to divide the training data set and then distribute it to each node. Each node contains a copy of the complete training model. Each node trains the model based on the local data allocated to it and transfers model parameters between each node. Model parallelism is the opposite. It divides the model and distributes it to each node. Each node contains a complete data set, and each node is trained by the local model allocated to it. The present invention adopts the data parallel method.

[0003] Currently, traditional distributed training architectures are significantly affected by heterogeneous environments, have high communication costs, and significantly increase training time in heterogeneous environments, especially in edge scenarios such as Parameter Server and Ring All-Reduce. The Prague architecture is an emerging distributed training architecture that consists of multiple nodes and a group generator. The group generator is like a central scheduler, mainly grouping nodes, and the nodes are mainly used for training. The training process of the Prague training architecture is mainly as follows: first, a grouping is requested. When a grouping is requested, all nodes that meet the grouping conditions are randomly grouped; then each node performs gradient calculations and updates parameters; finally, communicates with group members to transfer parameters and update the model; and the above steps are repeated until the model converges. Experimental results of the Prague training architecture show that it has good performance in both homogeneous and computing heterogeneous environments, but still has high communication costs in network heterogeneous environments, which increases the overall model training time. Summary of the Invention

[0004] In view of this, the present invention provides an asynchronous distributed training system and method, which are applied to a data-parallel distributed training method to solve the problem of how to reduce the communication cost of distributed training and shorten the overall time of model training.

[0005] In order to solve the above technical problems, one aspect of the present invention is to provide an asynchronous distributed training system, characterized in that it includes a group manager and n machine nodes node1 to node 2 connected to the group manager via a network. n , the group manager includes a group generation unit, a group clearing unit and a node information table; wherein,

[0006] Each machine node i It is configured to iteratively train the model based on the local dataset;

[0007] The group generation unit is configured to be based on the machine node node i The grouping request is grouped according to the preset grouping conditions. i Grouping and updating the node information table;

[0008] The group clearing unit is configured to be based on the machine node node i The release request clears the machine node node i group information and updating the node information table;

[0009] The node information table is used to record the n machine nodes node1 to node n The state information can be shared by the group generation unit and the group clearing unit;

[0010] The preset grouping condition is that the following conditions (A) and (B) are met at the same time:

[0011] (A) Able to communicate with the machine node that makes the group request i Machine nodes divided into the same group j In the ungrouped state, and the machine node node j The number of group requests and the machine node node i The difference in the number of group requests is not greater than the difference threshold;

[0012] (B) setting the number of machine nodes in the same group to a fixed value, and when the number of machine nodes that meet condition (A) is less than the fixed value, grouping is performed without waiting;

[0013] Here, n is an integer greater than or equal to 2, i=0 to n, j=0 to n, and j≠i.

[0014] In a specific solution, the difference threshold value ranges from 10 to 20.

[0015] In a specific solution, the fixed value ranges from 3 to 10.

[0016] In a specific solution, the machine node node i Each iterative training of the model includes the following steps in sequence: gradient calculation, model update, group request, intra-group synchronization, and group release.

[0017] In a specific solution, the machine node nodei Gradient calculation is performed based on the local data set and the model is updated according to the calculation results, and then a grouping request is sent to the group generation unit. After receiving the group information of the group, the model update parameters are synchronized within the group, and then a release group information is sent to the group clearing unit.

[0018] In a specific solution, the status information recorded in the node information table includes at least: the number of each machine node, information on whether each machine node has been grouped, the number of grouping requests for each machine node, and group information of machine nodes grouped in the same group.

[0019] In order to solve the above technical problems, another aspect of the present invention is to provide an asynchronous distributed training method. Based on the above asynchronous distributed training system, the method includes:

[0020] The machine node node i Performing gradient calculation based on the local data set and updating the model according to the calculation results, and then issuing a grouping request to the group generation unit;

[0021] The group generation unit determines the machine node node i Is it grouped? If yes, read the corresponding group information from the node information table and send it to the machine node node i And update the node information table; if not, the machine node node is grouped according to the preset grouping conditions i Group and send group information to the machine node node i And update the node information table;

[0022] The machine node node i Synchronize the model update parameters within the group according to the received group information, and then send a release group information to the group clearing unit;

[0023] The group clearing unit clears the machine node node according to the released group information i group information and updating the node information table;

[0024] The machine node node i Repeat the above steps until the model converges.

[0025] In a specific solution, the group generation unit updates the node information table by: updating the node information table with the machine node node i All machine nodes that are grouped in the same group are set to grouped state, and the machine node node i The number of group requests increases by 1, and the machine node node is recorded i Group information.

[0026] In a specific solution, the group clearing unit updates the node information table by: i Set to ungrouped state, and set the machine node node i The group information is set to empty.

[0027] The asynchronous distributed training system and method provided by the embodiment of the present invention: Each machine node node i After the gradient calculation and model update are completed, a grouping request is initiated. The group generation unit only groups the machine nodes that request grouping, forming a non-overlapping grouping method. Furthermore, the number of machine nodes in the same group is set to a fixed value, and when the number of machine nodes that meet the grouping conditions does not reach the fixed value requirement, grouping is performed without waiting, forming a no-wait grouping mechanism. Based on the non-overlapping grouping method and no-wait grouping mechanism described above, the system and method provided by the present invention reduce the communication waiting time of distributed training, reduce the overall time of distributed training, and thus improve the performance of distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the structure of the asynchronous distributed training system in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the present invention more apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Examples of these preferred embodiments are illustrated in the accompanying drawings. The embodiments of the present invention shown in and described with reference to the accompanying drawings are merely exemplary, and the present invention is not limited to these embodiments.

[0030] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0031] The embodiment of the present invention first provides an asynchronous distributed training system. Figure 1 The asynchronous distributed training system includes a group manager and machine nodes connected to the group manager via a network. The group manager includes a group generation unit, a group clearing unit, and a node information table. The number of machine nodes is n, including machine nodes node1 to node n ( Figure 1 Only one of the machine nodes is shown as an example), where n is an integer greater than 2.

[0032] Among them, each machine nodei (i=0~n) is configured to iteratively train the model based on the local data set. Specifically, the machine node node i Each iterative training of the model includes the following steps in sequence: gradient calculation, model update, group request, intra-group synchronization, and group release.

[0033] More specifically, the machine node node i Gradient calculation is performed based on the local data set and the model is updated according to the calculation results, and then a grouping request is sent to the group generation unit. After receiving the group information of the group, the model update parameters are synchronized within the group, and then a release group information is sent to the group clearing unit.

[0034] Wherein, the group generation unit is configured to be based on the machine node node i The grouping request is grouped according to the preset grouping conditions. i Grouping is performed and the node information table is updated.

[0035] Each machine node i After the gradient calculation and model update are completed, a grouping request is initiated. The group generation unit only groups the machine nodes that request grouping, forming a non-overlapping grouping method, reducing the communication waiting time of distributed training and lowering the communication cost.

[0036] Specifically, if Figure 1 As shown, the working process of the group generation unit includes: according to the received machine node node i The group request information first determines the machine node node i Whether it has been grouped, if it has been grouped, then read the corresponding group information from the node information table and send it to the machine node node i And update the node information table; if not, the machine node node is grouped according to the preset grouping conditions i Group and send group information to the machine node node i And update the node information table.

[0037] The preset grouping condition is that the following conditions (A) and (B) are met at the same time:

[0038] Condition (A): Can communicate with the machine node that makes the group request i Machine nodes divided into the same group j (j=0~n and j≠i) is in the ungrouped state, and the machine node node j The number of group requests and the machine node node iThe difference in the number of group requests is not greater than the difference threshold.

[0039] Condition (B): The number of machine nodes in the same group is set to a fixed value, and when the number of machine nodes that meet condition (A) is less than the fixed value, grouping is performed without waiting.

[0040] Since each machine node i After the gradient calculation and model update are completed, a group request is initiated. The number of group requests can reflect the machine node i Therefore, based on condition (A), by setting the difference in the number of group requests to be no greater than the difference threshold, we can filter out machine nodes with relatively consistent training progress (or training performance) for grouping, which can improve the overall efficiency and performance of distributed training.

[0041] The difference threshold in condition (A) can be set according to actual needs, and is generally determined based on the total number of machine nodes included in the system. In a preferred embodiment, the difference threshold ranges from 10 to 20, for example, 10, 11, 12, 13, 14, 15, 16, 17, 19, 19, or 20.

[0042] Based on condition (B), a wait-free grouping mechanism is formed, which further reduces the communication waiting time of distributed training, reduces the communication cost, reduces the overall time of distributed training, and improves the performance of distributed training.

[0043] The fixed value in condition (B) can be set according to actual needs, and is usually determined based on the total number of machine nodes included in the system. In a preferred embodiment, the fixed value ranges from 3 to 10, for example, 3, 4, 5, 6, 7, 8, 9, or 10.

[0044] Wherein, the group clearing unit is configured to be based on the machine node node i The release request clears the machine node node i The group information and update the node information table. Specifically, Figure 1 As shown, the working process of the group clearing unit includes: according to the received machine node node i The group release information of the machine node node i The group information is cleared and the node information table is updated.

[0045] The node information table is used to record the n machine nodes node1 to node n The status information can be shared by the group generation unit and the group clearing unit. Figure 1As shown, the status information recorded in the node information table includes at least: the number of each machine node, information on whether each machine node has been grouped, the number of grouping requests for each machine node, and group information of machine nodes grouped in the same group.

[0046] Specifically, in the embodiment of the present invention, Figure 1 As shown in FIG, in the information on whether each machine node has been grouped, “1” indicates grouped and “0” indicates not grouped. In some other embodiments, “1” may indicate not grouped and “0” may indicate grouped. The group information in the node information table mainly displays the machine nodes that are grouped in the same group, for example Figure 1 In the example, machine nodes numbered 0, 1, and 2 are grouped in the same group, while machine node numbered 3 is in an ungrouped state, and the group information is empty (None).

[0047] Based on the above-described asynchronous distributed training system, an embodiment of the present invention further provides an asynchronous distributed training method, which includes the following steps:

[0048] Step 1: the machine node i Gradient calculation is performed based on the local data set, and the model is updated according to the calculation result, and then a grouping request is sent to the group generation unit.

[0049] Step 2: The group generation unit determines the machine node node i Is it grouped? If yes, read the corresponding group information from the node information table and send it to the machine node node i And update the node information table; if not, the machine node node is grouped according to the preset grouping conditions i Group and send group information to the machine node node i And update the node information table.

[0050] Step 3: The machine node i According to the received group information, the model update parameters are synchronized within the group, and then a release group information is sent to the group clearing unit.

[0051] Step 4: The group clearing unit clears the machine node node according to the released group information i and updates the node information table.

[0052] Step 5: The machine node i Repeat the above steps until the model converges.

[0053] Specifically, see Figure 1 The working steps performed in the machine node are as follows:

[0054] Step 11, Gradient calculation: Calculate the gradient using the local data set;

[0055] Step 12, model update: update the trained model using the gradient calculated in step 11;

[0056] Step 13, requesting grouping: sending a grouping request to the group generation unit and waiting for the grouping result;

[0057] Step 14, Intra-group synchronization: Based on the group information received in step 13, parameters are transferred among the machine nodes in the group for synchronization;

[0058] Step 15, releasing the group: sending a release group message to the group clearing unit to clear the group information of the machine node.

[0059] Specifically, see Figure 1 The working steps performed in the group generation unit are as follows:

[0060] Step 21, get the group request message: that is, receive the message from the machine node node i The group request message sent;

[0061] Step 22, determine whether the group has been formed: read the machine node node from the node information table i If the grouping status information is "0", that is, not grouped, proceed to step 23; if it is "1", that is, grouped, proceed to step 25;

[0062] Step 23, filter the grouping machine nodes: According to the grouping condition (A) described above, filter out the machine nodes that can make the grouping request i Machine nodes divided into the same group j ;

[0063] Step 24, Grouping: Group the request nodes according to the grouping condition (B) described above i Group and continue to Step 26;

[0064] Step 25: Get group information from the node information table and continue to Step 26.

[0065] Step 26. Send group information: Send the divided groups to the machine node that requested the grouping i ;

[0066] Step 27, update the node information table. The group generation unit updates the node information table including: iAll machine nodes that are grouped in the same group are set to grouped state, and the machine node node i The number of group requests increases by 1, and the machine node node is recorded i Group information.

[0067] Specifically, see Figure 1 The working steps performed in the group clearing unit are as follows:

[0068] Step 31, get the group release message: that is, receive the message from the machine node node i Release packet message sent;

[0069] Step 32, clear group information: clear machine node i Group information;

[0070] Step 33, update the node information table. The group clearing unit updates the node information table by: i Set to ungrouped state, and set the machine node node i The group information is set to empty.

[0071] Based on the asynchronous distributed training system and method provided in the above embodiments, five neural networks, VGG-16, ResNet-50, MobileNet, DenseNet-121 and GoogleNet, were trained respectively in the embodiments of the present invention. The training results show that the asynchronous distributed training system and method of the present invention perform well in a homogeneous environment, and significantly shorten the communication time in a heterogeneous environment, especially a network heterogeneous environment, thereby improving the overall training performance.

[0072] The asynchronous distributed training system and method provided by the embodiments of the present invention, based on a non-overlapping grouping method and a no-wait grouping mechanism, reduces the communication waiting time of distributed training, reduces the overall time of distributed training, and thus improves the performance of distributed training.

[0073] It should be noted that the above embodiments are merely illustrative of the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent variations or modifications made in accordance with the spirit and substance of the present invention are intended to be encompassed within the scope of protection of the present invention.

Claims

1. An asynchronous distributed training system, characterized in that: It includes a group manager and n machine nodes node1 to node 2 connected to the group manager via a network. n , the group manager includes a group generation unit, a group clearing unit and a node information table; wherein, Each machine node i It is configured to iteratively train the model based on the local dataset; The group generation unit is configured to be based on the machine node node i The grouping request is grouped according to the preset grouping conditions. i Grouping and updating the node information table; The group clearing unit is configured to be based on the machine node node i The release request clears the machine node node i group information and updating the node information table; The node information table is used to record the n machine nodes node1 to node n The state information can be shared by the group generation unit and the group clearing unit; The preset grouping condition is that the following conditions (A) and (B) are met at the same time: (A) Able to communicate with the machine node that makes the group request i Machine nodes divided into the same group j In the ungrouped state, and the machine node node j The number of group requests and the machine node node i The difference in the number of group requests is not greater than the difference threshold; (B) setting the number of machine nodes in the same group to a fixed value, and when the number of machine nodes that meet condition (A) is less than the fixed value, grouping is performed without waiting; Here, n is an integer greater than or equal to 2, i=0 to n, j=0 to n, and j≠i.

2. The asynchronous distributed training system according to claim 1, characterized in that: The difference threshold value ranges from 10 to 20.

3. The asynchronous distributed training system according to claim 1, characterized in that: The fixed value ranges from 3 to 10.

4. The asynchronous distributed training system according to claim 1, characterized in that: The machine node node i Each iterative training of the model includes the following steps in sequence: gradient calculation, model update, group request, intra-group synchronization, and group release.

5. The asynchronous distributed training system according to claim 4, characterized in that: The machine node node i Gradient calculation is performed based on the local data set and the model is updated according to the calculation results, and then a grouping request is sent to the group generation unit. After receiving the group information of the group, the model update parameters are synchronized within the group, and then a release group information is sent to the group clearing unit.

6. The asynchronous distributed training system according to claim 1, characterized in that: The status information recorded in the node information table includes at least: the number of each machine node, information on whether each machine node has been grouped, the number of grouping requests for each machine node, and group information of machine nodes grouped in the same group.

7. An asynchronous distributed training method, characterized in that: Based on the asynchronous distributed training system according to any one of claims 1 to 6, the method includes: The machine node node i Performing gradient calculation based on the local data set and updating the model according to the calculation results, and then issuing a grouping request to the group generation unit; The group generation unit determines the machine node node i Is it grouped? If yes, read the corresponding group information from the node information table and send it to the machine node node i And update the node information table; if not, the machine node node is grouped according to the preset grouping conditions i Group and send group information to the machine node node i And update the node information table; The machine node node i Synchronize the model update parameters within the group according to the received group information, and then send a release group information to the group clearing unit; The group clearing unit clears the machine node node according to the released group information i group information and updating the node information table; The machine node node i Repeat the above steps until the model converges.

8. The asynchronous distributed training method according to claim 7, characterized in that: The group generation unit updates the node information table including: i All machine nodes that are grouped in the same group are set to grouped state, and the machine node node i The number of group requests increases by 1, and the machine node node is recorded i Group information.

9. The asynchronous distributed training method according to claim 7, characterized in that: The group clearing unit updates the node information table by: i Set to ungrouped state, and set the machine node node i The group information is set to empty.

Images