A method for accelerating distributed training based on a packet asynchronous parallel strategy

Abstract

The application discloses a kind of based on grouping asynchronous parallel strategy acceleration distributed training method, comprising: machine performance data is collected and the performance data is calculated in real time;Grouping is carried out to node based on the performance data, and the grouping result is issued to all nodes;Each group is separately trained using asynchronous strategy, and global model is obtained;Loop iteration until global model converges.The application groups nodes adaptively through fast natural breakpoint clustering algorithm, avoids the influence of obsolete gradient and thereby improves the training efficiency of distributed system, and the method is not only suitable for static heterogeneous cluster but also suitable for dynamic heterogeneous cluster.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and deep learning, and in particular to a method for accelerating distributed training based on a grouped asynchronous parallel strategy. Background Technology

[0002] Deep learning has been widely applied in various scenarios, including computer vision, recommender systems, speech recognition, and natural language processing. As the size of training datasets and network models continues to expand, training tasks require increasingly more computing resources, which a single machine cannot handle. It may take weeks or even months to understand the characteristics of a model with millions of neurons from a terabyte (TB) dataset. To address this challenge, distributed deep learning has been proposed, where a set of computing nodes trains a neural model in parallel. This parallelism can be achieved in two ways: data parallelism and model parallelism. Data parallelism divides the training data into different fragments, with each node responsible for training one or more fragments. Model parallelism divides the neural network structure into different subnetworks, which can be trained by different nodes. Data parallelism is the most commonly used method and can be performed synchronously or asynchronously. In synchronous training, each node needs to collect and aggregate gradients from others and update the model weights in each iteration. In asynchronous training, nodes do not need to synchronize with each other; gradients are typically aggregated by a parameter server. The parameter server node collects gradients from the computing nodes, updates the global model weights, and sends new parameters to the computing nodes. With the help of the parameter server, gradient sending and aggregation can be performed asynchronously. More precisely, after sending gradients to the parameter server, a compute node can immediately begin a new iteration without waiting for other nodes. However, in the real world, most clusters are inherently heterogeneous, composed of various types of CPUs (Central Processing Units) and GPUs (Graphics Processing Units). Due to performance differences, the training progress between the slowest and fastest compute nodes can vary significantly. More precisely, the gradients calculated by the slowest compute node conflict with the latest model on the parameter server, severely affecting the model's convergence direction. This makes it difficult for the overall training to converge, requiring more iterations and longer training time for the model to converge. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is that the gradient calculated by the slowest computing node conflicts with the latest model on the parameter server, which seriously affects the convergence direction of the model and makes it difficult for the overall training to converge.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: collecting machine performance data and performing real-time calculations on the performance data; grouping nodes based on the performance data and distributing the grouping results to all nodes; employing an asynchronous strategy to train each group individually and obtaining a global model; iterating in a loop until the global model converges.

[0007] As a preferred embodiment of the distributed training method accelerated by the group asynchronous parallel strategy described in this invention, a heterogeneous distributed GPU server is constructed, and a parameter server architecture is used for gradient communication.

[0008] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, the parameter server is used to collect performance data, and a performance evaluator is used to perform real-time calculation on the machine performance data, using the number of calculable samples per second as a quantification indicator.

[0009] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, wherein: the grouping includes,

[0010] Based on the performance data, the control node uses the natural breakpoint clustering algorithm to group the nodes and sends the grouping results to all nodes.

[0011] As a preferred embodiment of the distributed training method based on the grouped asynchronous parallel strategy described in this invention, the grouping principle includes:

[0012] Group nodes with similar performance together, divide all nodes in the cluster into several groups, and use variance to measure similarity.

[0013] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, the optimal grouping result is obtained by calculating the variance of each group, then calculating the sum of these variances.

[0014] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, wherein: based on the optimal grouping result, the computing nodes in each group independently update the local models between groups using an asynchronous strategy, and each group has an independent parameter server that maintains the group-level local models.

[0015] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, wherein: an staleness threshold is set, and the parameter server is used to track the version number of the local model of each group.

[0016] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, the method includes: if the version difference between the fastest group and the slowest group exceeds the set obsolescence threshold, then pausing the update of the local model of the fastest group and performing a full group synchronization, obtaining the global model by weighted averaging of the models of all groups, and all computing nodes continuing to train using the global model.

[0017] As a preferred embodiment of the distributed training method based on the asynchronous parallel strategy of grouping described in this invention, the training is terminated by iterating in a loop until the global model converges.

[0018] The beneficial effects of this invention are as follows: This invention adaptively groups nodes using a fast natural breakpoint clustering algorithm, avoiding the influence of outdated gradients and thereby improving the training efficiency of distributed systems. This method is applicable not only to static heterogeneous clusters but also to dynamic heterogeneous clusters, such as online multi-tenant cloud clusters. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0020] Figure 1 A basic flowchart of a distributed training method based on a grouped asynchronous parallel strategy is provided as an embodiment of the present invention.

[0021] Figure 2 A schematic diagram of the training process of a distributed training method based on a grouped asynchronous parallel strategy is provided as an embodiment of the present invention;

[0022] Figure 3 A comparison chart of loss values ​​of various methods for training a ResNet101 model based on a grouped asynchronous parallel strategy to accelerate distributed training, provided as an embodiment of the present invention.

[0023] Figure 4 This is a comparison chart of the training accuracy of various methods for training a ResNet010 model based on a grouped asynchronous parallel strategy to accelerate distributed training, as provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0028] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integrated connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0030] Example 1

[0031] Reference Figure 1and Figure 2 As an embodiment of the present invention, a method for accelerating distributed training based on a grouped asynchronous parallel strategy is provided, comprising:

[0032] S1: Collects machine performance data and performs real-time calculations on the performance data. It should be noted that:

[0033] Construct a heterogeneous distributed GPU server and use a parameter server architecture for gradient communication.

[0034] Performance data is collected using a parameter server, and performance evaluation is typically performed during the first epoch. In this stage, no grouping strategy is used; instead, the most basic asynchronous strategy is employed, with each computing node training independently to ensure the accuracy of the sampling.

[0035] S2: Group nodes based on performance data and distribute the grouping results to all nodes. It should be noted that grouping includes...

[0036] Based on performance data, the control node uses the natural breakpoint clustering algorithm to group the nodes and sends the grouping results to all nodes.

[0037] Different groups are distinguished by different keys, which are integer variables. For example, if compute nodes 1 and 2 are in group 1, then the key they receive is 1. If compute nodes 3 and 4 are in group 2, then the key they receive is 2. Finally, the parameter server performs aggregation operations on the models of all groups based on the different keys. In the parameter server, each group model is assigned a version number. The initial version number is set to 1, and the version number is incremented by 1 for each aggregation operation.

[0038] Grouping principles include,

[0039] Nodes with similar performance are grouped together, all nodes in the cluster are divided into several groups, and variance is used to measure similarity; by calculating the variance of each group, the sum of these variances is calculated, and the optimal grouping result is obtained.

[0040] The quality of grouping results can be compared by using the sum of variances. The grouping result that is the smallest sum of variances is the optimal grouping result.

[0041] S3: An asynchronous strategy is used to train each group individually and obtain the global model. It should be noted that:

[0042] Based on the optimal grouping results, the computing nodes within each group adopt an asynchronous strategy to independently update the local models between groups, and each group has an independent parameter server to maintain the group-level local models.

[0043] Set an obsolescence threshold and use a parameter server to track the version number of the local model for each group.

[0044] If the version difference between the fastest and slowest groups exceeds a set obsolescence threshold, then updates to the local model of the fastest group will be paused. Figure 2 For example, G1_M1 represents the local model with version number 1 in group 1, G2_M1 represents the local model with version number 1 in group 2, and GM1 represents the global model with version number 1. At the start of training, the staleness threshold is set to δ. At a certain moment, the local model of group 1 is updated to version M, while the local model of group 2 is updated to version N. If the difference between M and N is greater than δ, the local model update of the fastest group is paused. It should be noted that only the local model update of the fastest group is paused here to ensure that the local models of slower and faster groups do not become too skewed. A full group synchronization is then performed, and the global model is obtained by weighted averaging of the models of all groups. All computing nodes continue training using the global model. The weighted average is specifically: the latest version number of the group / the sum of all version numbers.

[0045] S4: Iterate through the loop until the global model converges. It should be noted that:

[0046] Repeat step S3 until the global model converges, at which point the training can be terminated.

[0047] This invention adaptively groups nodes using a fast natural breakpoint clustering algorithm, avoiding the influence of outdated gradients and thereby improving the training efficiency of distributed systems. This method is applicable not only to static heterogeneous clusters but also to dynamic heterogeneous clusters, such as online multi-tenant cloud clusters.

[0048] Example 2

[0049] Reference Figure 3 and Figure 4 This is the second embodiment of the present invention. Unlike the first embodiment, this embodiment provides a verification test of a distributed training method accelerated by a grouped asynchronous parallel strategy. To verify and explain the technical effects of the method, this embodiment compares the traditional technical solution with the method of the present invention, and compares the experimental results with scientific demonstration methods to verify the real effect of the method.

[0050] The test environment uses four heterogeneous graphics cards: one NVIDIA 2080Ti, two NVIDIA T4s, and one NVIDIA 1080Ti. The four graphics cards are placed on four servers, and the ResNet101 network is trained using the Cifar10 dataset.

[0051] In this embodiment, the classic asynchronous SGD and the subsequent improved method K-batch asynchronous SGD will be used to compare the training accuracy, loss value and training time during the training process with the present method.

[0052] like Figure 3 As shown, this invention is mainly applicable to the training of sparse networks, such as convolutional neural network models or CTR (click-through rate) recommendation models. The figure shows that this invention reduces the loss value more quickly within a shorter training time, reaching a relatively stable value. In contrast, asynchronous SGD and K-batch asynchronous SGD, due to their inability to correctly utilize stale gradients, exhibit significantly slower loss value reduction rates compared to this invention.

[0053] pass Figure 4 The figure shows that the present invention has a faster convergence speed and a faster accuracy improvement speed, while asynchronous SGD and K-batch asynchronous SGD are difficult to converge and have a slower accuracy improvement because they cannot properly handle stale gradients. In comparison, the present invention requires more training time to achieve the same accuracy.

[0054] The results are as follows: Figure 3 , 4 As shown, this embodiment compares the convergence speed of these three methods by training a ResNet101 model on the Cifar10 dataset. In heterogeneous clusters, traditional asynchronous SGD has the worst statistical efficiency because it uses outdated parameters, making it difficult for the model to converge quickly. K-batch asynchronous SGD needs to wait for the gradients of the Top-K nodes before each parameter update, and the time for these K nodes to complete gradient calculations varies with heterogeneity and lacks similarity, resulting in low hardware efficiency. Although all nodes in K-batch asynchronous SGD use the same parameters, the global model lacks some sample data because only the gradients of a subset of nodes are taken each time, leading to a decrease in accuracy. The group-based method proposed in this paper has similar performance among different nodes within a group, eliminating the problem of gradient staleness. Furthermore, the model consistency is guaranteed under the constraint of a global aggregator, thus achieving higher statistical efficiency than these existing methods.

[0055] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for accelerating distributed training based on a grouped asynchronous parallel strategy, characterized in that, include: Collect machine performance data and perform real-time calculations on the performance data; The nodes are grouped based on the performance data, and the grouping results are sent to all nodes. The grouping includes, Based on the performance data, the control node uses the natural breakpoint clustering algorithm to group the nodes and sends the grouping results to all nodes; Grouping principles include, Group nodes with similar performance together, divide all nodes in the cluster into several groups, and use variance to measure similarity; The optimal grouping result is obtained by calculating the variance of each group, summing the variances, and then calculating the variance of each group. Based on the optimal grouping results, the computing nodes in each group update the local models between groups independently using an asynchronous strategy, and each group has an independent parameter server that maintains the group-level local models. Set an obsolescence threshold and use the parameter server to track the version number of the local model for each group; An asynchronous strategy is used to train each group separately and obtain the global model. This includes pausing the local model update of the fastest group and the slowest group if the version difference between them exceeds the set obsolescence threshold, and performing a full group synchronization. The global model is obtained by weighted averaging of the models of all groups, and all computing nodes continue training using the global model. Repeat the loop until the global model converges.

2. The method for accelerating distributed training based on a grouped asynchronous parallel strategy as described in claim 1, characterized in that: Construct a heterogeneous distributed GPU server and use a parameter server architecture for gradient communication.

3. The method for accelerating distributed training based on a grouped asynchronous parallel strategy as described in claim 1 or 2, characterized in that: The parameter server is used to collect performance data, and a performance evaluator is used to perform real-time calculations on the machine performance data, using the number of samples that can be calculated per second as a quantitative indicator.

4. The method for accelerating distributed training based on a grouped asynchronous parallel strategy as described in claim 3, characterized in that: The training process continues iteratively until the global model converges, at which point the training can be terminated.

Images