An adaptive synchronization mechanism to accelerate distributed machine learning

Abstract

The application relates to the technical field of distributed machine learning, and discloses an adaptive synchronization mechanism for accelerating distributed machine learning, which adopts different parallel training schemes for work nodes with different training speeds adaptively under the target of improving gradient staleness, and improves existing BSP and ASP schemes. Based on the staleness value between the fastest work node and the slowest work node, the mechanism adaptively adds a relaxed synchronization barrier for the faster work node to limit or reduce the staleness value, thereby improving the gradient staleness. In order to reduce the influence of the slow gradient in the synchronization, a differentiated weighted gradient aggregation method is used for aggregating the gradient. While limiting the fast work node, the mechanism adopts an asynchronous parallel training scheme for the slow work node to accelerate the training speed of the slow work node. Through the above strategies, the mechanism improves the gradient staleness problem in the ASP, overcomes the'straggler' problem of the BSP, and can improve the training speed of the distributed machine learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of accelerating distributed machine learning, and particularly relates to an adaptive synchronization mechanism for accelerating distributed machine learning. BACKGROUND

[0002] The method of training machine learning model by multi-machine or multi-GPU cluster is called distributed training, which effectively solves the problem of long training time and low training efficiency of single-machine training large model or large data set. Parameter server (PS) and All-Reduce are two main system architectures of distributed training. At the same time, according to the different division of data set and model, distributed training can be divided into data parallel and model parallel. In data parallel distributed training based on parameter server, Bulk Synchronous Parallel (BSP) and Asynchronous Parallel (ASP) are two mainstream parallel training schemes of distributed training. BSP can usually achieve the highest model accuracy and stability, but due to the existence of the problem of "straggler", the training time of BSP is longer. In each iteration, due to the difference in computing speed between work nodes or network delay, the work node with the fastest speed needs to wait for the work node with slower speed. In this waiting time, the expensive computing resources will be idle, and these work nodes with slower speed are called "stragglers". Compared with BSP, ASP can usually achieve the shortest training time in a fixed epoch (one complete data set traversal is called an epoch), but due to the lack of any synchronization, ASP will cause serious gradient staleness problem, which will affect the accuracy and stability of model convergence, and even cause the model to eventually fail to converge. In ASP, due to the lack of any synchronization, the version of global model parameters and the version of gradient pushed by work nodes may be inconsistent, which is called gradient staleness.

[0003] Obviously, BSP and ASP have their own advantages and disadvantages.

[0004] Prior art document

[0005] [1]. A. V. Gerbessiotis and L. G. Valiant, "Direct bulk-synchronous parallel algorithms," Journal of parallel and distributed computing, vol. 22, no. 2, pp. 251-267, 1994.

[0006] [2] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems, 2012, pp. 1223-1231.

[0007] [3] S. U. Stich, “Local SGD converges fast and communicates little,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.

[0008] [4] Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing, “More effective distributed ml via a stale synchronous parallel parameter server,” in Advances in Neural Information Processing Systems, 2013, pp. 1223-1231.

[0009] [5] Xing Zhao*, Aijun An*, Junfeng Liu , Bao Xin Chen*, “Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning,” in 2019 IEEE 39th International Conference on Distributed Computing Systems.

[0010] [6]. Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.

[0011] [7]. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of ICML, pages 448-456, 2015.

[0012] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017.

[0013] [3][4][5] tried to improve BSP and ASP. Local-SGD proposed in [3] reduces the time of communication waiting by reducing the frequency of communication synchronization in BSP, but this method may damage the model accuracy, and the threshold of the frequency of communication synchronization is difficult to determine. SSP proposed in [4] improves the problem of gradient staleness in ASP by specifying that the gradient staleness between the worker node with the fastest training speed and the worker node with the slowest training speed does not exceed a certain threshold. However, this method requires the user to specify the staleness threshold. Users often do not know how to set the staleness threshold, and it is often necessary to debug frequently to determine an optimal staleness threshold, which is time-consuming. DSSP proposed in [5] changes the staleness threshold in SSP to dynamic, thereby solving the problem of manually specifying the staleness threshold in SSP. However, the dynamic threshold is estimated according to the iteration time interval of the worker node, and when the network changes greatly, the estimation of the staleness threshold may have a large error.

[0014] The application is similar to the SSP and the DSSP, and is used for controlling the time lag between the fastest working node and the slowest working node, but the application does not force the fastest working node to stop and wait for the slowest working node, because the expensive computing resources will be idle during the forced waiting process. The application adopts an adaptive synchronization method to avoid the time lag between the fastest working node and the slowest working node being too large. Specifically, the application is based on the time lag between the fastest working node and the slowest working node, and adopts a relaxed synchronous parallel training strategy for the working node with a faster training speed, and adopts a completely asynchronous parallel training strategy for the remaining working nodes with a slower training speed. Meanwhile, in the relaxed synchronous parallel training strategy, the training iteration numbers of the synchronized working nodes may be different, so that the traditional average method cannot be used to aggregate the gradient in the synchronization, and the difference in the iteration numbers of the working nodes should be considered for the weighted aggregation. Therefore, the application adopts a differentiated weighted gradient aggregation method in the aggregation process of the synchronization, effectively reduces the influence of the slow gradient, and improves the model performance.

[0015] Because the relaxed synchronous parallel training strategy is adopted for the faster working nodes, a relaxed synchronization barrier is added in the faster working nodes, so that the speed of the faster working nodes is slowed down. Because the synchronization is relaxed, the training speed of the faster working nodes will not be slowed down too much (to achieve a lighter gradient time lag with less synchronization), and the asynchronous parallel training strategy is adopted for the remaining slower working nodes, so that the slower working nodes can maintain a faster speed. The process is adaptively changed towards the goal of improving the gradient time lag, and this is a mixed way of synchronization and asynchronization, so that higher stability and faster training speed can be achieved. SUMMARY

[0016] (I) Technical problems solved

[0017] The application aims to provide an adaptive synchronization mechanism for accelerating distributed machine learning, and achieves the purposes of improving the parallel training efficiency and improving the gradient time lag problem in the ASP.

[0018] (II) Technical solutions

[0019] The application provides the following technical solutions: an adaptive synchronization mechanism for accelerating distributed machine learning, comprising the following steps:

[0020] S1, in the initial training stage, each working node adopts the ASP scheme for training;

[0021] S2, the server detects the stale value s between the fastest worker node and the slowest worker node in each epoch, when s is greater than 1 epoch, the server adopts a relaxed BSP scheme for the top s ranked worker nodes, unlike BSP, the relaxed BSP scheme allows the server to relax the synchronization condition, that is, the server can directly aggregate the received gradient after waiting for a certain number of iterations, without waiting for all gradients to be received before aggregation, and in the relaxed BSP scheme, a differentiated weighted gradient aggregation method is used to aggregate the gradient, and for the remaining worker nodes, because their ranking is later, the training speed is slower, the server adopts the ASP scheme for them;

[0022] S3, in each round of training, s is detected, and according to s and the change of the training speed of the worker node, different parallel training schemes (relaxed BSP or ASP) are adaptively adopted for each worker node.

[0023] Preferably 1, the steps of implementing the adaptive synchronization mechanism are as follows:

[0024] S11, the implementation steps of the worker node are:

[0025] (1) each worker node independently loads a part of the data set, and calculates the loss (Cost), accuracy (Accuracy) and gradient (Gradient);

[0026] (2) the worker node pushes the gradient to the parameter server through the TCP communication protocol;

[0027] (3) wait to receive the latest model parameters returned from the parameter server;

[0028] (4) replace the local model parameters with the latest model parameters, so as to perform the next iteration training, that is, cyclically execute steps (1), (2), (3) and (4).

[0029] S11, the implementation of the parameter server is divided into two stages: initial stage and adaptive synchronization stage, the operation of the parameter server is started from the initial stage, and then gradually transits to the adaptive synchronization stage, and according to the severity of the gradient staleness, the initial stage and the adaptive synchronization stage will adaptively switch to each other:

[0030] Initial stage:

[0031] (1) wait to receive the gradient from the worker node, and record the iteration number and the training round number (the training round number refers to the epoch, and one complete traversal of the data set is recorded as an epoch);

[0032] (2) According to the number of iterations, the parameter server detects whether all worker nodes have completed a complete epoch, if not, after executing the following 1) and 2) steps, then return to the initial stage step (1);

[0033] 1) The parameter server directly uses the received gradient to update the global model parameters, and the updating method uses stochastic gradient descent (SGD, Stochastic Gradient Descent), and the stochastic gradient descent is shown in formula (1):

[0034] W t+1 =W t -decay_α*g i (1)

[0035] Wherein, W t+1 represents the global model parameters saved in the parameter server in the t+1 iteration, decay_α represents the decay learning rate, g i represents the gradient from the i-th worker node;

[0036] 2) The latest global model parameters W t+1 are unicast to the corresponding i-th worker node;

[0037] After the above step (2), if yes (that is, all worker nodes have completed a complete epoch), the parameter server ranks all worker nodes according to the recorded training epoch, and calculates the epoch difference s of the fastest worker node and the slowest worker node, wherein:

[0038] 3) If s <= 1, the parameter server directly executes 1) and 2) steps, and then returns to the initial stage step (1); if s > 1, the server puts the top s worker nodes into the synchronous group, and puts the remaining worker nodes into the asynchronous group, then executes 1) and 2) steps in (2), and the parameter server enters the adaptive synchronization stage:

[0039] Adaptive synchronization stage:

[0040] (3) Wait to receive the gradient from the worker node, continue to record the iteration number and the training round (epoch) of the worker node while inheriting the initial stage;

[0041] (4) If there is at least one gradient in the aggregation list (that is, the aggregation list described in the preferred 2) at this time, the parameter server starts the relaxation counter, and the relaxation counter counts plus 1 for each received gradient;

[0042] (5) Judge whether the received gradient is from the worker node in the synchronous group;

[0043] (6) If the gradient fraction is from a worker node in the synchronous group, the following procedures are executed according to the preferred 2, 3, 4, 5 and 6;

[0044] (7) If the gradient fraction is from a worker node in the asynchronous group, the steps 1) and 2) in the initial stage are executed;

[0045] (8) The steps (3) (4) (5) (6) (7) are executed in a loop until the training of the worker node ends.

[0046] Preferred 2, after the step (5) in the adaptive synchronous stage, if the gradient fraction is from a worker node in the synchronous group, the following operations are executed:

[0047] 1) The parameter server does not immediately update the global model parameters, but adds the gradient fraction to an aggregation list.

[0048] 2) If the number of gradient fractions in the aggregation list is not equal to the number of all worker nodes in the synchronous group or the relaxation counter count is less than or equal to the preset relaxation factor, return to the step (3) in the adaptive synchronous stage.

[0049] Preferred 3, after the step 2) in the preferred 2, if the number of gradient fractions in the aggregation list is equal to the number of all worker nodes in the synchronous group or the relaxation counter count is greater than the preset relaxation factor, the gradients in the aggregation list are aggregated at this time, and the steps ① ② ③ ④ are executed:

[0050] ① A differentiated weighted gradient aggregation method is used to aggregate the gradients, which is shown in the formulas (2) (3):

[0051]

[0052]

[0053] Wherein, iteration i represents the iteration number of the i-th worker node, s' represents the number of gradient fractions received by the parameter server and needing to be synchronized and aggregated, wherein s'≤s, s is the epoch difference between the fastest worker node and the slowest worker node, weight i represents the differentiated weighting proportion of the i-th worker node based on the iteration number, g i represents the gradient of the i-th worker node, and g represents the aggregated gradient.

[0054] ② The aggregated gradient g is used to update the global model parameters, and the updating method is shown in the formula (1), and the updated global model parameters are sent to the corresponding s worker nodes;

[0055] ③ The relaxation counter count is reset to 0;

[0056] (4) According to the number of iterations, it is detected whether all worker nodes have completed a complete epoch.

[0057] Preferred 4, in preferred 3, if yes (i.e. all worker nodes have completed a complete epoch), then: the parameter server ranks all worker nodes according to the recorded training epochs, and calculates the epoch difference s of the fastest worker node and the slowest worker node.

[0058] Preferred 5, in preferred 4, if s>1, then update the synchronous group and the asynchronous group, put the top s worker nodes into the synchronous group, and put the remaining worker nodes into the asynchronous group, otherwise s<=1, return to step (1) of the initial stage.

[0059] Preferred 6, in preferred 3, if there is a worker node that has not completed a complete epoch, then return to step (3) of the adaptive synchronization stage.

[0060] Preferred 7, relaxation mechanism, we observed that if a complete synchronization (i.e. BSP) strategy is adopted for fast workers, the PS usually needs to spend a lot of time in this synchronization waiting, so that the slow worker with a higher ranking becomes the fastest worker and the gap with the slowest worker is widened. In order to coordinate the speed between workers and avoid unnecessary waiting, we adopt a relaxed synchronization strategy for fast workers, i.e. according to experimental observation, we artificially set a suitable relaxation factor. After the parameter server receives at least one gradient of the synchronous group, the relaxation counter starts counting. When the count is greater than the preset relaxation factor, the received gradient can be directly aggregated without waiting for all fast workers to receive the gradient before aggregation. Experimental observation shows that when the relaxation factor is 3 or 5, the best training speed and accuracy can be achieved.

[0061] Preferred 8, model performance evaluation mechanism, equations (4), (5), (6) are used to compare the performance of BSP, ASP and adaptive synchronization mechanism. The essence is to evaluate the convergence accuracy and convergence time of the distributed training scheme according to the sliding average value of the difference between the accuracy of the previous and the next epoch.

[0062] Δacc i =acc i -acc i-1 (4)

[0063] move i =β*move i-1 -(1-β)*Δacci (5)

[0064] biased_acc i =move i / 1-β i (6)

[0065] where acc i denotes the accuracy at the i-th epoch, β is used to control the length of the sliding window, move i and biased_acc i are the moving average and the corrected bias of the accuracy, respectively. When biased_acc i is very small, it means that the model has converged.

[0066] Preferably 9, a dynamic adjustment mechanism of learning rate, dynamically decays the learning rate α according to equation (7),

[0067]

[0068] where α is the initial learning rate, decay_rate is the decay rate, which is less than 1, and the greater the value, the faster the initial learning rate decays; e min takes the minimum training round number in the worker node; decay_α represents the decayed learning rate, which is used to guide the update proportion of the model parameters in each round.

[0069] Preferably 10, an adaptive synchronization barrier mechanism, whose technical principle is as follows: in each epoch, all workers are ranked according to the number of training epochs, and the difference between the fastest worker and the slowest worker is calculated, denoted as s. If s>1, then in the next training epoch, the top s workers (fast workers) adopt a relaxed synchronous parallel training scheme (relaxed BSP), and in the synchronization, a differentiated gradient aggregation method is used to aggregate the gradient; the workers other than the top s workers (slow workers) adopt an asynchronous parallel training scheme (ASP).

[0070] Compared with the prior art, the application provides an adaptive synchronization mechanism for accelerating distributed machine learning, which realizes a slight gradient staleness with as few synchronizations as possible, and has the following beneficial effects:

[0071] (1) The application is an adaptive method, in which whether BSP or ASP is adopted by each worker is adaptively changed in each training round, without human setting.

[0072] (2), the application absorbs the advantages of BSP and ASP, avoids their respective problems, has a faster training speed than BSP, and has higher stability than ASP.

[0073] (3), the application can accelerate distributed training like ASP, and does not cause serious gradient staleness problems.

[0074] (4), the application proposes an adaptive synchronization mechanism, which is a hybrid parallel training scheme, which adopts different parallel training schemes (relaxed BSP or ASP) for work nodes with different training speeds, thereby improving the parallel training efficiency.

[0075] (5), the application improves the gradient staleness problem in ASP, wherein, in order to reduce the influence of fast gradient, an adaptive synchronization barrier is added to the work node with faster training speed, and a differentiated weighted gradient aggregation method is proposed to reduce the influence of slow gradient in the synchronization barrier.

[0076] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is a general schematic diagram of the adaptive synchronization mechanism of the application;

[0078] Figure 2 is a runtime schematic diagram of the adaptive synchronization mechanism of the four work nodes of the application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments of the application.

[0080] As shown in Figures 1-2 in order to absorb the advantages of BSP and ASP and avoid their respective problems, the application proposes an adaptive synchronization mechanism based on the parameter server architecture, and the system overall design of the mechanism is as shown in Figure 1The ASP scheme is used for training in the initial training stage. Then, the parameter server detects the outdated value s between the fastest worker node and the slowest worker node in each epoch. When s is greater than 1 epoch, the parameter server uses the relaxed BSP scheme for the top s worker nodes. Unlike the BSP scheme, the relaxed BSP scheme allows the server to relax the synchronization condition, that is, the server can directly aggregate the received gradient after waiting for a certain number of iterations, without waiting for all gradients to be received before aggregation. In the BSP scheme, a differentiated weighted gradient aggregation method is used to aggregate the gradient. For the remaining worker nodes, because their ranking is later and the training speed is slower, the parameter server uses the ASP scheme for them. Finally, the parameter server detects s in each round of training, and adaptively uses different parallel training schemes (relaxed BSP or ASP) for each worker node according to s and the change of the training speed of the worker node.

[0081] The application provides an adaptive synchronization mechanism for accelerating distributed machine learning, comprising the following steps:

[0082] S1, using the ASP scheme for training in the initial training stage of each worker node;

[0083] S2, the server detects the outdated value s between the fastest worker node and the slowest worker node in each epoch. When s is greater than 1 epoch, the server uses the relaxed BSP scheme for the top s worker nodes. Unlike the BSP scheme, the relaxed BSP scheme allows the server to relax the synchronization condition, that is, the server can directly aggregate the received gradient after waiting for a certain number of iterations, without waiting for all gradients to be received before aggregation. In the BSP scheme, a differentiated weighted gradient aggregation method is used to aggregate the gradient. For the remaining worker nodes, because their ranking is later and the training speed is slower, the server uses the ASP scheme for them.

[0084] S3, detecting s in each round of training, and adaptively using different parallel training schemes (relaxed BSP or ASP) for each worker node according to s and the change of the training speed of the worker node.

[0085] Further, the steps of implementing the adaptive synchronization mechanism are as follows:

[0086] S11, implementation of the worker node:

[0087] (1) Each worker node independently loads a part of the data set and calculates the loss (Cost), accuracy (Accuracy) and gradient (Gradient);

[0088] (2) The worker node pushes the gradient to the parameter server through the TCP communication protocol;

[0089] (3) Wait to receive the latest model parameters returned from the parameter server;

[0090] (4) Replace the local model parameters with the latest model parameters, and proceed to the next iteration training, that is, cyclically execute steps (1), (2), (3), and (4);

[0091] S12, the implementation of the parameter server is divided into two stages: an initial stage and an adaptive synchronization stage, the operation of the parameter server is started by the initial stage, and then gradually transitions to the adaptive synchronization stage, and the initial stage and the adaptive synchronization stage will adaptively switch to each other according to the severity of the gradient staleness:

[0092] Initial stage:

[0093] (1) Wait to receive the gradient from the worker node, and record the iteration number and the training epoch;

[0094] (2) According to the iteration number, detect whether all worker nodes have completed a complete epoch, if not, execute the following 1) and 2) steps, and then return to the initial stage step (1);

[0095] 1) The parameter server directly updates the global model parameters using the received gradient, and the update method uses stochastic gradient descent, which is shown in formula (1):

[0096] W t+1 = W t -decay_α*g i (1)

[0097] Where, W t+1 represents the global model parameters saved in the server at the t+1 iteration, decay_α represents the decay learning rate, and decay_α is calculated as shown in formula (7), g i represents the gradient from the i-th worker node.

[0098] 2) Unicast the latest global model parameters W t+1 to the corresponding i-th worker node;

[0099] After step (2), if yes (that is, all worker nodes have completed a complete epoch), rank all worker nodes according to the recorded training epoch, and calculate the epoch difference s of the fastest worker node and the slowest worker node, where:

[0100] 3) If s<=1, the server directly executes the steps 1) and 2), and then returns to the initial stage step (1); if s>1, the server puts the top s workers into the synchronous group, and the rest of the workers into the asynchronous group, and then executes the steps 1) and 2) in (2), and then the server enters the adaptive synchronization stage:

[0101] Adaptive synchronization stage:

[0102] (3) Wait to receive the gradient from the worker, and continue to record the iteration number and the training epoch of the worker while inheriting the initial stage;

[0103] (4) If there is at least one gradient in the aggregation list (i.e., the aggregation list in step (6) of the adaptive synchronization stage) at this time, the parameter server starts a relaxation counter, and the relaxation counter counts up by 1 for each received gradient;

[0104] (5) Determine whether the received gradient is from the worker in the synchronous group;

[0105] (6) If the gradient is from the worker in the synchronous group, the following operations are performed:

[0106] 1) The server does not immediately update the global model parameters, but adds the gradient to an aggregation list;

[0107] 2) If the number of gradients in the aggregation list is not equal to the number of workers in the synchronous group or the relaxation counter count is less than or equal to the preset relaxation factor, return to step (3) of the adaptive synchronization stage; otherwise, aggregate the gradients in the aggregation list at this time, and execute the steps ① ② ③ ④:

[0108] ① A differentiated weighted gradient aggregation method is used to aggregate the gradients, which is shown in the formulas (2) and (3):

[0109]

[0110]

[0111] wherein, iteration i represents the iteration number of the i-th worker, s' represents the number of gradients received by the parameter server that need to be synchronized and aggregated, wherein s'<=s, s is the epoch difference between the fastest worker and the slowest worker, weight i represents the differentiated proportion of the i-th worker based on the iteration number, g i represents the gradient of the i-th worker, and g represents the aggregated gradient.

[0112] The gradient g after the aggregation is used to update the global model parameters, and the method for updating is shown in formula (1), and the global model parameters after the updating are sent to the corresponding s working nodes.

[0113] The relaxation counter is reset to 0.

[0114] The iteration number is detected to determine whether all the working nodes have completed a complete epoch.

[0115] If yes, that is, all the working nodes have completed a complete epoch, then the following is performed: all the working nodes are ranked according to the recorded training epochs, and the epoch difference s between the fastest working node and the slowest working node is calculated, wherein: if s>1, the synchronous group and the asynchronous group are updated, the working nodes ranked in the top s are put into the synchronous group, and the remaining working nodes are put into the asynchronous group; otherwise s<=1, and the initial stage step (1) is returned.

[0116] In step 4 in (5), otherwise: the adaptive stage step (3) is returned.

[0117] If the gradient is from the working node in the asynchronous group, then steps 1) and 2) in the initial stage are performed.

[0118] Steps (3), (4), (5), (6) and (7) are cyclically performed until the working node training is completed.

[0119] The above are the detailed implementation steps of the adaptive synchronization mechanism of the present application. In order to better understand the running mechanism of the present application, an example of the adaptive synchronization mechanism of the present application when four working nodes are running is given in the following. Figure 2 Figure 2 ​As shown, the initial stage 4 workers (corresponding to the workers in the figure) all start from the diagonal striped matrix block, which means they perform the ASP scheme, corresponding to steps (1) (2). When the server detects that the training epoch of the fastest worker worker1 is greater than 1 than the slowest worker worker4 (worker1 runs 3 more epochs than worker4 in the figure), the top 3 workers (worker1 for 6 epochs, worker2 for 5 epochs, and worker3 for 4 epochs) are increased with the synchronization barrier, that is, they perform the relaxed BSP scheme, so they change from the diagonal striped matrix block to the vertical striped matrix block, which corresponds to steps (3) (4) (5). While the remaining slow worker4 (running 3 epochs) performs ASP, so worker4 is still a diagonal striped matrix block, corresponding to step (6). Since the faster workers are increased with the synchronization barrier, their speed will slow down, and the slower worker performs ASP, so it can run more iterations than the fast worker in the same time. Therefore, it can be seen that after multiple iterations, the gap between worker4 and worker1 is narrowed from 3 epochs to 2 epochs. This process is self-adapting, and which workers perform BSP and which workers perform ASP is self-adapting according to the epoch difference between the fastest worker and the slowest worker and the ranking of the workers, without the need for human setting.

[0120] Running process of the adaptive synchronization mechanism:

[0121] (1) According to the set mini-batch size (manually set), the data set is randomly divided into several parts.

[0122] (2) Each worker independently reads 1 / N of the mini-batch and calculates the gradient, loss, etc., N being the number of workers.

[0123] (3) The worker sends the gradient to the parameter server and then waits until the parameter server returns the latest model parameters.

[0124] (4) The parameter server receives the gradient and performs the corresponding work according to the two stages mentioned above.

[0125] (5) The worker receives the latest model parameters returned by the parameter server and replaces the local model parameters with the latest model parameters for the next iteration training.

[0126] (6) Iterative training: return to process (1) after running processes (2) (3) (4) (5) for training-set-size / mini-batch-size times, until the model meets the convergence condition, and then end the training, wherein training-set-size is the training set size, and mini-batch-size is the size of the mini-batch.

[0127] Experimental verification:

[0128] (1) Hardware setup

[0129] The present application is tested on an OpenStack machine cluster, which is built by three Dawning I840-G30 servers. One of the servers is used as a control node, and the other two servers are used as computing nodes. The configuration of the control node is 8-core 2.1GHz CPU, 16GB RAM and 1TB DISK. Each computing node has 8-core 2.1GHz CPU, 64GB RAM and 1.8TB DISK. The total bandwidth of the physical link between computing node 1 and computing node 2 is 10Gbps. In the OpenStack machine cluster, we virtualize a Fat-tree network topology with k = 4 using NFV technology, and 9 hosts are connected to the topology, one of which is used as a parameter server and the other 8 hosts are used as worker nodes. Each Router in the Fat-tree network topology is configured with 1V CPU, 512MB RAM, 2GB DISK, and the 9 hosts are configured with 4V CPU, 10.5GB RAM, 128GB DISK.

[0130] (2) Dataset

[0131] The present application uses MNIST, CIFAR10 and CIFAR100 datasets for image classification tasks. The MNIST dataset has 55000 training images (training set), 5000 validation images (validation set), and 10000 test images (test set), and MNIST is a 10-class image classification task. While the CIFAR10 and CIFAR100 datasets both have 50000 training images and 10000 test images. CIFAR10 has 10 classes, while CIFAR100 has 100 classes. In order to improve the generalization ability of the model, we use the Mixup[6] data augmentation method for both CIFAR10 and CIFAR100 training datasets.

[0132] (3) Benchmark

[0133] The present application uses the current mainstream distributed training schemes BSP, ASP, SSP and DSSP as the benchmark for comparison, wherein the hyperparameter settings in SSP and DSSP are set according to the optimal hyperparameters given in the original paper, the outdated threshold of SSP is set to 3, and the outdated range of DSSP is set to [3, 12].

[0134] (4) Model structure

[0135] The present application builds LeNet-5, VGG-Net-small and ResNet-18-small neural network structures to evaluate BSP, ASP, SSP, DSSP and adaptive synchronization mechanism. The present application shows the model structure of VGG-Net-small and ResNet-18-small in Table 1 (LeNet-5 is a classic network, which is not shown here), wherein convX-Y represents that the size of the convolution kernel is X and the number of convolution kernels is Y; BN represents Batch Normalization [7], and SE represents SE-Net [8]; avgpool and maxpool represent average pooling layer and maximum pooling layer, respectively; in ResNet-18-small, every 2 convolution layers have a shortcut connected to the next layer

[0136] Table 1 Model structure of VGG-Net-small and ResNet-18-small

[0137]

[0138]

[0139] (5) Convergence judgment method

[0140] The present application uses the moving average of the difference between the two epochs of the test set accuracy calculated by equations (4) (5) (6) to evaluate the convergence accuracy and convergence time of the distributed training scheme,

[0141] Deltaacc i = acc i - acc i-1 (4)

[0142] Move i = beta * move i-1 - (1-beta) * deltaacc i (5)

[0143] Biased_acc i = move i / 1-beta i (6)

[0144] where acc i denotes the accuracy at the i-th epoch, and β is used to control the length of the sliding window, move i and biased_acc i are the moving average and the corrected bias of the accuracy, respectively. When biased_acc i is very small, it means that the model has converged. In the experiment, β is set to 0.9, which means that the fluctuation of the test set accuracy can be observed for about 10 epochs, move i and biased_acc i are the moving average and the corrected bias of the test set accuracy, respectively. When biased_acc i is very small, it means that the model has converged, because the fluctuation of the test set accuracy is very small in about 10 epochs. The traditional method usually sets a very large training epoch in advance to ensure the convergence of the model. However, this method is not optimal, because the model may have converged in the early stage, and the additional epochs will increase the training time. In addition, if the training time is too long, it is also easy to cause the model to overfit. The convergence judgment method according to the model precision trend can make a more accurate judgment on the convergence of the model, and can save the training time and avoid overfitting.

[0145] (6) Learning rate and batch size setting

[0146] In all experiments, the initial learning rate is 0.9, and the batch size read by the worker node is 125. The learning rate is decayed according to equation (7),

[0147]

[0148] where α is the initial learning rate, that is, 0.9; decay_rate is the decay rate, which is set to 0.99. The larger the value, the faster the learning rate decays; e min is the minimum training round in the worker node; and decay_α is used to guide the update proportion of the model parameters in each round. Through the dynamic adjustment of the learning rate, the training can be more stably transitioned to the convergence state.

[0149] (7) Experimental results

[0150] The experimental results are given in Tables 2, 3, 4 and 5, respectively. In the experiment, the adaptive synchronization mechanism of the present application is denoted as A2S. The experimental results show that the convergence accuracy and convergence speed of the adaptive synchronization mechanism (A2S) are superior to the current mainstream distributed parallel training schemes (BSP, ASP, SSP and DSSP).

[0151] Table 2 shows the biased_acc i Experimental results of training LeNet-5 on the MNIST dataset with a relaxation factor of >0.002 (A2S relaxation factor set to 8).

[0152]

[0153]

[0154] Table 3 shows the biased_acc i Experimental results of training VGG-Net-small on the CIFAR10 dataset with a relaxation factor of 5 (A2S relaxation factor set to 5) under the condition of >0.002.

[0155]

[0156] Table 4 shows the biased_acc i Experimental results of training VGG-Net-small on the CIFAR100 dataset with a relaxation factor of 5 (A2S relaxation factor set to 5) under the condition of >0.002.

[0157]

[0158]

[0159] Table 5 shows the biased_acc i Experimental results of training ResNet-18-small on the CIFAR10 dataset with a relaxation factor of 3, where the value is greater than 0.002.

[0160]

Claims

1. An adaptive synchronization method for accelerating distributed machine learning, the method comprising: The method comprises the following steps: S1, in the initial training stage, each worker node adopts an ASP scheme for training; S2, the server detects the staleness value s between the fastest worker node and the slowest worker node in each epoch, when s is greater than 1 epoch, the server adopts a relaxed BSP scheme for the top s worker nodes, which is different from the BSP scheme, the relaxed BSP scheme allows the server to relax the synchronization condition, that is, the server waits for a certain number of iterations and directly aggregates the received gradient without waiting for all gradients to be aggregated, and in the relaxed BSP scheme, a differentiated weighted gradient aggregation method is used to aggregate the gradient, and for the remaining worker nodes, because their ranking is later and the training speed is slower, the server adopts the ASP scheme for them; S3, in each round of training, s is detected, and different parallel training schemes are adaptively adopted for each worker node according to s and the change of the training speed of the worker node; The specific steps of the adaptive synchronization method are as follows: S11, the implementation steps of the worker node are as follows: (1) each worker node independently loads a part of the data set and calculates the loss (Cost), accuracy (Accuracy) and gradient (Gradient); (2) the worker node pushes the gradient to the parameter server through the TCP communication protocol; (3) wait to receive the latest model parameters returned from the parameter server; (4) replace the local model parameters with the latest model parameters, and thus perform the next iteration training, that is, loop execution; S21, the implementation of the parameter server is divided into two stages: the initial stage and the adaptive synchronization stage, the operation of the parameter server is started from the initial stage, and then gradually transitions to the adaptive synchronization stage, and the initial stage and the adaptive synchronization stage will adaptively switch to each other according to the severity of the gradient staleness: Initial stage: (10) wait to receive the gradient from the worker node, and record the iteration number and the training round number at the same time, the training round number is referred to as an epoch, and one complete traversal of the data set is referred to as an epoch; (20) according to the iteration number, the parameter server detects whether all worker nodes have completed a complete epoch, if not, the following steps 1) and 2) are executed, and then the initial stage step (10) is returned; 1) the parameter server directly updates the global model parameters using the received gradient, and the update method uses the stochastic gradient descent (SGD), and the stochastic gradient descent is shown in formula (1): W t+1 = W t -decay_α*g i (1) where W t+1 denotes the global model parameters saved in the parameter server at the t+1th iteration, decay_α denotes the decayed learning rate, g i denotes the gradient from the i-th worker node; 2) send the latest global model parameters W t+1 to the corresponding ith worker node; After step (20), if yes, that is, all worker nodes have completed a complete epoch, the parameter server ranks all worker nodes according to the recorded training epoch, and calculates the epoch difference s between the fastest worker node and the slowest worker node, wherein: 3) If s<=1, the parameter server directly executes the steps 1) and 2), and then returns to the initial stage step (10); if s>1, the server puts the top s workers into the synchronous group and the rest into the asynchronous group, and then executes the steps 1) and 2) in (20) and enters the adaptive synchronization stage: Adaptive synchronization stage: (30) Wait for receiving the gradient from the worker, and continue to record the iteration number and the training epoch of the worker while inheriting the initial stage; (40) If there is at least one gradient in the aggregation list at this time, the parameter server starts a relaxation counter, and the relaxation counter counts plus 1 for each received gradient; (50) Judge whether the received gradient is from the worker in the synchronous group; (60) If the gradient is from the worker in the synchronous group, the following operations are performed: 1) The parameter server does not immediately update the global model parameters, but adds the gradient to an aggregation list; 2) If the number of gradients in the aggregation list is not equal to the number of workers in the synchronous group or the relaxation counter count is less than or equal to the preset relaxation factor, return to the step (30) of the adaptive synchronization stage; (70) If the gradient is from the worker in the asynchronous group, execute the steps 1) and 2) in the initial stage; (80) Loop the steps (30), (40), (50), (60), (70) until the training of the worker is completed.

2. The method of claim 1, wherein, If the number of gradients in the aggregation list is equal to the number of workers in the synchronous group or the relaxation counter count is greater than the preset relaxation factor, aggregate the gradients in the aggregation list, and execute the steps ①, ②, ③, ④: ① A differentiated weighted gradient aggregation method is used to aggregate the gradients, which is shown in the formulas (2) and (3): wherein iteration i represents the iteration number of the i-th worker node, s' represents the number of gradient shares received by the parameter server that need to be synchronously aggregated, wherein s'≤s, s is the epoch difference between the fastest worker node and the slowest worker node, weight i represents the differentiated weighting ratio calculated by the i-th worker node based on the iteration number, g i represents the gradient of the i-th worker node, g represents the aggregated gradient; ② The aggregated gradient g is used to update the global model parameters, and the updated global model parameters are sent to the corresponding s' workers; ③ The relaxation counter count is reset to 0; ④ Detect whether all workers have completed a complete epoch according to the iteration number.

3. The method of claim 2, wherein, In the step ④, if all workers have completed a complete epoch, execute: the parameter server ranks all workers according to the recorded training epoch, and calculates the epoch difference s between the fastest worker and the slowest worker.

4. The method of claim 3, wherein, If the difference s>1, update the synchronous group and the asynchronous group, put the top s workers into the synchronous group and the rest into the asynchronous group, otherwise s<=1, return to the initial stage step (1).

5. The method of claim 4, wherein, In the step ④, if there is a worker that has not completed a complete epoch, return to the adaptive synchronization stage step (3).

6. The method of claim 1 to 5, wherein, The relaxation method, if a complete synchronization strategy is adopted for the fast worker, the PS usually needs to spend a lot of time in the synchronization waiting, so that the slow worker in the early stage becomes the fastest worker and the gap with the slowest worker is widened. In order to coordinate the speed of the workers and avoid unnecessary waiting, the synchronization strategy of the fast worker is relaxed, that is, according to the experimental observation, a suitable relaxation factor is artificially set. After the parameter server receives at least one gradient of the synchronization group, the relaxation counter starts counting. When the count is greater than the preset relaxation factor, the parameter server is allowed to directly aggregate the received gradient without waiting for all the fast workers to aggregate. When the relaxation factor is 3 or 5, the best training speed and accuracy can be obtained.

7. The method of claim 1 to 5, wherein, The model performance evaluation method uses equations (4), (5) and (6) to compare the performance of BSP, ASP, SSP, DSSP and adaptive synchronization method. The essence is to evaluate the convergence accuracy and convergence time of the distributed training scheme according to the sliding average value of the difference between the accuracies of the previous and next two epochs. Δacc i = acc i -acc i-1 (4) move i = β*move i-1 -(1-β)*Δacc i (5) biased_acc i = move i / 1-β i (6) where acc i represents the accuracy at the i-th epoch, β is used to control the length of the sliding window, move i and biased_acc i are the moving average of the accuracy and the corrected bias, respectively, when biased_acc i is very small, it means that the model has converged.

8. The method of claim 1 to 5, wherein, The dynamic adjustment method of learning rate dynamically decays the learning rate α according to equation (7). Wherein, a is the initial learning rate, decay_rate is the decay rate, which is less than 1, the greater the value, the slower the initial learning rate decays; e min The value is the minimum training round in the worker node; decay_a indicates the decayed learning rate, which is used to guide the update proportion of the model parameters in each round.

9. The method of claim 1 to 5, wherein, The adaptive synchronization barrier method has the following technical principle: in each epoch, all workers are ranked according to the number of training epochs, and the training epoch difference between the fastest worker and the slowest worker is calculated, denoted as s. If s>1, the top s workers are trained using the relaxed synchronization parallel training scheme in the next training epoch, and the differential gradient aggregation method is used to aggregate the gradient in the synchronization. Workers other than the top s workers are trained using the asynchronous parallel training scheme.

Images