Asynchronous distributed machine learning method based on ordered momentum
By introducing an ordered momentum mechanism on the server side, the problem of lack of efficient driving force in asynchronous distributed machine learning is solved, and training speed and model performance are improved in heterogeneous computing scenarios.
Patent Information
- Application Number
- CN202411000289.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-24
AI Technical Summary
The existing asynchronous distributed machine learning methods lack reliable and efficient driving force implementation, resulting in limited training speed in scenarios where computing capabilities in training clusters are heterogeneous.
The ordered momentum mechanism is introduced on the server side, and the gradient iterative index is organized into ordered momentum, and parameters and momentum are updated between working nodes to avoid storage overhead and improve the convergence performance and model generalization performance of the asynchronous stochastic gradient descent method.
The speed of distributed training is significantly improved, especially in scenarios with heterogeneous computing capabilities, which improves the convergence and generalization performance of the model and reduces the training time overhead.
Smart Images

Figure CN118917435A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an asynchronous distributed machine learning method based on ordered momentum, which is a method for performing asynchronous distributed learning on working nodes in a training cluster. Background Art
[0002] Many machine learning models can be formalized as optimization problems of the following finite sum form:
[0003]
[0004] Where w represents the model parameter, d represents the model parameter dimension, is the training sample set, and f(w;ξ) represents the loss function corresponding to the sample ξ.
[0005] Stochastic gradient descent (SGD) and its variants are widely used to solve the above problems. In each iteration, SGD samples one or a small batch of training samples to calculate the stochastic gradient as an estimate of the full gradient to update the model parameters. In practical applications, momentum is often introduced into SGD as a key technology to obtain faster convergence and better generalization performance. Many popular machine learning libraries (such as TensorFlow and PyTorch) include SGD with momentum (SGDm) as one of the optimizers.
[0006] In recent years, the size of machine learning models and data sets has increased rapidly, and a single machine is often not enough to complete the training task of machine learning models within an acceptable time. Distributed learning accelerates the training process by distributing computing tasks on multiple machines. Due to its necessity for large-scale machine learning training tasks, distributed learning has become a hot research topic in recent years. Existing distributed learning methods can be divided into two main types: synchronous distributed learning methods and asynchronous distributed learning methods. In synchronous distributed learning methods, the working nodes that have completed the calculation must remain idle to wait for other working nodes. Therefore, the speed of synchronous distributed learning methods is often affected by slow working nodes. In contrast, asynchronous distributed learning methods aim to solve the problem that slow working nodes slow down the training speed. The most representative asynchronous distributed learning methods are asynchronous stochastic gradient descent (ASGD) and its variants. In ASGD, once a working node completes the gradient calculation, the gradient is used immediately for parameter update without waiting for other working nodes to complete the calculation.
[0007] In deep model training, momentum technology is widely recognized for its advantages in optimization and generalization. In synchronous distributed learning methods, momentum is widely used in various fields, including decentralized methods, communication compression algorithms, and so on. In synchronous distributed learning methods, nodes with fast training speeds need to wait for nodes with slow training speeds, so the training speed of the entire distributed system will be restricted by the slow nodes in the cluster. In comparison, the training speed of asynchronous distributed learning methods will not be restricted by slow nodes in the cluster, but there is currently a lack of reliable and efficient asynchronous distributed learning methods with momentum. Summary of the invention
[0008] Purpose of the invention: In view of the problems and shortcomings in the prior art, the present invention provides an asynchronous distributed machine learning method based on ordered momentum. This method introduces momentum into the asynchronous stochastic gradient descent method, which can improve the convergence performance of the asynchronous stochastic gradient descent method and make the trained machine learning model have better generalization performance. Compared with the existing synchronous distributed machine learning method with momentum, this method can greatly improve the speed of distributed training, especially in the scenario where the computing power of each working node in the training cluster is heterogeneous, the speed advantage of this method is more obvious.
[0009] In the method of the present invention, the gradients are organized into ordered momentum in the order of their iteration indexes on the server side. After the gradients arrive at the server side, they are placed in the corresponding gradient storage buckets according to their iteration indexes. It is worth noting that the gradient storage buckets here do not exist physically, and this method does not introduce additional storage overhead except for the momentum term.
[0010] Technical solution: An asynchronous distributed machine learning method based on ordered momentum, including the following contents:
[0011] (11) Initialize model parameters, momentum, iteration index, and the latest gradient bucket index on the server side. The server sends the initial model parameters and iteration index to all working nodes.
[0012] (12) The server determines whether the gradient belonging to the next gradient bucket is being calculated by a certain working node based on the latest gradient bucket index; if so, it updates the parameters, momentum and the latest gradient bucket index; otherwise, the parameters, momentum and the latest gradient bucket index remain unchanged; receives the gradient and the iteration index of the gradient from a certain working node, uses the gradient to update the momentum and parameters, and updates the iteration index; returns the updated parameters and iteration index to the working node; until the model training is completed;
[0013] (21) The specific steps of the training process on a certain working node are:
[0014] Input training data;
[0015] Receive the model parameters and iteration index sent by the server node; randomly sample training data from the local data set, calculate the stochastic gradient based on the selected training data, send the calculated stochastic gradient and iteration index to the server node, and determine whether the stop working message sent by the server node is received. If not, continue to receive the model parameters and iteration index sent by the server node to perform gradient calculation and participate in model training; otherwise, end the training.
[0016] The method (11) comprises the following steps:
[0017] Step 100, input the number of iterations T, momentum coefficient β, learning rate η, total number of working nodes K, latest gradient bucket index b0=0, and other hyperparameters;
[0018] Step 101, initializing model parameters w0 and momentum u0=0;
[0019] Step 102, initializing iteration index t=0;
[0020] Step 103: Send the initial model parameter w0 and iteration index t=0 to all working nodes.
[0021] The method (12) comprises the following steps:
[0022] Step 104: determine whether the gradient belonging to the next gradient bucket is being calculated by a working node based on the latest bucket index. Then execute the following steps to update the parameters, momentum, and the latest gradient bucket index; otherwise, the parameters, momentum, and the latest gradient bucket index remain unchanged: b t+1 =b t , jump directly to step 107; here K refers to the total number of working nodes, b t 、w t 、u t They refer to the latest gradient bucket index, model parameters, and momentum respectively; and They are the intermediate variables generated in the process of parameter and momentum update;
[0023] Step 105, update parameters using momentum
[0024] Step 106, multiply the momentum by the momentum coefficient And move the latest gradient bucket index forward by one position b t+1 =b t +1;
[0025] Step 107: Receive gradients from a worker node and the iteration index of the gradient ite(k t ,t), where k t Indicates the index of the working node;
[0026] Step 108, using the gradient to update momentum Here Indicates the gradient bucket index to which the gradient belongs;
[0027] Step 109: Use the gradient to update the parameters
[0028] Step 110, update iteration index t=t+1;
[0029] Step 111: update the latest parameter w t+1 and its iteration index t+1 is returned to the worker node k t ;
[0030] Step 112, determine whether the stop condition t=T is met, if not, jump to step 104;
[0031] Step 113, notify each working node to stop training.
[0032] In (21), the specific steps of the training process of the method of the present invention on the kth working node are:
[0033] Step 200, input training data;
[0034] Step 201: Receive the model parameter w sent by the server node t′ and its iteration index t′;
[0035] Step 202: randomly sample training data ξ from the local data set k ;
[0036] Step 203, based on the selected training data ξ k , calculate the stochastic gradient
[0037] Step 204, send the calculated stochastic gradient and its iteration index t′ to the server node;
[0038] Step 205, determining whether a stop working message sent by the server node is received, if not, returning to step 201 to continue training; if yes, ending the training.
[0039] The gradient buckets in steps 100, 104, 106, and 108 are defined as follows: The 0th gradient bucket is The i-th gradient bucket is defined as Here i ≥ 1. For the gradient of iteration index t For example, the gradient bucket index to which it belongs is The latest gradient bucket of momentum refers to the gradient bucket used to store the latest gradient in the momentum. t+1 For example, its latest gradient bucket index is
[0040] Beneficial effects: Compared with the prior art, the asynchronous distributed machine learning method based on ordered momentum provided by the present invention organizes the gradients into ordered momentum in the order of their iteration indexes on the server side. The ordered momentum here is the weighted sum of several gradient storage buckets. After the gradient arrives at the server side, it is placed in the corresponding gradient storage bucket according to its iteration index. This method introduces momentum into the asynchronous stochastic gradient descent method, which can improve the convergence performance of the algorithm and the generalization performance of the model. Compared with the existing synchronous distributed machine learning methods with momentum, this method can greatly improve the speed of distributed training, especially in the scenario where the computing power of each working node in the training cluster is heterogeneous, the speed advantage of this method is more obvious. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flowchart of the asynchronous distributed machine learning method based on ordered momentum implemented by the present invention on a server node;
[0042] Figure 2 The flowchart of the asynchronous distributed machine learning method based on ordered momentum implemented in the present invention on the working node. DETAILED DESCRIPTION
[0043] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0044] The asynchronous distributed machine learning method based on ordered momentum provided by the present invention can be applied to a variety of tasks such as image classification, text classification, and speech recognition. Taking the image classification task as an example, in the method of the present invention, the image data is stored on several working nodes, and the machine learning model parameters are maintained by the server node. The specific workflow of the method of the present invention in this task is as follows:
[0045] The asynchronous distributed machine learning method based on ordered momentum has the following workflow on the server node: Figure 1As shown. First, input the number of iterations T, momentum coefficient β, learning rate η, total number of working nodes K, and the latest gradient bucket index b0=0 (step 100); initialize the model parameters w0 and momentum u0=0. The input of the model is the image data to be classified, and the output is the predicted image category (step 101); initialize the iteration index t=0 (step 102); send the initial model parameters and iteration index to all working nodes (step 103). Then enter the iteration phase of model training: based on the latest bucket index, determine whether the gradient belonging to the next gradient bucket is being calculated by a certain working node. If Then perform the following steps to update the parameters, momentum, and the latest gradient bucket index, otherwise the parameters, momentum, and the latest gradient bucket index remain unchanged: b t+1 =b t , jump directly to step 107 (step 104); use momentum to update parameters (Step 105); Multiply the momentum by the momentum coefficient And move the latest gradient bucket index forward by one position b t+1 =b t +1 (step 106); receive gradients from a worker node and the iteration index of the gradient ite(k t ,t), where k t represents the working node index (step 107); use this gradient to update the momentum Here Indicates the gradient bucket index to which the gradient belongs (step 108); uses the gradient to update the parameter (Step 109); Update the iteration index t=t+1 (Step 110); Set the latest parameter w t+1 and its iteration index t+1 is returned to the worker node k t (Step 111); determine whether the stop condition t=T is reached, if not, jump to step 104 (step 112); notify each working node to stop training (step 113), and end the training.
[0046] The asynchronous distributed machine learning method based on ordered momentum has the following workflow on the kth node: Figure 2 As shown in FIG. 1 , the model on the kth working node is the same as the model on the server node. The input of the model is the image data to be classified, and the output is the predicted image category. First, the image data on the kth node is input (step 200); then the model training phase begins: the model parameters w sent by the server node are received. t′ and its iteration index t′ (step 201); randomly sample image data ξ from the local image dataset k(Step 202); Based on the selected image data ξ k , calculate the stochastic gradient (Step 203); Send the calculated stochastic gradient and its iteration index t′ to the server node (step 204); determine whether a stop working message sent by the server node is received, if not, return to step 201 and continue training; if so, end the training (step 205).
[0047] The method of the present invention was experimented on an image classification dataset, and the method of the present invention was compared with an existing asynchronous distributed machine learning method with momentum. The experimental results show that the method of the present invention can achieve the best convergence performance and better model generalization performance. At the same time, the method of the present invention was also compared with a synchronous distributed machine learning method with momentum, and the experimental results show that the method of the present invention can significantly reduce the time overhead of model training.
Claims
1. An asynchronous distributed machine learning method based on ordered momentum, characterized in that: It includes the following: (11) Initialize model parameters, momentum, iteration index, and the latest gradient bucket index on the server side. The server sends the initial model parameters and iteration index to all working nodes. (12) The server determines whether the gradient belonging to the next gradient bucket is being calculated by a certain working node based on the latest gradient bucket index; if so, it updates the parameters, momentum and the latest gradient bucket index; otherwise, the parameters, momentum and the latest gradient bucket index remain unchanged; receives the gradient and the iteration index of the gradient from a certain working node, uses the gradient to update the momentum and parameters, and updates the iteration index; returns the updated parameters and iteration index to the working node; until the model training is completed; (21) The specific steps of the training process on a certain working node are: Input training data; Receive the model parameters and iteration index sent by the server node; randomly sample training data from the local data set, calculate the stochastic gradient based on the selected training data, send the calculated stochastic gradient and iteration index to the server node, and determine whether the stop working message sent by the server node is received. If not, continue to receive the model parameters and iteration index sent by the server node to perform gradient calculation and participate in model training; otherwise, end the training.
2. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that: The method (11) comprises the following steps: Step 100, input the number of iterations T, momentum coefficient β, learning rate η, total number of working nodes K, latest gradient bucket index b0=0, and other hyperparameters; Step 101, initializing model parameters w0 and momentum u0=0; Step 102, initializing iteration index t=0; Step 103: Send the initial model parameter w0 and iteration index t=0 to all working nodes.
3. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that: The method (12) comprises the following steps: Step 104: determine whether the gradient belonging to the next gradient bucket is being calculated by a working node based on the latest bucket index. Then perform the following steps to update the parameters, momentum, and the latest gradient bucket index; otherwise, the parameters, momentum, and the latest gradient bucket index remain unchanged: b t+1 =b t , jump directly to step 107; K refers to the total number of working nodes, b t 、w t 、u t They refer to the latest gradient bucket index, model parameters, and momentum respectively; and They are the intermediate variables generated in the process of parameter and momentum update; Step 105, update parameters using momentum Step 106, multiply the momentum by the momentum coefficient And move the latest gradient bucket index forward one position b t+1 =b t +1; Step 107: Receive gradients from a worker node and the iteration index of the gradient ite(k t ,t), where k t Indicates the index of the working node; Step 108, using the gradient to update momentum Here Indicates the gradient bucket index to which the gradient belongs; Step 109: Use the gradient to update the parameters Step 110, update the iteration index t=t+1; Step 111: The latest parameter w t+1 and its iteration index t+1 is returned to the worker node k t ; Step 112, determine whether the stop condition t=T is reached, if not, jump to step 104; Step 113, notify each working node to stop training.
4. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that: In (21), the specific steps of the training process on the kth working node are: Step 200, input training data; Step 201: Receive the model parameter w sent by the server node t′ and its iteration index t′; Step 202: randomly sample training data ξ from the local data set k ; Step 203, based on the selected training data ξ k , calculate the stochastic gradient Step 204, send the calculated stochastic gradient and its iteration index t′ to the server node; Step 205, determining whether a stop working message sent by the server node is received, if not, returning to step 201 to continue training; if yes, ending the training.
5. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that: The gradient bucket is defined as follows: The 0th gradient bucket is The i-th gradient bucket is defined as Here i ≥ 1; for the gradient with iteration index t For example, the gradient bucket index to which it belongs is The latest gradient bucket of momentum refers to the gradient bucket used to store the latest gradient in the momentum. t+1 For example, its latest gradient bucket index is
Citation Information
Patent Citations
Distributed depth learning system based on momentum and pruning
CN109299781A
Method for reducing distributed machine learning communication overhead
CN110287031A
Large-batch decentralized distributed image classifier training method and system
CN114186671A
Batch size adjusting method in distributed data parallel online asynchronous training
CN118245809A
Method, device, and system for determining distributed training algorithm framework configuration
WO2023123275A1
