An asynchronous distributed machine learning method based on ordered momentum

Abstract

This invention discloses an asynchronous distributed machine learning method based on ordered momentum. It incorporates momentum into the asynchronous stochastic gradient descent method, building upon the concept of ordered momentum. On the server side, gradients are organized into ordered momentum according to their iteration indices. Here, ordered momentum is a weighted sum of several gradient buckets; after a gradient reaches the server, it is placed into the corresponding gradient bucket based on its iteration index. Compared to asynchronous stochastic gradient descent, this method does not introduce additional storage overhead besides the momentum term and can effectively improve the algorithm's convergence performance and the model's generalization performance. Compared to existing synchronous distributed machine learning methods using momentum, this method can significantly improve the speed of distributed training in a training cluster, especially in scenarios where the computing power of the worker nodes in the training cluster is heterogeneous, where the speed advantage of this method is even more pronounced.

Description

Technical Field

[0001] This invention relates to an asynchronous distributed machine learning method based on ordered momentum, which is a method for asynchronous distributed learning of each worker node in a training cluster. Background Technology

[0002] Many machine learning models can be formalized into optimization problems of the following finite sum form:

[0003]

[0004] Where w represents the model parameters and d represents the dimension of the model parameters. Let f(w; ξ) be the training sample set, and let f(w; ξ) represent the loss function corresponding to sample ξ.

[0005] Stochastic gradient descent (SGD) and its variants are widely used to solve the problems mentioned above. In each iteration, SGD samples one or a mini-batch of training samples to compute the stochastic gradient as an estimate of the full gradient to update the model parameters. In practical applications, momentum is often introduced into SGD as a key technique to achieve faster convergence and better generalization performance. Many popular machine learning libraries (such as TensorFlow and PyTorch) include SGD with momentum (SGDm) as one of their optimizers.

[0006] In recent years, the size of machine learning models and datasets has been increasing rapidly, often making it insufficient for a single machine to complete the training task within an acceptable timeframe. Distributed learning accelerates the training process by distributing computational tasks across multiple machines. Due to its necessity for large-scale machine learning training tasks, distributed learning has become a hot research topic in recent years. Existing distributed learning methods can be divided into two main types: synchronous distributed learning methods and asynchronous distributed learning methods. In synchronous distributed learning methods, worker nodes that have completed computation must remain idle while waiting for other worker nodes. Therefore, the speed of synchronous distributed learning methods is often affected by slow worker nodes. In contrast, asynchronous distributed learning methods aim to solve the problem of slow worker nodes slowing down training. The most representative asynchronous distributed learning method is Asynchronous Stochastic Gradient Descent (ASGD) and its variants. In ASGD, once a worker node completes gradient calculation, it immediately uses the gradient to update parameters without waiting for other worker nodes to complete computation.

[0007] In deep model training, momentum techniques are widely recognized for their advantages in optimization and generalization. In synchronous distributed learning methods, momentum is extensively applied across various fields, including decentralized methods and communication compression algorithms. However, in synchronous distributed learning, faster training nodes must wait for slower nodes, thus the overall training speed of the distributed system is constrained by the slower nodes in the cluster. In contrast, asynchronous distributed learning methods are not constrained by slower nodes, but currently, reliable and efficient asynchronous distributed learning methods using momentum are lacking. Summary of the Invention

[0008] Objective: To address the problems and shortcomings of existing technologies, this invention provides an asynchronous distributed machine learning method based on ordered momentum. This method introduces momentum into the asynchronous stochastic gradient descent (ASTD) method, which can improve the convergence performance of ASTD and enable the trained machine learning model to have better generalization performance. Compared with existing synchronous distributed machine learning methods using momentum, this method can significantly improve the speed of distributed training, especially in scenarios where the computing power of each worker node in the training cluster is heterogeneous, where the speed advantage of this method is even more obvious.

[0009] In the method of this invention, gradients are organized into ordered momentum on the server side according to their iteration indices. After the gradients arrive at the server side, they are placed into the corresponding gradient storage buckets according to their iteration indices. It is worth noting that the gradient storage buckets here do not physically exist; this method does not introduce additional storage overhead except for the momentum term.

[0010] Technical solution: An asynchronous distributed machine learning method based on ordered momentum, comprising the following:

[0011] (11) Initialize the model parameters, momentum, iteration index and latest gradient bucket index on the server side. The server sends the initial model parameters and iteration index to all worker nodes.

[0012] (12) The server determines whether the gradient belonging to the next gradient bucket is being computed by a worker node based on the latest gradient bucket index. If so, it updates the parameters, momentum, and latest gradient bucket index. Otherwise, the parameters, momentum, and latest gradient bucket index remain unchanged. The server receives the gradient and its iteration index from a worker node, uses the gradient to update the momentum and parameters, and updates the iteration index. The server returns the updated parameters and iteration index to the worker node. This process continues until the model training ends.

[0013] (21) The specific steps of training the process at a certain working node are as follows:

[0014] Input training data;

[0015] Receive model parameters and their iteration indexes sent by the server node; randomly sample training data from the local dataset, calculate the stochastic gradient based on the selected training data, send the calculated stochastic gradient and iteration indexes to the server node, determine whether a stop message has been received from the server node, if not, continue to receive model parameters and their iteration indexes sent by the server node to perform gradient calculations and participate in model training; otherwise, end training.

[0016] The steps in (11) are as follows:

[0017] Step 100: Input the number of iterations T, momentum coefficient β, learning rate η, total number of worker nodes K, latest gradient bucket index b0 = 0, and other hyperparameters;

[0018] Step 101: Initialize model parameters w0 and momentum u0 = 0;

[0019] Step 102, initialize the iteration index t = 0;

[0020] Step 103: Send the initial model parameters w0 and iteration index t=0 to all working nodes.

[0021] The steps in (12) include the following:

[0022] Step 104: Based on the latest bucket index, determine whether the gradient belonging to the next gradient bucket is being calculated by a worker node. If... Then perform the following steps to update the parameters, momentum, and latest gradient bucket index; otherwise, the parameters, momentum, and latest gradient bucket index remain unchanged: b t+1 =b t Proceed directly to step 107; here K refers to the total number of worker nodes, b t w t u t These refer to the latest gradient bucket index, model parameters, and momentum, respectively. and These are intermediate variables generated during the parameter and momentum update process, respectively.

[0023] Step 105, update parameters using momentum

[0024] Step 106, multiply the momentum by the momentum coefficient. And shift the latest gradient bucket index forward by one position (b). t+1 =b t +1;

[0025] Step 107: Receive gradients from a working node and the iterative index of the gradient ite(k) t ,t), where k t Indicates the index of this worker node;

[0026] Step 108, update momentum using the gradient. Here This indicates the index of the gradient bucket to which the gradient belongs;

[0027] Step 109, update the parameters using the gradient.

[0028] Step 110, update the iteration index t = t + 1;

[0029] Step 111, change the latest parameter w t+1 The iteration index t+1 is returned to the worker node k. t ;

[0030] Step 112: Determine whether the stopping condition t=T has been met. If not, proceed to step 104.

[0031] Step 113: Notify each working node to stop training.

[0032] In (21), the specific steps of the training process of the method of the present invention on the k-th working node are as follows:

[0033] Step 200: Input training data;

[0034] Step 201: Receive model parameters w sent by the server node. t′ and its iterative index t′;

[0035] Step 202: Randomly sample training data ξ from the local dataset. k ;

[0036] Step 203, based on the selected training data ξ k Calculate the stochastic gradient

[0037] Step 204: Send the calculated stochastic gradient. and its iterative index t′ to the server node;

[0038] Step 205: Determine whether a stop-work message has been received from the server node. If not, return to step 201 and continue training; if yes, end training.

[0039] The gradient storage buckets in steps 100, 104, 106, and 108 are defined as follows: the 0th gradient storage bucket is... The i-th gradient storage bucket is defined as Here, i ≥ 1. For the gradient at iteration index t... For example, the gradient bucket index it belongs to is The latest gradient bucket for momentum refers to the gradient bucket used to store the latest gradient in that momentum. For momentum u t+1 For example, its latest gradient bucket index is

[0040] Beneficial Effects: Compared with existing technologies, the asynchronous distributed machine learning method based on ordered momentum provided by this invention organizes gradients into ordered momentum on the server side according to their iteration indices. Here, the ordered momentum is a weighted sum of several gradient buckets; after a gradient reaches the server, it is placed into the corresponding gradient bucket according to its iteration index. This method introduces momentum into the asynchronous stochastic gradient descent method, which can improve the convergence performance of the algorithm and the generalization performance of the model. Compared with existing synchronous distributed machine learning methods using momentum, this method can significantly improve the speed of distributed training, especially in scenarios where the computing power of each worker node in the training cluster is heterogeneous, where the speed advantage of this method is even more obvious. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the workflow of the asynchronous distributed machine learning method based on ordered momentum implemented in this invention on a server node.

[0042] Figure 2 This is a flowchart illustrating the workflow of the asynchronous distributed machine learning method based on ordered momentum implemented in this invention on the worker node. Detailed Implementation

[0043] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0044] The asynchronous distributed machine learning method based on ordered momentum provided by this invention is applicable to various tasks such as image classification, text classification, and speech recognition. Taking image classification as an example, in the method of this invention, image data is stored on several working nodes, while the machine learning model parameters are maintained by the server node. The specific workflow of the method of this invention in this task is as follows:

[0045] The asynchronous distributed machine learning method based on ordered momentum has the following workflow on the server node: Figure 1As shown in the diagram. First, input the iteration count T, momentum coefficient β, learning rate η, total number of worker nodes K, and the latest gradient bucket index b0 = 0 (step 100); initialize the model parameters w0 and momentum u0 = 0. The input of this model is the image data to be classified, and the output is the predicted image category (step 101); initialize the iteration index t = 0 (step 102); send the initial model parameters and iteration index to all worker nodes (step 103). Then, proceed to the iterative phase of model training: based on the latest bucket index, determine whether the gradient belonging to the next gradient bucket is being calculated by a worker node. If... Then perform the following steps to update the parameters, momentum, and latest gradient bucket index; otherwise, the parameters, momentum, and latest gradient bucket index remain unchanged: b t+1 =b t Proceed directly to step 107 (step 104); update parameters using momentum. (Step 105); Multiply the momentum by the momentum coefficient And shift the latest gradient bucket index forward by one position (b). t+1 =b t +1 (Step 106); Receive gradient from a working node and the iterative index of the gradient ite(k) t ,t), here k t Indicate the working node index (step 107); use this gradient to update momentum. Here Indicate the gradient bucket index to which the gradient belongs (step 108); use this gradient to update the parameters. (Step 109); Update the iteration index t = t + 1 (Step 110); Set the latest parameter w t+1 The iteration index t+1 is returned to the worker node k. t (Step 111); Determine whether the stopping condition t=T has been met. If not, jump to step 104 (Step 112); Notify each working node to stop training (Step 113), and end training.

[0046] The asynchronous distributed machine learning method based on ordered momentum has the following workflow at the k-th node: Figure 2 As shown, the model on the k-th worker node is the same as the model on the server node. The input of the model is the image data to be classified, and the output is the predicted image category. First, the image data on the k-th node is input (step 200); then the model training phase begins: the model parameters w sent by the server node are received. t′ and its iterative index t′ (step 201); randomly sample image data ξ from the local image dataset. k(Step 202); Based on the selected image data ξ k Calculate the stochastic gradient (Step 203); Send the calculated stochastic gradient The iterative index t′ is traced to the server node (step 204); it is determined whether a stop-work message has been received from the server node. If not, the process returns to step 201 and continues training; if so, the training ends (step 205).

[0047] The method of this invention was experimentally tested on an image classification dataset, and compared with existing asynchronous distributed machine learning methods using momentum. Experimental results show that the method of this invention achieves the best convergence performance and better model generalization performance. Simultaneously, the method of this invention was also compared with synchronous distributed machine learning methods using momentum; experimental results show that the method of this invention can significantly reduce the time cost of model training.

Claims

1. An asynchronous distributed machine learning method based on ordered momentum, characterized in that, Includes the following: (11) Initialize the model parameters, momentum, iteration index and latest gradient bucket index on the server side, and send the initial model parameters and iteration index to all worker nodes; (12) The server determines whether the gradient belonging to the next gradient bucket is being computed by a worker node based on the latest gradient bucket index; if so, it updates the parameters, momentum and the latest gradient bucket index; otherwise, the parameters, momentum and the latest gradient bucket index remain unchanged; it receives the gradient and the iteration index of the gradient from a worker node, uses the gradient to update the momentum and parameters, and updates the iteration index; it returns the updated parameters and iteration index to the worker node; until the model training ends. (21) The specific steps of training the process at a certain working node are as follows: Input training data; Receive model parameters and their iteration index sent by the server node; randomly sample training data from the local dataset, calculate the stochastic gradient based on the selected training data, send the calculated stochastic gradient and iteration index to the server node, determine whether a stop message has been received from the server node, if not, continue to receive model parameters and their iteration index sent by the server node to perform gradient calculation and participate in model training; otherwise, end training. The steps in (11) are as follows: Step 100, Input the number of iterations momentum coefficient Learning rate Total number of working nodes Latest gradient bucket index and other hyperparameters; Step 101, Initialize model parameters and momentum ; Step 102, Initialize the iteration index ; Step 103: Send the initial model parameters to all worker nodes. and iterative index ; The steps in (12) are as follows: Step 104: Based on the latest bucket index, determine whether the gradient belonging to the next gradient bucket is being calculated by a worker node. If... If the condition is met, then the following steps are performed to update the parameters, momentum, and latest gradient bucket index; otherwise, the parameters, momentum, and latest gradient bucket index remain unchanged: , , Proceed directly to step 107; This refers to the total number of working nodes. , , These refer to the latest gradient bucket index, model parameters, and momentum, respectively. and These are intermediate variables generated during the parameter and momentum update process, respectively. Step 105, update parameters using momentum ; Step 106, multiply the momentum by the momentum coefficient. And shift the latest gradient bucket index one position to the front. ; Step 107: Receive gradients from a working node and the iteration index of the gradient , here Indicates the index of this worker node; Step 108, update momentum using the gradient. , here This indicates the index of the gradient bucket to which the gradient belongs; Step 109, update the parameters using the gradient. ; Step 110, Update the iteration index ; Step 111, add the latest parameters and its iterative index Return to work node ; Step 112: Determine if the stopping condition has been met. If not, proceed to step 104; Step 113: Notify each working node to stop training.

2. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that, In (21), in the first The specific steps of the training process on each working node are as follows: Step 200: Input training data; Step 201: Receive model parameters sent by the server node. and its iterative index ; Step 202: Randomly sample training data from the local dataset. ; Step 203, based on the selected training data Calculate the stochastic gradient ; Step 204: Send the calculated stochastic gradient. and its iterative index To the server node; Step 205: Determine whether a stop-work message has been received from the server node. If not, return to step 201 and continue training; if yes, end training.

3. The asynchronous distributed machine learning method based on ordered momentum according to claim 1, characterized in that, The gradient storage bucket is defined as follows: the 0th gradient storage bucket is... , No. Each gradient storage bucket is defined as... ,here For iteration index is gradient For example, the gradient bucket index it belongs to is The latest gradient bucket for momentum refers to the gradient bucket used to store the latest gradient in that momentum. For example, its latest gradient bucket index is .

Images