Method for reducing distributed machine learning communication overhead

Abstract

The invention discloses a method for reducing distributed machine learning communication overhead, which is based on a parameter server architecture, is suitable for multi-machine cluster distributed machine learning of a data center, and is also suitable for end cloud collaborative distributed machine learning with a server as a cloud end, a mobile phone or an embedded device as a terminal. The method comprises the following steps: firstly, calculating gradients of all working nodes, solving a global momentum by combining two rounds of parameter difference, summing the global momentum and a previous round of memory gradient to obtain a new round of memory gradient, sending a part of the new round of memory gradient to a server node, and accumulating the rest part; then enabling the server node to accumulate all sparse memory gradient sums so as to update parameters and broadcast the parameter difference of two rounds to all working nodes; and finally, enab;omh the working node to receive the two rounds of parameter difference and update the parameters. The method is based on global gradient compression, and only a part of global momentum is transmitted when the working node communicates with the server node, so that the communication overhead in distributed machine learning is reduced.

Description

technical field [0001] The invention provides a method for reducing the communication overhead of distributed machine learning, which can effectively reduce the communication overhead in distributed machine learning. Background technique [0002] Most machine learning models can be formalized as the following optimization problems: [0003] [0004] Where w represents the parameters of the model, n represents the total number of training samples, ξ i Represents the i-th sample, f(w; ξ i ) represents the loss function corresponding to the i-th sample, and d represents the model size. To solve the above optimization problems, stochastic gradient descent (SGD) and its variants are currently the most widely used methods. The momentum gradient descent method (MSGD) in the variant uses exponential weighted average for the gradient, which reduces the influence of the gradient this time, reduces the fluctuation, and converges more stably when it is close to the minimum value. ...

AI Technical Summary

Problems solved by technology

However, as the scale of the distributed cluster becomes larger and larger, the transfer of gradients and the synchronization of parameters prolong the communication time and become a bottleneck for further efficiency improvement.

Images