Distributed machine learning method based on local learning strategy

Abstract

The invention discloses a distributed machine learning method based on a local learning strategy, which is based on a parameter server architecture, is suitable for multi-machine cluster distributed machine learning of a data center, and is also suitable for client-cloud collaborative distributed machine learning taking a server as a cloud end, and a mobile phone or embedded equipment as a terminal. The method comprises the following steps: firstly, a server node accumulating a local gradient sum calculated by all working nodes to obtain a full gradient, and broadcasting the full gradient to all the working nodes; each working node updating the parameters for several times and then sending the local parameters to the server node; finally, the server node averaging the parameters collectedfrom the working nodes and broadcasting the average value as the latest parameter to all the working nodes; iterating the above process for multiple rounds until a convergence condition is achieved. The method is based on a local learning strategy, communication does not need to be carried out after parameters of the working nodes are updated every time, and therefore the communication expenditurein distributed machine learning is reduced.

Description

technical field [0001] The invention provides a distributed machine learning method based on a local learning strategy, relates to a distributed algorithm in the field of machine learning, and can effectively reduce communication overhead in distributed machine learning. Background technique [0002] Most machine learning models can be formalized as the following optimization problems: [0003] [0004] Where w represents the parameters of the model, n represents the total number of training samples, f i (·) represents the loss function corresponding to the i-th sample. To solve the above optimization problems, stochastic gradient descent (SGD) and its variants are currently the most widely used methods. As the amount of training data increases, the training process of many machine learning problems takes a lot of time. The distributed algorithm disperses the training data to multiple nodes for parallel training to speed up the training process of machine learning. [...

AI Technical Summary

Problems solved by technology

[0007] With the increase of the machine learning model and the number of nodes participating in distributed co

Images