Distributed machine learning method capable of tolerating untrusted nodes

Abstract

The invention discloses a distributed machine learning method capable of tolerating untrusted nodes, which comprises the following steps that: each working node acquires latest parameters from a server node, calculates a gradient according to locally stored data, sends the gradient to the server node, and repeats the step until a stop message of a server is received. The server node is provided with a certain number of buffers, after gradient information is received each time, corresponding buffer numbers are calculated according to the numbers of the sender working nodes, and values in the buffers are updated to be average values of all gradients received corresponding to the buffers; judging whether all buffers have gradients or not, if yes, calculating a final gradient according to thegradients in all the buffers through an aggregation function, updating model parameters, and emptying all the buffers; the latest parameters are sent to the working node; and continuously repeating the training steps until a stop condition is met, and notifying each working node to stop.

Description

technical field [0001] The invention relates to a distributed machine learning method that can tolerate untrusted nodes, which can effectively reduce the negative effects brought by the erroneous gradient information of untrusted nodes in distributed machine learning, and improve the robustness of distributed machine learning. Background technique [0002] Many machine learning models can be formulated as finite and optimization problems: [0003] [0004] where w is the parameter of the model, n is the total number of training samples, ξ i represents the ith sample, f(w;ξ i ) represents the loss function corresponding to the ith sample, and d is the size of the model dimension. Stochastic Gradient Descent (SGD) and its variants are currently the most widely used methods for solving the above-mentioned finite and optimization problems. [0005] Parameter server architecture (Parameter Server) is a commonly used architecture in distributed machine learning. The paramet...

AI Technical Summary

Problems solved by technology

In this case the worker nodes may be untrusted

In addition, in applications such as edge computing and federated learning, the server organizer has weak control over the worker nodes, making it difficult to guarantee the credibility of the worker nodes

In the case that these working nodes may not be trusted, the existing asynchronous distributed machine learning has not been able to solve this problem well

Images