Neural network asynchronous training-oriented learning rate adjustment method

Abstract

The invention discloses a learning rate adjustment method for asynchronous training of a neural network. The method comprises the following steps: initializing parameters; sending parameters of the neural network to all idle computing nodes until c calculation results are received; respectively adjusting the learning rates of the c received calculation gradients; s4, using the learning rate obtained in the step S4 and the received c gradients to perform one-step gradient descent update on the network; and judging whether the network precision meets the requirement. If the requirements are met, completing training, sending the response 2 to all the computing nodes, and carrying out exiting; and otherwise, returning to the step B, and sending the response 1 to all nodes which are calculated in the current round, and carrying out the training of the next round of circulation. The method has the beneficial effects that the learning rate of the delay gradient does not linearly rise along with the increase of the number, meanwhile, the delay condition of other currently received gradients and the sample batch size are considered during calculation, and the overall learning rate is adjusted to be more balanced and more scientific.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence-neural network optimization technology research, in particular to a learning rate adjustment method for asynchronous training of neural networks. [0002] technical background [0003] As the current data set is expanding day by day, the parameters of the trained model (such as deep neural network) are also increasing, stochastic gradient descent optimization (SGD) has become the core of the current supervised learning algorithm. [0004] This training method is composed of several rounds of optimization. In each cycle, randomly select several samples from the training set, let them pass through the neural network, and calculate the loss (LOSS) according to the difference between the neural network's calculation results and the actual results; then reverse the network according to this loss Propagate, calculate the gradient of each parameter in the neural network for the loss, and ...

AI Technical Summary

Problems solved by technology

[0010] 1. There is no exact theoretical basis for numerical setting, but only heuristic setting

And the hyperparameters are also difficult to define and can only be selected through experiments.

[0011] 2. This type of method ignores other influencing factors, such as minibatch size, the impact of the delay of other gradients in the current minibatch

[0013] Because of these problems, these simple learning rate adjustment methods can only have good results when the number of workers is small, or the size of the minibatch calculated by each worker per round is extremely small.

Once these two conditions cannot be fully met, the accuracy of the network produced by training will be greatly reduced

The phenomenon is explained in detail in the experiment, and it is pointed out that the larger the value of (Number of workers N*batchsize of each worker), the worse the effect of asynchronous update

Images