Gradient synchronization method and device in distributed training

Abstract

The invention provides a gradient synchronization method and device in distributed training. The gradient synchronization method in distributed training comprises the following steps: grouping training data on each training node in a distributed training cluster to obtain a plurality of sub-training data on each training node, the training nodes in the distributed training cluster being in annularconnection; calculating the sub-training average gradient of each piece of sub-training data in the training nodes of the distributed training cluster; obtaining a sub-training accumulation gradientcorresponding to the sub-training average gradient according to the sub-training average gradient; and synchronizing the sub-training accumulation gradient to each training node of the distributed training cluster. The average gradient of different batches of training data is calculated on each training node, so that the oscillation range of the gradient is relatively small, the gradient descending direction can be determined more accurately, the training speed of the model is increased, and the training efficiency of the model is improved.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a method and device for gradient synchronization in distributed training, a computing device, a computer-readable storage medium and a chip. Background technique [0002] At present, with the rapid development of computer technology, deep learning technology has also made rapid progress. With the deepening of deep learning technology, more and more complex algorithms have been developed. These algorithms require a large amount of data and consume a lot of time to be effective. Complete training, so distributed training was developed. [0003] In the model optimization of deep learning, it is necessary to use the gradient descent method to calculate the gradient to find the minimum loss function, so as to train the model and speed up the convergence of the model. In the current distributed training, it is necessary to transfer gradient information and synchronize ...

AI Technical Summary

Problems solved by technology

In the current distributed training, it is necessary to transfer gradient information and synchronize gradient information every time training is completed, so as to share the gradient on the distributed training nodes and find the minimum loss function. The problem of high-frequency gradient information transmission and the large amount of information transmitted leads to long model training time and large span, which seriously delays the speed of model training

Images