Gradient fusion method and device and computer readable storage medium

Abstract

The invention discloses a gradient fusion method, which includes: the input tensor on each node in the depth learning distributed training architecture is determined, then the input tensor is transferred to the fusion buffer one by one to determine whether the number of input tensors in the fusion buffer is greater than the number of preset tensors, if the number of input tensors in the fusion buffer is greater than the number of preset tensors, the input tensors in the fusion buffer are processed by an allreduce algorithm to obtain the output tensors. As can be seen, the present invention determines whether to continue transferring tensor according to the magnitude of the number of tensors in the fusion buffer and the number of preset tensors. Compared with a method of gradient fusion based on the magnitude of all tensors in the buffer, the method of the invention solves the problem that the threshold is difficult to select due to the large difference of tensor magnitude, avoids the process of calculating and accumulating the magnitude of each tensor in the fusion buffer, thus simplifies the process of gradient fusion and improves the efficiency. The invention also provides a gradient fusion device and a computer-readable storage medium, the function of which corresponds to the function of the method.

Description

technical field [0001] The present invention relates to the field of deep learning, in particular to a gradient fusion method, device and computer-readable storage medium. Background technique [0002] Deep learning has made great progress in the past few years, especially in the fields of voice, image, machine translation, and natural language processing. Deep learning training requires massive data, which requires ultra-large-scale parameters. network model fitting. If the training data is insufficient, if the network model parameters are too few, it will cause underfitting and the model accuracy will be low. At present, the common network model parameters have reached hundreds of millions, and the parameter size has reached several gigabytes. The data parallel training method requires each GPU node to have a complete copy of the model parameters, and to send and receive complete gradient data when merging gradients. The huge amount of communication data brings huge netw...

AI Technical Summary

Problems solved by technology

[0009] The purpose of the present invention is to provide a gradient fusion method, device and computer-readable storage medium to solve the problem that the selection of the buffer threshold in the traditional gradient fusion method is difficult and the selection process is complicated

Images