Method and device for neural network machine learning model training

Abstract

The invention discloses a method and a device for neural network machine learning model training, and the method comprises the steps: applying the method and device to a distributed computing frame which comprises a plurality of computing nodes, segmenting training data into training data slices in advance, and enabling the number of the segmented slices to be the same as the number of the computing nodes participating in the calculation; a computing node obtaining training data slices and trains local model parameters; the computing node transmits the trained local model parameters to a parameter server; and the computing node updating local model parameters according to the global model parameters returned by the parameter server and continues to train the local model parameters. According to the method, the calculation acceleration ratio at multiple nodes can almost reach a linear ideal value, and the period of model training is greatly shortened.

Description

technical field [0001] This application relates to but is not limited to computer technology, especially a training method and device for a neural network machine learning model. Background technique [0002] After the Neural Machine Translation (NMT, Neural Machine Translation) model was proposed, it has been continuously developed in recent years due to the obvious improvement in translation effect. At present, in some languages ​​and scenarios, the translation quality can even reach the level of human translation. [0003] However, due to the complex structure of the NMT model, and the training process of the deep neural network model itself generally involves a large amount of calculations, the NMT system often requires a long training period. For example, using 30 million training data in a processor such as a single Training on a graphics processing unit (GPU) card requires more than 20 days of training to obtain a preliminary usable model. [0004] The existing neur...

AI Technical Summary

Problems solved by technology

In the worst case, multi-card expansion is not even possible due to the relatively small computing communication at the beginning

On the other hand, if the mini-batch size (size) is doubled in order to improve the calculation and communication ratio, when it is higher than the optimal empirical value, the model convergence accuracy will be greatly reduced

Furthermore, as the parallel scale increases, the computing performance of a single GPU cannot be fully utilized, which also causes waste

Images