Distributed training method for large-scale deep neural network

Abstract

The invention belongs to the crossing field of high-performance calculation and artificial intelligence, and particularly relates to a distributed training method for a large-scale deep neural network, which specifically comprises the following steps of: realizing overlapping of a communication process and a calculation process through a layer-by-layer scheduling parameter synchronization process and reverse error propagation so as to hide communication overhead and accelerate model training; in addition, in the parameter synchronization process of each layer, dynamically deciding the data to be transmitted according to the sparsity of different data blocks and the data compression overhead to realize finer-grained control of Ring-All-Reduce communication, so that the communication overhead of the parameter synchronization process is minimized, the performance is greatly improved, and under the condition that the model accuracy and the convergence rate are not influenced, distributed training of any deep neural network is close to linear acceleration, and high-efficiency expansion of clusters is facilitated.

Description

technical field [0001] The invention belongs to the intersection field of high-performance computing and artificial intelligence, and in particular relates to a distributed training method for large-scale deep neural networks. Background technique [0002] Deep neural network is one of the most effective technologies of artificial intelligence, which has excellent accuracy and generalization performance in many applications such as image classification, speech recognition, and text processing. In real-world applications, large-scale deep neural networks with tens of millions or even billions of parameters often yield higher accuracy and robustness. With the deepening of the number of deep neural network layers and the expansion of the parameter scale of a single layer, the computing and storage capabilities of a single CPU or GPU and other hardware accelerators gradually cannot meet the training needs. A straightforward way to break this limitation is to use multiple hardwa...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the above-mentioned technical problems in the prior art and reduce the communication overhead of distributed deep learning training to make it closer to linear acceleration, the present invention provides a distributed training method for large-scale deep neural networks to solve the current There is a communication bottleneck in the deep learning parallel technology, and the distributed training of the deep learning model is accelerated. The specific technical plan is as follows:

Images