Training system of back propagation neural network DNN (Deep Neural Network)

Abstract

The invention provides a training system of back propagation neural network DNN (Deep Neural Network). The training system of the back propagation neural network DNN comprises a first graphics processor assembly, a second graphics processor assembly and a controller assembly, wherein the first graphics processor assembly is used for performing DNN forward calculation and weight update calculation; the second graphics processor assembly is used for perform DNN forward calculation and DNN back calculation; the controller assembly is used for controlling the first graphics processor assembly and the second graphics processor assembly to perform Nth-layer DNN forward calculation respectively according to respective input data, and after the completion of the forward calculation, controlling the first graphics processor assembly to perform the weight update calculation and controlling the second graphics processor assembly to perform the DNN back calculation; and N is a positive integer. The training system provided by the invention has the advantages of high training speed and low data transmission cost, so that the training speed of the back propagation neural network DNN is promoted.

Description

technical field [0001] The invention relates to the technical field of backpropagation neural network, in particular to a training system of backpropagation neural network DNN. Background technique [0002] Backpropagation neural network (DNN) is widely used in many important Internet applications, such as speech recognition, image recognition, natural language processing, etc. It can greatly improve the accuracy of speech recognition, so it is widely used in speech recognition products of many companies. [0003] DNN training calculation has two characteristics: high computational complexity and incapable of large-scale parallelism. The calculation of DNN training is mainly the multiplication of floating-point matrix, and the calculation complexity is O(N3), which is a typical calculation-intensive type. Due to the limitations of the algorithm, the DNN training calculation cannot be parallelized on a large scale, so the traditional multi-server method cannot be used to im...

AI Technical Summary

Problems solved by technology

However, the above method has the following disadvantages: the data synchronization overhead is large

At present, the PCI-E2.0 bus used between multiple GPUs has a one-way transmission bandwidth of 4 to 5GB / S. If 4 GPUs need to synchronize 200MB of weight data respectively, the method of group synchronization (GPU 1-2 synchronization, 3-4 synchronization, 1-3 synchronization, 2-4 synchronization), two rounds of transmission are required, and the communication overhead will reach hundreds of milliseconds

In contrast, the current large-scale DNN training takes only tens of milliseconds for a calculation on a single GPU, which means that using multiple cards is slower than single-card training

Images