Multiple GPUs-based BPNN training method and apparatus

Abstract

The invention provides a multiple graphics processing unit (GPU)s-based back-propagation neural network (BPNN) training method and apparatus. The method comprises the following steps: S1, controlling all GPUs to carry out BPNN forward calculation and synchronizing forward calculation outputs among all GPUs; S2, controlling all GPUs to carry out BPNN backward error calculation and synchronizing backward error calculation outputs among all GPUs; S3, controlling all GPUs to update the weight of the BPNN according to the forward calculation outputs obtained by synchronization and backward error calculation outputs obtained by synchronization. According to the invention, data synchronization costs of multiple GPUs during the BPNN training can be lowered; and the BPNN training efficiency of the multiple GPUs can be improved.

Description

【Technical field】 [0001] The invention relates to neural network training technology, in particular to a multi-GPU-based BPNN training method and device. 【Background technique】 [0002] BPNN (Back-Propagation Nueral Networks) backpropagation neural network is a multi-layer feedforward network trained by the error backpropagation algorithm proposed by a team of scientists headed by Rumelhart and McCelland in 1986. It is currently the most widely used neural network model. one. [0003] The topology of the BPNN model includes an input layer (input), a hidden layer (hide layer) and an output layer (output layer). The input layer is responsible for receiving input data from the outside world and passing it to the hidden layer; the hidden layer is the internal information processing layer, responsible for data processing, and the hidden layer can be designed as a single hidden layer or a multi-hidden layer structure; the last hidden layer is passed to the output After further p...

AI Technical Summary

Problems solved by technology

[0006] However, in the above-mentioned multi-GPU BPNN training method, there is a large overhead when synchronizing the synchronization weight data between the BPNNs of the GPUs. The weight value data of a large-scale BPNN can reach hundreds of megabytes. The communication time overhead of these BPNN weight values ​​can reach hundreds of milliseconds, resulting in the inefficiency of using multiple GPUs to train BPNNs, and the training process on a single GPU usually only takes tens of milliseconds. It can be seen that due to multiple GPUs The overhead of data synchronization between them is high, resulting in low efficiency of using multiple GPUs to train BPNN, and sometimes it is not as good as using a single GPU for BPNN training

Images