A multi-gpu-based bpnn training method and device

Abstract

The present invention provides a backpropagation neural network (BPNN) training method and device based on multiple graphics processing units (GPUs), wherein the method includes: S1, controlling each GPU to perform forward calculation of BPNN, and synchronizing between each GPU Forward calculation output; S2, controlling each GPU to perform reverse error calculation of BPNN, and synchronizing the reverse error calculation output between each GPU; S3, controlling each GPU to obtain the forward calculation output according to the synchronization and the synchronization obtained The output of the reverse error calculation updates the weights of the BPNN. The invention can reduce the data synchronization overhead during multi-GPU BPNN training, and improve the multi-GPU BPNN training efficiency.

Description

【Technical field】 [0001] The invention relates to neural network training technology, in particular to a multi-GPU-based BPNN training method and device. 【Background technique】 [0002] BPNN (Back-Propagation Nueral Networks) backpropagation neural network is a multi-layer feedforward network trained by the error backpropagation algorithm proposed by a team of scientists headed by Rumelhart and McCelland in 1986. It is currently the most widely used neural network model. one. [0003] The topology of the BPNN model includes an input layer (input), a hidden layer (hide layer) and an output layer (outputlayer). The input layer is responsible for receiving input data from the outside world and passing it to the hidden layer; the hidden layer is the internal information processing layer, responsible for data processing, and the hidden layer can be designed as a single hidden layer or a multi-hidden layer structure; the last hidden layer is passed to the output After further pr...

AI Technical Summary

Problems solved by technology

[0006] However, in the above-mentioned multi-GPU BPNN training method, there is a large overhead when synchronizing the synchronization weight data between the BPNNs of the GPUs. The weight value data of a large-scale BPNN can reach hundreds of megabytes. The communication time overhead of these BPNN weight values ​​can reach hundreds of milliseconds, resulting in the inefficiency of using multiple GPUs to train BPNNs, and the training process on a single GPU usually only takes tens of milliseconds. It can be seen that due to multiple GPUs The overhead of data synchronization between them is high, resulting in low efficiency of using multiple GPUs to train BPNN, and sometimes it is not as good as using a single GPU for BPNN training

Images