Parallel model processing method and device based on multiple graphics processing units

Abstract

The invention relates to a parallel model processing method based on multiple graphics processing units (GPUs). The method includes the steps: creating multiple Workers used for respectively controlling multiple Worker Groups in a central processing unit (CPU), wherein each Worker Group comprises the GPUs; binding each Worker with one corresponding GPU; loading one Batch of training data from a nonvolatile memory into a GPU video memory corresponding to one Worker Group; transmitting data, needed by the GPUs for data processing, among the GPUs corresponding to one Worker Group in a Peer to Peer manner; controlling the GPUs to perform data processing in parallel through the Workers. By the method, efficiency of parallel data processing of the GPUs can be improved. Besides, the invention further provides a parallel data processing device.

Description

technical field [0001] The invention relates to the technical field of computing, in particular to a data parallel processing method and device based on multiple graphics processing units (GPUs). Background technique [0002] Deep neural network technology is currently a relatively popular technology in the field of machine learning. There have been successful cases in academia and industry, including speech recognition, image recognition, natural language processing, and advertising recommendation. The article "Large-scale Deep Unsupervised Learning using Graphics Processors" published by Rajat Raina, Anand Madhavan, Andrew Y.Ng et al. at the 26th International Machine Learning Conference (Montreal, Canada, 2009) introduces the use of a single graphics processor ( GPU) training method and system implementation of deep neural network; Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Image Classification (ImageNet Classification with Deep Convolutional Neural Networks)" introduc...

AI Technical Summary

Problems solved by technology

[0008] The point-to-point data exchange model has serious shortcomings in performance: when more than 2 parallel units are required, more data exchange cycles are required, and there is waiting in each data exchange cycle, and the idle bus bandwidth is not fully utilized.

The existing technology uses a fixed learning rate to update parameters, and the training experiment process involves a lot of manual adjustment of the learning rate and judging the convergence work, which is complicated and cumbersome and low in efficiency

Images