Large category deep learning GPU parallel acceleration method

Abstract

A large category deep learning GPU parallel acceleration method provided by the present invention comprises the steps of adopting a model to train the model parameters of the softmax layers in a deepneural network structure parallelly, using each GPU to train the respective model fragmentations, enabling the softmax layers of the GPUs to interact the data characteristics of the model parameters,thereby finishing the deep learning. The method of the present invention adopts a mixed architecture, namely all levels before the softmax layers still adopt a data parallel mode, and the softmax layers adopt a model parallel mode, thereby breaking through the bottleneck of the large category deep learning parallel operation, overcoming the problems that the communication cost is high and the expended communication time is too much by interacting the parameters in the last full-link layer in the deep neural network structure, being able to improve the model learning efficiency substantially while keeping an original deep learning effect, and reducing the GPU occupancy rate.

Description

technical field [0001] The invention relates to the field of computers and applications thereof, in particular to a method for parallel acceleration of large-scale deep learning GPUs. Background technique [0002] At present, deep learning has achieved breakthroughs in several major fields: in the field of speech recognition, in the field of image recognition, and in the field of natural language processing. It can be said that so far, deep learning is the closest intelligent learning method to the human brain. However, the deep learning model has many parameters, a large amount of calculation, and a larger scale of training data, which consumes a lot of computing resources. If the training can be accelerated, the work efficiency will be significantly improved, and for large-scale training data and models, difficult tasks can be made possible. [0003] With the continuous advancement of GPU's massively parallel architecture support, general-purpose computing-oriented GPU (...

AI Technical Summary

Problems solved by technology

However, for data with a large number of categories, the communication cost for parameter interaction is too high on the last full-link layer in the deep neural network structure. Therefore, a new technical means is needed, which can greatly improve the model learning efficiency and reduce the GPU occupancy rate while maintaining the original deep learning effect.

Images