Training method and device for acceleration distributed deep neural network

Abstract

The embodiment of the invention provides a training method and device for an acceleration distributed deep neural network. The method comprises the following steps that: on the basis of parallel training, designing the training of the deep neural network into a distributed training pattern, and dividing a deep neural network model to be trained into a plurality of sub-networks; dividing a trainingsample set into a plurality of sub-sample sets; and on the basis of distributed cluster architecture and a preset scheduling method, utilizing the plurality of sub-sample sets to train the deep neural network, wherein each piece of training is carried out by a plurality of sub-networks so as to finish the distributed training of the deep neural network. Since the influence of network delay for the distributed training sub-network can be reduced through data localization on the basis of the distributed cluster architecture and the preset scheduling method, a training strategy is regulated in real time and the progresses of parallel-training sub-networks can be synchronized, the training completion time of the distributed deep neural network can be shortened, and the training of the deep neural network is accelerated.

Description

technical field [0001] The invention relates to the technical field of deep neural network training, in particular to a training method and device for accelerating a distributed deep neural network. Background technique [0002] Deep neural networks have been successfully applied in many fields, including image recognition, texture classification, speech recognition and other fields. In recent years, the performance of deep neural networks has been significantly improved due to the use of deeper network architectures and larger training sample sets for training. However, in the training process, a serious problem also arises. With the explosive growth of network parameters and training samples, the training time of deep neural network is very long. [0003] In order to solve this problem, researchers have proposed a method for parallel training of deep neural networks, mainly using multiple graphics processor cores to achieve parallel training, thereby reducing training tim...

AI Technical Summary

Problems solved by technology

However, during the training process, a serious problem also occurred. With the explosive growth of network parameters and training samples, the training time of deep neural network was very long.

Images