Distributed deep learning method based on pipeline annular parameter communication

Abstract

In order to overcome the defects of low cluster training speed and high training time overhead, the invention provides a distributed deep learning method based on pipeline ring parameter communication. The method comprises the following steps: obtaining a training model, and initializing computing nodes in a cluster by adopting the training model; performing distributed training on computational nodes in the cluster by adopting a pipeline stochastic gradient descent method, executing training model updating and gradient calculation, and executing gradient communication in parallel during the period; after the node completes the ith round of gradient calculation locally, compressing gradient data, then starting a communication thread to execute an annular AllReduce operation, and starting the (i + 1)-th round of iterative training at the same time until the iterative training is completed. According to the method, an annular AllReduce algorithm is adopted, the problem of communication congestion of server nodes of an image parameter server framework is avoided through annular communication, and time consumption is reduced through parallel overlapping calculation and communication of a local assembly line.

Description

technical field [0001] The present invention relates to the technical field of deep learning, more specifically, to a distributed deep learning method based on pipeline ring parameter communication. Background technique [0002] Distributed deep learning for cluster parallel computing on multiple machines has gradually become the focus of technological innovation and development. Distributed deep learning requires frequent communication and the exchange of a large amount of data, and the bandwidth of the network interface is limited, resulting in most of the training time of the neural network being used for data transmission. If the GPU is used for acceleration, the communication volume remains unchanged due to the reduction of calculation time. The time proportion of communication consumption will further increase, which becomes a bottleneck restricting the development of parallelization. [0003] For the acceleration of model training, there are currently two main soluti...

AI Technical Summary

Problems solved by technology

The general distributed AllReduce algorithm uses a machine to collect the gradient data of each node, and then sends the updated gradient back to each node. However, in this method, the communication time of the gradient collection node increases linearly with the increase in the number of cluster nodes. Defects, there are problems of long calculation time and low cluster training speed

Images