Parameter exchange method, device, server and storage medium for deep learning

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses a parameter exchange method, a device, a server and a storage medium for deep learning. The invention comprises the following steps: obtaining matching training data; training according to the matched training data and a known first weight value to obtain a first training parameter; the first training parameter is compressed and exchanged with other processors to obtain a second training parameter. When deep learning networks are trained by processors, by acquiring matching training data and a known first weight value for training,the first training parameter is obtained, the first training parameter is compressed and exchanged with other processors to obtain the second training parameter, and the speed of parameter exchange between processors is improved by the mode of transmission after compression, thereby saving the time consumed in parameter exchange.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of artificial intelligence, and in particular to a parameter exchange method, device, server and storage medium for deep learning. Background technique [0002] Deep learning network is a research hotspot in the field of machine learning in recent years, and it has been widely used in various industries. The deep learning network has a deep structure and tens of millions of parameters need to be learned, which makes it very time-consuming. The Graphics Processing Unit (GPU) has powerful computing power and is suitable for accelerating deep learning network training. At present, the acceleration methods used for deep learning networks mainly include data parallelism and model parallelism, and the current mainstream computing frameworks generally support data parallelism. In the data parallel mode, each GPU device trains for multiple iterations, and parameters need to be exchanged t...

AI Technical Summary

Problems solved by technology

[0003] The inventors have found that at least the following problems exist in the prior art: in the prior art, when training according to the data set, the parameter exchange is generally performed in a synchronous or asynchronous manner to improve the performance of the parameter exchange, but it is necessary to exchange all the parameters of the model every time. Parameters, if the model is large, especially in the case of cross-node exchange through the network, the time spent on the exchange will become the bottleneck of the entire model training

Images