Method and system for distributed deep learning parameter quantification communication optimization

Abstract

The invention discloses a method and system for distributed deep learning parameter quantization communication optimization based on discrete cosine transform, and the method comprises the steps: carrying out the discrete cosine transform of a gradient value in the distributed deep learning, carrying out the compression processing, carrying out the inverse quantization operation when the weight isupdated, and forming a distributed deep learning system with the high communication efficiency. Before a working node sends gradient data to a parameter server, gradient values are processed by adopting gradient division, GPU parallel computing, discrete cosine transform and quantization and high-frequency filtering compression methods, and then the processed gradient values are pushed to the parameter server. And the working node obtains the weight from the parameter server through a pull operation, and then updates the weight on the current working node by adopting inverse discrete cosine transform and inverse quantization and error compensation updating methods. According to the method, the communication efficiency between the working node and the parameter server in an existing distributed deep learning framework can be effectively improved, and the model training speed is increased.

Description

technical field [0001] The invention belongs to the technical field of deep learning, and more specifically relates to a distributed deep learning parameter quantization communication optimization method and system based on discrete cosine transform. Background technique [0002] Deep Neural Network (DNN) is an artificial neural network (Artificial Neural Network, ANN) composed of an input layer, multiple hidden layers and an output layer. Each layer is composed of multiple neuron nodes. The neuron nodes of the front layer and the back layer are connected to each other, and each connection corresponds to a weight parameter. like figure 1 As shown, layer1 represents the input layer, layer4 represents the output layer, layer2 and layer3 represent the hidden layer, and the connection between neurons corresponds to a weight parameter Among them, l represents the l-th layer, j represents the j-th neuron in the layer before l, and k represents the k-th neuron in the l-layer. T...

AI Technical Summary

Problems solved by technology

[0019] In view of the above defects or improvement needs of the prior art, the present invention provides a distributed deep learning parameter quantization communication optimization method and system based on discrete cosine transform, thereby solving the problem that the existing distributed deep learning model training speed has certain limitations technical problem

Images