Gradient compression method for distributed DNN training in edge computing environment

Abstract

The invention discloses a gradient compression method for distributed DNN training in an edge computing environment. The method comprises the steps: building a selection standard based on a gradient number, and screening a gradient network layer which meets a model compression standard; evaluating gradient importance according to the gradient entropy, adaptively selecting a gradient sparsification threshold value, and performing gradient sparsification compression based on the flexible threshold value; according to gradient residual errors and a momentum correction mechanism, accumulating and optimizing the gradient residual errors, thereby reducing the performance loss of a training model caused by gradient sparsity; quantizing the sparse gradient according to a ternary quantization compression scheme to obtain a sparse ternary tensor; and according to a lossless coding technology, recording a distance of a non-zero gradient in the transmission tensor, performing optimization coding on the distance, and outputting a sparse ternary gradient. According to the sparse ternary gradient compression algorithm based on the gradient number and the gradient entropy, the gradient size of a gradient exchange stage in distributed DNN training can be adaptively compressed; therefore, the communication efficiency of distributed DNN training is effectively improved.

Description

technical field [0001] The invention relates to a gradient compression method for distributed DNN training in an edge computing environment, in particular to an adaptive sparse ternary gradient compression method for distributed DNN training in an edge computing environment, belonging to the technical field of edge computing. Background technique [0002] With the rapid development of artificial intelligence, Deep Neural Network (DNN) is widely used in various fields of intelligence, including computer vision, natural language processing and big data analysis. In every domain, the high accuracy of deep learning comes at the cost of high computational and storage requirements during the training phase. However, DNN models need to iteratively optimize millions of parameters over multiple time periods, which makes training deep neural network models both time-consuming and computationally expensive. Edge computing can meet the training requirements of DNN to a certain extent, ...

AI Technical Summary

Problems solved by technology

Since quantization and sparsification are at the expense of model accuracy in exchange for communication efficiency, gradient compression schemes with different degrees of quantization and sparsification will have significant differences. Poor accuracy optimization

Images