Gradient Compression Method for Distributed DNN Training in Edge Computing Environment

Abstract

The invention discloses a gradient compression method for distributed DNN training in an edge computing environment, establishes a selection standard based on the number of gradients, and screens the gradient network layer that meets the model compression standard; evaluates the importance of the gradient according to the gradient entropy, and selects it adaptively The threshold of gradient sparsification is based on the flexible threshold to compress the gradient sparsification; according to the gradient residual and momentum correction mechanism, the gradient residual is accumulated and optimized to reduce the performance loss of the training model caused by gradient sparsity; according to the ternary quantization compression scheme, quantization The sparse gradient is obtained to obtain a sparse ternary tensor; according to the lossless coding technology, the distance of the non-zero gradient in the transfer tensor is recorded, and the optimized encoding is performed to output the sparse ternary gradient. The sparse ternary gradient compression algorithm based on the gradient quantity and gradient entropy of the present invention can adaptively compress the gradient size of the gradient exchange stage in the distributed DNN training, and effectively improve the communication efficiency of the distributed DNN training.

Description

technical field [0001] The invention relates to a gradient compression method for distributed DNN training in an edge computing environment, in particular to an adaptive sparse ternary gradient compression method for distributed DNN training in an edge computing environment, belonging to the technical field of edge computing. Background technique [0002] With the rapid development of artificial intelligence, deep neural network (DNN) is widely used in various intelligent fields, including computer vision, natural language processing, and big data analysis. In every domain, the high accuracy of deep learning comes at the expense of high computational and storage requirements during the training phase. DNN models, on the other hand, need to iteratively optimize millions of parameters over multiple time periods, which makes training deep neural network models both time and computationally expensive. Edge computing can meet the training requirements of DNN to a certain extent,...

AI Technical Summary

Problems solved by technology

Since quantization and sparsification are at the expense of model accuracy in exchange for communication efficiency, gradient compression schemes with different degrees of quantization and sparsification will have significant differences. Poor accuracy optimization

Images