A Federated Learning Adaptive Gradient Quantization Method

Abstract

The invention discloses a federated learning adaptive gradient quantization method, which initializes training samples and local models of each working node, uses the training samples to train the local model, obtains local gradients, and quantifies the local gradients according to the quantization levels obtained by each working node ; Upload the local gradient to the parameter server for gradient aggregation, and transmit the aggregation result back to each working node; each working node uses the quantized aggregation gradient to update the local model parameters; judge whether the number of iteration rounds meets the preset interval threshold, If it is satisfied, the link status of each working node is broadcast, and the quantization level of itself is adjusted in time, otherwise it enters the iterative training process until the preset condition is reached, and the training is ended; the present invention adaptively adjusts the quantization bit of the gradient according to the real-time bandwidth of the node link, Effectively alleviate the straggler problem. On the basis of completing the task of reducing communication overhead with traditional quantization methods, it improves the utilization of bandwidth resources and completes more efficient federated learning training.

Description

A Federated Learning Adaptive Gradient Quantization Method technical field [0001] The present invention relates to the technical field of gradient quantization, in particular to a federated learning adaptive gradient quantization method. Background technique Due to the continuous expansion of data volume and model scale, traditional machine learning cannot meet application requirements, so the distribution of Machine learning becomes mainstream. In order to complete multi-machine cooperation, communication between nodes is essential. But with model, neural The scale of the network is getting larger and larger, and the amount of parameters to be transmitted each time is also very large, which may cause the communication time to be too long, or even Because the increased communication time offsets the computational time saved by parallelism. Therefore, how to reduce the communication cost has become an important issue. A widely studied topic in the field of distribu...

AI Technical Summary

Problems solved by technology

In the case of a large required global model, network bandwidth limitations and the number of working nodes will exacerbate the communication bottleneck of federated learning, thereby slowing down the overall training process, and heterogeneous and dynamic networks will cause client devices to drop / exit The Straggler Problem

At this time, if the gradient quantization algorithm with uniform precision is adopted, the communication time between the fast and slow nodes will differ greatly, and the process of the fast nodes waiting for the slow nodes to complete the parameter synchronization will cause a lot of waste of computing resources and communication resources, which intensifies the straggler question

At the same time, for nodes with good link status, if the same low-precision quantization gradient is used as nodes with poor link status, the accuracy of the final trained model will also decrease.

Images