Stochastic gradient descent optimization method based on distributed coding

Abstract

The invention discloses a stochastic gradient descent optimization method based on distributed coding. For the problems of gradient delay and efficiency reduction caused by computing node heterogeneity and communication load bottleneck on a large-scale distributed cluster, the invention provides an asynchronous stochastic gradient descent optimization algorithm adapting to node load balance basedon distributed coding. Coding optimization is mainly carried out on data communication generated during parameter updating in the data exchange stage, a strategy based on load balancing is used for carrying out real-time estimation on the node computing capacity, task allocation between nodes is optimized, and the gradient delay problem is solved. According to the algorithm, the problem that the loss function of the deep neural network is difficult to converge due to the gradient delay problem can be relieved, and the training performance of the large-scale neural network can be better improved, so that the performance of the distributed neural network training algorithm is improved.

Description

technical field [0001] The invention relates to a distributed computing architecture, in particular to a distributed coding-based distributed stochastic gradient descent optimization method. Background technique [0002] The neural network training method based on the gradient descent algorithm has received extensive attention in recent years. However, due to the upper limit of the performance that a single machine can achieve, distributed clusters can improve the speed of deep neural network training. At present, the widely used distributed computing method of deep neural network is the asynchronous gradient descent method, which can better ensure the accuracy of training compared with the parameter averaging method, but the total amount of information communication required by the asynchronous stochastic gradient descent method is compared with Parameter averaging has been greatly increased. At the same time, the asynchronous stochastic gradient descent method has a signi...

AI Technical Summary

Problems solved by technology

[0003] Purpose of the invention: The purpose of the invention is to provide an asynchronous stochastic gradient descent optimization method based on distributed coding and node load balancing strategy, mainly by targeting at the data exchange stage The data communication generated when the parameters are updated is encoded and optimized, and a load-balancing strategy is used to estimate the computing power of the nodes in real time to optimize the task allocation between nodes, improve the gradient delay problem, and solve the problems faced by large-scale distributed clusters. The problem of gradient delay and efficiency drop caused by heterogeneous computing nodes and communication load bottlenecks

Images