Distributed deep learning multi-step delay updating method based on communication operation sparsification

Abstract

The invention discloses a distributed deep learning multi-step delay updating method based on communication operation sparsification, and the method comprises the specific steps: warm-up training: carrying out the training of a certain number of iterations of a deep learning model through employing a synchronous random gradient descent method before multi-step delay iteration training; in the switching stage, switching a synchronous stochastic gradient descent updating method into a multi-step delay training mode, wherein the local parameter updating operation adopts a local updating method based on global gradient, so as to relieve weight delay and ensure the convergence precision of the model; and the multi-step delay training specifically comprising three steps of global parameter updating, local parameter updating and communication operation sparsification. By adopting communication operation sparsification, network congestion is relieved, synchronization expenditure is eliminated,communication expenditure in the distributed training process is reduced to a great extent, and communication expenditure in the training process is optimized.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a training update method for distributed deep learning. Background technique [0002] Deep learning has recently achieved great success in various fields such as computer vision, natural language processing, autonomous driving, and intelligent medical care. The rise of deep learning is mainly due to two conditions. One is the emergence of general-purpose and customized hardware accelerators (GPU, NPU, TPU, etc.), which have brought great progress in computing power. Open source of general training datasets like CIFAR. However, with the rapid growth of deep neural networks and datasets, the computing power of the machines used for training becomes a bottleneck, and it takes days or weeks to complete the training of a large neural network model. In this case, distributed Training has become a common practice, which greatly improves training efficiency and speeds u...

AI Technical Summary

Problems solved by technology

Considering the impact of the weight delay problem on the model training accuracy, the key to optimizing the ASGD method is to ensure the convergence accuracy of the model. Researchers have proposed different optimization measures based on the asynchronous update mechanism. Although the final convergence accuracy of the model has been improved, an additional The limitation or operation of ∆ reduces the speed of distributed training to some extent, making it impossible to train faster than the original ASGD method

Images