Multi-step delayed update method for distributed deep learning based on sparsification of communication operations

Abstract

The invention discloses a distributed deep learning multi-step delay update method based on communication operation sparsification. The specific steps include: warm-up training, using a synchronous stochastic gradient descent method to perform a certain depth learning model before performing multi-step delay iterative training. The training of the number of iterations; the switching stage, the purpose is to switch the synchronous stochastic gradient descent update method to the multi-step delay training mode; the local parameter update operation adopts the local update method based on the global gradient, the purpose is to alleviate the weight delay and ensure the model. Convergence accuracy; multi-step delay training, which specifically includes three steps of global parameter update, local parameter update and communication operation sparsification. The present invention slows down network congestion by adopting sparse communication operations, eliminates synchronization overhead, greatly reduces communication overhead in the distributed training process, and optimizes the communication overhead in the training process.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a training update method for distributed deep learning. Background technique [0002] Deep learning has recently achieved great success in various fields such as computer vision, natural language processing, autonomous driving, and intelligent medical care. The rise of deep learning is mainly due to two conditions. One is the emergence of general-purpose and customized hardware accelerators (GPU, NPU, TPU, etc.), which have brought great progress in computing power. Open source of general training datasets like CIFAR. However, with the rapid growth of deep neural networks and datasets, the computing power of the machines used for training becomes a bottleneck, and it takes days or weeks to complete the training of a large neural network model. In this case, distributed Training has become a common practice, which greatly improves training efficiency and speeds u...

AI Technical Summary

Problems solved by technology

Considering the impact of the weight delay problem on the model training accuracy, the key to optimizing the ASGD method is to ensure the convergence accuracy of the model. Researchers have proposed different optimization measures based on the asynchronous update mechanism. Although the final convergence accuracy of the model has been improved, an additional The limitation or operation of ∆ reduces the speed of distributed training to some extent, making it impossible to train faster than the original ASGD method

Images