Single-step delay stochastic gradient descent training method for machine learning

Abstract

The invention discloses a single-step delay stochastic gradient descent training method for machine learning, and the method comprises the following steps: defining a global updating function and a local updating function, enabling the updating function to run in a parameter server in a distributed training mode, and enabling the updating function to run in a computing node in a single-node training mode; adjusting an execution sequence of calculation and communication operations in the calculation node, and in an ODSGD training mode, performing an update operation of a local weight by the calculation node while executing a gradient sending operation; introducing a new parameter variable to break the original data dependence relationship; and executing a distributed training task, settingthe number of iterations of a preheating stage, and additionally specifying an algorithm for local updating and corresponding hyper-parameters. According to the method, the utilization rate of computing resources in the distributed deep learning training process can be increased, meanwhile, on the premise of ensuring the training precision, the distributed training speed is increased, and the training time of the neural network model is shortened.

Description

technical field [0001] The invention relates to the field of artificial intelligence, in particular to an optimization method for distributed training. Background technique [0002] Deep learning has recently achieved great success in various fields such as computer vision, natural language processing, autonomous driving, and intelligent medical care. The rise of deep learning is mainly due to two conditions. One is the emergence of general-purpose and customized hardware accelerators (GPU, NPU, TPU, etc.), which have brought great progress in computing power. Open source of general training datasets like CIFAR. However, with the rapid growth of deep neural networks and datasets, the computing power of the machines used for training becomes a bottleneck, and it takes days or weeks to complete the training of a large neural network model, for example, based on the ImageNet dataset, using It takes 21 hours to train ResNet-50 for 90 rounds with 8 P100 GPUs, which makes real-t...

AI Technical Summary

Problems solved by technology

Although ASGD achieves better training performance by alleviating the synchronization overhead in SSGD, it tends to converge less accurately than SSGD due to the use of delayed gradient information for training.

Images