Parameter updating method, device and apparatus of AI distributed training system

Abstract

The invention discloses a parameter updating method of an AI distributed training system. The method comprises: starting a training task of an AI algorithm model on a target worker node of the distributed heterogeneous system, controlling the node to load model parameters, randomly selecting sample data of kth iterative training for the node, carrying out gradient updating on the model parameters,randomly creating a target node set, carrying out non-zero value updating on an adjacent matrix by utilizing the set, and updating the model parameters on each node in the set by utilizing the updated adjacent matrix; and when the kth iterative training is completed, if the AI algorithm model converges, repeatedly carrying out iterative training on the node until the node completes M times of iterative training, and judging that the distributed heterogeneous system completes the AI acceleration task. By utilizing the method, the requirement of each worker node in the distributed computing cluster on the communication bandwidth during parameter synchronization can be reduced while the hybrid heterogeneous distributed computing environment is supported.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a parameter updating method, device, equipment and medium of an AI distributed training system. Background technique [0002] In practical applications, distributed clusters are often used to accelerate the training tasks of AI (Artificial Intelligence, artificial intelligence) algorithm models. When using multiple worker nodes in distributed clusters to perform data parallel training on AI algorithm models , firstly, the same AI algorithm model will be deployed on each worker node, and the marked training data will be iteratively processed in batches. In each iteration, a batch of training data needs to be processed according to The number of worker nodes is divided into N micro-batches, and then the training data of these N micro-batches are distributed to different worker nodes for model training. Finally, when all worker nodes complete the training data of each micr...

AI Technical Summary

Problems solved by technology

However, in practical applications, various new computing devices will be continuously added to the worker nodes of the AI ​​distributed training system. In this case, if the above three algorithms are used to adjust the parameters in the AI ​​distributed training system When updating, the overall performance of the AI ​​distributed training system will be limited by the worker node with the slowest computing performance or the communication link with the slowest transmission in the distributed computing environment

At present, there is no effective solution to this technical problem

Images