Method and apparatus for distributed training of machine learning model

Abstract

The invention relates to a method and apparatus for distributed training of a machine learning model. The method comprises the following steps of: in a preset number of iteration steps of distributedtraining, obtaining the sequence of the current iteration steps, the preset number and the N being in a logarithmic relationship; based on the sequence, obtaining one node from the N nodes as a targetnode of the current node in the current iteration steps; communicating with the target node to obtain model information shared by the target node; updating the machine learning model of the current node according to the model information of the current node and the model information of the target node, so as to synchronously obtain the model information of the N nodes by the machine learning model of each node after completing the preset number of iteration steps. According to the method and apparatus of the embodiments, the convergence efficiency of distributed training can be ensured; meanwhile, the communication traffic of a single node can be reduced to S, and the effect of shortening the communication time is achieved; in addition, each node only shares information with its respective target node, so that the load balance among the nodes can be ensured.

Description

technical field [0001] The present disclosure relates to the technical field of machine learning, and in particular to a method and device for distributed training of machine learning models. Background technique [0002] As the data and models for machine learning become larger and larger, the storage and computing capabilities of a single card / computer can no longer meet the training needs of big data and large models, and distributed machine learning emerges as the times require. Currently, distributed machine learning in related technologies: [0003] Taking the Ring-allreduce architecture as an example, N computing devices form a ring, and each computing device is a worker node. In an iterative process, each node divides the model information / gradient to be communicated into N parts, each worker completes its own training data (mini-batch) training, calculates the gradient, and passes the gradient to the next one in the ring worker, and it also receives the gradient f...

AI Technical Summary

Problems solved by technology

At this time, the communication volume of a single node depends on the number of nodes selected for communication, and the amount of information shared by each node in each iteration of other nodes is limited, requiring an average of log 2 Only N~N / 2 iterations can share the information of all nodes, resulting in insufficient information sharing between nodes, and the overall convergence process of distributed training slows down

Images