Method and apparatus for distributed training of machine learning models

Abstract

The present disclosure relates to a method and apparatus for distributed training of machine learning models. The method includes: in the preset number of iteration steps of distributed training, obtaining the order of the current iteration step; the preset number has a logarithmic relationship with the N; based on the order, obtaining a node from N nodes as The target node of the current node in the current iteration step; communicate with the target node to obtain the model information shared by the target node; update the machine learning model of the current node according to the model information of the current node and the model information of the target node to complete the preset amount After the iteration step, the machine learning models of each node obtain the model information of the N nodes synchronously. In this embodiment, the convergence efficiency of distributed training can be guaranteed; at the same time, the communication volume of a single node can be reduced to S, achieving the effect of shortening the communication time; in addition, each node only shares information with its own target node, which can ensure that each node Load balancing among them.

Description

technical field [0001] The present disclosure relates to the technical field of machine learning, and in particular to a method and device for distributed training of machine learning models. Background technique [0002] As the data and models for machine learning become larger and larger, the storage and computing capabilities of a single card / computer can no longer meet the training needs of big data and large models, and distributed machine learning emerges as the times require. Currently, distributed machine learning in related technologies: [0003] Taking the Ring-allreduce architecture as an example, N computing devices form a ring, and each computing device is a worker node. In an iterative process, each node divides the model information / gradient to be communicated into N parts, each worker completes its own training data (mini-batch) training, calculates the gradient, and passes the gradient to the next one in the ring worker, and it also receives the gradient f...

AI Technical Summary

Problems solved by technology

At this time, the communication volume of a single node depends on the number of nodes selected for communication, and the amount of information shared by each node in each iteration of other nodes is limited, requiring an average of log 2 Only N~N / 2 iterations can share the information of all nodes, resulting in insufficient information sharing between nodes, and the overall convergence process of distributed training slows down

Images