Distributed machine learning system and communication scheduling method suitable for same

Abstract

The invention discloses a distributed machine learning system and a communication scheduling method suitable for the same and the method comprises the steps: additionally arranging an automaton on a parameter server of the distributed machine learning system or a network chip of a switcher, and recognizing the parameter server and a working node in the distributed machine learning system through the automaton; enabling the parameter server to sequentially send the correspondingly distributed parameters to each working node, and only send the correspondingly distributed parameters to one working node at the same time point; after each working node pulls the parameter from the parameter server, immediately starting to calculate the gradient according to the parameter; and after each workingnode completes gradient calculation, judging whether the parameter server receives gradients pushed by other working nodes or not at the moment, and if not, pushing the gradient calculated by the current working node to the parameter server. According to the invention, the communication of the distributed machine learning system is reasonably scheduled, and the communication time cost of distributed machine learning is effectively reduced.

Description

technical field [0001] The invention belongs to the cross technical field of distributed computing and machine learning, and specifically relates to a distributed machine learning system and a communication scheduling method applicable thereto. Background technique [0002] With the advent of the era of big data, machine learning algorithms, especially deep learning algorithms suitable for large-scale data, are receiving more and more attention and applications, including speech recognition, image recognition, and natural language processing. However, with the increase of the input training data (a type of data used to solve the neural network model in machine learning) and the neural network model, there are memory limitations and weeks or even months of training time for single-node machine learning training. problem, distributed machine learning came into being. Distributed machine learning has received widespread attention in both industry and academia. For example, Goo...

AI Technical Summary

Problems solved by technology

In the above process, communication and computing do not overlap each other, resulting in waste of computing and communication resources

[0010] Moreover, as the amount of training data increases, the increase of nodes can linearly decrease the calculation time. Therefore, it is an inevitable trend to set more working nodes in the distributed machine learning system to train the model, but at the same time, the increase of working nodes Make the communication time grow nonlinearly and rapidly, causing the communication time to become the bottleneck of distributed machine learning

Images