Asynchronous training of machine learning models

Abstract

Embodiments of the present disclosure relate to asynchronous training of machine learning models. The server receives feedback data generated by training the machine learning model from the worker machine. These feedback data are obtained by the worker using its own training data and are associated with the previous values ​​of the parameter sets of the machine learning model at this particular worker. The server determines a difference between the previous value and a current value of the parameter set at the server. This current value may have been updated one or more times due to the actions of other workers. The server can then update the current value of the parameter set based on the difference between the feedback data and the value of the parameter set. Thus, this update not only takes into account the training results of each worker, but also properly compensates for the delay between different workers.

Description

Background technique [0001] Machine learning has wide-ranging applications in fields such as speech recognition, computer vision, and natural language processing. For example, Deep Neural Networks (DNN) can train machine learning models with multiple layers and parameters in parallel on the basis of big data and powerful computing resources. In the training phase, one or more parameters of the model need to be trained according to the given training data set and optimization goal. For example, for the training of neural networks, stochastic gradient descent methods can be used. [0002] It is known that the training dataset can be distributed among multiple workers. These worker machines use their respective training data to optimize model parameters and return their results to a central server. However, the core problem of distributed or asynchronous model training is the mismatch between individual workers. For example, when a worker returns its parameter updates, the mo...

AI Technical Summary

Problems solved by technology

However, the core problem of distributed or asynchronous model training is the mismatch between individual workers

Images