Parameter communication optimization method for distributed machine learning

Abstract

The invention discloses a parameter communication optimization method for distributed machine learning. According to the method, the fault-tolerant characteristic of the machine learning iteration-convergence algorithm is expanded; a dynamic finite fault tolerance characteristic is provided, a distributed machine learning parameter communication optimization strategy is realized based on the dynamic finite fault tolerance, the performance of each computing node is fully utilized by dynamically adjusting the synchronization strategy of each computing node and a parameter server in combination with a performance detection model, and the accuracy of the machine learning model is ensured; sufficient computing resources are guaranteed, and the training process of the model is not affected by dynamic changes of the distributed computing resources; a training algorithm and system hardware resources are decoupled, the process that developers manually allocate computing resources and adjust andoptimize data communication according to experience is liberated, and the expansibility and high execution efficiency of a program in various cluster environments are effectively improved. The methodcan be applied to the fields of optimization of distributed machine learning parameter communication, optimization of cluster computing performance and the like.

Description

technical field [0001] The invention belongs to the field of machine learning and high-performance parallel computing, in particular to a parameter communication optimization method oriented to distributed machine learning. Background technique [0002] With the advent of the big data era, distributed machine learning has become a research hotspot because it can adapt to the complexity of big data, obtain higher prediction accuracy, and support smarter tasks. [0003] The main purpose of distributing machine learning is: (1) to solve the insufficient memory of a single computing node, so as to ensure that the data volume of TB level and above can be processed; (2) to use parallel acceleration model training to reduce the training time of several months shorten. One of the most important issues is how to achieve parallel acceleration of the training process. Data parallelization based on parameter server is a common parallelization scheme in distributed machine learning, in...

AI Technical Summary

Problems solved by technology

The overall synchronous parallel method will lead to unbalanced iterative calculation load due to the difference in the performance of computing nodes

[0004] At present, there are some methods to solve the above problems, such as the distributed machine learning asynchronous iteration scheme, in which the computing nodes can use the local model parameters to execute the next iteration before receiving the global model parameters. It may fall into a local optimum, and it cannot guarantee that the machine learning model will eventually converge to the optimal solution,

Images