Parameter synchronization optimization method and system suitable for distributed machine learning

Abstract

The invention provides a parameter synchronization optimization method and system suitable for distributed machine learning. A machine learning algorithm achieved in a parameter server distribution mode is used for overcoming bottlenecks, such as a large amount of parallel machine learning training time delay caused by insufficient network and parameter server resources, of an existing algorithm in the parameter synchronization process. The system comprises a resource monitoring and distributing module at the parameter server end, a parameter maintaining module at the parameter server end, server resource request modules of all working nodes, parameter synchronization time interval control modules of the working nodes, non-synchronization time accumulation modules of the working nodes, parameter calculation modules of the working nodes and parameter synchronization modules of the working node. According to the parameter synchronization optimization method and system, different synchronization time intervals are selected for the different working nodes to avoid request emergency situations by monitoring resource occupancy conditions of a parameter server; meanwhile, it is guaranteed that the selected time intervals can meet the requirements for communication frequency reducing and training accurate rate guaranteeing at the same time, and the bottlenecks of an existing distributed machine learning system in the parameter synchronization process are effectively avoided.

Description

technical field [0001] The invention belongs to the interdisciplinary technical field of distributed computing and machine learning, and specifically relates to a parameter synchronization optimization method and system suitable for distributed machine learning. Background technique [0002] With the advent of the era of big data, machine learning algorithms, especially deep learning algorithms suitable for large-scale data, are receiving more and more attention and applications, including speech recognition, image recognition, and natural language processing. However, with the increase of the input training data (a type of data used to solve the neural network model in machine learning) and the neural network model, there are memory limitations and weeks or even months of training time for single-node machine learning training. problem, distributed machine learning came into being. Distributed machine learning has received widespread attention in both industry and academia...

AI Technical Summary

Problems solved by technology

By monitoring the resource occupancy of the parameter server, different synchronization time intervals are selected for different working nodes to avoid request bursts, and at the same time ensure that the selected time interval can meet the requirements of reducing communication frequency and ensuring training accuracy. The above method and system can effectively solve the problem. The Problem of Parameter Synchronization Bottleneck in Existing Distributed Machine Learning Systems

Images