A parameter synchronization optimization method and system suitable for distributed machine learning

A technology of machine learning and optimization methods, which is applied in the direction of program synchronization, resource allocation, multi-programming devices, etc., and can solve problems such as reducing communication frequency and parameter synchronization bottlenecks

CN104714852BActive Publication Date: 2018-05-22HUAZHONG UNIV OF SCI & TECH
5 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2018-05-22

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a parameter synchronization optimization method and system suitable for distributed machine learning. A machine learning algorithm achieved in a parameter server distribution mode is used for overcoming bottlenecks, such as a large amount of parallel machine learning training time delay caused by insufficient network and parameter server resources, of an existing algorithm in the parameter synchronization process. The system comprises a resource monitoring and distributing module at the parameter server end, a parameter maintaining module at the parameter server end, server resource request modules of all working nodes, parameter synchronization time interval control modules of the working nodes, non-synchronization time accumulation modules of the working nodes, parameter calculation modules of the working nodes and parameter synchronization modules of the working node. According to the parameter synchronization optimization method and system, different synchronization time intervals are selected for the different working nodes to avoid request emergency situations by monitoring resource occupancy conditions of a parameter server; meanwhile, it is guaranteed that the selected time intervals can meet the requirements for communication frequency reducing and training accurate rate guaranteeing at the same time, and the bottlenecks of an existing distributed machine learning system in the parameter synchronization process are effectively avoided.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the interdisciplinary technical field of distributed computing and machine learning, and specifically relates to a parameter synchronization optimization method and system suitable for distributed machine learning. Background technique

[0002] With the advent of the era of big data, machine learning algorithms, especially deep learning algorithms suitable for large-scale data, are receiving more and more attention and applications, including speech recognition, image recognition, and natural language processing. However, with the increase of the input training data (a type of data used to solve the neural network model in machine learning) and the neural network model, there are memory limitations and weeks or even months of training time for single-node machine learning training. problem, distributed machine learning came into being. Distributed machine learning has received widespread attention in both industry and academia...

Examples

Embodiment Construction

[0063] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

[0064] figure 1 It is a structural block diagram of the parameter synchronous optimization system of the present invention. Such as figure 1 As shown, the parameter synchronization optimization system of the present invention includes a resource monitoring and allocation module and a parameter maintenance module located at the parameter server end, a server resource request module located at e...