SGD load balancing method and device based on parallel computing and storage medium

Abstract

The invention discloses an SGD load balancing method based on parallel computing. The method comprises the steps that distributed parallel gpu computing is achieved based on a design mode combining model parallelism and data parallelism; a semaphore mechanism is adopted to realize synchronous communication between a main node and sub-nodes, and an optimizer in a sub-container adopts a stochastic gradient descent algorithm to update the weight. The main node constructs a minimum spanning tree by taking errors in the sub-node control table as weights, key nodes in the graph nodes are discovered,irrelevant nodes are eliminated in sequence, and hardware resources of the irrelevant nodes are reallocated. A plurality of model copies can process different subsets of training samples at the sametime, the model copies are interactively combined periodically, and a distributed algorithm is optimized. A new architecture thought is provided to achieve a strategy of load balancing calculation, the model development efficiency is improved, the development cost is reduced, the algorithm has relatively good adaptability to the data scale, and meanwhile, asynchronous communication between dynamicmanagement sub-containers is realized.

Description

technical field [0001] The invention relates to the field of machine learning, in particular to a parallel computing-based SGD load balancing method, device and storage medium. Background technique [0002] At present, people have already experienced the great advantages of artificial intelligence in many fields. Machine learning is an important part of artificial intelligence. It helps people make decisions by modeling and training massive amounts of data. [0003] However, with the rise of big data, the scale of data is getting larger and larger, and the storage and computing capabilities in stand-alone mode can no longer meet the requirements of massive data. Distributed machine learning emerged as the times require. Using distributed machine learning to speed up model convergence has become the mainstream method in the industry. Currently, there are two common methods for distributed machine learning: model parallelism and data parallelism. [0004] However, the curren...

AI Technical Summary

Problems solved by technology

[0004] However, the current parallel computing is limited by the barrel effect, and it is often necessary to wait for the slowest node to complete the calculation before proceeding to the next step

Images