A decentralized distributed training topology, training system and method

Abstract

The invention discloses a decentralized distributed training topology, comprising: the topology is an n-dimensional hypersquare topology, which is a closed, compact, and convex undirected graph, and the topology consists of a finite non- Composed of empty node sets and finite edge sets, the 1-dimensional skeleton of the topology is composed of a group of equal-length line segments aligned in each dimension in its space, where the opposite line segments are parallel to each other and intersect at a point The line segments are orthogonal to each other. The present invention focuses on decentralized distributed performance training, "homogenizes" the training tasks, and evenly distributes the training task load to each training node in the distributed training system. The system performance no longer depends on Based on the performance of a single training node, it has the advantages of short iteration time, data localization, and high communication effectiveness.

Description

technical field [0001] The invention relates to the technical field of distributed training in computer technology, in particular to a decentralized distributed training topology, training system and method. Background technique [0002] With the massive growth of data and the sharp increase in the size of deep models in artificial intelligence, a large amount of training time and computing resources are required to complete effective training, so the field of distributed training has received a lot of attention. [0003] The traditional centralized distributed training system has the shortcomings of long iteration time and high bandwidth requirements, and the performance of the distributed training system is completely dependent on the performance of the central training node. In addition, the decentralized distributed training system has a variety of topological structures, and different structures determine Different communication frequency, communication times and commun...

AI Technical Summary

Problems solved by technology

[0003] The traditional centralized distributed training system has the shortcomings of long iteration time and high bandwidth requirements, and the performance of the distributed training system is completely dependent on the performance of the central training node. In addition, the decentralized distributed training system has a variety of topological structures, and different structures determine Different communication frequencies, communication times and communication volumes

These have greatly affected the performance of the decentralized distributed training system, resulting in slower training tasks and longer time

Images