Decentralized distributed training topological structure, training system and method

Abstract

The invention discloses a decentralized distributed training topological structure, and the structure is characterized in that the structure is an n-dimensional hypersquare topological structure, is a closed, compact and convex undirected graph, and is composed of a finite non-empty node set and a finite edge set; a one-dimensional skeleton of the topological structure is composed of a group of line segments which are orderly arranged in a space where the one-dimensional skeleton is located and aligned with each dimension and are equal in length, wherein the opposite line segments are parallel to each other, and the line segments intersecting at one point are orthogonal to each other. The method focuses on decentralized distributed performance training, the training tasks are 'homogenized', the load of the training tasks is uniformly distributed to each training node in the distributed training system, the system performance does not depend on the performance of a single training node any more, and the method has the advantages of short iteration time consumption, data localization and higher communication effectiveness.

Description

technical field [0001] The invention relates to the technical field of distributed training in computer technology, in particular to a decentralized distributed training topology, a training system and a method. Background technique [0002] With the massive growth of data and the dramatic increase in the size of deep models in artificial intelligence, a lot of training time and computing resources are required to complete effective training, so the field of distributed training has received a lot of attention. [0003] The traditional centralized distributed training system has the shortcomings of long iteration time and high bandwidth requirements, and the performance of the distributed training system is completely dependent on the performance of the central training node. In addition, the decentralized distributed training system has various topological structures, which are determined by different structures. different communication frequencies, times of communication, ...

AI Technical Summary

Problems solved by technology

[0003] The traditional centralized distributed training system has the shortcomings of long iteration time and high bandwidth requirements, and the performance of the distributed training system is completely dependent on the performance of the central training node. In addition, the decentralized distributed training system has a variety of topological structures, and different structures determine Different communication frequencies, communication times and communication volumes

These have greatly affected the performance of the decentralized distributed training system, resulting in slower training tasks and longer time

Images