Data synchronization method and device, equipment and storage medium

Abstract

The invention relates to the technical field of model training, and discloses a data synchronization method and device, equipment and a storage medium, and the method comprises the steps: constructing a first-stage physical topology between acceleration equipment of the same type, and constructing a second-stage physical topology between acceleration equipment of different types; the acceleration devices in the secondary physical topology are connected through a cache coherence protocol; first processing is carried out on data to be synchronized in the acceleration equipment in a scatterreduce communication mode according to the first-level physical topology, and second processing is carried out on the data, subjected to the first processing, in the acceleration equipment according to the second-level physical topology; and performing third processing on the data subjected to the second processing in the acceleration equipment through an allgather communication mode according to the second-level physical topology, and performing fourth processing on the data subjected to the third processing in the acceleration equipment according to the first-level physical topology. Deep learning data parallelism based on various heterogeneous acceleration devices can be realized, and the hardware resource utilization rate and the data communication efficiency are improved.

Description

technical field [0001] The present invention relates to the technical field of model training, in particular to a data synchronization method, device, equipment and storage medium. Background technique [0002] With the wide application of deep neural network, its model size becomes larger and larger, and this growth makes efficient model training more important, and distributed training emerges as the times require. The current distributed model training methods include data parallelism and model parallelism. One of the most common and widely used methods is data parallelism. The data parallel method divides the input data to be trained, and trains multiple batches of data simultaneously on multiple acceleration devices during each training iteration. Data parallelism is divided into two methods: synchronous data parallelism and asynchronous data parallelism. Among them, in the synchronous data parallel method, after all the acceleration devices calculate the gradient of...

AI Technical Summary

Problems solved by technology

However, in modern data centers, acceleration devices such as GPUs and FPGAs are widely deployed. If only one type of device can be used for each data parallel training, it will inevitably cause idle and waste of resources.

Images