Model training method, server and computer readable storage medium

Abstract

The invention discloses a model training method, a server and a computer readable storage medium, and belongs to the field of artificial intelligence calculation. The model training method comprises the steps: after a training job is received, acquiring job information; copying a model to each GPU, and synchronizing the initial values of model parameters of each GPU; extracting a part of sample data each time of iteration, splitting the extracted sample data, distributing the split sample data to different GPUs for training, performing global protocol operation on parameter gradients obtainedby training of all the GPUs, and updating the model parameters on the GPUs according to the parameter gradients after protocol; and after iteration of the iteration times is completed, selecting the model parameters of any GPU and storing the model parameters as a model training result. According to the model training method, the bandwidth bottleneck and the computing power bottleneck between thecomputing nodes are eliminated by fully utilizing the high-speed data transmission bandwidth of the GPU-GPU, so that the synchronous training efficiency and the speed-up ratio of the model on the GPUcluster are improved.

Description

technical field [0001] The invention relates to the field of artificial intelligence computing, in particular to a model training method, a server and a computer-readable storage medium. Background technique [0002] The training of deep learning models requires huge computing resources, and it can take days or even months to complete a training session. In order to speed up model training, it is often necessary to use large-scale GPU (Graphic Processing Unit, Graphics Processing Unit) clusters for parallel training. At the same time, deep learning training tasks often use some parallel algorithms to distribute training tasks to multiple computing nodes to run simultaneously. Including data parallel and model parallel two types of parallel algorithms, data parallel is more commonly used. [0003] In data parallel algorithms, such as figure 1 Shown, usually using PS-Worker (Parameter [0004] Server-Worker (parameter server-computing node) architecture is deployed on GPU c...

AI Technical Summary

Problems solved by technology

Due to the large amount of interactive data, the bandwidth and CPU processing power between PS-Worker often become the bottleneck, and the powerful parallel computing capability of the GPU on the Worker cannot be used, resulting in low system acceleration ratio, poor scalability, and waste of hardware resources.

Images