Distributed parallel training method and device, and readable medium

Abstract

The invention discloses a distributed parallel training method. The method comprises the following steps: distributing GPUs for a plurality of processes; setting the plurality of GPUs under the plurality of processes to adopt the same optimizer and network state parameters; under multiple processes, distributing different network layers of the training model to different GPUs; and allocating the training set to a plurality of processes, and performing training in parallel through a plurality of GPUs under the plurality of processes. The invention further discloses computer equipment and a readable storage medium. Different training layers of the training model are allocated to different GPUs to realize model parallel training, data are allocated to different threads to realize data parallel training, and different threads allocate multiple GPUs for training, so that model parallel and data parallel hybrid training is realized, and the data processing efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of deep learning, in particular to a distributed parallel training method, device and readable medium. Background technique [0002] With the development of scientific computing technology, the neural network has achieved rapid development. At the same time, due to the continuous enhancement of the programmability of the GPU, the application capability of the GPU has far exceeded the graphics rendering task, and has been continuously applied to new fields. With excellent parallel processing capabilities, GPUs are widely used in training and reasoning of neural networks. [0003] In terms of GPU training, there are mainly two training methods, one is model parallel training, and the other is data parallel training, in which model parallel training mainly assigns different layers in the model to different GPUs, and trains the model in a pipelined manner. the weight of. And data parallelism means that...

AI Technical Summary

Problems solved by technology

Model parallelism and data parallelism each have their own advantages and disadvantages. Model parallelism is relatively large in communication data, which is not conducive to improving the efficiency of GPU during training. Data parallelism is only suitable for networks with relatively small models. When training large network structures, it will Insufficient video memory occurs

[0004] At present, there are many network architectures that support neural network training, including tensorflow, pytorch and other architectures, but most of them prefer to use data parallelism to train neural networks, and the efficiency of general training is not particularly high. Taking tensorflow as an example, in When using standard tensorflow to test on 128 Pascal GPUs, both Inception V3 and ResNet-101 waste nearly half of the GPU computing power

Images