Distributed training method based on hybrid parallelism

Abstract

The invention discloses a distributed model training method based on hybrid parallelism, relates to the technical field of deep neural network model training, and solves the problems by adopting a hybrid parallelism mode of data parallelism and model parallelism and using multiple nodes and multiple GPUs. Firstly, for the problem of long training time, a distributed clustering method is adopted toperform parallel computing on mass data, and the training speed is increased; secondly, for the problem that a classification layer model occupies too much video memory during training, a model parallel mode is adopted, the classification layer model is segmented into a plurality of parts and deployed on a plurality of GPUs of a plurality of nodes in a cluster, meanwhile, the number of the nodescan be dynamically adjusted according to the size of the classification layer model, and classification model training under the large ID condition is met. According to the method, a hybrid parallel mode based on data parallelism and model parallelism is used, distributed cluster training is used, the model training efficiency can be greatly improved while the original deep learning training effect is kept, and classification model training under a large ID is met.

Description

technical field [0001] The invention relates to the technical field of deep learning for deep neural network model training, in particular to a hybrid parallel-based distributed training method. Background technique [0002] The concept of deep learning originates from the research of artificial neural network. It is a system science based on computer neural network theory and machine learning theory. It extracts and expresses information through multi-layer neural network, and combines the underlying features to form more abstract high-level features. , to learn the underlying regularity of the data samples. [0003] With the continuous improvement of industrial application requirements, large-scale model structure design and massive data model training have become mainstream methods, which makes the complexity and cost of deep learning continue to increase. For example, when training a large number of ID face recognition models, it takes nearly a day to train hundreds of ...

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is to provide a distributed training method based on hybrid parallelism, which uses data parallelism and model parallelism in order to solve the problems of long training time in mass ID data training, model parameters exceeding the GPU memory, and the model cannot be trained. Hybrid parallel method, using multi-nodes and multi-GPUs to solve the above problems

Images