Deep neural network model parallel mode selection method

Abstract

The invention discloses a deep neural network model parallel mode selection method. The method comprises the following steps: S1, calculating the total data volume of a whole neural network model; S2,judging whether the total data volume of the neural network model obtained in the step S1 exceeds the total available memory volume of a single computing node to be trained, if not, executing the step S3, and if yes, executing the step S4; S3, selecting a data parallel mode; S4, segmenting a network layer of the neural network model, obtaining the number of computing nodes needing to be distributed by the neural network model according to a segmentation result, executing S5 if the number of the computing nodes in the input parameters is less than twice of the number of the nodes needed by model segmentation, and otherwise, executing S6; S5, selecting a model parallel mode; S6, selecting a hybrid parallel mode including data parallelism and model parallelism. According to the invention, through information acquisition and analysis of model parameters, hyper-parameters and data volume, automatic selection of a distributed extended parallel mode is realized, and high parallel performanceis ensured.

Description

technical field [0001] The invention relates to a method for selecting a parallel mode of a deep neural network model, which belongs to the technical field of deep learning. Background technique [0002] The distributed training of the data parallel mode stores a backup of a model on each computing node, and processes different parts of the data set on each computing node. The data parallel mode training method needs to combine the results of each working node, and between nodes Synchronize model parameters between. The distributed training of the model parallel mode assigns different network layers of the neural network model to different computing nodes, or assigns different parameters within the same layer to different computing nodes, and different computing nodes are responsible for the training of different parts of the network model. The hybrid parallel mode is to have both model parallelism and data parallelism in a batch of computing nodes for distributed training....

AI Technical Summary

Problems solved by technology

Although the deep network greatly improves the accuracy rate, it also makes the network model parameters more and more, and the training time is getting longer and longer, which has become a major obstacle to the rapid development and wide application of deep learning technology.

Images