High-performance pipeline parallel deep neural network training

Abstract

Layers of a deep neural network (DNN) are partitioned into stages by using a profile of the DNN. Each of the stages includes one or more of the layers of the DNN. The partitioning of the layers of theDNN into stages is optimized in various ways including optimizing the partitioning to minimize training time, to minimize data communication between worker computing devices used to train the DNN, orto ensure that the worker computing devices perform an approximately equal amount of the processing for training the DNN. The stages are assigned to the worker computing devices. The worker computingdevices process batches of training data by using a scheduling policy that causes the workers to alternate between forward processing of the batches of the DNN training data and backward processing of the batches of the DNN training data. The stages can be configured for model parallel processing or data parallel processing.

Description

Background technique [0001] In biological nervous systems such as the human brain, deep neural networks ("DNNs") are loosely modeled after information processing and communication patterns. DNNs can be used to solve complex classification problems such as but not limited to object detection, semantic labeling and feature extraction. As a result, DNNs form the basis of many artificial intelligence ("AI") applications, such as computer vision, speech recognition, and machine translation. In many domains, DNNs can match or even exceed human accuracy. [0002] The advanced performance of DNNs stems from their ability to extract advanced features from input data after using statistical learning on large datasets to obtain an efficient representation of the input space. However, the superior performance of DNNs comes at the cost of high computational complexity. High-performance general-purpose processors, such as graphics processing units ("GPUs"), are often used to provide the ...

AI Technical Summary

Problems solved by technology

This growth not only puts pressure on the already time- and resource-intensive DNN training process, but also breaks down common parallelization methods used to train DNNs

Images