High-performance pipeline-parallel deep neural network training

Abstract

The layers of a deep neural network (DNN) are divided into multiple stages using a profile of the DNN. Each stage in the stage includes one or more layers in the layers of the DNN. The division of the layers of the DNN into multiple stages is optimized in various ways, including optimizing the division to minimize training time, minimizing data communication between worker computing devices used to train the DNN, or ensuring that the worker computing devices perform approximately equal amounts of processing to train the DNN. Stages are assigned to worker computing devices. The worker computing devices process batches of training data using a scheduling strategy that causes workers to alternate between forward processing of batches of DNN training data and backward processing of batches of DNN training data. These stages can be configured for model parallel processing or data parallel processing.

Description

Background Art

[0001] Deep neural networks ("DNNs") are loosely modeled after the information processing and communication patterns in biological neural systems such as the human brain. DNNs can be used to solve complex classification problems such as, but not limited to, object detection, semantic tagging, and feature extraction. As a result, DNNs form the basis of many artificial intelligence ("AI") applications, such as computer vision, speech recognition, and machine translation. In many areas, DNNs can match or even exceed human accuracy.

[0002] The high performance of DNNs stems from their ability to extract high-level features from input data after using statistical learning on large data sets to obtain an effective representation of the input space. However, the superior performance of DNNs comes at the cost of high computational complexity. High-performance general-purpose processors such as graphics processing units ("GPUs") are often used to provide the high levels of computational performance required by many DNN applications.

[0003] However, as DNNs become more widely developed and used, model sizes have increased to improve performance. Today's models have tens to hundreds of layers, typically with a total of 10 to 20 million parameters. This growth not only puts pressure on the already time- and resource-intensive DNN training process, but also makes the commonly used parallelization methods for training DNNs collapse.

[0004] It is with respect to these and other technical challenges that the disclosure made herein is presented. Summary of the invention

[0005] Disclosed herein is a technique for high-performance pipeline parallel DNN model training. The disclosed DNN model (which may be referred to herein as "DNN") training system parallelizes the training of the DNN model by pipelineizing various aspects of the training process across computing devices, wherein the computing devices are configured to process a range of DNN layers. In addition to other technical benefits, the disclosed pipeline parallel computing technique can also eliminate the performance impact caused by previous parallelization methods when training large DNN models or when limited network bandwidth causes a high communication-to-computation ratio.

[0006] For large DNN models, the disclosed pipeline-parallel DNN training techniques can also reduce communication overhead by up to ninety-five percent (95%) relative to data-parallel training by enabling overlapping communication and computation. Additionally, the disclosed techniques can maintain GPU productivity by partitioning DNN layers between pipeline stages to balance work and minimize communication, versioning model parameters for backward pass correctness, and scheduling forward and backward passes of a bidirectional training pipeline.

[0007] Using the mechanisms briefly described above and described more fully below, embodiments of the disclosed technology have been shown to be five times faster than data parallel training in terms of "time to target accuracy" for DNN training. This efficiency improvement can reduce the utilization of various types of computing resources, including but not limited to memory, processor cycles, network bandwidth, and power. Other technical benefits not specifically identified herein may also be achieved through embodiments of the disclosed technology.

[0008] To achieve the technical benefits briefly mentioned above and potential other benefits, the disclosed technology utilizes a combination of pipeline mechanisms, model parallelism, and data parallelism. This combination is referred to herein as "pipeline parallel" DNN training. In order to achieve pipeline parallel DNN training, a profile of the DNN model is generated. The DNN profile can be generated by executing the DNN on a small number of computing devices (e.g., one) using a subset of the DNN training data (e.g., several thousand mini-batches).

[0009] Once the DNN profile has been generated, the layers of the DNN model are divided into stages based on the profile. Each stage in the stage includes one or more layers of the DNN model. In some embodiments, the partitioning of the DNN is optimized to minimize the time to train the DNN model to a desired level of accuracy.

[0010] The partitioning of the DNN model may also or alternatively be optimized to minimize data communication between computing devices, or to configure the computing devices used to train the DNN model to perform approximately the same amount of processing during training. The partitioning of the layers of the DNN model may also include calculating the optimal number of batches of DNN training data provided to the computing devices used for training to maximize their processing efficiency.

[0011] Once the DNN model is divided into stages, the stages are individually assigned to the computing devices that will train the DNN model. Some or all stages can be configured for model parallel processing, and some or all stages can be configured for data parallel processing.

[0012] In some configurations, computing devices are configured using a forward-once, backward-once ("1F1B") scheduling strategy. The 1F1B scheduling strategy configures the computing devices to alternate between forward processing of batches of DNN training data and backward processing of batches of DNN training data. Once the computing devices are configured in this manner, they can begin processing DNN training data to train a DNN model.

[0013] It should be understood that the above subject matter can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or an article such as a computer-readable medium. These and various other features will become apparent by reading the following detailed description and viewing the associated drawings.

[0014] This Summary is provided to introduce in simplified form some aspects of the disclosed technology that are further described below in the Detailed Description. This Summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a computing architecture diagram illustrating aspects of a mechanism disclosed herein for configuring a computing device to implement pipelined parallel DNN training;

[0016] Figure 2 is a flow chart showing a routine illustrating aspects of an illustrative computer-implemented process for generating a profile of a DNN;

[0017] Figure 3 is a flow chart showing a routine illustrating aspects of an illustrative computer-implemented process for optimizing allocation of layers of a DNN to computing devices for pipeline-parallel training of the DNN;

[0018] Figure 4A and Figure 4B is a computing system diagram showing several example allocations of layers of a DNN to computing devices for pipeline-parallel training of the DNN;

[0019] Figure 5 is a workflow diagram illustrating aspects of a forward-once-backward scheduling strategy disclosed herein for allocating work to computing devices for pipeline-parallel DNN training;

[0020] Figure 6 is a flow chart showing a routine illustrating aspects of an illustrative computer-implemented process for performing pipelined parallel DNN training using a forward-once-backward scheduling strategy;

[0021] Figure 7 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that may implement aspects of the techniques presented herein; and

[0022] Figure 8 is a network diagram illustrating a distributed computing environment in which aspects of the disclosed technology may be implemented. DETAILED DESCRIPTION

[0023] The following detailed description relates to techniques for high-performance pipeline parallel DNN model training. In addition to other technical benefits, the disclosed techniques can also eliminate the performance impact caused by previous parallelization techniques when training large DNN models or when network bandwidth causes a high communication-to-computation ratio. The disclosed techniques can also divide the layers of the DNN model between pipeline stages to balance work and minimize network communication, and efficiently schedule the forward and backward passes of the bidirectional DNN training pipeline. These and other aspects of the disclosed techniques can reduce the utilization of various types of computing resources, including but not limited to memory, processor cycles, network bandwidth, and power. Other technical benefits not specifically identified herein can also be achieved through implementations of the disclosed techniques.

[0024] Before describing the disclosed techniques for pipeline parallel DNN training, a brief overview of several methods for DNN models, DNN model training, and parallel training of DNN models will be provided. DNN models are typically composed of a sequence of different types of layers (e.g., convolutional layers, fully connected layers, and pooling layers). DNN models are typically trained using labeled datasets (e.g., a set of images that have been labeled with descriptions of what is in the images). DNN models are trained across multiple epochs. In each epoch, the DNN model is trained in multiple steps through all the training data in the dataset. In each step, the current model first makes predictions on a subset of the training data (which may be referred to herein as a "mini-batch" or "batch"). This step is typically referred to as a "forward pass."

[0025] To make predictions, input data from a mini-batch is fed to the first layer of the DNN model, which is often referred to as the "input layer." Each layer of the DNN model then computes a function on its input, typically using learned parameters or weights, to produce input for the next layer. The output of the last layer (often referred to as the "output layer") is a class prediction. Based on the labels predicted by the DNN model and the actual labels for each instance of the training data, the output layer computes a loss or error function. In the "backward pass" of the DNN model, each layer of the DNN model will compute the error and gradient for the previous layer, or update the weights of that layer that move the DNN model's prediction toward the desired output.

[0026] One goal of DNN training is to obtain a DNN model with a desired level of accuracy in as little time as possible. This goal can be quantified by two metrics: statistical efficiency (i.e., the number of generations required to achieve the desired level of accuracy) and hardware efficiency (i.e., the time required to complete a single generation). The total training time to achieve the desired level of accuracy is the product of these two metrics. The result of training a DNN model is a set of parameters called "weights" or "kernels". These parameters represent transformation functions that can be applied to the input, and the result is a classified or semantically labeled output.

[0027] In order to train large models in a reasonable amount of time, one of the following two methods can be used to perform training in parallel across multiple GPUs: model parallelism or data parallelism. In the case of using model parallelism or model parallel processing, the entire DNN model is copied to multiple GPU-equipped computing devices (which may be referred to as "workers" or "worker devices" in this article), where each worker processes a different subset of the training data. The weight updates calculated on individual worker devices are aggregated to obtain a final weight update that reflects the update across all training data. During and after each aggregation, the amount of data communicated between worker devices is proportional to the size of the DNN model.

[0028] Although model parallelism enables training of very large models, conventional model parallelism is inefficient for training DNN models because DNN training requires that the forward pass traverse all layers before the backward pass can determine parameter updates. As a result, conventional model parallelism can lead to severe underutilization of computational resources because it either actively uses only one worker device at a time (if partitioned across layers) or fails to overlap computation and communication (if partitioned per layer).

[0029] In data parallelism or data parallel processing, the training dataset is partitioned across multiple GPUs. Each GPU maintains a complete copy of the DNN model and trains on its own partition of the training data while periodically synchronizing weights with other GPUs. The frequency of weight synchronization affects both statistical and hardware efficiency. The synchronization of weights at the end of each micro-batch (which may be referred to as batch synchronous parallelism or "BSP") reduces staleness in training, thereby ensuring statistical efficiency. However, BSP requires each GPU to wait for gradients from other GPUs, which greatly reduces hardware efficiency.

[0030] Although data-parallel DNN training works well with some DNN models with high computation-to-communication ratios, two trends threaten its effectiveness. First, the growing size of DNN models increases the communication of each aggregate network. In fact, some current DNN models are large enough that the data communication overhead exceeds the computation time of the GPU, which limits expansion and almost dominates the entire DNN training time. Second, the rapid improvement of GPU computing power further shifts the bottleneck of DNN training to data communication for all types of DNN models. The technology disclosed in this article addresses these and potentially other considerations.

[0031] Referring now to the drawings, in which like reference numerals represent like elements throughout the several drawings, aspects of various techniques for high-performance pipeline parallel DNN training will be described. In the following detailed description, reference is made to the accompanying drawings that form a part thereof, and specific configurations or examples are shown by way of illustration.

[0032] Figure 1 is a computational architecture diagram illustrating aspects of the system disclosed herein for pipeline parallel DNN training. As will be discussed in more detail below, Figure 1 The system illustrated in the figure partitions the DNN and assigns subsets of the DNN's layers to different worker devices for training. The system also pipelines the processing of mini-batches of training data, injecting multiple mini-batches into the worker device processing the first DNN layer before the processing of the previous DNN layer is completed. This can keep the processing pipeline full and ensure concurrent processing on the worker devices (i.e., each worker processes a different mini-batch at any particular point in time). The disclosed system also uses data parallelism for selected subsets of layers to balance the computational load among the worker devices. This combination of pipelining, model parallelism, and data parallelism is referred to herein as "pipeline-parallel" DNN training.

[0033] like Figure 1 As shown in , DNN definition 102 is utilized in some configurations describing the architecture of DNN model 100. For example, but not limited to, DNN definition 102 may include data describing the architecture of DNN model 100, including but not limited to: layers of DNN model 100, configurations of layers of DNN model 100, and the type and amount of training data 103 to be used to train DNN model 100. DNN definition 102 is provided to a DNN analyzer in one configuration.

[0034] The DNN analyzer 104 is a software or hardware component that determines an optimal partitioning of the layers of the DNN model 100 among the worker computing devices 112 used to train the DNN model 100. The inputs to the DNN analyzer 104 are the DNN definition 102, the training data 103 to be used by the DNN model 100, and data identifying the number of worker computing devices 112 to be used to train the DNN model 100. The DNN analyzer 104 determines the optimal partitioning of the layers of the DNN model 100 by briefly training the DNN model 100 on a subset of the worker computing devices 112 (e.g., a single worker computing device 112) and observing the performance characteristics of the DNN model 100 in some configurations. The DNN analyzer 104 trains the DNN model 100 using a subset of mini-batches from the training data 103 (e.g., 1000 mini-batches). The DNN analyzer 104 outputs a DNN profile 106, which includes data describing the performance characteristics of the DNN model 100.

[0035] Temporary reference Figure 2 , a flowchart showing a routine 200 will be described, which shows aspects of an illustrative computer-implemented process for generating a DNN profile 106. It should be understood that Figure 2 The logical operations described in the and other figures may be implemented as (1) a sequence of computer-implemented actions or program modules running on a computing device and / or (2) interconnected machine logic circuits or circuit modules within a computing device.

[0036] The specific implementation of the technology disclosed herein is a matter of choice depending on the performance and other requirements of the computing device. Therefore, the logical operations described herein are variously referred to as states, operations, structural devices, actions, or modules. These states, operations, structural devices, actions, and modules can be implemented in hardware, software, firmware, dedicated digital logic, and any combination thereof. It should be understood that more or less operations than those shown in the accompanying drawings and described herein may be performed. These operations may also be performed in an order different from the order described herein.

[0037] The routine 200 begins at operation 202, where the DNN analyzer 104 calculates the total computation time for each layer in the DNN model 100. The total computation time for each layer is calculated as the sum of the amount of time required for the forward pass and the backward pass of the layer. The routine 200 then proceeds from operation 202 to operation 204, where the DNN analyzer 104 calculates the size of the output activation from each layer to the next layer in the DNN model 100. This also matches the size of the input gradient in the backward pass.

[0038] From operation 204, the routine 200 proceeds to operation 206, where the DNN analyzer 104 determines the size of the weights for each layer of the DNN model. From operation 206, the routine 200 proceeds to operation 208, where the DNN analyzer 104 stores the DNN profile 106. The DNN profile 106 includes data describing the total computation time for each layer, the size of the output activations from each layer to the next layer, and the size of the weights for each layer. The DNN profile 106 may include other data describing the performance characteristics of the DNN model 100 in other configurations. From operation 208, the routine 200 proceeds to operation 210, which ends there.

[0039] like Figure 1 As shown in , in some configurations, the DNN analyzer 104 provides the DNN profile 106 to the DNN optimizer 108. The DNN optimizer 108 is a software or hardware component that utilizes the DNN profile 106 to generate an optimized layer allocation 110. In order to generate the optimized layer allocation 110, the DNN optimizer 108 divides the DNN model 100 into a plurality of stages. Each of the plurality of stages includes one or more layers of the DNN model 100.

[0040] In some embodiments, the partitioning of the DNN model 100 is optimized to minimize the time required to train the DNN model 100 to a desired level of accuracy. The partitioning of the DNN model 100 may also, or alternatively, be optimized to minimize data communication between computing devices 112, or to configure computing devices 112 used to train the DNN model 100 to perform approximately the same amount of processing during training. The partitioning of the layers of the DNN model 100 may also include calculating the optimal number of batches of DNN training data 103 provided to the computing devices 112 for training to maximize their processing efficiency.

[0041] Some or all stages may be configured for model parallel processing. Some or all stages may be configured for data parallel processing. When data parallel processing is used, multiple worker computing devices 112 may be assigned to a given stage, with each worker computing device processing a different micro-batch during execution.

[0042] Once the DNN model is divided into multiple stages, the stages are individually assigned to the GPUs 118 in the worker computing devices 112 that will train the DNN model 100. Each stage is mapped to a separate GPU 118, which performs a forward pass and a backward pass for all layers of the stage. A stage containing an input layer may be referred to herein as an input stage, and a stage containing an output layer may be referred to herein as an output stage.

[0043] Temporary reference Figure 3, a flowchart showing a routine 300 will be described, which illustrates aspects of an illustrative computer-implemented process for optimizing the allocation of layers of a DNN model 100 to computing devices 112 for pipeline-parallel training of a DNN. The routine 300 begins at operation 302, where the DNN optimizer 108 calculates an optimal partitioning of the layers of the DNN model 100 into multiple stages. As described above, the partitioning of the DNN model 100 can be optimized to minimize the time to train the DNN model 100, minimize data communication between computing devices 112, or configure the computing devices 112 used to train the DNN model 100 to perform approximately the same amount of processing during training. In other configurations, the partitioning of the DNN model 100 can be optimized for other metrics.

[0044] From operation 302, the routine 300 proceeds to operation 304, where the DNN optimizer 108 calculates the replication factor for each stage. Then, the routine 300 proceeds from operation 304 to operation 306, where the DNN optimizer 108 calculates the optimal number of mini-batches of the DNN training data 103 to provide to the computing device 112 for training to maximize its processing efficiency. Then, the routine 300 proceeds from operation 306 to operation 308, where the DNN optimizer 108 stores the optimized layer allocation 110, including but not limited to: data defining the optimal partitioning of the layers of the DNN model 100, the replication factor, and the optimal number of mini-batches of the DNN training data 103 to provide to the computing device 112 for training to maximize its processing efficiency. Then, the routine 300 proceeds from operation 308 to operation 310, where it ends.

[0045] like Figure 1 As shown in , in some configurations, the worker computing device 112 is configured with a 1F1B scheduling strategy 116. The 1F1B scheduling strategy configures the computing device 112 to alternate between forward processing of batches of DNN training data 103 and backward processing of batches of DNN training data 103. Figure 5 and Figure 6 Additional details about the 1F1B scheduling strategy are described. Once the computing devices are configured with the optimized layer allocation 110 and the 1F1B scheduling strategy 116, they can begin processing DNN training data to perform DNN training 114.

[0046] Figure 4A and Figure 4B 1 is a computing system diagram showing several example allocations of layers of a DNN model 100 to computing devices 112 for pipelined parallel training of the DNN 100. Figure 4AIn the example configuration shown in , the DNN model 100 includes seven layers 402A-402G, including an input layer 402A and an output layer 402G. In this example, the DNN optimizer 108 has generated an optimized layer allocation 110 including three stages 404A-404C. The input stage 404A includes layers 402A-402C, the stage 404B includes layer 402D, and the output stage 404C includes layers 402E-402G. Stage 404A is assigned to a worker computing device 112A, stage 404B is assigned to a worker computing device 112B, and stage 404C is assigned to a worker computing device 112C.

[0047] exist Figure 4A In the example shown in , each of stages 404A-404C is implemented using parallel model processing. Figure 4B In the example shown in , data parallel processing is used to implement stage 404B. In this example, two worker computing devices 112B and 112D implement stage 404B by operating on different mini-batches of DNN training data 103.

[0048] Figure 5 1 is a workflow diagram illustrating aspects of a 1F1B scheduling strategy 116 disclosed herein for allocating mini-batches of training data 103 to computing devices 112 for pipelined parallel DNN training. As briefly discussed above, the 1F1B scheduling strategy 116 configures the computing devices 112 to alternate between forward processing of batches of DNN training data 103 and backward processing of batches of DNN training data 103.

[0049] Unlike conventional unidirectional pipelines, DNN training is bidirectional (i.e., a forward pass is followed by a backward pass, going through the same layers in reverse order). The 1F1B scheduling strategy 116 interleaves forward and backward mini-batch processing on each worker computing device 112, and on the backward pass, mini-batches of training data 103 are routed through the same worker 112. This helps keep all worker computing devices 112 busy without stalling the pipeline, while preventing too many in-flight mini-batches.

[0050] exist Figure 5In the example shown in , four worker computing devices 112 implement four stages of the DNN model 100 and process mini-batches of the training data 103. The row labeled 'A' in the workflow diagram corresponds to the first worker 112, the row labeled 'B' corresponds to the second worker 112, the row 'C' corresponds to the third worker 112, and the row 'D' corresponds to the fourth worker 112. The number in the box in each row indicates the number of the mini-batch currently being processed by the corresponding worker 112. Boxes with diagonal lines indicate mini-batches being processed in the forward direction, boxes with hatched lines indicate mini-batches being processed in the backward direction, and boxes without lines indicate that the worker 112 is idle during the corresponding time period.

[0051] In the startup state, the input stage allows a sufficient number of mini-batches of training data 103 (four in this example) to keep it full while the pipeline enters a steady state. These mini-batches propagate on their way to the output stage. Once the forward pass for the first mini-batch is completed in the output stage, a backward pass is performed for the same mini-batch, and then for subsequent mini-batches, it begins alternating between performing forward passes and performing backward passes. This can be done in Figure 5 As the backward pass begins to propagate to the previous stages of the pipeline, each stage begins to alternate between the forward pass and the backward pass for different mini-batches. Thus, in the steady state, each worker 112 is busy, performing one of the forward pass or the backward pass for a mini-batch.

[0052] Upon completing the forward pass for a mini-batch, each stage asynchronously sends its output activations to the next stage and starts performing work for another mini-batch. Similarly, upon completing the backward pass for a mini-batch, each stage asynchronously sends gradients to the previous stage and starts computation for another mini-batch.

[0053] When the stage is run in a data-parallel configuration replicated across multiple GPUs (such as Figure 4B In the example shown in ), deterministic round-robin load balancing can be used to distribute work from previous stages across GPUs. This deterministic load balancing ensures that the backward work for a mini-batch goes through the same stages that the mini-batch went through in its forward work stage. Both the 1F1B scheduling strategy for stages in the pipeline and the round-robin scheduling strategy for load balancing across replicated stages are static strategies. Therefore, they can be executed independently by each worker 112 without expensive distributed coordination.

[0054] Figure 66 is a flow chart showing a routine 600 illustrating aspects of an illustrative computer-implemented process for performing pipelined parallel DNN training using the 1F1B scheduling strategy 116. The routine 600 begins at operation 602, where the worker computing device 112 begins processing mini-batches of the training data 103.

[0055] The routine 600 then proceeds from operation 602 to operation 604, where the worker computing device 112 determines whether it has reached a steady state. If so, the routine 600 proceeds from operation 604 to operation 606. If not, the routine 600 proceeds back to operation 602. At operation 606, the worker computing device 112 performs forward processing on a mini-batch of training data 103. The routine 600 then proceeds from operation 606 to operation 608, where the worker computing device 112 performs backward processing on the mini-batch. The routine 600 then proceeds to operation 610, where the worker computing device 112 determines whether all of the training data 103 has been processed. If not, the routine returns to operation 606. Once all of the training data 103 has been processed, the routine 600 proceeds from operation 610 to operation 612, where it ends.

[0056] The techniques disclosed herein may also enable other optimizations of the DNN model 100 training pipeline. In particular, weight storage is utilized in some configurations to maintain multiple versions of weights, one for each active mini-batch.

[0057] When performing forward work, each stage uses the latest version of the available weights to process the mini-batch. After completing the forward work, a copy of the weights is stored as part of the intermediate state for that mini-batch. When performing a backward pass for a mini-batch, the same version of the weights will be used to calculate the weight gradients. In this way, weight caching ensures that within a stage, both forward and backward work for a given mini-batch use the same version of the model parameters. Other optimizations can also be used, including but not limited to pre-allocating all GPU memory at the beginning of training to minimize dynamic memory allocation.

[0058] Figure 7 is a computer architecture diagram that shows an illustrative computer hardware and software architecture for a computing device that can implement the various techniques presented herein. In particular, Figure 7 The architecture illustrated in the figure may be used to implement a server computer, a mobile phone, an e-reader, a smart phone, a desktop computer, an alternate reality or virtual reality (“AR / VR”) device, a tablet computer, a laptop computer, or another type of computing device.

[0059] Although the subject matter described herein is presented in the general context of a server computer that performs parallelized training of a DNN model, those skilled in the art will recognize that other embodiments may be performed in conjunction with other types of computing systems and modules. Those skilled in the art will also recognize that the subject matter described herein may be practiced with other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices (such as wearable computing devices, automobiles, home automation, etc.), microcomputers, mainframe computers, etc.

[0060] Figure 7 The illustrated computer 700 includes one or more central processing units 702 ("CPUs"), system memory 704 (including random access memory 706 ("RAM") and read-only memory ("ROM") 708), and a system bus 710 that couples memory 704 to CPU 702. A basic input / output system ("BIOS" or "firmware") containing basic routines to help transfer information between elements within the computer 700, such as during startup, may be stored in ROM 708. The computer 700 also includes a mass storage device 712 for storing an operating system 722, application programs, and other types of programs. The mass storage device 712 may also be configured to store other types of programs and data, such as DNN definitions 102, DNN analyzers 104, DNN profiles 106, DNN optimizers 108, and optimized layer allocations 110 ( Figure 7 not shown).

[0061] The mass storage device 712 is connected to the bus 710 through a mass storage controller ( Figure 7 700). The mass storage device 712 and its associated computer readable media provide non-volatile storage for the computer 700. Although the description of computer readable media contained herein refers to a mass storage device, such as a hard disk, a CD-ROM drive, a DVD-ROM drive, or a USB memory key, it should be understood by those skilled in the art that the computer readable media can be any available computer storage media or communication media that can be accessed by the computer 700.

[0062] Communication media includes computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any delivery medium. The term "modulated data signal" refers to a signal that has one or more of its characteristics changed or set in a manner that can encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0063] By way of example and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computer 700. For the purposes of the claims, the phrase "computer storage media" and its variations do not include waves or signals themselves or communications media.

[0064] According to various configurations, the computer 700 can operate in a networked environment using logical connections to remote computers through a network such as network 720. The computer 700 can be connected to the network 720 via a network interface unit 716 connected to the bus 710. It should be understood that the network interface unit 716 can also be used to connect to other types of networks and remote computer systems. The computer 700 can also include an input / output controller 718 for receiving and processing input from a plurality of other devices (including keyboards, mice, touch inputs, electronic pens ( Figure 7 Similarly, the input / output controller 718 can provide input to a display screen or other type of output device (not shown in FIG. Figure 7 (not shown in the figure) provides an output.

[0065] It should be understood that the software components described herein, when loaded into the CPU 702 and executed, can convert the CPU 702 and the entire computer 700 from a general-purpose computing device to a special-purpose computing device customized to facilitate the functions described herein. The CPU 702 can be composed of any number of transistors or other discrete circuit elements that can present any number of states individually or collectively. More specifically, in response to the executable instructions contained in the software modules disclosed herein, the CPU 702 can operate as a finite state machine. These computer executable instructions can convert the CPU 702 by specifying how the CPU 702 converts between states, thereby converting the transistors or other discrete hardware elements that constitute the CPU 702.

[0066] Encoding the software modules proposed herein can also transform the physical structure of the computer-readable medium proposed herein. In different implementations of this specification, the specific transformation of the physical structure depends on various factors. Examples of such factors include, but are not limited to, the technology used to implement the computer-readable medium, whether the characteristics of the computer-readable medium are primary storage or auxiliary storage, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also convert the physical state of these components to store data thereon.

[0067] As another example, the computer storage medium disclosed herein can be implemented using magnetic or optical technology. In this embodiment, when software is encoded therein, the software proposed herein can transform the physical state of magnetic or optical media. These transformations can include changing the magnetic properties of specific locations in a given magnetic medium. These transformations can also include changing the physical characteristics or properties of specific locations in a given optical medium to change the optical properties of these locations. Without departing from the scope and spirit of this specification, other transformations of physical media are possible, wherein the aforementioned examples are provided only to facilitate this discussion.

[0068] In view of the above, it should be understood that many types of physical transformations occur in the computer 700 in order to store and execute the software components presented herein. It should also be understood that Figure 7 The architecture shown for computer 700 or similar architectures may be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices (such as smartphones, tablets, and AR / VR devices), and other types of computing devices known to those skilled in the art. It is also contemplated that computer 700 may not include Figure 7All components shown may include Figure 7 Other components not explicitly shown in the figure or the like may be used Figure 7 The architecture shown is a completely different architecture.

[0069] Figure 8 8 is a network diagram illustrating a distributed network computing environment 800 in which aspects of the disclosed technology may be implemented according to various configurations presented herein. Figure 8 As shown in the figure, one or more server computers 800A can be interconnected with multiple client computing devices (such as but not limited to tablet computers 800B, game consoles 800C, smart watches 800D, phones 800E (such as smart phones), personal computers 800F and AR / VR devices 800G) via a communication network 720 (which can be a fixed wired or wireless LAN, WAN, intranet, extranet, peer-to-peer network, virtual private network, Internet, Bluetooth communication network, proprietary low-voltage communication network or other communication network).

[0070] For example, in a network environment where the communication network 720 is the Internet, the server computer 800A may be a dedicated server computer operable to process and communicate data with the client computing devices 800B-800G via any of a variety of known protocols, such as the Hypertext Transfer Protocol (“HTTP”), the File Transfer Protocol (“FTP”), or the Simple Object Access Protocol (“SOAP”). In addition, the network computing environment 800 may utilize various data security protocols, such as Secure Sockets Layer (“SSL”) or Pretty Good Privacy (“PGP”). Each client computing device 800B-800G may be equipped with an operating system operable to support one or more computing applications or terminal sessions, such as a web browser ( Figure 8 ) or other graphical user interface ( Figure 8 ) or a mobile desktop environment ( Figure 8 ), to gain access to server computer 800A.

[0071] The server computer 800A can be communicatively coupled to other computing environments ( Figure 8 ), and receives data about the interaction / resource network of participating users. In illustrative operation, a user ( Figure 8 ) can interact with computing applications running on client computing devices 800B-800G to obtain desired data and / or execute other computing applications.

[0072] Data and / or computing applications may be stored on one or more servers 800A and transmitted to collaborating users via client computing devices 800B-800G via exemplary communication network 720. Figure 8 800A) can request access to all or part of specific data and applications hosted on the server computer 800A. Such data can be transmitted between the client computing devices 800B-800G and the server computer 800A for processing and storage.

[0073] The server computer 800A can host computing applications, processes and applets for data, application generation, authentication, encryption and communication, and can communicate with other server computing environments ( Figure 8 Not shown), third-party service providers ( Figure 8 (not shown), network attached storage ("NAS"), and storage area network ("SAN") collaborate to enable application / data transactions.

[0074] It should be understood that Figure 7 The computational architecture shown in Figure 8 The distributed network computing environment shown in is simplified for ease of discussion. It should also be understood that the computing architecture and distributed computing network may include and utilize more computing components, devices, software programs, network devices and other components not specifically described herein.

[0075] The disclosure presented herein also encompasses the subject matter set forth in the following examples:

[0076] Example A: A computer-implemented method for parallel training of a DNN model, comprising: generating a profile of a deep neural network (DNN) model, the DNN model comprising a plurality of layers; dividing the layers of the DNN model into a plurality of stages based on the profile, wherein each of the plurality of stages comprises one or more of the layers of the DNN model, and wherein the division is optimized to minimize the time to train the DNN model; and enabling a plurality of computing devices to train the DNN model.

[0077] Example B: The computer-implemented method of Example A, wherein the partitioning is further optimized to minimize data communication between the computing devices.

[0078] Example C: A computer-implemented method according to any of Examples AB, wherein the partitioning is further optimized so that each of the multiple computing devices performs approximately the same amount of processing during training of the DNN model.

[0079] Example D: A computer-implemented method according to any one of Examples AC, wherein the partitioning of the layers of the DNN model further includes: calculating the optimal number of batches of DNN training data to be provided to the multiple computing devices to maximize the processing efficiency of the multiple computing devices.

[0080] Example E: The computer-implemented method according to any one of Examples AD further includes assigning at least one of the multiple stages to each computing device among a plurality of computing devices, wherein the computing devices are configured to process batches of the DNN training data to train the DNN model by alternating between forward processing of the batches of the DNN training data and backward processing of the batches of the DNN training data.

[0081] Example F: A computer-implemented method according to any of Examples AE, wherein at least one of the multiple stages is configured for model parallel processing.

[0082] Example G: The computer-implemented method of any of Examples AF, wherein at least one of the plurality of stages is configured for data parallel processing.

[0083] Example H: A computer-implemented method according to any of Examples AG, wherein at least one of the multiple stages is configured for model parallel processing, and wherein at least one of the multiple stages is configured for data parallel processing.

[0084] Example I: A computing device comprising: one or more processors; and at least one computer storage medium having computer executable instructions stored thereon, which, when executed by the one or more processors, will cause the computing device to: divide the layers of the DNN model into multiple stages, wherein each of the multiple stages includes one or more of the layers of the DNN model, and wherein the division is optimized to minimize the time for training the DNN model; and assign at least one of the multiple stages to each worker computing device in a plurality of worker computing devices, the computing devices being configured to: process batches of the DNN training data to train the DNN model by alternating between forward processing of the batches of the DNN training data and backward processing of the batches of the DNN training data.

[0085] Example J: A computing device according to Example I, wherein the partitioning is further optimized to minimize data communication between the worker computing devices.

[0086] Example K: A computing device according to any of Examples IJ, wherein the partitioning is further optimized so that each of the multiple computing devices performs approximately the same amount of processing during training of the DNN model.

[0087] Example L: A computing device according to any of Examples IK, wherein at least one of the multiple stages is configured for model parallel processing, and wherein at least one of the multiple stages is configured for data parallel processing.

[0088] Example M: ​​A computing device according to any one of Examples IL, wherein the at least one computer storage medium has additional computer executable instructions stored thereon for: generating a profile of the deep neural network (DNN) model; and based on the profile, dividing the layers of the DNN module into the multiple stages.

[0089] Example N: A computing device according to any of Examples IM, wherein the profile of the DNN module is generated by training the DNN model on a subset of the multiple worker computing devices for a predetermined time period using a subset of the DNN training data.

[0090] Example O: A computer storage medium having computer executable instructions stored thereon, which, when executed by one or more processors of a computing device, will cause the computing device to: divide the layers of a deep neural network (DNN) model into multiple stages, wherein each of the multiple stages includes one or more of the layers of the DNN model, and wherein the division is optimized to minimize the time to train the DNN model; and assign at least one of the multiple stages to each worker computing device of a plurality of worker computing devices, the computing devices being configured to: process batches of the DNN training data to train the DNN model by alternating between forward processing of the batches of the DNN training data and backward processing of the batches of the DNN training data.

[0091] Example P: A computer storage medium according to Example O, wherein the partitioning is further optimized to minimize data communication between the worker computing devices.

[0092] Example Q: The computer storage medium of any of Examples OP, wherein the partitioning is further optimized so that each of the plurality of computing devices performs approximately the same amount of processing during training of the DNN model.

[0093] Example R: A computer storage medium according to any one of Examples OQ, wherein the partitioning of the layers of the DNN model further comprises: calculating the optimal number of batches of the DNN training data to be provided to the multiple computing devices so as to maximize the processing efficiency of the multiple computing devices.

[0094] Example S: A computer storage medium according to any one of Examples OR, wherein the computer storage medium has additional computer executable instructions stored thereon for: generating a profile of the DNN model; and based on the profile, dividing the layers of the DNN module into the multiple stages.

[0095] Example T: A computer storage medium according to any of the example OS, wherein the profile of the DNN module is generated by training the DNN model on a subset of the plurality of worker computing devices for a predetermined period of time using a subset of the DNN training data.

[0096] Based on the foregoing, it should be understood that techniques for high-performance pipeline parallel DNN training have been disclosed herein. Although the subject matter presented herein has been described in a language specific to computer structural features, methods and transformation actions, specific computing machines, and computer-readable media, it should be understood that the subject matter set forth in the appended claims is not necessarily limited to the specific features, actions, or media described herein. Instead, specific features, actions, and media are disclosed as example forms of implementing the claimed subject matter.

[0097] The above subject matter is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example configurations and applications shown and described, and without departing from the scope of the present disclosure set forth in the appended claims.

Claims

1. A computer-implemented method for training a deep neural network (DNN) model using a processing pipeline, the method comprising: generating a profile of the DNN model by executing the DNN model on a plurality of worker computing devices using a subset of DNN training data for a predetermined time period, the DNN training data comprising a plurality of mini-batches and the DNN model comprising a plurality of layers; dividing the plurality of layers of the DNN model into a plurality of stages based on the profile, wherein each stage of the plurality of stages includes one or more layers of the plurality of layers of the DNN model, and wherein the division minimizes a time to train the DNN model; assigning the plurality of stages to the plurality of worker computing devices based on the partitioning; in a startup state, admitting a subset of the plurality of mini-batches of the DNN training data to the processing pipeline, wherein the subset of the plurality of mini-batches comprises a number of mini-batches such that a single worker computing device of the plurality of worker computing devices is not idle during a steady state; and The DNN model is trained using a one-forward, one-backward scheduling strategy, whereby in the stable state: a first worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a first distribution phase, and a second worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a second distribution phase, the second distribution phase being different from the first distribution phase, the first of the plurality of worker computing devices alternately performing forward processing of a first mini-batch in the subset of the plurality of mini-batches of training data and backward processing of the first mini-batch in the subset of the plurality of mini-batches of training data, and The second one of the plurality of worker computing devices alternately performs forward processing of a second one of the subset of the plurality of mini-batches of training data and backward processing of a third one of the subset of the plurality of mini-batches of training data, the third one of the subset of the plurality of mini-batches of training data being different from the second one of the subset of the plurality of mini-batches of training data.

2. The computer-implemented method of claim 1, wherein the partitioning also minimizes data communication between the plurality of worker computing devices.

3. A computer-implemented method according to claim 1, wherein the partitioning causes each of the plurality of worker computing devices to perform the same amount of processing during training of the DNN model.

4. A computer-implemented method according to claim 1, wherein at least one of the multiple worker computing devices is configured for model parallel processing, whereby the DNN model is copied to the at least one of the multiple worker computing devices for training.

5. The computer-implemented method of claim 1 , wherein several of the plurality of worker computing devices are configured for data parallel processing, whereby the several of the plurality of worker computing devices are assigned at a stage to process the one or more of the plurality of layers of the DNN, and each of the several of the plurality of worker computing devices processes a different mini-batch of the plurality of mini-batches of training data during training.

6. The computer-implemented method of claim 1 , wherein at least one of the plurality of worker computing devices is configured for model parallel processing, whereby the DNN model is copied to the at least one of the plurality of worker computing devices for training, and wherein several of the plurality of worker computing devices are configured for data parallel processing, whereby the several of the plurality of worker computing devices are configured at a stage to process one or more of the plurality of layers of the DNN, and each of the several of the plurality of worker computing devices processes a different mini-batch of the plurality of mini-batches of training data during training.

7. A computing device for training a deep neural network (DNN) model using a processing pipeline, the computing device comprising: one or more processors; as well as at least one computer storage medium having computer executable instructions stored thereon, which when executed by the one or more processors will cause the computing device to: generating a profile of the DNN model by executing the DNN model on a plurality of worker computing devices using a subset of DNN training data for a predetermined time period, the DNN training data comprising a plurality of mini-batches and the DNN model comprising a plurality of layers; dividing the plurality of layers of the DNN model into a plurality of stages, wherein each stage of the plurality of stages includes one or more layers of the plurality of layers of the DNN model, and wherein the division minimizes the time to train the DNN model; assigning the plurality of stages to the plurality of worker computing devices based on the partitioning; in a startup state, admitting a subset of the plurality of mini-batches of the DNN training data to the processing pipeline, wherein the subset of the plurality of mini-batches comprises a number of mini-batches such that a single worker computing device of the plurality of worker computing devices is not idle during a steady state; and The DNN model is trained using a one-forward, one-backward scheduling strategy, whereby in the stable state: a first worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a first distribution phase, and a second worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a second distribution phase, the second distribution phase being different from the first distribution phase, the first of the plurality of worker computing devices alternately performing forward processing of a first mini-batch in the subset of the plurality of mini-batches of training data and backward processing of the first mini-batch in the subset of the plurality of mini-batches of training data, and The second one of the plurality of worker computing devices alternately performs forward processing of a second one of the subset of the plurality of mini-batches of training data and backward processing of a third one of the subset of the plurality of mini-batches of training data, the third one of the subset of the plurality of mini-batches of training data being different from the second one of the subset of the plurality of mini-batches of training data.

8. The computing device of claim 7, wherein the partitioning further minimizes data communications between the plurality of worker computing devices.

9. A computing device according to claim 7, wherein the partitioning causes each of the plurality of worker computing devices to perform the same amount of processing during training of the DNN model.

10. The computing device of claim 7, wherein at least one of the plurality of worker computing devices performs model parallel processing, whereby the DNN model is copied to at least one of the plurality of worker computing devices for training, and wherein several of the plurality of worker computing devices perform data parallel processing, whereby the several of the plurality of worker computing devices processes one or more of the plurality of layers of the DNN in a stage, and each of the several of the plurality of worker computing devices processes a different mini-batch of the plurality of mini-batches of the DNN training data during training.

11. A computer storage medium having stored thereon computer executable instructions which, when executed by one or more processors of a computing device, cause the computing device to: generating a profile of the DNN model by executing the DNN model on a plurality of worker computing devices using a subset of DNN training data for a predetermined time period, the DNN training data comprising a plurality of mini-batches and the DNN model comprising a plurality of layers; dividing the plurality of layers of the DNN model into a plurality of stages, wherein each stage of the plurality of stages includes one or more layers of the plurality of layers of the DNN model, and wherein the division minimizes the time to train the DNN model; assigning the plurality of stages to the plurality of worker computing devices based on the partitioning; In a startup state, admitting a subset of the plurality of mini-batches of the DNN training data to a processing pipeline, wherein the subset of the plurality of mini-batches comprises a number of mini-batches such that a single worker computing device of the plurality of worker computing devices is not idle during a steady state; and The DNN model is trained using a one-forward, one-backward scheduling strategy, whereby in the stable state: a first worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a first distribution phase, and a second worker computing device among the plurality of worker computing devices performs forward processing and backward processing on the plurality of layers of the DNN model in a second distribution phase, the second distribution phase being different from the first distribution phase, the first of the plurality of worker computing devices alternately performing forward processing of a first mini-batch in the subset of the plurality of mini-batches of training data and backward processing of the first mini-batch in the subset of the plurality of mini-batches of training data, and The second one of the plurality of worker computing devices alternately performs forward processing of a second one of the subset of the plurality of mini-batches of training data and backward processing of a third one of the subset of the plurality of mini-batches of training data, the third one of the subset of the plurality of mini-batches of training data being different from the second one of the subset of the plurality of mini-batches of training data.

12. The computer storage medium of claim 11, wherein the partitioning further minimizes data communications between the plurality of worker computing devices.

13. The computer storage medium of claim 11, wherein the partitioning causes each of the plurality of worker computing devices to perform the same amount of processing during training of the DNN model.

Images