Pipelined neural network processing with persistent and asynchronous updates

Abstract

Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. The method for processing a neural network comprising L layers (where L is an integer greater than 2) includes: partitioning the network into L layers among a set of computational resources, which are configured to process forward and backward passes associated with each of the L layers. The method further includes: initiating the processing of forward and backward passes using the set of computational resources. The method also includes: after completing a first set of forward and backward passes associated with the first layer of the L layers, initiating an update of the parameters associated with the first layer when gradients are available to update the parameters associated with the first layer, without waiting for the computation of gradients associated with any of the remaining layers of the L layers.

Description

Background Technology

[0001] Neural network technology is used to perform complex tasks such as reading comprehension, language translation, image recognition, or speech recognition. Machine learning services (such as those based on Natural Language Processing (NLP), Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) neural networks, or Gated Recurrent Units (GRUs)) have already been deployed to perform these complex tasks. While these types of neural networks have been deployed, further improvements are needed in the underlying architecture to perform these complex tasks. Summary of the Invention

[0002] In one example, this disclosure relates to a method for processing a neural network comprising L layers, where L is an integer greater than two. The method may include partitioning the L layers among a set of computational resources configured to process forward and backward passes associated with each of the L layers. The method may also include using the set of computational resources to initiate the processing of the forward and backward passes. The method may further include, after completing a first set of forward and backward passes associated with a first layer of the L layers, initiating an update of the parameters associated with the first layer when gradients are available to update the parameters associated with the first layer, without waiting for the computation of gradients associated with any of the remaining layers of the L layers.

[0003] In another example, this disclosure relates to a system for processing a neural network comprising L layers, where L is an integer greater than two. The system may include a set of computational resources configured to handle forward and backward propagations associated with each of the L layers, wherein the L layers are partitioned within the set of computational resources, and each computational resource is configured to asynchronously initiate an update of the weights associated with that layer when a gradient pair updates the weights associated with that layer, without necessarily receiving gradients associated with any of the other layers in the L.

[0004] In yet another example, this disclosure relates to a method for processing a neural network comprising L layers, where L is an integer greater than two. The method may include, as part of a first stage of a pipeline, using a first computational resource to initiate processing of a first forward pass associated with a first layer of the L layers. The method may also include, as part of a second stage of a pipeline: (1) using the first computational resource to initiate processing of a first forward pass associated with a second layer of the L layers, and (2) using a second computational resource to initiate processing of a second forward pass associated with the first layer of the L layers. The method may also include, as part of a third stage of a pipeline: (1) using the first computational resource to initiate processing of a first forward pass associated with a third layer of the L layers, (2) using the second computational resource to initiate processing of a second forward pass associated with the second layer of the L layers, and (3) using a third computational resource to initiate processing of a third forward pass associated with the first layer of the L layers. The method may further include, as part of a fourth stage of the pipeline: (1) using a first computational resource, initiating a first forward pass associated with the fourth layer of the L layers; (2) using a second computational resource, initiating a second forward pass associated with the second layer of the L layers; and (3) using a third computational resource, initiating a backward pass associated with the first layer of the L layers. The method may further include, as part of a fifth stage of the pipeline: (1) using a first computational resource, initiating a first forward pass associated with the fifth layer of the L layers; (2) using a second computational resource, initiating a second backward pass associated with the first layer of the L layers; and (3) using a third computational resource, updating the weights associated with the first layer of the L layers when the gradients associated with the weights associated with the first layer of the L layers are accumulated.

[0005] This summary provides a simplified overview of some concepts, which are further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0006] This disclosure is illustrated by way of example and is not limited to the accompanying drawings, wherein like reference numerals indicate like elements. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale.

[0007] Figure 1 An example architecture of the bidirectional encoder representation (BERT) implementation from the transformer is shown, which can be trained using pipelined model parallelism;

[0008] Figure 2 A block diagram of a converter based on an example is shown;

[0009] Figure 3 The system shown is based on an example, which has a converter partitioned across multiple computing resources;

[0010] Figure 4 An implementation of computational resources configured to process a neural network model is shown, based on an example.

[0011] Figure 5 The diagram illustrates gradient accumulation for training a neural network model, based on an example.

[0012] Figure 6 The diagram illustrates a pipeline for continuous and asynchronous weight updates during the training of a neural network model, based on an example.

[0013] Figure 7 A flowchart illustrating a method for initiating weight updates during the training of a neural network model, based on an example, is shown; and

[0014] Figure 8A and Figure 8B A flowchart is shown, based on an example, of another method for updating weights during the training of a neural network model. Detailed Implementation

[0015] The examples disclosed in this document relate to neural network processing with continuous asynchronous updates. Some examples involve neural network processing where layers, sublayers, or even smaller parts of the neural network are partitioned to achieve parallelism. Some examples also involve model parallelism. As an example, in model parallelism, different machines in the system can be responsible for computation in different parts of a single network—for example, each layer, sublayer, or even smaller part of the neural network can be assigned to different machines in the system. Therefore, as part of model parallelism, the neural network model can be split across different machines (e.g., CPUs, GPUs, IPUs, FPGAs, or other types of machines), but each machine can use the same data. As an example, model parallelism can be achieved by splitting weights across machines.

[0016] Typically, neural networks consist of many numerical operations that need to be efficiently partitioned across computational resources. Many approaches exist for this partitioning, each highly specific to the model and architecture. One approach is to split neural network layers across sequential computational resources to achieve pipelined parallelism. Pipeline parallelism can be very efficient for processing feedforward networks, but it becomes much more complex when feedback and weight updates are applied. One technique for neural networks is to update weights based on mini-batches. However, this approach is inefficient for pipelined models because it requires flushing content from the pipeline before proceeding. The flushing operation requires the pipeline to be emptied at the end of a mini-batch and refilled at the beginning of the next mini-batch, leading to inefficiency.

[0017] Some examples in this disclosure involve continuously and asynchronously updating weights during the training of a neural network model. Weight updates are applied during backpropagation. As an example, the gradient of the loss function with respect to the weights in the network (or a portion of the network) is computed. The gradient is then fed to an optimization method that uses the gradient to update the weights to minimize the loss function. The goal of backpropagation is to update each (or at least some) of the weights in the network such that the weights make the actual output closer to the target output, thereby minimizing the error of each output neuron and the network as a whole.

[0018] As part of model parallelism, the network can be divided into different neural computation units, each of which can be processed independently by computational resources. Each computational resource can be implemented as a separate device (e.g., GPU, TPU, or IPU), or the computational resources can share a device. Thus, in one example, each device (e.g., IPU) may include two computational resources that can handle two different computational units (e.g., two different layers) associated with the neural network model.

[0019] The parallelism of pipelined models is illustrated using an example from the Bidirectional Encoder Representation (BERT) model derived from the transformer. The BERT model can be used for sentence word completion and other natural language processing (NLP) related problems. Figure 1An example architecture of a BERT implementation 100 that can be trained using pipelined model parallelism is shown. The BERT implementation 100 can be used to learn contextual relationships between words in text. The BERT implementation 100 may include an input stage comprising an embedding layer 104 and a post-processing layer 106. The BERT implementation 100 may also include a transformer 110, comprising transformers 0 and 112, 1114, and L and 116. The BERT implementation 100 may also include a decoding / loss layer 120. The embedding layer 104 can convert each input lexical (e.g., words in a sequence) into a vector representation. The vector size can be a 768-dimensional vector or a 1024-dimensional vector. The post-processing layer 106 can perform additional operations, including positional embedding, sentence embedding, and lexical masking. Positional embedding can be used to indicate the position of lexical terms within a sequence. Sentence embedding can be used to map sentences to vectors. Lexical masking can include replacing a certain percentage of words in each sequence with masked lexical terms. The transformers will be described in detail later. Decoding / loss layer 120 can generate predictions based on training.

[0020] Figure 2 A block diagram of a converter 200 according to an example is shown. Converter 200 can be used with... Figure 1 Transformer 200 corresponds to any one of Transformer 0112, Transformer 1114, and Transformer L116. Transformer 200 can use self-attention to transform the input sequence into an output sequence. Transformer 200 can be configured to process a batch of layers with a batch size of B. The sequence (S) length can refer to the number of words being processed in each sequence. Transformer 200 can be configured to have 12 or 24 hidden (H) layers. The vector size (also known as the hidden (H) size) can specify whether each word is transformed into a 768-dimensional vector or a 1024-dimensional vector. The intermediate (I) size can be a 3072-dimensional vector or a 4096-dimensional vector.

[0021] Continue to refer to Figure 2 Transformer 200 may include fully connected network (FCN) layers, including FCN (query) 202, FCN (key) 204, and FCN (value) 206. Self-attention can map the query and a set of key-value pairs to the output. The attention function can be implemented using SoftMax block 210. Transformer 200 may also include additional fully connected network (FCN) layers, including FCN (projection) 214, FCN (intermediate) 218, and FCN (output) 220. FCN (intermediate) 218 ​​may involve intermediate representations of the same terms as those represented in the output (FCN (output) 220). Fully connected layers are a major factor in the complexity of the operations associated with the transformer.

[0022] Figure 3A system 300 according to an example is shown, in which converters are allocated to computing resources. In this example, computing resources can be any combination of processing cores and memory. The pipeline can be constructed in such a way that each computing resource in the pipeline (e.g., a processor or a combination of processors and memory) handles approximately the same amount of operations. This can help ensure efficient pipeline utilization. In system 300, the embedding layer 312 and post-processing layer 314 of the BERT model can be configured to be processed by computing resource P0 310. Converter 0 322 can be configured to be processed by computing resource P1 320. Converter 1 332 can be configured to be processed by computing resource P2 330. Converter 5 342 can be configured to be processed by computing resource P5 340. Converter 6 352 can be configured to be processed by computing resource P6 350. Converter 7 362 can be configured to be processed by computing resource P7 360. Converter 11 372 can be configured to be processed by computing resource P12 370. The decoding / loss layer 382 of the BERT model can be configured to be processed by computational resource P13 380. Although Figure 3 Each converter is shown as being partitioned into individual computing resources, but converters can share computing resources. Therefore, as an example, each computing resource can be partitioned into two converters. Each computing resource can be implemented using any number of graphics processing units (GPUs), computer processing units (CPUs), memory (e.g., SRAM or other types of memory), or field-programmable gate arrays (FPGAs). Application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, and general-purpose array logic (GAL) devices can also be used to implement computing resources. Furthermore, although... Figure 3 The partitioning of the BERT model is described, but any serializable neural network model can be partitioned and processed as long as it uses gradient descent with backpropagation.

[0023] Figure 4 An implementation of computing resource 400 configured to process a neural network model is shown according to an example. In this example, computing resource 400 can be used as... Figure 3Each computing resource in the computing resources described in this section corresponds to a specific computing resource. Computing resource 400 may include two sets of processing cores and memory, coupled via a switching structure 440. The first set 410 may include processing cores P412, P414, P416, P418, P420, P422, P424, and P426. The processing cores may be arranged in columns and may access local memory (e.g., SRAM or BRAM). The first set 410 may include local memories M430, M432, M434, and M436. The second set 450 may include processing cores P452, P454, P456, P458, P460, P462, P464, and P466. The processing cores may be arranged in columns and may access local memory (e.g., SRAM or BRAM). The second set 450 may include local memories M 470, M 472, M 474, and M 476. These two sets can communicate via a switching structure 440. As an example, a processing core can communicate with other processing cores via message passing or other techniques. Although Figure 4 A certain number of components of computing resources 400 arranged in a certain way are shown, but there may be more or fewer components arranged in different ways. Processing cores can be implemented using FPGAs, CPUs, GPUs, or other processing arrangements. Image files can be used to configure or reconfigure the FPGA. Image files or similar files or programs can be delivered from the host CPU via network links or local links (e.g., PCIe). Information included in the image file can be used to program the logic blocks of the FPGA and reconfigurable interconnects to achieve the desired functionality. The desired functionality can be implemented to support any service that can be provided via a combination of computing, networking, and storage resources, such as via data centers or other infrastructure used for service delivery.

[0024] The trained model can be deployed in a cloud computing environment. Cloud computing can refer to a model that enables on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be adopted in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources can be rapidly provisioned via virtualization and published with low management effort or service provider interaction, and then scaled accordingly. Cloud computing models can include various characteristics such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so on. Cloud computing models can be used to expose various service models, such as Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Different deployment models (such as private cloud, community cloud, public cloud, hybrid cloud, etc.) can also be used to deploy cloud computing models.

[0025] Besides the BERT model, machine learning services, such as those based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can also be implemented using the examples described in this disclosure. In one example, service-related content or other information (such as words, sentences, images, videos, or other such content / information) can be converted into vector representations. These vector representations can correspond to techniques such as RNNs, CNNs, LSTMs, or GRUs. In one example, the neural network model can include multiple layers, and each layer can be encoded as a matrix or vector of weights, represented as coefficients or constants obtained through offline training of the neural network. Programmable hardware logic blocks in the nodes can process the matrices or vectors to perform various operations, including multiplication, addition, and other operations on input vectors representing the encoded information related to the service. In one example, the matrices or vectors of weights can be partitioned and fixed across multiple nodes using techniques such as graph partitioning. As part of this process, a large neural network can be converted into an intermediate representation (e.g., a graph), which can then be divided into smaller representations (e.g., subgraphs), and each matrix in the weight matrix corresponding to each subgraph can be fixed to the on-chip memory of the nodes. In one example, the model can be converted into fixed-size matrices and vectors. This allows processor resources to operate on the fixed-size matrices and vectors in parallel.

[0026] Taking LSTM as an example, an LSTM network can include a sequence of repeating RNN layers or other types of layers. Each layer of an LSTM network can consume input (e.g., the state from a layer in a previous time step) at a given time step and can produce a new set of outputs or states. When using LSTM, a single content chunk can be encoded into a single vector or multiple vectors. As an example, a word or combination of words (e.g., a phrase, sentence, or paragraph) can be encoded into a single vector. Each chunk can be encoded into a separate layer of the LSTM network (e.g., at a specific time step). An LSTM layer can be described using a set of equations, such as the following:

[0027] i t =σ(W xi xt+W hi h t-1 +W ci c t-1 +b i

[0028] f t =σ(W xf x t +W hf h t-1 +W cf c t-1 +b f )

[0029] c t =f t c t-1 i t tanh(W xc x t +W hc h t-1 +b c )

[0030] o t =σ(W xo x t +W h oh t-1 +W co c t +b o )

[0031] h t =o t tanh(c t )

[0032] In this example, within each LSTM layer, a combination of vector operations (e.g., dot product, inner product, or vector addition) and nonlinear functions (e.g., sigmoid, hyperbola, and tangent) can be used to process the input and hidden states. In some cases, most of the computationally intensive operations may come from the dot product, which can be implemented using dense matrix vectors and matrix-matrix multiplication routines. In one example, the processing of vector operations and nonlinear functions can be performed in parallel.

[0033] Figure 5 Figure 500 illustrates gradient accumulation for training a neural network model according to an example. One technique for updating weights in a neural network is batch stochastic gradient descent, which involves running multiple samples through the network before updating the weights. Different methods exist to handle larger batch sizes, but they all require significant memory. In this example, gradient accumulation can be used to reduce memory requirements. As part of gradient accumulation, after each backpropagation, the computed gradient (e.g., dW) can be added to the previously accumulated gradient (e.g., W). The following equation can be used:

[0034]

[0035] Gradient accumulation can be performed in hardware (e.g., by implementing the equations above), and the hardware can be included in the computational resources described above. The hardware implementation may include adder 510 and block 512, which can compute a block average. The block average can be reset at time K or at another time. Alternatively, the block average can be reset after a specified number of backpropagations. The use of gradient accumulation can allow computational resources (e.g., Figure 3 The processor shown maintains the changes to the on-chip weights. After a weight update, the on-chip accumulated gradients can be flushed, and the accumulation process can restart. Although Figure 5 One way to accumulate gradients is shown, but other methods can also be used. Furthermore, the accumulated gradients can be used to update parameters other than the weights. Such parameters include any adaptive parameters associated with the neural network model, including bias and batch normalization parameters.

[0036] Figure 6A pipeline diagram 600 illustrating continuous and asynchronous weight updates during the training of a neural network model, based on an example, is shown. This pipeline diagram 600 illustrates the execution of forward passes (indicated by the letter M), backward passes (indicated by the letter B), and weight updates (indicated by the letter W) for each pipeline stage. In this example, three processors, P1, P2, and P3, are scheduled to perform these operations. As previously described, using model parallelism, the neural network model can be partitioned across computational resources (e.g., any number of processors). In pipeline stage 602, processor P1 initiates the processing of a first forward pass (M1) for the first layer of the neural network model being processed. In one example, this forward pass may correspond to one of the transformer layers described previously. Processors P2 and P3 are both idle in this pipeline stage. In pipeline stage 604, after completing the first forward pass (M1) for the first layer of the neural network model, processor P1 initiates the processing of a first forward pass (M2) for the second layer of the neural network model. Furthermore, in pipeline stage 604, for the first layer of the neural network model, processor P2 initiates the processing of the second forward pass (M1). At this pipeline stage, processor P3 remains idle. In pipeline stage 606, after completing the first forward pass (M2) for the second layer, processor P1 initiates the processing of the first forward pass (M3) for the third layer of the neural network model. Furthermore, in pipeline stage 606, for the second layer of the neural network model, processor P2 initiates the processing of the second forward pass (M2). Furthermore, in pipeline stage 606, for the first layer of the neural network model, processor P3 initiates the processing of the third forward pass (M1). This example assumes that the model (or a subset of the model (e.g., mini-batch)) requires three forward passes per layer. Therefore, after completing the third forward pass (M1) for the first layer of the neural network model, backpropagation can begin.

[0037] Continue to refer to Figure 6In stage 608, for the fourth layer of the neural network model, processor P1 initiates the first forward pass (M4). Furthermore, in pipeline stage 608, for the third layer of the neural network model, processor P2 initiates the second forward pass (M3). Furthermore, in pipeline stage 608, for the first layer of the neural network model, processor P3 initiates the first backward pass (B1). In stage 610, for the fifth layer of the neural network model, processor P1 initiates the first forward pass (M5). Furthermore, in pipeline stage 610, for the first layer of the neural network model, processor P2 initiates the second backward pass (B1). Furthermore, in pipeline stage 610, for the first layer of the neural network model, processor P3 initiates the weight (W1) update. As shown in pipeline diagram 600, weight updates are initiated asynchronously and continuously. Accumulated gradients can be used to perform weight updates. In stage 612, for the first layer of the neural network model, processor P1 initiates the third backward pass (B1). Furthermore, in pipeline stage 612, for the first layer of the neural network model, processor P2 initiates an update of the weights (W1). Also in pipeline stage 612, for the third layer of the neural network model, processor P3 initiates a third forward pass. Forward passes, backward passes, and weight updates are processed as shown in pipeline diagram 600. As an example, in pipeline stage 620, processor P1 initiates a weight update for the third layer based on the third backward pass associated with the third layer. Furthermore, in this stage, processor P2 initiates a first forward pass associated with the sixth layer of the neural network model, and processor P3 initiates a first backward pass associated with the fourth layer of the neural network model. Each computation performed by any of the processors may take less or more time than other computations performed in the pipeline. However, in general, as previously stated, the neural network model is processed in a way that divides the workload of processors (or other computing resources) into similar amounts at each pipeline stage.

[0038] Figure 7 A flowchart 700 is shown illustrating a method for initiating weight updates during the training of a neural network model, according to an example. Step 710 may include partitioning a set of computational resources into L layers, which are configured to handle the forward and backward propagations associated with each of the L layers. Machine language frameworks configured to be used (such as Tensorflow, Apache MXNet, and...) can be employed. The Cognitive Toolkit (CNTK) uses code to partition the model. As an example, this step could include partitioning a BERT model, such as... Figure 3As shown in the diagram. Therefore, the individual layers of a BERT model can be assigned to be processed using different processors (or similar computing resources). In this way, the various parameters associated with a layer can be processed in parallel. Thus, for an example BERT model with 12 layers, there might be 110 million parameters. Each computing resource can process approximately 9 million parameters per layer. For a larger BERT model with 24 layers, there might be 340 million parameters. If spanning 8 computing resources (e.g., similar to...) Figure 4 If the BERT model is partitioned across 400 computing resources, each computing resource can handle approximately 64 million parameters. In another example, if the BERT model is partitioned across 16 computing resources, each computing resource can handle approximately 32 million parameters. As part of this step, other types of neural network models can also be partitioned for processing.

[0039] Step 720 may include initiating forward and backward pass processing using a set of computing resources. As part of this step, a scheduler (or some other software) may initiate forward and backward pass processing. In this example, the processing may be similar to... Figure 6 The described pipeline-like process occurs.

[0040] Step 730 may include, after completing the first set of forward and backward propagations associated with the first of the L layers, initiating an update of the parameters associated with the first layer when the gradient pairs are available to update the parameters associated with the first layer, without waiting for the gradients associated with any of the remaining L layers to be computed. In this example, the parameters may include weights. As part of this step, similar to Figure 6 As shown, weight updates associated with the first layer can be initiated without waiting for the gradients associated with the remaining layers to be computed. Weight updates can be based on accumulated gradients (e.g., as per the gradient descent). Figure 5 (As explained). Although Figure 7 It describes a specific number of steps to be performed in a particular order, but additional or fewer steps may be performed in a different order.

[0041] Figure 8A and Figure 8B A flowchart 800 illustrates another method for updating weights during the training of a neural network model, according to one example. Step 810 may include: as part of the first stage of the pipeline, using first computational resources to initiate processing of a first forward pass associated with a first layer of L layers. In this example, this step may be combined with... Figure 6This corresponds to pipeline state 602. Therefore, as part of this step, processor P1 can initiate a first forward pass (M1) for the first layer of the neural network model being processed. In one example, this forward pass can correspond to one of the transformer layers described earlier.

[0042] Step 820 may include, as part of a second stage of the pipeline, (1) using first computing resources, initiating a first forward pass associated with the second layer of the L layers, and (2) using second computing resources, initiating a second forward pass associated with the first layer of the L layers. In this example, this step may be combined with... Figure 6 The pipeline stage 604 corresponds to this. Therefore, as part of this step, for the second layer of the neural network model, processor P1 can initiate the first forward pass (M2) process, and for the first layer of the neural network model, processor P2 can initiate the second forward pass (M1) process.

[0043] Step 830 may include, as part of a third stage of the pipeline, (1) using a first computing resource, initiating a first forward pass associated with the third layer of the L layers, (2) using a second computing resource, initiating a second forward pass associated with the second layer of the L layers, and (3) using a third computing resource, initiating a third forward pass associated with the first layer of the L layers. In this example, this step may correspond to Figure 6 The pipeline stage 606. Therefore, as part of this step, for the third layer of the neural network model, processor P1 can initiate the first forward pass (M3) process; for the second layer of the neural network model, processor P2 can initiate the second forward pass (M2) process; and for the first layer of the neural network model, processor P3 can initiate the third forward pass (M1) process.

[0044] Step 840 may include, as part of a fourth stage of the pipeline, (1) using a first computing resource, initiating a first forward pass associated with the fourth layer of the L layers, (2) using a second computing resource, initiating a second forward pass associated with the second layer of the L layers, and (3) using a third computing resource, initiating a backward pass associated with the first layer of the L layers. In this example, this step may be combined with... Figure 6 The pipeline stage 608 corresponds to this. Therefore, as part of this step, for the fourth layer of the neural network model, processor P1 can initiate the first forward pass (M4) process; for the third layer of the neural network model, processor P2 can initiate the second forward pass (M3) process; and for the first layer of the neural network model, processor P3 can initiate the first backward pass (B1) process.

[0045] Step 850 may include, as part of the fifth stage of the pipeline, (1) using a first computational resource, initiating a first forward pass associated with the fifth layer of the L layers, (2) using a second computational resource, initiating a second backward pass associated with the first layer of the L layers, and (3) using a third computational resource, updating the weights associated with the first layer of the L layers when the gradients associated with the weights associated with the first layer of the L layers are accumulated. In this example, this step may be combined with... Figure 6 The pipeline stage 608 corresponds to this. Therefore, as part of this step, for the fifth layer of the neural network model, processor P1 can initiate the first forward pass (M5) processing; for the first layer of the neural network model, processor P2 can initiate the second backward pass (B1) processing; and for the first layer of the neural network model, processor P3 can initiate the weight (W1) update. Although Figure 8A and Figure 8B It describes a specific number of steps to be performed in a particular order, but additional or fewer steps may be performed in a different order.

[0046] In summary, this disclosure relates to a method for processing a neural network comprising L layers, where L is an integer greater than two. The method may include partitioning the L layers among a set of computational resources configured to process forward and backward passes associated with each of the L layers. The method may also include using the set of computational resources to initiate the processing of the forward and backward passes. The method may further include, after completing a first set of forward and backward passes associated with a first layer among the L layers, initiating an update of the parameters associated with the first layer when gradients are available to update the parameters associated with the first layer, without waiting for the computation of gradients associated with any of the remaining layers among the L layers.

[0047] Dividing a set of computing resources into L layers can include: distributing the parameters associated with the neural network in a substantially equal manner across the set of computing resources. The neural network can be configured to perform natural language processing. The neural network can also include an input stage coupled to provide input to the L layers, and an output stage coupled to receive output from the L layers.

[0048] The parameters may include weights, and the method may further include: initiating an update of the weights associated with the second layer when the gradient update for the weights associated with the second layer of the L layers is available, wherein initiating the update of the weights associated with the second layer of the L layers is asynchronous with initiating the update of the weights associated with the first layer of the L layers. The set of computational resources may include a first, second, and third computational resource arranged sequentially such that forward and backward propagation are processed in a pipelined manner. The method may further include accumulating gradients over a specified number of backward propagations.

[0049] In another example, this disclosure relates to a system for processing a neural network comprising L layers, where L is an integer greater than two. The system may include a set of computational resources configured to handle forward and backward propagations associated with each of the L layers, wherein the L layers are partitioned within the set of computational resources, and each of the computational resources is configured to asynchronously initiate an update of the weights associated with a layer when a gradient pair updates the weights associated with that layer, without necessarily receiving gradients associated with any of the other layers in the L.

[0050] The weights associated with a neural network can be distributed substantially equally among a set of computational resources. The neural network can also be configured to perform natural language processing. It can further include an input stage coupled to provide input to L layers, and an output stage coupled to receive outputs from the L layers.

[0051] A set of computational resources may include a first, second, and third computational resource arranged sequentially, such that forward and backward passes are processed in a pipelined manner. A subset of the set of computational resources may also include an accumulator configured to accumulate gradients over a specified number of backward passes.

[0052] In yet another example, this disclosure relates to a method for processing a neural network comprising L layers, where L is an integer greater than two. The method may include: as part of a first stage of a pipeline, using a first computational resource, initiating processing of a first forward pass associated with a first layer of the L layers. The method may also include: as part of a second stage of a pipeline, (1) using the first computational resource, initiating processing of a first forward pass associated with a second layer of the L layers, and (2) using a second computational resource, initiating processing of a second forward pass associated with the first layer of the L layers. The method may also include: as part of a third stage of a pipeline, (1) using the first computational resource, initiating processing of a first forward pass associated with a third layer of the L layers, (2) using the second computational resource, initiating processing of a second forward pass associated with the second layer of the L layers, and (3) using a third computational resource, initiating processing of a third forward pass associated with the first layer of the L layers. The method may further include, as part of a fourth stage of the pipeline, (1) using a first computational resource, initiating a first forward pass associated with the fourth layer of the L layers, (2) using a second computational resource, initiating a second forward pass associated with the second layer of the L layers, and (3) using a third computational resource, initiating a backward pass associated with the first layer of the L layers. The method may further include, as part of a fifth stage of the pipeline, (1) using a first computational resource, initiating a first forward pass associated with the fifth layer of the L layers, (2) using a second computational resource, initiating a second backward pass associated with the first layer of the L layers, and (3) using a third computational resource, updating the weights associated with the first layer of the L layers when the gradients associated with the weights associated with the first layer of the L layers are accumulated.

[0053] Each of the first, second, third, fourth, and fifth layers can correspond to a transformer. The method may also include dividing at least a first, second, and third computational resource into L layers that are substantially equally distributed. The neural network can also be configured to perform natural language processing. The neural network may further include an input stage coupled to provide input to the L layers, and an output stage coupled to receive outputs from the L layers.

[0054] The method may also include accumulating gradients over a specified number of backpropagations. The method may also include accumulating gradients over a specified number of loops associated with the pipeline.

[0055] It should be understood that the methods, modules, and components described herein are merely exemplary. Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), Systems-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract but still definitive sense, any arrangement of components implementing the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two components combined herein to achieve a particular functionality can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered “operably connected” or “coupled” with each other to achieve the desired functionality.

[0056] The functionality associated with some of the examples described in this disclosure may also include instructions stored in a non-transient medium. As used herein, the term "non-transient medium" refers to any medium that stores data and / or instructions that cause the machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or magnetic tapes, optical disks or optical tapes, flash memory, EPROM, NVRAM, PRAM, or other such media, or networking versions of such media. Volatile media include, for example, dynamic memory, such as DRAM, SRAM, cache, or other such media. Non-transient media are distinct from transmission media but may be used in conjunction with transmission media. Transmission media are used to transmit data and / or instructions to or from the machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media (e.g., radio waves).

[0057] Furthermore, those skilled in the art will recognize that the boundaries between the functions of the above operations are merely illustrative. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed across other operations. Additionally, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be varied in various other embodiments.

[0058] While specific examples are provided in this disclosure, various modifications and variations may be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure. Any benefits, advantages, or solutions to problems described herein with respect to specific examples are not intended to be construed as key, essential, or necessary features or elements of any or all claims.

[0059] Furthermore, the term "a" or "an" as used herein is defined as one or more. Moreover, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be construed as implying that another claim element introduced by the indefinite article "a" or "an" limits any particular claim containing such an element to an invention containing only one such element, even when the same claim includes the introductory phrase "one or more" or "at least one" along with indefinite articles such as "a" or "an." This is also true for the use of definite articles.

[0060] Unless otherwise stated, terms such as “first” and “second” are used to arbitrarily distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of these elements.

Claims

1. A method for processing a neural network comprising L layers, wherein L is an integer greater than two, the method comprising: The set of computing resources is divided into L layers, which are configured to process forward and backward passes associated with each of the L layers, wherein dividing the L layers into the set of computing resources includes: splitting the parameters associated with the neural network substantially equally among the set of computing resources; Using the aforementioned set of computing resources, the forward pass and the backward pass processes are initiated for each given layer among the L layers; After completing the first set of forward and backward propagations associated with the first layer in the L layers, when the first set of accumulated gradients is available to update the first set of parameters associated with the first layer, an update of the first set of parameters associated with the first layer assigned to the first computing resource in the set of computing resources is initiated, without waiting to compute gradients associated with any of the remaining L-1 layers, and after completing the update of the first set of parameters associated with the first layer, the first set of accumulated gradients is flushed; and After completing the second set of forward passes and the second set of backward passes associated with the second layer in the L layers, when the second set of accumulated gradients is available to update the second set of parameters associated with the second layer, an update of the second set of parameters associated with the second layer assigned to the second computing resource in the set of computing resources is initiated without waiting to compute gradients associated with any of the remaining L-2 layers, and after completing the update of the second set of parameters associated with the second layer, the second set of accumulated gradients is flushed, wherein the initiation of the update of the second set of parameters associated with the second layer is asynchronous with the initiation of the update of the first set of parameters associated with the first layer.

2. The method of claim 1, wherein the neural network is configured to perform natural language processing.

3. The method of claim 1, wherein the neural network further comprises an input stage coupled to provide input to the L layers, and wherein the neural network further comprises an output stage coupled to receive output from the L layers.

4. The method according to claim 1, wherein each of the first set of parameters and the second set of parameters includes weights.

5. The method of claim 1, wherein the set of computing resources is arranged sequentially such that the forward pass and the reverse pass are processed in a pipelined manner.

6. The method of claim 1, further comprising resetting the first block average of the first group of accumulated gradients after a first specified number of backpropagations, and resetting the second block average of the second group of accumulated gradients after the first specified number of backpropagations.

7. The method of claim 1, wherein at least a subset of the L layers corresponds to a converter.

8. A system for processing a neural network comprising L layers, wherein L is an integer greater than two, the system comprising: A set of computational resources is configured to process forward and backward propagations associated with each of the L layers, wherein the L layers are partitioned within the set of computational resources by substantially equally splitting the parameters associated with the neural network, and wherein each of the set of computational resources is configured to asynchronously initiate an update of the weights associated with the layer when a gradient is available to update the weights associated with the layer among the L layers, without having to receive gradients associated with any of the remaining layers among the L layers, wherein the set of computational resources includes a first computational resource and a second computational resource, the first and second computational resources being arranged such that the forward and backward propagations are processed in a pipelined manner, wherein the system is configured as follows: (1) After completing the first set of forward propagation and the first set of backward propagation associated with the first layer in the L layers, when the first set of accumulated gradients is available to update the first set of parameters associated with the first layer, an update of the first set of parameters associated with the first layer assigned to the first computing resource in the set of computing resources is initiated without waiting to compute the gradients associated with any of the remaining L-1 layers, and after completing the update of the first set of parameters associated with the first layer, the first set of accumulated gradients is flushed. as well as (2) After completing the second set of forward propagation and the second set of backward propagation associated with the second layer in the L layers, when the second set of accumulated gradients is available to update the second set of parameters associated with the second layer, an update of the second set of parameters associated with the second layer assigned to the second computing resource in the set of computing resources is initiated without waiting to compute the gradients associated with any of the remaining L-2 layers, and after completing the update of the second set of parameters associated with the second layer, the second set of accumulated gradients is flushed.

9. The system of claim 8, wherein the neural network is configured to perform natural language processing.

10. The system of claim 8, wherein the neural network further comprises an input stage coupled to provide input to the L layers, and wherein the neural network further comprises an output stage coupled to receive output from the L layers.

11. The system of claim 8, wherein the subset of the set of computing resources further includes an accumulator configured to accumulate gradients over a specified number of backpropagations.

12. The system of claim 8, wherein at least a subset of the L layers corresponds to a converter.

13. The system of claim 8, wherein the system is further configured to: reset the first block average of the first set of accumulated gradients after a first specified number of backpropagations, and reset the second block average of the second set of accumulated gradients after the first specified number of backpropagations.

14. A method for processing a neural network comprising L layers, wherein L is an integer greater than two, the method comprising: As part of the first stage of the pipeline: using the first computing resources, initiate the processing of the first forward pass associated with the first of the L layers; As part of the second stage of the pipeline: (1) using the first computing resources, initiate a first forward pass process associated with the second layer of the L layers, and (2) using the second computing resources, initiate a second forward pass process associated with the first layer of the L layers; As part of the third stage of the pipeline: (1) using the first computing resources, initiate a first forward pass process associated with the third layer of the L layers; (2) using the second computing resources, initiate a second forward pass process associated with the second layer of the L layers; and (3) using the third computing resources, initiate a third forward pass process associated with the first layer of the L layers. As part of the fourth stage of the pipeline: (1) using the first computing resource, initiate a first forward pass process associated with the fourth layer of the L layers; (2) using the second computing resource, initiate a second forward pass process associated with the third layer of the L layers; and (3) using the third computing resource, initiate a first reverse pass associated with the first layer of the L layers. as well as As part of the fifth stage of the pipeline: (1) using the first computing resource, initiate a first forward pass process associated with the fifth layer of the L layers; (2) using the second computing resource, initiate a second backward pass process associated with the first layer of the L layers; and (3) when the gradient associated with the weights of the first layer of the L layers is accumulated, update the weights using the third computing resource without waiting to compute the gradients associated with any of the remaining L-1 layers, and flush the accumulated gradients of the first layer after the update of the weights associated with the first layer of the L layers is completed, wherein the update of the weights associated with the L layers is performed asynchronously.

15. The method of claim 14, wherein each of the first layer, the second layer, the third layer, the fourth layer, and the fifth layer corresponds to a converter.

16. The method of claim 14, further comprising: The L layers are divided substantially equally among at least the first computing resource, the second computing resource, and the third computing resource.

17. The method of claim 14, wherein the neural network is configured to perform natural language processing.

18. The method of claim 14, wherein the neural network further comprises an input stage coupled to provide input to the L layers, and wherein the neural network further comprises an output stage coupled to receive output from the L layers.

19. The method of claim 14, further comprising: Accumulate the gradient over a specified number of backpropagations.

20. The method of claim 14, further comprising: Accumulate the gradient over a specified number of loops associated with the pipeline.

Images