Neural network training in distributed systems

Abstract

Methods and systems are provided for performing training operations on a neural network. In one example, a method includes: performing a backpropagation computation for a second layer of the neural network to generate a second weight gradient; splitting the second weight gradient into several parts; causing a hardware interface to exchange a first part of the second weight gradient with a second computer system; while the exchange of the first part of the second weight gradient is in progress, performing a backpropagation computation for a first layer of the neural network to generate a first weight gradient, the first layer being a layer lower than the second layer in the neural network; causing the hardware interface to transmit the first weight gradient to the second computer system; and causing the hardware interface to transmit the remaining part of the second weight gradient to the second computer system.

Description

Background Technology

[0001] Artificial neural networks (ANNs) are computational systems with an architecture based on biological neural networks. An ANN may contain a set of weights. Throughout the computation, the weights can be combined with input data to extract information, and decisions can be made based on that information. For example, in computer vision applications for object detection, an ANN can combine weights with an input image to extract certain features of the object from the image. Based on the extracted features, the ANN can make a decision as to whether the object is in the image.

[0002] The weight set of an artificial neural network can be generated / updated through a training process, during which the artificial neural network learns how to perform a computational task for an application. The training process involves feeding the artificial neural network training input data, which may be labeled with a reference output set supporting a specific decision (e.g., detecting or not detecting an object in an image). The artificial neural network can perform computations to combine the weights with the training input data to produce a training output data set, and can compare the training output data set with the reference output data set to produce a difference set. During training, different training input data sets can be provided to the artificial neural network to produce different training output data sets. The weight set of the artificial neural network can be adjusted based on, for example, the goal of minimizing the difference between the training output data set and the reference output data set.

[0003] To improve the likelihood of artificial neural networks making correct decisions, they are typically trained using a large amount of training input data that covers a wide range of operational scenarios. Therefore, training usually requires significant time and computational resources. Attached Figure Description

[0004] Various embodiments according to this disclosure will be described with reference to the figures, in which:

[0005] Figure 1 An example of a classifier device that processes data using the techniques disclosed herein is shown;

[0006] Figure 2A-2B This is a simplified block diagram illustrating a predictive model and calculation using the techniques disclosed herein, according to certain aspects of this disclosure;

[0007] Figures 3A-3C An example illustrating the training process of a neural network;

[0008] Figures 4A-4C Examples are shown of a process for training a neural network in a distributed system according to certain aspects of this disclosure;

[0009] Figures 5A-5E Showing the support Figures 4A-4CThe instance training process is an instance of a component of the computational system.

[0010] Figure 6 Examples of methods for performing a neural network training process in a distributed system according to certain aspects of this disclosure are shown;

[0011] Figure 7 Showing support Figures 4A-4C An example of an integrated circuit device in the training process;

[0012] Figure 8 Showing support Figures 4A-4C The instance of the host system during the instance training process; and

[0013] Figure 9 A diagram containing instance networks. Detailed Implementation

[0014] Examples of this disclosure relate to neural network processing, and more specifically, to the training process of a neural network in a distributed system.

[0015] Artificial neural networks (hereinafter referred to as neural networks) are typically implemented in computing systems with an architecture based on biological neural networks and process input data in a manner similar to that of biological neural networks. Neural networks typically consist of several cascaded neural network layers, each containing a set of weights. In an inference operation, a first neural network layer receives an input data set, combines the input data set with the weights (e.g., by multiplying the input data set with the weights and then summing the products) to produce a first output data set for the neural network layer, and propagates the output data set to a second neural network layer in a forward propagation operation. The second neural network layer performs another forward propagation operation on the first output data set from the first neural network layer to produce a second output data set, and propagates the second output data set to higher neural network layers. The forward propagation operation can begin at the first neural network layer and end at the highest neural network layer. The forward propagation operation at each neural network layer can represent different stages of the extraction and processing of information from the input data set. A decision can then be made based on the output data of the highest neural network layer. For example, each neural network layer can extract and / or process features from an image, and a decision on whether an object is in the image can be made based on the results of processing the extracted features at the neural network layer.

[0016] The weight set of a neural network can be generated and / or updated during the training process to improve the likelihood that the neural network will make correct decisions. An example training process can use a gradient descent scheme. Specifically, as part of the training process, a forward propagation operation can be performed on the training input data set using the weight set at each layer of the neural network to produce a training output data set at the highest-level neural network layer. The training output data set can be compared with a reference output data set that supports a specific decision. A set of gradients for the input data can be generated, for example, based on the difference between the training output data set and the reference output data set.

[0017] As part of the training process, each neural network layer can subsequently perform a backpropagation process to adjust the weight set at each neural network layer. Specifically, the highest neural network layer can receive the input data gradient set, and in the backpropagation operation, a first data gradient set and a first weight gradient set are calculated based on applying the weight set to the input data gradient in a mathematical operation similar to that of the forward propagation operation. The highest neural network layer can adjust the layer's weight set based on the first weight gradient set, and the first data gradient set can be propagated to the second highest neural network layer to influence the adjustment of the weight set of the preceding neural network layer. The backpropagation operation can begin at the highest neural network layer and end at the first neural network layer. The weight set at each neural network layer can be adjusted to complete one iteration of the training process. The training process can be repeated several times for the same input data set until the loss objective (e.g., thresholding the input data gradient) is achieved.

[0018] Training is typically a very time-consuming process due to the sequential nature of the operations involved and the data dependencies between them. Specifically, as described above, training first involves performing a forward propagation operation at each neural network layer to compute the training output dataset, followed by computing the input data gradients based on the training output dataset (and the reference output dataset), and then performing a backpropagation operation at each neural network layer to compute the weight gradients, followed by weight updates at each neural network layer. Since the backpropagation operation depends on the forward propagation operation, the two sets of operations cannot be performed in parallel. Furthermore, due to the data dependencies between neural network layers, the forward and backpropagation operations must be performed sequentially for each neural network layer. This lack of parallelism significantly increases training time, which increases further when multiple iterations of the training process are performed on the same input dataset to achieve the loss objective. Moreover, the training process typically involves feeding the neural network multiple training input datasets to cover different operating conditions, allowing the neural network to be trained to make correct decisions under those different operating conditions. The computational system implementing the neural network will need to perform additional training processes to process these additional input datasets, which will further increase training time. Given that the training process typically requires higher precision than the inference operation, a slow training process can put a lot of pressure on computational resources.

[0019] One way to accelerate the training process is to use a distributed system to distribute the training process across multiple computing systems, each configured as a worker node. The training input data set can be split into multiple parts, each of which will be processed by a worker node. Each worker node can independently and in parallel perform forward and backward propagation operations based on a portion of the training input data to produce a set of weight gradients for each neural network layer. Each worker node can exchange its set of weight gradients with other worker nodes and average its set of weight gradients with those received from other worker nodes. Each computing node can have the same averaged set of weight gradients and can subsequently update the set of weights for each neural network layer based on the averaged weight gradients.

[0020] Distributing the training process across multiple worker nodes reduces the amount of training input data processed at each worker node, which in turn reduces the execution time of forward and backward propagation operations at each neural network layer and accelerates the training process. However, the exchange of weight gradients between worker nodes introduces a substantial bottleneck. For example, in a distributed system on a cloud infrastructure where worker nodes exchange weight gradients by sending network packets, network latency can be substantial relative to the execution time of forward / backward propagation operations. Network latency can negate the training time reduction brought about by a distributed system, or even increase the training time.

[0021] Examples of this disclosure relate to neural network processing, and more specifically, to performing a training process for a neural network in a distributed system. In one example, the distributed system comprises multiple worker nodes. Each worker node receives a portion of the training input data set and performs a forward propagation computation on the corresponding portion to generate an input data gradient. After the input data gradient is generated, each worker node may perform a backpropagation operation to generate a data gradient and a weight gradient, and exchange weight gradients with other worker nodes.

[0022] To accelerate the training process, worker nodes can perform backpropagation operations and weight gradient swaps in parallel. Specifically, a worker node can perform a backpropagation operation on a higher neural network layer (e.g., a second neural network layer) based on the input data gradient to generate a second data gradient and a second weight gradient, and then perform a backpropagation operation on a lower neural network layer (e.g., a first neural network layer) to generate a first data gradient and a first weight gradient. While the backpropagation operation on the first neural network layer is in progress, the worker node can also perform the swapping of the second weight gradient between worker nodes, and subsequent updates to the weight set at the second neural network layer based on the swapped second weight gradient. Therefore, the backpropagation operation for the neural network layer can be performed in parallel with the weight gradient swapping and weight updates for the higher neural network layer. This parallelism does not affect the weight update operation for the higher neural network layer or the backpropagation operation for the lower neural network layer, because the two operations are not data-dependent on each other. Furthermore, by parallelizing the weight gradient swapping of the higher neural network layer with the backpropagation operation of the lower neural network layer, rather than serializing these two operations, the impact of network latency experienced during weight gradient swapping can be reduced, which can further accelerate the training process.

[0023] In some instances, the swapping of first weight gradients in lower neural network layers can be prioritized relative to the swapping of second weight gradients in higher neural network layers. This prioritization can be based on, for example, worker nodes swapping at least some of the first weight gradients, followed by at least some of the second weight gradients, even if the second weight gradients are generated at the worker nodes before the first weight gradients. Such an arrangement can help reduce training time. Specifically, due to data dependencies, the swapping of weight gradients and the updating of weights for a neural network layer must be serialized with the forward propagation operation at said neural network layer. Since forward and backward propagation operations are performed across neural network layers in reverse order, the time window between backward and forward propagation operations is generally shorter for lower layers than for higher layers. By prioritizing lower neural network layers to perform weight gradient swapping first, the waiting time for weight gradient swapping in lower neural networks can be reduced. Furthermore, the swapping of weight gradients for higher neural network layers can be performed in parallel with the forward propagation of lower neural network layers. Therefore, the overall training time can be reduced.

[0024] In some instances, to support the aforementioned prioritization scheme, the weight gradient generated for each neural network node can be split into multiple parts, each associated with a swap task. The size of a portion of the weight gradient included in a swap task can be based on various factors, including, for example, a predetermined threshold level of network efficiency when transmitting weight gradients, a target time for training completion, etc. Each swap task includes the associated portion of the weight gradient transmitted by the worker node and the corresponding portion received from each of the other worker nodes. Swap tasks can be stored in a buffer managed by an arbitrator. The buffer contains multiple entries. Each entry can store the swap task and its completion status. The buffer also includes write pointers and read pointers. The write pointer points to an empty entry for receiving the next swap task, while the read pointer points to the most recently stored swap task. In some instances, the buffer can be implemented as a stack. In other instances, other data structures such as queues, linked lists, or other types of buffers can be used to store and manage swap tasks.

[0025] After generating the second weight gradient for higher neural network layers, the worker node can split the weight gradient into several parts and sequentially store the first swap task set for each part into a buffer. Since each first swap task is stored, the read and write pointers are also adjusted accordingly. Before storing the next swap task set, the worker node can begin processing the most recently stored first swap task indicated by the read pointer. After the first swap is completed (where the worker node has transmitted a portion of the second weight gradient and received the corresponding portion of the second weight gradient from each of the other worker nodes), the entry's completion status can be marked as complete, and the worker node can process the second most recently stored first swap task. The worker node can process the remaining parts of the first swap tasks in reverse order of task storage.

[0026] While a worker node is processing the first set of swap tasks, it can also store the second set of swap tasks associated with the first weight gradient into a buffer. The read and write pointers can then be moved based on the storage of the second set of swap tasks. The read pointer can point to the most recently stored second swap task. Due to the storage of the new swap task, the worker node can complete the first swap task in progress and then pause processing the remaining first swap tasks. The worker node can then begin processing the most recently stored second swap task. Before storing the next set of swap tasks (for the next lower layer) into the buffer, the worker node can process the remainder of the second swap task while performing a forward propagation operation, followed by processing the remaining first swap tasks. The read pointer can traverse the buffer in reverse order of task storage.

[0027] In the above examples, swapping tasks stored later are processed first by moving the read pointer along with the write pointer. This arrangement allows swapping tasks from lower neural network layers to be prioritized over those from higher neural network layers, since swapping tasks from lower neural network layers are typically generated later (and therefore stored in the buffer later) than those from higher neural network layers. In some instances, the read pointer may also be controlled by an arbitrator that determines the priority of newly stored swapping tasks. If the arbitrator determines that a newly stored swapping task has the highest priority, then the arbitrator can move the read pointer to the entry storing high-priority swapping tasks to process those tasks first.

[0028] The described technique enables at least some of the weight gradient swapping operations in a distributed system to be executed in parallel with backpropagation and forward propagation operations, which further accelerates the training process. Furthermore, by prioritizing the swapping of weight gradients in lower neural network layers over higher neural network layers, the waiting time between two iterations of the training process caused by the time spent by the lowest neural network layer completing the weight gradient swapping and weight updates can be reduced, further accelerating the training process. All of these improvements enhance the efficiency of the training process and reduce the pressure on computational resources.

[0029] In the following description, various examples will be described. Specific configurations and details are elaborated for illustrative purposes to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without the specific details described. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments.

[0030] Figure 1 An example classifier device 100 is shown that processes data using the techniques disclosed herein. Classifier device 100 may be, for example, a computing device that operates a software application 102 and a prediction model 103 to predict information included in a data sequence and perform predetermined functions based on the predictions. For example, classifier device 100 may be part of an image recognition service configured to identify certain objects (e.g., text, people, etc.) from images. It should be understood that the image recognition service is provided only as an illustrative example, and the techniques disclosed herein can be used in other data processing applications, including, for example, text-based data processing (e.g., processing of search queries), audio data processing, etc. Furthermore, classifier device 100 may operate multiple different prediction models to process different input data simultaneously or at different times.

[0031] In some instances, image recognition services may be set up in a multi-tenant computing service system. A multi-tenant computing service system typically includes multiple servers that can host data and can be used by multiple clients or organizations to run instances, such as virtual machine instances or bare metal instances (e.g., an operating system running directly on server hardware). In most cases, multi-tenant computing service systems, such as bare metal or virtual machine instances, can allocate resources to clients when needed and decouple them when no longer needed, allowing resources to be reallocated to other clients. In this disclosure, the terms “tenant,” “client,” and “customer” are used interchangeably, but such terms do not necessarily imply any particular business arrangement. The term “instance” may refer to, for example, an instance running directly on server hardware or as a virtual machine. Different types of instances typically correspond to different hardware capabilities and / or hardware arrangements (e.g., different amounts of available memory and / or processing hardware). Figure 1In this instance, the multi-tenant computing service system can provide image recognition services when clients need them and deactivate them when no longer needed, allowing resources supporting the image recognition service (e.g., access to software application 102 and the underlying hardware resources used to process software application 102) to be reallocated to other clients. Different clients (or a single client) can request application 102 to use the same or different prediction models, including prediction model 103, to perform processing on different input data.

[0032] exist Figure 1 In one example, software application 102 may receive pixel data of image 104 from a user. Image 104 may comprise an array of pixels. Software application 102 may perform analysis on the pixel data and predict one or more objects 106 depicted in image 104. The analysis may include, for example, comparing the pixel data with a predetermined set of feature data. The predetermined feature data may include data associated with a predetermined set of visual image features, such as a nose object, a mouth object, etc. The predetermined feature data may also include data associated with non-visual image features or a combination of visual and non-visual image features. As will be discussed in more detail below, software application 102 may employ prediction model 103 to calculate a set of scores based on the pixel data of image 104. The set of scores may represent, for example, the probability that image 104 contains image features represented by the feature data. Software application 102 may then determine other information about the content of image 104 based on the scores. For example, based on the scores, software application 102 may determine that image 104 is an image of, for example, a panda, a cat, or other object.

[0033] The prediction model 103 may take the form of an artificial neural network. The artificial neural network may include multiple processing nodes, each of which is configured to process a portion of the input pixel data or further process intermediate outputs from other processing nodes. Figure 1 An example of a prediction model 103 using the techniques disclosed herein is shown. Figure 1 In this model, prediction model 103 can be a multi-layer neural network, such as a deep neural network (DNN) or a convolutional neural network (CNN). Prediction model 103 may include an input layer 207, a set of intermediate layers including intermediate layers 209 and 211, and an output layer (…). Figure 2A (Not shown in the image). It should be understood that the prediction model 103 may also include other different types of neural networks, including, for example, long-term memory (LSTM), multilayer perceptron (MTP), multi-scale dense network (MSDNET), etc.

[0034] Layer 207 can process pixel data representing different parts of image 104. For example, in Figure 2AIn this example, layer 207 can process pixel data of image 204. Each processing node of layer 207 is specified to receive pixel values ​​(e.g., x0, x1, x2...x1) corresponding to predetermined pixels within image 104. n The process nodes of layer 207 transmit one or more weights containing the received pixel values ​​to layer 209. If the prediction model 203 is a DNN, a weight set defined by matrix W1 can be assigned to each processing node of layer 207. Each processing node of layer 207 can send the received pixel values ​​and the assigned weights to each processing node of layer 209. If the prediction model 103 is a CNN, the processing node groups of layer 207 can share a weight set, and each group can send the weight set and pixel values ​​received by the processing node group to a single processing node of layer 209. Different neural network models may contain different topologies (e.g., different numbers of layers, different connections between layers, etc.) and / or different weight sets for each layer.

[0035] Layer 209 can process scaled output from layer 207 to produce an intermediate output set. For example, assuming that processing node 210a of layer 209 is connected to n processing nodes in layer 207, processing node 210a can produce the sum of scaled output received from layer 207 based on the following equation:

[0036]

[0037] Here, sum 210a This represents the intermediate output generated by processing node 210a. W1 i ×x i This represents the scaling of a specific pixel value (e.g., x0) by the processing nodes of layer 207 using associated weights (e.g., W10). In the case that prediction model 103 is a DNN, each processing node of layer 209 can produce a sum based on the scaling of pixel values ​​from each processing node of layer 207, and then produce a sum (e.g., a summation) by summing the scaled pixel values. 210a The sum can also represent the dot product between an input vector comprising multiple elements (e.g., pixel values) and a weight vector (e.g., W1). In some instances, a bias can also be added to the scaled output to produce an intermediate output.

[0038] In the case that prediction model 103 is a CNN, each processing node of layer 209 can produce an intermediate output based on a scaled set of pixel values ​​from processing nodes of layer 207. The intermediate output can represent the convolution result between the group of pixel values ​​and a filter including weight values. Figure 2B This shows an example of the convolution operations that can be performed on layer 209. Figure 2BIn this process, filter 230 may contain a two-dimensional array of weights. The weights in filter 230 may represent the spatial distribution of pixels for certain features to be detected from the image. The two-dimensional array may have a height of R rows and a width of S columns, and is typically smaller than an input image having a height of H pixels and a width of W pixels. Each weight may be mapped to pixels in a rectangular block having pixel values ​​with the same R rows and S columns. A processing node of layer 209 (e.g., processing node 210a) may receive a group 240 of pixel values ​​corresponding to a first rectangular block of pixels from the input image from a group of processing nodes of input layer 207, the group 240 of pixel values ​​corresponding to the first step frame position of filter 230, and generate a convolution output 242 according to Equation 1 based on the sum of the results of multiplying each weight of filter 230 with each corresponding pixel in group 240, to produce a dot product between the matrix represented by filter 230 and the matrix represented by group 240. Another processing node of layer 209 may also receive a group 244 of pixel values ​​from another group of processing nodes of input layer 207, the group 244 of pixel values ​​corresponding to a second rectangular block of pixels from the input image corresponding to the second stride position of filter 230, and generate a convolution output 246 according to Equation 1 based on the sum of the results of multiplication between each weight of filter 230 and each corresponding pixel in group 244, to produce a dot product between the matrix of filter 230 and the matrix represented by group 240. In some instances, Figure 2B Each convolutional output (e.g., convolutional output 242, convolutional output 346, etc.) may correspond to the output of a processing node in layer 209. In some instances, the pixel data in the input image may be referred to as the input feature map to indicate that the pixels are processed by the same filter (or the same set of filters) corresponding to a certain feature. The convolutional output may be referred to as the output feature map to indicate that the output is the result of processing the input feature map through the filter.

[0039] like Figure 2B As shown, the convolution operation can be arranged in a sliding window such that the second rectangular block overlaps with the first rectangular block in the input image, or is otherwise adjacent to the first rectangular block in the input image. For example, in Figure 2BIn this example, D can be the stride (in pixels) of the sliding window used for each convolution operation, such that the pixel block corresponding to group 244 can be located at a distance D (in pixels) from the pixel block corresponding to group 240, and the next block of the pixel can also be located at the same distance D from group 244. Other processing nodes of layer 209 can also receive pixel groups corresponding to other rectangular blocks and produce other intermediate outputs. The convolution output can be part of a convolution output array. The convolution output array can have a smaller height and a smaller width than the input image. The rectangular blocks of the convolution output can be further grouped, and convolution operations can be performed at layer 211 between the convolution output groups and another set of filter weights to produce another set of convolution outputs.

[0040] Return to reference Figure 2A A processing node in layer 209 can be configured to produce convolutional output elements of a convolutional output array, and the set M of processing nodes in layer 209 can correspond to the set M of convolutional output arrays. The processing nodes in layer 209 can also process each convolutional output with an activation function to produce an activation output. The activation function can transform the convolutional output into a decision on whether to forward the convolutional output to intermediate layer 211 to influence the classifier's decision (similar to the firing of biological neurons). An instance of the activation function can be a rectified linear unit (ReLU) defined according to the following equation:

[0041]

[0042] Besides ReLU, other activation functions can be used, including soft addition functions (which can be a smooth approximation of the ReLU function), hyperbolic tangent function (tanh), arc tangent function (arctan), sigmoid function, Gaussian function, etc.

[0043] The processing nodes of layer 209 (e.g., processing node 210a) can process the sum with the ReLU function to produce the first output of layer 209 based on the following equation:

[0044] First Output 210a =ReLU(sum) 210a (Equation 3)

[0045] Optionally, prediction model 103 may include pooling layers to reduce the intermediate outputs of layer 209 (e.g., summation). 210aThe number of pooling layers. Pooling layers can group intermediate outputs and perform pooling operations on each group. Pooling operations can include, for example, max pooling (e.g., selecting the largest intermediate output within a group), min pooling (e.g., selecting the smallest intermediate output), average pooling (e.g., finding the average of each group), summation pooling (finding the sum of each group), etc., and the reduced intermediate outputs can be processed by the activation function to produce the first output of layer 209. Pooling operations can be performed to reduce the computational cost associated with activation function processing.

[0046] Layer 211 can further process the scaled intermediate output from layer 209 by performing additional convolution operations, for example, based on different sets of filters. The output from each processing node of layer 211 can be forwarded to other higher intermediate layers, or forwarded to the output layer. Figure 2A (Not shown in the image). The output layer can form an output vector representing, for example, the probability that a specific feature is contained in image 104 and / or the probability that image 204 contains an image of a panda. For example, the output vector can be compared with a reference vector associated with a panda's nose or a reference vector associated with a panda. A decision about whether image 104 is an image of a panda can be determined based on the comparison result.

[0047] exist Figure 2A and Figure 2B The weights and filter coefficients described herein can be generated and updated during the training process to improve the likelihood that the prediction model 103 will make correct decisions. (Reference) Figure 2A and Figure 2B An example of this can be used to train the prediction module 103 based on a set of training images. The training images may include images of different pandas, images of other animals, and other artifacts. The prediction model 103 can process those images and produce different output vectors. The prediction model 103 can update the weights in the neural network layers of the prediction model 103 to maximize the number of correct decisions (e.g., pandas are detected in training images containing pandas, and pandas are not detected in training images not containing pandas).

[0048] Figure 3A An example of a training process 300 for training a neural network comprising a prediction model 103 is shown. The training process can be performed, for example, by a neural network hardware accelerator implementing a neural network that supports the arithmetic operations involved in the neural network processing described above, a general-purpose hardware processor, or other suitable computing system. Training can be based on a gradient descent scheme, which includes forward propagation, loss gradient operations, and backpropagation operations. Specifically, as... Figure 3A As shown, a forward propagation operation can be performed for each layer of the neural network, for example, for the lowest layer 1 (which can correspond to...). Figure 2A The forward propagation operation 302a of the input layer 207) is used for layer 2 (which may correspond to Figure 2A The forward propagation operation 302a of layer 209 is used for the highest layer n (which may correspond to layer 209). Figure 2A The forward propagation operation 302n of layer 211, etc. The forward propagation operation at a neural network layer may include multiplication and summation calculations between the input data and the weight set for that layer, followed by activation function processing as described in Equations 1 and 2 above to produce output data. The output data can then be propagated to the next neural network layer as input to the forward propagation operation at that layer. For example, as... Figure 3A As shown, forward propagation operation 302a combines training input data with the W1 weights of layer 1 to produce output data out1, which is propagated to layer 2 as input. Forward propagation operation 302b combines data out1 with the W2 weights of layer 2 to produce output data out2, which can then be propagated to the next layer. At the highest layer n, forward propagation operation 302n receives data from layer n-1 ( Figure 3A The data outn-1 (not shown in the figure) is combined with the Wn weight of layer n to produce the output data outn.

[0049] The loss gradient operation 304 compares the output data outn of layer n with the reference output data refoutn to produce the input data gradient din. The input data gradient din measures the ratio of the difference between outn and refoutn relative to each data element of the output data outn. In some instances, the training objective is to minimize the difference between outn and refoutn so that the input data gradient din becomes close to zero.

[0050] After generating the input data gradient din through the loss gradient operation 304, a backpropagation operation 306 can be performed for each neural network layer. For example, a backpropagation operation 306n can be performed at the highest layer n, a backpropagation operation 306b can be performed at layer 2, and a backpropagation operation 306a can be performed at layer 1. The backpropagation operation at the neural network layer can be based on the weights of the neural network layer, the data gradient input to the neural network layer, and the input to the forward propagation operation of the layer. For example, for layer n, the backpropagation operation 306n can receive the weights wn, the input data outn-1 (from the forward propagation operation at neural network layer n-1), and the input data gradient din as inputs. The backpropagation operation can perform multiplication and summation calculations on the inputs similar to those in Equation 1 to generate the output data gradient ( Figure 3A (dn-1, d2, d1, etc.) and weight gradient wgrad( Figure 3A(dwn, dw2, dw1, etc. in the model). The output data gradient can be forwarded to the next lower neural network layer as input to the backpropagation operation in that layer, while the weight gradient can represent the change that will be applied to the weights at the neural network layer. The weights at layer n can be updated based on the weight gradient dwn according to the following equation via update operation 308 (e.g., update operation 308n for layer n):

[0051] wn′=wn-α×dwn (Equation 4)

[0052] In Equation 4, wn′ can refer to the updated weight wn, while α can contain a predetermined set of constants.

[0053] The output data gradient dn-1 generated by layer n can then be propagated to the next lower neural network layer n-1 as input to the backpropagation operation at that layer. The backpropagation operation 302b of layer 2 operates on the data gradient d2, weights w2, and input data out1 to produce the output data gradient d1 and the weight gradient dw2. The weight gradient dw2 can be used by the update operation 308b to update the weights of w2 based on Equation 4. The data gradient d1 can be propagated to layer 1. The backpropagation operation 302a of layer 1 operates on the data gradient d2, weights w1, and training input data to produce the weight gradient dw1. The weight gradient dw1 can be used by the update operation 308a to update the weights of w1 based on Equation 4.

[0054] The training process typically involves feeding a neural network multiple sets of training input data to cover different operating conditions, enabling the network to be trained to make correct decisions under those different operating conditions. Due to limited computational resources, computing systems (e.g., neural network hardware accelerators) often lack the ability to perform training simultaneously using all sets of training input data. In practice, the training input data can be divided into multiple parts. The computing system can then perform the training process sequentially in batches, with each batch operating on a portion of the training input data.

[0055] Figure 3B Example 310 illustrates the batch training process relative to time. (e.g.) Figure 3BAs shown, a batch 1 training operation can be performed, including operations 312, 314, and 316. In operation 312, a forward propagation operation can be performed at each neural network layer based on the batch 1 training input data and initial weights. Then, in operation 314, a backpropagation operation can be performed at each neural network layer based on the output of operation 312, followed by operation 316, where the weights at each neural network layer are updated based on the weight gradients from the backpropagation operation in operation 314. After the batch 1 training operation is completed, a batch 2 training operation can be performed, including operations 322, 324, and 326. Operation 322 includes a forward propagation operation based on the batch 2 training input data and the updated weights from the batch 1 training operation; operation 324 includes a backpropagation operation based on the output of operation 322; followed by operation 326, where the weights at each neural network layer are further updated based on the weight gradients from the backpropagation operation in operation 324. After the completion of batch 2, other batches of training operations can be performed on other portions of the training input data. The training operation can also be repeated in batches over multiple iterations until, for example, the gradient din of the input data from the loss gradient operation 304 becomes close to zero, which indicates that the difference between the training output outn and the reference output refout is minimized.

[0056] like Figure 3B As shown, the training process performed on a single computing system is extremely time-consuming due to the sequential nature of the training process. Specifically, as described above, the training process first performs forward propagation at each neural network layer to compute the training output data set, then computes the input data gradient based on the training output data set (and the reference output data set), and then performs backpropagation at each neural network layer to compute the weight gradient, followed by weight updates at each neural network layer. Since the backpropagation operation depends on the forward propagation operation, the two sets of operations cannot be performed in parallel. This is in... Figure 3B It is evident that the backpropagation operation is performed after the forward propagation operation, and the weight update operation is performed after the backpropagation operation. Furthermore, due to the data dependencies between neural network layers, forward and backpropagation operations must be performed sequentially for each neural network layer. The lack of parallelism can significantly increase training time, and this training time increases further when multiple batches of the training process are performed on different parts of the training input dataset, and these batches are repeated iteratively to converge toward the minimum data gradient.

[0057] One way to accelerate the training process is to use a distributed system to distribute the training process across multiple computing systems, each configured as a worker node. Specifically, refer to the above. Figure 3BIn batch instances, each part of the training input data can be further subdivided into sub-parts, each of which will be processed by each worker node of the distributed system during parallel training. Figure 3C This illustrates an example of a parallel training process in a distributed system. For example... Figure 3C As shown, a portion of the training input data used for batch training operations can be further subdivided into sub-parts 0, 1, p, etc. Parallel training processes (e.g., training processes 330a, 330b, 330p, etc.) can be performed at each worker node. For example, approximately simultaneously, each worker node can perform operation 312 (e.g., 312a, 312b, 312p, etc.) based on the corresponding sub-part of the training input data and the same set of initial weights, which includes a forward propagation operation. After the forward propagation operation is completed, each worker node can perform operation 314 (e.g., 314a, 314b, 314p, etc.), which includes a backward propagation operation. Each worker node can perform its corresponding operation 314 approximately simultaneously to generate weight gradients for each neural network layer. Each worker node generates weight gradients based on the corresponding sub-part of the training input data received by the worker node.

[0058] After the backpropagation operation is completed, each worker node may perform an exchange operation 332 (e.g., 332a, 332b, 332p, etc.) to exchange the weight gradients generated at each worker node with other worker nodes. Each exchange operation involves the worker node transmitting the set of weight gradients for each neural network layer to each of the other worker nodes and receiving the set of weight gradients for each neural network from each of the other worker nodes. The worker node may average its set of weight gradients and the set of weight gradients received from the other worker nodes. At the end of the exchange operation 332, each worker node may have the same set of averaged weight gradients. Each worker node may then perform operation 316 (e.g., 316a, 316b, 316p, etc.) to update the weights for each neural network layer based on the averaged weight gradients, and then begin batch 2 forward propagation operation 322 (e.g., 322a, 322b, 322p, etc.) in parallel based on the updated weights.

[0059] Distributing the training process across multiple worker nodes reduces the amount of training data processed at each worker node, which reduces the execution time of forward and backward propagation operations and speeds up the training process. For example, because the amount of training data processed by each worker node is reduced, therefore... Figure 3C The durations of the forward propagation operation 312 and the backward propagation operation 314 in the middle can be shorter than Figure 3BThe duration of the same operation in operation 332 is significantly longer than that of the forward and backward propagation operations. However, in operation 332, the exchange of weight gradients between worker nodes introduces a substantial bottleneck. For example, in a distributed system on a cloud infrastructure where each worker node exchanges weight gradients with each other by sending network packets, network latency can be substantial relative to the execution time of the forward / backward propagation operations. Therefore, the duration of the weight gradient exchange operation 332 can be much longer than the duration of the forward and backward propagation operations 312 and 314. Since the weight gradient exchange operation 332 is serialized with the forward and backward propagation operations 312 and 314, it significantly delays the training process and increases training time.

[0060] Figure 4A , Figure 4B and Figure 4C A distributed system 400 is shown that can perform a training process for a neural network. For example... Figure 4A As shown, the distributed system 400 includes multiple computing systems 402a, 402b, 402p, etc. Each computing system may include a hardware interface for communicating with each other via a network 404. The network 404 may be an Ethernet network, a point-to-point interconnect, a ring-based interconnect, etc. Each computing system may represent a worker node and contain computing resources to perform tasks. Figures 3A-3C The training process includes operations such as forward propagation 312, backward propagation 314, and weight update 316. Computational resources may include, for example, neural network hardware accelerators, general-purpose hardware processors, or other suitable computing systems that support the arithmetic operations involved in the training process. Each computing device may communicate with each other device via network 404 to perform a swap operation 332 by exchanging weight gradients, and performs the weight update operation 316 after the swap operation 332 is completed.

[0061] Figure 4B Examples of internal components of computing system 402 are shown. For example... Figure 4B As shown, the computing system 402 includes a neural network processor 412, a switching processor 414, a weight update module 416, and a hardware interface 418. In some instances, the switching processor 414 and the weight update module 416 can be implemented in software. In some instances, the switching processor 414 and the weight update module 416 can also be implemented as hardware components, such as parts of an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. The computing system 402 further includes a controller (…). Figure 4B (Not shown in the figure) to coordinate the operation between the neural network processor 412, the exchange processor 414, the weight update module 416 and the hardware interface 418.

[0062] The neural network processor 412 can perform a forward propagation operation 312 for all neural network layers, followed by a backward propagation operation 314 for all neural network layers. After the neural network processor 412 generates a weight gradient (e.g., weight gradient 420) for a neural network layer, it sends the weight gradient to a swap processor 414, which manages the swapping operation of the weight gradient. As part of the swapping operation, the swap processor 414 can forward the weight gradient to a hardware interface 418 for transmission to other worker nodes via a network 404. The swap processor 414 can also receive other weight gradients from other worker nodes via a network 404 and a hardware interface 418. The swap processor 414 can average weight gradients 420 and 422 to produce an averaged weight gradient 425 for the neural network layer, and then send the averaged weight gradient 425 to a weight update module 416.

[0063] Weight update module 416 can perform weight update operation 316. Weight update module 416 may include software instructions and / or hardware circuitry (e.g., arithmetic circuitry) to update weights based on Equation 4 above. As part of weight update operation 316, weight update module 416 may receive weights 428 for the neural network layer from neural network processor 412 (or from another source), update the weights based on the averaged weight gradient 425 from exchange processor 414 according to Equation 4 above, and send the updated weights 429 to neural network processor 412 before the next batch of forward propagation operation 322.

[0064] Simultaneously, during the exchange of weight gradients and the updating of weights, the neural network processor 412 can begin the backpropagation operation for the next (lower) neural network layer. The neural network processor 412 can then subsequently send another set of weight gradients for the next neural network layer to the exchange processor 414 after the backpropagation operation for the next neural network layer is completed.

[0065] In some instances, the exchange processor 414 may include a buffer 430 to store weight gradients received from the neural network processor 412 as exchange tasks for neural network layers. The buffer 430 may be implemented in memory coupled to or part of the exchange processor 414. The buffer 430 may contain multiple entries. Each entry may store an exchange task 424 containing the weight gradients of the neural network layer, and an indication 426 indicating whether the exchange task has been completed or not. Figure 4BAs shown, buffer 430 may, for example, store swapping tasks for layer n, swapping tasks for layer n-1, etc. Swap processor 414 can execute swapping tasks by releasing the weight gradients of the swapping tasks to hardware interface 418 whenever hardware interface 418 is available (e.g., no other data packets are being sent). In some instances, swap processor 414 may execute swapping tasks in the order they are stored in buffer 430, such that buffer 430 operates as a first-in, first-out buffer. In some instances, as described below, swap processor 414 may execute swapping tasks based on prioritizing lower neural network layers, such that at least some of the swapping tasks are not executed in the order they are stored in buffer 430.

[0066] To perform the exchange task, hardware interface 418 can establish a communication channel with each of the other worker nodes, exchange weight gradients with each of the other worker nodes using the communication channel, and forward the received weight gradients back to exchange processor 414. When exchange processor 414 detects that all weight gradients for the neural network layer have been received, exchange processor 414 can determine that the exchange task for the neural network layer is complete, and forward the complete set of weight gradients to weight update module 416 to perform weight update operation 316a as described above.

[0067] Figure 4B The arrangement allows the exchange of weight gradients and the backpropagation operation to be performed in parallel. Figure 4C An example of training process 440 is shown. For example... Figure 4C As shown, after the backpropagation operation 442 for layer n is completed, the operation 444, which includes weight gradient swapping (dwn) for layer n and weight update for layer n, can be executed in parallel with the backpropagation operation 446 for layer n-1. Similarly, after the backpropagation operation 452 for layer 2 is completed, the operation 454, which includes weight gradient swapping (dw2) for layer 2 and weight update for layer 2, can be executed in parallel with the backpropagation operation 456 for layer 1. This is achieved by serializing the weight gradient swapping and backpropagation operations. Figure 3C compared to, Figure 4B The arrangement can significantly reduce training time and further accelerate the training process. Furthermore, parallelism does not affect weight update operations for higher neural network layers (e.g., layer n) or backpropagation operations for lower neural network layers (e.g., layer n-1), because the two operations are not data-dependent on each other.

[0068] Although Figure 4BThe arrangement of weight gradients can accelerate the training process, but training time can be further reduced by prioritizing the weight gradient swaps used for lower layers. For example, the swapping of at least some of the weight gradients (dw1) of the lowest layer 1 can be performed first, followed by the swapping of weight gradients of other layers including layers 2 and n. Prioritizing the weight gradient swapping of lower layers over higher layers shortens training time. Furthermore, the second batch forward propagation operation (operation 460) of layer 1 depends on the swapping of weight gradients dw1 in the previous first batch backward propagation operation of layer 1 and the weight update operation 462 of layer 1, as shown by the data dependency arrow 464. Therefore, the update operation 462 of layer 1 must be serialized with the second batch forward propagation operation 460 of layer 1. Thus, the time taken to complete operation 462 can determine the waiting time between the first batch training process and the second batch training process, and can significantly contribute to the overall training time. Prioritizing the weight gradient swapping and weight operation 462 updates for Layer 1 reduces the likelihood of pausing operation 462 while waiting for weight gradient swapping in other layers to complete. This pausing would further delay the completion of operation 462 and the start of the second batch training process. Specifically, operation 462 can begin once the batch 1 backward operation 456 for Layer 1 is completed. Therefore, the time spent completing operation 462 can be reduced, which can further reduce the overall training time.

[0069] Meanwhile, delaying the weight gradient swapping for higher layers does not necessarily increase the overall training time. This is because higher layers have a larger timing window to perform weight gradient swapping and weight updates before the updated weights are needed in the next batch of forward propagation operations for higher layers. This is because forward and backward propagation operations are performed in reverse order between neural network layers. As described above, backward propagation begins at the highest neural network layer and ends at the lowest neural network layer. After updating the weights, a new iteration of the training process can begin, where forward propagation begins at the lowest neural network layer and ends at the highest neural network layer. Since the lowest neural network layer is the last layer used to generate weight gradients but is also the first layer to use updated weights in the forward propagation operation for the next iteration of the training process, the timing window provided to the lowest neural network layer for weight gradient swapping and subsequent weight updates is the shortest among all neural network layers. Figure 4C As shown, while layer n has a window of T1 to complete weight gradient swapping and weight updates before the updated weights are needed for batch 2 forward operation 470 for layer n, as indicated by data dependency arrow 472, layer 1 has a much shorter window of T2 to complete weight gradient swapping and weight updates. Weight gradient swapping in higher layers can be delayed, allowing them to be performed in parallel with the batch 2 forward propagation operation of lower layers (which has no data dependency on the updated weights of higher layers), further reducing training time.

[0070] Figure 5A Additional components of a swap processor 414 are shown that can prioritize swap tasks for lower neural network layers. Specifically, the weight gradients to be swapped for each layer can be split into several parts, each associated with a swap task. When a new swap task for a lower neural network layer is generated, the execution of the swap task for a higher neural network layer can be paused to allow the execution of the new swap task. Since the lower neural network layer has a shorter window to complete weight gradient swapping and weight updates, this arrangement ensures that the weight gradient swapping and weight update operations of the lower neural network layer begin as soon as possible after the weight gradients are generated. Therefore, the delay introduced to the training operation can be reduced by pausing the forward propagation computation of the next batch of the lower neural network while waiting for the weight gradient swapping to complete.

[0071] See Figure 5A In addition to buffer 430, switching processor 414 also includes a controller ( Figure 5A (Not shown in the diagram) The weight gradient splitter 502 and arbitrator 504 are controlled by the neural network processor 412. The weight gradient splitter 502 splits the weight gradients provided by the neural network processor 412 for neural network layers into multiple equal parts, each of which can be associated with a swapping task. As described below, the size of a portion of the weight gradient included in a swapping task can be based on various factors, including, for example, a predetermined threshold level of network efficiency for transmitting weight gradients, a target time for completing the training process, etc. Additionally, the arbitrator 504 controls the execution order of swapping tasks stored in the buffer 430. The buffer 430 is further associated with a write pointer 506 and a read pointer 508. The write pointer 506 can point to an empty entry for storing the next new swapping task and can be adjusted for each newly stored swapping task. The read pointer 508 indicates the entry storing the currently executing task. As described below, the arbitrator 504 can control the read pointer 508 to prioritize swapping tasks for lower neural network layers and / or prioritize newly stored swapping tasks.

[0072] See Figure 5A After the neural network processor 412 completes the backpropagation operation of the neural network layer (e.g., layer n) and provides the weight gradients (e.g., weight gradient 420) to the weight gradient splitter 502, the weight gradient splitter 502 may include an internal buffer for storing the weight gradients. Figure 5A(Not shown in the figure). Based on configuration information specifying the number of weight gradients to be included in each portion, the weight gradient splitter 502 can retrieve portions of the weight gradient 420 from an internal buffer and send the portions of the weight gradient 420 along with the swap task identifier associated with said portions to a buffer 430. The buffer 430 can store each portion of the weight gradient 420 identified based on the associated swap task identifier as a swap task, and a first plurality of swap tasks 510 can be stored in the buffer 430.

[0073] As an illustrative example, for layer n, the weight gradient splitter 502 can split the weight gradient into portions 0 to k. n For layer n-1, the weight gradient splitter 502 can split the weight gradient into parts 0 to k. n-1 One swap task can be stored in an entry of buffer 430 at a time, with write pointer 506 adjusted to point to the next entry for storing the next swap task. After the first plurality of swap tasks are stored in buffer 430, arbitrator 504 can begin executing the first plurality of swap tasks from buffer 430 based on the detection that hardware interface 418 is idle. Arbitrator 504 can set read pointer 508 to the most recently stored execution task and then execute tasks in reverse order of their storage in buffer 430 until buffer 430 stores a second plurality of swap tasks 512 for different sets of weight gradients from different neural network layers (e.g., layer n-1).

[0074] When buffer 430 stores a second plurality of swap tasks 512, arbitrator 504 can determine whether the second plurality of swap tasks are used for a neural network layer lower than the layer associated with the first plurality of swap tasks 510. Figure 5A In this instance, arbitrator 504 can determine that they are such. Arbitrator 504 can then wait until the in-progress exchange tasks of the first plurality of exchange tasks 510 have been completed. When those exchange tasks are completed (e.g., for layer n split k), n When task 510a) is completed, arbitrator 504 may pause the execution of the remaining portion of the first plurality of swap tasks, set the read pointer to the last of the second plurality of swap tasks, and begin executing the second plurality of swap tasks, thus giving the second plurality of swap tasks a higher priority. After the second plurality of swap tasks are completed, the remaining first plurality of swap tasks can be executed in parallel with the forward propagation operation of the lower layer (layer n-1).

[0075] Giving higher priority to the second plurality of swapping tasks can be based, for example, on the fact that the second plurality of swapping tasks are generated for neural network layers lower than the first plurality of swapping tasks, and / or that the second plurality of swapping tasks are stored in buffer 430 later than the first plurality of swapping tasks. Just like the first plurality of swapping tasks, the second plurality of swapping tasks can also be executed in reverse order of being stored in buffer 430, followed by the first plurality of swapping tasks and skipping all completed swapping tasks until buffer 430 stores a third plurality of swapping tasks for another set of weight gradients from yet another different neural network layer. Figure 5A In this arrangement, the switching tasks of lower neural network layers can be prioritized over those of higher neural network layers, which can reduce the training time as described above.

[0076] The size of a portion of the weight gradients included in the swapping task is determined based on the total number of weight gradients that will be transmitted by worker nodes and the number of swapping tasks that will be performed to exchange weight gradients. The number of swapping tasks (and the number of weight gradients in the portion associated with the swapping task) can be determined based on various factors to minimize the overall training time.

[0077] In one instance, to minimize or at least reduce the overall training time, a threshold level for network efficiency in transmitting weight gradients can be determined such that the exchange task can be completed within a target time period in a network with a certain network bandwidth, and the entire training process can be completed within the target training time. Specifically, each exchange task includes establishing a communication channel between worker nodes and preparing data for transmission over the network, as well as exchanging weight gradients via the communication channel. The establishment of the communication channel and the preparation of data for transmission incur overhead time. Increasing the number of exchange tasks can lead to the accumulation of overhead time, which can reduce the share of time the network spends transmitting weight gradients, and this share of time defines the network efficiency in transmitting weight gradients. The number of exchange tasks can be determined such that the share of time spent transmitting weight gradients, compared to the share of timeouts, meets a predetermined threshold level for network efficiency.

[0078] Splitting the weight gradient into several parts and assigning these parts to different switching tasks can also improve the utilization of hardware interface 418 and network 404, while reducing latency to lower-level switching tasks due to prioritization. Specifically, by splitting the weight gradient into several parts and assigning these parts to different switching tasks, at least some of the higher-level weight gradients can be switched in parallel with the backpropagation operation of the lower layers. Compared to an arrangement where all higher-level switching tasks are paused until the lowest-level switching task is completed, Figure 5AThis arrangement reduces the likelihood of hardware interfaces 418 and network 404 being idle while waiting for the lowest-level switching tasks, which improves the utilization of hardware resources and maintains the efficiency of the network between worker nodes. Furthermore, since at least some of the higher-level switching tasks can be executed first instead of being all paused to wait for the lowest-level switching tasks to complete, the possibility of higher-level switching tasks becoming bottlenecks due to prioritization is also reduced.

[0079] Figures 5B to 5E The instance states of buffer 430 at different stages of the training process are shown. Figure 5B This indicates the state of buffer 430 at the end of the backpropagation operation in layer n. The weight gradient is split into parts 0 to k, each associated with a swap task, and the swap tasks are stored sequentially from part 0 to part k. The write pointer 506 can point to the state used for part k of layer n ("layer n split k"). n The first empty entry after the swap task, while read pointer 508 can point to the storage used for layer n part k. n The entry for the swap task that is in progress (written above the pointer 506). Based on the reverse order of the tasks stored in buffer 430, the next swap task to be executed can be layer n split k-1 ("layer n split k"). n -1”).

[0080] Figure 5C This shows the state of buffer 430 at the end of the backpropagation operation in layer n-1. The weight gradient is split into parts 0 to 2, each associated with a swap task, and the swap tasks are stored sequentially from part 0 to part 2. The write pointer 506 can point to the first empty entry after the swap task for part 2 of layer n-1 (“layer n-1 split 2”). While storing the swap task for layer n-1, the execution of layer n split k is complete, and layer n split k... n-1 The execution of -1 is in progress. Arbitrator 504 may wait until layer n splits into k. n-1 The execution of -1 is complete, and then the execution of the remaining part of the swap task for layer n is paused, and the read pointer 508 is moved to layer n-1 split 2 so that it can be executed next.

[0081] Figure 5DThis shows the state of buffer 430 at the end of the backpropagation operation in layer 1 (the lowest layer). The weight gradient is split into parts 0 to 1, each associated with a swap task, and the swap tasks are stored sequentially from part 0 to part 1. The write pointer 506 can point to the first empty entry after the swap task for layer 1 part 1 (“layer 1 split 1”). When storing the swap task for layer 1, the execution of layer n split k, layer n split k-1, layer n-1 split 2, and layer n-1 split 1 has been completed. The execution of these swap tasks can be performed in parallel with the backpropagation operation of higher layers, as explained above. The arbiter 504 can move the read pointer 508 to layer 1 part 1 where execution is in progress.

[0082] Figure 5E This shows the state of buffer 430 at the end of the next batch of forward propagation operations in layer 1. (Example) Figure 5E As shown, the execution of swap tasks Layer 1 Split 0 and Layer 1 Split 1 has been completed, and the weight update module 416 has a complete set of weight gradients for Layer 1 to update the weights for Layer 1. The weight update module 416 can provide the updated weights for Layer 1 to the neural network processor 412, which can then execute the next batch of forward propagation operations for Layer 1. Simultaneously, the arbitrator 504 can move the read pointer 508 to traverse through the remaining unfinished swap tasks in reverse order of the task storage, while skipping completed swap tasks. Figure 5E In this process, arbitrator 504 can move the read pointer 508 to the entry of the layer n-1 split 0 that is in progress during storage execution.

[0083] Figure 5E The status of buffer 430 is displayed, indicating that all exchange tasks have been completed. Arbitrator 504 can reset buffer 430 to prepare for the next batch or iteration of the backward propagation operation.

[0084] Figure 6 This diagram illustrates a method for training a neural network in a distributed system, such as distributed system 400. Distributed system 400 includes multiple computing systems with worker nodes configured to be connected via network 404. A training process can be performed on an input dataset. The input dataset can be divided into multiple parts. Each part is then divided into several sub-parts. Each worker node can execute a sub-part of the input data in the first batch of the training process, followed by another sub-part in the second batch, until all sub-parts of the input data have been used. Each batch of the training process can be executed by neural network hardware and can include forward propagation operations for each layer, followed by backward propagation operations for each layer. Figure 6 In this example, the neural network may contain multiple neural network layers, including a first neural network layer and a second neural network layer. The second neural network layer receives the output of the first neural network as input.

[0085] Method 600 begins in step 602, wherein the neural network processor of the first worker node of the distributed system 400 performs a backpropagation computation for a second neural network layer to generate the second layer data gradient and the second layer weight gradient. The backpropagation computation can be a first batch. This can occur after the first batch of forward propagation operations for the first and second neural network layers has been completed and the input data gradients ( Figure 3A After din has been generated, step 602 is executed. The first batch of backpropagation computations can be performed based on the gradient of the input data or the gradient of the data output by higher neural network layers (if the neural network has more than two neural network layers).

[0086] In step 604, the first worker node generates a first plurality of exchange tasks, each corresponding to the exchange of a portion of the second-layer weight gradient with other worker nodes on network 404. Specifically, the weight gradient splitter 502 can divide the second-layer weight gradient into a plurality of portions, each containing a predetermined number of second-layer weight gradients. Each of the first plurality of exchange tasks may include the worker node establishing a communication channel with each of the other worker nodes, and transmitting a portion of the second-layer weight gradient to each of the other worker nodes via the communication channel and receiving a portion of the second-layer weight gradient generated at each of the other worker nodes. The size of the portion of the weight gradient contained in the exchange task is determined based on the total number of weight gradients to be transmitted by the worker nodes and the number of exchange tasks to be performed to exchange weight gradients. The number of exchange tasks (and the number of weight gradients in the portions associated with the exchange tasks) can be determined based on various factors to minimize the overall training time, such as network efficiency based on maintaining a certain level of weight gradient transmission.

[0087] In some instances, the weight gradient splitter 502 includes an internal buffer to store weight gradients. Based on configuration information indicating the number of weight gradients to be included for each swap task, the weight gradient splitter 502 may obtain the number of second-layer weight gradients from the internal buffer to form a portion for the swap task, and provide the second-layer weight gradients and an execution task identifier to the swap processor 414. The swap processor 414 may identify each portion of the second-layer weight gradient based on the execution task identifier and store the portion in an entry of the buffer 430. The swap processor 414 may sequentially store the plurality of portions of the second-layer weight gradients associated with the first plurality of swap tasks in a plurality of entries of the buffer 430. The swap processor 414 may maintain read pointers (e.g., read pointer 508) and write pointers (e.g., write pointer 506) to track the storage of swap tasks. The read pointer 508 may point to the entry with the most recently stored execution task, while the write pointer 506 may point to the next entry after the read pointer 508 to receive the next swap task.

[0088] In step 606, the first worker node executes the first of the plurality of exchange tasks to exchange a first portion of the second weight gradient with the second worker node. The first worker node may obtain the first portion of the second weight gradient from a first entry in buffer 430. The first entry may be the entry pointed to by read pointer 508. To execute the first exchange task, exchange processor 414 may control hardware interface 418 to establish a communication channel with each of the other worker nodes and transmit the first portion of the second weight gradient to the other worker nodes via the communication channel. Hardware interface 418 may also receive the same number of second weight gradients as the first portion from each of the other worker nodes as part of the first exchange task.

[0089] In step 608, the neural network processor of the first worker node performs backpropagation computation for the first neural network layer in parallel with the first exchange task based on the second layer data gradient (from step 602) to generate the first layer data gradient and the first layer weight gradient. The backpropagation computation for the first neural network layer is performed in the same batch (e.g., the first batch) as the backpropagation computation for the second neural network layer in step 602.

[0090] In step 610, the first worker node generates a second plurality of exchange tasks, each corresponding to an exchange of a portion of the first layer weight gradient with other worker nodes. The weight gradient splitter 502 can divide the first layer weight gradient into multiple parts, each part containing a predetermined number of first layer weight gradients and associated with an exchange task. The number of first layer weight gradients included in each of the second plurality of exchange tasks may be the same as or different from the number of second layer weight gradients included in each of the first plurality of exchange tasks. As in step 604, each of the second plurality of exchange tasks may include the worker node establishing a communication channel with each of the other worker nodes, and transmitting a portion of the first layer weight gradient to each of the other worker nodes via the communication channel and receiving a portion of the first layer weight gradient generated at each of the other worker nodes.

[0091] In step 612, after the first exchange task is completed, the first worker node executes the second plurality of exchange tasks to exchange the first layer weight gradient with the second worker node. Specifically, the exchange processor 414 may execute the exchange tasks based on the order in which the tasks are stored in the buffer 430, but may prioritize the execution of lower-level exchange tasks over higher-level ones once the lower-level exchange tasks are stored in the buffer 430. For example, after the completion of the first exchange task (exchanging the first part of the second layer weight gradient), the exchange processor 414 may move the read pointer 508 to the entry with the most recently stored exchange task (for the first layer weight gradient) among the second plurality of exchange tasks to execute the exchange task. The exchange processor 414 may then traverse the entries storing the remaining parts of the second plurality of exchange tasks to complete the exchange of the first layer weight gradient. After the exchange of the first layer weight gradient is completed, the exchange processor 414 may determine the averaged first layer weight gradient based on averaging the first layer weight gradient generated by the first worker node and the first layer weight gradient received from other worker nodes.

[0092] In step 614, after the exchange of the first layer weight gradients is completed, the weight update module 416 of the first worker node updates the weights used for the first neural network layer based on the averaged first layer weight gradients. This update can be based on Equation 4 above.

[0093] In step 616, the first worker node performs a forward propagation computation for the first neural network layer based on the updated weights. This forward propagation computation is a second batch and is based on a sub-part of input data different from the first batch.

[0094] In step 618, the first worker node executes the remaining exchange tasks among the first plurality of exchange tasks to exchange the remaining portion of the second-layer weight gradient with other worker nodes. At least some of these remaining exchange tasks are executed in parallel with the update of the weights for the first neural network layer and the forward propagation computation for the first neural network layer. After the exchange of the first-layer weight gradient is completed in step 614, the first worker node may begin executing the remaining exchange tasks among the first plurality of exchange tasks in parallel with the update of the weights for the first neural network layer in step 616, followed by the forward propagation computation for the first neural network layer. After the exchange of the second weight gradient is completed, the exchange processor 414 may determine the averaged second-layer weight gradient based on averaging the second-layer weight gradient generated by the first worker node and the second-layer weight gradient received from other worker nodes.

[0095] In step 620, the first worker node updates the weights for the second neural network layer based on the averaged second-layer weight gradient. This update can be based on Equation 4 above. After updating the weights for the second neural network layer, the first worker node can perform a second batch forward propagation operation for the second neural network layer.

[0096] Embodiments of this disclosure may also be described in accordance with the following terms:

[0097] 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to:

[0098] Perform backpropagation computation for the second layer of the neural network to generate the second weight gradient;

[0099] The second weight gradient is split into several parts;

[0100] This causes the hardware interface to exchange the first part of the second weight gradient with the second computer system;

[0101] While the exchange of the first portion of the second weight gradient is in progress, a backpropagation computation for the first layer of the neural network is performed to generate a first weight gradient, the first layer being a lower layer in the neural network than the second layer.

[0102] After the transmission of the first portion of the second weight gradient is completed, the hardware interface is caused to transmit the first weight gradient to the second computer system; and

[0103] After the transmission of the first weight gradient is completed, the hardware interface causes the remaining portion of the second weight gradient to be transmitted to the second computer system.

[0104] 2. The non-transitory computer-readable medium as described in Clause 1, wherein the second weight gradient is generated prior to the first weight gradient.

[0105] 3. A non-transitory computer-readable medium according to clause 1 or 2, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to:

[0106] Each part of the second weight gradient is assigned to each of the plurality of first exchange tasks;

[0107] The plurality of first switching tasks are stored sequentially in the buffer; and

[0108] Receive an indication that the hardware interface is idle:

[0109] Retrieve a first exchange task from the plurality of first exchange tasks from the buffer; and

[0110] The first exchange task will be performed based on the following:

[0111] This causes the hardware interface to transmit the first portion of the second weight gradient assigned to one of the plurality of first switching tasks to the second computer system; and

[0112] The third weight gradient is received from the second computer system via the hardware interface.

[0113] 4. The non-transitory computer-readable medium as described in Clause 3, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to:

[0114] The first weight gradient is divided into multiple parts;

[0115] Each part of the first weight gradient is assigned to each of the plurality of second exchange tasks;

[0116] The plurality of second switching tasks are sequentially stored in the buffer after the plurality of first switching tasks; and

[0117] In response to the completion of the transmission of the portion of the second weight gradient and the reception of the third weight gradient:

[0118] Sequentially retrieve each of the plurality of second switching tasks from the buffer; and

[0119] Each of the plurality of second exchange tasks is executed based on causing the hardware interface to transmit each portion of the first weight gradient assigned to each of the plurality of second exchange tasks;

[0120] Sequentially retrieve each of the remaining first exchange tasks from the buffer; and

[0121] Each of the remaining first exchange tasks is executed based on each portion of the second weight gradient assigned to each of the remaining first exchange tasks by causing the hardware interface to transmit.

[0122] 5. The non-transitory computer-readable medium according to Clause 4, wherein the plurality of second exchange tasks are retrieved from the buffer prior to the remaining first exchange task based on the fact that the first layer is a lower layer in the neural network than the second layer.

[0123] 6. A non-transitory computer-readable medium according to clause 4 or 5, wherein the plurality of second switching tasks are retrieved from the buffer prior to the remaining first switching tasks based on the fact that the plurality of second switching tasks are stored in the buffer later than the plurality of first switching tasks.

[0124] 7. A non-transitory computer-readable medium according to any one of clauses 4 to 6, wherein the plurality of second switching tasks are retrieved sequentially from the buffer in reverse order of the order in which they are stored in the buffer; and

[0125] Each portion of the first weight gradient is transmitted in the order in which the plurality of second exchange tasks are retrieved from the buffer.

[0126] 8. A non-transitory computer-readable medium according to any one of clauses 4 to 7, wherein the magnitude of each portion of the first weight gradient and the second weight gradient is determined based on at least one of the following: network efficiency in transmitting the weight gradient to the second computer system, or the target completion time for training the neural network.

[0127] 9. A non-transitory computer-readable medium according to any one of clauses 4 to 8, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to:

[0128] When the backpropagation computation for the first layer is performed, the hardware interface is caused to transmit at least a portion of the second weight gradient to the second computer system.

[0129] The third weight gradient is transmitted by the second computer system in response to receiving at least a portion of the second weight gradient.

[0130] 10. A non-transitory computer-readable medium according to any one of clauses 1 to 9, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to:

[0131] Perform forward propagation calculations for the first layer; and

[0132] When the forward propagation computation for the first layer is performed, the hardware interface causes at least a portion of the second weight gradient to be transmitted to the second computer system.

[0133] 11. An apparatus comprising:

[0134] Neural network processor;

[0135] Hardware interface;

[0136] Integrated circuits, including weighted gradient splitters and switching processors; and

[0137] The controller is configured to:

[0138] The neural network processor is configured to perform backpropagation computation for the second layer of the neural network to generate a second weight gradient;

[0139] Control the weight gradient splitter to split the second weight gradient into several parts;

[0140] The hardware interface is controlled via the switching processor to exchange a first portion of the second weight gradient with the second computer system;

[0141] The neural network processor is controlled to perform backpropagation computation for a first layer of the neural network to generate a first weight gradient while the exchange of the first portion of the second weight gradient is in progress. The first layer is a lower layer in the neural network than the second layer.

[0142] After the first portion of the second weight gradient is transmitted, the first weight gradient is transmitted to the second computer system via the hardware interface controlled by the switching processor; and

[0143] After the transmission of the first weight gradient is completed, the remaining part of the second weight gradient is transmitted to the second computer system via the hardware interface controlled by the switching processor.

[0144] 12. The apparatus of claim 11, wherein the controller is configured to control the neural network processor to perform forward propagation computation for the second layer in parallel with the transfer of at least a portion of the remaining portion of the second weight gradient to the second computer system.

[0145] 13. The device according to clause 11 or 12, further comprising a memory for storing a buffer;

[0146] The switching processor is configured to:

[0147] Each part of the second weight gradient is assigned to each of the plurality of first exchange tasks;

[0148] The plurality of first switching tasks are stored sequentially in the buffer; and

[0149] Receive an indication that the hardware interface is idle:

[0150] Retrieve a first exchange task from the plurality of first exchange tasks from the buffer; and

[0151] The first exchange task will be performed based on the following:

[0152] The hardware interface controls the transmission of the first portion of the second weight gradient assigned to one of the plurality of first switching tasks to the second computer system; and

[0153] The third weight gradient is received from the second computer system via the hardware interface.

[0154] 14. The device according to Clause 13, wherein the plurality of second switching tasks are retrieved from the buffer prior to the remaining first switching tasks, based on the fact that the plurality of second switching tasks are stored in the buffer later than the plurality of first switching tasks.

[0155] 15. The device according to clause 13 or 14, wherein the plurality of second exchange tasks are retrieved from the buffer prior to the remaining first exchange task based on the fact that the first layer is a lower layer in the neural network than the second layer.

[0156] 16. A method for training a neural network model in a distributed system, the distributed system comprising a first worker node and a second worker node, the neural network model comprising a first neural network layer and a second neural network layer, the method being executed by the first worker node and comprising:

[0157] Perform backpropagation computation for the second neural network layer to generate the second layer data gradient and the second layer weight gradient;

[0158] Generate a first plurality of exchange tasks, each corresponding to the exchange of a portion of the second layer weight gradient with the second worker node;

[0159] Execute the first exchange task in the plurality of exchange tasks to exchange a first portion of the second layer weight gradient with the second worker node;

[0160] Backpropagation computation for the first neural network layer is performed based on the second layer data gradient to generate the first layer data gradient and the first layer weight gradient;

[0161] A second plurality of exchange tasks are generated, each corresponding to the exchange of a portion of the first layer weight gradient with the second worker node;

[0162] After the first exchange task is completed, the second plurality of exchange tasks are executed to exchange the first layer weight gradient with the second worker node;

[0163] The weights of the first neural network layer are updated based on the first layer weight gradient of the exchange.

[0164] The first worker node performs forward propagation computation for the first neural network layer based on the updated weights;

[0165] Execute the remaining exchange tasks in the first plurality of exchange tasks to exchange the remaining portion of the second layer weight gradient with the second worker node; and

[0166] The weights of the second neural network layer are updated based on the second layer weight gradient of the exchange.

[0167] 17. The method according to Clause 16, wherein the backpropagation computation for the first neural network layer and the exchange of the first portion of the weight gradient of the second layer with the second worker node are performed in parallel.

[0168] 18. The method according to clause 16 or 17, wherein the exchange of a portion of the second layer weight gradient with the second worker node comprises:

[0169] Transmit the first portion of the second layer weight gradient generated at the first worker node to the second worker node;

[0170] The first worker node receives a first portion of the second layer weight gradient generated at the second worker node from the second worker node;

[0171] The first worker node determines the averaged second weight gradient based on averaging the first portion of the second layer weight gradient generated at the first worker node and the first portion of the second layer weight gradient generated at the second worker node; and

[0172] The weights used for the second neural network layer are updated using the averaged second weight gradient.

[0173] 19. The method according to any one of clauses 16 to 18, further comprising:

[0174] Store the first plurality of exchange tasks in the buffer;

[0175] Retrieve the first exchange task from the buffer for execution; and

[0176] The buffer stores the second plurality of exchange tasks; and

[0177] In response to the second plurality of swap tasks being stored in the buffer and the completion of the execution of the first plurality of swap tasks, the second plurality of swap tasks are retrieved from the buffer for execution before the remaining swap tasks in the first plurality of swap tasks are executed.

[0178] 20. The method according to any one of clauses 16 to 19, wherein the first weight gradient and the second weight gradient are exchanged on the network; and

[0179] The magnitude of each part of the first weight gradient and the second weight gradient is determined based on at least one of the following: a certain degree of network efficiency in transmitting the weight gradient between the first worker node and the second worker node, or the target completion time for training the neural network model.

[0180] 21. The method according to any one of clauses 16 to 20, further comprising:

[0181] Forward propagation computation for the first neural network layer is performed based on the updated weights used for the first neural network layer.

[0182] The forward propagation computation for the first neural network layer is performed in parallel with the exchange of at least some of the remaining portions of the weight gradients for the second layer.

[0183] 22. A method for training a neural network model in a distributed system, the distributed system comprising a first worker node and a second worker node, the neural network model comprising a first neural network layer and a second neural network layer, the method being executed by the first worker node and comprising:

[0184] Perform backpropagation computation for the second neural network layer to generate the second layer data gradient and the second layer weight gradient;

[0185] Generate a first plurality of exchange tasks, each corresponding to the exchange of a portion of the second layer weight gradient with the second worker node;

[0186] Execute the first exchange task in the plurality of exchange tasks to exchange a first portion of the second layer weight gradient with the second worker node;

[0187] Backpropagation computation for the first neural network layer is performed based on the second layer data gradient to generate the first layer data gradient and the first layer weight gradient;

[0188] A second plurality of exchange tasks are generated, each corresponding to the exchange of a portion of the first layer weight gradient with the second worker node;

[0189] After the first exchange task is completed, the second plurality of exchange tasks are executed to exchange the first layer weight gradient with the second worker node;

[0190] The weights of the first neural network layer are updated based on the first layer weight gradient of the exchange.

[0191] The first worker node performs forward propagation computation for the first neural network layer based on the updated weights;

[0192] Execute the remaining exchange tasks in the first plurality of exchange tasks to exchange the remaining portion of the second layer weight gradient with the second worker node; and

[0193] The weights of the second neural network layer are updated based on the second layer weight gradient of the exchange.

[0194] 23. The method according to Clause 22, wherein the backpropagation computation for the first neural network layer and the first portion of the weight gradient of the second layer are performed in parallel with the exchange of the second worker node.

[0195] 24. The method according to clause 22 or 23, wherein the exchange of a portion of the second layer weight gradient with the second worker node comprises:

[0196] Transmit the first portion of the second layer weight gradient generated at the first worker node to the second worker node;

[0197] The first worker node receives a first portion of the second layer weight gradient generated at the second worker node from the second worker node;

[0198] The first worker node determines the averaged second weight gradient based on averaging the first portion of the second layer weight gradient generated at the first worker node and the first portion of the second layer weight gradient generated at the second worker node; and

[0199] The weights used for the second neural network layer are updated using the averaged second weight gradient.

[0200] 25. The method according to any one of clauses 22 to 24, further comprising:

[0201] Store the first plurality of exchange tasks in the buffer;

[0202] The first exchange task is retrieved from the buffer for execution;

[0203] The buffer stores the second plurality of exchange tasks; and

[0204] In response to the second plurality of swap tasks being stored in the buffer and the completion of the execution of the first plurality of swap tasks, the second plurality of swap tasks are retrieved from the buffer for execution before the remaining swap tasks in the first plurality of swap tasks are executed.

[0205] 26. The method according to any one of clauses 22 to 25, wherein the first weight gradient and the second weight gradient are exchanged on the network; and

[0206] The magnitude of each part of the first weight gradient and the second weight gradient is determined based on at least one of the following: a certain degree of network efficiency in transmitting the weight gradient between the first worker node and the second worker node, or the target completion time for training the neural network model.

[0207] 27. The method according to any one of clauses 22 to 27, further comprising:

[0208] Forward propagation computation for the first neural network layer is performed based on the updated weights used for the first neural network layer.

[0209] The forward propagation computation for the first neural network layer is performed in parallel with the exchange of at least some of the remaining portions of the weight gradients for the second layer.

[0210] Figure 7 This shows that it can be configured to perform, for example... Figures 2A to 3B The various types of neural network operations described herein and can be used as Figures 4A to 4C A block diagram of an example of an integrated circuit device comprising a portion of a neural network processor 412. Figure 7An example is shown for accelerator 702. In various instances, for a set of input data (e.g., input data 750), accelerator 702 can perform computations using processing engine array 710, activation engine 716, and / or pooling engine 718. The processor may have other integrated circuit components, including additional accelerator engines. Accelerator 702 may include controller 722 to control the operation of processing engine array 710, activation engine 716, and / or pooling engine 718.

[0211] In various embodiments, the memory subsystem 704 may include multiple memory banks 714. In these embodiments, each memory bank 714 can be accessed independently, meaning that a read of one memory bank does not depend on a read of another memory bank. Similarly, writing to one memory bank does not affect or limit writing to different memory banks. In some cases, each memory bank can be read and written simultaneously. Various techniques can be used for memory banks 714 that can be accessed independently. For example, each memory bank may be a physically independent memory component with a separate address space that is independent of the address space of each other memory bank. In this example, each memory bank may have at least one read channel and may have at least one separate write channel that can be used simultaneously. In these examples, the memory subsystem 704 may allow simultaneous access to the read or write channels of multiple memory banks. As another example, the memory subsystem 704 may include arbitration logic such that arbitration, for example, between the outputs of multiple memory banks 714, may allow the outputs of more than one memory bank to be used. In these and other instances, although managed globally by the memory subsystem 704, each memory bank can be operated independently of any other memory bank.

[0212] Making memory bank 714 independently accessible can improve the efficiency of accelerator 702. For example, values ​​can be read and served to each row of processing engine array 710 simultaneously, allowing the entire processing engine array 710 to be used in a single clock cycle. As another example, memory bank 714 can be read while the result calculated by processing engine array 710 is written to memory subsystem 704. In contrast, a single memory may only be able to serve one read or write at a time. In the case of a single memory, multiple clock cycles might be required to read the input data for each row of processing engine array 710 before processing engine array 710 can begin.

[0213] In various implementations, the memory subsystem 704 can be configured to simultaneously serve multiple clients, including the processing engine array 710, the activation engine 716, the pooling engine 718, and any external clients accessing the memory subsystem 704 via the communication structure 720. In some implementations, the ability to serve multiple clients may mean that the memory subsystem 704 has at least as many memory banks as the number of clients. In some cases, each row of the processing engine array 710 can be considered a separate client. In some cases, each column of the processing engine array 710 can output results, such that each column can be considered a separate write client. In some cases, output from the processing engine array 710 can be written to memory bank 714, which can then provide input data to the processing engine array 710. As another example, the activation engine 716 and the pooling engine 718 can include multiple execution channels, each of which can be a separate memory client. For example, memory bank 714 can be implemented using static random access memory (SRAM).

[0214] In various implementations, the memory subsystem 704 may include control logic. For example, the control logic may track the address space of each of the memory banks 714, identify which memory bank 714 to read from or write to, and / or move data between memory banks 714. In some implementations, the memory banks 714 may be hardwired to specific clients. For example, a set of memory banks 714 may be hardwired to provide values ​​to rows of the processing engine array 710, with one memory bank serving one row. As another example, a set of memory banks may be hardwired to receive values ​​from columns of the processing engine array 710, with one memory bank receiving data for each column.

[0215] The processing engine array 710 is the computation matrix of the instance accelerator 702. For example, the processing engine array 710 can perform parallel integration, convolution, correlation, and / or matrix multiplication, etc. The processing engine array 710 includes multiple processing engines 711 arranged in rows and columns, such that the output of one processing engine 711 can be directly input into another processing engine 711. Therefore, processing engines 711 not located on the outer edge of the processing engine array 710 can receive data from other processing engines 711, rather than from the memory subsystem 704, for operation.

[0216] In various instances, the processing engine array 710 uses pulsed execution, where data arrives at each processing engine 711 from different directions at regular intervals. In some instances, input data can flow into the processing engine array 710 from the left, and weight values ​​can be loaded at the top. In some instances, weights and input data can flow from the left, and partial sums can flow from the top to the bottom. In these and other instances, multiplication and accumulation operations move diagonally through the processing engine array 710 as wavefronts, where data moves right and down on the array. Control signals can be input simultaneously with the weights at the left and can flow through and down along with the computation.

[0217] In various implementations, the number of columns in the processing engine array 710 determines the computing capacity of the processing engine array 710, and the number of rows determines the required memory bandwidth to achieve maximum utilization of the processing engine array 710. The processing engine array 710 may have, for example, 64 columns and 64 rows, or some other number of columns and rows.

[0218] Figure 7 An example of processing engine 711 is illustrated in the illustration. As illustrated by this example, processing engine 711 may include multiplier-accumulator circuitry. The input from the left may include, for example, input data i and weight values ​​w, where the input data is a set of values ​​taken from a set of input data or a set of intermediate results, and the weight values ​​are a set of weight values ​​that connect one layer of the neural network to the next layer. For example, a set of input data may be an image submitted for recognition or object identification, an audio clip provided for speech recognition, a string of text for natural language processing or machine translation, or the current state of a game that needs to be analyzed to determine the next move, etc. In some instances, the input data and weight values ​​are output to the right for input to the next processing engine 711.

[0219] In the illustrated example, the input from above may contain a partial sum p_in, which is provided from another processing engine 711 or from the previous round of computation via the processing engine array 710. When computation begins for a new set of input data, the top row of the processing engine array 710 may receive a fixed value for p_in, such as zero. As illustrated in this example, i and w are multiplied together, and the result is summed with p_in to produce a new partial sum p_out, which can be input into another processing engine 711. Various other implementations of the processing engine 411 are possible.

[0220] The output from the last row in the processing engine array 710 can be temporarily stored in the summation buffer 712. The result can be an intermediate result, which can be written to memory storage 714 to be provided to the processing engine array 710 for further computation. Alternatively, the result can be a final result, which, once written to memory storage 714, can be read from memory subsystem 704 via communication structure 720 for system output.

[0221] In some implementations, accelerator 702 includes activation engine 716. In these implementations, activation engine 716 can combine results from processing engine array 710 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other instances, it may be necessary to accumulate results from one or more columns in processing engine array 710 to produce an output activation for a single node in the neural network. In some instances, activation engine 716 can be bypassed.

[0222] In various instances, the activation engine 716 may comprise multiple individual execution channels. In these instances, the execution channels may correspond to columns of the processing engine array 710 and can perform operations on the outputs of the columns, the results of which can be stored in the memory subsystem 704. In these instances, the activation engine 716 is capable of performing parallel computations ranging from one to n, where n equals the number of columns in the processing engine array 710. In some cases, one or more of the computations can be performed simultaneously. Examples of computations that each execution channel can perform include exponential, square, square root, identity, binary step, bipolar step, sigmoidal, and ramp operations, among others.

[0223] In some implementations, accelerator 702 may include a pooling engine 718. Pooling is a combination of the outputs of columns from processing engine array 710. Combinations may include, for example, calculating maximum, minimum, average, median, sum, multiplication, or another logical or mathematical combination. In various instances, pooling engine 718 may include multiple execution channels that can operate on values ​​from corresponding columns of processing engine array 710. In these examples, pooling engine 718 is capable of performing parallel computations between one and n, where n equals the number of columns in processing engine array 710. In various instances, the execution channels of pooling engine 718 can operate in parallel and / or simultaneously. In some instances, pooling engine 718 may be bypassed.

[0224] In this paper, the activation engine 716 and the pooling engine 718 can be collectively referred to as the execution engine. The processing engine array 710 is another instance of the execution engine. Another instance of the execution engine is a direct memory access (DMA) engine that can be located outside the accelerator 702.

[0225] Input data 750 can reach communication structure 720. Communication structure 720 can connect accelerometer 702 to other components of the processor, such as a DMA engine, memory drive, or network interface that can obtain input data 750 from input / output (I / O) devices. Input data 750 can be, for example, one-dimensional data, such as a string or sequence of numbers, or two-dimensional data, such as an array of pixel values ​​for an image or the frequency and amplitude over time for an audio signal. In some instances, input data 750 can be three-dimensional, such as for contextual information used in autonomous vehicles or virtual reality data. In some embodiments, memory subsystem 704 may include a separate buffer for input data 750. In some embodiments, when accelerometer 702 receives input data 750, input data 750 may be stored in memory storage 714.

[0226] In some instances, accelerator 702 may implement a neural network processing engine. Accelerator 702 can execute a neural network to perform tasks such as training the neural network on a set of input data 750, for example, performing an inference task. Accelerator 702 can also perform various operations of the training process, such as forward propagation operations, back propagation operations, weight update operations, etc., to update the weights used in the neural network. For example, the processing engine array 710 may be controlled to perform forward propagation computations according to Equation 1 to support the inference task, and to perform both forward and back propagation computations based on Equation 1 to support the training process.

[0227] The weights of the neural network can be stored together with the input data 750 that the neural network will operate on in the memory subsystem 704. The addresses of the weights and input data 750 in the memory subsystem 704 can be based on or mapped to the coordinates of the weights and input data 750 in the weight data array and input data array, respectively, allowing retrieval of the weights and input data based on addresses derived from their coordinates. The neural network may also contain instructions that can be executed by the controller 722 to control the processing engine array 710 to perform various computations on the weights and input data to support inference tasks or training processes. These instructions may be generated by a compiler and may also be stored in the memory subsystem 704, the memory bank 714, or a separate instruction buffer. The processing engine array 710 can output intermediate results representing the outputs of individual layers of the neural network. In some cases, the activation engine 716 and / or the pooling engine 718 may be enabled for computations invoked by specific layers of the neural network. The accelerator 702 can store intermediate results in the memory subsystem 704 to be input into the processing engine array 710 to compute the results of the next layer of the neural network. The processing engine array 710 can further output the final result from the last layer of the neural network. The final result can be stored in the memory subsystem 704 and then copied to the host processor memory or another location.

[0228] Figure 8 Includes a block diagram showing an instance of a host system 800 in which the acceleration engine 860 can be used. Figure 8 The acceleration engine 860 can include, for example Figure 7 Examples of one or more accelerator devices described herein. The host system 800 can be configured as a worker node and can be... Figure 4A The computing system 402. Figure 8 The instance host system 800 includes an acceleration engine 860, a host processor 872, DRAM 830 or processor memory, I / O devices 832, and a support system 874. In various embodiments, the host system 800 may include other hardware not described herein.

[0229] The host processor 872 is a general-purpose integrated circuit capable of executing program instructions. In some instances, the host processor 872 may contain multiple processing cores. A multi-core processor may contain multiple processing units within the same processor. In some instances, the host system 800 may contain more than one host processor 872. In some instances, the host processor 872 and the acceleration engine 860 may be a single chip, such as one or more integrated circuits within the same package.

[0230] In various embodiments, the host processor 872 may communicate with other components in the host system 800 via one or more communication channels. For example, the host system 800 may include a host processor bus, through which the host processor 872 may communicate with, for example, DRAM 830. As another example, the host system 800 may include an I / O bus, such as a PCI-based bus, through which the host processor 872 may communicate with, for example, accelerator engine 860 and / or I / O device 832. In various embodiments, alternatively or additionally, the host system 800 may include other communication channels or buses, such as a serial bus, power management bus, memory device bus, etc.

[0231] In some instances, a software program executing on host processor 872 can receive or generate input for processing by acceleration engine 860. In some instances, for a given input, the program can select a suitable neural network to execute. For example, the program can be used for language translation and can select one or more neural networks capable of speech recognition and / or machine translation. In these and other instances, the program can configure acceleration engine 860 by the neural network to be executed, and / or select a neural network processing engine on acceleration engine 860 that has been previously configured to execute the desired neural network. In some instances, once acceleration engine 860 has begun inference on the input data, host processor 872 can manage the movement of data (e.g., weights, instructions, intermediate results, results of conditional layers, and / or final results) into or out of acceleration engine 860.

[0232] In some instances, software programs using the acceleration engine 860 for inference can read results from the conditional layer of the acceleration engine 860 and / or from storage locations such as DRAM 830. In these instances, the program can determine what action the neural network should take next. For example, the program can determine to terminate the inference. As another instance, the program can determine to change the direction of inference, which can be executed by lower-level code and / or the neural network processor moving to the next layer. In these and other instances, the execution flow of the neural network can be coordinated by software.

[0233] DRAM 830 is a memory used by the host processor 872 to store program code being executed by the host processor 872 and values ​​being operated on. In some instances, neural network data (e.g., weight values, instructions, and other data) may be stored entirely or partially in DRAM 830. DRAM is a common data source for processor memory, and although DRAM is volatile memory, processor memory can be volatile and / or non-volatile. Although not stated herein, the host system 800 may include other volatile and non-volatile memories for other purposes. For example, the host system 800 may include read-only memory (ROM) storing boot code and / or Basic Input / Output System (BIOS) code for powering on the host system 800.

[0234] Although not described herein, DRAM 830 may store instructions for various programs that may be loaded into and executed by host processor 872. For example, DRAM 830 may store instructions for an operating system, one or more data storage areas, one or more application programs, one or more drives, and / or services for implementing the features disclosed herein.

[0235] The operating system manages and schedules the overall operation of the host system 800, such as scheduling tasks, executing applications and / or controller peripherals, and other operations. In some instances, the host system 800 may host one or more virtual machines. In these instances, each virtual machine may be configured to run its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, and so on. Alternatively or additionally, the operating system may be a proprietary operating system.

[0236] Data storage areas may contain permanent or temporary data used and / or manipulated by operating systems, applications, or drives. Examples of such data include web pages, video data, audio data, images, user data, etc. In some instances, information in the data storage area may be made available to user devices via a network. In some cases, the data storage area may additionally or alternatively contain stored applications and / or drives. Alternatively or additionally, the data storage area may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. Information stored in the data storage area may be machine-readable object code, source code, interpreted code, or intermediate code.

[0237] A drive may contain programs that facilitate communication between components within the host system 800. For example, some drives may provide communication between the operating system and peripheral devices or I / O devices 832. Alternatively or additionally, some drives may provide communication between an application and the operating system, and / or between an application and peripheral devices accessible by the host system 800. In many cases, a drive may contain functions that are easy to understand (e.g., printer drive, monitor drive, hard disk drive, solid-state drive, etc.). In other cases, a drive may provide proprietary or special-purpose functions.

[0238] I / O device 832 may include hardware for connecting to user input and output devices, such as a keyboard, mouse, stylus, tablet computer, voice input device, touch input device, display or monitor, speakers, and printer, as well as other devices. I / O device 832 may also include a storage drive and / or a network interface for connecting to network 880. For example, host system 800 may use the network interface to communicate with storage devices, user terminals, other computing devices or servers and / or other networks and various instances.

[0239] In various instances, one or more of the I / O devices 832 may be storage devices. In these instances, the storage device includes non-volatile memory and may store program instructions and / or data. Examples of storage devices include magnetic storage devices, optical disks, solid-state drives, flash memory, and / or magnetic tape storage devices, etc. The storage device may be housed in the same chassis as the host system 800, or it may be housed in an external enclosure. The storage device may be fixed (e.g., attached by screws) or removable (e.g., having a physical release mechanism and possibly a hot-swappable mechanism).

[0240] Storage devices, DRAM 830, and any other memory components in host system 800 are examples of computer-readable storage media. A computer-readable storage medium is a physical medium capable of storing data in a format readable by a device such as host processor 872. Computer-readable storage media can be non-transitory. A non-transitory computer-readable medium can retain data stored thereon when no power is applied to the medium. Examples of non-transitory computer-readable media include ROM devices, disks, magnetic tapes, optical discs, flash drives, and solid-state drives, etc. As used herein, computer-readable storage media does not include computer-readable communication media.

[0241] In various instances, data stored on a computer-readable storage medium may include program instructions, data structures, program modules, libraries, other software program components, and / or other data that may be transmitted within data signals such as carrier waves or other transmissions. Alternatively or additionally, the computer-readable storage medium may include documents, images, video, audio, and other data that can be manipulated or controlled using software programs.

[0242] Figure 9 A diagram including instance network 900, which may contain one or more host systems, for example... Figure 8 The host system shown. For example. Figure 9 The instance network 900 contains multiple nodes 902a-902h, one or more of which may be, for example, in Figure 8 The host system is shown in the diagram. The other nodes in nodes 902a-902h can be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing instructions, and a network interface for connecting to network 900.

[0243] In various instances, network 900 can be used to process data. For example, input data can be received at one of nodes 902a-902h or from other networks 908 with which network 900 can communicate. In this instance, input data can be directed to a node in network 900 that contains an acceleration engine, where the acceleration engine operates and produces results. The results can then be transmitted to said node or other networks from which input data was received. In various instances, input data can be accumulated from various sources, including one or more of nodes 902a-902h and / or computing devices located in other networks 908, and the accumulated input data can be directed to one or more host systems in network 900. Results from the host systems can then be distributed back to the sources from which the input data was collected.

[0244] In various instances, one or more of nodes 902a-902h may be responsible for operations such as accumulating input data for host system operation, tracking which host systems are busy and which host systems can accept more work, determining whether host systems are operating correctly and / or most efficiently, monitoring network security and / or other administrative operations.

[0245] exist Figure 9In this example, nodes 902a-902h are interconnected using a switching architecture with point-to-point links. This switching architecture comprises multiple switches 904a-904d, which can be deployed in a multi-layered network such as a Clos network. A network device that filters and forwards packets between local area network (LAN) segments is called a switch. Switches typically operate at the data link layer (Layer 2), and sometimes at the network layer (Layer 3) of the Open System Interconnection (OSI) reference model, and can support several packet protocols. Figure 9 The 904a-904d switches can connect to nodes 902a-902h and provide multiple paths between any two nodes.

[0246] Network 900 may also include one or more network devices, such as router 906, for connecting to other networks 908. Routers use headers and forwarding tables to determine the best path for forwarding packets and communicate with each other using protocols such as the Internet Control Message Protocol (ICMP) to configure the best route between any two devices. Figure 9 The 906 router can be used to connect to other networks 908, such as subnets, LANs, wide area networks (WANs), and / or the Internet.

[0247] In some instances, network 900 may comprise any or a combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks. Interconnecting switches 904a-904d and router 906 (if present) may be referred to as switch architecture 910, architecture, network architecture, or simply network. In the context of computer networks, the terms "architecture" and "network" are used interchangeably herein.

[0248] Nodes 902a-902h can be any combination of host systems, processor nodes, storage subsystems, and I / O chassis representing user devices, service provider computers, or third-party computers.

[0249] The user device may include a computing device that accesses an application 932 (e.g., a web browser or mobile device application). In some aspects, the application 932 may be hosted, managed, and / or provided by a computing resource service or service provider. The application 932 may allow a user to interact with the service provider's computer to, for example, access web content (e.g., web pages, music, videos, etc.). The user device may be a computing device, such as a mobile phone, smartphone, personal digital assistant (PDA), laptop computer, netbook computer, desktop computer, lightweight client device, tablet computer, e-book reader, game console, etc. In some instances, the user device may communicate with the service provider's computer via another network 908. Additionally, the user device may be part of a distributed system managed, controlled, or otherwise integrated with the service provider's computer (e.g., a console device integrated with the service provider's computer).

[0250] Figure 9 The nodes may also represent one or more service provider computers. One or more service provider computers can provide native applications configured to run on user devices that can interact with the user. Service provider computers may provide computing resources in some instances, such as, but not limited to, client entities, low-latency data storage devices, durable data storage devices, data access, management, virtualization, cloud-based software solutions, eContent performance management, etc. Service provider computers can also be used to provide users with web hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, etc. In some instances, service provider computers may be provided as one or more virtual machines implemented in a managed computing environment. A managed computing environment may contain one or more computing resources that can be rapidly provisioned and released. These computing resources may include computing, network connectivity, and / or storage devices. A managed computing environment may also be referred to as a cloud computing environment. Service provider computers may contain one or more servers, which may be deployed in a cluster, as a server cluster, or as individual servers that are not interconnected, and may host applications and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some respects, the service provider computer may additionally or alternatively include computing devices such as mobile phones, smartphones, personal digital assistants (PDAs), laptops, desktop computers, netbooks, server computers, lightweight client devices, tablets, game consoles, and so on. In some cases, the service provider computer may communicate with one or more third-party computers.

[0251] In one instance configuration, nodes 902a-902h may include at least one memory 918 and one or more processing units (or processors 920). Processor 920 may be implemented in hardware, computer-executable instructions, firmware, or a combination thereof. The computer-executable instructions or firmware implementation of processor 920 may contain computer-executable or machine-executable instructions written in any suitable programming language for performing the various functions described.

[0252] In some cases, the hardware processor 920 can be a single-core processor or a multi-core processor. A multi-core processor can contain multiple processing units within the same processor. In some instances, multi-core processors can share certain resources, such as buses and second- or third-level caches. In some cases, each core in a single-core or multi-core processor can also include multiple execution logic processors (or execution threads). In such cores (e.g., cores with multiple logic processors), several stages of the execution pipeline and lower-level caches may also be shared.

[0253] Memory 918 may store program instructions that can be loaded and executed on processor 920, as well as data generated during the execution of these programs. Depending on the configuration and type of nodes 902a-902h, memory 918 may be volatile (e.g., RAM) and / or non-volatile (e.g., ROM, flash memory, etc.). Memory 918 may contain operating system 928, one or more data storage areas 930, one or more application programs 932, one or more drivers 934, and / or services for implementing the features disclosed herein.

[0254] Operating system 928 can support the basic functions of nodes 902a-902h, such as scheduling tasks, executing applications, and / or controller peripherals. In some implementations, the service provider computer can host one or more virtual machines. In these implementations, each virtual machine can be configured to run its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, and so on. Operating system 928 can also be a proprietary operating system.

[0255] Data storage area 930 may contain permanent or temporary data used and / or operated on by operating system 928, application 932, or drive 934. Examples of such data include web pages, video data, audio data, images, user data, etc. Information in data storage area 930 may be made available to a user device via network 908 in some embodiments. In some cases, data storage area 930 may additionally or alternatively contain stored applications and / or drives. Alternatively or additionally, data storage area 930 may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. Information stored in data storage area 930 may be machine-readable object code, source code, interpreted code, or intermediate code.

[0256] Drive 934 contains programs that enable communication between components within a node. For example, some drives 934 enable communication between operating system 928 and additional storage device 922, network device 924, and / or I / O device 926. Alternatively or additionally, some drives 934 enable communication between application 932 and operating system 928, and / or application 932 and peripheral devices accessible by a service provider's computer. In many cases, drive 934 may contain drives that provide easily understood functionality (e.g., printer drive, monitor drive, hard disk drive, solid-state drive). In other cases, drive 934 may provide proprietary or special-purpose functionality.

[0257] The service provider's computer or server may also include an additional storage device 922, which may include removable and / or non-removable storage devices. The additional storage device 922 may include magnetic storage devices, optical discs, solid-state drives, flash memory, and / or magnetic tape storage devices. The additional storage device 922 may be housed in the same chassis as nodes 902a-902h, or it may be housed in an external enclosure. Memory 918 and / or the additional storage device 922, along with their associated computer-readable media, provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some embodiments, memory 918 may include various types of memory, such as SRAM, DRAM, or ROM.

[0258] Removable and non-removable memory 918 and additional storage device 922 are examples of computer-readable storage media. For example, a computer-readable storage medium may comprise volatile or non-volatile, removable or non-removable media implemented in a method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Memory 918 and additional storage device 922 are examples of computer storage media. Additional types of computer storage media that may exist in nodes 902a-902h may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives, or some other media that can be used to store desired information and are accessible by nodes 902a-902h. Computer-readable media also include any combination of the above media types, comprising multiple units of one media type.

[0259] Alternatively or additionally, a computer-readable communication medium may contain computer-readable instructions, program modules, or other data transmitted within data signals such as carrier waves or other transmissions. However, as used herein, a computer-readable storage medium does not include a computer-readable communication medium.

[0260] Nodes 902a-902h may also include I / O devices 926, such as keyboards, mice, styluses, voice input devices, touch input devices, displays, speakers, printers, and the like. Nodes 902a-902h may also include one or more communication channels 936. The communication channels 936 provide a medium through which various components of nodes 902a-902h can communicate. The communication channels 936 may be in the form of a bus, ring, switching structure, or network.

[0261] Nodes 902a-902h may also contain a network device 924, which allows nodes 902a-902h to communicate with a stored database, another computing device or server, user terminals and / or other devices on network 900.

[0262] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If a module is a software module, it may be embodied on a non-transitory computer-readable medium and processed by a processor in any computer system described herein. It should be noted that the described processes and architectures may be executed in real-time or asynchronously prior to any user interaction. Modules may be configured as recommended in the preceding figures, and / or the functionality described herein may be provided by one or more modules existing as separate modules, and / or the module functionality described herein may be distributed across multiple modules.

[0263] Therefore, this specification and drawings should be considered in an illustrative rather than restrictive sense. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the claims.

[0264] Other variations are within the spirit of this disclosure. Therefore, while the disclosed technology is open to various modifications and alternative constructions, some illustrative examples have been shown in the figures and described in detail above. However, it should be understood that this disclosure is not intended to be limited to the specific forms disclosed, but rather is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0265] Unless otherwise stated herein or clearly contradicted by the context, the use of the words “a / an” and “the,” and similar indicators, in the context of describing the disclosed examples (especially in the context of the appended claims), shall be understood to cover both singular and plural. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” shall be understood as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” shall be understood as being partially or wholly contained, attached to, or joined together, even in the presence of intermediates. Unless otherwise stated herein, references to ranges of values ​​herein are intended only as a way of individually referring to each individual value belonging to the range, and each individual value is incorporated into this specification as if it were individually referenced herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein may be performed in any suitable order. Unless otherwise claimed, the use of any and all instances or exemplary language (e.g., “for example”) provided herein is intended only to better illustrate instances of this disclosure and does not constitute a limitation on the scope of this disclosure. Nothing in this specification should be construed as indicating that any unclaimed element is necessary for the practice of this disclosure.

[0266] Unless otherwise specifically stated, delimited language such as the phrase “at least one of X, Y or Z” is intended to be understood in the context in which it is used to generally represent items, terms, etc., that can be X, Y or Z, or any combination thereof (e.g., X, Y and / or Z). Therefore, such delimited language generally does not intend and should not imply that certain instances require the existence of at least one of X, at least one of Y, or at least one of Z.

[0267] Various examples of this disclosure are described herein, including the best mode known to the inventors for carrying out this disclosure. Variations of those examples will become apparent to those skilled in the art upon reading the foregoing description. The inventors expect those skilled in the art to use such variations where appropriate, and the inventors intend to practice this disclosure in ways different from those specifically described herein. Therefore, this invention encompasses all modifications and equivalents of the subject matter set forth in the appended claims as permitted by applicable law. Furthermore, unless otherwise indicated herein or otherwise clearly contradicted by the context, this disclosure covers any combination of the elements described above with all their possible variations.

Claims

1. A non-transitory computer-readable medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to: Perform backpropagation computation for the second layer of the neural network to generate the second weight gradient; The second weight gradient is split into several parts; This causes the hardware interface to exchange the first part of the second weight gradient with the second computer system; While the exchange of the first portion of the second weight gradient is in progress, a backpropagation computation for the first layer of the neural network is performed to generate a first weight gradient, the first layer being a lower layer in the neural network than the second layer. After the transmission of the first portion of the second weight gradient is completed, the hardware interface is caused to transmit the first weight gradient to the second computer system. as well as After the transmission of the first weight gradient is completed, the hardware interface causes the remaining portion of the second weight gradient to be transmitted to the second computer system.

2. The non-transitory computer-readable medium of claim 1, wherein the second weight gradient is generated prior to the first weight gradient.

3. The non-transitory computer-readable medium according to claim 1 or 2, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to: Each part of the second weight gradient is assigned to each of the plurality of first exchange tasks; The plurality of first exchange tasks are stored sequentially in the buffer; as well as Receive an indication that the hardware interface is idle: Retrieve the first exchange task from the plurality of first exchange tasks from the buffer; as well as The first exchange task will be performed based on the following: This causes the hardware interface to transmit the first portion of the second weight gradient assigned to one of the plurality of first switching tasks to the second computer system; and The third weight gradient is received from the second computer system via the hardware interface.

4. The non-transitory computer-readable medium of claim 3, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to: The first weight gradient is divided into multiple parts; Each part of the first weight gradient is assigned to each of the plurality of second exchange tasks; The plurality of second switching tasks are sequentially stored in the buffer after the plurality of first switching tasks; as well as In response to the completion of the transmission of the portion of the second weight gradient and the reception of the third weight gradient: Each of the plurality of second exchange tasks is retrieved sequentially from the buffer; as well as Each of the plurality of second exchange tasks is executed based on causing the hardware interface to transmit each portion of the first weight gradient assigned to each of the plurality of second exchange tasks; Each of the remaining first exchange tasks is retrieved sequentially from the buffer; as well as Each of the remaining first exchange tasks is executed based on each portion of the second weight gradient assigned to each of the remaining first exchange tasks by causing the hardware interface to transmit.

5. The non-transitory computer-readable medium of claim 4, wherein the plurality of second exchange tasks are retrieved from the buffer prior to the remaining first exchange task based on the fact that the first layer is a lower layer in the neural network than the second layer.

6. The non-transitory computer-readable medium of claim 4, wherein the plurality of second switching tasks are retrieved from the buffer prior to the remaining first switching tasks, based on the fact that the plurality of second switching tasks are stored in the buffer later than the plurality of first switching tasks.

7. The non-transitory computer-readable medium of claim 4, wherein the plurality of second switching tasks are retrieved sequentially from the buffer in reverse order of their storage in the buffer; and Each portion of the first weight gradient is transmitted in the order in which the plurality of second exchange tasks are retrieved from the buffer.

8. The non-transitory computer-readable medium of claim 4, wherein the size of each portion of the first weight gradient and the second weight gradient is determined based on at least one of the following: a certain degree of network efficiency in transmitting the weight gradient to the second computer system, or the target completion time for training the neural network.

9. The non-transitory computer-readable medium of claim 4, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to: When the backpropagation computation for the first layer is performed, the hardware interface is caused to transmit at least a portion of the second weight gradient to the second computer system. The third weight gradient is transmitted by the second computer system in response to receiving at least a portion of the second weight gradient.

10. The non-transitory computer-readable medium of claim 1 or 2, further storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to: Perform forward propagation calculations for the first layer; and When the forward propagation computation for the first layer is performed, the hardware interface causes at least a portion of the second weight gradient to be transmitted to the second computer system.

11. An apparatus comprising: Neural network processor; Hardware interface; Integrated circuit, which includes a weighted gradient splitter and a switching processor; as well as The controller is configured to: The neural network processor is configured to perform backpropagation computation for the second layer of the neural network to generate a second weight gradient; Control the weight gradient splitter to split the second weight gradient into several parts; The hardware interface is controlled via the switching processor to exchange a first portion of the second weight gradient with the second computer system; The neural network processor is controlled to perform backpropagation computation for a first layer of the neural network to generate a first weight gradient while the exchange of the first portion of the second weight gradient is in progress. The first layer is a lower layer in the neural network than the second layer. After the first portion of the second weight gradient is transmitted, the first weight gradient is transmitted to the second computer system via the hardware interface controlled by the switching processor. as well as After the transmission of the first weight gradient is completed, the remaining part of the second weight gradient is transmitted to the second computer system via the hardware interface controlled by the switching processor.

12. The apparatus of claim 11, wherein the controller is configured to control the neural network processor to perform forward propagation computation for the second layer in parallel with the transmission of at least a portion of the remaining portion of the second weight gradient to the second computer system.

13. The device according to claim 11 or 12, further comprising a memory for storing a buffer; The switching processor is configured to: Each part of the second weight gradient is assigned to each of the plurality of first exchange tasks; The plurality of first exchange tasks are stored sequentially in the buffer; as well as Receive an indication that the hardware interface is idle: Retrieve the first exchange task from the plurality of first exchange tasks from the buffer; as well as The first exchange task will be performed based on the following: The hardware interface controls the transmission of the first portion of the second weight gradient assigned to one of the plurality of first switching tasks to the second computer system; and The third weight gradient is received from the second computer system via the hardware interface.

14. The device of claim 13, wherein the plurality of second switching tasks are retrieved from the buffer prior to the remaining first switching tasks, based on the fact that the plurality of second switching tasks are stored in the buffer later than the plurality of first switching tasks.

15. The device of claim 13, wherein the plurality of second switching tasks are retrieved from the buffer prior to the remaining first switching tasks based on the fact that the first layer is a lower layer in the neural network than the second layer.

Images