Speculative training using partial gradient updates

Abstract

This application discloses speculative training using partial gradient updates. The exchange of weight gradients between processing nodes can present a substantial bottleneck to the training process. Rather than remaining idle during the weight gradient exchange process, a processing node can update its own set of weights using the local weight gradient of the processing node for the next iteration of the training process. By using these speculative weights until the weight gradient exchange process is complete and the global weight update is available, the next iteration of the training can begin. If the speculative weights are sufficiently close to the weight values from the global weight update, the training process at the processing node can continue training using the results computed from the speculative weights to reduce the overall training time.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of priority to U.S. Patent Application No. 16 / 831060, filed on March 26, 2020. Background Technology

[0003] Neural networks can be used to perform tasks such as recognizing objects in images. In a neural network, input data is combined with weights to derive output data using an activation function. For example, a neural network can take an image as input data and output a decision or probability that some object is in the image. The set of weights used in a neural network is determined through a training process, during which the neural network learns how to perform certain computational tasks for an application. The training process involves providing the neural network with training input data and corresponding reference outputs that support a specific decision (e.g., whether an object is detected or not detected in an image). The neural network can perform computations to combine the weights with the training input data to generate training output data, which is compared to the reference output data to evaluate the accuracy of the neural network model. Different training input datasets can be provided during training to generate different training output datasets. The weights of the neural network can be adjusted to minimize the difference between the training output data and the reference output data. To improve the probability that the neural network generates correct decisions, a large amount of training input data covering a wide range of scenarios can be used to train the neural network. Therefore, training a neural network can consume a significant amount of time and computational resources. Attached Figure Description

[0004] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0005] Figure 1 The diagram illustrates an example of a classifier device;

[0006] Figure 2 The diagram shows a simplified block diagram of an example of a prediction model;

[0007] Figure 3 The diagram shows a simplified block diagram illustrating an example of an operation performed using a predictive model;

[0008] Figure 4 The diagram illustrates an example of the training process for a neural network;

[0009] Figure 5 The diagram illustrates an example of a distributed system;

[0010] Figure 6 The figure shows a timeline diagram illustrating an example of the training process;

[0011] Figure 7 The figure shows a timing diagram of another example of the training process;

[0012] Figure 8 A timing diagram illustrating a further example of the training process;

[0013] Figure 9 The flowchart illustrates an example of the training process;

[0014] Figure 10 The flowchart illustrates another example of the training process;

[0015] Figure 11 The diagram shows a block diagram of an example computing device;

[0016] Figure 12 The diagram shows another example of a computing device;

[0017] Figure 13 The diagram shows a block diagram of an example of an integrated circuit device;

[0018] Figure 14 A block diagram including an example of an acceleration engine;

[0019] Figure 15 Includes a block diagram of an example host system; and

[0020] Figure 16 Includes a diagram of an example network. Detailed Implementation

[0021] Neural networks typically consist of multiple cascaded neural network layers, each associated with a set of weights. In an inference operation, a first neural network layer receives an input dataset, combines it with the weights (e.g., by multiplying the input dataset by the weights and then summing the products) to generate a first output dataset for that layer, and propagates this output dataset to a second neural network layer in a forward propagation operation. The second neural network layer performs another set of forward propagation operations on the first output dataset from the first layer to generate a second output dataset, and propagates this second output dataset to higher neural network layers. Forward propagation operations can begin at the first neural network layer and end at the highest neural network layer. The forward propagation operations at each neural network layer can represent different stages of extracting and processing information from the input dataset. Decisions can then be made based on the output data of the highest neural network layer. For example, each neural network layer can extract and / or process features from an image, and a decision about whether an object exists in the image can be generated based on the results of processing the extracted features at the neural network layer.

[0022] The weight set of a neural network can be generated and / or updated during the training process to improve the likelihood that the neural network will make correct decisions. An example training process can use a gradient descent scheme. As part of the training process, a forward propagation operation can be performed on the training input dataset using the weight set at each layer of the neural network to generate a training output dataset at the highest-level neural network layer. The training output dataset can be compared with a reference output dataset to determine the error of the neural network model operating on the training input data, and this error can be used to adjust the model's weights to improve the accuracy of the neural network.

[0023] As part of the training process, each neural network layer can perform a backpropagation operation to adjust the set of weights at each layer. During backpropagation, the error or difference between the training output dataset and the reference output dataset propagates backward from the highest-level neural network layer and back towards the first-level neural network layer. At each layer, a set of weight gradients is computed based on the error to determine the amount of adjustment to be made for each weight value. One iteration of the training process is completed when the weights of each layer have been adjusted. The next iteration of the training process can then be performed using the updated weights, and the training process can be repeated for multiple iterations until a loss objective, such as minimizing the error or reducing the error to a certain threshold, is achieved.

[0024] The training process can be very time-consuming due to the sequential nature and data dependencies between the operations involved. As mentioned above, a forward propagation operation is first performed to compute the training output dataset, and then a backpropagation operation is performed to compute the weight gradients. The weights at each layer of the neural network can then be updated using these gradients. Because of the data dependencies between the backpropagation and forward propagation operations, these two sets of operations cannot be performed in parallel. Furthermore, the next iteration of the training process cannot begin until the weight values ​​have been updated. This lack of parallelism drastically increases training time, and this time increases further when multiple iterations are performed on the training process to achieve the loss objective. In addition, the training process typically involves feeding the neural network multiple sets of training input data to cover different input scenarios, allowing the neural network to be trained to provide correct decisions over a wide range of inputs. The computational system implementing the neural network will require training on a large number of input datasets, which further increases training time. The training process may also require higher precision than the inference operation, leading to additional pressure on computational resources.

[0025] One way to accelerate the training process is by using a distributed system where the training is distributed across multiple computing devices, each of which can be configured as a processing node. The training input dataset can be partitioned into multiple parts, each of which is processed by a processing node. Each processing node can independently perform forward and backward propagation operations based on its portion of the training input data to generate a set of weight gradients for each neural network layer. At each iteration of the training process, each processing node can exchange its set of weight gradients with other processing nodes and average its set with the weight gradients received from the other processing nodes. Each processing node can then update its set of weights for each neural network layer based on the average weight gradient computed for that layer.

[0026] Distributing the training process across multiple processing nodes can reduce the amount of training data processed at each node, which in turn reduces the completion time of forward and backpropagation operations at each neural network layer, thus accelerating the training process. However, the exchange of weight gradients between processing nodes can introduce a substantial bottleneck to the training process. For example, in a scenario where the distributed system is located in a cloud infrastructure and processing nodes exchange weight gradients with each other by sending packets of network data, the network latency relative to the time required to complete forward / backpropagation operations can be quite significant. This can result in each processing node being idle while waiting for the next set of weights to become available for the next iteration of the training process. Consequently, the network latency for exchanging weight gradients diminishes the reduction in training time brought about by the distributed system, or even increases the training time if a large number of processing nodes are used.

[0027] In some implementations, to improve system throughput, each processing node can update its own set of weights using local weight gradients to generate a speculative set of weights, instead of remaining idle while waiting for the next set of weights to become available through the gradient exchange process. The next iteration of the training process can be executed using the speculative set of weights while the gradient exchange process is being performed in parallel to obtain the next set of weights. When the next set of weights becomes available, it can be compared with the speculative set of weights obtained from the gradient exchange process to determine the difference between the two sets. This difference represents the difference between the speculative set of weights updated using the local weight gradients of the processing node and the set of weights updated using the average gradients obtained across the processing nodes in the system. If the difference is below a threshold, the speculative set of weights used for the local processing node is sufficiently close to the globally updated weights, and the training process at the processing node can continue using the result of the speculative weights. Therefore, each processing node can initiate and execute the next iteration of the training process using the locally updated speculative weights, instead of remaining idle while waiting for global weight updates.

[0028] In some instances, the speculative weight set may become too far removed from the globally updated weights if the difference between the speculative weight set and the globally updated weights exceeds a threshold. In such instances, the computation performed using the speculative weight set can be discarded, and the iterations of the training process can be repeated using the globally updated weights instead of the speculative weights. Although there is some overhead associated with restarting the iterations of the training process, the number of instances in which the iterations of the training process must be repeated is expected to be small compared to the number of instances in which the results from using speculative weights are available. Thus, despite the possibility of having to repeat the iterations of the training process, the overall training can be completed much faster than in a system where processing nodes remain idle during the gradient exchange process.

[0029] In the following description, various examples will be described. Specific configurations and details are set forth for illustrative purposes to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0030] Figure 1The illustration shows an example of a classifier device 100, in which the techniques disclosed herein can be used to process data. The classifier device 100 may be, for example, a computing device that executes a software application 102 and a predictive model to predict information included in a data sequence and perform predetermined functions based on the predictions. For example, the classifier device 100 may be part of an image recognition service provided to identify certain objects (e.g., text, people, etc.) from images. It should be understood that the image recognition service is provided merely as an illustrative example, and the techniques disclosed herein can be used in other data processing applications, including, for example, text-based data processing (e.g., processing of search queries), audio data processing, etc. Furthermore, the classifier device 100 can process different input data by operating many different predictive models in parallel or at different times.

[0031] In some examples, image recognition services can be provided in a multi-tenant computing service system. A multi-tenant computing service system typically includes multiple servers that manage data and are used by multiple clients or organizations to run instances, such as virtual machine instances or bare metal instances (e.g., an operating system running directly on server hardware). In most instances (such as bare metal instances or virtual machine instances), the multi-tenant computing service system can be allocated to clients when needed and deactivated when no longer needed, allowing resources to be reallocated to other clients. In this disclosure, the terms “tenant,” “client,” and “customer” are used interchangeably, although these terms do not necessarily imply any particular business arrangement. The term “instance” can refer to, for example, an instance that executes directly on server hardware or as a virtual machine. Different types of instances generally correspond to different hardware capabilities and / or hardware arrangements (e.g., different amounts of available memory and / or processing hardware). Figure 1 In the example, the multi-tenant computing service system can provide image recognition services when clients need them and deactivate them when they are no longer needed, allowing resources supporting the image recognition service (e.g., access to software application 102 and the underlying hardware resources used to process software application 102) to be reallocated to other clients. Different clients (or a single client) can request application 102 to use the same or different prediction models (including prediction model 103) to perform processing on different input data.

[0032] exist Figure 1In the example, software application 102 may receive pixel data of image 104 from a user. Image 104 may include an array of pixels. Software application 102 may perform analysis on the pixel data and predict one or more objects 106 depicted in image 104. This analysis may include, for example, comparing the pixel data with a predetermined set of feature data. The predetermined feature data may include data associated with a predetermined set of visual image features (such as a nose object, a mouth object, etc.). The predetermined feature data may also include data associated with non-visual image features, or a combination of visual image features and non-visual image features. As discussed in more detail below, software application 102 may employ prediction model 103 to calculate a set of scores based on the pixel data of image 104. This set of scores may represent, for example, the probability that image 104 includes image features represented by the feature data. Software application 102 may then determine other information related to the content of image 104 based on the scores. For example, software application 102 may determine based on the scores that image 104 is an image of, for example, a panda, a cat, or other object.

[0033] The prediction model 103 can be in the form of an artificial neural network. The neural network can include multiple processing nodes, each of which is configured to process a portion of the input pixel data, or further to process intermediate outputs from other processing nodes. Figure 1 The figure illustrates an example of a prediction model 103 that can utilize the techniques disclosed herein. Figure 1 In this context, the prediction model 103 can be a multi-layer neural network, such as a deep neural network (DNN) or a convolutional neural network (CNN).

[0034] Reference Figure 2 The prediction model 203 may include an input layer 207, a set of intermediate layers, and an output layer ( Figure 2 (Not shown in the diagram), the set of intermediate layers includes intermediate layer 209 and intermediate layer 211. It should be understood that the prediction model 203 may also include other different types of neural networks, including, for example, Long Short-Term Memory (LSTM), Multilayer Perceptron (MTP), Multiscale Dense Network (MSDNET), etc.

[0035] Layer 207 can process pixel data representing different parts of the image being analyzed. For example, in Figure 2 In the example, layer 207 can process pixel data of image 204. Each processing node of layer 207 is assigned to receive pixel values ​​(e.g., x0, x1, x2, ... x) corresponding to predetermined pixels within image 204. nThe process nodes of layer 207 can send one or more weights along with the received pixel values ​​to layer 209. In the case of a DNN prediction model 203, a set of weights defined based on matrix W1 can be assigned to each processing node of layer 207. Each processing node of layer 207 can send the received pixel values ​​and the assigned weights to each processing node of layer 209. In the case of a CNN prediction model 203, groups of processing nodes in layer 207 can share the set of weights, and each group can send the set of weights and the pixel values ​​received by that group of processing nodes to a single processing node in layer 209. Different neural network models can include different topologies (e.g., different numbers of layers, different connections between layers, etc.) and / or different sets of weights for each layer.

[0036] Layer 209 can process the scaled output from layer 207 to generate a set of intermediate outputs. For example, assuming that processing node 210a of layer 209 is connected to n processing nodes in layer 207, processing node 210a can generate the sum of the scaled outputs received from layer 207 based on the following equation:

[0037]

[0038] Here, sum 210a This represents the intermediate output generated by processing node 210a. W1 i ×x i This represents the scaling of a specific pixel value (e.g., x0) by the processing nodes of layer 207 using associated weights (e.g., W10). In the case that prediction model 203 is a DNN, each processing node of layer 209 can generate a sum based on the scaling of pixel values ​​from each processing node of layer 207, and then generate a sum (e.g., summation) by summing the scaled pixel values. 210a This sum can also represent the dot product between the input vector and the weight vector (e.g., W1), which includes multiple elements (e.g., pixel values). In some examples, a bias can also be added to the scaled output to generate an intermediate output.

[0039] In the case that prediction model 203 is a CNN, each processing node of layer 209 can generate an intermediate output based on a scaling of pixel values ​​from a set of processing nodes in layer 207. The intermediate output can represent the result of a convolution between a set of pixel values ​​and a filter including weight values.

[0040] Figure 3 The diagram illustrates an example of the convolution operations that layer 209 can perform. Figure 3In this process, filter 330 may include a two-dimensional array of weights. The weights in filter 330 may represent the spatial distribution of pixels for certain features to be detected from the image. The two-dimensional array may have a height of R rows and a width of S columns, and is typically smaller than an input image having a height of H pixels and a width of W pixels. Each weight may be mapped to pixels in a rectangular block of pixel values ​​having the same R rows and S columns. A processing node of layer 209 (e.g., processing node 210a) may receive a group 340 of pixel values ​​corresponding to a first rectangular block of pixel values ​​from the input image from a set of processing nodes of input layer 207, the group 340 of pixel values ​​corresponding to a first span position of filter 330, and the processing node may generate a convolutional output 342 based on summing the results of multiplying each weight of filter 330 with each corresponding pixel in group 340 according to Equation 1, to generate a dot product between the matrix represented by filter 330 and the matrix represented by group 340. Another processing node of layer 209 can also receive a group 344 of pixel values ​​corresponding to a second rectangular pixel block from the input image from another group of processing nodes of input layer 207. This group 344 of pixel values ​​corresponds to a second span position of filter 330, and this other processing node generates a convolution output 346 based on summing the results of multiplying each weight of filter 330 with each corresponding pixel in group 344 according to Equation 1, to generate a dot product between the matrix of filter 330 and the matrix represented by group 340. In some examples, Figure 3 Each convolutional output (e.g., convolutional output 342, convolutional output 346, etc.) can correspond to the output of the processing node in layer 209. In some examples, the pixel data in the input image can be referred to as the input feature map to indicate that the pixels were processed by the same filter (or the same filter bank) corresponding to a certain feature(s). The convolutional output can be referred to as the output feature map to indicate that the output is the result of processing the input feature map using the filter(s).

[0041] like Figure 3 As shown, the convolution operation can be arranged in a sliding window such that the second rectangular block overlaps with or is otherwise adjacent to the first rectangular block in the input image. For example, in Figure 3In the example, D can be the distance (in pixels) of the span of the sliding window for each convolution operation, such that the pixel block corresponding to group 344 can be located at a distance D (in pixels) from the pixel block corresponding to group 340, and the next pixel block can also be located at the same distance D from group 344. Other processing nodes in layer 309 can also receive pixel groups corresponding to other rectangular blocks and generate other intermediate outputs. The convolution output can be part of a convolution output array. The array of convolution outputs can have a smaller height and a smaller width than the input image. The rectangular blocks of the convolution outputs can be further grouped, and at layer 211, a convolution operation can be performed between the groups of convolution outputs and another set of filter weights to generate another set of convolution outputs.

[0042] Return to reference Figure 2 A processing node in layer 209 can be configured to generate convolutional output elements of a convolutional output array, and a set of M processing nodes in layer 209 can correspond to a set of M convolutional output arrays. The processing nodes in layer 209 can also process each convolutional output using an activation function to generate an activation output. The activation function can transform the convolutional output into a decision on whether to forward the convolutional output to the intermediate layer 211 to influence the classifier's decision (similar to the firing of biological neurons). An example of an activation function could be a Modified Linear Unit (ReLU) defined according to the following equation:

[0043]

[0044] Besides ReLU, other activation functions can be used, such as the softplus function (which is a smooth approximation of ReLU), the hyperbolic tangent (tanh), the arctangent (arctan), the sigmoid function, and the Gaussian function.

[0045] The processing node of layer 209 (e.g., processing node 210a) can use the ReLU function to process the sum based on the following equation to generate the first output of layer 209:

[0046] First Output 210a =ReLU(sum) 210a (Equation 3)

[0047] Optionally, the prediction model 203 may include intermediate outputs (e.g., summation) for reducing the outputs of layer 209. 210aThe pooling layers can group intermediate outputs and perform a pooling operation on each group. Pooling operations can include max pooling (e.g., selecting the largest intermediate output within a group), min pooling (e.g., selecting the smallest intermediate output), average pooling (e.g., calculating the average of each group), summation pooling (calculating the sum of each group), etc., and the reduced intermediate outputs can be processed by an activation function to generate the first output of layer 209. Pooling operations can be performed to reduce the computational cost associated with activation function processing.

[0048] Layer 211 can further process the scaled intermediate output from layer 209 by performing additional convolution operations, for example, based on different filter banks. The output from each processing node of layer 211 can be forwarded to other higher intermediate layers, or forwarded to the output layer. Figure 2 (Not shown in the image). The output layer can form an output vector that, for example, represents the probability that certain features are included in image 204, and / or the probability that image 204 includes an image of a panda. For example, the output vector can be compared with a reference vector associated with a panda's nose object or a reference vector associated with a panda. A decision about whether image 204 is an image of a panda can be determined based on the comparison result.

[0049] Figures 2-3 The weights and filter coefficients described herein can be generated and updated through the training process to improve the likelihood that prediction model 203 will generate correct decisions. (Reference) Figures 2-3 For example, prediction module 203 can be trained based on a set of training images. Training images can include images of different pandas, images of other animals, and other artifacts. Prediction model 203 can process these images and generate different output vectors. The weights in the neural network layers of prediction model 203 can be updated to maximize the number of correct decisions made by prediction model 203 (e.g., detecting a panda in training images containing a panda, not detecting a panda in training images not containing a panda, etc.).

[0050] Figure 4 The diagram illustrates an example of a training process 400 for training a neural network, which may include a neural network of a prediction model 203. The training process can be performed by, for example, a neural network hardware accelerator, a general-purpose hardware processor, or other suitable computing system that supports the arithmetic operations involved in the aforementioned neural network processing. Training can be based on a gradient descent scheme, which includes forward propagation, loss gradient operations, and backpropagation operations. Specifically, as... Figure 4 As shown, a forward propagation operation can be performed for each layer of the neural network, such as forward propagation operation 402a for the lowest layer (the lowest layer can be...). Figure 2(corresponding to the input layer 207), for the forward propagation operation 402a of the second layer (the second layer can be...) Figure 2 (corresponding to layer 209), for the forward propagation operation 402n of the highest nth layer (the nth layer can be associated with...) Figure 2 (corresponding to layer 211), etc. The forward propagation operation at a neural network layer may include multiplication and summation between the input data and the set of weights of that layer, followed by activation function processing (as described in Equations 1 and 2 above) to generate output data. This output data can then be propagated to the next neural network layer as input to the forward propagation operation at that layer. For example, as Figure 4 As shown, forward propagation operation 402a combines the training input data with the W1 weights of layer 1 to generate output data out1 (output 1), which is then propagated as input to layer 2. Forward propagation operation 402b combines out1 with the W2 weights of layer 2 to generate output data out2 (output 2), which can then be propagated to the next layer. At the highest nth layer, forward propagation operation 402n receives data outn-1 (output n-1) from layer n-1. Figure 4 (not shown in the diagram), combined with the Wn weights of the nth layer, and generate output data outn (output n).

[0051] The loss gradient operation 404 compares the output data outn of the nth layer with the reference output data refoutn to generate the input data gradient din. The input data gradient din measures the ratio of the difference between outn and refoutn to each data element of the output data outn. In some examples, the training objective is to minimize the difference between outn and refoutn, making the input data gradient din close to zero.

[0052] After generating the input data gradient din through the loss gradient operation 404, a backpropagation operation 406 can be performed on each neural network layer. For example, a backpropagation operation 406n can be performed at the highest nth layer, a backpropagation operation 406b at the second layer, and a backpropagation operation 406a at the first layer. The backpropagation operation at the neural network layer can be based on the weights of that neural network layer, the data gradient input of that neural network layer, and the input of the forward propagation operation of that layer. For example, for layer n, the backpropagation operation 406n can receive the weights wn, the input data outn-1 (from the forward propagation operation at layer n-1), and the input data gradient din as input. The backpropagation operation can perform a multiplication and summation calculation on the input similar to the multiplication and summation calculation in Equation 1 to generate the output data gradient ( Figure 4(dn-1, d2, d1, etc.) and weight gradient wgrad( Figure 4 (e.g., dwn, dw2, dw1, etc. in the neural network). The output data gradient can be forwarded to the next lower neural network layer as input to the backpropagation operation in that layer, while the weight gradient can represent the change in weights to be applied to the neural network layer. The weights at the nth layer can be updated using update operation 408 (e.g., update operation 408n for the nth layer) based on the weight gradient dwn, according to the following equation:

[0053] wn′=wn-α×dwn (Equation 4)

[0054] In Equation 4, wn' can refer to the updated weight wn, while α can include a predetermined set of constants.

[0055] The output data gradient dn-1 generated by the nth layer can then be propagated to the next lower neural network layer n-1 as input to the backpropagation operation of that layer. The backpropagation operation 402b of the second layer operates on the data gradient d2, weights w2, and input data out1 to generate the output data gradient d1 and weight gradient dw2. The update operation 408b uses the weight gradient dw2 to update the weights w2 based on Equation 4. The data gradient d1 can be propagated to the first layer. The backpropagation operation 402a of the first layer operates on the data gradient d2, weights w1, and training input data to generate the weight gradient dw1. The weight gradient dw1 can be used by the update operation 408a to update the weights w1 based on Equation 4.

[0056] The training process typically involves feeding a neural network a set of training input data to cover different operating conditions, enabling the network to be trained to make correct decisions under these varying conditions. Due to limited computational resources, computational systems (e.g., neural network hardware accelerators) often lack the ability to perform training using all the training input datasets at once. Instead, the training input data can be partitioned into multiple parts. The computational system can then perform the training process in batches, sequentially, with each batch operating on a portion of the training input data.

[0057] Due to the sequential nature of the training process, training performed on a single computational system can be extremely time-consuming. Specifically, as mentioned above, during training, forward propagation is first performed at each neural network layer to compute the training output dataset, followed by backward propagation at each neural network layer to compute the weight gradients, and then weight updates at each neural network layer. Since the backward propagation operation depends on the forward propagation operation, the set of two operations cannot be performed in parallel. Furthermore, due to the data dependencies between neural network layers, the forward and backward propagation operations for each neural network layer also need to be performed sequentially. This lack of parallelism drastically increases training time, and the training time increases further when multiple batches of training are performed on different parts of the training input dataset, and these batches are repeated in multiple iterations to converge toward the minimum data gradient.

[0058] One way to accelerate the training process is by using a distributed system, distributing the training process across multiple computing devices, each of which can be configured as a processing node. Distributing the training process across multiple processing nodes reduces the amount of training data to be processed on each processing node, thereby reducing the completion time of forward and backpropagation operations and accelerating the training process. For example, because the amount of training data processed by each processing node has been reduced, the duration of forward and backpropagation operations can be shorter.

[0059] Figure 5 The diagram illustrates a distributed system, implemented in various ways, capable of performing the training process for a neural network. For example... Figure 5 As shown, the distributed system may include multiple computing devices 504-1, 504-2, ... to 504-n, etc. Each computing device may include a communication interface for communicating with each other via a computer network 506. Each computing device may represent a processing node and includes computing resources for performing operations of the training process, including forward propagation operations, back propagation operations, weight update operations, etc. Computing resources may include, for example, neural network processors, neural network accelerators, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), processors or coprocessors, application-specific integrated circuits (ASICs), and / or other suitable computing circuitry supporting the arithmetic operations involved in the training process. Each computing device may communicate with other computing devices via the computer network 506 to perform exchange operations by exchanging weight gradients, and to perform weight update operations after the exchange operations are completed.

[0060] To update the weight values ​​after each iteration of the training process, the weight gradients computed by each processing node are exchanged with those of other processing nodes, allowing the average set of weight gradients to be computed and used to update the system's weights. This exchange of weight gradients between processing nodes can introduce a substantial bottleneck to the training process. For example, in a distributed system situated in a cloud infrastructure, where each processing node exchanges weight gradients with the others by sending network data packets, network latency can be substantial relative to the time required to complete the forward / backward propagation operations. Therefore, the duration of the weight gradient exchange operation can be significantly longer than the duration of the forward and backward propagation operations. Because the weight gradient exchange operation is serialized with the forward and backward propagation operations, it adds significant latency to the training process and increases training time.

[0061] Figure 6 The figure illustrates a timing diagram of an example training process performed in a distributed neural network training system. The distributed neural network training system may include two or more processing nodes 604-1 to 604-n. Processing nodes 604-1 to 604-n can be implemented using any combination of neural network processors, neural network accelerators, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), processors, and / or application-specific integrated circuit (ASIC) devices.

[0062] To accelerate the training process, each of the processing nodes 604-1 to 604-n can perform training on a subset of the entire training dataset in parallel. For example, if the training dataset includes 10,000 images and there are ten processing nodes, each processing node can train on 1,000 images. Although the training dataset can be evenly distributed among the processing nodes, some systems can distribute the workload differently (e.g., processing nodes with faster hardware may be assigned a larger workload).

[0063] Reference Figure 6Processing node 604-1 can perform a first iteration of the training process on a portion of the training data (such as the batch 1-1 input dataset). The first iteration of the training process may include performing forward propagation 612-1 and backpropagation 614-1 using the neural network computation circuitry of the processing node. During forward propagation 612-1, computation is performed on the batch 1-1 input dataset from the lowest to the highest neural network layer using a set of weights to generate an output dataset. The output dataset can be compared to a reference output dataset to determine the amount of error between the output of the neural network model and the expected output when operating on the batch 1-1 input dataset. During backpropagation 614-1, the error of the neural network model operating on the batch 1-1 input dataset is propagated from the highest neural network layer back to the lowest neural network layer to determine a set of local weight gradients for processing node 604-1, which represents the amount by which the weight values ​​of each layer are adjusted.

[0064] Similarly, processing node 604-n can perform the first iteration of the training process on different portions of the training data (such as the batch 1-n input dataset). The first iteration of the training process at processing node 604-n may include performing forward propagation 612-n and backpropagation 614-n. During forward propagation 612-n, computation is performed on the batch 1-n input dataset using the same set of weights as in processing node 604-1 to generate an output dataset. The output dataset can be compared with a reference output dataset to determine the amount of error between the output of the neural network model and the expected output when operating on the batch 1-n input dataset. During backpropagation 614-n, the error of the neural network model operating on the batch 1-n input dataset is propagated from the highest neural network layer back to the lowest neural network layer to determine a set of local weight gradients for processing node 604-n, which represents the amount by which the weight values ​​of each layer are adjusted.

[0065] Since each of the processing nodes 604-1 to 604-n operates on a different part of the training dataset, the amount of error from the first iteration of the training process can vary across different processing nodes. To improve the accuracy of the neural network model across different training data, the local weight gradients computed by each processing node can be accumulated and then averaged to derive a set of average weight gradients. For example, if the neural network model uses twenty weight values, the first iteration of the training process at each processing node will produce twenty local weight gradients. The first local weight gradients from each processing node can be summed and divided by the number of processing nodes to derive the average weight gradient for the first weight values. Computation can be performed on each of the twenty weight gradients to derive a set of twenty average weight gradients.

[0066] The weights of the neural network model can then be updated using the average weight gradient set to derive the next set of weights to be used in the next iteration of the training process. For example, each weight value can be updated by multiplying the corresponding average weight gradient by a scalar representing the learning rate and adding the result to the weight value. To obtain the next set of weights to be used in the next iteration of the training process, processing node 604-1 can perform gradient swapping and global weight update procedure 616-1, and processing node 604-n can similarly perform gradient swapping and global weight update procedure 616-n to obtain the next set of weights. Figure 6 As shown, the second iteration of the training process at each processing node may not begin until the processing node obtains the next set of weights.

[0067] Once the next set of weights becomes available, processing node 604-1 can perform a second iteration of the training process, including forward propagation 622-1 and backpropagation 624-1, on the batch 2-1 input dataset. In some implementations, the batch 2-1 input dataset can be the same as the batch 1-1 input dataset if the system is to evaluate the accuracy of the updated model relative to the same input dataset used in the first iteration. In some implementations, the batch 2-1 input dataset can be a subset of the training dataset that is different from the batch 1-1 input dataset. Once processing node 604-n obtains the next set of weights from the global weight update process 616-n, processing node 604-n can similarly perform a second iteration of the training process, including forward propagation 622-n and backpropagation 624-n, on the batch 2-n input dataset. The training process can be repeated an arbitrary number of iterations until the error of the neural network model across the training dataset is reduced to a certain error threshold or reaches a minimum error.

[0068] The gradient swapping and global weight update process described above can be executed using hardware different from the hardware used to perform forward and backward propagation. For example, the gradient swapping and global weight update process can be executed using a host processor and communication interface, while forward and backward propagation can be executed using neural network computation circuitry. The gradient swapping and global weight update process can involve each processing node transmitting its corresponding set of local weight gradients to another processing node, which accumulates the local weight gradients from other processing nodes in the system and calculates the average of each weight gradient. The processing node receiving the local weight gradients from other processing nodes can then update each weight value of the neural network model using the corresponding average weight gradient to derive the next set of weights. The next set of weights can then be broadcast from that processing node to the other processing nodes in the system. Alternatively, the local weight gradients from each processing node can be broadcast to other processing nodes, and each processing node can calculate its own average set of weight gradients and update its set of weights accordingly using the set of local weight gradients received from other processing nodes.

[0069] Depending on the number of processing nodes in the system, obtaining the next set of weights can take a significant amount of time because the system must collect the local weight gradients from each processing node across the network to compute the next set of weights. Figure 6 As shown, the neural network computation circuits that perform forward and backward propagation operations are idle during the time they wait for the next set of weights to become available, because the next iteration of the training process may not begin until the processing node obtains the next set of weights.

[0070] One way to reduce the waiting time in the weight update process is to initiate gradient swapping as soon as the local weight gradients for the neural network layers become available. For example, during backpropagation 614-1, as the error propagates through the highest neural network layer, the local weight gradients associated with that layer can be computed and swapped with other processing nodes. Thus, the gradient swapping and global weight update process 616-1 performed by processing node 604-1 does not need to wait for the entire backpropagation 614-1 to complete; instead, it can be initiated as soon as the local weight gradients for the highest neural network layer become available. Accordingly, the gradient swapping and global weight update processes 616-1 and 616-n are performed in... Figure 6 The gradient exchange is shown to be initiated before the corresponding backpropagation processes 614-1 and 614-n are completed. Although initiating gradient exchange early in the manner described above can reduce the global weight update wait time, there can still be a significant amount of idle time between iterations of the training process (e.g., the idle time between backpropagation 614-1 and forward propagation 622-1).

[0071] Figure 7 The figure illustrates a timing diagram of another example of a training process executed in a processing node of a neural network training system. Processing node 704-1 can be one of multiple processing nodes in the neural network training system. Although in Figure 7 Only one processing node 704-1 is shown in the figure, but it should be understood that the other processing nodes in the system can operate in a similar manner.

[0072] Processing node 704-1 can begin training by performing the first iteration of the training process on a portion of the training data (such as a batch 1-1 input dataset). The first iteration of the training process may include performing forward propagation 712-1 and backpropagation 714-1. During forward propagation 712-1, computation is performed on the batch 1-1 input dataset from the lowest to the highest neural network layers using a set of weights to generate an output dataset. The output dataset can be compared to a reference output dataset to determine the amount of error between the neural network model's output and the expected output when operating on the batch 1-1 input dataset. During backpropagation 714-1, the error of the neural network model operating on the batch 1-1 input dataset is propagated back from the highest neural network layer to the lowest neural network layer to determine a set of local weight gradients for processing node 704-1, representing the amount of adjustment to the weight values ​​of each layer.

[0073] Once the local weight gradients for the highest neural network layer are generated via backpropagation 714-1, processing node 704-1 can initiate a local weight update process 718-1. This local weight update process 718-1 generates a speculative weight set by updating the weight set using the set of local weight gradients from processing node 704-1, without considering local weight gradients from other processing nodes in the system. Therefore, the local weight update process 718-1 can execute much faster than global weight updates because it does not need to exchange weight gradients with other processing nodes in the system. Once the local weight update process 718-1 has updated the weight set using the local weight gradients, processing node 704-1 can begin the second iteration of the training process on the batch 2-1 input dataset (which may be the same as or different from the batch 1-1 input dataset). The second iteration of the training process may include: performing a forward propagation 722-1 using a speculative set of weights to generate an output dataset; comparing the output dataset with a reference output dataset to determine the error; and performing a backpropagation 724-1 using the error to derive the next set of local weight gradients.

[0074] Processing node 704-1 can also initiate gradient exchange and weight comparison process 716-1 in parallel with the local weight update process 718-1. The gradient exchange and weight comparison process 716-1 may involve: processing node 704-1 transmitting its set of local weight gradients to one of the processing nodes, which accumulates the local weight gradients from the processing nodes in the system and calculates the average of each weight gradient. The processing node receiving the local weight gradients from other processing nodes in the system can update each weight value of the neural network model using the corresponding average weight gradient to derive the next weight set. Processing node 704-1 can then receive the next weight set in communication from the processing node that determined the next weight set. Alternatively, processing node 704-1 can broadcast its set of local weight gradients to other processing nodes and receive the set of local weight gradients from each of the other processing nodes, allowing processing node 704-1 to calculate its own average weight gradient set and generate the next weight set accordingly.

[0075] Once processing node 704-1 has obtained the next set of weights (e.g., by receiving the next set of weights in communication with another processing node, or by computing the next set of weights itself), processing node 704-1 can compare the speculative set of weights generated from the local weight gradient with the set of weights generated from the average weight gradient to determine the difference between the two sets of weights. In some implementations, the difference between the two sets of weights can be determined as, for example, the average of the differences between corresponding weight values ​​from the two sets, the median of the differences between corresponding weight values ​​from the two sets, the maximum of the differences between corresponding weight values ​​from the two sets, or the root mean square of the differences between corresponding weight values ​​from the two sets, etc. In other implementations, other metrics can be used to measure the difference between the two sets of weights.

[0076] Then, the gradient swapping and weight comparison process 716-1 compares the difference between the two weight sets with a threshold difference to determine how close the speculative weight set is to the actual weights calculated from the average weight gradient. If the difference between the two weight sets is at or below the threshold difference, the speculative weights are close enough to the actual weights, and the training process can continue using the results obtained from the second iteration of the training process using the speculative weight set. In other words, processing node 704-1 can continue training the neural network model using the results obtained in forward propagation 722-1 and backpropagation 724-1 using the speculative weight set, and processing node 704-1 can continue to perform further iterations of the training process.

[0077] like Figure 7As shown, the training process can continue as follows: Local weight updates (728-1) are performed using the set of local weight gradients determined from backpropagation (724-1) so that a third iteration of the training process can be initiated on the batch 3-1 input dataset while waiting for the next set of weights to become available from the gradient exchange and weight comparison process (726-1). The training process can be repeated an arbitrary number of iterations until the error of the neural network model across the training dataset decreases to a certain error threshold, or reaches a minimum error.

[0078] By Figure 7 and Figure 6 The comparison shows that using speculative weights derived from the local weight gradient set of the processing node itself to allow the training process to continue reduces the idle time of the neural network's computational circuitry between the backpropagation of the previous iteration and the forward propagation of the next iteration. Subsequent training iterations can be initiated without waiting for gradient swapping and global weight updates to complete. This improves the overall throughput of the system and allows for faster training across the entire training dataset.

[0079] Figure 8 The figure shows a timing diagram illustrating a further example of the training process performed in the processing node of a neural network training system. Figure 8 The initial part of the training process shown is related to Figure 7 The initial part of the training process shown is similar, and therefore there is no need to repeat its description. Figure 8 The training process shown is the same as Figure 7 The difference in the training process lies in the gradient exchange and weight comparison process, as shown in the results of 826-1.

[0080] exist Figure 8 In the training process shown, the following two differences are identified as exceeding a threshold difference: (1) the speculative weight set derived from the set of local weight gradients of the processing node 804-1 itself obtained during backpropagation 824-1, and (2) the weight set obtained from the gradient exchange procedure 826-1 (which is derived from the average weight gradients of the processing nodes across the system). In this case, the speculative weights may differ too much from the actual weights that should be used in the next iteration of the training process. Therefore, instead of continuing the training process using the results obtained from the computation performed on the batch 3-1 input dataset using the speculative weight set, the iteration of the training process on the batch 3-1 input dataset is restarted using the weight set obtained from the gradient exchange procedure 826-1. Thus, as Figure 8 As shown, processing node 804-1 can terminate forward propagation 832-1a using a speculative weight set, and restart the training iteration by repeating the computation using the weight set obtained from the gradient exchange process 826-1, utilizing forward propagation 832-1b.

[0081] While restarting the training process iteratively may introduce some latency during training, the latency attributable to restarting the training iteration is not expected to be significantly greater than the latency of waiting for the next set of weights to become available from the gradient exchange process. Furthermore, the number of instances where the training iteration must be restarted is expected to be fewer than the number of instances where speculative weights are acceptable. Therefore, despite the few potential scenarios where training iterations must be repeated, using speculative weights to reduce idle time in the system can still improve the overall training time of the neural network.

[0082] In some implementations, because each processing node in the system can continue updating its weight set using its own local weight gradient instead of the average weight gradient across all processing nodes, it is possible that each processing node will end up with a different set of weight values ​​as the training process progresses. Although the differences between the different weight sets in the processing nodes are expected to be small (e.g., within a threshold difference), the system can improve the likelihood of convergence by performing a weight synchronization process after a predetermined number of iterations of the training process to synchronize the weight values ​​across all processing nodes in the neural network training system. For example, a weight synchronization process can be performed after ten iterations of the training process. The weight synchronization process updates the weight set in each processing node to a weight set derived based on the average weight gradient of the most recent training iteration. In this way, the system can ensure that each processing node will revert to a consistent weight set during training.

[0083] Figure 9 The diagram illustrates a flowchart of an example training process 900 that can be executed in a neural network training system with multiple processing nodes, according to some implementations. Process 900 can be executed in any one or more processing nodes of the neural network training system. In some implementations, each processing node of the neural network training system may use a neural network processor, neural network accelerator, graphics processing unit (GPU), field-programmable gate array (FPGA), coprocessor, or application-specific integrated circuit (ASIC) to implement neural network computation circuitry to perform neural network computations, such as matrix multiplication or dot product. In some implementations, each processing node may be a computing device incorporating one or more of these components to perform neural network computations. The processing nodes of the neural network training system may be communicatively coupled to each other using chip or system interconnects, via networks, or combinations thereof. Furthermore, not every processing node of the neural network training system needs to use the same type of hardware to perform neural network computations. For example, the neural network training system may include a GPU to perform computations for one processing node and an FPGA to perform computations for another processing node, and so on.

[0084] Process 900 may begin at box 902: The processing node performs a first iteration of the training process using a first set of weights to generate a first output dataset. The input dataset used to generate the first output dataset may be a subset or part of the complete training dataset. The first iteration of the training process may include a forward propagation process, in which computations performed on the input dataset are propagated from the lowest layer to the highest layer of the neural network model to generate the first output dataset. The first output dataset may be compared with a reference output dataset (e.g., the expected output) to determine the error of the neural network model operating on that particular input dataset. The first iteration of the training process may also include a backpropagation process, in which the error is propagated back from the highest layer to the lowest layer of the neural network model to determine how the weight values ​​at each layer should be adjusted to improve the accuracy of the neural network model.

[0085] At box 904, a set of local weight gradients can be derived by the processing node based on a comparison of the first output dataset with the reference output dataset during the backpropagation process. The set of local weight gradients can include the weight gradient of each of the weight values ​​in the neural network model. Each weight gradient can be represented as the partial derivative of the model's cost function (e.g., the sum of squared errors, the average of squared errors, etc.) with respect to the corresponding weight.

[0086] At box 906, the first set of weights used in the first iteration of the training process can be updated using the set of local weight gradients of the processing nodes obtained in box 904 to derive a speculative set of weights for the processing nodes. For example, each speculative weight value can be derived by multiplying the corresponding local weight gradient by a scalar representing the learning rate and adding the result to the corresponding weight value in the first set of weights. The weight update performed at box 906 can be performed locally at the processing node and does not need to consider the local weight gradients from other processing nodes in the system. Therefore, the update at box 906 can be performed much faster than waiting for the gradient exchange process between processing nodes to complete.

[0087] At box 908, while waiting to obtain the second set of weights for the second iteration of the training process, the second iteration of the training process is performed using the neural network computation circuitry and the speculative weight set to generate a second output dataset. Similar to the first iteration of the training process, the second iteration may include forward and backward propagation processes. The input dataset operated on in the second iteration may be the same as or different from the input dataset used in the first iteration. Unlike speculative weights, which only consider the local weight gradients of the processing nodes themselves, the second set of weights considers the local weight gradients from all processing nodes in the system, and is derived from the average set of weight gradients computed on the set of local weight gradients generated by the system's processing nodes. By initiating the second iteration of the training process using the speculative weight set instead of the second set of weights, the idle time between training iterations can be reduced because the speculative weight set can be computed faster than the time it would take for the system to perform a gradient exchange process to obtain the second set of weights.

[0088] Figure 10 The diagram illustrates a flowchart of another example of a training process 1000 that can be executed in a neural network training system with multiple processing nodes, according to some implementations. Process 1000 can be executed in each of the processing nodes in the neural network training system. In some implementations, each processing node of the neural network training system may use a neural network processor, neural network accelerator, graphics processing unit (GPU), field-programmable gate array (FPGA), coprocessor, or application-specific integrated circuit (ASIC) to implement neural network computing circuitry to perform neural network computations, such as matrix multiplication or dot product. In some implementations, each processing node may be a computing device incorporating one or more of these components to perform neural network computations. The processing nodes of the neural network training system may be communicatively coupled to each other using chip or system interconnects, via networks, or combinations thereof. Furthermore, not every processing node of the neural network training system needs to use the same type of hardware to perform neural network computations. For example, the neural network training system may include a GPU performing computations for one processing node and an FPGA for another processing node, and so on.

[0089] Process 1000 may begin at box 1002 by performing a first iteration of the training process using the neural network computation circuitry of a processing node and a first set of weights to generate a first output dataset from the input dataset associated with the processing node. The input dataset associated with the processing node may be a subset or part of the complete training dataset. In other words, the complete training dataset may be divided into batches, allowing each processing node of the system to operate on different subsets of the training data to accelerate the training process. The first iteration of the training process may include a forward propagation process, in which computations performed on the input dataset are propagated from the lowest layer to the highest layer of the neural network model to generate the first output dataset. The first output dataset may be compared to a reference output dataset (e.g., the expected output) to determine the error of the neural network model operating on that particular input dataset. The first iteration of the training process may also include a backpropagation process, in which the error is propagated back from the highest layer to the lowest layer of the neural network model to determine how the weight values ​​at each layer should be adjusted to improve the accuracy of the neural network model.

[0090] At box 1004, a set of local weight gradients for processing nodes can be derived based on a comparison of the first output dataset with the reference output dataset during the backpropagation process. The set of local weight gradients can include the weight gradient of each of the weight values ​​in the neural network model. Each weight gradient can be represented as the partial derivative of the model's cost function (e.g., the sum of squared errors, the average of squared errors, etc.) with respect to the corresponding weight.

[0091] At box 1006, the first set of weights used in the first iteration of the training process can be updated using the set of local weight gradients of the processing nodes obtained in box 1004 to derive a speculative set of weights for the processing nodes. For example, each speculative weight value can be derived by multiplying the corresponding local weight gradient by a scalar representing the learning rate and adding the result to the corresponding weight value in the first set of weights. This weight update at box 1006 can be performed locally and does not require consideration of local weight gradients from other processing nodes in the system. Therefore, the update at box 1006 can be performed much faster than waiting for the gradient exchange process between processing nodes to complete.

[0092] At box 1008, a second iteration of the training process is performed using neural network computation circuitry and a speculative set of weights to generate a second output dataset. Similar to the first iteration of the training process, the second iteration may include forward and backward propagation processes. The input dataset operated on in the second iteration may be the same as or different from the input dataset used in the first iteration.

[0093] At box 1010, a second set of weights is obtained for the second iteration of the training process. This second set of weights can be obtained either by receiving a communication containing the second set of weights from another processing node in the neural network training system or by computing the second set of weights within the processing node itself. This second set of weights is derived from the average set of weight gradients calculated from the set of local weight gradients generated by the processing nodes of the system. Therefore, unlike speculative weights that only consider the local weight gradients of the processing node itself, this second set of weights considers the local weight gradients from all processing nodes in the system. It should be noted that box 1010 can be executed concurrently with boxes 1006 and 1008, and the second iteration of the training process can be executed in box 1008 while waiting for the second set of weights to become available in the neural network training system.

[0094] In some implementations, one of the processing nodes in the neural network training system can be configured to compute a second set of weights and broadcast it to the other processing nodes in the neural network training system. For example, a gradient exchange process can be performed, where each processing node sends its local weight gradients to a designated processing node. The designated processing node can accumulate the local weight gradient sets (where each set is derived by the corresponding processing node in the neural network training system), average the local weight gradient sets to derive an average weight gradient set, and use the average weight gradient set to update the first weight set to generate the second weight set. Each of the average weight gradients can be computed by summing the corresponding local weight gradients from each processing node and dividing the sum by the number of processing nodes to derive the average weight gradient for the corresponding weight value.

[0095] In some implementations, each of the processing nodes in a neural network training system can be configured to compute a second set of weights. For example, a gradient exchange process can be performed, in which each processing node broadcasts its local set of weight gradients to the other processing nodes in the system, and each processing node in the system can then use the local weight gradients received from the other processing nodes to compute an average set of weight gradients (e.g., as described above) and update the first set of weights accordingly.

[0096] At box 1012, a second set of weights derived from the average weight gradient set is compared with the speculative weight set of the processing node to determine the difference between the two weight sets. In some implementations, the difference between the two weight sets can be determined as, for example, the average of the differences between corresponding weight values ​​from the two sets, the median of the differences between corresponding weight values ​​from the two sets, the maximum of the differences between corresponding weight values ​​from the two sets, or the root mean square of the differences between corresponding weight values ​​from the two sets. In other implementations, other metrics can be used to determine the difference between the two weight sets. This difference provides a measure of how close or far the speculative weight set is from the weights calculated using the average weight gradient.

[0097] At box 1014, the difference between the second set of weights derived from the average weight gradient set and the speculative set of weights for the processing node is compared to a threshold difference. For example, the threshold difference can be chosen based on the system's accuracy tolerance. For instance, in systems where higher inaccuracies are tolerable, the threshold difference can be set to a higher value, while in systems that may require higher accuracy, the threshold difference can be set to a lower value.

[0098] At box 1016, if it is determined that the difference between the second weight set and the speculative weight set exceeds a threshold difference, the speculative weight set may be too far from the second weight set. In such cases, the results of the second iteration of the training process using the speculative weight set can be discarded, and the second iteration of the training process can be restarted and repeated using the second weight set instead of the speculative weight set. At box 1018, if it is determined that the difference between the second weight set and the speculative weight set is at or below the threshold difference, the speculative weight set is sufficiently close to the second weight set, and the training process can continue using the second output dataset generated from the speculative weight set.

[0099] In some implementations, a weight synchronization process can be performed after a predetermined number of iterations of the training process to synchronize weight values ​​across all processing nodes of the neural network training system. For example, the weight synchronization process can be performed after ten iterations of the training process. The weight synchronization process updates the set of weights in each processing node with weight values ​​derived from the average weight gradient computed in the most recent training iteration. In this way, the system can ensure that each processing node will revert to a consistent set of weights during training.

[0100] Figure 11 The illustration shows an example of the internal components of a computing device 1100 that can be used as a processing node in a neural network training system. Figure 11As shown, the computing device 1100 may include a neural network computing circuit 1112, a processor 1114, a weight update module 1116, and a communication interface 1118. In some implementations, the weight update module 1116 may be implemented as software executed by the processor 1114. In some implementations, the weight update module 1116 may be implemented using an application-specific integrated circuit (ASIC) or integrated as part of the neural network computing circuit 1112.

[0101] The neural network computing circuit 1112 can be implemented using a neural network processor, a neural network accelerator, a graphics processing unit (GPU), a field-programmable gate array (FPGA), a coprocessor, or an application-specific integrated circuit (ASIC). The neural network computing circuit 1112 can perform forward propagation and backward propagation operations. After the neural network computing circuit 1112 generates local weight gradients for the neural network layers, these local weight gradients can be sent to the processor 1114, which manages the exchange operation of the weight gradients. As part of the exchange operation, the processor 1114 can forward the local weight gradients to the communication interface 1118 for transmission to other processing nodes via a computer network.

[0102] When the weight update module 1116 receives a local weight gradient from the neural network computing circuit 1112, it can use the local weight gradient to calculate speculative weights and send these speculative weights as updated weights to the neural network computing circuit 1112, allowing training for the next batch of data to begin. When the processor 1114 receives a global weight update, the weight update module 1116 can compare the difference between the weight value from the global weight update and the speculative weight value. If the difference is at or below a threshold difference, the training process can continue using the speculative weights. If the difference exceeds the threshold difference, the weight update module 1116 can send the weight value from the global weight update to the neural network computing circuit 1112 and restart the training iteration using the weights from the global weight update.

[0103] In some implementations, processor 1114 may also receive weight gradients from other processing nodes via communication interface 1118. Weight update module 1116 may accumulate and average the weight gradients from the system's processing nodes to generate the average weight gradient of the neural network layer, use the average weight gradient to calculate updated weight values, and determine whether the training process can continue using speculative weights or be updated using weight values ​​derived from the average weight gradient.

[0104] Figure 12 The diagram shows a block diagram of another example of a computing device 1200 that can be used as a processing node in a neural network training system. The various components of the computing device 1200 are referenced above. Figure 11 The components discussed are similar, and therefore there is no need to repeat detailed descriptions of these components. Computing device 1200 and... Figure 11 The computing device 1100 shown differs from the computing device 1200 because the computing device 1200 further includes a weight comparison circuit 1222. For example, the weight comparison circuit 1222 can be implemented using an arithmetic logic unit (ALU) or other suitable circuitry dedicated to comparing and / or calculating weight values. The weight comparison circuit 1222 may include circuitry configured to compare the difference between a speculative set of weights calculated from a local gradient set of a processing node and a set of weights derived from local gradient sets of all processing nodes in the system with a difference threshold. In some implementations, the weight comparison circuit 1222 can also be used to calculate the difference between the two sets of weights. In some implementations, the weight comparison circuit 1222 can be further used to calculate speculative weight values ​​from the local weight gradients of processing nodes, and / or to calculate globally updated weight values ​​from the local weight gradients of all processing nodes.

[0105] By incorporating the weight comparison circuit 1222 into the computing device 1200, computations involving comparing and / or calculating weight values ​​can be offloaded from other components of the computing device 1200 (e.g., neural network computing circuit 1212 and / or processor 1214) to the weight comparison circuit 1222. The use of the weight comparison circuit 1222 can further reduce training time because the neural network computing circuit 1212 can continue training computations without interruption while the weight comparison circuit 1222 is comparing and / or calculating weight values. In this implementation, the weight update module 1216 may only need to transmit local weight gradients to other processing nodes in the system and receive global weight updates from the training system.

[0106] Figure 13 This is a block diagram illustrating an example of an integrated circuit device that can be used as a neural network computing circuit. Figure 13 The example illustration shows accelerator 1302. In various examples, for a set of input data (e.g., input data 1350), accelerator 1302 can perform computations using processing engine array 1310, activation engine 1316, and / or pooling engine 1318. In some examples, example accelerator 1302 can be an integrated circuit component of a processor such as a neural network processor. The processor may have other integrated circuit components, including additional accelerator engines.

[0107] In various implementations, the memory subsystem 1304 may include multiple memory blocks 1314. In these implementations, each memory block 1314 may be independently accessible, meaning that a read of one memory block does not depend on a read of another memory block. Similarly, a write to one memory block does not affect or limit writes to different memory blocks. In some cases, each memory block may be read and written simultaneously. Various techniques can be used to have independently accessible memory blocks 1314. For example, each memory block may be a physically separate memory component with an address space separate and independent from the address space of each other memory block. In this example, each memory block may have at least one read channel and may have at least one separate write channel that can be used simultaneously. In these examples, the memory subsystem 1304 may allow simultaneous access to the read or write channels of multiple memory blocks. As another example, the memory subsystem 1304 may include arbitration logic such that arbitration, for example, between the outputs of multiple memory blocks 1314, may result in the use of the output of more than one memory block. In these and other examples, although managed globally by the memory subsystem 1304, each memory block can operate independently of each other.

[0108] Making memory block 1314 independently accessible can improve the efficiency of accelerator 1302. For example, values ​​can be read and served to each row of processing engine array 1310 simultaneously, allowing the entire processing engine array 1310 to be in use within a single clock cycle. As another example, memory block 1314 can be read while the results computed by processing engine array 1310 are being written to memory subsystem 1304. Conversely, a single memory may only be able to serve one read or write at a time. For example, in the case of a single memory, multiple clock cycles might be required to read input data for each row of processing engine array 1310 before processing engine array 1310 can be started.

[0109] In various implementations, the memory subsystem 1304 can be configured to serve multiple clients simultaneously, including the processing engine array 1310, the activation engine 1316, the pooling engine 1318, and any external clients accessing the memory subsystem 1304 via the communication structure 1320. In some implementations, the ability to serve multiple clients may mean that the memory subsystem 1304 has at least as many memory blocks as the number of clients. In some cases, each row of the processing engine array 1310 can be considered a separate client. In some cases, each column of the processing engine array 1310 can output results, making each column a separate write client. In some cases, output from the processing engine array 1310 can be written to memory block 1314, which can then be used to provide input data for the processing engine array 1310. As another example, the activation engine 1316 and the pooling engine 1318 may include multiple execution channels, each of which can be a separate memory client. For example, memory block 1314 can be implemented using static random access memory (SRAM).

[0110] In various implementations, the memory subsystem 1304 may include control logic. For example, the control logic may maintain a track of the address space of each of the memory blocks 1314, identify memory blocks 1314 to be read from or written to, and / or move data between memory blocks 1314. In some implementations, memory blocks 1314 may be hardwired to specific clients. For example, a set of memory blocks 1314 may be hardwired to provide values ​​to rows of the processing engine array 1310, with one memory block serving each row. As another example, a set of memory blocks may be hardwired to receive values ​​from columns of the processing engine array 1310, with one memory block receiving data for each column.

[0111] The processing engine array 1310 is the computation matrix of the example accelerator 1302. For example, the processing engine array 1310 can perform parallel integration, convolution, correlation, and / or matrix multiplication, etc. The processing engine array 1310 includes multiple processing engines 1311 arranged in rows and columns, such that the output of one processing engine 1311 can be directly input into another processing engine 1311. Therefore, a processing engine 1311 not located on the outer edge of the processing engine array 1310 can receive data to be operated on from other processing engines 1311 instead of from the memory subsystem 1304.

[0112] In various examples, the processing engine array 1310 uses pulsed execution, where data arrives at each processing engine 1311 from different directions at regular intervals. In some examples, input data can flow into the processing engine array 1310 from the left, and weight values ​​can be loaded at the top. In some examples, weights and input data can flow from the left, and partial sums can flow from the top to the bottom. In these and other examples, multiplication and accumulation operations move diagonally across the processing engine array 1310 as wavefronts, where data moves right and down across the array. Control signals can be input simultaneously with the weights at the left and can flow down across the computation and along with the computation.

[0113] In various implementations, the number of columns in the processing engine array 1310 determines the computing power of the processing engine array 1310, and the number of rows determines the memory bandwidth required to achieve maximum utilization of the processing engine array 1310. For example, the processing engine array 1310 may have 64 columns and 428 rows, or some other number of columns and rows.

[0114] Example of processing engine 1311 in Figure 13 The illustration shows that, as illustrated in the figure, processing engine 1311 may include multiplier-accumulator circuitry. For example, the input from the left may include input data i and weight values ​​w, where the input data is a value derived from a set of input data or a set of intermediate results, and the weight values ​​are a set of weights that connect one layer of a neural network to the next. For example, the set of input data may be an image submitted for identification or object recognition, an audio clip provided for speech recognition, a text string for natural language processing or machine translation, or the current state of a game requiring analysis to determine the next action, etc. In some examples, the input data and weight values ​​are output to the right to be fed into the next processing engine 1311.

[0115] In the example shown, the input from above can include a partial sum p_in, which is provided by another processing engine 1311 or by the previous round of computation of the processing engine array 1310. When computation begins for a new set of input data, the top row of the processing engine array 1310 can receive a fixed value for p_in, such as zero. As shown in this example, i and w are multiplied, and the result is added to p_in to produce a new partial sum p_out, which can be input into another processing engine 1311. Various other implementations of the processing engine 1311 are possible.

[0116] The output from the last row in the processing engine array 1310 can be temporarily stored in the result buffer 1312. The result can be an intermediate result, which can be written to memory block 1314 to be provided to the processing engine array 1310 for additional computation. Alternatively, the result can be a final result, which, once written to memory block 1314, can be read from memory subsystem 1304 via communication structure 1320 for use as system output.

[0117] In some implementations, accelerator 1302 includes activation engine 1316. In these implementations, activation engine 1316 can combine results from processing engine array 1310 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce a single-channel output activation. In other examples, it may be necessary to accumulate results from one or more columns in processing engine array 1310 to produce the output activation of a single node in the neural network. In some examples, activation engine 1316 can be bypassed.

[0118] In various examples, activation engine 1316 may include multiple separate execution channels. In these examples, the execution channels may correspond to columns of processing engine array 1310 and may perform operations on the output of the columns, the results of which may be stored in memory subsystem 1304. In these examples, activation engine 1316 may be able to perform parallel computations between 1 and n, where n is equal to the number of columns in processing engine array 1310. In some cases, one or more computations may be performed simultaneously. Examples of computations that each execution channel can perform include exponentiation, squares, square roots, identities, binary steps, bipolar steps, sigmoid types, and ramps, etc.

[0119] In some implementations, accelerator 1302 may include pooling engine 1318. Pooling is the combination of the outputs of columns from processing engine array 1310. For example, combination may include calculating the maximum, minimum, average, median, sum, multiplication, or another logical or mathematical combination. In various examples, pooling engine 1318 may include multiple execution channels that can operate on the values ​​from corresponding columns of processing engine array 1310. In these examples, pooling engine 1318 may be able to perform parallel computations between 1 and n, where n equals the number of columns in processing engine array 1310. In various examples, the execution channels of pooling engine 1318 may operate in parallel and / or simultaneously. In some examples, pooling engine 1318 may be bypassed.

[0120] Here, activation engine 1316 and pooling engine 1318 can be collectively referred to as execution engines. Processing engine array 1310 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine, which may be located outside of accelerator 1302.

[0121] Input data 1350 can be reached via communication structure 1320. Communication structure 1320 can connect accelerator 1302 to other components of the processor, such as a DMA engine that can obtain input data 1350 from input / output (I / O) devices, memory drives, or network interfaces. For example, input data 1350 can be one-dimensional data such as strings or sequences of numbers, or two-dimensional data such as an array of pixel values ​​of an image or an array of frequency and amplitude values ​​of an audio signal over time. In some examples, input data 1350 can be three-dimensional, for example, in the case of contextual information or virtual reality data used by an autonomous vehicle. In some implementations, memory subsystem 1304 may include a separate buffer for input data 1350. In some implementations, when accelerator 1302 receives input data 1350, input data 1350 can be stored in memory block 1314.

[0122] In some examples, accelerator 1302 can implement a neural network processing engine. In these examples, for a set of input data 1350, accelerator 1302 can execute a neural network to perform a task, the neural network being trained for that task. Executing a neural network on a set of input data can be referred to as inference or performing inference.

[0123] The weights of the neural network and the input data 1350 that the neural network will operate on can be stored in the memory subsystem 1304. The neural network can also include instructions that program the processing engine array 1310 to perform various computations on the weights and input data. These instructions can also be stored in the memory subsystem 1304, in memory block 1314, or in a separate instruction buffer. The processing engine array 1310 can output intermediate results representing the outputs of individual layers of the neural network. In some cases, the activation engine 1316 and / or pooling engine 1318 can be enabled for computations invoked by certain layers of the neural network. The accelerator 1302 can store intermediate results in the memory subsystem 1304 for input into the processing engine array 1310 to compute the results of the next layer of the neural network. The processing engine array 1310 can further output the final result from the last layer of the neural network. The final result can be stored in the memory subsystem 1304 and then copied out to host processor memory or to another location.

[0124] Figure 14 This includes a block diagram illustrating an example of an acceleration engine 1400. The acceleration engine 1400 is an example of an integrated circuit that may include components that can interact with... Figure 11 The accelerator shown is similar to one or more accelerators 1402a-1402n.

[0125] exist Figure 14 In the examples, the acceleration engine 1400 includes multiple accelerators 1402a-1402n, each of which can perform a set of operations. In various examples, accelerators 1402a-1402n are used for specific types of operations, allowing them to perform operations much faster than when similar operations are performed by a general-purpose processor. In various examples, to perform a set of operations, the input data for which the operations are to be performed must first be moved into accelerators 1402a-1402n. Furthermore, in some cases, program code is also moved into accelerators 1402a-1402n, which programs the operations that accelerators 1402a-1402n will perform on the data. In the example shown, the acceleration engine 1400 includes n accelerators 1402a-1402n. Examples of accelerators that can be included in the acceleration engine 1400 include graphics accelerators, floating-point accelerators, neural network accelerators, and other accelerators. In various examples, accelerators 1402a-1402n can each be the same (e.g., each of them is a graphics accelerator) or they can be different (e.g., accelerators 1402a-1402n include a graphics accelerator, a floating-point accelerator, and a neural network accelerator).

[0126] The example acceleration engine 1400 further includes DRAM controllers 1442a-1442k for communicating with external memory. In this example, the external memory is implemented using DRAM 1430. In the illustrated example, the acceleration engine 1400 includes k DRAM controllers 1442a-1442k, each of which may be able to communicate with an independent set of blocks of DRAM. In other examples, other types of RAM technologies may be used for the external memory. The DRAM controllers 1442a-1442k may also be referred to as memory controllers.

[0127] In various examples, the input data and / or program code for accelerators 1402a-1402n can be stored in DRAM 1430. Different programs can cause accelerators 1402a-1402n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program can configure the neural network accelerator to perform speech recognition, while another program can configure the neural network accelerator to perform image recognition. In various examples, different accelerators 1402a-1402n can be programmed with different programs, causing each accelerator to perform a different set of operations. In various examples, processors 1448a-1448s can manage the movement of program code from DRAM 1430 to accelerators 1402a-1402n.

[0128] The example acceleration engine 1400 further includes I / O controllers 1444a-1444p for communicating with I / O devices 1432 in the system. For example, the acceleration engine 1400 can communicate with I / O devices via a processor bus. In some examples, the processor bus can be implemented using Peripheral Component Interconnect (PCI) and / or a variant of the PCI bus protocol. The processor bus can connect the acceleration engine 1400 to I / O devices (such as, for example, input and output devices), memory controllers, storage devices, and / or network interface cards, etc. In some examples, I / O controllers 1444-1444p can enable the acceleration engine 1400 to act as an I / O device for the host processor. For example, the acceleration engine 1400 can be a recipient of input data from the host processor and commands instructing the operation to be performed on the input data (e.g., a specific calculation or analysis). In the example shown, the acceleration engine 1400 includes p I / O controllers 1444a-1444p, each of which may include a separate root complex and can communicate with a separate set of I / O devices 1432. In other examples, other standardized bus protocols such as Hyperpath Interconnect (UPI) may be used on the host bus. In other examples, proprietary bus protocols may be used.

[0129] The movement of data within the acceleration engine 1400 can be managed by one or more processors 1448a-1448s, which may also be referred to as data management processors. Figure 14In one example, the acceleration engine 1400 includes S processors 1448a-1448s incorporated into the device (e.g., on the same silicon die). In other examples, the processors 1448a-1448s may be external to the acceleration engine 1400 (e.g., on a different die and / or in a different package). In some examples, the processors 1448a-1448s may manage the movement of data from I / O device 1432 to accelerators 1402a-1402n or DRAM 1430. For example, input data may be located at I / O device 1432 or in processor memory, and the processors 1448a-1448s may move input from I / O device 1432 or processor memory to the accelerator or to DRAM 1430. As another example, the program code for accelerators 1402a-1402n may be located on I / O device 1432 or in processor memory.

[0130] Example acceleration engine 1400 further includes DMA engines 1446a-1446d, which can move data between accelerators 1402a-1402n, DRAM controllers 1442a-1442k, and I / O controllers 1444a-1444p. In the illustrated example, acceleration engine 1400 includes d DMA engines 1446a-1446d. In some implementations, DMA engines 1446a-1446d can be assigned to specific tasks, such as moving data from DRAM controllers 1442a-1442d to accelerators 1402a-1402n, or moving data between I / O controllers 1444a-1444p and accelerators 1402a-1402n. For example, these tasks can be assigned by queuing descriptors for DMA engines 1446a-1446d, where the descriptors identify the address of a data block and the operation to be performed (e.g., read or write). For example, a descriptor can instruct the DMA engine to read data blocks from DRAM 1430. As a further example, a descriptor can instruct the DMA engine to write data read by the DMA controller to the accelerator. Further descriptors can be used to move data from the accelerator to DRAM 1430.

[0131] In various examples, each of processors 1448a-1448s can be responsible for managing data movement to different accelerators. In some examples, a processor can manage data movement to more than one accelerator. Similarly, in various examples, each of processors 1448a-1448s can be assigned to one or more DMA engines 1446a-1446d. In these and other examples, the association between processors 1448a-1448s, accelerators 1402a-1402n, and DMA engines 1446a-1446d is determined by the program code executed by each corresponding processor.

[0132] In the example acceleration engine 1400, various components can communicate via chip interconnect 1420. Chip interconnect 1420 primarily includes wiring for routing data between the components of acceleration engine 1400. In some cases, chip interconnect 1420 may include a minimal amount of logic, such as multiplexers for controlling data direction, flip-flops for handling clock domain crossings, and timing logic.

[0133] Figure 15 The block diagram includes an example of a host system 1500, in which an acceleration engine 1560 can be used. Figure 15 The acceleration engine 1560 is an example of a device that may include one or more accelerators, such as Figure 12 As shown. Figure 15 The example host system 1500 includes an acceleration engine 1560, a host processor 1572, DRAM 1530 or processor memory, I / O devices 1532, and a support system 1574. In various implementations, the host system 1500 may include other hardware not shown herein.

[0134] The host processor 1572 is a general-purpose integrated circuit capable of executing program instructions. In some examples, the host processor 1572 may include multiple processing cores. A multi-core processor may include multiple processing units within the same processor. In some examples, the host system 1500 may include more than one host processor 1572. In some examples, the host processor 1572 and the acceleration engine 1560 may be a single chip, such as one or more integrated circuits within the same package.

[0135] In various examples, host processor 1572 may communicate with other components in host system 1500 via one or more communication channels. For example, host system 1500 may include a host processor bus, which host processor 1572 may use to communicate with DRAM 1530. As another example, host system 1500 may include an I / O bus, such as a PCI-based bus, through which host processor 1572 may communicate with acceleration engine 1560 and / or I / O device 1532. In various examples, alternatively or additionally, host system 1500 may include other communication channels or buses, such as a serial bus, power management bus, storage device bus, etc.

[0136] In some examples, a software program executing on host processor 1572 can receive or generate input for processing by acceleration engine 1560. In some examples, the program can select an appropriate neural network to execute for a given input. For example, the program can be used for language translation and can select one or more neural networks capable of performing speech recognition and / or machine translation. In these and other examples, the program can configure acceleration engine 1560 with the neural network to be executed and / or select a neural network processing engine on acceleration engine 1560 that has been previously configured to execute the desired neural network. In some examples, once acceleration engine 1560 has begun inference on the input data, host processor 1572 can manage the movement of data (such as weights, instructions, intermediate results, results of conditional layers, and / or final results) into or out of acceleration engine 1560.

[0137] In some examples, a software program using the acceleration engine 1560 for inference can read results from the conditional layer from the acceleration engine 1560 and / or from a storage location (such as DRAM 1530). In these examples, the program can determine what action the neural network should take next. For example, the program can determine to terminate the inference. As another example, the program can determine to change the direction of the inference, which can be shifted to the next layer by lower-level code and / or the neural network processor. In these and other examples, the execution flow of the neural network can be coordinated by software.

[0138] DRAM 1530 is a memory used by host processor 1572 for storing program code being executed by host processor 1572 and the values ​​being manipulated. In some examples, data for neural networks (e.g., weight values, instructions, and other data) may be stored wholly or partially in DRAM 1530. DRAM is a common term for processor memory; while DRAM is volatile memory, processor memory can be volatile and / or non-volatile. Although not shown here, host system 1500 may include other volatile and non-volatile memories for other purposes. For example, host system 1500 may include read-only memory (ROM) storing boot code for booting host system 1500 upon power-up and / or storing Basic Input / Output System (BIOS) code.

[0139] Although not shown here, DRAM 1530 can store instructions for various programs that can be loaded into and executed by host processor 1572. For example, DRAM 1530 can store instructions for: operating system, one or more data stores, one or more applications, one or more drives, and / or services for implementing the features disclosed herein.

[0140] The operating system can manage and orchestrate the overall operation of the host system 1500, such as scheduling tasks, executing applications, and / or controlling peripheral devices, as well as other operations. In some examples, the host system 1500 can control one or more virtual machines. In these examples, each virtual machine can be configured to run its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. Alternatively or additionally, the operating system can be a proprietary operating system.

[0141] Data storage can include persistent or transient data used and / or manipulated by an operating system, applications, or drives. Examples of such data include web pages, video data, audio data, images, user data, etc. In some examples, the information in the data storage can be made available to user devices via networks(s). In some cases, additionally or alternatively, data storage may include stored applications and / or drives. Alternatively or additionally, data storage may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data storage may be machine-readable object code, source code, interpreted code, or intermediate code.

[0142] A drive may include programs that facilitate communication between components within the host system 1500. For example, some drives may facilitate communication between the operating system and peripheral devices or I / O devices 1532. Alternatively or additionally, some drives may facilitate communication between an application and the operating system, and / or between an application and peripheral devices accessible to the host system 1500. In many cases, a drive may include a drive that provides well-understood functionality (e.g., a printer drive, a monitor drive, a hard disk drive, a solid-state drive, etc.). In other cases, a drive may provide proprietary or specialized functionality.

[0143] I / O device 1532 may include hardware for connecting to user input and output devices such as keyboards, mice, pens, tablets, voice input devices, touch input devices, displays or monitors, speakers, printers, and other devices. I / O device 1532 may also include storage drives and / or network interfaces for connecting to network 1580. For example, host system 1500 may use the network interface to communicate with storage devices, user terminals, other computing devices or servers, and / or other networks, and various other examples.

[0144] In various examples, one or more of the I / O devices 1532 may be storage devices. In these examples, the storage devices include non-volatile memory and may store program instructions and / or data. Examples of storage devices include magnetic storage, optical disks, solid-state drives, flash memory, and / or magnetic tape storage, as well as other storage devices. The storage devices may be housed in the same rack as the host system 1500, or they may be in an external enclosure. The storage devices may be fixed (e.g., attached by screws) or removable (e.g., having a physical release mechanism and possibly a hot-swappable mechanism).

[0145] Storage devices, DRAM 1530, and any other memory components in host system 1500 are examples of computer-readable storage media. A computer-readable storage medium is a physical medium capable of storing data in a format readable by a device such as host processor 1572. A computer-readable storage medium can be non-transitory. A non-transitory computer-readable medium can retain stored data thereon when no power is applied to it. Examples of non-transitory computer-readable media include ROM devices, disks, magnetic tapes, optical discs, flash memory devices, and solid-state drives. As used herein, computer-readable storage media do not include computer-readable communication media.

[0146] In various examples, data stored on a computer-readable storage medium may include program instructions, data structures, program modules, libraries, other software program components, and / or other data that may be transmitted within a data signal such as a carrier wave or other transmission. Additionally or alternatively, a computer-readable storage medium may include documents, images, videos, audio, and other data that can be operated or manipulated using a software program.

[0147] In various examples, one or more of the I / O devices 1532 may be PCI-based devices. In these examples, the PCI-based I / O devices include a PCI interface for communicating with the host system 1500. The term “PCI” or “PCI-based” can be used to describe any protocol in the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and PCI Express (PCIe), or any other improved or derived protocols based on the PCI protocols discussed herein. PCI-based protocols are standard bus protocols used to connect devices such as local peripherals to host devices. Standard bus protocols are data transfer protocols for which specifications have been defined and adopted by manufacturers. Manufacturers ensure that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using Non-Volatile Memory Express (NVMe). NVMe is a device interface specification for accessing non-volatile storage media attached to a computing system using PCIe.

[0148] PCI-based devices may include one or more functions. A "function" describes the hardware and / or software that can provide operation by a PCI-based device. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some implementations, a PCI-based device may have up to eight functions.

[0149] In some examples, PCI-based devices may include Single Root I / O Virtualization (SR-IOV). SR-IOV is an extension capability that can be included in PCI-based devices. SR-IOV allows physical resources (e.g., a single network interface controller) to appear as multiple virtual resources (e.g., sixty-four network interface controllers). Therefore, a PCI-based device (e.g., a network interface controller) providing a certain function can appear as multiple devices providing the same function to devices using that PCI-based device. The functionality of storage adapter devices with SR-IOV capabilities can be classified as physical functions (PF) or virtual functions (VF). Physical functions are fully characteristic functions of a device that can be discovered, managed, and manipulated. Physical functions have configuration resources that can be used to configure or control the storage adapter device. Physical functions include the same configuration address space and memory address space that non-virtualized devices would have. A physical function can have multiple virtual functions associated with it. Virtual functions are similar to physical functions, but virtual functions are lightweight functions that may often lack configuration resources and are typically controlled by the configuration of their underlying physical functions. Each of the physical and / or virtual functions can be assigned to a corresponding execution thread (such as, for example, a virtual machine) running on the host device.

[0150] In various implementations, the support system 1574 may include hardware for coordinating the operation of the acceleration engine 1560. For example, the support system 1574 may include a microprocessor that coordinates the activities of the acceleration engine 1560, including moving data around on the acceleration engine 1560. In this example, the microprocessor may be an integrated circuit that executes microcode. Microcode is program code that allows the integrated circuit to have some flexibility in the operations it can perform, but because the program code uses a limited instruction set, the microprocessor may have more limited capabilities than the host processor 1572. In some examples, the program executed by the microprocessor is stored on the microprocessor's hardware or on a non-volatile memory chip in the host system 1500. In some examples, the microprocessor and the acceleration engine 1560 may be on a single integrated circuit, such as on the same die and in the same package.

[0151] In some examples, when a program executing on host processor 1572 requests to execute a neural network, support system 1574 can be responsible for fetching instructions from host processor 1572. For example, host processor 1572 can provide support system 1574 with a set of input data and a task to be performed on that set of input data. In this example, support system 1574 can identify the neural network capable of performing the task and can program acceleration engine 1560 to execute the neural network on the set of input data. In some examples, support system 1574 only needs to select the appropriate neural network processing engine for the neural network processor. In some examples, support system 1574 may need to load data for the neural network onto acceleration engine 1560 before acceleration engine 1560 can begin executing the neural network. In these and other examples, support system 1574 can further receive the output of executing the neural network and provide the output back to host processor 1572.

[0152] In some examples, the operation of supporting system 1574 can be handled by host processor 1572. In these examples, supporting system 1574 may not be required, and supporting system 1574 can be omitted from host system 1500.

[0153] In various examples, host system 1500 may include a combination of host system, processor node, storage subsystem, and I / O racks representing user equipment, service provider computer, or third-party computer.

[0154] User devices may include computing devices used to access applications (e.g., web browsers or mobile device applications). In some examples, the applications may be controlled, managed, and / or provided by a computing resource service or service provider. Applications may enable users to interact with the service provider's computer to, for example, access web content (e.g., web pages, music, videos, etc.). User devices may be computing devices such as, for example, mobile phones, smartphones, personal digital assistants (PDAs), laptops, netbooks, desktop computers, thin client devices, tablets, e-book readers, game consoles, etc. In some examples, user devices may communicate with the service provider's computer over one or more networks. Furthermore, user devices may be part of a distributed system that is managed by, controlled by, or otherwise integrated with the service provider's computer (e.g., a console device integrated with the service provider's computer).

[0155] The host system 1500 may also represent one or more service provider computers. Service provider computers can provide native applications configured to run on user devices, allowing users to interact with them. In some examples, service provider computers can provide computing resources such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, e-content performance management, etc. Service provider computers may also be operable to provide web hosting, database operations, computer application development and / or implementation platforms, combinations of the foregoing, etc. In some examples, service provider computers may be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that can be rapidly provisioned and released. These computing resources may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. Service provider computers may include one or more servers, perhaps deployed in a cluster as a server farm, or as individual servers that are not interconnected, and may host applications and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some examples, additionally or alternatively, the service provider computer may include computing devices such as, for example, mobile phones, smartphones, personal digital assistants (PDAs), laptops, desktop computers, netbooks, server computers, thin client devices, tablet computers, game consoles, etc. In some instances, the service provider computer may communicate with one or more third-party computers.

[0156] Figure 16 This includes a diagram of an example network 1600, which may include one or more host systems, such as Figure 15 The host system shown. For example. Figure 16 Example network 1600 includes multiple nodes 1602a-1602h, one or more of which can be, for example, Figure 15 The host system is shown. The other nodes in nodes 1602a-1602h may be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing instructions, and a network interface for connecting to network 1600.

[0157] In various examples, network 1600 can be used to process data. For example, input data can be received at one of nodes 1602a-1602h or from other networks 1608 with which network 1600 can communicate. In this example, the input data can be directed to a node in network 1600 that includes an acceleration engine, where the acceleration engine operates on it and produces results. The results can then be transmitted to the node from which the input data was received or to another network. In various examples, input data from various sources can be accumulated, including nodes 1602a-1602h and / or one or more computing devices located in other networks 1608, and the accumulated input data can be directed to one or more host systems in network 1600. The results from the host systems can then be distributed back to the sources from which the input data was collected.

[0158] In various examples, one or more of nodes 1602a-1602h may be responsible for operations such as accumulating input data for host systems to operate, keeping track of which host systems are busy and which host systems can accept more work, determining whether host systems are operating correctly and / or most efficiently, monitoring network security, and / or other administrative operations.

[0159] exist Figure 16 In the example, nodes 1602a-1602h are interconnected using a switched architecture with point-to-point links. The switched architecture includes multiple switches 1604a-1604d, which can be deployed in a multi-layered network such as a Clos network. A network device that filters packets and forwards them between segments of a local area network (LAN) can be called a switch. Switches typically operate at the data link layer (Layer 2) of the Open Systems Interconnection (OSI) reference model, and sometimes at the network layer (Layer 3) of the OSI reference model, and can support several packet protocols. Figure 16 The 1604a-1604d switches can connect to nodes 1602a-1602h and provide multiple paths between any two nodes.

[0160] Network 1600 may also include one or more network devices, such as router 1606, for connecting to other networks 1608. Routers use headers and forwarding tables to determine the best path for forwarding packets and use protocols such as Internet Control Message Protocol (ICMP) to communicate with each other and configure the best route between any two devices. Figure 16 The 1606 router can be used to connect to other networks 1608, such as subnets, LANs, wide area networks (WANs), and / or the Internet.

[0161] In some examples, network 1600 may include any one or a combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks. If present, the interconnected switches 1604a-1604d and router 1606 may be referred to as switch architecture 1610, architecture, network architecture, or simply network. In the context of computer networks, the terms "architecture" and "network" are used interchangeably herein.

[0162] Nodes 1602a-1602h can be any combination of host systems, processor nodes, storage subsystems, and I / O racks representing user equipment, service provider computers, or third-party computers.

[0163] User devices may include computing devices for accessing applications 1632 (e.g., web browsers or mobile device applications). In some aspects, application 1632 may be controlled, managed, and / or provided by a computing resource service or service provider. Application 1632 may allow (multiple) users to interact with (multiple) service provider computers to, for example, access web content (e.g., web pages, music, videos, etc.). The (multiple) user devices may be computing devices such as, for example, mobile phones, smartphones, personal digital assistants (PDAs), laptops, netbooks, desktop computers, thin client devices, tablets, e-book readers, game consoles, etc. In some examples, the (multiple) user devices may communicate with (multiple) service provider computers via (multiple) other networks 1608. Furthermore, the (multiple) user devices may be part of a distributed system managed by (multiple) service provider computers, controlled by (multiple) service provider computers, or otherwise integrated as part of (multiple) service provider computers (e.g., a console device integrated with a service provider computer).

[0164] Figure 16The (multiple) nodes may also represent one or more service provider computers. One or more service provider computers may provide native applications configured to run on user devices, with which (multiple) users may interact. In some examples, the (multiple) service provider computers may provide computing resources, such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, e-content performance management, etc. The (multiple) service provider computers may also be operable to provide (multiple) users with web hosting, database operations, computer application development and / or implementation platforms, combinations of the foregoing, etc. In some examples, the (multiple) service provider computers may be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources. These computing resources may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. The (multiple) service provider computers may include one or more servers, perhaps deployed in a cluster as a server farm, or as individual servers unrelated to each other, and may host applications and / or cloud-based software services. These servers can be configured as part of an integrated distributed computing environment. In some respects, additionally or alternatively, the service provider computers may include computing devices such as, for example, mobile phones, smartphones, personal digital assistants (PDAs), laptops, desktop computers, netbooks, server computers, thin client devices, tablet computers, game consoles, etc. In some instances, the service provider computers may communicate with one or more third-party computers.

[0165] In one example configuration, nodes(s) 1602a-1602h may include at least one memory 1618 and one or more processing units (or processors(s) 1620). The processors(s) 1620 may be implemented in hardware, computer-executable instructions, firmware, or a combination thereof. The computer-executable instructions or firmware implementation of the processors(s) 1620 may include computer-executable or machine-executable instructions written in any suitable programming language for performing the various functions described.

[0166] In some instances, the (multiple) hardware processors 1620 can be single-core or multi-core processors. A multi-core processor can include multiple processing units within the same processor. In some examples, multi-core processors can share certain resources, such as buses and second- or third-level caches. In some instances, each core in a single-core or multi-core processor can also include multiple execution logic processors (or execution threads). In such cores (e.g., those with multiple logical processors), several stages of the execution pipeline and lower-level caches can also be shared.

[0167] Memory 1618 may store program instructions that are loadable and executable on processor(s) 1620, as well as data generated during the execution of these programs. Depending on the configuration and type of nodes(s) 1602a-1602h, memory 1618 may be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). Memory 1618 may include operating system 1628, one or more data stores 1630, one or more application programs 1632, one or more drivers 1634, and / or services for implementing the features disclosed herein.

[0168] Operating system 1628 can support the basic functions of nodes 1602a-1602h, such as scheduling tasks, executing applications, and / or controlling peripheral devices. In some implementations, the service provider computer can master one or more virtual machines. In these implementations, each virtual machine can be configured to run its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. Operating system 1628 can also be a proprietary operating system.

[0169] Data storage 1630 may include permanent or transient data used and / or operated by operating system 1628, application 1632, or drive 1634. Examples of such data include web pages, video data, audio data, images, user data, etc. In some implementations, information in data storage 1630 may be made available to user devices via networks(s)1608. In some cases, additionally or alternatively, data storage 1630 may include stored applications and / or drives. Alternatively or additionally, data storage 1630 may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in data storage 1630 may be machine-readable object code, source code, interpreted code, or intermediate code.

[0170] Driver 1634 includes programs that enable communication between components within a node. For example, some drivers 1634 may enable communication between operating system 1628 and attached storage 1622, network device 1624, and / or I / O device 1626. Alternatively or additionally, some drivers 1634 may enable communication between application 1632 and operating system 1628, and / or between application 1632 and peripheral devices accessible to the service provider computer. In many cases, driver 1634 may include drivers that provide well-understood functionality (e.g., printer driver, monitor driver, hard disk drive, solid-state device driver). In other cases, driver 1634 may provide proprietary or specialized functionality.

[0171] The (multiple) service provider computers or servers may also include additional storage 1622, which may include removable storage and / or non-removable storage. The additional storage 1622 may include magnetic storage, optical disk, solid-state drive, flash memory, and / or magnetic tape storage. The additional storage 1622 may be housed in the same rack as the (multiple) nodes 1602a-1602h, or it may be housed in an external enclosure. Memory 1618 and / or additional storage 1622 and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computing devices. In some implementations, memory 1618 may include multiple different types of memory, such as SRAM, DRAM, or ROM.

[0172] Memory 1618 and additional storage 1622, in both removable and non-removable cases, are examples of computer-readable storage media. For example, a computer-readable storage medium may include volatile or non-volatile, removable or non-removable media implemented in methods or techniques for storing information, including, for example, computer-readable instructions, data structures, program modules, or other data. Memory 1618 and additional storage 1622 are examples of computer storage media. Additional types of computer storage media that may be present in nodes(s) 1602a-1602h may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, solid-state drives, or some other medium that can be used to store desired information and can be accessed by nodes(s) 1602a-1602h. Computer-readable media also include combinations of any of the above media types, comprising multiple units of one media type.

[0173] Alternatively or additionally, a computer-readable communication medium may include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave or other transmission. However, as used herein, a computer-readable storage medium does not include a computer-readable communication medium.

[0174] Multiple nodes 1602a-1602h may also include multiple I / O devices 1626, such as keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, etc. Multiple nodes 1602a-1602h may also include one or more communication channels 1636. The communication channel 1636 provides a medium through which various components of the multiple nodes 1602a-1602h can communicate. The one or more communication channels 1636 may take the form of a bus, ring, switched structure, or network.

[0175] Nodes 1602a-1602h may also include network devices 1624, which allow nodes 1602a-1602h to communicate with databases stored on network 1600, another computing device or server, user terminals and / or other devices.

[0176] In some implementations, network device 1624 is a peripheral device, such as a PCI-based device. In these implementations, network device 1624 includes a PCI interface for communicating with a host device. The term "PCI" or "PCI-based" can be used to describe any protocol in the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and PCI Express (PCIe), or any other improved or derived protocols based on the PCI protocols discussed herein. PCI-based protocols are standard bus protocols used to connect devices, such as local peripherals, to host devices. Standard bus protocols are data transfer protocols for which various manufacturers have defined and adopted specifications. Manufacturers ensure that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using Non-Volatile Memory Express (NVMe). NVMe is a device interface specification for accessing non-volatile storage media attached to a computing system using PCIe. For example, the bus interface module can implement NVMe, and the network device 1624 can be connected to the computing system using a PCIe interface.

[0177] PCI-based devices may include one or more functions. A "function" describes the operation that can be provided by the network device 1624. Examples of functions include a mass storage controller, a network controller, a display controller, a memory controller, a serial bus controller, a wireless controller, and encryption and decryption controllers. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some implementations, a PCI-based device may have up to eight functions.

[0178] In some implementations, network device 1624 may include single root I / O virtualization (SR-IOV). SR-IOV is an extension capability that can be included in PCI-based devices. SR-IOV allows physical resources (e.g., a single network interface controller) to appear as multiple resources (e.g., sixty-four network interface controllers). Therefore, a PCI-based device (e.g., a network interface controller) providing a certain function can appear as multiple devices providing the same function to devices using that PCI-based device. The functionality of a storage adapter device with SR-IOV capability can be classified as physical functions (PF) or virtual functions (VF). Physical functions are fully characteristic functions of a device that can be discovered, managed, and manipulated. Physical functions have configuration resources that can be used to configure or control the storage adapter device. Physical functions include the same configuration address space and memory address space that non-virtualized devices would have. A physical function can have multiple virtual functions associated with it. Virtual functions are similar to physical functions, but virtual functions are lightweight functions that may typically lack configuration resources and are usually controlled by the configuration of their underlying physical functions. Each of the physical and / or virtual functions can be assigned to a corresponding execution thread (such as, for example, a virtual machine) running on the host device.

[0179] Examples of implementation methods can be described in the following numbered clauses:

[0180] Clause 1. A distributed neural network training system, comprising:

[0181] Multiple processing nodes, each of which includes:

[0182] Communication interface; and

[0183] Neural network computing circuit, wherein each processing node is configured to:

[0184] The training process is first iterated using a neural network computing circuit and a first set of weights to generate a first output dataset from the input dataset associated with the processing node; a set of local weight gradients for the processing node is derived based on a comparison of the first output dataset and a reference output dataset.

[0185] The first weight set is updated using the set of local weight gradients of the processing node to derive a speculative weight set for the processing node.

[0186] The second iteration of the training process is performed using a neural network computing circuit and a speculative set of weights to generate a second output dataset.

[0187] A second set of weights is obtained for the second iteration of the training process, wherein the second set of weights is calculated from the average set of weight gradients calculated from multiple sets of local weight gradients, wherein each set of local weight gradients is derived from a corresponding processing node among multiple processing nodes;

[0188] The second set of weights is compared with the speculative set of weights for the processing nodes;

[0189] Determine whether the difference between the second weight set and the speculative weight set is at or below a threshold difference; and

[0190] The training process continues using a second output dataset generated from the speculative weight set.

[0191] Clause 2. A distributed neural network training system as described in Clause 1, wherein each of the processing nodes is further configured to perform a weight synchronization operation using a communication interface after a predetermined number of iterations in the training process, in order to synchronize weight values ​​across multiple processing nodes of the distributed neural network training system.

[0192] Clause 3. A distributed neural network training system as described in any of Clauses 1-2, wherein one of the processing nodes of the distributed neural network training system is configured to compute a second set of weights and broadcast the second set of weights to the other processing nodes of the distributed neural network training system.

[0193] Clause 4. A distributed neural network training system as described in any of Clauses 1-2, wherein each of the processing nodes of the distributed neural network training system is configured to compute a second set of weights.

[0194] Clause 5. A method comprising:

[0195] The first iteration of the training process is performed by one of the processing nodes in the neural network training system using a first set of weights to generate a first output dataset;

[0196] The processing node derives a set of local weight gradients based on a comparison between the first output dataset and the reference output dataset;

[0197] The processing node updates the first weight set using the local weight gradient set to derive the speculative weight set; and

[0198] While the processing node waits to obtain the second set of weights for the second iteration of the training process, it performs the second iteration of the training process using the speculative set of weights to generate the second output dataset.

[0199] Clause 6. The method as described in Clause 5 further includes:

[0200] Obtain the second set of weights for the second iteration of the training process; and

[0201] Determine the difference between the second set of weights and the speculative set of weights.

[0202] Clause 7. The method as described in Clause 6 further includes:

[0203] Determine whether the difference between the second weight set and the speculative weight set is at or below a threshold difference; and

[0204] The training process continues using a second output dataset generated from the speculative weight set.

[0205] Clause 8. The method as described in Clause 6 further includes:

[0206] Determine that the difference between the second set of weights and the speculative set of weights exceeds a threshold difference; and repeat the second iteration of the training process using the second set of weights instead of the speculative set of weights.

[0207] Clause 9. The method of any one of Clauses 5-8, wherein the second set of weights is obtained from another processing node of the neural network training system.

[0208] Clause 10. The method described in any of Clauses 5-8, wherein the second weight set is obtained through the following steps:

[0209] Multiple sets of local weight gradients are accumulated, and each set of local weight gradients is derived from the corresponding processing node among the multiple processing nodes of the neural network training system.

[0210] Average the local weight gradient set to derive the average weight gradient set; and

[0211] The first weight set is updated using the average weight gradient set to generate the second weight set.

[0212] Clause 11. The method as described in any of Clauses 5-8 further includes:

[0213] A weight synchronization operation is performed after a predetermined number of iterations in the training process to synchronize weight values ​​across multiple processing nodes of the neural network training system.

[0214] Clause 12. The method of any of Clauses 5-8, wherein multiple processing nodes are coupled to each other in a communicative manner via a network.

[0215] Clause 13. A computing device comprising:

[0216] Neural network computing circuit;

[0217] processor, and

[0218] Memory that stores code, which, when executed by a processor, enables a computing device to perform operations including:

[0219] The first iteration of the training process is performed using a neural network computing circuit and a first set of weights to generate a first output dataset.

[0220] The set of local weight gradients is derived by comparing the first output dataset with the expected output dataset.

[0221] The first weight set is updated using the local weight gradient set to derive the speculative weight set;

[0222] When the second set of weights, awaiting the second iteration of the training process, becomes available, the second iteration of the training process is executed using the neural network computation circuitry and the speculative weight set to generate the second output dataset; and

[0223] A second set of weights is obtained for the second iteration of the training process, wherein the second set of weights is generated by averaging multiple sets of local weight gradients, wherein each set of local weight gradients is derived by the corresponding processing node of the neural network training system.

[0224] Clause 14. The computing device as described in Clause 13, wherein operation further includes:

[0225] Determine whether the difference between the second weight set and the speculative weight set is at or below a threshold difference; and

[0226] The training process can continue using a second output dataset generated from the speculative weight set.

[0227] Clause 15. As with Clause 13, the computing device, wherein operation further includes:

[0228] Determine that the difference between the second set of weights and the speculative set of weights exceeds a difference threshold; and repeat the second iteration of the training process using the second set of weights instead of the speculative set of weights.

[0229] Clause 16. A computing device as described in any of Clauses 13-15, wherein the second set of weights is obtained from one of the processing nodes of the neural network training system.

[0230] Clause 17. A computing device as described in any of Clauses 13-15, wherein the second weight set is obtained by: accumulating multiple sets of local weight gradients; averaging the sets of local weight gradients to derive an average set of weight gradients; and updating the first weight set with the average set of weight gradients to generate the second weight set.

[0231] Clause 18. A computing device as described in any of Clauses 13-15, wherein the neural network computing circuitry is a neural network processor, a neural network accelerator, a graphics processing unit (GPU), a field-programmable gate array (FPGA), a coprocessor, or an application-specific integrated circuit (ASIC).

[0232] Clause 19. A computing device as described in any of Clauses 13-15, wherein the operation further includes performing a weight synchronization process after a predetermined number of iterations of the training process to synchronize weight values ​​across multiple processing nodes of the neural network training system.

[0233] Clause 20. The computing device of any of Clauses 13-15 further includes a weight comparison circuit configured to compare the difference between a second set of weights and a speculative set of weights with a threshold difference.

[0234] Clause 21. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a computing device in a neural network training system, cause the computing device to perform operations including:

[0235] The first iteration of the training process is performed using the first set of weights to generate the first output dataset;

[0236] The set of local weight gradients is derived by comparing the first output dataset with the expected output dataset.

[0237] Update the first weight set using the local weight gradient set to derive the speculative weight set; and

[0238] While waiting to obtain the second set of weights for the second iteration of the training process, the second iteration of the training process is performed using the speculative set of weights to generate the second output dataset.

[0239] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If a module is a software module, it may be embodied on a non-transitory computer-readable medium and processed by a processor in any computer system described herein. It should be noted that the processes and architectures described may be executed in real-time or asynchronously prior to any user interaction. Modules may be configured in the manner suggested in the foregoing figures, and / or the functionality described herein may be provided by one or more modules that exist as separate modules, and / or the module functionality described herein may be distributed across multiple modules.

[0240] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the claims.

[0241] Other variations are also within the spirit of this disclosure. Thus, while the disclosed technology is readily adaptable to various modifications and substitutions, some of the illustrated examples are shown in the accompanying drawings and have been described in detail above. However, it should be understood that this is not intended to limit this disclosure to the one or more specific forms disclosed, but rather to cover all modifications, substitutions, and equivalents falling within the spirit and scope of this disclosure as defined by the appended claims.

[0242] In the context of describing the disclosed examples (especially in the context of the following claims), the terms “a,” “an,” and “the,” and similar appellations are intended to cover both singular and plural forms, unless otherwise stated herein or obviously contradicted by the context. The terms “comprising,” “having,” “containing,” and “including” should be interpreted as open-ended terms (i.e., meaning “including, but not limited to”), unless otherwise stated. The term “connected” should be interpreted as partially or wholly contained, attached to, or combined with, even if something in between exists. The recitation of value ranges herein is intended only as a shorthand method for individually referring to each individual value falling within that range, unless otherwise stated herein, and each individual value is incorporated into this specification as if it were separately recited herein. All methods described herein may be performed in any suitable order, unless otherwise stated herein or obviously contradicted by the context. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate examples of this disclosure and does not constitute a limitation on the scope of this disclosure, unless otherwise required. The language used in this specification should not be construed as indicating that any non-claimed element is necessary for implementing this disclosure.

[0243] Unless otherwise specifically stated, delimiters such as “at least one of X, Y, or Z” are used in a context where items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such delimiters are generally not intended to imply, and should not imply, that certain examples require the presence of at least one X, at least one Y, or at least one Z each.

[0244] This document describes various examples of this disclosure, including the best mode known to the inventors for carrying out this disclosure. Variations of these examples may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect those skilled in the art to appropriately employ such variations, and the inventors intend to practice this disclosure in ways other than those specifically described herein. Therefore, this disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, any combination of the foregoing elements in all possible variations is covered by this disclosure unless otherwise stated herein or clearly contradicted by the context.

Claims

1. A method comprising: performing, by a processing node of a plurality of processing nodes in a neural network training system, a first iteration of a training process using a first set of weights to generate a first output dataset; deriving, by the processing node, a set of local weight gradients based on a comparison of the first output dataset to a reference output dataset; updating, by the processing node, the first set of weights using the set of local weight gradients to derive a speculative set of weights; and performing, by the processing node, the second iteration of the training process using the speculative set of weights to generate a second output dataset while waiting to obtain a second set of weights for a second iteration of the training process.

2. The method of claim 1, further comprising: obtaining the second set of weights for the second iteration of the training process; and determining a difference between the second set of weights and the speculative set of weights.

3. The method of claim 2, further comprising: determining that the difference between the second set of weights and the speculative set of weights is at or below a threshold difference; and continuing the training process using the second output dataset generated from the speculative set of weights.

4. The method of claim 2, further comprising: determining that the difference between the second set of weights and the speculative set of weights exceeds a threshold difference; and repeating the second iteration of the training process using the second set of weights instead of the speculative set of weights. the second set of weights is obtained from another processing node of the neural network training system. the second set of weights is obtained by: accumulating a plurality of sets of local weight gradients, each set of local weight gradients derived by a corresponding processing node of the plurality of processing nodes of the neural network training system; averaging the sets of local weight gradients to derive a set of average weight gradients; and 5. The method of any one of claims 1-4, wherein, updating the first set of weights using the set of average weight gradients to generate the second set of weights.

6. The method of any one of claims 1-4, wherein, 7. The method of any of claims 1-4, further comprising: performing a weight synchronization operation to synchronize weight values across the plurality of processing nodes of the neural network training system after a predetermined number of iterations of the training process. the plurality of processing nodes are communicatively coupled to each other via a network.

9. A computing device comprising: a neural network computation circuit; a processor, and memory storing code that, when executed by the processor, causes the computing device to implement operations comprising:

8. The method of any one of claims 1-4, wherein, performing, using the neural network computation circuit, a first iteration of a training process with a first set of weights to generate a first output dataset; deriving a set of local weight gradients based on a comparison of the first output dataset to an expected output dataset; updating the first set of weights using the set of local weight gradients to derive a speculative set of weights; and ​ ​ ​ ​ ​ performing, using the neural network compute circuit, a second iteration of the training process with the speculative weight set to generate a second output data set when a second weight set for the second iteration of the training process becomes available; and obtaining the second weight set for the second iteration of the training process, wherein the second weight set is generated by averaging a plurality of local weight gradient sets, wherein each local weight gradient set is derived by a corresponding processing node of a neural network training system.

10. The computing device of claim 9, wherein, The operations further include: determining that a difference between the second weight set and the speculative weight set is at or below a threshold difference; and allowing the training process to continue using the second output data set generated from the speculative weight set.

11. The computing device of claim 9, wherein, The operations further include: determining that a difference between the second weight set and the speculative weight set exceeds a difference threshold; and repeating the second iteration of the training process using the second weight set instead of the speculative weight set.

12. The computing device of any of claims 9-11, wherein, The second weight set is obtained from one of the processing nodes of the neural network training system.

13. The computing device of any of claims 9-11, wherein, The second weight set is obtained by accumulating a plurality of local weight gradient sets, averaging the local weight gradient sets to derive an average weight gradient set, and updating the first weight set using the average weight gradient set to generate the second weight set.

14. The computing device of any of claims 9-11, wherein, The neural network compute circuit is a neural network processor, a neural network accelerator, a graphics processing unit (GPU), a field programmable gate array (FPGA), a co-processor, or an application specific integrated circuit (ASIC).

15. The computing device of any of claims 9-11, wherein, The operations further include performing a weight synchronization process to synchronize weight values across the plurality of processing nodes of the neural network training system after a predetermined number of iterations of the training process.

16. The computing device of any one of claims 9-11, further comprising a weight comparison circuit configured to compare a difference between the second weight set and the speculative weight set to a threshold difference.

17. A non-transitory computer-readable medium having stored therein instructions which, when executed by a computing device in a neural network training system, cause the computing device to perform operations comprising: performing a first iteration of a training process using a first weight set to generate a first output data set; deriving a local weight gradient set based on a comparison of the first output data set to an expected output data set; updating the first weight set using the local weight gradient set to derive a speculative weight set; and performing a second iteration of the training process using the speculative weight set to generate a second output data set when a second weight set for the second iteration of the training process is obtained.

Images