Gradient compression for distributed training

Abstract

A technique for exchanging compressed gradient data within a distributed system is disclosed. A gradient set is calculated on a first worker node of the distributed system using a neural network model and a weight set associated with the neural network model. Each of the gradient sets having a value less than a threshold is clipped, resulting in unclipped data elements and clipped data elements. A mapping is generated that indicates which of the gradient sets correspond to unclipped data elements and which correspond to clipped data elements. Compressed data is generated based on the unclipped data elements. The mapping and the compressed data are transmitted from the first worker node to a second worker node of the distributed system.

Description

Background Art

[0001] An artificial neural network (often referred to simply as a neural network) is a computing system with an architecture based on biological neural networks. A neural network can be trained using training data to learn how to perform specific tasks, such as identifying or classifying physical objects, activities, people, etc. from images or videos. A neural network may include multiple layers of processing nodes. Each processing node in a layer may perform calculations on input data generated by a processing node in the previous layer to generate output data. For example, a processing node may perform a set of arithmetic operations (such as multiplication and addition) to generate intermediate outputs, or perform post-processing operations on intermediate outputs to generate final outputs. A neural network may include thousands or more processing nodes and millions or more parameters.

[0002] The architecture of a neural network can include an input layer, an output layer, and many intermediate layers, often called hidden layers. Each layer performs calculations on the output of the previous layer, with the final layer (the output layer) providing the final result. In theory, neural networks can use more layers to perform more complex tasks, such as language translation and identifying (or classifying) the content of an image. Neural networks with more than three hidden layers are sometimes called deep neural networks. Deep neural networks can have many hidden layers, for example, from five to more than a thousand.

[0003] Neural networks can use central processing units (CPUs) to perform calculations. However, CPUs are always optimized for sequential calculations rather than parallel calculations, so they may suffer from poor response time. Graphics processing units (GPUs) are optimized for parallel calculations, but they do not necessarily provide results from one calculation unit directly to another. Typically, the results must be written to storage first and then read. Although the response time of GPUs is better than that of CPUs, there is still a need to improve the execution time of neural networks. Recently, application-specific integrated circuit devices, such as neural network processors or accelerators, have been developed that can perform neural network calculations more efficiently than CPUs or GPUs. These devices use a spatial architecture in which the arithmetic logic unit (ALU) can pass data directly from one unit to another, while in the temporal architecture used by CPUs and GPUs, the ALUs can only obtain data from the storage layer structure and cannot communicate directly with each other.

[0004] When training a neural network to perform a specific function, the parameters of the neural network (e.g., its weights, representing the strength of connections between different processing nodes) are adjusted in multiple iterations. The training process includes providing training data to the neural network, which may include training input data and corresponding reference output data that can support a specific decision (e.g., whether an object in an image is detectable or undetectable). The neural network can perform calculations to combine the weights with the training input data to generate training output data, and can compare the training output data with the reference output data to generate error data (representing the difference between the two). During training, different training input data can be provided to the neural network to generate different training output data. The weights of the neural network can be adjusted based on some goal, such as minimizing the difference between the training output data and the reference output data. In order to increase the likelihood that the neural network will generate correct decisions, the neural network is usually trained using massive training input data covering a large number of computing scenarios. As a result, the training operation usually requires a lot of time and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various exemplary embodiments according to the present invention are described with reference to the accompanying drawings, in which:

[0006] Figure 1 Examples of computational flow models for neural networks;

[0007] Figure 2 An example of the training process used to train a neural network;

[0008] Figure 3 Examples of distributed systems that can perform neural network training processes;

[0009] Figures 4A-4C Examples of various training steps performed by the distributed system;

[0010] Figures 5A-5C Examples of various training steps performed by the distributed system;

[0011] Fig. 6A and Figure 6B Corresponding to Figures 4A-4C and Figures 5A-5C An example of a timing diagram for

[0012] Figure 7 An example of a timing diagram for neural network model training;

[0013] Figure 8 An example of gradient set communication between a sending worker node and a receiving worker node;

[0014] Fig. 9 Schematic example of the steps for exchanging compressed gradient data within a distributed system;

[0015] Fig.10 Example of a timing diagram for sending compressed gradient data within a distributed system;

[0016] Fig.11A and 11B Methods for exchanging compressed gradient data within a distributed system;

[0017] Fig.12 An example of an accelerator;

[0018] Fig.13 An example of a speed-up engine;

[0019] Fig.14 An example of a host system;

[0020] Fig.15 An example of a network. DETAILED DESCRIPTION

[0021] During training of a neural network, a first neural network layer may receive training input data, combine the training input data with weights (e.g., by multiplying the training input data with the weights and then summing the products), generate first output data for the neural network layer, and propagate the output data to a second neural network layer in a forward propagation operation. The second neural network layer performs another forward propagation operation on the first output data from the first neural network layer to generate second output data, and propagates the second output data to a higher neural network layer. The forward propagation operation may start from the first neural network layer and end at the highest neural network layer. The forward propagation operation of each neural network layer may represent a different stage of extracting and processing information from the training input data. A decision may then be made based on the output data of the highest neural network layer.

[0022] The training process can generate and / or update a set of weights for a neural network to increase the likelihood that the neural network will generate a correct decision. An example training process can use a descent gradient scheme. Specifically, as part of the training process, a forward propagation operation can be performed on the training input data using the set of weights for each neural network layer to generate training output data at the highest neural network layer. The training output data can be compared to reference output data that supports a particular decision. For example, a set of gradients can be generated based on the difference between the training output data and the reference output data.

[0023] As part of the training process, each neural network layer can then perform a back-propagation process to adjust the weight set of each neural network layer. Specifically, the highest neural network layer can receive the gradient set and, in a back-propagation operation, calculate a first data gradient set and a first weight gradient set based on applying the weight set to the input data gradient using a similar mathematical operation method as the forward propagation operation. The highest neural network layer can adjust the weight set of the layer based on the first weight gradient set, and the first data gradient set can be propagated to the next highest neural network layer to affect the adjustment of the weight set of the previous neural network layer. The back-propagation operation can start from the highest neural network layer and end at the first neural network layer. The weight set of each neural network layer can be adjusted to complete one iteration of the training process. The training process can be repeated for multiple iterations on the same training data until a loss target (e.g., a threshold input data gradient) is reached.

[0024] Due to the sequential nature and data dependencies between the operations involved in the training process, the training process is usually very time-consuming. Specifically, during the training process, a forward propagation operation is first performed at each neural network layer to calculate the output data, followed by calculating the input data gradient based on the output data (and the reference output data), followed by a backward propagation operation at each neural network layer to calculate the weight gradient, and then the weight is updated at each neural network layer. Since the backward propagation operation depends on the forward propagation operation, the two sets of operations may not be performed in parallel. In addition, due to the data dependencies between the neural network layers, the forward propagation operation and the backward propagation operation need to be performed sequentially for each neural network layer. The lack of parallelism will greatly increase the training time, and the training time will be further increased when multiple iterations of the training process are performed on the same training input data to achieve the loss target. In addition, the training process usually involves providing multiple sets of training data to the neural network to cover different operating conditions, so that the neural network can be trained to provide correct decisions under different operating conditions. The computing system that implements the neural network may need to perform other training processes to process other training input data, which will further increase the training time. In addition, the training process usually requires higher precision than the inference operation, and the slow training process will put a lot of pressure on computing resources.

[0025] Distributed systems can speed up the training process by distributing the training process across multiple computing systems (called worker nodes). The training data can be divided into multiple parts, each of which will be processed by a worker node. Each worker node can perform forward and backward propagation operations independently and in parallel with each other based on a portion of the training input data, generating a set of weight gradients for each neural network layer. Each worker node can exchange its weight gradient set with other worker nodes and average its own weight gradient set with the weight gradient sets received from other worker nodes. Each computing node can have the same average weight gradient set, and then a weight set can be updated for each neural network layer based on the average weight gradient.

[0026] Distributing the training process across multiple worker nodes can reduce the amount of training input data to be processed on each worker node, which can reduce the execution time of the forward and backward propagation operations of each neural network layer and speed up the training process. However, since distributed learning is usually implemented on relatively slow networks, the exchange of weight gradients between worker nodes can become a serious bottleneck. For example, when the distributed system is located in a cloud architecture and the worker nodes exchange weight gradients with each other by sending network packets, the network latency can be quite large relative to the execution time of the forward / backward propagation operations. Network latency can offset the reduction in training time brought by the distributed system, or even increase training time. Therefore, technical means are needed to reduce network latency in distributed training.

[0027] The embodiments described herein are directed to systems, methods, and other techniques for exchanging compressed gradient data within a distributed system. The embodiments described reduce the throughput requirements for exchanging gradients between nodes by clipping some of the calculated gradients and applying compression techniques to the remaining data before transmitting to remote nodes. Although some embodiments have a small increase in overhead due to the addition of a compression header accompanying the compressed data, this increase significantly decreases with the amount of compressed data.

[0028] Some embodiments typically provide for the communication of weight gradients (or simply "gradients") between a sending work node and a receiving work node in a distributed system. The transmitting work node can use a neural network model to calculate a set of gradients, which will be transmitted to the receiving work node for synchronization. Before transmission, a sparsity analysis can be performed on the calculated gradients on the sending work node. The sparsity analysis can determine various statistics about the gradients, such as the mean or standard deviation. The sparsity analysis can identify different gradient values ​​at which different sparsity percentages are obtained by clipping the gradients. For example, a clipping threshold T can be set at a sparsity percentage of 50% so that 50% of the gradients are below the threshold. The sparsity percentage can be based on the number of gradients in each layer, so that for all layers, the compression time and / or transmission time can be better aligned or reduced.

[0029] After the clipping threshold value T is determined, the gradient can be clipped by clipping each gradient with a value less than the clipping threshold value T (e.g., by setting its value to zero). In this way, a gradient set including unclipped data elements and clipped data elements can be obtained. Next, a mapping is generated that represents compressed data of the unclipped data elements and identifies the position of the unclipped data elements in the gradient set. The mapping allows the receiving worker node to decompress and reconstruct the gradient set by combining the unclipped data elements in the compressed data with the clipped data elements (e.g., zero) using the mapping.

[0030] Various advantages are achieved by the described techniques. For example, for almost all sparse percentages, the described techniques provide better compression ratios than conventional compression algorithms. For example, as demonstrated by the experimental data described herein, for the lowest sparse percentage case, the compressed data provided by the described techniques is less than or close to the original data size, while for the higher sparse percentage case, the described techniques provide significant compression ratios. The described techniques provide less complex hardware implementations because there is no need to track row / column indexes and pointers. In addition, the computable compressed header has a predictable length, which results in more direct sliding window buffer management. In addition to inter-node communication, the described compression techniques can be used when transmitting data within a working node (e.g., from local memory to a computing engine).

[0031] Various examples will be described below. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the examples may be practiced without the specific details. In addition, well-known functions may be omitted or simplified to avoid confusion with the exemplary implementations described.

[0032] Figure 1An example of a computational flow model for a neural network 100. Neural networks are inspired by the workings of the human brain. According to various models of the brain, the primary computational unit of the brain is the neuron. Neurons are connected together in many units, the units that enter the neuron are called dendrites, and the single element that leaves the neuron is called an axon. Neurons receive signals through the dendrites, perform computations on the signals, and then output signals on the axon. The input and output signals are called activations. The axon of one neuron can branch and connect to the dendrites of multiple neurons. The connections between the branches of the axon and the dendrites are called synapses.

[0033] Synapses scale the signals that cross them. The scaling factors are called weights, and are thought to be how the brain is able to learn: different responses to inputs result in different weights. Learning can change the weights, but the organization of neurons and synapses does not need to change to achieve learning. Therefore, the static structure of the brain can be used as a model for a program, with the weights reflecting the tasks the program has learned to perform.

[0034] The working principle of a neural network is that the computation of a neuron involves taking a weighted sum of the input values. These weighted sums correspond to the scaling of the values ​​performed by the synapses and the combining of these values ​​in the neuron. A function is performed on the combined inputs within the neuron. In the brain model, this operation appears to be a nonlinear function that causes the neuron to generate an output only when the input exceeds a certain threshold value. So by analogy, the nodes of a neural network can apply a nonlinear function to the weighted sum of the values ​​input to the node.

[0035] In the example shown, the neural network 100 includes an input layer 104, one or more intermediate layers (often referred to as hidden layers) 106, and an output layer 108. Each layer includes a certain number of nodes 102. In this example, each node 102 of the input layer 104 is connected to each node 102 of the hidden layer 106-1. These connections, which are called synapses in the brain model, are called weights 110. Also in this example, each node 102 of the hidden layer 106-N has a connection or weight 110 with each node 102 of the output layer 108. The input layer 104 can receive input and can propagate the input to the hidden layer 106-1. The weighted sum calculated by the hidden layer 106-1 is propagated to the remaining hidden layers 106 and then to the output layer 108, which can present the final output to the user. Consistent with the brain model, the output of the node 102 can be called an activation.

[0036] The following are examples of calculations that might occur at each layer in the example neural network 100:

[0037]

[0038] In the above equation, W ijis the weight, x i is the input activation signal, y j is the output activation signal, f() is a nonlinear function, and b is a bias term. Various nonlinear functions can be used to achieve different purposes.

[0039] The neural network 100 model may be referred to as a directed weighted graph. In a directed graph, each connection to or from a node represents a direction (e.g., entering a node or leaving a node). In a weighted graph, each connection has a weight. Neural network development tools can visualize a neural network as a directed weighted graph for easier understanding and debugging. In some cases, these tools can also be used to train the neural network and output the trained weights. Executing the neural network is then simply a matter of performing calculations on the input data using the weights.

[0040] Neural networks with multiple layers are able to learn higher-level features that are more complex and abstract than shallow networks. For example, a neural network can be trained to recognize images. In this example, the pixels of an image are fed into the input layer of the neural network, and the first layer outputs indicate the presence of low-level features in the image, such as lines and edges. In subsequent layers, these features can be combined to weigh the likely presence of higher-level features: lines can be combined into shapes, and further combined into sets of shapes. Given all this information, the neural network can output the probability that the high-level feature represents a particular object or scene. For example, the neural network can output whether there is a cat in the image.

[0041] The learning phase of a neural network refers to neural network training. During training, the neural network 100 is trained to perform a task. As the task is learned, the values ​​of the weights 110 (and possibly biases) are determined. The underlying model architecture of the neural network (e.g., how the nodes are organized into layers, the connections between the nodes of each layer, the calculations performed by each node) does not change during training. After training, the neural network 100 can calculate results using the weights 110 values ​​determined during training in order to perform the task. The process of running a neural network program is called inference.

[0042] As described above, the training process of the neural network 100 may occur on multiple worker nodes in a distributed system, such as Figure 1120. In various implementations, worker node 120 may be a neural network hardware accelerator, a general-purpose hardware processor, or other appropriate computing system that supports the arithmetic operations involved in the above-mentioned neural network processing. Worker node 120 may include a hardware interface for communicating with other worker nodes over a network. Worker node 120 may include computing resources for performing training process operations, which may include forward propagation operations, loss gradient operations, and backpropagation operations. Worker node 120 may receive training data, including training input data and reference output data. Based on the training input data, worker node 120 may use model architecture 112 and weights 110 to calculate output data. Then, worker node 120 can calculate error data by comparing the output data with the reference output data, which can be used to calculate a set of gradients. Gradients can be distributed to other nodes in the distributed system for gradient synchronization. Then, worker node 120 can receive synchronized gradients and / or gradient adjustments that may be applicable to weights 110.

[0043] Because the training data operated by the worker node 120 is different from that of other worker nodes in the distributed system (e.g., different parts of a training data set), the amount of iterative error in the training process of different worker nodes may be different. To improve the accuracy of the neural network model on different training data, the local gradients calculated by each worker node can be accumulated and then averaged to obtain a set of average gradients. For example, if the neural network model uses 20 weight values, the first iteration of the training process of each worker node will produce 20 local gradients. The first local gradient of each worker node can be added together and then divided by the number of worker nodes to obtain the average gradient of the first value. Each of the 20 gradients can be calculated to obtain a set of 20 average gradients.

[0044] Figure 2 An example of a training process 200 for training a neural network, such as neural network 100, is shown. Figure 2 As shown, a forward propagation operation can be performed for each neural network layer, such as the lowest layer 1 (which may correspond to Figure 1 104 layers) forward propagation operation 202a, the second layer forward propagation operation 202a (corresponding to Figure 1 106-1 layer in), the highest layer n forward propagation operation 202n (corresponding to Figure 1 108 layers in the neural network), etc. The forward propagation operation of a neural network layer may include multiplication and summation calculations between the input data of the layer and a set of weights, followed by activation function processing to generate output data. The output data can then be propagated to the next neural network layer and used as the input of the forward propagation operation of this layer. For example, Figure 2As shown, the forward propagation operation 202a can combine the training data with the weight w1 of the first layer to generate output data out1, which will be propagated as input to the second layer. The forward propagation operation 202b can combine the data out1 with the weight w2 of the second layer to generate output data out2, which can then be propagated to the next layer. In the highest layer n, the forward propagation operation 202n is from the n-1 layer ( Figure 2 (not shown) receives data outn-1 and then combines the data with the weight wn of layer n to generate output data outn.

[0045] The loss gradient operation 204 may compare the output data outn of the n layer with the reference output data refoutn to generate an input data gradient din. The input data gradient din may measure the difference rate between outn and refoutn for each data element of the output data outn. In some examples, the purpose of training is to minimize the difference between outn and refoutn so that the input data gradient din approaches zero.

[0046] After the input data gradient din is generated by the loss gradient operation 204, a back propagation operation 206 may be performed for each neural network layer. For example, the back propagation operation 206n may be performed on the highest layer n, the back propagation operation 206b may be performed on the 2nd layer, and the back propagation operation 206a may be performed on the 1st layer. The back propagation operation of the neural network layer may be based on the weights, data gradient inputs, and forward propagation operation inputs of the neural network layer. For example, for layer n, the back propagation operation 206n may receive as input the weight wn, the input data outn-1 (from the forward propagation operation of the neural network layer n-1), and the input data gradient din. The back propagation operation may perform multiplication and summation calculations on the inputs to generate output data gradients ( Figure 2 dn-1, d2, d1, etc.) and weight gradient wgrad ( Figure 2 The output data gradients may be forwarded to the next lower neural network layer as input to the backpropagation operation of that layer, and the weight gradients may represent the changes to be applied to the weights in the neural network layer.

[0047] The weights of the n layers may be updated by an update operation 208 (e.g., update operation 208n for the n layer) based on the weight gradient dwn and the following formula:

[0048] wn′=wn-α×dwn

[0049] In this formula, wn' may represent the updated weight wn, and α may include a set of predetermined constants.

[0050] Then, the output data gradient dn-1 generated by layer n can be propagated to the next lower neural network layer n-1 as the back propagation operation input of the layer. The back propagation operation 202b of layer 2 can be operated based on the data gradient d2, the weight w2 and the input data out1 to generate the output data gradient d1 and the weight gradient dw2. The weight gradient dw2 can be used for the update operation 208b to update the w2 weight. The data gradient d1 can be propagated to layer 1. The back propagation operation 202a of the first layer can be calculated based on the data gradient d2, the weight w1 and the training input data to generate the weight gradient dw1. The weight gradient dw1 can be used for the update operation 208a to update the w1 weight.

[0051] Due to the sequential nature of the training process, the training process performed on a single computing system can be very time-consuming. Specifically, as described above, during the training process, forward propagation is first performed on each neural network to calculate the training output data, and then backpropagation is performed on each neural network layer to calculate the weight gradient, followed by updating the weights of each neural network layer. Since the backpropagation operation is based on the forward propagation operation, these two sets of operations cannot be performed in parallel. In addition, due to the data dependency between the neural network layers, each neural network layer also needs to perform forward propagation operations and backpropagation operations sequentially. The lack of parallelization can significantly increase the training time, which is further increased when multiple batches of training processes are performed on different parts of the training data, and the batches are repeated in multiple iterations to approach the minimum data gradient.

[0052] As described above, one way to speed up the training process is to use a distributed system to distribute the training process to multiple computing devices, each of which can be configured as a worker node. By distributing the training process to multiple worker nodes, the amount of training data to be processed on each worker node can be reduced, which can shorten the completion time of forward and backward propagation operations and speed up the training process. For example, as the amount of training data processed by each worker node is reduced, the duration of the forward feedback operation and the backward feedback operation can be shortened.

[0053] Figure 3 An example of a distributed system 300 is shown that can perform a training process for a neural network according to some implementations. Figure 3As shown, the distributed system 300 may include many worker nodes (such as computing devices) 120-1, 120-2... to 120-n, etc. Each worker node 120 may include a communication interface to communicate with each other through a computer network 306. Each worker node 120 may include computing resources to perform forward propagation operations, backpropagation operations, weight update operations, etc. For example, the computing resources may include a neural network processor, a neural network accelerator, a graphics processing unit (GPU), a field programmable gate array (FPGA), a processor or coprocessor, an application-specific integrated circuit (ASIC) and / or other appropriate computing circuits that support arithmetic operations involved in the training process. Each worker node 120 can communicate with other worker nodes 120 through a computer network 306 to perform an exchange operation by exchanging weight gradients, and perform an update weight operation after the exchange operation is completed.

[0054] The distributed system 300 may be initialized by a coordination agent 304. In one example, the coordination agent 304 may receive a list of jobs 302 to be performed. The coordination agent 304 may determine which worker nodes 120 may process the list of jobs 302, select multiple worker nodes 120 to process the list of jobs 302, and provide instructions to each selected worker node 120 for the list of jobs 302 to be completed. The instructions provided to each selected worker node 120 may include the identification of other selected worker nodes and / or the identification of the next or previous worker node in the logical ring (for a logical ring topology). After completing the list of jobs 302, the selected worker node 120 may alert the coordination agent 304 that any subsequent jobs may be processed.

[0055] Figures 4A-4C Various example training steps performed by the distributed system for the first synchronization scheme are shown, where the gradients are synchronized on a single worker node. Since the gradients are calculated by the various worker nodes 120-1, 120-2, and 120-3, the gradients are distributed to the worker node 120-4, as shown in Figure 4A Next, the worker node 120-4 performs gradient synchronization, as shown in Figure 4B After synchronizing the gradient, the gradient is distributed from the worker node 120-4 to each of the worker nodes 120-1, 120-2, and 120-3, as shown in FIG. Figure 4C shown.

[0056] Figures 5A-5C Various example training steps performed by the distributed system for the second synchronization scheme are shown, where the gradients are synchronized on each worker node. Since the gradients are completed by the various worker nodes 120-1, 120-2, 120-3, and 120-4, the gradient exchange is thoroughly performed so that each worker node receives the gradients from other worker nodes, such as Figure 5A Next, each worker node 120-1, 120-2, 120-3 and 120-4 performs gradient synchronization, as shown in FIG. Figure 5B As shown. Since each worker node 120-1, 120-2, 120-3 and 120-4 calculates its own synchronization gradient, there is no synchronized gradient allocation, such as Figure 5C shown.

[0057] Fig. 6A and 6B Shown respectively with Figures 4A-4C and Figures 5A-5C The timing diagrams for the first and second synchronization schemes are shown in . Fig. 6A In the example shown in , during the first training period, training data is loaded into each worker node 120 (optionally worker node 120-4), gradients are calculated by each worker node 120 (optionally worker node 120-4), gradients are passed from worker nodes 120-1, 120-2, and 120-3 to worker node 120-4 (as indicated by the downward arrow), which synchronizes the received gradients (and the gradients calculated by worker node 120-4 (optional)), and then the synchronized gradients are passed from worker node 120-4 to worker nodes 120-1, 120-2, and 120-3 (as indicated by the upward arrow). Prior to the second training period (repeating the same process), weights associated with the neural network model are updated based on the synchronized gradients.

[0058] exist Figure 6B In the example shown in , during the first training epoch, training data is loaded into each worker node 120, gradients are calculated by each worker node 120, gradients are transferred (e.g., exchanged) between each worker node 120 (as indicated by the downward and upward arrows), and each worker node 120 synchronizes the received gradients (as well as the gradients calculated on the specific worker node). Prior to the second training epoch (repeating the same process), weights associated with the neural network model are updated according to the synchronized gradients.

[0059] Figure 7 shows an example timing diagram of neural network model training. Figure 7In the example shown in , during the first training period, the training data loaded into the worker node 120-1 is used to train the first neural network model having layers 1, 2, and 3. Next, the gradients are calculated by the worker node 120-1 by performing forward propagation operations on layers 1, 2, and 3, followed by backpropagation operations for layers 1, 2, and 3. Since each gradient for each layer has been calculated and available, these gradients can be passed from the worker node 120-1 to the worker node 120-4 (as indicated by the downward arrow). For example, the gradient of layer 3 is calculated first and then passed to the worker node 120-4, the gradient of layer 2 is calculated next and then passed to the worker node 120-4, and finally the gradient of layer 1 is calculated and then passed to the worker node 120-4.

[0060] Worker node 120-4 synchronizes the gradients it received from worker node 120-1 with the other gradients it received. For example, first the layer 3 gradients are received and synchronized, then the layer 2 gradients are received and synchronized, and finally the layer 1 gradients are received and synchronized. The synchronized gradients are transmitted from worker node 120-4 to worker node 120-1 (as indicated by the upward arrow). Prior to the second training period (during which the same process is repeated), the weights associated with the first neural network model are updated according to the synchronized gradients.

[0061] Figure 8 An example communication of a gradient group between a transmitting worker node 802 and a receiving worker node 804 is illustrated. After the transmitting worker node 802 calculates the uncompressed gradient 806, but before transmission, the uncompressed gradient 806 is compressed at the transmitting worker node 802 by a compression module 812 to generate a compressed gradient 808. The compressed gradient 808 is then transmitted from the transmitting worker node 802 to the receiving worker node 804. Upon receiving the compressed gradient 808, the compressed gradient 808 is decompressed at the receiving worker node 804 by a decompression module 814 to generate a decompressed gradient 810.

[0062] Fig. 9 is an example diagram showing the steps of exchanging compressed gradient data within a distributed system. A gradient group 902 is calculated at a transmitting worker node using a neural network model, a set of weights, and training data. The 902 gradient group includes data elements N0, N1, ..., N7. A sparsity analysis will be performed on the 902 gradient groups to determine a clipping threshold T. Clipping will be performed on the 902 gradient groups by clipping each 902 gradient group whose value is less than the clipping threshold T.

[0063] In the illustrated example, it has been determined that the values ​​of data elements N0, N4, and N7 are all greater than the clipping threshold T, and the values ​​of data elements N1, N2, N3, N5, and N6 are all less than the clipping threshold T. Therefore, by performing gradient clipping 906, data elements N1, N2, N3, N5, and N6 are clipped, thereby generating 908 a gradient group including unclipped data elements N0, N4, and N7 and clipped data elements, which are all equal to zero at the positions of data elements N1, N2, N3, N5, and N6. In some examples, the values ​​of the clipped data elements may not be equal to zero, such as some predetermined value greater than zero.

[0064] Gradient compression 910 is performed on the 908 gradient groups to generate a header 912 and compressed data 918. The header may include a map 914 and an original length 916. The map 914 may indicate which 908 gradient groups correspond to non-clipped data elements and / or which 908 gradient groups correspond to clipped data elements. In the illustrated example, the map 914 includes an 8-bit bitmap with 1's at the locations of the non-clipped data elements and 0's at the locations of the clipped data elements. In other examples, the 1's and 0's may be flipped. In some examples, the map 914 may include a 2D bitmap or matrix containing binary values ​​(e.g., 1's or 0's) indicating the locations of the non-clipped data elements. The map 914 may include an original length 916 indicating the number of non-clipped data elements and clipped data elements in the 908 gradient group. In the illustrated example, the original length 916 is equal to 8 because there are 3 non-clipped data elements and 5 clipped data elements.

[0065] The compressed data 918 may include untrimmed data elements (and / or descriptions of untrimmed data elements) from the gradient group 908: N0, N4, and N7. The header 912 and the compressed data 918 are transmitted by the transmitting worker node to the receiving worker node. Gradient decompression 920 may be performed on the received header 912 and compressed data 918 to generate decompressed data including the gradient group 922. During the gradient decompression 920, the untrimmed data elements included (or depicted) in the compressed data 918 are positioned at the locations of the untrimmed data elements identified by the mapping 914 to form the gradient group 922. Therefore, the gradient group 922 may exactly match the gradient 908 group.

[0066] Fig.10 is an example timing diagram illustrating the transfer of compressed gradient data within a distributed system. Fig.10In the example shown in , training data is loaded into worker node 120-1 for training a neural network model with layers 1, 2, and 3. Each layer gradient is calculated by worker node 120-1 by performing a forward propagation operation on layers 1, 2, and 3, and then performing a backward propagation operation on layers 3, 2, and 1. As each gradient for each layer is calculated and becomes available, it is compressed by referring to Fig. 9 904, gradient clipping 906, and gradient compression 910 are performed as described in. For example, the following operations are performed on the gradients of layer 3, layer 2, and layer 1 in sequence: first, they are calculated, then clipped, compressed, and transmitted to the worker node 120-4.

[0067] In some embodiments, the DMA controller 1002 at the worker node 120-1 may perform one or more compression tasks. For example, the DMA controller 1002 may utilize a gradient compression engine (GCE) to perform sparsity analysis, gradient pruning, quantization, and / or compression. The GCE may construct a compression header to record the original length (the number of unpruned and pruned data elements) and the compression bitmap.

[0068] Worker node 120-4 decompresses the compressed gradients received from worker node 120-1 and then synchronizes the gradients with the gradients received from other worker nodes (e.g., worker nodes 120-2 and 120-3). For example, after receiving the 3rd layer gradients, it is decompressed and synchronization begins, then the 2nd layer gradients are received, decompressed and synchronization begins, and finally the 1st layer gradients are received, decompressed and synchronization begins. The synchronized gradients are transmitted from worker node 120-4 to worker node 120-1 (as indicated by the upward arrow). The weights associated with the neural network model are updated based on the synchronized gradients. In some embodiments, the same compression / decompression operation may be performed on the synchronized gradients before and after the synchronized gradient group is transmitted from worker node 120-4 to the first worker node 120-1, such as Fig.10 as shown in .

[0069] In some embodiments, the DMA controller 1004 at the worker node 120-4 may perform one or more decompression (and / or compression) tasks. For example, the DMA controller 1004 may utilize a gradient decompression engine to read the compressed data and headers, insert zero data elements between untrimmed (e.g., non-zero) data elements, and then write out the decompressed data elements.

[0070] Fig.11A and 11BA method 1100 for exchanging compressed gradient data within a distributed system is illustrated. During the execution of the method 1100, one or more steps of the method 1100 may be omitted, and the steps of the method 1100 need not be executed in the order shown. One or more steps of the method 1100 may be executed by one or more processors, such as a neural network processor or a component thereof (e.g., a compression or decompression accelerator). The method 1100 may be implemented as a computer-readable medium or a computer program product, wherein the instructions contained therein may cause one or more computers to execute the steps of the method 1100 when the program is executed by one or more computers. Such computer program products may be transmitted via a wired or wireless network in a data carrier signal carrying the computer program product.

[0071] In step 1102, a neural network model is received at a first worker node of a distributed system. Also, in step 1102, a set of weights associated with the neural network model may be received at the first worker node. A particular architecture of the neural network model may have a particular number of layers.

[0072] In step 1104, training data is received at a first worker node. The training data may include training input data and corresponding reference output data. The training data may be used to train a neural network model.

[0073] In step 1106, a gradient group is calculated at the first worker node. The gradient group may be calculated using the neural network model, the weight group, and the training data. The gradient group may include the gradients of the first layer, the second layer, and the third layer of the neural network model. The gradients of the third layer, the second layer, and the first layer will be calculated in sequence.

[0074] In step 1108, a sparsity analysis is performed on the gradient group. Performing the sparsity analysis may include determining statistics associated with the gradient. For example, a mean and standard deviation of the gradient group may be calculated. In some embodiments, different standard deviations from the mean may be calculated to determine different accumulation levels. In some embodiments, the sparsity analysis may determine different sparsity percentages. For example, a 50% sparsity percentage may correspond to a value at which 50% of the gradients are set to zero if a clipping threshold is applied. Similarly, a 90% sparsity percentage may correspond to a value at which 90% of the gradients are set to zero if a clipping threshold is applied. In various implementations, the sparsity analysis may determine 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90% or 95% sparsity percentages and / or any percentages therebetween. In some embodiments, the sparsity analysis may be performed at the first worker node (eg, by a DMA controller at the first worker node).

[0075] In step 1110, a trimming threshold T is determined based on the sparsity analysis. The trimming threshold T can be set to a value associated with the sparsity percentage found in step 1108. For example, the trimming threshold T can be set to a 50% sparsity percentage or any sparsity percentage listed in step 1108. In some embodiments, the trimming threshold T can be determined based on the sparsity analysis of a previous layer of the neural network model. For example, in some embodiments, a sparsity analysis can be performed on the third layer of the neural network, and the trimming thresholds T of the third layer, the second layer, and the first layer can be determined based on the same sparsity analysis, thereby improving the compression speed.

[0076] In some embodiments, the pruning threshold T can be set to different sparsity percentages for different layers of the neural network. In some cases, the more gradients a layer has, the greater its sparsity percentage; the fewer gradients a layer has, the smaller its sparsity percentage, so as to better adjust the compression time and / or transmission time for all layers. In one example, a 40% sparsity percentage can be used for the third layer of the neural network model, a 60% sparsity percentage can be used for the second layer of the neural network model (the second layer has more gradients than the third layer), and an 80% sparsity percentage can be used for the first layer of the neural network model (the first layer has more gradients than the second layer). In some embodiments, the pruning threshold T can be determined at the first worker node (e.g., by a DMA controller at the first worker node).

[0077] In step 1112, the set of gradients is gradient clipped using a clipping threshold T by clipping each of the set of gradients having a value less than a clipping threshold T. Clipping gradients may have their values ​​set to zero or to some predetermined value (such as the clipping threshold T). In some embodiments, gradient clipping the set of gradients results in the set of gradients including unclipped data elements and clipped data elements. In some embodiments, gradient clipping may be performed at the first worker node (e.g., by a DMA controller at the first worker node).

[0078] In step 1114, a map is generated that indicates which of the set of gradients correspond to untrimmed data elements and which of the set of gradients correspond to trimmed data elements. The element map may include a bitmap having binary values ​​indicating the positions of the untrimmed data elements and the trimmed data elements. For example, the bitmap may include 1s corresponding to the positions of the untrimmed data elements and 0s corresponding to the positions of the trimmed data elements. In some embodiments, the map may be generated at the first worker node (e.g., by a DMA controller at the first worker node).

[0079] In step 1116, a header is formed containing the mapping. In some embodiments, method 1100 may further include generating or determining the original length of the set of gradients. In such embodiments, the header may further include the original length. In some embodiments, the header may be formed at the first worker node (e.g., by a DMA controller at the first worker node).

[0080] In step 1118, compressed data including (or representing) the untrimmed data elements may be generated. In some embodiments, the compressed data may be generated at the first worker node (eg, by a DMA controller at the first worker node).

[0081] In step 1120, the header and compressed data are transmitted from the first worker node to the second worker node of the distributed system. In some embodiments, the header and compressed data of each layer of the neural network model may be transmitted in the order in which the gradient of each layer is calculated. For example, the header and compressed data of the third layer may be transmitted first, the header and compressed data of the second layer may be transmitted next, and the header and compressed data of the first layer may be transmitted last.

[0082] In step 1122, decompressed data is generated by combining untrimmed data elements from the compressed data with trimmed data elements using a mapping. Once generated, the decompressed data may include the set of gradients including untrimmed data elements and trimmed data elements. In some embodiments, the decompressed data may be generated at a second worker node (e.g., by a DMA controller at the second worker node).

[0083] In step 1124, a set of synchronized gradients is calculated at the second worker node based on the set of gradients from the decompressed data and other received gradients. The set of synchronized gradients can be calculated as each of the set of gradients is received. For example, the synchronized gradients of the third layer of the first neural network model can be calculated first, the synchronized gradients of the second layer of the first neural network model can be calculated next, and finally the synchronized gradients of the first layer of the first neural network model can be calculated.

[0084] In some embodiments, method 1100 may further include transmitting the set of synchronized gradients from the second worker node to the first worker node, and adjusting the set of weights based on the set of synchronized gradients. In some embodiments, the synchronized gradients may be subjected to the same compression / decompression operation before and after transmitting the set of synchronized gradients from the second worker node to the first worker node.

[0085] In an experimental environment, the performance of method 1100 is compared with a compressed row storage (CRS) algorithm to demonstrate its feasibility. The compression ratio CR (i.e., the ratio between the compressed data and the original uncompressed data) is calculated for method 1100 as follows:

[0086] CR=(NNZ×DS×8+m×n+log2(m×n)) / (m×n×DS×8),

[0087] Where NNZ is the number of nonzero elements, m is the number of columns in the original uncompressed data matrix, n is the number of rows in the original uncompressed data matrix (m×n is the total number of elements in the data), and DS is the element data size.

[0088] As shown in the table below, compared to the CRS algorithm, method 1100 has better CR in both high sparse percentage matrices and low sparse percentage matrices. For sparse percentages of 50% and below, the CRS algorithm provides a negative compression ratio, that is, the compressed data is larger than the original data. In the case of some low sparse percentages (such as 5%), the CRS algorithm may even double the original data size. Method 1100 provides a high CR of 87% at a sparse percentage of 90% (where 90% of the elements are zero elements), and provides a CR above 0% at a very low sparse percentage of 5% (where only 5% of the elements are zero elements).

[0089] For a 32×128 matrix of FP16 data type with m=32 and n=128 and an original data size of 8192, the following experimental results were obtained:

[0090] Sparse percentage = 90% Compressed header NNZ CR Method 1100 256.451545 819.2 0.86869488 CRS Algorithm 885.2 819.2 0.791943359

[0091] Sparse percentage = 75% Compressed header NNZ CR Method 1100 256.451545 2048 0.71869488 CRS Algorithm 2114 2048 0.491943359

[0092] Sparse percentage = 50% Compressed header NNZ CR Method 1100 256.451545 4096 0.46869488 CRS Algorithm 4162 4096 -0.008056641

[0093] Sparse percentage = 5% Compressed header NNZ CR Method 1100 256.451545 7782.4 0.01869488 CRS Algorithm 7848.4 7782.4 -0.908056641

[0094] Sparse percentage = 0% Compressed header NNZ CR Method 1100 256.451545 8192 -0.03130512 method 8258 8192 -1.008056641

[0095] It was determined that if the sparsity percentage is greater than 96.875%, then the CRS algorithm outperforms the 1100 algorithm. For all other sparsity percentages, the method 1100 outperforms the CRS algorithm.

[0096] Fig.12An example of an accelerator 1202 is shown, which can be an integrated circuit of a processor (such as a neural network processor). The processor can have other integrated circuit components, including additional accelerator engines. In various examples, for a set of input data (e.g., input data 1250), the accelerator 1202 can use the processing engine array 1210, the activation engine 1216, and / or the pooling engine 1218 to perform calculations.

[0097] In various implementations, the memory subsystem 1204 may include multiple memory banks 1214. In these implementations, each memory bank 1214 may be independently accessible, meaning that the reading of one memory bank does not depend on the reading of another memory bank. Similarly, writing to one memory bank does not affect or limit writing to a different memory bank. In some cases, each memory bank may be read and written simultaneously. Various techniques may be used to have independently accessible memory banks 1214. For example, each memory bank may be a physically separate memory component having an address space that is separate and independent from the address space of each other memory bank. In this example, each memory bank may have at least one read channel and may have at least one separate write channel that may be used simultaneously. In these examples, the memory subsystem 1204 may allow simultaneous access to the read or write channels of multiple memory banks. As another example, the memory subsystem 1204 may include arbitration logic so that, for example, arbitration between the outputs of multiple memory banks 1214 may result in the use of the outputs of more than one memory bank. In these and other examples, while managed globally by the memory subsystem 1204, each memory bank can operate independently of any other memory bank.

[0098] Making the memory banks 1214 independently accessible can improve the efficiency of the accelerator 1202. For example, a value can be read and provided to each row of the processing engine array 1210 simultaneously so that the entire processing engine array 1210 can be used in one clock cycle. As another example, the memory banks 1214 can be read while the results calculated by the processing engine array 1210 are written to the memory subsystem 1204. In contrast, a single memory may only be able to service one read or write at a time. With a single memory, multiple clock cycles may be required, for example, to read the input data for each row of the processing engine array 1210 before the processing engine array 1210 can be started.

[0099] In various implementations, the memory subsystem 1204 can be configured to simultaneously serve multiple clients, including the processing engine array 1210, the activation engine 1216, the pooling engine 1218, and any external client that accesses the memory subsystem 1204 through the communication structure 1220. In some implementations, being able to serve multiple clients means that the memory subsystem 1204 has at least as many memory banks as clients. In some cases, each row of the processing engine array 1210 can be counted as a separate client. In some cases, each column of the processing engine array 1210 can output a result so that each column can be counted as a separate write client. In some cases, the output from the processing engine array 1210 can be written to the memory bank 1214, which can then provide input data for the processing engine array 1210. As another example, the activation engine 1216 and the pooling engine 1218 can include multiple execution channels, each of which can be a separate memory client. The memory bank 1214 can be implemented, for example, using static random access memory (SRAM).

[0100] In various implementations, the memory subsystem 1204 may include control logic. The control logic may, for example, track the address space of each of the memory banks 1214, identify the memory bank 1214 to be read or written, and / or move data between the memory banks 1214. In some implementations, the memory banks 1214 may be hardwired to specific clients. For example, a group of memory banks 1214 may be hardwired to provide values ​​to the rows of the processing engine array 1210, with one memory bank servicing each row. As another example, a group of memory banks may be hardwired to receive values ​​from the columns of the processing engine array 1210, with one memory bank receiving data for each column.

[0101] The processing engine array 1210 is a computational matrix of the example accelerator 1202. The processing engine array 1210 can, for example, perform parallel integration, convolution, correlation, and / or matrix multiplication, among others. The processing engine array 1210 includes a plurality of processing engines 1211 arranged in rows and columns so that a result output by one processing engine 1211 can be directly input to another processing engine 1211. Thus, a processing engine 1211 that is not on the outer edge of the processing engine array 1210 can receive data to be operated on from other processing engines 1211 instead of from the memory subsystem 1204.

[0102] In various examples, the processing engine array 1210 uses systolic execution, where data arrives at each processing engine 1211 from different directions at fixed intervals. In some examples, input data can flow into the processing engine array 1210 from the left, and weight values ​​can be loaded at the top. In some examples, weights and input data can flow from the left, and partial sums can flow from top to bottom. In these and other examples, multiplication and accumulation operations move through the processing engine array 1210 as a diagonal wave front, where data moves right and down across the array. Control signals can be input on the left at the same time as weights, and can flow across and down as calculations are performed.

[0103] In various implementations, the number of columns in processing engine array 1210 determines the computational power of processing engine array 1210, and the number of rows determines the memory bandwidth required to achieve maximum utilization of processing engine array 1210. Processing engine array 1210 may have, for example, 64 columns and 428 rows, or some other number of columns and rows.

[0104] An example of processing engine 1211 is shown in the illustration Fig.12 . As shown in this example, the processing engine 1211 can include a multiplier-accumulator circuit. The input from the left side can include, for example, input data i and weight values ​​w, where the input data is a value taken from a set of input data or a set of intermediate results, and the weight values ​​come from a set of weight values ​​that connect one layer of the neural network to the next layer. A set of input data can be, for example, an image submitted for identification or object recognition, an audio clip provided for speech recognition, a text string for natural language processing or machine translation, or a current state of the game that needs to be analyzed to determine the next step, and others. In some examples, the input data and weight values ​​are output to the right to be input to the next processing engine 1211.

[0105] In the example shown, the input from above may include a partial sum p_in, which is provided from another processing engine 1211 or from a previous round of calculations in the processing engine array 1210. When calculations for a new set of input data are started, the top row of the processing engine array 1210 may receive a fixed value for p_in, such as zero. As shown in this example, i and w are multiplied together and the result is added to p_in to produce a new partial sum p_out, which may be input into another processing engine 1211. Various other implementations of the processing engine 1211 are possible.

[0106] The output from the last row in the processing engine array 1210 may be temporarily stored in the result buffer 1212. The results may be intermediate results that may be written to the memory bank 1214 to be provided to the processing engine array 1210 for additional calculations. Alternatively, the results may be final results that, once written to the memory bank 1214, may be read from the memory subsystem 1204 via the communication fabric 1220 to be output by the system.

[0107] In some implementations, the accelerator 1202 includes an activation engine 1216. In these implementations, the activation engine 1216 can combine the results from the processing engine array 1210 into one or more output activation signals. For example, for a convolutional neural network, convolutions from multiple channels can be added to produce an output activation signal for a single channel. In other examples, it may be necessary to accumulate results from one or more columns in the processing engine array 1210 to produce an output activation signal for a single node in the neural network. In some examples, the activation engine 1216 can be bypassed.

[0108] In various examples, the activation engine 1216 may include multiple separate execution channels. In these examples, the execution channels may correspond to columns of the processing engine array 1210, and may operate on the outputs of the columns, the results of which may be stored in the memory subsystem 1204. In these examples, the activation engine 1216 may be capable of performing 1 to n parallel calculations, where n is equal to the number of columns in the processing engine array 1210. In some cases, one or more of these calculations may be performed simultaneously. Examples of calculations that each execution channel may perform include exponential, square, square root, identity, bipolar step, bipolar step, sigmoidal, and ramp, among others.

[0109] In some implementations, the accelerator 1202 may include a pooling engine 1218. Pooling is a combination of the outputs of the columns of the processing engine array 1210. The combination may include, for example, calculating a maximum value, a minimum value, an average value, a median value, addition, multiplication, or another logical or mathematical combination. In various examples, the pooling engine 1218 may include a plurality of execution channels that may operate on the values ​​of the corresponding columns from the processing engine array 1210. In these examples, the pooling engine 1218 may be capable of performing 1 to n parallel calculations, where n is equal to the number of columns in the processing engine array 1210. In various examples, the execution channels of the pooling engine 1218 may operate in parallel and / or simultaneously. In some examples, the pooling engine 1218 may be bypassed.

[0110] Herein, activation engine 1216 and pooling engine 1218 may be collectively referred to as an execution engine. Processing engine array 1210 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine that may be located outside of accelerator 1202.

[0111] Input data 1250 may arrive via communication fabric 1220. Communication fabric 1220 may connect accelerator 1202 to other components of the processor, such as a DMA engine that may obtain input data 1250 from an input / output (I / O) device, a storage drive, or a network interface. Input data 1250 may be, for example, one-dimensional data (such as a string or sequence of numbers) or two-dimensional data (such as an array of pixel values ​​of an image or frequency and amplitude values ​​of an audio signal that varies over time). In some examples, input data 1250 may be three-dimensional, which may be the case, for example, of contextual information or virtual reality data used by self-driving cars. In some implementations, memory subsystem 1204 may include a separate buffer for input data 1250. In some implementations, when input data 1250 is received by accelerator 1202, input data 1250 may be stored in storage body 1214.

[0112] In some examples, accelerator 1202 can implement a neural network processing engine. In these examples, for a set of input data 1250, accelerator 1202 can execute a neural network to perform the task for which the neural network is trained. Executing a neural network on a set of input data can be referred to as inference or performing inference.

[0113] The weights of the neural network can be stored in the memory subsystem 1204 along with the input data 1250 that the neural network will operate on. The neural network can also contain instructions that can program the processing engine array 1210 to perform various calculations on the weights and input data. The instructions can also be stored in the memory subsystem 1204, the storage body 1214, or a separate instruction buffer. The processing engine array 1210 can output intermediate results representing the outputs of various layers of the neural network. In some cases, the activation engine 1216 and / or the pooling engine 1218 can be enabled for calculations required by certain layers of the neural network. The accelerator 1202 can store the intermediate results in the memory subsystem 1204 for input to the processing engine array 1210 to calculate the results of the next layer of the neural network. The processing engine array 1210 can further output the final result from the last layer of the neural network. The final result can be stored in the memory subsystem 1204 and then copied to the host processor memory or another location.

[0114] Fig.13An example of an acceleration engine 1300 is shown. Acceleration engine 1300 is an example of an integrated circuit that may include one or more accelerators 1302a to 1302n, which may be similar to Fig.12 The accelerator shown.

[0115] exist Fig.13 In the example of, acceleration engine 1300 includes multiple accelerators 1302a to 1302n, each of which can perform a set of operations. In various examples, accelerators 1302a to 1302n are used for specific types of operations so that accelerators 1302a to 1302n can be operated much faster than similar operations performed by a general-purpose processor. In various examples, in order to perform a set of operations, the input data to be operated must first be moved to accelerators 1302a to 1302n. In addition, in some cases, program code is also moved to accelerators 1302a to 1302n, and the program code will program the operations performed by accelerators 1302a to 1302n on the data. In the example shown, acceleration engine 1300 includes n accelerators 1302a to 1302n. The examples of accelerators that can be included in acceleration engine 1300 include graphics accelerators, floating-point accelerators, neural network accelerators, etc. In various examples, accelerators 1302a through 1302n may each be identical (eg, each accelerator is a graphics accelerator) or may be different (eg, accelerators 1302a through 1302n include a graphics accelerator, a floating point accelerator, and a neural network accelerator).

[0116] The example acceleration engine 1300 further includes DRAM controllers 1342a to 1342k for communicating with external memory. In this example, DRAM 1330 is used to implement the external memory. In the example shown, the acceleration engine 1300 includes k DRAM controllers 1342a to 1342k, each of which can communicate with a group of independent DRAM groups. In other examples, other types of RAM technologies can be used for external memory. DRAM controllers 1342a to 1342k can also be called memory controllers.

[0117] In various examples, input data and / or program code for accelerators 1302a to 1302n may be stored in DRAM 1330. Different programs may cause accelerators 1302a to 1302n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program may configure the neural network accelerator to perform speech recognition, while another program may configure the neural network accelerator to perform image recognition. In various examples, different accelerators 1302a to 1302n may be programmed with different programs so that each accelerator performs a different set of operations. In various examples, processors 1348a to 1348s may manage the movement of program code from DRAM 1330 to accelerators 1302a to 1302n.

[0118] The example acceleration engine 1300 further includes I / O controllers 1344a to 1344p for communicating with an I / O device 1332 in the system. The acceleration engine 1300 can communicate with the I / O device through, for example, a processor bus. In some examples, the processor bus can be implemented using a peripheral component interconnect (PCI) and / or a variant of the PCI bus protocol. The processor bus can connect the acceleration engine 1300 to I / O devices, such as input and output devices, memory controllers, storage devices, and / or network interface cards, among others. In some examples, the I / O controllers 1344 to 1344p can enable the acceleration engine 1300 to act as an I / O device for a host processor. For example, the acceleration engine 1300 can be a recipient of input data from a host processor and a command indicating an operation to be performed on the input data (e.g., a specific calculation or analysis). In the illustrated example, the acceleration engine 1300 includes p I / O controllers 1344a to 1344p, each of which may include a separate root complex and may communicate with a separate set of I / O devices 1332. In other examples, other standardized bus protocols, such as Ultra Path Interconnect (UPI), may be used for the host bus. In other examples, a dedicated bus protocol may be used.

[0119] The movement of data within the acceleration engine 1300 may be managed by one or more processors 1348a through 1348s, which may also be referred to as data management processors. Fig.13In the example of , the acceleration engine 1300 includes s processors 1348a to 1348s incorporated into the device (e.g., on the same silicon die). In other examples, the processors 1348a to 1348s may be external to the acceleration engine 1300 (e.g., on different dies and / or in different packages). In some examples, the processors 1348a to 1348s may manage the movement of data from the I / O device 1332 to the accelerators 1302a to 1302n or the DRAM 1330. For example, the input data may be located in the I / O device 1332 or in the processor memory, and the processors 1348a to 1348s may move the input from the I / O device 1332 or the processor memory to the accelerator or the DRAM 1330. As another example, the program code of the accelerators 1302a to 1302n may be located on the I / O device 1332 or in the processor memory.

[0120] The example acceleration engine 1300 further includes DMA engines 1346a to 1346d, which can move data between accelerators 1302a to 1302n, DRAM controllers 1342a to 1342k, and I / O controllers 1344a to 1344p. In the example shown, the acceleration engine 1300 includes d DMA engines 1346a to 1346d. In some implementations, DMA engines 1346a to 1346d can be assigned to specific tasks, such as moving data from DRAM controllers 1342a to 1342d to accelerators 1302a to 1302n, or moving data between I / O controllers 1344a to 1344p and accelerators 1302a to 1302n. For example, these tasks can be assigned by queuing descriptors with DMA engines 1346a to 1346d, where the descriptors identify the address of the data block and the operation to be performed (e.g., read or write). For example, the descriptor may direct the DMA engine to instruct the DMA controller to read a block of data from DRAM 1330. As another example, the descriptor may instruct the DMA engine to write the data read by the DMA controller to the accelerator. Other descriptors may be used to move data from the accelerator to DRAM 1330.

[0121] In various examples, each of the processors 1348a to 1348s can be responsible for managing data movement of different accelerators. In some examples, a processor can manage data movement of more than one accelerator. Similarly, in various examples, each of the processors 1348a to 1348s can be assigned to one or more DMA engines 1346a to 1346d. In these and other examples, the association between the processors 1348a to 1348s, the accelerators 1302a to 1302n, and the DMA engines 1346a to 1346d is determined by the program code being executed by each respective processor.

[0122] In the example acceleration engine 1300, various components may communicate through a chip interconnect 1320. The chip interconnect 1320 primarily includes wiring for passing data between components of the acceleration engine 1300. In some cases, the chip interconnect 1320 may include a minimal amount of logic, such as multiplexers for controlling data direction, flip-flops for handling clock domain crossings, and timing logic.

[0123] Fig.14 An example of a host system 1400 that may use an acceleration engine 1460 is shown. Fig.14 The acceleration engine 1460 may include one or more accelerators (such as Fig.13 An example of a device as shown). Fig.14 The example host system 1400 includes an acceleration engine 1460, a host processor 1472, DRAM 1430 or processor memory, I / O devices 1432, and a support system 1474. In various implementations, the host system 1400 may include other hardware not shown here.

[0124] Host processor 1472 is a general purpose integrated circuit capable of executing program instructions. In some examples, host processor 1472 may include multiple processing cores. A multi-core processor may include multiple processing units within the same processor. In some examples, host system 1400 may include more than one host processor 1472. In some examples, host processor 1472 and acceleration engine 1460 may be a chip, such as one or more integrated circuits within the same package.

[0125] In various examples, the host processor 1472 can communicate with other components in the host system 1400 through one or more communication channels. For example, the host system 1400 can include a host processor bus, which the host processor 1472 can use, for example, to communicate with the DRAM 1430. As another example, the host system 1400 can include an I / O bus (such as a PCI-based bus), through which the host processor 1472 can communicate, for example, with the acceleration engine 1460 and / or the I / O device 1432. In various examples, the host system 1400 can alternatively or additionally include other communication channels or buses, such as a serial bus, a power management bus, a storage device bus, etc.

[0126] In some examples, a software program executed on the host processor 1472 can receive or generate input for processing by the acceleration engine 1460. In some examples, the program can select the appropriate neural network to be executed for a given input. For example, the program can be used for language translation, and one or more neural networks capable of speech recognition and / or machine translation can be selected. In these and other examples, the program can configure the acceleration engine 1460 with the neural network to be executed, and / or a neural network processing engine that has been previously configured to execute the desired neural network can be selected on the acceleration engine 1460. In some examples, once the acceleration engine 1460 has begun reasoning about the input data, the host processor 1472 can manage the movement of data (such as weights, instructions, intermediate results, results of conditional layers, and / or final results) into and out of the acceleration engine 1460.

[0127] In some examples, a software program that is using acceleration engine 1460 for inference can read results from a conditional layer of acceleration engine 1460 and / or from a storage location such as DRAM 1430. In these examples, the program can determine what action the neural network should take next. For example, the program can determine to terminate the inference. As another example, the program can determine to change the direction of the inference, which can be translated by lower-level code and / or the neural network processor into the next layer to execute. In these and other examples, the execution flow of the neural network can be coordinated by software.

[0128] DRAM 1430 is memory used by host processor 1472 to store program code being executed by host processor 1472 and values ​​being operated on. In some examples, data for a neural network (e.g., weight values, instructions, and other data) can be stored in whole or in part in DRAM 1430. DRAM is a common term for processor memory, and although DRAM is volatile memory, processor memory can be volatile and / or non-volatile. Although not shown here, host system 1400 may include other volatile and non-volatile memory for other purposes. For example, host system 1400 may include read-only memory (ROM) that stores boot code and / or basic input / output system (BIOS) code for starting host system 1400 at power-on.

[0129] Although not shown here, DRAM 1430 may store instructions for various programs that may be loaded into and executed by host processor 1472. For example, DRAM 1430 may store instructions for an operating system, one or more data stores, one or more application programs, one or more drivers, and / or services for implementing the functionality disclosed herein.

[0130] The operating system can manage and coordinate the overall operation of the host system 1400, for example, scheduling tasks, executing applications, and / or controlling peripheral devices, among other operations. In some examples, the host system 1400 can host one or more virtual machines. In these examples, each virtual machine can be configured to execute its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. The operating system can be an alternative or additional proprietary operating system.

[0131] The data store may contain permanent or temporary data used and / or operated by an operating system, application, or driver. Examples of processed data include web pages, video data, audio data, images, user data, etc. In some examples, the information in the data store may be provided to a user device via a network. In some cases, the data store may additionally or alternatively include stored applications and / or drivers. The data store may alternatively or additionally store standard and / or proprietary software libraries and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data store may be computer-readable object code, source code, interpreted code, or intermediate code.

[0132] Drivers may include programs that provide communication between components of the host system 1400. For example, some drivers may provide communication between the operating system and peripheral devices or I / O devices 1432. Some drivers may alternatively or additionally provide communication between applications and the operating system and / or between applications and peripheral devices accessible to the host system 1400. In many cases, drivers may include drivers that provide well-understood functionality (e.g., printer drivers, display drivers, hard disk drivers, solid-state device drivers, etc.). In other cases, drivers may provide proprietary or specialized functionality.

[0133] I / O devices 1432 may include devices such as hardware for connecting to user input and output devices (e.g., keyboard, mouse, pen, tablet, voice input device, touch input device, display or monitor, speaker, and printer, etc.). I / O devices 1432 may also include storage drives and / or network interfaces for connecting to network 1480. For example, in various examples, host system 1400 may use a network interface to communicate with storage devices, user terminals, other computing devices or servers, and / or other networks.

[0134] In various examples, one or more I / O devices 1432 may be storage devices. In these examples, the storage device includes non-volatile memory and may store program instructions and / or data. Examples of storage devices include magnetic storage, optical disks, solid-state drives, flash memory, and / or tape storage, among others. The storage device may be housed in the same chassis as the host system 1400, or it may be stored in an external housing. The storage device may be fixed (e.g., connected by screws) or removable (e.g., with a physical release mechanism and possibly a hot-swap mechanism).

[0135] The storage devices, DRAM 1430, and any other memory components in the host system 1400 are examples of computer-readable storage media. Computer-readable storage media are physical media that can store data in a format that can be read by a device such as the host processor 1472. Computer-readable storage media can be non-transitory. Non-transitory computer-readable media can retain data stored thereon when the media has no power. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tapes, optical disks, flash memory devices, and solid-state hard drives, among others. As used herein, computer-readable storage media do not include computer-readable communication media.

[0136] In various examples, data stored on a computer-readable storage medium may include program instructions, data structures, program modules, libraries, other software program components, and / or other data that may be transmitted within a data signal (such as a carrier wave or other transmission). A computer-readable storage medium may additionally or alternatively include documents, images, video, audio, and other data that may be operated on or manipulated using a software program.

[0137] In various examples, one or more I / O devices 1432 may be PCI-based devices. In these examples, PCI-based I / O devices include a PCI interface for communicating with the host system 1400. The term "PCI" or "PCI-based" may be used to describe any protocol in the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and PCI-Express (PCIe), or any other improved or derived protocol based on the PCI protocol discussed herein. A PCI-based protocol is a standard bus protocol for connecting devices (such as local peripherals) to a host device. A standard bus protocol is a data transfer protocol whose specifications have been specified and adopted by various manufacturers. Manufacturers ensure that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using non-volatile memory express (NVMe). NVMe is a device interface specification for accessing non-volatile storage media connected to a computing system using PCIe.

[0138] A PCI-based device may include one or more functions. "Function" describes the hardware and / or software that can be operated by a PCI-based device. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers. In some cases, a PCI-based device may include multiple functions. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some embodiments, a PCI-based device may have up to 8 functions.

[0139] In some examples, a PCI-based device may include single root I / O virtualization (SR-IOV). SR-IOV is an extended feature that can be included in a PCI-based device. SR-IOV allows a physical resource (such as a single network interface controller) to appear as multiple virtual resources (such as 64 network interface controllers). Therefore, from the perspective of a device using a PCI-based device, a PCI-based device (such as a network interface controller) that provides a certain function may be multiple devices that provide the same function. The functions of a storage adapter device with SR-IOV capabilities can be classified as a physical function (PF) or a virtual function (VF). A physical function is a fully functional function of a device that can be discovered, managed, and manipulated. A physical function has configuration resources that can be used to configure or control a storage adapter device. A physical function includes the same configuration address space and memory address space as a non-virtualized device. A physical function may have many virtual functions associated with it. A virtual function is similar to a physical function, but is a lightweight function that may typically lack configuration resources and is typically controlled by the configuration of its underlying physical function. Each of the physical functions and / or virtual functions can be assigned to a corresponding execution thread (such as a virtual machine) running on a host device.

[0140] In various embodiments, the support system 1474 can include hardware for coordinating the operation of the acceleration engine 1460. For example, the support system 1474 can contain a microprocessor that coordinates the activities of the acceleration engine 1460, including moving data around on the acceleration engine 1460. In this example, the microprocessor can be an integrated circuit that can execute microcode. Microcode is a type of program code that can give an integrated circuit some flexibility in the operations that the integrated circuit can perform, but because the program code uses a limited set of instructions, the functionality of the microprocessor may be more limited than the functionality of the host processor 1472. In some examples, the program executed by the microprocessor is stored on the hardware of the microprocessor or on a non-volatile memory chip in the host system 1400. In some examples, the microprocessor and the acceleration engine 1460 can be on a chip, for example, an integrated circuit on the same die and in the same package.

[0141] In some examples, support system 1474 may be responsible for receiving instructions from host processor 1472 when a program executing on host processor 1472 requests execution of a neural network. For example, host processor 1472 may provide support system 1474 with a set of input data and a task to be performed on the set of input data. In this example, support system 1474 may identify a neural network that can perform the task and may program acceleration engine 1460 to execute the neural network on the input data set. In some examples, support system 1474 need only select the appropriate neural network processing engine of the neural network processor. In some examples, support system 1474 may need to load data for the neural network onto acceleration engine 1460 before acceleration engine 1460 can begin executing the neural network. In these and other examples, support system 1474 may further receive the output of executing the neural network and provide the output back to host processor 1472.

[0142] In some examples, the operations of support system 1474 may be handled by host processor 1472. In these examples, support system 1474 may not be required and may be omitted from host system 1400.

[0143] In various examples, host system 1400 may include a combination of a host system, processor nodes, storage subsystems, and I / O chassis representing a user device, a service provider computer, or a third party computer.

[0144] A user device may include a computing device for accessing an application (e.g., a web browser or a mobile device application). In some examples, the application may be hosted, managed, and / or provided by a computing resource service or service provider. The application may enable a user to interact with a service provider computer, for example, to access web content such as web pages, music, videos, etc. A user device may be a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a portable computer, a netbook computer, a desktop computer, a thin client device, a tablet computer, an e-book reader, a game console, etc. In some examples, a user device may communicate with a service provider computer via one or more networks. In addition, a user device may be part of a distributed system managed and controlled by a service provider computer (e.g., a console device integrated with a service provider computer), or may be part of a service provider computer.

[0145] The host system 1400 may also represent one or more service provider computers. The service provider computer may provide a native application configured to run on a user device, with which the user may interact. In some examples, the service provider computer may provide computing resources, such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, and the like. The service provider computer may also be used to provide website hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, and the like. In some examples, the service provider computer may be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that are rapidly provisioned and released. These computing resources may include computing, network, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. The service provider computer may include one or more servers, which may be arranged in a cluster as a server farm or as individual servers that are not associated with each other, and may host applications and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some examples, the service provider computer may additionally or alternatively include a computing device such as a mobile phone, a smart phone, a personal digital assistant (PDA), a portable computer, a netbook computer, a desktop computer, a thin client device, a tablet computer, a gaming console, etc. In some cases, the service provider computer may communicate with one or more third-party computers.

[0146] Fig.15 An example network 1500 is shown, which may include one or more host systems, such as Fig.14 The host system shown in 14. For example, Fig.15The example network 1500 includes a plurality of nodes 1502a-1502h, one or more of which may be Fig.14 The host system shown. Other nodes in the nodes 1502a-1502h may be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing instructions, and a network interface for connecting to the network 1500.

[0147] In various examples, network 1500 can be used to process data. For example, input data can be received at one of nodes 1502a-1502h or from other networks 1508 with which network 1500 can communicate. In this example, the input data can be directed to a node in network 1500 that includes an acceleration engine so that the acceleration engine operates and produces a result. The result can then be transmitted to a node or other network from which the input data is received. In various examples, input data can be accumulated from various sources including one or more of nodes 1502a-1502h and / or computing devices located in other networks 1508, and the accumulated input data can be directed to one or more host systems in network 1500. The results from the host system can then be distributed back to the source that collected the input data.

[0148] In various examples, one or more of nodes 1502a-1502h may be responsible for accumulating input data required for operation of the host systems, tracking which host systems are busy and which host systems can accept more work, determining whether the host systems are operating correctly and / or most efficiently, monitoring network security, and / or other management operations.

[0149] exist Fig.15 In the example of , nodes 1502a-1502h are connected to each other using a switching architecture with point-to-point links. The switching architecture includes multiple switches 1504a-1504d, which can be arranged in a multi-layer network of a Clos network. A network device that filters and forwards packets between local area network (LAN) segments can be called a switch. Switches usually operate at the data link layer (layer 2) of the open system interconnection (OSI) reference model, but sometimes at the network layer (layer 3), and can support multiple packet protocols. Fig.15 The switches 1504a-1504d can be connected to the nodes 1502a-1502h and provide multiple paths between any two nodes.

[0150] The network 1500 may also include one or more network devices (such as routers 1506) for connecting to other networks 1508. Routers use headers and forwarding tables to determine the best path to forward packets, communicate with each other using protocols such as the Internet Control Message Protocol (ICMP), and configure the best route between any two devices. Fig.15 The router 1506 may be used to connect to other networks 1508 such as a subnet, a LAN, a wide area network (WAN), and / or the Internet.

[0151] In some examples, the network 1500 may include any one or more combinations of many different types of networks, such as cable television networks, the Internet, wireless networks, mobile phone networks, and other private and / or public networks. The interconnected switches 1504a-1504d and routers 1506 (if any) may be referred to as a switching fabric 1510, a fabric, a network fabric, or simply a network. In the context of computer networks, the terms "fabric" and "network" may be used interchangeably herein.

[0152] Nodes 1502a-1502h may be any combination of host systems, processor nodes, storage subsystems, and I / O chassis representing user equipment, service provider computers, or third-party computers.

[0153] The user device may include a computing device for accessing an application 1532 (e.g., a web browser or mobile device application). In some aspects, the application 1532 may be hosted, managed, and / or provided by a computing resource service or service provider. The application 1532 may allow a user to interact with a service provider computer, for example, to access web content such as web pages, music, and videos. The user device may be a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a portable computer, a netbook computer, a desktop computer, a thin client device, a tablet computer, an e-book reader, a game console, and the like. In some examples, the user device may communicate with the service provider computer via other networks 1508. In addition, the user device may be part of a distributed system managed and controlled by a service provider computer (e.g., a console device integrated with the service provider computer), or may be part of a service provider computer.

[0154] Fig.15The nodes may also represent one or more service provider computers. One or more service provider computers may provide a native application that is configured to run on a user device, and the user may interact with it. In some examples, the service provider computer may provide computing resources, such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, etc. The service provider computer may also be used to provide website hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, etc. to users. In some examples, the service provider computer may be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that are quickly provisioned and released. These computing resources may include computing, network and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. The service provider computer may include one or more servers, which may be arranged in a cluster as a server farm or as individual servers that are not associated with each other, and may host applications 1532 and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some aspects, the service provider computer may additionally or alternatively include a computing device such as a mobile phone, a smart phone, a personal digital assistant (PDA), a portable computer, a netbook computer, a desktop computer, a thin client device, a tablet computer, a gaming console, etc. In some cases, the service provider computer may communicate with one or more third-party computers.

[0155] In one example configuration, nodes 1502a-1502h may include at least one memory 1518 and one or more processing units (or processors 1520). Processor 1520 may be implemented in hardware, computer executable instructions, firmware, or a combination thereof. Computer executable instructions or firmware implementations of processor 1520 may include computer executable or machine executable instructions written in any suitable programming language to perform the various functions described.

[0156] In some cases, the hardware processor 1520 can be a single-core processor or a multi-core processor. A multi-core processor can include multiple processing units within the same processor. In some examples, the multi-core processors can share certain resources, such as a bus and a second or third level cache. In some cases, each core in a single-core or multi-core processor can also include multiple execution logical processors (or execution threads). In such cores (e.g., those servers with multiple logical processors), multiple stages of the execution pipeline and lower-level caches can also be shared.

[0157] The memory 1518 may store program instructions that are loadable and executable on the processor 1520, as well as data generated during the execution of these programs. Depending on the configuration and type of the nodes 1502a-1502h, the flash memory 1518 may be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). The memory 1518 may include an operating system 1528, one or more data stores 1530, one or more application programs 1532, one or more drivers 1534, and / or services for implementing the functionality disclosed herein.

[0158] The operating system 1528 can support the basic functions of the nodes 1502a-1502h, such as scheduling tasks, executing applications and / or controlling peripheral devices. In some embodiments, the service provider computer can host one or more virtual machines. In these embodiments, each virtual machine can be configured to execute its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. The operating system 1528 can also be a proprietary operating system.

[0159] The data store 1530 may contain permanent or temporary data used and / or operated by the operating system 1528, the application 1532, or the driver 1534. Examples of processed data include web pages, video data, audio data, images, user data, etc. In some embodiments, the information in the data store 1530 may be provided to a user device via the network 1508. In some cases, the data store 1530 may additionally or alternatively include stored applications and / or drivers. The data store 1530 may alternatively or additionally store standard and / or proprietary software libraries and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data store 1530 may be computer-readable object code, source code, interpreted code, or intermediate code.

[0160] Driver 1534 includes programs that can provide communication between node components. For example, some drivers 1534 can provide communication between operating system 1528 and additional storage 1522, network device 1524 and / or I / O device 1526. Some drivers 1534 can provide communication between application program 1532 and operating system 1528 and / or application program 1532 and service provider computer accessible peripheral device alternatively or additionally. In many cases, driver 1534 can include the driver program (for example, printer driver, display driver, hard disk driver, solid state device driver) that provides easy-to-understand function. In other cases, driver 1534 can provide proprietary or special function.

[0161] The service provider computer or server may also include additional memory 1522, which may include removable memory and / or non-removable memory. Additional memory 1522 may include tape storage, optical disks, solid-state disks, flash memory, and / or tape storage. Additional memory 1522 may be housed in the same chassis as nodes 1502a-1502h, or in an external housing. Memory 1518 and / or additional memory 1522 and its associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some embodiments, memory 1518 may include multiple different types of memory, such as SRAM, DRAM, or ROM.

[0162] Removable and non-removable memory 1518 and additional storage 1522 are examples of computer-readable storage media. For example, computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in a method or technology for storing information, including computer-readable instructions, data structures, program modules or other data, etc. Memory 1518 and additional storage 1522 are examples of computer storage media. Other types of computer storage media that may be present in nodes 1502a-1502h may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical storage devices, cassettes, magnetic tapes, disk storage or other magnetic storage devices, solid-state hard drives, or some other medium that can be used to store the desired information and can be accessed by nodes 1502a-1502h. Computer-readable media also include combinations of any of the above media types, including multiple units of a media type.

[0163] Computer readable communication media may alternatively or additionally include computer readable instructions, program modules or other data transmitted in a data signal (such as a carrier wave or other transmission). However, as used herein, computer readable storage media does not include computer readable communication media.

[0164] Nodes 1502a-1502h may also include I / O devices 1526, such as keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, etc. Nodes 1502a-1502h may also include one or more communication channels 1536. Communication channels 1536 may provide a medium through which the various components of nodes 1502a-1502h can communicate. One or more communication channels 1536 may take the form of a bus, a ring network, a switching structure, or a network.

[0165] The nodes 1502a - 1502h may also include a network device 1524 that allows the nodes 1502a - 1502h to communicate with a stored database, another computing device or server, a user terminal, and / or other devices on the network 1500 .

[0166] In some embodiments, the network device 1524 is a peripheral device (such as a PCI-based device). In these embodiments, the network device 1524 includes a PCI interface for communicating with a host device. The term "PCI" or "PCI-based" can be used to describe any protocol in the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP) and PCI-Express (PCIe) or any other improved or derived protocol based on the PCI protocol discussed herein. The PCI-based protocol is a standard bus protocol for connecting devices (such as local peripherals) to host devices. The standard bus protocol is a data transmission protocol whose specifications have been specified and adopted by various manufacturers. The manufacturer ensures that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using non-volatile memory express (NVMe). NVMe is a device interface specification for accessing non-volatile storage media connected to a computing system using PCIe. For example, the bus interface module can implement NVMe, and the network device 1524 can be connected to the computing system using the PCIe interface.

[0167] A PCI-based device may include one or more functions. "Function" describes an operation that may be provided by network device 1524. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers, etc. In some cases, a PCI-based device may include multiple functions. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some embodiments, a PCI-based device may have up to 8 functions.

[0168] In some embodiments, the network device 1524 may include single root I / O virtualization (SR-IOV). SR-IOV is an extended function that can be included in PCI-based devices. SR-IOV allows a physical resource (such as a single network interface controller) to appear as multiple resources (such as 64 network interface controllers). Therefore, from the perspective of a device using a PCI-based device, a PCI-based device (such as a network interface controller) that provides a certain function may be multiple devices that provide the same function. The functions of a storage adapter device with SR-IOV function can be classified as a physical function (PF) or a virtual function (VF). A physical function is a fully functional function of a device that can be discovered, managed, and manipulated. A physical function has configuration resources that can be used to configure or control a storage adapter device. A physical function includes the same configuration address space and memory address space as a non-virtualized device. A physical function may have many virtual functions associated with it. A virtual function is similar to a physical function, but is a lightweight function that may typically lack configuration resources and is typically controlled by the configuration of its underlying physical function. Each of the physical functions and / or virtual functions can be assigned to a corresponding execution thread (such as a virtual machine) running on a host device.

[0169] The modules described herein may be software modules, hardware modules, or a suitable combination thereof. If software modules, the modules may be embodied on a non-transitory computer-readable medium and processed by a processor in any computer system described herein. It should be noted that the described processes and architectures may be executed in real time or in an asynchronous mode prior to any user interaction. The modules may be configured as suggested in the above-listed figures, and / or the functionality described herein may be provided by one or more modules that exist in the form of separate modules, and / or the module functionality described herein may be distributed across multiple modules.

[0170] The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

[0171] Other variations are also within the spirit and scope of the invention. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated examples thereof are shown in the drawings and have been described in detail above. However, it should be understood that it is not intended to limit the invention to the specific form or forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions and equivalent forms that fall within the spirit and scope of the invention, as defined by the appended claims.

[0172] In the context of describing the disclosed examples (especially in the context of the following claims), the use of the terms "a" and "an" and "the" and similar referents should be interpreted as covering both singular and plural forms, unless otherwise specified herein or clearly contradicted by the context. The terms "comprising", "having", "including" and "containing" should be interpreted as open terms (i.e., meaning "including but not limited to"), unless otherwise specified. The term "connected" should be interpreted as partially or completely included in, connected to or connected together, even if there are intermediates inserted. Unless otherwise specified herein, the enumeration of numerical ranges herein is intended only to be used as a shorthand method of referring to each individual value belonging to the range separately, and each individual value is incorporated into the specification as if it were individually described in the specification. Unless otherwise specified herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. Unless otherwise required, the use of any and all examples or exemplary language (e.g., "for example") provided herein is intended only to better illustrate the examples of the present invention and does not constitute a limitation on the scope of the present invention. Any language in the specification should not be interpreted as indicating that any unclaimed content is essential for implementing the present invention.

[0173] Unless expressly stated otherwise, use of disjunctive language herein, such as the phrase "at least one of X, Y, or Z," should generally be understood to mean that a certain item, term, etc., can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain examples require that at least one of X, at least one of Y, or at least one of Z each be present.

[0174] Various examples of the present invention are described herein, including the best mode known to the inventor for carrying out the present invention. After reading the foregoing description, variations of those examples will become apparent to those of ordinary skill in the art. The inventor expects that skilled technicians will appropriately adopt such variations, and the inventor wishes to practice the present invention in a manner different from that specifically described herein. Therefore, as permitted by applicable law, the present invention includes all modifications and equivalents of the subject matter described in the appended claims. In addition, unless otherwise indicated herein or clearly contradictory to the context, the present invention encompasses any combination of the above in all possible variations thereof.

Claims

1. A method for exchanging compressed gradient data in a distributed system for training a neural network model, the method comprising: At a transmission worker node of a distributed system, computing a gradient set of a neural network model and a weight set associated with the neural network model, wherein the neural network model is used for recognition or object identification and the gradient set and the weight set are associated with an image submitted for recognition or object identification, the neural network model is used for speech recognition and the gradient set and the weight set are associated with an audio clip provided for speech recognition, the neural network model is used for natural language processing or machine translation and the gradient set and the weight set are associated with a text string used for natural language processing or machine translation, or the neural network model is used for a game and the gradient set and the weight set are associated with a current state of a game that requires analysis to determine a next step; On the transmission worker node, perform sparsity analysis of the gradient set to determine the threshold; At the transmission work node, each of the clipped gradient sets has a value less than a threshold value, thereby causing the gradient set to include non-clipped data elements and clipped data elements; generating a map at a transfer worker node, the map indicating which of the gradient sets correspond to unclipped data elements and which correspond to clipped data elements; At a transmission worker node, generating compressed data including untrimmed data elements from the gradient set; Transmitting the mapped and compressed data from the transmitting worker nodes to the receiving worker nodes of the distributed system; combining unclipped data elements with clipped data elements in the compressed data using a map to generate decompressed data at a receiving worker node such that the decompressed data includes a gradient set including the unclipped data elements and the clipped data elements; and On the receiving worker, a synchronized set of gradients is computed based on the set of gradients and other gradients received at the receiving worker.

2. The method of claim 1, further comprising: At the transmitting worker node, a header is formed including a map and an original length of the gradient set corresponding to the plurality of untrimmed data elements and the trimmed data elements.

3. The method of claim 1, wherein the mapping comprises a bitmap having binary values ​​indicating locations of unclipped data elements and clipped data elements. 4 . The method of claim 1 , wherein clipping of each gradient set comprises setting a value equal to zero such that clipped data elements are zero data elements and unclipped data elements are non-zero data elements.

5. A method for gradient compression in distributed training, the method comprising: At a first transmission worker node of the distributed system, a set of gradients using a neural network model and a set of weights associated with the neural network model are calculated, wherein the neural network model is used for recognition or object identification and the set of gradients and the set of weights are associated with an image submitted for recognition or object identification, the neural network model is used for speech recognition and the set of gradients and the set of weights are associated with an audio clip provided for speech recognition, the neural network model is used for natural language processing or machine translation and the set of gradients and the set of weights are associated with a text string used for natural language processing or machine translation, or the neural network model is used for a game and the set of gradients and the set of weights are associated with a current state of a game that requires analysis to determine a next step; Clipping each of the gradient sets to have a value less than a threshold value, thereby causing the gradient set to include unclipped data elements and clipped data elements, wherein the threshold value is based on a sparsity analysis of the gradient set; generating a map indicating which of the gradient sets correspond to unclipped data elements and which correspond to clipped data elements; generating compressed data from unclipped data elements from the gradient set; and The mapped and compressed data is transferred from a first worker node to a second worker node of the distributed system.

6. The method of claim 5, further comprising: Unclipped data elements in the compressed data are combined with the clipped data elements using mapping, and decompressed data is generated at the second working node to obtain a gradient set including the unclipped data elements and the clipped data elements.

7. The method of claim 5, further comprising: A header including the mapping is formed, wherein the header and the compressed data are transmitted from the first working node to the second working node.

8. The method of claim 7, wherein the header further comprises an original length of the gradient set, the original length corresponding to the number of the untrimmed data elements and the trimmed data elements.

9. The method of claim 5, wherein the mapping comprises a bitmap having binary values ​​indicating locations of the unclipped data elements and the clipped data elements.

10. The method of claim 5, wherein the clipping of each gradient set comprises setting a value equal to zero such that the clipped data elements are zero data elements and the non-clipped data elements are non-zero data elements.

11. The method of claim 5, further comprising: The sparsity analysis is performed on a set of gradients to determine the threshold.

12. The method of claim 11, wherein performing sparsity analysis comprises: Calculate the average of the gradient set; Calculate the standard deviation of the gradient set; as well as The threshold is determined based on the mean and standard deviation.

13. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: At a first transmission working node of the distributed system, a set of gradients using a neural network model and a set of weights associated with the neural network model are calculated, wherein: The neural network model is used for recognition or object identification and the set of gradients and the set of weights are associated with an image submitted for recognition or object identification, the neural network model is used for speech recognition and the set of gradients and the set of weights are associated with an audio clip provided for speech recognition, the neural network model is used for natural language processing or machine translation and the set of gradients and the set of weights are associated with a text string used for natural language processing or machine translation, or the neural network model is used for a game and the set of gradients and the set of weights are associated with a current state of a game that requires analysis to determine a next step; Clipping each of the gradient sets to have a value less than a threshold value, thereby causing the gradient set to include unclipped data elements and clipped data elements, wherein the threshold value is based on a sparsity analysis of the gradient set; generating a map indicating which of the gradient sets correspond to unclipped data elements and which correspond to clipped data elements; generating compressed data from unclipped data elements from the gradient set; as well as The mapped and compressed data is transferred from a first worker node to a second worker node of the distributed system.

14. The non-transitory computer readable medium of claim 13, wherein the operations further comprise: Unclipped data elements in the compressed data are combined with the clipped data elements using mapping, and decompressed data is generated at the second working node to obtain a gradient set including the unclipped data elements and the clipped data elements.

15. The non-transitory computer readable medium of claim 13, wherein the operations further comprise: A header including the mapping is formed, wherein the header and the compressed data are transmitted from the first working node to the second working node.

16. The non-transitory computer-readable medium of claim 15, wherein the header further comprises an original length of the gradient set, the original length corresponding to a number of the untrimmed data elements and the trimmed data elements.

17. The non-transitory computer readable medium of claim 13, wherein the map comprises a bitmap having binary values ​​indicating locations of untrimmed data elements and trimmed data elements.

18. The non-transitory computer-readable medium of claim 13, wherein clipping of each gradient set comprises setting a value equal to zero such that clipped data elements are zero data elements and non-clipped data elements are non-zero data elements.

19. The non-transitory computer readable medium of claim 13, wherein the operations further comprise: The sparsity analysis is performed on a set of gradients to determine the threshold.

20. The non-transitory computer readable medium of claim 19, wherein the sparsity analysis performed comprises: Calculate the average of the gradient set; Calculate the standard deviation of the gradient set; as well as The threshold is determined based on the mean and standard deviation.

Images