Dynamic Processing Element Array Expansion

Abstract

A computer-implemented method includes receiving a neural network model including a tensor operation and dividing the tensor operation into sub-operations. The sub-operations include at least two sub-operations with no data dependencies between the two sub-operations. The computer-implemented method further includes assigning a first of the two sub-operations to a first compute engine, assigning a second of the two sub-operations to a second compute engine, and generating instructions for executing the first sub-operation by the first compute engine and the second sub-operation by the second compute engine in parallel. Inference is then performed based on a result of the first sub-operation, a result of the second sub-operation, or both. The first compute engine and the second compute engine are in the same integrated circuit device or in two different integrated circuit devices.

Description

Background Art

[0001] Artificial neural networks are computing systems with an architecture based on biological neural networks. They can be trained using training data to learn how to perform specific tasks, such as identifying or classifying physical objects, activities, or characters from images or videos. Artificial neural networks can include multiple layers of processing nodes. Each processing node in a layer can perform computations on input data generated by processing nodes in the previous layer to generate output data. For example, a processing node can perform a set of arithmetic operations, such as multiplication and addition, to generate intermediate outputs, or perform post-processing operations on the intermediate outputs.

[0002] Complex artificial neural networks, such as deep neural networks, may include thousands or more processing nodes and millions or more parameters, and therefore each inference may include a large number of computations. For example, in a ResNet-50 neural network, each inference may include approximately 8 billion arithmetic operations. Therefore, the latency for performing inferences may be long because integrated circuits used to implement neural networks may include limited resources, so that many operations (e.g., convolution operations or other tensor operations) may need to be performed in a serial manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0004] Figure 1 An example of a multi-layer artificial neural network is shown;

[0005] Figure 2 An example of a convolutional neural network (CNN) is shown;

[0006] Figure 3A and Figure 3B shows a convolution operation performed on an input pixel array by an example of a convolutional layer in a convolutional neural network;

[0007] Figures 4A to 4E shows examples of convolution, non-linear activation, and pooling operations performed on examples of input pixel data;

[0008] Figure 5 An example of a model for a convolutional layer of a convolutional neural network is shown;

[0009] Figure 6 An example of a convolution operation involving a batch (N=1) of C channels of input data and M sets of C filters is shown;

[0010] Figure 7 is a simplified block diagram illustrating an example of an integrated circuit device for performing neural network operations according to certain embodiments;

[0011] Figure 8 shows a simplified example of a computing engine including an array of processing elements according to certain embodiments;

[0012] Figure 9 shows a simplified example of weight-fixed convolution using an example of a computation engine having an array of processing elements, according to certain embodiments;

[0013] Figure 10A shows a series of operations for performing inference using a neural network model;

[0014] Figure 10B An example of using multiple compute engines for parallel inference is shown;

[0015] Figure 11 An example of operations performed by multiple computing engines to perform parallel inference is shown;

[0016] Figure 12 An example of using multiple computation engines to perform inference by distributing operations of a neural network to the multiple computation engines is shown in accordance with certain embodiments;

[0017] Figure 13 illustrates examples of operations performed by multiple computation engines to perform inferences according to certain embodiments;

[0018] Figure 14 An example of using multiple computation engines to perform inference by splitting the operation of a neural network into sub-operations to be performed in parallel by the multiple computation engines is shown in accordance with certain embodiments;

[0019] Figure 15 illustrates examples of operations performed by multiple computation engines to perform inferences according to certain embodiments;

[0020] Figure 16 An example of using multiple computation engines to perform inference by splitting the operation of a neural network into sub-operations that can be performed in parallel by the multiple computation engines is shown in accordance with certain embodiments;

[0021] Figure 17 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple computing engines is shown according to certain embodiments;

[0022] Figure 18 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple computing engines is shown according to certain embodiments;

[0023] Figure 19 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple computing engines is shown according to certain embodiments;

[0024] Figure 20 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple computing engines is shown according to certain embodiments;

[0025] Figure 21 is a flow chart illustrating an example of a method for accelerating a tensor operation by executing sub-operations of the tensor operation in parallel on multiple compute engines, according to certain embodiments;

[0026] Figure 22A shows an example of a converter that may use the techniques disclosed herein, according to certain embodiments; Figure 22B Shows an example of scaled dot-product attention boxes in the transformer; Figure 22C Shows examples of multi-head attention sublayers in the encoder and decoder for a Transformer;

[0027] Figure 23 shows an example of a Bidirectional Encoder Representation from Transformer (BERT) model that can use some of the techniques disclosed herein, according to certain embodiments;

[0028] Figure 24 A block diagram including an example of a host system;

[0029] Figure 25 A block diagram including an example of an acceleration engine; and

[0030] Figure 26 Includes a diagram of an example of a network. DETAILED DESCRIPTION

[0031] The technology disclosed herein relates generally to artificial neural networks, and more particularly to accelerating inference based on neural network models using multiple compute engines that perform sub-operations of the neural network operation in parallel.

[0032] An artificial neural network may include multiple processing nodes arranged in two or more layers, where processing nodes on one layer may be connected to processing nodes on another layer. Each processing node on a layer may receive a stream of input data elements, multiply each input data element by a weight, calculate a weighted sum of the input data elements, and forward the weighted sum to the next layer. An artificial neural network may include thousands or more processing nodes and millions or more weights and input data elements. In some neural networks, such as convolutional neural networks, transformers including multi-head attention models, multilayer perceptrons, or other neural network models based on tensor operations, large input tensors may be processed to generate new output tensors (e.g., tensor products). Therefore, large memory spaces and processing units with high parallel processing performance may be required. However, the underlying hardware used to implement neural networks may have limited resources, such as a limited number of processing elements and / or limited memory space or memory bandwidth. Therefore, some tensor operations of the neural network may need to be performed serially, which may significantly increase the time period required to perform the operation, and thus the waiting time for inference may be longer than expected, for example in some real-time applications.

[0033] According to certain embodiments, tensor operations such as convolution operations, multi-head attention operations, or multi-layer perceptron operations can be split into sub-operations to be performed in parallel by multiple computing engines in certain ways, so that each computing engine can perform the sub-operations to generate a portion of the final result of the tensor operation (e.g., an output tensor) in a shorter period of time. The portion of the output tensor generated by the sub-operations can be used for early inference and can be cascaded to form the output tensor without the need for additional arithmetic operations. In one example, each computing engine can perform a convolution operation on a portion of the input feature map in a shorter period of time to generate a portion of each of the output feature maps in the tensor output. In another example, each computing engine can perform a convolution operation on the input feature map in a shorter period of time using a portion of the filter for a portion of the output channel to generate a portion of the number of output feature maps. Portions of the output feature maps can be used individually or in combination to make early predictions or decisions. In this way, the latency for performing inference can be significantly reduced, for example, by the number of computing engines used to perform the sub-operations in parallel. This can also effectively create computing engines with higher computing power.

[0034] In the following description, various examples will be described. For the purpose of explanation, specific configurations and details will be set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without the specific details. In addition, in order not to obscure the described embodiments, well-known features may be omitted or simplified. The drawings and descriptions are not intended to be limiting. The terms and expressions adopted in this disclosure are used as descriptive and non-restrictive terms, and when such terms and expressions are used, it is not intended to exclude any equivalents or portions thereof of the features shown and described. The word "example" is used herein to mean "used as an example, instance or illustration." Any embodiment or design described herein as an "example" is not necessarily to be interpreted as being preferred or advantageous over other embodiments or designs.

[0035] Artificial neural networks (also referred to as "neural networks") have been used in machine learning research and industrial applications and have achieved many breakthroughs in areas such as image recognition, speech recognition, computer vision, and text processing. The basic computational unit in a neural network is a neuron (also referred to as a node). A neuron can receive inputs from other neurons or external sources and compute an output. Each input can have an associated weight (w), which can be assigned based on the importance of the input relative to other inputs. A neuron can also apply a function (e.g., a nonlinear function) to the weighted sum of its inputs.

[0036] An artificial neural network may include a plurality of processing nodes arranged in two or more layers, wherein a processing node on one layer may be connected to a processing node on another layer. The processing nodes may be divided into a plurality of layers, including, for example, an input layer, a plurality of intermediate layers (also referred to as hidden layers), and an output layer. Each processing node on a layer (e.g., an input layer, an intermediate layer, etc.) may receive a sequential stream of input data elements, multiply each input data element by a weight, calculate a weighted sum of the input data elements, and forward the weighted sum to the next layer. An artificial neural network, such as a convolutional neural network, may include thousands or more processing nodes and millions or more weights and input data elements.

[0037] A feedforward neural network is a type of artificial neural network. It can include multiple nodes arranged in multiple layers. Nodes from adjacent layers can have connections or edges between them. These connections can have corresponding weights associated with them. Information can flow from input nodes through hidden nodes (if any) and to output nodes. There are no rings or loops in the network. In contrast, a recurrent neural network (e.g., a long short-term memory (LSTM) network) can include connections between nodes to form rings or loops.

[0038] Figure 1An example of a multi-layer neural network 100 is shown. The multi-layer neural network 100 may include an input layer 110, a hidden (or intermediate) layer 120, and an output layer 130. In many implementations, the multi-layer neural network 100 may include two or more hidden layers and may be referred to as a deep neural network. A neural network with a single hidden layer may generally be sufficient to model any continuous function. However, when compared to a neural network with multiple hidden layers, such a network may require an exponentially larger number of nodes. It has been shown that deeper neural networks can be trained to achieve performance far superior to relatively shallow networks.

[0039] The input layer 110 may include multiple input nodes (e.g., nodes 112, 114, and 116), which may provide information (e.g., input data) from the outside world to the network. The input nodes may pass information to the next layer, and the input nodes may not perform calculations. The hidden layer 120 may include multiple nodes, such as nodes 122, 124, and 126. The nodes in the hidden layer may not have direct contact with the outside world (hence the name "hidden"). They may perform calculations and transmit information from the input nodes to the next layer (e.g., another hidden layer or output layer 130). Although a feedforward neural network may have a single input layer and a single output layer, it may have zero or more hidden layers. The output layer 130 may include multiple output nodes, which are responsible for computing and transmitting information from the network to the outside world, such as identifying certain objects or activities, or determining conditions or actions.

[0040] like Figure 1 As shown in Figure 1, in a feedforward neural network, a node (except the bias node, if any) can be connected to all nodes in the immediately previous layer and the immediately next layer (except the bias node, if any). Therefore, these layers can be called fully connected layers. All connections between nodes can have weights associated with them, but Figure 1 Only some of these weights are shown in .For a complex network, there may be hundreds or thousands of nodes, and thousands or millions of connections between nodes.

[0041] As described above, a feedforward neural network may include zero hidden layers (referred to as a single-layer perceptron) or one or more hidden layers (referred to as a multi-layer perceptron (MLP)). Figure 1 Only a single hidden layer in a multilayer perceptron is shown, but a multilayer perceptron can include one or more hidden layers (in addition to an input layer and an output layer). A feedforward neural network with many hidden layers is referred to as a deep neural network. A single-layer perceptron can only learn linear functions, while a multilayer perceptron can learn nonlinear functions.

[0042] exist Figure 1In the example shown, node 112 can be a bias node with a value of 1 or can be a regular input node. Nodes 114 and 116 can take external inputs X1 and X2, which can be numerical values ​​depending on the input data set. As discussed above, no computation is performed on the input layer 110, and therefore the outputs from nodes 112, 114, and 116 on the input layer 110 are 1, X1, and X2, respectively, which are fed to the hidden layer 120.

[0043] exist Figure 1 In the example shown, node 122 may be a bias node with a value of 1 or may be a regular network node. The outputs of nodes 124 and 126 in hidden layer 120 may depend on the outputs from input layer 110 (e.g., 1, X1, X2, etc.) and the weights associated with connection 115. For example, node 124 may take numeric inputs X1 and X2 and may have weights w1 and w2 associated with these inputs. Additionally, node 124 may have another input (called a bias), such as 1, with a weight w0 associated with the input. The main role of the bias is to provide a trainable constant value for each node (in addition to the normal inputs that the node receives). The bias value allows one to shift the activation function to the left or right. It should be noted that although Figure 1 Only three inputs to node 124 are shown in FIG, but in various implementations, a node may include tens, hundreds, thousands, or more inputs and associated weights.

[0044] The output Y from node 124 can be calculated as follows:

[0045] Y=f(w1×X1+w2×X2+w0×deviation), (1)

[0046] Wherein the function f may be a nonlinear function commonly referred to as an activation function. When a node has K inputs, the output from the node may be calculated as follows:

[0047]

[0048] Thus, the computation on each neural network layer can be described as the multiplication of the input matrix and the weight matrix and the activation function applied to the product of the matrix multiplication. The output from the nodes on the intermediate layer can then be fed to the nodes on the next layer, such as the output layer 130.

[0049] The purpose of an activation function is to introduce nonlinearity into the output of a neural network node, since most real-world functions are nonlinear, and it is expected that neurons can learn these nonlinear representations. An activation function can take a single number and perform some fixed mathematical operation on it. Several activation functions can be used in artificial neural networks. One example activation function is the sigmoid function σ(x), which takes a real-valued input and transforms it into a value between 0 and 1. Another example activation function is the hyperbolic tangent function, which takes a real-valued input and transforms it into a value in the range [-1, 1]. A third example activation function is the rectified linear unit (ReLU) function. The ReLU function takes a real-valued input and sets a threshold above zero on the real-valued input (e.g., replacing negative values ​​with zero). Another example activation function is the leaky ReLU function.

[0050] Figure 1 The output layer 130 in the example shown may include nodes 132 and 134, which may take input from the hidden layer 120 and use the weights associated with the connections 125 to perform calculations similar to those of the hidden nodes. The results of the calculations (Y1 and Y2) are the outputs of the multilayer perceptron. In some implementations, a Softmax function may be used as an activation function in the output layer in an MLP for classification. The Softmax function takes a vector of real-valued scores and maps the vector to a vector of values ​​between 0 and 1 that sum to 1.

[0051] Given a set of input features X = (x1, x2, ...) and a target Y, a multilayer perceptron can learn the relationship between the features and the target to perform classification or regression. Multilayer perceptrons can use the backpropagation algorithm to learn. Error backpropagation (commonly referred to as BackProp) is one of several ways to train artificial neural networks. BackProp can be a supervised training scheme that learns from labeled training data and errors at the nodes by changing the parameters of the neural network to reduce the error.

[0052] As described above, the connections between nodes in adjacent layers of an artificial neural network have weights associated with them, where the weights determine what output vector is generated for a given input vector. The learning or training process can assign appropriate weights to these connections. In some implementations, the initial values ​​of the weights can be randomly assigned. For each input in the training data set, the output of the artificial neural network can be observed and compared with the expected output, and the error between the expected output and the observed output can be backpropagated to the previous layer. The weights can be adjusted accordingly based on the error. This process is repeated until the output error falls below a predetermined threshold.

[0053] In many cases, it may be impractical to use a feedforward neural network as described above for real-world applications such as image classification. For example, for a two-dimensional (2-D) image with 200×200 pixels, 40,000 input nodes may be used in the neural network. If the hidden layer has 20,000 nodes, the size of the weight matrix will be 40,000×20,000 (or 800 million elements). If each weight is a 32-bit (i.e., 4-byte) floating point value, the total memory used for the weights will be 3.2GB. This is just for the first layer. As the number of layers increases, the size of the weights may also increase. In addition, vectorizing the image using individual pixels may ignore the complex multidimensional spatial structure of the image.

[0054] One way to overcome these problems is to use a convolutional neural network, which uses smaller convolution filters rather than large matrix multiplications as described above to perform convolution. Learning a set of convolution filters (e.g., 11×11 matrices) may be much easier and faster than learning large matrices (e.g., 40,000×20,000). Multidimensional convolution or other tensor operations can also naturally take the multidimensional structure of the image into account. Convolutional neural networks can be thought of as feedforward neural networks with local connections and weight sharing. Local connections refer to the fact that the convolution filters may have dimensions much smaller than the image they operate on. Weight sharing is due to the fact that the same filter may be used on the entire image when performing convolution, which means that the same local filter is used at many locations in the image. In other words, the weights between all filters at different locations in the image are shared.

[0055] A convolutional neural network (ConvNet or CNN) can perform operations including, for example, (1) convolution; (2) nonlinear (or activation) functions (e.g., ReLU); (3) pooling or subsampling; and (4) classification. Different CNNs may have different combinations of these four main operations, as well as other additional operations. For example, a ResNet-50 network may include network layers that primarily include convolutional layers and some pooling layers, and may also perform residual addition operations for residual learning.

[0056] Figure 2An example of a convolutional neural network (CNN) 200 for classifying images or other objects is shown. As described above, CNN 200 can perform four types of operations, including convolution, nonlinear (or activation) functions (e.g., ReLU), pooling or subsampling, and classification (fully connected layers). An object 210 to be classified, such as one or more input images or other input data sets (referred to as input feature maps), can be represented by a matrix of pixel values. For example, object 210 can include multiple channels (e.g., multiple input feature maps), each channel representing a component of object 210. For example, an image from a digital camera can have at least a red channel, a green channel, and a blue channel, each of which can be represented by a 2-D pixel matrix having pixel values ​​in a range of, for example, 0 to 255 (i.e., 8 bits). A grayscale image may have only one channel. In the following description, processing of a single image channel using CNN 200 is described. Other channels can be processed similarly.

[0057] like Figure 2 As shown, an object 210 (e.g., an input image) may first be processed by a first convolutional layer 215 using a first set of filters, wherein the first convolutional layer 215 may perform a convolution between a matrix representing the input image and a matrix representing each filter in the first set of filters. The convolution may include multiple matrix multiplications. The first convolutional layer 215 may also perform a nonlinear activation function (e.g., ReLU). The output matrix 220 from the first convolutional layer 215 may have a smaller dimension than the input image and may be referred to as a convolution feature, activation map, or feature map. The first convolutional layer 215 may perform a convolution on the input image using the first set of filters to generate a plurality of output matrices 220, which may be referred to as output feature maps of the first convolutional layer 215. The number of filters used may be referred to as the depth of the convolutional layer. Figure 2 In the example shown, the depth of the first convolutional layer 215 may be 3. Each output matrix 220 (eg, output feature map) may be passed to a pooling layer 225 , where each output matrix 220 may be subsampled or downsampled to generate a matrix 230 .

[0058] Each matrix 230 may be processed by a second convolutional layer 235 using a second set of filters. A non-linear activation function (e.g., ReLU) may also be performed by the second convolutional layer 235 as described above. The output matrix 240 (e.g., output feature map) from the second convolutional layer 235 may have a smaller dimension than the matrix 230. The second convolutional layer 235 may perform convolution on the matrix 230 using a second set of filters to generate a plurality of output matrices 240. Figure 2 In the example shown, the depth of the second convolutional layer 235 may be 6. Each output matrix 240 may be passed to a pooling layer 245 , where each output matrix 240 may be subsampled or downsampled to generate an output matrix 250 .

[0059] The output matrix 250 from the pooling layer 245 can be flattened into a vector by the flattening layer 255 and passed through the fully connected layer 260 (e.g., a multilayer perceptron (MLP)). The fully connected layer 260 may include an input layer 270, which takes the 2-D output vector from the flattening layer 255. The fully connected layer 260 may also include a hidden layer and an output layer 290. The fully connected layer 260 may use the feature map or output matrix 250 and, for example, a softmax function to classify objects in the input image into one of several categories. The operation of the fully connected layer can be represented by matrix multiplication. For example, if there are M nodes on the input layer 270 and N nodes on the hidden layer 280, and the weights of the connections between the M nodes on the input layer 270 and the N nodes on the hidden layer 280 can be represented by a matrix W, then the output Y of the hidden layer 280 can be determined by Y=X×W.

[0060] The convolution operation in CNN can be used to extract features from the input image. The convolution operation can preserve the spatial relationship between pixels by extracting image features using small areas of the input image. In convolution, a matrix (called a filter, kernel, or feature detector) can be slid across the input image (or feature map) with a specific step size (called the stride). For each position (or each step), the element-by-element multiplication between the filter matrix and the overlap matrix in the input image can be calculated and summed to obtain the final value representing a single element of the output matrix (e.g., feature map). The filter can be used to detect certain features from the original input image.

[0061] Convolution of an input pixel array using one filter can be used to produce one feature map, and convolution of the same input pixel array using another filter can produce a different feature map. In practice, CNNs can learn the weights of the filters themselves during the training process based on some user-specified parameters (which can be called hyperparameters) such as the number of filters, filter size, and the architecture of the network. The greater the number of filters used, the more image features that can be extracted, and the better the network may be at recognizing patterns in new images.

[0062] The size of the output feature map can be determined based on parameters such as depth, stride, and zero padding. As described above, the depth can correspond to the number of filters used for the convolution operation. For example, in Figure 2In the illustrated CNN 200, three different filters are used in the first convolutional layer 215 to perform a convolution operation on the input image, thereby producing three different output matrices (or feature maps) 220. The stride is the number of pixels that the filter matrix slides across the input pixel array. For example, when the stride is 1, the filter matrix moves one pixel at a time. When the stride is 2, the filter matrix moves two pixels at a time. Having a larger stride can produce smaller feature maps. In some implementations, the input matrix can be padded with zeros around the borders so that the filter matrix can be applied to the border elements of the input pixel array. Zero padding can allow the size of the feature map to be controlled.

[0063] like Figure 2 As shown, an additional nonlinear operation using an activation function (e.g., ReLU) can be used after each convolution operation. ReLU is an element-wise operation that replaces all negative pixel values ​​in the feature map with zero. The purpose of the ReLU operation is to introduce nonlinearity into the CNN. Other nonlinear functions such as the hyperbolic tangent or sigmoid function can also be used, but it has been found that ReLU performs better in many cases.

[0064] Spatial pooling (also known as subsampling or downsampling) reduces the dimensionality of each feature map while retaining the most important information. In particular, pooling makes the feature dimensions smaller and more manageable, and reduces the number of parameters and computations in the network. Pooling also makes the network invariant to small transformations, distortions, and translations in the input image, so that small distortions in the input image may not change the pooled output because the maximum or average value in the local neighborhood is used. Therefore, pooling can help achieve an isovariant representation of the input image, so that objects in the image can be detected regardless of their location. Spatial pooling can be performed in different ways, such as maximum pooling, average pooling, and sum pooling. In maximum pooling, the maximum element in each spatial neighborhood (e.g., a 2×2 window) can be used to represent the spatial neighborhood. Instead of using the maximum element, the average (for average pooling) or sum (for sum pooling) of all elements in each window can be used to represent the spatial neighborhood. In many applications, maximum pooling may be more effective than other pooling techniques.

[0065] exist Figure 2 In the example shown, two sets of convolutional and pooling layers are used. It should be noted that these operations can be repeated any number of times in a single CNN. In addition, a pooling layer may not be used after each convolutional layer. For example, in some implementations, a CNN may perform multiple convolution and ReLU operations before performing a pooling operation.

[0066] The training process of a convolutional neural network such as CNN 200 can be similar to the training process of any feedforward neural network. First, all parameters and weights (including weights in filters and weights in fully connected layers) can be initialized with random values ​​(or known parameters of the neural network). Second, the convolutional neural network can use training samples (e.g., training images) as input, perform forward propagation steps (including convolution, nonlinear activation and pooling operations, as well as forward propagation operations in fully connected layers), and determine the output probability of each possible class. Since the parameters of the convolutional neural network, such as weights, are randomly assigned for the training examples, the output probabilities may also be random.

[0067] At the end of the training process, all weights and parameters of the CNN may have been optimized to correctly classify training samples from the training dataset. When unseen samples (e.g., test samples or new samples) are input into the CNN, the CNN can use the trained weights and parameters to complete the forward propagation step and output the probability of each category, which can be compared with the training process. This can be called an inference (or prediction) process. If the training dataset is sufficient, the trained network can classify unseen samples into the correct category.

[0068] Figure 3A and Figure 3B A convolution operation is shown performed on an input pixel array 320 by a convolutional layer in a convolutional neural network using a filter 310. The input pixel array 320 may include an input image, channels of an input image, or a feature map generated by another convolutional layer or a pooling layer. Figure 3A A convolution operation is shown performed on a first region 322 of the input pixel array 320 at a first step. Figure 3B A convolution operation is shown performed on a second region 324 of the input pixel array 320 at a second step after sliding the filter 310 by a certain step.

[0069] The filter 310 may include a two-dimensional matrix, and each element of the 2-D matrix represents a weight. The weights in the filter 310 may be designed or trained to detect or extract certain features from the spatial distribution of pixel values ​​in the image. The extracted features may or may not be meaningful to the human eye. Different filters can be used to detect or extract different features from the input pixel array. For example, some filters can be used to detect edges in an image, or to sharpen or blur an image. The filter 310 may have R rows (height) and S columns (width), and may typically be smaller than the input pixel array 320, which may have a height of H pixels and a width of W pixels. Each weight in the filter 310 may be mapped to a pixel in an area of ​​the input pixel array 320 having R rows and S columns. For example, Figure 3AAs shown, the convolution layer (e.g., the first convolution layer 215 or the second convolution layer 235) or the processing node of the convolution layer may receive pixel values ​​of a region 322 (including R×S pixels) of the input pixel array 320, perform element-by-element multiplications between the filter 310 and corresponding elements in the region 322, and sum the products of the element-by-element multiplications to generate a convolution output value 332. In other words, the convolution output value 332 may be a value based on The sum of the multiplication results between the weights in the filter 310 and the corresponding pixels in the area 322, that is, the dot product between the matrix W representing the filter 310 and the matrix X representing the pixel values ​​of the area 322.

[0070] Similarly, if Figure 3B As shown, the convolution layer (e.g., another processing node of the convolution layer) may receive pixel values ​​of region 324 (including R×S pixels) of input pixel array 320, perform element-wise multiplications between corresponding elements in filter 310 and region 324, and sum the products of the element-wise multiplications to generate a convolution output value 334. Figure 3B As shown, the convolution operation can be performed in a sliding window manner with a predetermined stride D. The stride is the number of pixels that the filter matrix slides across the input pixel array. For example, Figure 3B In the example shown, region 324 may be a distance D (in pixels) from region 322, and the next region for the next convolution operation may be located at the same distance D from region 324. The stride D may be smaller or larger than the width S of the filter 310.

[0071] The output of the convolution operation may form a convolution output matrix 330 having a height of E rows and a width of F columns. As described above, the matrix 330 may be referred to as a feature map. The dimensions of the matrix 330 may be smaller than the input pixel array 320 and may be determined based on the dimensions of the input pixel array 320, the dimensions of the filter 310, and the stride D. As described above, in some implementations, the input pixel array 320 may be padded with zeros around the boundaries so that the filter 310 may be applied to the boundary elements of the input pixel array 320. Zero padding may allow the size of the feature map (e.g., the matrix 330) to be controlled. When the padding size on each side of the 2-D input pixel array 320 is P, the height E of the matrix 330 is And the width F of the matrix 330 is For example, if the stride D is equal to one pixel in both the horizontal and vertical directions, then E may be equal to H - R + 2P + 1, and F may be equal to W - S + 2P + 1. Having a larger stride D may produce a smaller feature map.

[0072] Figures 4A to 4EExamples of convolution, non-linear activation, and pooling operations performed on examples of input pixel data are shown. The input pixel data may represent, for example, a digital image, a channel of a digital image, or a feature map generated by a previous layer in a convolutional neural network. Figure 4A An example input matrix 410 including example input pixel data is shown. The input matrix 410 may include a 6x6 array of pixels, where each element of the pixel array may include a real number, such as an integer or a floating point number. Figure 4B An example filter 420 is shown. The filter 420 may include a 3×3 matrix, where each element of the matrix represents a weight of the filter. The filter 420 may be used to extract certain features from the input matrix 410. For example, Figure 4B The example filter 420 shown may be a filter used to detect edges in an image.

[0073] The input matrix 410 and the filter 420 may be convolved to generate Figure 4C Output matrix 430 is shown. Each element in output matrix 430 may be the sum of element-by-element multiplications (e.g., dot products) between corresponding elements in overlapping region 412 of filter 420 and input matrix 410, and may be determined in each step (a window having the same dimensions as filter 420) as filter 420 (e.g., 3×3) slides across input matrix 410 with a particular stride (e.g., 1 element in both the horizontal and / or vertical directions). For example, the value of element 432 in row 1 and column 3 of output matrix 430 may be the dot product between the matrix representing filter 420 and the matrix representing region 412 of input matrix 410, where 2×0+1×1+0×0+5×1+3×(−4)+2×1+2×0+1×1+1×0=1+5−12+2+1=−3. Similarly, the value of element 434 in row 4 and column 1 of output matrix 430 may be the dot product between the matrix representing filter 420 and the matrix representing region 414 of input matrix 410, where 0×0+2×1+1×0+0×1+0×(−4)+1×1+5×0+3×1+2×0=2+1+3=6. For input matrix 410 having a 6×6 pixel array and filter 420 represented by a 3×3 matrix, when the stride used is one element or pixel, output matrix 430 may be a 4×4 matrix.

[0074] A non-linear activation function (e.g., ReLU, sigmoid, hyperbolic tangent, etc.) may then be applied to the output matrix 430 to generate Figure 4D The matrix 440 is shown. Figure 4DIn the example shown, the ReLU function is used, and therefore all negative values ​​in the output matrix 430 are replaced by 0 in the matrix 440. A pooling operation (e.g., a maximum, average, or sum pooling operation) may be applied to the matrix 440 to subsample or downsample the data in the matrix 440. Figure 4D and Figure 4E In the example shown, a maximum pooling operation can be applied to matrix 440, where the 4×4 matrix 440 can be divided into four 2×2 regions 442, 444, 446, and 448. The maximum value of each region can be selected to represent a subsample of each region. For example, the maximum value 9 is selected from region 442, the maximum value 2 is selected from region 444, the maximum value 5 is selected from region 446, and the maximum value 6 is selected from region 448. Thus, a feature map 450 having four elements 9, 2, 6, and 5 can be generated from the 6×6 input matrix 410 after convolution, nonlinear activation, and pooling operations.

[0075] Figure 5 An example of a model 500 for a convolutional layer of a convolutional neural network, such as in image processing, is shown. As shown in the example, the convolutional layer may have multiple (e.g., N) 3-D inputs 520-1, ..., and 520-N. Each 3-D input may include C channels of a 2-D input feature map (having dimensions H×W). For the first convolutional layer in a CNN, the 3-D input may include, for example, three channels of a 2-D image, such as red, green, and blue channels. A plurality (e.g., M) of 3-D filters 510-1, ..., and 510-M (each having C 2-D filters of dimension R×S) may be convolved with N 3-D inputs 520-1, ..., and 520-N (e.g., N batches of C input feature maps of dimension H×W) to generate a plurality (e.g., N) of 3-D outputs 530-1, ..., and 530-N, wherein each of the 3-D outputs 530-1, ..., and 530-N may include M output feature maps (also referred to as output channels). Each 3-D filter 510-1, ..., or 510-M (having dimensions C×R×S) may be applied to a 3-D input 520-1, ..., or 520-N (having dimensions C×H×W) to generate an output feature map (having dimensions as described above with respect to Figure 3A and Figure 3B, or 530-N, the 3-D outputs comprising M output feature maps, and thus M 3-D filters may be used to generate the M output feature maps in the 3-D outputs for a 3-D input. For example, 3-D filter 510-1 may be applied to 3-D input 520-1 to generate output feature maps 530-1-1, ..., and 3-D filter 510-M may be applied to 3-D input 520-1 to generate output feature maps 530-1-M. The same M 3-D filters 510-1, ..., and 510-M may be applied to each 3-D input 520-1, ..., or 520-N to generate a respective 3-D output 530-1, ..., or 530-N comprising M output feature maps. For example, 3-D filter 510-1 can be applied to 3-D input 520-N to generate output feature map 530-N-1, and 3-D filter 510-M can be applied to 3-D input 520-N to generate output feature map 530-NM. Thus, there are N 3-D inputs and N 3-D outputs, where each 3-D output includes M output feature maps.

[0076] More specifically, if Figure 5 As shown, for a 3-D input 520-1, ..., or 520-N and a 3-D filter 510-1, ..., or 510-M, C 2-D filters (each having a dimension of R×S) in the 3-D filter 510-m may correspond to C channels of a 2-D input feature map (each having a dimension of H×W) in the 3-D input, and a convolution operation may be performed between each of the C 2-D filters and a corresponding channel of the C channels of the 2-D input feature map. The convolution results of the C pairs of 2-D filters and the corresponding 2-D input feature maps may be summed to generate a convolution output (e.g., a pixel) on an output feature map indexed at m in the M output feature maps in the 3-D output 530-1, ..., or 530-N as follows: e,f m :

[0077]

[0078] Where m corresponds to the index of the output feature map and the index of the 3-D filter in the M 3-D filters. c eD+r,fD+sis the value of the pixel with horizontal pixel coordinate eD+r and vertical pixel coordinate fD+s in the input feature map indexed by C in the C channels of the 2-D input feature map in the 3-D input. D is the sliding window stride distance. e and f are the coordinates of the output pixel in the corresponding output feature map in the M output feature maps and may correspond to a specific sliding window. r and s correspond to a specific position (e.g., pixel or element) within the sliding window or 2-D filter. W c,m r,s is the weight of the pixel at position (r, s) of the 2-D filter indexed by C in the 3-D filter indexed by m. Equation (3) indicates that in order to calculate each convolution output (e.g., pixel) at position (e, f) on the output feature map m, e,f m , each pixel X in the sliding window in the input feature map indexed by C can be c eD+r,fD+s Multiply by the corresponding weight W c,m r,s To generate the product, the partial sum of the product for the pixel within each sliding window in the input feature map indexed by C may be calculated, and then the sum of the partial sums of all C input feature maps may be calculated to determine the value of the pixel at position (e, f) in the corresponding output feature map indexed by m in the M output feature maps. e,f m .

[0079] In one example, for a 3-D filter 510-1 and a 3-D input 520-1, each of the C 2-D filters in the 3-D filter 510-1 may correspond to a corresponding input feature map 522 in the 3-D input 520-1 and may be used to convolve (e.g., filter) with the corresponding input feature map 522, wherein each pixel in a sliding window 524 in the input feature map 522 may be multiplied by the corresponding pixel in the 2-D filter 512 to generate a product, and the products for all pixels in the sliding window 524 may be summed to generate a partial sum. The partial sums of the C 2-D filters 512 (and the corresponding input feature maps 522) may be added together to generate an output pixel 532 at a position (e, f) on the output feature map 530-1-1 in the 3-D output 530-1. The sliding window 524 may shift across all C input feature maps 522 in the 3-D input 520-1 based on a stride D in two dimensions to generate another output pixel 532 at a different position in the output feature map 530-1-1 of the 3-D output 530-1. The sliding window 524 may repeatedly shift across all C input feature maps 522 together until all output pixels 532 on the output feature map 530-1-1 in the 3-D output 530-1 are generated.

[0080] Each 3-D filter 510-2, ..., or 510-M can be used to convolve with the 3-D input 520-1 as described above with respect to the 3-D filter 510-1 to generate a respective output feature map 530-1-2, ..., or 530-1-M in the 3-D output 530-1. Similarly, each 3-D filter 510-1, ..., or 510-M can be used to convolve with the 3-D input 520-N as described above with respect to the 3-D filter 510-1 and the 3-D input 520-1 to generate a respective output feature map 530-N-1, ..., or 530-NM in the 3-D output 530-N.

[0081] Figure 6 An example of a convolution operation involving a batch (N=1) of C channels (C=3) of input data 620 and M groups (M=2) of C filters (C=3) is shown. Figure 6 The example shown may be relative to Figure 5 A specific example of the model 500 is described, where the number of batches N is 1. As shown, the input data 620 includes three input feature maps 622, 624, and 626 (e.g., input channels), each input feature map corresponding to one input channel. The filter includes a first set of filters 610-1 and a second set of filters 610-2, where the first set of filters 610-1 may include three 2-D filters 612-1, 614-1, and 616-1, and the second set of filters 610-2 may include three 2-D filters 612-2, 614-2, and 616-2.

[0082] Each 2-D filter 612-1, 614-1, or 616-1 in the first set of filters 610-1 may be convolved with the corresponding input feature map 622, 624, or 626, and the results of the convolution of the three input feature maps may be added together to generate an output feature map 630-1 in the output feature map 630. For example, a pixel in filter 612-1 may be multiplied by a corresponding pixel in a window 622-1 on the input feature map 622, and the products may be added together to generate a first partial sum. A pixel in filter 614-1 may be multiplied by a corresponding pixel in a window 624-1 on the input feature map 624, and the products may be added together to generate a second partial sum. A pixel in filter 616-1 may be multiplied by a corresponding pixel in a window 626-1 on the input feature map 626, and the products may be added together to generate a third partial sum. The first, second, and third partial sums may be added together to generate an output pixel 632-1 on the output feature map 630-1. Other output pixels on the output feature map 630-1 may be generated in the same manner by shifting windows or filters together on the input feature map.

[0083] Similarly, each 2-D filter 612-2, 614-2, or 616-2 in the second set of filters 610-2 may be convolved with the corresponding input feature map 622, 624, or 626, and the results of the convolution of the three input feature maps may be summed to generate an output feature map 630-2 in the output feature map 630. For example, a pixel in filter 612-2 may be multiplied by a corresponding pixel in window 622-1 on the input feature map 622, and the products may be added together to generate a first partial sum. A pixel in filter 614-2 may be multiplied by a corresponding pixel in window 624-1 on the input feature map 624, and the products may be added together to generate a second partial sum. A pixel in filter 616-2 may be multiplied by a corresponding pixel in window 626-1 on the input feature map 626, and the products may be added together to generate a third partial sum. The first, second, and third partial sums may be added together to generate an output pixel 632-2 on the output feature map 630-2. Other output pixels on the output feature map 630-2 may be generated in the same manner by shifting windows or filters together on the input feature map.

[0084] As shown in the model discussed above, the operation of a neural network (e.g., performing inference) typically involves obtaining input data or input activations, performing multiplication and accumulation operations in parallel for each node in the layer, and providing output activations. Optimal performance of neural networks, as measured by response time, is achieved when the hardware architecture is capable of highly parallelized computation. Central processing units (CPUs) (also known as general-purpose processing units) can have multiple cores (e.g., 2 to 64 or more cores) and can increase parallelism by using multiple execution threads. However, CPU cores are often optimized for sequential processing. For this and other reasons, CPUs often have slow response times when performing inference for neural networks. Graphics processing units (GPUs) achieve parallelism by having thousands of small, efficient cores specifically configured for parallel computation. Consequently, GPUs can achieve significantly better performance than CPUs when executing neural networks. However, individual GPU compute engines can still be primarily sequential in nature, requiring memory operations to provide the output of one compute engine to the input of another. The speed of neural networks can also be limited by memory latency or data transfer bandwidth.

[0085] Dedicated or domain-specific neural network processors can achieve better performance than both CPUs and GPUs when executing neural networks. A neural network processor can employ a spatial architecture that includes an array of processing elements (PEs), where the processing elements can form a processing chain and data can be passed directly from one processing element to another. This can significantly reduce the number of memory transactions. In some examples, weights or inputs can be preloaded into the array of processing elements. In some examples, the neural network processor can also include an on-chip buffer that can store values ​​read from the processor memory and can distribute the values ​​to multiple computing engines in the processor. The computing engine can also include a small local register file (e.g., a small memory) for storing intermediate results. Having an on-chip memory hierarchy can improve the operating efficiency of the neural network by reducing memory latency.

[0086] Figure 7 is a block diagram illustrating an example of an integrated circuit device for performing neural network operations such as tensor operations, according to certain embodiments. Figure 7 The example shown includes an accelerator 702. In various examples, the accelerator 702 can use a processing element array 710, an activation engine 716, and / or a pooling engine 718 to perform computations on a set of input data (e.g., input data 750). In some examples, the accelerator 702 can be an integrated circuit component of a processor, such as a neural network processor. The processor can have other integrated circuit components, including additional accelerator engines.

[0087] In some embodiments, the accelerator 702 may include a memory subsystem 704 (e.g., a status buffer) comprising multiple memory banks 714. Each memory bank 714 may be independently accessible, meaning that a read from one memory bank is independent of a read from another. Similarly, a write to one memory bank does not affect or restrict a write to a different memory bank. In some cases, each memory bank may be read from and written to simultaneously. Various techniques may be used to provide independently accessible memory banks 714. For example, each memory bank may be a physically separate memory component having an address space separate and independent from the address space of each other memory bank. In this example, each memory bank may have at least one read channel and at least one separate write channel that can be used simultaneously. In these examples, the memory subsystem 704 may grant simultaneous access to the read or write channels of multiple memory banks. As another example, the memory subsystem 704 may include arbitration logic such that, for example, arbitration between the outputs of multiple memory banks 714 may result in the use of the outputs of more than one memory bank. In these and other examples, while globally managed by memory subsystem 704 , each memory bank can operate independently of any other memory bank.

[0088] Making memory banks 714 independently accessible can improve the efficiency of accelerator 702. For example, values ​​can be read and provided to each row of processing element array 710 simultaneously, allowing the entire processing element array 710 to be used within a single clock cycle. As another example, memory banks 714 can be read while results computed by processing element array 710 are being written to memory subsystem 704. In contrast, a single memory may only be able to service one read or write at a time. For example, with a single memory, multiple clock cycles may be required to read input data for each row of processing element array 710 before processing element array 710 can be enabled.

[0089] In various implementations, memory subsystem 704 can be configured to simultaneously service multiple clients, including processing element array 710, activation engine 716, pooling engine 718, and any external clients accessing memory subsystem 704 through communication fabric 720. In some implementations, being able to service multiple clients may mean that memory subsystem 704 has at least as many memory banks as clients. In some cases, each row of processing element array 710 can be considered a separate client. In some cases, each column of processing element array 710 can output a result, such that each column can be considered a separate write client. In some cases, output from processing element array 710 can be written to memory bank 714, which can then provide input data for processing element array 710. As another example, activation engine 716 and pooling engine 718 can include multiple execution channels, each of which can be a separate memory client. For example, memory bank 714 can be implemented using static random access memory (SRAM).

[0090] In various implementations, memory subsystem 704 may include control logic. The control logic may, for example, track the address space of each of memory banks 714, identify the memory banks 714 to be read from or written to, and / or move data between memory banks 714. In some implementations, memory banks 714 may be hardwired to specific clients. For example, a group of memory banks 714 may be hardwired to provide values ​​to each row of processing element array 710, with one memory bank servicing one row. As another example, a group of memory banks may be hardwired to receive values ​​from each column of processing element array 710, with one memory bank receiving data for one column.

[0091] Processing element array 710 is the computational matrix of accelerator 702. Processing element array 710 can, for example, perform parallel integration, convolution, correlation, and / or matrix multiplication. Processing element array 710 includes a plurality of processing elements 711 arranged in rows and columns so that a result output by one processing element 711 can be directly input to another processing element 711. Processing elements 711 not on the outer edges of processing element array 710 can therefore receive data from other processing elements 711 rather than from memory subsystem 704 to operate on the data.

[0092] In various examples, processing element array 710 uses systolic execution, where data arrives at each processing element 711 at regular intervals from different directions. In some examples, input data can flow into processing element array 710 from the left and weight values ​​can be loaded at the top. In some examples, weights and input data can flow from the left, and partial sums can flow from top to bottom. In these and other examples, multiplication and accumulation operations move through processing element array 710 as diagonal wavefronts, with data moving right and down through the array. Control signals can be input from the left at the same time as weights and can flow horizontally and vertically along with the calculations.

[0093] In various implementations, the number of columns in processing element array 710 determines the computational power of processing element array 710, and the number of rows determines the memory bandwidth required to achieve maximum utilization of processing element array 710. Processing element array 710 may have, for example, 64 columns and 128 rows, or some other number of columns and rows.

[0094] exist Figure 7 An example of a processing element 711 is illustrated in FIG. As shown in this example, the processing element 711 may include a multiplier-accumulator circuit. The input from the left side may include, for example, input data i and weight values ​​w, where the input data is a value obtained from a set of input data or a set of intermediate results, and the weight values ​​are a set of weight values ​​from connecting one layer of the neural network to the next layer. A set of input data may be, for example, an image submitted for recognition or object identification, an audio clip provided for speech identification, a text string for natural language processing or machine translation, or the current state of a game that needs to be analyzed to determine the next move. In some examples, the input data and weight values ​​are output to the right side to be input to the next processing element 711.

[0095] In the example shown, the input from above may include a partial sum p_in, which is provided from another processing element 711 or from a previous round of calculations in processing element array 710. When starting calculations on a new set of input data, the top row of processing element array 710 may receive a fixed value for p_in, such as zero. As shown in this example, i and w are multiplied together and the result is added to p_in to produce a new partial sum p_out, which may be input to another processing element 711. Various other implementations of processing element 711 are possible.

[0096] The output from the last row of processing element array 710 may be temporarily stored in a result buffer 712 (e.g., a partial sum (PSUM) buffer). The result may be an intermediate result that can be written to memory bank 714 to be provided to processing element array 710 for additional calculations. Alternatively, the result may be a final result that, once written to memory bank 714, can be read from memory subsystem 704 via communication fabric 720 for output by the system.

[0097] In some implementations, accelerator 702 includes an activation engine 716. In these implementations, activation engine 716 can combine results from processing element array 710 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other examples, it may be necessary to accumulate results from one or more columns in processing element array 710 to produce an output activation for a single node in the neural network. In some examples, activation engine 716 can be bypassed.

[0098] In various examples, activation engine 716 may include multiple separate execution channels. In these examples, an execution channel may correspond to a column of processing element array 710 and may perform operations on the output of the column, the results of which may be stored in memory subsystem 704. In these examples, activation engine 716 may be capable of performing 1 to N parallel calculations, where N is equal to the number of columns in processing element array 710. In some cases, one or more of the calculations may be performed simultaneously. Examples of calculations that each execution channel may perform include exponential, square, square root, identity, binary step, bipolar step, sigmoidal, and ramp, among other examples.

[0099] In some implementations, accelerator 702 may include a pooling engine 718. Pooling is the combination of the outputs of the columns of processing element array 710. Combining may include, for example, calculating a maximum, minimum, average, median, summation, multiplication, or another logical or mathematical combination. In various examples, pooling engine 718 may include multiple execution paths that can operate on values ​​from corresponding columns of processing element array 710. In these examples, pooling engine 718 may be capable of performing 1 to N parallel calculations, where N is equal to the number of columns in processing element array 710. In various examples, the execution paths of pooling engine 718 may operate in parallel and / or simultaneously. In some examples, pooling engine 718 may be bypassed.

[0100] Activation engine 716 and pooling engine 718 may be collectively referred to herein as an execution engine. Processing element array 710 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine, which may be located outside accelerator 702.

[0101] Input data 750 may arrive via the communication fabric 720. The communication fabric 720 may connect the accelerator 702 to other components of the processor, such as a DMA engine that may obtain the input data 750 from an input / output (I / O) device, a storage drive, or a network interface. The input data 750 may be, for example, one-dimensional data, such as a string or sequence of numbers, or two-dimensional data, such as an array of pixel values ​​for an image or an array of frequency and amplitude values ​​of an audio signal varying over time. In some examples, the input data 750 may be three-dimensional, as may be the case for contextual information or virtual reality data used by self-driving cars. In some implementations, the memory subsystem 704 may include a separate buffer for the input data 750. In some implementations, when the accelerator 702 receives the input data 750, the input data 750 may be stored in the memory bank 714.

[0102] In some examples, accelerator 702 can implement a neural network processing engine. In these examples, accelerator 702 can execute a neural network to perform a task for which the neural network was trained, given a set of input data 750. Executing a neural network on a set of input data can be referred to as inference or performing inference.

[0103] The weights of the neural network can be stored in the memory subsystem 704 along with the input data 750 on which the neural network will operate. The neural network can also include instructions that program the processing element array 710 to perform various calculations on the weights and input data. Instructions can also be stored in the memory subsystem 704, in the memory bank 714, or in a separate instruction buffer. The processing element array 710 can output intermediate results representing the outputs of individual layers of the neural network. In some cases, the activation engine 716 and / or the pooling engine 718 can be enabled to perform the calculations required by certain layers of the neural network. The accelerator 702 can store the intermediate results in the memory subsystem 704 for input to the processing element array 710 to calculate the results of the next layer of the neural network. The processing element array 710 can also output the final results from the last layer of the neural network. The final results can be stored in the memory subsystem 704 and then copied from there to the host processor memory or another location.

[0104] Figure 8 A simplified example of a compute engine 800 including a processing element (PE) array 820 is shown according to certain embodiments. The compute engine 800 may be a simplified example of an accelerator 702 or a PE array 710. Figure 8 In the example shown, processing element array 820 may include an N×M array of PEs, with N PEs in each of the M columns and M PEs in each of the N rows. In one example, N may be 128 and M may be 64, such that PE array 820 may include 8,192 PEs.

[0105] Each PE may include a row input bus 822, a column input bus 824, a column output bus 826, and a row output bus 828. A PE may receive input from the previous PE in the same row (on the left) (or from an external circuit) via the row input bus 822. A PE may also receive input from the previous PE in the same column (on the top) (or from an external circuit) via the column input bus 824. A PE may perform floating-point or integer arithmetic operations (e.g., multiplication and accumulation (MAC)) based on the inputs and transmit the results of the arithmetic operations to the next (lower) PE in the same column (or an external circuit) via the column output bus 826. A PE may also forward inputs to the next PE in the same row (forwarding to the right) via the row output bus 828.

[0106] Each row of PE array 820 can process an input data set comprising multiple input data elements, such as a one-dimensional vector representing a flattened multidimensional matrix. Each column of PE array 820 can generate a weighted sum of input data elements from a different input data set. For example, when compute engine 800 is about to process N input pixel groups 810, 812, 814, ..., and 816, the first row can receive input data elements from pixel group 810, the second row can receive input data elements from input pixel group 812, the third row can receive input data elements from input pixel group 814, ..., and the Mth row can receive input data elements from input group 816. Each PE can include a multiplier 823 that processes one input data element at a time. For example, a PE can receive an input data element and a weight (e.g., from row input bus 822) and use multiplier 823 to generate a multiplication product representing the weighted input data element. The weight can be an element of a convolution kernel (e.g., a filter) or a connection weight between nodes on two layers of a fully connected layer. In addition, the PE may also receive a partial weighted sum from the previous PE in the same column (e.g., from the column input bus 824). The partial weighted sum represents the weighted sum of the input data elements of the input data set received by the PEs in the rows above the PE. The PE may include an adder 825 that can add the weighted input data elements to the partial weighted sum and pass the updated partial weighted sum to the following PE in the same column (e.g., via the column output bus 826). The PE at the bottom row of each column can generate a weighted sum of the input data elements received by all PEs in the column.

[0107] In some embodiments, the operation of each PE of PE array 820 can be synchronized with a clock signal to improve interoperability between PE array 820 and other components of the neural network processor (e.g., accelerator 702). Each PE can also include sequential logic circuitry (e.g., registers, latches, flip-flops, state machines, etc.) to store input data, weights, and output data for adder and multiplier circuits and synchronize the flow of data into and out of the circuits. The sequential logic circuitry of each PE can be clocked by the same clock signal or a copy of the clock signal so that data can be sequentially shifted into and / or out of the PE synchronously during clock cycles. For example, in some embodiments, during a first clock cycle, PE 820b in the second row can receive a first input data element of pixel group 812 and a partial sum of the first input data element including the weights of pixel group 810 from PE 820a in the first row. During the first clock cycle, PE 820b can multiply the input data element by the weight, add the product of the multiplication to the partial sum to generate an updated partial sum, and store the updated partial sum in an internal register. In the second clock cycle, PE 820b may forward the updated partial sum to PE 820c in the third row below, which may perform multiplication and accumulation to generate the updated partial sum. In the third clock cycle, PE 820c may forward the updated partial sum to the next PE in the fourth row below, which may perform multiplication and accumulation to generate the updated partial sum. The updated partial sum may propagate down each column until the updated partial sum is output to output buffer 830 (also referred to as a PSUM buffer) by PE 820d in the Mth row in the Mth clock cycle.

[0108] In some implementations, each column of the computation engine 800 may correspond to a processing node of a neural network layer and may apply a different set of weights {w i} for each input data set {x i Generate different weighted sums

[0109] In some embodiments, the above with respect to Figure 5 and Figure 6And mapping the tensor operation described by equation (3) to a PE array (e.g., PE array 820) for execution may include mapping each of the M 3-D filters to a corresponding column of the PE array, and mapping each of the C input feature maps (e.g., C channels) in the 3-D input to a corresponding row of the PE array. For example, the H×W pixels in each 2-D input feature map may be flattened to form a one-dimensional vector and mapped to a row of the PE array. The C×R×S weights or pixels in each 3-D filter may be flattened to form a one-dimensional vector and mapped to a column of the PE array. Partial sums may be accumulated vertically in each column. In the case of processing a batch of N 3-D inputs (each including C channels), each row of the PE array may be mapped to N 2-D input feature maps.

[0110] As described above, the movement of data such as input pixels, filter weights, and partial sums to be accumulated between PEs can reduce accesses to the state buffer or off-chip memory. In some embodiments, the input feature map can be fixed and the weights of the filter can be offset, which can be referred to as a "image-fixed" model. In some embodiments, a "weight-fixed" model can be used, where the weights of the filter are fixed (preloaded from the state buffer into registers in the PE array) and the image is moved (loaded from the state buffer during calculation) to minimize the cost of moving the weights. In some embodiments, the output of the PE can be stored in a register at the PE and kept fixed to minimize the cost of moving the partial sums, where the input feature map and weights can be moved through the PE array and state buffer.

[0111] Figure 9 A simplified example of a weight-fixed convolution operation using an example of a computation engine including processing element array 910 is shown in accordance with certain embodiments. Figure 9 In the example shown, the processing element array 910 includes four rows and four columns of processing elements 912. The input 920 of the processing element array 910 may include four (corresponding to C) input channels 922, 924, 926, and 928. Each input channel may correspond to an input feature map or one input feature map for each of the N (in the example N=1) inputs as described above. Each input feature map may include an 8×8 matrix in this example and may be flattened into a one-dimensional vector with 64 elements. The PE array 910 may generate four (corresponding to M) output feature maps, one output feature map from each column of the PE array 910.

[0112] During a convolution operation, the weights of each of the four 2-D filters (having dimensions R×S) in each of the four 3-D filters (having dimensions C×R×S) may be preloaded into the PE array 910. For example, Figure 9 As shown, the first element (r=0, s=0) in each of the four 2-D filters (corresponding to the first 3-D filter or m=0) for the first output feature map can be loaded into the corresponding PE 912 of the four PEs in the first column of the PE array 910, the first element (e.g., r=0, s=0) in each of the four 2-D filters (corresponding to the second 3-D filter or m=1) for the second output feature map can be loaded into the corresponding PE 912 of the four PEs in the second column of the PE array 910, the first element (r=0, s=0) in each of the four 2-D filters (corresponding to the third 3-D filter or m=2) for the third output feature map can be loaded into the corresponding PE 912 of the four PEs in the third column of the PE array 910, and the first element (r=0, s=0) in each of the four 2-D filters (corresponding to the fourth 3-D filter or m=3) for the fourth output feature map can be loaded into the corresponding PE 912 of the four PEs in the fourth column of the PE array 910. 912. Thus, 16 values ​​representing the first elements of the 16 2-D filters in the four 3-D filters are loaded into the PE array 910. The elements in the one-dimensional vector of each input feature map can then be shifted from, for example, the state buffer into the PE array 910 and can be multiplied by the preloaded weights in the PE array 910. The products in each column for the four channels 922, 924, 926, and 928 can be accumulated to generate four partial sum values. When the elements in the one-dimensional vector of each input feature map are shifted into the PE array 910, a first partial sum vector PSUM can be generated. 0,0 At 930, the first portion and vector may include four portions and sub-vectors of the four output feature maps. In some embodiments, the shifting of elements in the input feature maps may be based on a desired stride of the convolution, such that each portion and sub-vector of the output feature maps may include a desired number of elements (e.g., E×F as described above).

[0113] Sixteen (16) values ​​representing the second elements (e.g., r=0, s=1) of the 16 2-D filters in the four 3-D filters may then be loaded into the PE array 910. The elements in the one-dimensional vector of each input feature map may be shifted into the PE array 910 and may be multiplied by the pre-loaded weights in the PE array 910. The products in each column may be accumulated to generate the second partial sum vector PSUM 0,1 (932), the second partial sum vector includes four partial sum sub-vectors of the four output feature maps. Each element of the 16 2-D filters can be loaded into the PE array 910 and multiplied by the element in the one-dimensional vector to generate a partial sum vector including four partial sum sub-values ​​of the four output feature maps until the partial sum vector PSUM is generated. R-1,S-1(934), the partial sum vector corresponds to the element (R-1, S-1) in each 2-D filter and includes four partial sum subvectors of the four output feature maps. The partial sum vector PSUM 0,0 (930), PSUM 0,1 (932), ..., and PSUM R-1,S-1 The portions and sub-vectors in (934) corresponding to each respective output feature map are accumulated to generate a respective vector 940, 942, 944, or 946 that may correspond to the flattened output feature map.

[0114] As described above, the neural network model may include multiple convolutional layers, and thus the neural network may perform multiple convolution operations (e.g., tensor operations) to extract features and perform inference. In some embodiments, due to the limited size of the PE array in the accelerator (e.g., 128×64), the convolution operation may need to be split into multiple convolution operations to be performed serially.

[0115] Figure 10A An example of a series of operations for performing inference using a neural network model is shown. In the example shown in FIG10 , it may be necessary to perform two operations including operation A (1010) and operation B (1020) to perform inference. Operation A (1010) may, for example, generate 512 output feature maps from 512 input feature maps. Operation B (1020) may, for example, generate 512 output feature maps from 512 input feature maps. Operation A (1010) and operation B (1020) may be executed sequentially on a computing engine (e.g., an accelerator) within a time period T. In some embodiments, in order to increase the throughput of inference, multiple computing engines may be used to perform inference in parallel.

[0116] Figure 10B An example of using multiple computing engines for parallel inference is shown. For example, accelerator 1 can sequentially perform operation A (1010-1) and operation B (1020-1) during time period T. Accelerator 2 can sequentially perform operation A (1010-2) and operation B (1020-2) during the same time period T. Thus, two inferences can be performed during time period T. Therefore, by using multiple processing engines in parallel, the throughput of the system and the average time it takes to perform inference can be improved. However, the latency for performing inference remains unchanged (e.g., T).

[0117] Figure 11 It is shown that multiple computing engines are executed to perform Figure 10B11. An example of operations of parallel inference shown. The operations performed by the first computing engine can be shown by data flow graph 1100, and the operations performed by the second computing engine can be shown by data flow graph 1105. Data flow graph 1100 and data flow graph 1105 can be generated, for example, by a compiler. Multiple computing engines can be on the same integrated circuit device or on multiple integrated circuit devices. Examples of types of computing engines can include processing element arrays, activation engines, and pooling engines, among other examples. In various examples, each node in the data flow graph can represent an operation to be performed by a particular computing engine of the integrated circuit device. The operations can include, for example, memory access (e.g., read / write), computation, and decision making.

[0118] exist Figure 11 In the example shown, the dataflow graph 1100 may include a node 1110, which may include a memory access operation (e.g., a memory read operation); a node 1120, which may include a convolution operation A for a first inference (e.g., in a first convolutional layer); a node 1130, which may include a memory access operation (e.g., including memory read and write operations); a node 1140, which may include a convolution operation B for the first inference (e.g., in a second convolutional layer); and a node 1150, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation. Similarly, the data flow graph 1105 may include a node 1115, which may include a memory access operation (e.g., a memory read operation); a node 1125, which may include a convolution operation A for a second inference (e.g., in a first convolutional layer); a node 1135, which may include a memory access operation (e.g., including memory read and write operations); a node 1145, which may include a convolution operation B for a second inference (e.g., in a second convolutional layer); and a node 1155, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation.

[0119] The connections between the nodes in data flow graph 1100 and data flow graph 1105 may be referred to as edges and, in some cases, may represent data and / or resource dependencies between nodes. For example, when a computation engine uses the output of another computation engine as input for a computation, data dependency may exist. When neither operation requires the output data generated by the other operation as input data, the two operations do not have data dependency. For example, when a first computation engine requires the use of a memory location that a second computation engine also requires, resource dependency may exist. For example, the second computation engine may need to write a value to the memory location, while the first computation engine may need to read the value and use it in a computation at the same time. As another example, the first execution engine may need to write a value to a memory location (e.g., due to limited available memory) while the second computation engine is reading or writing a value to the same memory location. Due to data and / or resource dependencies between nodes, some operations may need to be executed sequentially, where one operation may only begin after an event occurs, and the event may occur when another operation has completed.

[0120] In some embodiments, there may not be data and / or resource dependencies between some operations, or the operations may be arranged so that there may not be data and / or resource dependencies between some operations. Therefore, these operations may not need to be performed sequentially. Similarly, these operations may be performed in parallel by multiple computing engines to reduce the waiting time for performing these operations.

[0121] Figure 12 An example of using multiple computation engines for inference according to certain embodiments is shown. Figure 12 In the example shown, it may be necessary to perform two operations including operation A and operation B to perform inference. Operation A may, for example, generate 512 output feature maps from 512 input feature maps. Operation B may, for example, generate 512 output feature maps from 512 input feature maps. Operation A and operation B may be designed so that there is no data and / or resource dependency between them. In order to perform the first inference, accelerator 1 may perform operation A (1210-1) and accelerator 2 may simultaneously perform operation B (1220-1). Therefore, the first inference may be performed within a time period T / 2. After the first inference, accelerator 1 may perform operation A (1210-2) within a time period T / 2 and accelerator 2 may simultaneously perform operation B (1220-2) to perform the second inference. Therefore, the throughput of the system and the average time to perform inference may be improved by using multiple processing engines in parallel, and the waiting time for performing inference may also be reduced (for example, to approximately T / 2). It should be noted that, without including other operations such as memory access operations, when only the computation time using the PE array is considered, the inference period or latency is T / 2.

[0122] Figure 13 According to certain embodiments, for example Figure 12 As shown above, the example of the operation performed by multiple computing engines to perform inference. Figure 12 As described, each inference may include operation A (e.g., a tensor operation, such as a convolution operation) and operation B (e.g., a tensor operation, such as a convolution operation). The operations performed by the first computing engine may be illustrated by dataflow graph 1300, while the operations performed by the second computing engine may be illustrated by dataflow graph 1305. Dataflow graphs 1300 and 1305 may be generated, for example, by a compiler. Multiple computing engines may be on the same integrated circuit device or on multiple integrated circuit devices.

[0123] Dataflow graph 1300 may include node 1310, which may include a memory access operation (e.g., a memory read operation); node 1320, which may include a convolution operation A for a first inference; node 1330, which may include a memory access operation (e.g., including a memory write and / or read operation) and / or a decision-making operation; node 1340, which may include a convolution operation A for a second inference; and node 1350, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation. Similarly, dataflow graph 1305 may include node 1315, which may include a memory access operation (e.g., a memory read operation); node 1325, which may include a convolution operation B for a first inference; node 1335, which may include a memory access operation (e.g., including a memory write and / or read operation) and / or a decision-making operation; node 1345, which may include a convolution operation B for a second inference; and node 1355, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation. In some embodiments, a prediction or decision may be made based on the combined operations at node 1330 and node 1335. In some embodiments, a prediction or decision may be made based on the combined operations at node 1350 and node 1355.

[0124] In some embodiments, in some applications (such as some applications where predictions or decisions can be used for real-time control or other real-time operations), it may be desirable to make predictions or decisions as quickly as possible. In some cases, a portion of the output feature maps, such as some but not all output feature maps, or a portion (e.g., a few rows) of each output feature map can be used to make early predictions or decisions. Therefore, a tensor operation such as a convolution operation can be divided into multiple sub-operations, each of which can be performed by a computing engine to generate a portion of an output feature map, and the results of the sub-operations can be used individually or in combination to make early predictions or decisions.

[0125] Figure 14 An example of using multiple computation engines to perform inference by splitting the operation of a neural network into sub-operations that can be performed in parallel by multiple computation engines is shown in accordance with certain embodiments. Figure 14 In the example shown, two operations, including Operation A and Operation B, may need to be performed to perform inference. Operation A may, for example, generate 512 output feature maps from 512 input feature maps. Operation B may, for example, generate 512 output feature maps from 512 input feature maps. There may or may not be data and / or resource dependencies between Operation A and Operation B.

[0126] To reduce inference latency, operation A can be divided into sub-operations with no data and / or resource dependencies between them, so that the sub-operations can be executed in parallel by different computing engines, where each sub-operation can be executed by a corresponding computing engine and can generate a portion of the output feature map. For example, operation A can be divided into two sub-operations with no data and / or resource dependencies between them, where neither sub-operation requires the use of output data generated by the other sub-operation as input data. Each sub-operation can be used to generate a portion of the result of operation A, such as generating 256 output feature maps from 512 input feature maps, or generating half of a row in each of 512 output feature maps from 512 input feature maps. Similarly, operation B can be divided into sub-operations with no data and / or resource dependencies between them, so that the sub-operations can be executed in parallel on different computing engines. Since the sub-operations of operation B may not be executed when the sub-operations of operation A are executed, operation A and operation B may have data and / or resource dependencies.

[0127] To perform the first inference, accelerator 1 may execute sub-operation OPA-1 (1410-1) of operation A, and accelerator 2 may simultaneously execute sub-operation OPA-2 (1410-2) of operation A. Therefore, excluding other operations such as memory access operations, operation A can be completed within a time period T / 4 when only the computation time using the PE array is considered. In some embodiments, the result of sub-operation OPA-1 (1410-1) and the result of sub-operation OPA-2 (1410-2) may be used separately to make predictions or decisions. For example, predictions or decisions may be made based on half the number of output feature maps or based on half of each output feature map. In some embodiments, the result of sub-operation OPA-1 (1410-1) and the result of sub-operation OPA-2 (1410-2) may be cascaded to form a complete result of operation A.

[0128] Similarly, accelerator 1 may execute sub-operation OPB-1 (1420-1) of operation B, and accelerator 2 may simultaneously execute sub-operation OPB-2 (1420-2) of operation B. Therefore, excluding other operations such as memory access operations, operation B can be completed within time period T / 4 when only the computation time using the PE array is considered. In some embodiments, the result of sub-operation OPB-1 (1420-1) and the result of sub-operation OPB-2 (1420-2) can be used separately to make predictions or decisions. For example, predictions or decisions can be made based on half the number of output feature maps or based on half of each output feature map of operation B. In some embodiments, the result of sub-operation OPB-1 (1420-1) and the result of sub-operation OPB-2 (1420-2) can be cascaded to form the complete result of operation B. Therefore, when other operations such as memory access operations are not included, and only the computation time using the PE array is considered, operations A and B for the first inference can be completed within a time period of approximately T / 2, or within a shorter time period when more than two computation engines are used to perform operations A and B.

[0129] After the first inference, a second inference may be performed similarly. For example, accelerator 1 may execute sub-operation OPA-1 (1430-1) of operation A for the second inference, and accelerator 2 may simultaneously execute sub-operation OPA-2 (1430-2) of operation A for the second inference. Accelerator 1 may then execute sub-operation OPB-1 (1440-1) of operation B for the second inference, and accelerator 2 may simultaneously execute sub-operation OPB-2 (1440-2) of operation B for the second inference. When other operations such as memory access operations are not included, and only the computation time using the PE array is considered, the total time to execute operations A and B for the second inference may be T / 2, or shorter when more than two computing engines are used to execute operations A and B. Therefore, the throughput of the system and the average time to perform inference may be improved by using multiple processing engines in parallel, and the waiting time for performing inference may also be reduced (e.g., T / 2, T / 4, or shorter).

[0130] Figure 15 According to certain embodiments, for example Figure 14 As shown above, the example of the operation performed by multiple computing engines to perform inference. Figure 14As described, each inference may include operation A (e.g., a tensor operation, such as a convolution operation) and operation B (e.g., a tensor operation, such as a convolution operation). Operation A and operation B may each be divided into K sub-operations, which may be performed by multiple (e.g., K) compute engines, such as the accelerators disclosed herein. The operations performed by the first compute engine may be illustrated by data flow graph 1500, while the operations performed by the Kth compute engine may be illustrated by data flow graph 1505. Data flow graphs 1500, ..., and 1505 may be generated, for example, by a compiler. Multiple compute engines may be on the same integrated circuit device or on multiple integrated circuit devices.

[0131] The data flow graph 1500 may include a node 1510, which may include a memory access operation (e.g., a memory read operation); a node 1520, which may include a sub-operation A-1 for an inferred operation A; a node 1530, which may include a memory access operation (e.g., including a memory write and / or read operation) and / or (part of) a decision-making operation; a node 1540, which may include a sub-operation B-1 for an inferred operation B; and a node 1550, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation. Similarly, data flow graph 1505 may include node 1515, which may include a memory access operation (e.g., a memory read operation); node 1525, which may include a sub-operation AK for an inferred operation A; node 1535, which may include a memory access operation (e.g., including a memory write and / or read operation) and / or a (partial) decision-making operation; node 1545, which may include a sub-operation BK for an inferred operation B; and node 1555, which may include a memory access operation (e.g., a memory write operation) and / or a decision-making operation. As described above, in some embodiments, a partial prediction or decision may be made independently at node 1530, node 1535, node 1550, or node 1555. In some embodiments, a partial prediction or decision may be made based on a combined operation at node 1530 and node 1535. In some embodiments, a prediction or decision may be made based on a combined operation at node 1550 and node 1555.

[0132] exist Figure 14 and Figure 15 In the example shown, when operation A and operation B do not have data and / or resource dependencies between them, two or more sub-operations of operation A and two or more sub-operations of operation B can be executed in parallel on four or more computing engines to further reduce the latency for performing inference, for example, to approximately T / 4 or less.

[0133] Figure 16An example of using multiple computing engines (e.g., accelerators) to perform inference by splitting the operation of a neural network into sub-operations that can be performed in parallel by multiple computing engines is shown in accordance with certain embodiments. Figure 16 In the example shown, two operations, including Operation A and Operation B, may need to be performed to perform inference. Operation A may, for example, generate 512 output feature maps from 512 input feature maps. Operation B may, for example, generate 512 output feature maps from 512 input feature maps. There may be no data and / or resource dependencies between Operation A and Operation B.

[0134] In order to reduce the waiting time for inference, operation A can be divided into two or more sub-operations, such as sub-operations OPA-1 and OPA-2. Operation B can be divided into two or more sub-operations, such as sub-operations OPB-1 and OPB-2. To perform the first inference, accelerator 1 can execute sub-operation OPA-1 (1610-1), accelerator 2 can execute sub-operation OPA-2 (1610-2), accelerator 3 can execute sub-operation OPB-1 (1620-1), and accelerator 4 can execute sub-operation OPB-2 (1620-2). Therefore, without excluding other operations such as memory access operations, when only the computing time using the PE array is considered, the first inference can be performed in parallel by four accelerators during the time period T / 4. Similarly, to perform the second inference, accelerator 1 may execute sub-operation OPA-1 (1630-1), accelerator 2 may execute sub-operation OPA-2 (1630-2), accelerator 3 may execute sub-operation OPB-1 (1640-1), and accelerator 4 may execute sub-operation OPB-2 (1640-2). Therefore, excluding other operations such as memory access operations, when only the computation time using the PE array is considered, the second inference can be performed in parallel by four accelerators during the time period T / 4.

[0135] As above relative to Figure 5 、 Figure 6 and Figure 9As described, a tensor operation such as a convolution operation may use an input tensor comprising N (e.g., one or more) 3-D inputs, each comprising C channels of an input feature map (each having dimensions H×W); and a filter comprising M 3-D filters, each comprising C channels of a 2-D filter (each having dimensions R×S). Thus, the input tensor may comprise N×C×H×W pixel values, and the filter may comprise a total of M×C×R×S weight values. Also as described above, the C input channels (each comprising N×H×W pixel values) may be mapped to rows of a PE array, and the M output channels or 3-D filters (each comprising C×R×S weight values) may be mapped to columns of the PE array. Thus, there may be many different ways of partitioning a tensor operation such as a convolution operation. In some embodiments, the tensor operation may be partitioned such that each sub-operation may generate a portion of an output tensor (e.g., an output feature map) that may be used to make a prediction or decision and / or may be saved to a memory space for output tensors (e.g., concatenated in the memory space), where additional arithmetic calculations (e.g., summation or accumulation) may not be required to generate the output feature map.

[0136] Figure 17 An example of splitting the operations of a neural network into sub-operations to be performed in parallel by multiple compute engines according to certain embodiments is shown. As shown, the convolution operation to be performed by the PE array 1720 may use N 3-D inputs, each 3-D input comprising C channels of a 2-D input feature map (each having dimensions H×W); and a filter comprising M 3-D filters, each 3-D filter comprising C channels of a 2-D filter (each having dimensions R×S). The input feature map 1730 may be flattened into C input channels, each comprising N×H×W pixel values, where each input channel may need to be mapped to a row in the PE array 1720. The filter 1740 may be flattened into M channels, each comprising C×R×S weight values, where each of the M channels may need to be mapped to a column in the PE array 1720. In Figure 17 In the example shown, M may be greater than the number of columns in PE array 1720. Thus, PE array 1720 may not be arranged as described above with respect to, for example, Figure 9 The described operations are used to perform the convolution operation.

[0137] According to certain embodiments, the M 3-D filters (corresponding to output channels) used for the convolution operation can be divided into K groups, where each group can include M / K 3-D filters or output channels. Thus, the convolution operation can be divided into K sub-operations, where each sub-operation can use M / K 3-D filters and input feature map 1730 (including C channels, each channel including N×H×W pixel values) to generate an output feature map on M / K output channels, where each output channel can include N output feature maps, each including E×F pixels. The K sub-operations can be executed in parallel by multiple (e.g., K) computation engines described above, where each sub-operation can be executed by one computation engine in a shorter period of time because there are fewer weights to be loaded and there are fewer columns for the input pixel values ​​in each channel of the input feature map to propagate through all columns of the PE array 1720.

[0138] For example, the first sub-operation may be performed by the first accelerator 1710-1 using the PE array 1720-1. The first accelerator 1710-1 may use the input feature map 1730 and the first group of 3-D filters 1740-1 including M / K 3-D filters to generate a first group of output feature maps 1750-1 in M / K output channels. The K-th sub-operation may be performed by the accelerator K (1710-K) using the PE array 1720-K. The accelerator K (1710-K) may use the input feature map 1730 and the K-th group of 3-D filters 1740-K including M / K 3-D filters to generate a K-th group of output feature maps 1750-K in M / K output channels. The output feature maps generated by the K accelerators are the final output feature maps of the convolution operation, and therefore no additional accumulation is required. The output feature map generated by each of the K accelerators may be saved into a portion of the memory space used for the output feature map of the convolution operation and may be used to make predictions or decisions.

[0139] Figure 18An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple compute engines according to certain embodiments is shown. As shown, a convolution operation to be performed by PE array 1820 can use N 3-D inputs, each 3-D input including C channels of 2-D input feature maps (each having dimensions H×W); and 3-D filters 1840, which include M 3-D filters, each including C channels of 2-D filters (each having dimensions R×S), to generate an output feature map 1850 including an output feature map of M output channels. Each output channel may include N output feature maps, each including E×F pixels. Input feature map 1830 can be flattened into C input channels, each including N×H×W pixel values, where each input channel may need to be mapped to a row in PE array 1820. The 3-D filter 1840 may be flattened into M channels, each channel comprising C×R×S weight values, where each of the M channels may need to be mapped to a column in the PE array 1820 .

[0140] According to certain embodiments, the C channels of the input feature map 1830 for the convolution operation may be divided into L groups, each of which may include N×H×W / L pixels. Thus, the convolution operation may be divided into L sub-operations, each of which may use M 3-D filters and a portion of the input feature map 1830 (including C channels, each of which includes N×H×W / L pixel values) to generate a portion (e.g., several rows) of each output feature map across M output channels, each of which may include N / L output feature maps, each including E×F pixels, or N output feature maps, each including E×F / L pixels. The L sub-operations may be executed in parallel by multiple (e.g., L) computation engines described above, wherein each sub-operation may be executed by one computation engine in a shorter period of time because fewer input pixel values ​​in each channel of the input feature map are propagated through the PEs in a row of the PE array 1820.

[0141] For example, the first sub-operation can be performed by the first accelerator 1810-1 using the PE array 1820-1. The first accelerator 1810-1 can use M 3-D filters 1840 and input 1830-1 (including C input channels, each input channel including N×H×W / L pixel values) to generate a first set of output feature maps 1850-1 including M output channels, where each output channel may include N / L output feature maps or N×E×F / L pixels. For example, when N=1, each output channel may include 1 / L (e.g., E / L rows) of output feature maps or E×F / L pixels. The Lth sub-operation can be performed by the accelerator L (1810-L) using the PE array 1820-L. Accelerator L (1810-L) can use M 3-D filters 1840 and input 1830-L (including C input channels, each input channel including N×H×W / L pixel values) to generate an L-th set of output feature maps 1850-L including M output channels, where each output channel may include N / L output feature maps. For example, when N=1, each output channel may include 1 / L (e.g., E / L rows) of output feature maps or E×F / L pixels. The output feature maps generated by the L accelerators are the final output feature maps of the convolution operation and therefore do not require additional accumulation. The output feature maps generated by each of the L accelerators can be saved in the memory space of the output feature map for the convolution operation and can be used to make predictions or decisions.

[0142] Figure 19 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple compute engines according to certain embodiments is shown. As shown, the convolution operation to be performed by PE array 1920 can use N 3-D inputs, each 3-D input including C channels of input feature maps (each having dimensions H×W); and 3-D filter 1940, which includes M 3-D filters, each 3-D filter including C channels of 2-D filters (each having dimensions R×S) to generate an output feature map including M output channels. Each output channel may include N output feature maps, each including E×F pixels. Input feature map 1930 can be flattened into C input channels, each including N×H×W pixel values, where each input channel may need to be mapped to a row in PE array 1920. The 3-D filter 1940 may be flattened into M channels, each channel comprising C×R×S weight values, where each of the M channels may need to be mapped to a column in the PE array 1920. Figure 19 In the example shown, M may be greater than the number of columns in PE array 1920. Therefore, PE array 1920 may not be arranged as described above with respect to, for example Figure 9 The described operations are used to perform the convolution operation.

[0143] According to certain embodiments, the C channels of the input feature map 1930 for the convolution operation may be divided into L groups, where each group may include N×H×W / L pixels. In addition, the M 3-D filters 1940 (corresponding to the output channels) used for the convolution operation may be divided into K groups, where each group may include M / K 3-D filters or output channels. Thus, the convolution operation may be divided into L×K sub-operations, where each sub-operation may use M / K 3-D filters and a portion of the input feature map 1930 (including C channels, each including N×H×W / L pixel values) to generate a portion (e.g., several rows) of each output feature map on M / K output channels, where each output channel may include N / L output feature maps, each including E×F pixels. The L×K sub-operations can be performed in parallel by multiple (e.g., L×K) computing engines described above, where each sub-operation can be performed by one computing engine in a shorter period of time because there will be fewer (e.g., 1 / L) input pixel values ​​in each channel of the input feature map to propagate through fewer (e.g., 1 / K) PEs in each row in the PE array 1920.

[0144] For example, the first sub-operation may be performed by the first accelerator 1910-1 using the PE array 1920-1. The first accelerator 1910-1 may use the M / K 3-D filters (1940-1) and the input 1930-1 (including C channels, each channel including N×H×W / L pixel values) to generate a first set of output feature maps 1950-1 in M / K output channels, where each output channel may include N / L output feature maps or N×E×F / L pixels. For example, when N=1, each output channel may include 1 / L (e.g., E / L rows) of output feature maps or E×F / L pixels. The (K×L)th sub-operation may be performed by the accelerator K×L (1910-K×L) using the PE array 1920-K×L. The K×L accelerators (1910-K×L) may use M / K 3-D filters (1940-K) and input 1930-L (comprising C channels, each channel comprising N / L input feature maps or N×H×W / L pixel values) to generate a (K×L)th set of output feature maps 1950-K×L in M / K output channels, where each output channel may include N / L output feature maps. For example, when N=1, each output channel may include 1 / L (e.g., E / L rows) of output feature maps or E×F / L pixels. The output feature maps generated by the K×L accelerators are the final output feature maps of the convolution operation, and therefore, no additional accumulation is required. The output feature map generated by each of the K×L accelerators may be an output feature map of 1 / (K×L) of the convolution operation and may be saved to the memory space for the output feature map of the convolution operation or may be used for prediction or decision making.

[0145] In the above relative to Figures 17 to 19 In the example described for splitting tensor operations of a neural network, each sub-operation can generate a portion of the final output feature map, which can be used to make predictions or decisions and / or can be saved to a memory space for output feature maps (e.g., concatenated in the memory space), where additional arithmetic calculations (e.g., summation or accumulation) may not be required to generate the final output feature map. In some embodiments, the tensor operation can be divided so that each sub-operation can generate a partial sum of the final output feature map, and the partial sums from the sub-operations can be summed in additional arithmetic calculations to generate the final output feature map or perform inference.

[0146] Figure 20 An example of splitting the operation of a neural network into sub-operations to be performed in parallel by multiple compute engines according to certain embodiments is shown. As shown, the convolution operation to be performed by PE array 2020 can use N 3-D inputs, each 3-D input including C channels of input feature maps (each having dimensions H×W); and 3-D filter 2040, which includes M 3-D filters, each 3-D filter including C channels of 2-D filters (each having dimensions R×S) to generate an output feature map including M output channels. Each output channel may include N output feature maps, each including E×F pixels. Input feature map 2030 can be flattened into C input channels, each including N×H×W pixel values, where each input channel may need to be mapped to a row in PE array 2020. The 3-D filter 2040 may be flattened into M channels, each channel comprising C×R×S weight values, where each of the M channels may need to be mapped to a column in the PE array 2020. Figure 20 In the example shown, C may be greater than the number of rows in the PE array 2020. Therefore, the PE array 2020 may not be as large as described above with respect to, for example Figure 9 The described operations are used to perform the convolution operation.

[0147] According to certain embodiments, the C channels of the input feature map 2030 for the convolution operation may be divided into K groups, where each group may include an input feature map of C / K channels. Furthermore, the M 3-D filters 2040 (corresponding to output channels) used for the convolution operation may be divided into K groups, where each group may include M 3-D filters, each 3-D filter including C / K 2-D filters. Thus, the convolution operation may be divided into K sub-operations, where each sub-operation may use C / K 2-D filters from each of the M 3-D filters and a portion of the input feature map 2030 (including the input feature map of C / K channels, each channel including N×H×W pixel values) to generate a portion and feature map of the output feature map on M output channels, where each output channel may include N portion and feature maps, each portion and feature map including E×F pixels. The K sub-operations can be performed in parallel by multiple (e.g., K) compute engines described above, where each sub-operation can be performed by one compute engine in a shorter period of time because fewer weights are loaded into the PE array and there are fewer (e.g., 1 / K) rows for partial sums to propagate through all rows in the PE array 2020.

[0148] For example, the first sub-operation may be performed by the first accelerator 2010-1 using the PE array 2020-1. The first accelerator 2010-1 may use the C / K 2-D filters in each of the M 3-D filters 2040 and a portion of the input feature map 2030 (including the input feature map of C / K channels) to generate a portion and feature map 2050-1 of the output feature map on M output channels, where each output channel may include N portions and feature maps, each portion and feature map including E×F pixels. The Kth sub-operation may be generated and performed by the accelerator K (2010-K) using the PE array 2020-K. Accelerator K (2010-K) may use C / K 2-D filters in each of M 3-D filters 2040 and a portion of the input feature map 2030 (including the input feature map of C / K channels) to generate a partial sum feature map 2050-K of the output feature map on M output channels, where each output channel may include N partial sum feature maps, each partial sum feature map including E×F pixels. The partial sum feature maps of the output feature maps generated by the K accelerators are not the final output feature maps of the convolution operation, and additional accumulation may be required to generate the final output feature map.

[0149] Figure 2121 is a flowchart 2100 illustrating an example of a method for accelerating tensor operations by executing sub-operations of tensor operations in parallel on multiple compute engines according to certain embodiments. The operations described in flowchart 2100 can be performed, for example, by a compiler running on a host system, such as the compiler and host system described below with respect to FIG. 22 . Although flowchart 2100 may describe the operations as a sequential process, in various embodiments, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. The process may have additional steps not shown in the figure. In addition, the embodiment of the method can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the associated tasks may be stored in a computer-readable medium such as a storage medium.

[0150] At block 2110, the host system may receive a neural network model that includes a first tensor operation, such as a convolution operation. The tensor operation may be used to generate an output tensor including a set of output feature maps using a set of input feature maps and a set of filters. As described above (e.g., in Figure 5 、 Figure 6 、 Figure 9 and Figures 17 to 20 ), in some embodiments, the set of input feature maps may include C input feature maps for each of the N input data sets, the set of filters may include M filters each including C filter matrices, and the set of output feature maps may include M output feature maps for each of the N input data sets, wherein each of the M output feature maps for a particular input data set may be the result of a convolution between the C input feature maps of the input data set and the corresponding C filter matrices in a particular filter of the M filters. In some embodiments, M may be greater than the total number of columns of the two-dimensional array of processing elements in the computing engine. In some embodiments, C may be greater than the total number of rows of the two-dimensional array of processing elements in the computing engine.

[0151] At block 2120, a compiler executed on a host system may divide the first tensor operation into sub-operations. The sub-operations may include two or more sub-operations that do not have data dependencies between the two or more sub-operations. Each of the two or more sub-operations may generate a portion of the set of output feature maps. In some embodiments, the portion of the set of output feature maps may include a portion of the total number of output feature maps in the set of output feature maps, wherein the sub-operation may use the set of input feature maps and a portion of the total number of filters in the set of filters to generate the portion of the set of output feature maps. In some embodiments, the portion of the set of output feature maps may include a portion of each output feature map in at least a subset of the set of output feature maps, wherein the sub-operation may use the set of filters and a portion of each input feature map in at least a subset of the set of input feature maps to generate the portion of the set of output feature maps. In some embodiments, the portion of the set of output feature maps may include a portion of each output feature map in a portion of the total number of output feature maps in the set of output feature maps, wherein the sub-operation may use a portion of the total number of filters in the set of filters and a portion of each input feature map in at least a subset of the set of input feature maps to generate the portion of the set of output feature maps.

[0152] At block 2130, the compiler may assign a first sub-operation of the two or more sub-operations to a first computing engine (e.g., the accelerator described above) of two or more computing engines on the integrated circuit device or on two or more integrated circuit devices. At block 2140, the compiler may assign a second sub-operation of the two or more sub-operations to a second computing engine of the two or more computing engines. At block 2150, the compiler may generate instructions (e.g., machine code) for executing the first sub-operation in parallel by the first computing engine and executing the second sub-operation by the second computing engine. Optionally, at block 2160, the compiler may generate instructions for performing inference based on the result of the first sub-operation and / or the result of the second sub-operation. For example, inference may be performed based on a portion of the set of output feature maps generated by one sub-operation or based on the entire set of output feature maps generated by two or more sub-operations.

[0153] In some embodiments, the compiler may also identify a second tensor operation in the neural network model, wherein the second tensor operation and the first tensor operation may not have data dependencies. The compiler may divide the second tensor operation into sub-operations of the second tensor operation, wherein the sub-operations of the second tensor operation may not have data dependencies. The compiler may assign a first sub-operation of the sub-operations of the second tensor operation to a third compute engine, assign a second sub-operation of the sub-operations of the second tensor operation to a fourth compute engine, and generate instructions for executing the first sub-operation of the sub-operations of the second tensor operation by the third compute engine and executing the second sub-operation of the sub-operations of the second tensor operation by the fourth compute engine in parallel with the first sub-operation of the first tensor operation and the second sub-operation of the first tensor operation. The first, second, third, and fourth compute engines may be on the same integrated circuit device, or on two or more integrated circuit devices. For example, the four compute engines may be four accelerators in the same acceleration engine.

[0154] The techniques described above can be applied to any tensor operation or any operation involving matrix multiplication, such as the one described above with respect to Figure 1 Operation of a multilayer perceptron as described. In one example, as an alternative to convolution, a transformer for natural language processing (NLP) may encode each position and apply an attention mechanism to associate two distant words, which can be parallelized to speed up training. An attention mechanism in a transformer is a way of computing the relevance of a set of values ​​(e.g., information) based on some key and query. An attention mechanism can be used by a transformer to focus on relevant information based on what is currently being processed. An attention weight may represent the relevance of an encoder hidden state (e.g., a value) when processing a decoder state (e.g., a query), and may be computed based on the encoder hidden state (e.g., a key) and the decoder hidden state (e.g., a query). A transformer can reduce the number of sequential operations by using a multi-headed attention mechanism that can model dependencies to associate two symbols from an input / output sequence with a constant number of O(1) operations, regardless of the distance between the symbols in the input or output sentence.

[0155] A transformer typically includes an encoder and a decoder. The encoder can map an input sequence of symbolic representations (x1, ..., xn) to a continuous representation sequence z = (z1, ..., zn). The decoder can generate one symbol at a time and generate an output sequence of symbols (y1, ..., ym) based on z. The encoder can use the embedding of the source sentence for its keys, values, and queries, while the decoder can use the output of the encoder for its keys and values, and the embedding of the target sentence for its queries. Transformers typically use stacked self-attention and point-by-point fully connected layers for both the encoder and decoder, where previously generated symbols are used as additional input when generating new symbols.

[0156] Figure 22A An example of a transformer 2200 that can use the techniques disclosed herein according to certain embodiments is shown. The transformer 2200 may include an encoder 2210 and a decoder 2220. The encoder 2210 may include a stack of N layers 2212. Each layer 2212 may include two sublayers that perform matrix multiplication and element-wise transformation. The first sublayer may include a multi-head self-attention network, and the second sublayer may include a position-by-position fully connected feedforward network. Residual connections may be used around each of the two sublayers, followed by layer normalization. Residual connections add the input to the output of the sublayer and are a way to make training deep networks easier. Layer normalization is a normalization method in deep learning that is similar to batch normalization. The output of each sublayer can be written as LayerNorm(x+Sublayer(x)), where Sublayer(x) is a function implemented by the sublayer. In the encoder stage, the transformer first generates initial input (e.g., input embedding and position encoding) for each word in the input sentence. For each word, self-attention aggregates information from all other words (pairs) in the context of the sentence to create a new representation for each word that is the attended representation of all other words in the sequence. This is repeated multiple times for each word in the sentence, continuously building updated representations on top of previous ones.

[0157] The decoder 2220 may also include a stack of N layers 2222. In addition to the two sub-layers in each encoder layer 2212 described above, each layer 2222 in the decoder 2220 may also include a third sub-layer that performs multi-head attention on the output of the encoder stack. Similar to the layer 2212 in the encoder 2210, residual connections around each of the sub-layers may be used in the layer 2222 in the decoder 2220, followed by layer normalization. The self-attention sub-layer in the decoder stack may be modified (labeled "masked multi-head attention") to mask the decoder input in future time steps and prevent positions from attending to subsequent positions. This masking, combined with the offset of the output embedding by one position, ensures that the prediction for position i can only depend on the known output at positions less than i. The decoder 2220 may generate one word at a time, from left to right. The first word generated at a layer may be based on the final representation of the encoder (offset by 1 position). Each subsequent word predicted may attend to the previously generated words at that layer of the decoder as well as the final representation of the encoder.

[0158] The attention function maps a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The query vector q encodes the focused word / position. The key vector k encodes the focused word. The key vector k and the query vector q together determine the attention score between the corresponding words. The output is computed as a weighted sum of the values, where the weight assigned to each value is calculated using a compatibility function between the query and the corresponding key.

[0159] Figure 22B An example of a scaled dot product attention box 2230 is shown. In the scaled dot product attention box 2230, the input includes all dimensions d k The query and key, and the dimension is d v The scaled dot product attention can be calculated for a set of queries simultaneously according to the following equation:

[0160]

[0161] Where Q is a matrix of queries packed together, and K and V are matrices of keys and values ​​packed together. Scaled dot product attention computes the dot product (attention score) of the query with all keys (“MatMul”), dividing each element of the dot product by a scaling factor ("scaling"), applying a softmax function to obtain weights for the values, and then using the weights to determine the weighted sum of the values.

[0162] When using only a single attention to compute a weighted sum of values, it can be difficult to capture the different aspects of the input. For example, in the sentence “I like cats more than dogs,” one might want to capture the fact that the sentence compares two entities while preserving the actual entities being compared. The Transformer can use a multi-headed self-attention sublayer to allow the encoder and decoder to view the entire input sequence simultaneously. To learn different representations, multi-headed attention applies different linear transformations to the value, key, and query for each attention head, where different weight matrices can be used for multiple attention heads, and the results of multiple attention heads can be concatenated.

[0163] Figure 22C An example of a multi-head attention sublayer 2240 used in the encoder 2210 and decoder 2220 of the transformer 2200 described above is shown. 模型 A single attention function is performed on the key, value, and query dimensions. The multi-head self-attention sublayer 2240 linearly projects the query, key, and value onto d dimensions multiple times (e.g., h times) using different learned linear projections. k d k and d vThe attention function is performed in parallel on h projected versions of the query, key, and value using multiple (e.g., h) scaled dot-product attentions, resulting in hd v dimensional output value. Each attention head can have Figure 22B The structure shown in , and can be characterized by three different projections given by the weight matrix:

[0164] The dimension is d 模型 xd k

[0165] The dimension is d 模型 xd k

[0166] The dimension is d 模型 xd v .

[0167] The outputs of multiple scaled dot-product attentions are concatenated to produce a matrix of dimension d i ×(h×d v ) matrix, where d i is the length of the input sequence. Then, we have a sequence with dimensions (h×d v )×d e A linear layer with a weight matrix W0 is applied to the concatenated result, resulting in a value of dimension d i ×d e The final result:

[0168]

[0169] where d e is the dimension of the word embedding. Multi-head attention allows the network to jointly attend to information from different representation subspaces at different positions. Multi-head attention can be performed using tensor operations, which can be split into multiple sub-operations (e.g., one sub-operation per head) and executed in parallel by multiple computation engines as described above.

[0170] Figure 23An example of a bidirectional encoder representation from a transformer (BERT) model 2300 that can use some of the techniques disclosed herein is shown according to certain embodiments. The BERT model may include multiple layers of bidirectional transformer encoders (rather than left-to-right transformer encoders) and does not include a transformer decoder because the BERT model is used to generate a language model. The BERT model is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on left and right context in all layers. The pre-trained BERT model can be fine-tuned with additional output layers to create state-of-the-art models for a variety of tasks (such as question answering and language inference) without requiring extensive task-specific architectural modifications. BERT alleviates the unidirectionality constraint by using a "masked language model" (MLM) pre-training object. A masked language model randomly masks some of the word units from the input, and the goal is to predict the original vocabulary identity (Id) of the masked word based solely on the context of the masked word. Unlike left-to-right language model pre-training, the MLM object enables the representation to incorporate left and right context, which allows pre-training of deep bidirectional transformers. In addition to masked language models, the next sentence prediction task can also be used to jointly pre-train text pair representations.

[0171] exist Figure 23 In the example shown, the BERT model 2300 uses an input including a word sequence 2306, which may include one or more sentences, such as a first sentence 2302 and a second sentence 2304. In some embodiments, some (e.g., about 15%) of the word units 2306 may be masked. The input word units 2306 may be embedded in vectors 2310 and processed by encoder layers 2320, 2330, ..., and 2340 to generate word sequences 2350, each represented by a vector. The encoder layers 2320, 2330, ..., and 2340 may form a multi-layer perceptron. Each encoder layer 2320, 2330, ..., or 2340 may be similar to the encoder layer 2212 and may include the following components as described above with respect to the encoder layer 2212: Figures 22A to 22C The multi-head attention model and / or fully connected layer described. The multi-head attention model may include multiple dot product attentions. The operation of each encoder layer 2320, 2330, ..., or 2340 may include tensor operations, which can be split into sub-operations that have no data dependencies on each other and can therefore be executed in parallel by multiple computing engines (e.g., accelerators) as described above.

[0172] Figure 24Included is a block diagram illustrating an example of a host system 2400 on which a compiler 2430, such as that described herein, may run. The illustrated host system 2400 is an example of a computing device and includes a processor 2402, processor memory 2404, at least one storage device 2406, various input / output (I / O) devices 2408, and at least one network interface 2410. Figure 24 In the example, host system 2400 also includes an acceleration engine 2412, which is an integrated circuit device that can accelerate certain operations or calculations performed by host system 2400. In various examples, host system 2400 can be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smartphone, among other examples. In some examples, operations or components discussed below as being executed or included in host system 2400 can be executed or included in other computer devices. For example, compiler 2430 can be executed on host system 2400, while acceleration engine 2412 is located on a different host system.

[0173] The processor 2402 is an integrated circuit device that can execute program code in the form of instructions. The program code can be used in various software applications or tools, such as an operating system 2420 or a compiler 2430 shown. When the processor 2402 is executing a program, the instructions for the program can be stored in the processor memory 2404. The instructions can also be stored in other locations, such as a storage device 2406, and can be loaded into the processor memory 2404 when needed by the processor 2402. The processor 2402 can also use the processor memory 2404 to temporarily store other data for which the processor 2402 operates. In various examples, the processor memory 2404 is a volatile memory type, such as a random access memory type, but a non-volatile memory type can alternatively or additionally be used for the processor memory 2404.

[0174] Storage device 2406 is an example of a device that may include non-volatile memory. For example, storage device 2406 may be a magnetic disk drive, a solid-state drive, or an optical disk drive, among other examples. Storage device 2406 may also be non-transitory, such that program code and other data stored on storage device 2406 remain present when storage device 2406 is not powered.

[0175] Storage device 2406 is an example of a peripheral device, which is a component that can be connected to host system 2400 to add functionality to host system 2400. Other examples of peripheral devices include input / output devices 2408 and network interface 2410. Input / output devices 2408 may include user input and output devices such as a keyboard, mouse, touch screen, microphone, display, speaker, printer, and scanner, among other examples. Network interface 2410, which may be implemented using a network interface card, may provide access to one or more networks. Network interface 2410 may include, for example, a physical port for connecting a network cable and / or a wireless antenna for communicating with Wi-Fi and / or cellular networks. Network interface 2410 may also be described as an I / O device.

[0176] The acceleration engine 2412 is also another type of peripheral device or I / O device. The acceleration engine 2412 is a device that is specifically built to perform certain operations that can be performed by the processor 2402 but can be performed faster by the acceleration engine 2412. For example, the acceleration engine 2412 may be a neural network accelerator and may therefore be able to perform massively parallel computations of neural networks more efficiently than when the processor 2402 performs the computations. As another example, the acceleration engine 2412 may be a graphics processing unit (GPU) and may be optimized to perform the computations required for graphics rendering. Other examples of devices that may be implemented by the acceleration engine 2412 include cryptographic accelerators, compression and decompression accelerators, 3-D accelerators, regular expression accelerators, security accelerators, and the like.

[0177] In various examples, the acceleration engine 2412 can execute program code to perform certain operations. For example, when the acceleration engine 2412 is a neural network accelerator, the acceleration engine 2412 can be programmed to execute a specific neural network, such as a neural network that performs image recognition or a neural network that performs machine translation. As another example, to support the execution of the neural network, the acceleration engine 2412 can be programmed to perform operations such as: copying data for the neural network from the processor memory 2404 (for example) to the acceleration engine 2412, copying input data for the neural network from the processor memory 2404 to the acceleration engine 2412, and / or copying results from the acceleration engine 2412 to the processor memory 2404, as well as other examples.

[0178] To generate program code for the acceleration engine 2412, in various examples, the host system 2400 may execute a compiler 2430. Generally speaking, a compiler is a software program that translates program code written in a human-readable language into a format (e.g., machine instructions) that can be read and processed by an integrated circuit device. Figure 24In the example of FIG, the acceleration engine 2412 is a neural network accelerator, and the compiler 2430 is used to compile the neural network description into instructions to be executed by the acceleration engine 2412. When the acceleration engine 2412 implements a different type of accelerator, another compiler can be used. As described above, in some embodiments, the compiler 2430 can be used to divide tensor operations into sub-operations and can schedule the execution of the sub-operations to different acceleration engines and / or different accelerators in the acceleration engine.

[0179] For example, when the operating system 2420 receives keyboard, mouse, touch screen, voice command, or other input from the input / output device 2408, the compiler 2430 may be activated. The input may also include parameters for the compiler 2430, such as input code 2442 that compiles and configures options for the compilation process. Once the compiler 2430 is activated, the processor 2402 may load instructions for the compiler 2430 into the processor memory 2404 and may execute the instructions.

[0180] exist Figure 24 In the example shown, compiler 2430 includes a first stage 2432, a second stage 2436, and a third stage 2440, each of which performs a different operation to produce compiled code 2444. In other examples, compiler 2430 may combine the operations of first stage 2432, second stage 2436, and / or third stage 2440 into fewer stages, or may divide the operations of one of the stages into multiple stages.

[0181] The first stage 2432 may receive and process input code 2442. The input code 2442 may describe a program in a high-level programming language such as Java, C++, or TensorFlow, among many other examples. The input code 2442 may describe, for example, the steps for performing image recognition, speech recognition, machine translation, or other operations. For example, the input code 2442 may be obtained from the storage device 2406. Alternatively, although not shown here, the input code 2442 may be located in the processor memory 2404 or may be obtained from a network location using the network interface 2410. Processing the input code 2442 may include classifying the operations described in the input code 2442 into various layers, where the output of one layer provides the input to the next layer. Processing may also include identifying steps to be performed by the processor 2402 rather than by the acceleration engine 2412. For example, the processor 2402 may be required to execute the driver 2422 to perform steps such as configuring direct memory access (DMA) descriptors to move data to or from the acceleration engine 2412, among other examples.

[0182] The output 2434 of the first stage 2432 can be organized, for example, into layers, nodes, and connections between nodes of a neural network. The second stage 2436 can perform intermediate processing on the output 2434. For example, the operations performed in any one layer or at any one node in a layer may be too numerous for the acceleration engine 2412 to perform simultaneously. The acceleration engine 2412 may, for example, have a limited amount of local storage for the data required for the computation, or the computation may exceed what the acceleration engine 2412 can perform at one time. In this example, the first stage 2432 can break down the operations of the layer or node into smaller operations that can fit within the acceleration engine's local memory and / or can fit within the acceleration engine's 2412 computational power. Processing the output 2434 of the first stage 2432 can include other steps, such as arranging or determining the order in which the acceleration engine 2412 and / or processor 2402 will perform the operations, as well as other examples.

[0183] In various examples, output 2438 of second stage 2436 includes various steps to be performed by components of acceleration engine 2412 in the order in which the steps are to be performed. Output 2438 can be represented as, for example, a data flow graph, where nodes in the graph represent memory operations, computations, and other operations, and edges or connections between nodes represent dependencies between nodes, such as data dependencies, memory dependencies, or operation dependencies, among other examples.

[0184] The third stage 2440 may operate on the output 2438 of the second stage 2436 and perform various steps before generating instructions to be executed by the acceleration engine 2412. These steps may include, for example, removing redundant dependencies, resolving or handling dependencies between nodes by inserting synchronization instructions into the code, identifying possible optimizations in memory usage or memory bandwidth usage, and other operations. In some examples, the third stage 2440 may include a data scheduler 2450.

[0185] The output of the third stage 2440 is compiled code 2444, which may include machine instructions in binary format. In some examples, the compiled code 2444 may be stored in the processor memory 2404. Alternatively or additionally, the compiled code 2444 may be copied to the storage device 2406 or a network location. As described above, the acceleration engine 2412 may be located on a different host system, in which case the compiled code 2444 may be sent to the other host system via the network interface 2410.

[0186] exist Figure 24In the example of FIG, the host system 2400 can execute a driver 2422, which can also be referred to as a device driver or runtime driver, to manage the acceleration engine 2412. The driver 2422 can provide an interface between an application executing on the host system 2400 (or another host system) and the acceleration engine 2412. For example, the driver 2422 can provide an application program interface (API) that defines functions for feeding input data to the acceleration engine 2412 and defining operations to be performed on the input data. In this and other examples, the driver 2422 can configure the acceleration engine 2412 to perform the operations. For example, the driver 2422 can identify the neural network that the acceleration engine 2412 will execute and the location of the compiled code 2444 of the neural network in the processor memory 2404 or on the storage device 2406. The driver 2422 may also load the acceleration engine 2412 or cause the acceleration engine 2412 to load the compiled code 2444, may load or cause the acceleration engine 2412 to load the input data on which the neural network operation will be performed, and / or may cause the acceleration engine 2412 to perform the operation on the input data. Once the acceleration engine 2412 is completed, the acceleration engine 2412 may notify the driver 2422, and the driver 2422 may transmit the result back to the application that requested the result.

[0187] Figure 25 Included is a block diagram illustrating an example of an acceleration engine 2500. Acceleration engine 2500 is an example of an integrated circuit that may include one or more accelerators 2502a-2502n, which may be similar to those described above with respect to, for example, Figure 7 For example, in one embodiment, the acceleration engine 2500 may include four accelerators that can be used to perform the four sub-operations of the convolution operation in parallel as described above.

[0188] exist Figure 25In the example of FIG, the acceleration engine 2500 includes a plurality of accelerators 2502a-2502n, each of which can perform a set of operations. In various examples, the accelerators 2502a-2502n are used for specific types of operations, so that the accelerators 2502a-2502n can perform operations faster than general-purpose processors when performing similar operations. In various examples, in order to perform a set of operations, the input data for which the operations will be performed must first be moved to the accelerators 2502a-2502n. Additionally, in some cases, program code is also moved to the accelerators 2502a-2502n, and the program code programs the operations that the accelerators 2502a-2502n will perform on the data. In the example shown, the acceleration engine 2500 includes N accelerators 2502a-2502n. Examples of accelerators that may be included in the acceleration engine 2500 include graphics accelerators, floating-point accelerators, neural network accelerators, etc. In various examples, accelerators 2502a-2502n can each be the same (e.g., each accelerator is a graphics accelerator) or can be different (e.g., accelerators 2502a-2502n include a graphics accelerator, a floating point accelerator, and a neural network accelerator).

[0189] The example acceleration engine 2500 also includes DRAM controllers 2542a-2542k for communicating with external memory. In this example, the external memory is implemented using DRAM 2530. In the example shown, the acceleration engine 2500 includes K DRAM controllers 2542a-2542k, each of which may be capable of communicating with a separate set of DRAM banks. In other examples, other types of RAM technologies may be used for the external memory. The DRAM controllers 2542a-2542k may also be referred to as memory controllers.

[0190] In various examples, input data and / or program code for the accelerators 2502a-2502n can be stored in DRAM 2530. Different programs can cause the accelerators 2502a-2502n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program can configure the neural network accelerator to perform speech recognition, while another program can configure the neural network accelerator to perform image recognition. In various examples, different accelerators 2502a-2502n can be programmed with different programs so that each accelerator performs a different set of operations. In various examples, the processors 2548a-2548s can manage the movement of program code from DRAM 2530 to the accelerators 2502a-2502n.

[0191] The example acceleration engine 2500 also includes I / O controllers 2544a-2544p for communicating with I / O devices 2532 in the system. The acceleration engine 2500 can communicate with the I / O devices via, for example, a processor bus. In some examples, the processor bus can be implemented using a peripheral component interconnect (PCI) and / or a variant of the PCI bus protocol. The processor bus can connect the acceleration engine 2500 to I / O devices, such as input and output devices, memory controllers, storage devices, and / or network interface cards, among other things. In some examples, the I / O controllers 2544a-2544p can enable the acceleration engine 2500 to act as an I / O device for a host processor. For example, the acceleration engine 2500 can be a recipient of input data from the host processor and commands indicating operations to be performed on the input data (e.g., specific calculations or analyses). In the example shown, the acceleration engine 2500 includes p I / O controllers 2544a-2544p, each of which may include a separate root complex and may communicate with a separate set of I / O devices 2532. In other examples, other standardized bus protocols such as Ultra Path Interconnect (UPI) may be used for the host bus. In other examples, a proprietary bus protocol may be used.

[0192] Data movement within the acceleration engine 2500 may be managed by one or more processors 2548a-2548s, which may also be referred to as data management processors. Figure 25 In the example of FIG, the acceleration engine 2500 includes s processors 2548a-2548s incorporated into the device (e.g., on the same silicon die). In other examples, the processors 2548a-2548s may be external to the acceleration engine 2500 (e.g., on different dies and / or in different packages). In some examples, the processors 2548a-2548s may manage the movement of data from the I / O device 2532 to the accelerators 2502a-2502n or the DRAM 2530. For example, input data may be located at the I / O device 2532 or in processor memory, and the processors 2548a-2548s may move input from the I / O device 2532 or processor memory to the accelerator or DRAM 2530. As another example, the program code for the accelerators 2502a-2502n may be located on the I / O device 2532 or in processor memory.

[0193] The example acceleration engine 2500 also includes DMA engines 2546a-2546d that can move data between the accelerators 2502a-2502n, the DRAM controllers 2542a-2542k, and the I / O controllers 2544a-2544p. In the example shown, the acceleration engine 2500 includes D DMA engines 2546a-2546d. In some implementations, the DMA engines 2546a-2546d can be assigned to specific tasks, such as moving data from the DRAM controllers 2542a-2542d to the accelerators 2502a-2502n, or moving data between the I / O controllers 2544a-2544p and the accelerators 2502a-2502n. For example, these tasks can be assigned by queuing descriptors using the DMA engines 2546a-2546d, where the descriptors identify the address of a data block and the operation to be performed (e.g., read or write). For example, a descriptor may direct the DMA engine to instruct the DMA controller to read a block of data from DRAM 2530. As another example, a descriptor may instruct the DMA engine to write data read by the DMA controller to the accelerator. Additional descriptors may be used to move data from the accelerator to DRAM 2530.

[0194] In various examples, each of the processors 2548a-2548s can be responsible for managing data movement for different accelerators. In some examples, a processor can manage data movement for more than one accelerator. Similarly, in various examples, each of the processors 2548a-2548s can be assigned to one or more DMA engines 2546a-2546d. In these and other examples, the association between the processors 2548a-2548s, the accelerators 2502a-2502n, and the DMA engines 2546a-2546d is determined by program code executed by each respective processor.

[0195] In the example acceleration engine 2500, various components can communicate via a chip interconnect 2520. The chip interconnect 2520 primarily includes wiring for routing data between components of the acceleration engine 2500. In some cases, the chip interconnect 2520 may include a minimal amount of logic, such as multiplexers to control the direction of data, flip-flops to handle crossing clock domains, and timing logic.

[0196] Figure 26 A diagram of an example network 2600 is included, which may include one or more host systems, such as Figure 24 The host system shown. For example, Figure 26 The example network 2600 includes a plurality of nodes 2602a-2602h, one or more of which may be, for example, Figure 24The other nodes 2602a-2602h may be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing instructions, and a network interface for connecting to the network 2600.

[0197] In various examples, network 2600 can be used to process data. For example, input data can be received at one of nodes 2602a-2602h or from other networks 2608 with which network 2600 can communicate. In this example, the input data can be directed to a node in network 2600 that includes an acceleration engine so that the acceleration engine can operate on the input data and generate a result. The result can then be transmitted to the node or other network from which the input data was received. In various examples, input data can be accumulated from various sources including one or more of nodes 2602a-2602h and / or computing devices located in other networks 2608, and the accumulated input data can be directed to one or more host systems in network 2600. The results from the host system can then be distributed back to the source from which the input data was collected.

[0198] In various examples, one or more of nodes 2602a-2602h may be responsible for operations such as accumulating input data for host systems to operate, tracking which host systems are busy and which can accept more work, determining whether host systems are operating correctly and / or most efficiently, monitoring network security, and / or other management operations.

[0199] exist Figure 26 In the example of FIG, nodes 2602a-2602h are interconnected using a switching fabric with point-to-point links. The switching fabric includes multiple switches 2604a-2604d, which may be arranged in a multi-layer network such as a Clos network. A network device that filters and forwards packets between local area network (LAN) segments may be referred to as a switch. Switches typically operate at the data link layer (Layer 2) and sometimes at the network layer (Layer 3) of the Open Systems Interconnection (OSI) reference model and may support a variety of packet protocols. Figure 26 The switches 2604a-2604d may be connected to the nodes 2602a-2602h and provide multiple paths between any two nodes.

[0200] The network 2600 may also include one or more network devices, such as routers 2606, for connecting to other networks 2608. Routers use headers and forwarding tables to determine the best path for forwarding data packets, and use protocols such as the Internet Control Message Protocol (ICMP) to communicate with each other and configure the best route between any two devices. Figure 26The router 2606 may be used to connect to other networks 2608, such as a subnet, a LAN, a wide area network (WAN), and / or the Internet.

[0201] In some examples, the network 2600 can include any one or a combination of many different types of networks, such as a cable network, the Internet, a wireless network, a cellular network, and other private and / or public networks. The interconnected switches 2604a-2604d and routers 2606 (if present) can be referred to as a switching fabric 2610, a fabric, a network fabric, or simply a network. In the context of computer networks, the terms "fabric" and "network" are used interchangeably herein.

[0202] Nodes 2602a-2602h may be any combination of host systems, processor nodes, storage subsystems, and I / O chassis representing user devices, service provider computers, or third-party computers.

[0203] The user device may include a computing device that accesses an application 2632 (e.g., a web browser or mobile device application). In some aspects, the application 2632 may be hosted, managed, and / or provided by a computing resource service or service provider. The application 2632 may allow one or more users to interact with one or more service provider computers to, for example, access web content (e.g., web pages, music, videos, etc.). The one or more user devices may be computing devices such as, for example, mobile phones, smartphones, personal digital assistants (PDAs), laptop computers, netbook computers, desktop computers, thin client devices, tablet computers, electronic book (e-book) readers, game consoles, etc. In some examples, the one or more user devices may communicate with the one or more service provider computers via one or more other networks 2608. Alternatively, the one or more user devices may be part of a distributed system that is managed, controlled, or otherwise part of one or more service provider computers (e.g., a console device integrated with the service provider computers).

[0204] Figure 26One or more nodes may also represent one or more service provider computers. One or more service provider computers may provide native applications configured to run on user devices, with which one or more users can interact. In some examples, one or more service provider computers may provide computing resources, such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, and the like. One or more service provider computers may also be operable to provide one or more users with web hosting, database generation, computer application development and / or implementation platforms, combinations of the foregoing, and the like. In some examples, one or more service provider computers may be provided as one or more virtual machines implemented in a hosted computing environment. A hosted computing environment may include one or more computing resources that can be rapidly provisioned and released. These computing resources may include computing, networking, and / or storage devices. A hosted computing environment may also be referred to as a cloud computing environment. One or more service provider computers may include one or more servers, which may be arranged in a cluster, as a server farm, or as separate, independent servers, and may host applications 2632 and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some aspects, the one or more service provider computers may additionally or alternatively comprise a computing device such as, for example, a mobile phone, a smartphone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a netbook computer, a server computer, a thin client device, a tablet computer, a gaming console, etc. In some cases, the one or more service provider computers may communicate with one or more third-party computers.

[0205] In one example configuration, one or more nodes 2602a-2602h may include at least one memory 2618 and one or more processing units (or one or more processors 2620). The one or more processors 2620 may be implemented in hardware, computer-executable instructions, firmware, or a combination thereof. The computer-executable instructions or firmware implementation of the one or more processors 2620 may include computer-executable instructions or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0206] In some cases, one or more hardware processors 2620 may be single-core processors or multi-core processors. A multi-core processor may include multiple processing units within the same processor. In some examples, the multi-core processors may share certain resources, such as a bus and second-level or third-level caches. In some cases, each core in a single-core or multi-core processor may also include multiple execution logical processors (or execution threads). Among such cores (e.g., those with multiple logical processors), the execution pipeline and several levels of lower-level caches may also be shared.

[0207] The memory 2618 may store program instructions that are loadable and executable on one or more processors 2620, as well as data generated during the execution of these programs. Depending on the configuration and type of one or more nodes 2602a-2602h, the memory 2618 may be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). The memory 2618 may include an operating system 2628, one or more data stores 2630, one or more applications 2632, one or more drivers 2634, and / or services for implementing the features disclosed herein.

[0208] Operating system 2628 may support basic functions of nodes 2602a-2602h, such as scheduling tasks, executing applications, and / or controlling peripheral devices. In some implementations, the service provider computer may host one or more virtual machines. In these implementations, each virtual machine may be configured to execute its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. Operating system 2628 may also be a proprietary operating system.

[0209] Data store 2630 may include permanent or temporary data used and / or operated by operating system 2628, applications 2632, or drivers 2634. Examples of such data include web pages, video data, audio data, images, user data, and the like. In some implementations, the information in data store 2630 may be provided to a user device via one or more networks 2608. In some cases, data store 2630 may additionally or alternatively include stored applications and / or drivers. Alternatively or additionally, data store 2630 may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in data store 2630 may be machine-readable object code, source code, interpreted code, or intermediate code.

[0210] Driver 2634 comprises the program that can provide communication between the parts in the node.For example, some drivers 2634 can provide communication between operating system 2628 and extra storage 2622, network device 2624 and / or I / O device 2626.Alternatively or additionally, some drivers 2634 can provide communication between application 2632 and operating system 2628 and / or between application 2632 and the accessible peripheral device of service provider computer.In many cases, driver 2634 can comprise the driver (for example, printer driver, display driver, hard disk driver, solid state device driver) that provides well-understood function.In other cases, driver 2634 can provide proprietary or specialized function.

[0211] One or more service provider computers or servers may also include additional storage 2622, which may include removable storage and / or non-removable storage. Additional storage 2622 may include magnetic storage, optical disks, solid-state disks, flash memory, and / or tape storage. Additional storage 2622 may be housed in the same chassis as one or more nodes 2602a-2602h or may be in a housing. Memory 2618 and / or additional storage 2622 and its associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some implementations, memory 2618 may include multiple different types of memory, such as SRAM, DRAM, or ROM.

[0212] Memory 2618 and additional storage 2622 (both removable and non-removable) are examples of computer-readable storage media. For example, a computer-readable storage medium may include volatile or non-volatile, removable or non-removable media implemented in a method or technology for storing information, the information including, for example, computer-readable instructions, data structures, program modules, or other data. Memory 2618 and additional storage 2622 are examples of computer storage media. Other types of computer storage media that may be present in one or more nodes 2602a-2602h may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, solid-state drives, or other media that can be used to store desired information and can be accessed by one or more nodes 2602a-2602h. Computer-readable media also includes any combination of the above media types, including multiple units of a media type.

[0213] Alternatively or additionally, computer-readable communication media may include computer-readable instructions, program modules, or other data transmitted within a data signal (such as a carrier wave or other transmission medium). However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0214] One or more nodes 2602a-2602h may also include I / O devices 2626, such as a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc. One or more nodes 2602a-2602h may also include one or more communication channels 2636. Communication channels 2636 may provide a medium through which various components of one or more nodes 2602a-2602h can communicate. One or more communication channels 2636 may take the form of a bus, ring, switch fabric, or network.

[0215] One or more nodes 2602a - 2602h may also include one or more network devices 2624 that allow one or more nodes 2602a - 2602h to communicate with a stored database, another computing device or server, a user terminal, and / or other devices on the one or more networks 2600 .

[0216] In some implementations, the network device 2624 is a peripheral device, such as a PCI-based device. In these implementations, the network device 2624 includes a PCI interface for communicating with a host device. The term "PCI" or "PCI-based" may be used to describe any protocol in the PCI bus protocol family, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and PCI-Express (PCIe), or any other improved or derived protocol based on the PCI protocol discussed herein. A PCI-based protocol is a standard bus protocol for connecting devices such as local peripheral devices to a host device. A standard bus protocol is a data transfer protocol whose specifications have been defined and adopted by various manufacturers. Manufacturers ensure that devices that comply with the specification are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using Non-Volatile Memory Express (NVMe). NVMe is a device interface specification for accessing non-volatile storage media attached to a computing system using PCIe. For example, the bus interface module may implement NVMe and the network device 2624 may connect to the computing system using a PCIe interface.

[0217] A PCI-based device may include one or more functions. A "function" describes an operation that can be provided by the network device 2624. Examples of functions include a mass storage controller, a network controller, a display controller, a memory controller, a serial bus controller, a wireless controller, and an encryption and decryption controller. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some implementations, a PCI-based device may have up to eight functions.

[0218] In some implementations, the network device 2624 may include single root I / O virtualization (SR-IOV). SR-IOV is an expansion capability that may be included in a PCI-based device. SR-IOV allows a physical resource (e.g., a single network interface controller) to appear as multiple resources (e.g., 64 network interface controllers). Therefore, a PCI-based device that provides a certain function (e.g., a network interface controller) may appear to be multiple devices providing the same function to a device utilizing the PCI-based device. The functions of a storage adapter device that supports SR-IOV may be classified as physical functions (PFs) or virtual functions (VFs). A physical function is a full-featured function of a device that can be searched, managed, and manipulated. A physical function has configuration resources that can be used to configure or control a storage adapter device. A physical function includes the same configuration address space and memory address space as a non-virtualized device. A physical function may have many virtual functions associated with it. A virtual function is similar to a physical function, but may typically lack lightweight functionality for configuration resources and is typically controlled by the configuration of its underlying physical function. Each of the physical functions and / or virtual functions may be assigned to a corresponding thread of execution running on a host device (such as, for example, a virtual machine).

[0219] The modules described herein may be software modules, hardware modules, or a suitable combination thereof. If the modules are software modules, they may be embodied on a non-transitory computer-readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the processes and architectures described may be executed in real time or in an asynchronous mode prior to any user interaction. The modules may be configured in the manner suggested in the aforementioned figures, and / or the functionality described herein may be provided by one or more modules as separate modules, and / or the module functionality described herein may be spread across multiple modules.

[0220] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the claims.

[0221] Other variations are also within the spirit of the present disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, specific examples thereof have been shown in the drawings and described in detail above. However, it should be understood that there is no intention to limit the present disclosure to the specific form or forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0222] In the context of describing the disclosed examples (especially in the context of the following claims), the use of the terms "a, an" and "the" and similar referents should be interpreted as covering both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. The terms "include," "have," "include," and "contain" should be interpreted as open-ended terms (i.e., meaning "including but not limited to"), unless otherwise indicated. The term "connected" should be interpreted as partially or completely included, attached, or combined together, even if there are intervening objects. Unless otherwise indicated herein, the description of the value range herein is merely intended to be used as a shorthand method for individually referring to each individual value belonging to the range, and each individual value is incorporated into this specification as if it were individually described herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any appropriate order. The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the examples of the present disclosure and, unless otherwise required, does not impose limitations on the scope of the present disclosure. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0223] Unless specifically stated otherwise, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood in context as generally used to indicate that an item, term, etc. may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain examples require the corresponding presence of at least one of X, at least one of Y, or at least one of Z.

[0224] Various examples of the present disclosure are described herein, including the best mode known to the inventor for carrying out the present disclosure. After reading the foregoing description, the variants of those examples may become apparent to those of ordinary skill in the art. The inventor hopes that the technician will adopt such variants as appropriate, and the inventor intends to practice the present disclosure in a manner different from that specifically described herein. Therefore, where permitted by applicable law, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto. In addition, unless otherwise specified herein or otherwise clearly contradicted by context, the present disclosure encompasses any combination of the elements described above in all possible variations thereof.

[0225] Certain embodiments are described in the following examples. As used hereinafter, any reference to a series of examples should be understood as a separate reference to each example in the series (e.g., "Examples 1 to 3" should be understood as "Example 1, Example 2, or Example 3").

[0226] In Example 1, a computer-implemented method includes receiving a neural network model including a first tensor operation; dividing the first tensor operation into sub-operations of the first tensor operation, the sub-operations including two sub-operations with no data dependency between the two sub-operations; assigning a first sub-operation of the two sub-operations to a first computing engine; assigning a second sub-operation of the two sub-operations to a second computing engine; and generating instructions for executing the first sub-operation by the first computing engine and executing the second sub-operation by the second computing engine in parallel.

[0227] In Example 2, the first tensor operation of the computer-implemented method of Example 1 includes a multilayer perceptron operation.

[0228] In Example 3, the first tensor operation of the computer-implemented method of any one of Examples 1-2 uses a set of input feature maps and a set of filters to generate an output tensor comprising a set of output feature maps.

[0229] Example 4 includes the computer-implemented method of Example 3, wherein: the set of input feature maps includes C input feature maps for each of N input data sets; the set of filters includes M filters, each including C filter matrices; and the set of output feature maps includes M output feature maps for each of the N input data sets, wherein each of the M output feature maps for a certain input data set is a result of a convolution between the C input feature maps of the input data set and the C filter matrices in a certain filter of the M filters.

[0230] In Example 5, the first computing engine of the computer-implemented method of Example 4 includes a two-dimensional processing element array, characterized in that: the total number of columns of the two-dimensional processing element array is less than M; or the total number of rows of the two-dimensional processing element array is less than C.

[0231] In Example 6, the first sub-operation of the computer-implemented method of any one of Examples 3 to 5 generates a portion of the set of output feature maps.

[0232] Example 7 includes the computer-implemented method of Example 6, wherein the portion of the set of output feature maps comprises at least one of: a portion of a total number of output feature maps in the set of output feature maps; a portion of each output feature map in at least a subset of the set of output feature maps; or a portion of each output feature map in a portion of the total number of output feature maps in the set of output feature maps.

[0233] In Example 8, the first sub-operation of the computer-implemented method of any one of Examples 6 to 7 generates the portion of the set of output feature maps using at least one of: a portion of the set of input feature maps and a total number of filters in the set of filters; a portion of the set of filters and each input feature map in at least a subset of the set of input feature maps; or a portion of the total number of filters in the set of filters and a portion of each input feature map in a subset of the set of input feature maps.

[0234] In Example 9, the computer-implemented method of any one of Examples 1 to 8 further includes generating instructions for performing an inference based on a result of the first sub-operation, a result of the second sub-operation, or both.

[0235] In Example 10, the computer-implemented method of any one of Examples 1 to 9 further includes: identifying a second tensor operation in the neural network model, the second tensor operation and the first tensor operation having no data dependency; dividing the second tensor operation into sub-operations of the second tensor operation, the sub-operations of the second tensor operation having no data dependency; assigning a first sub-operation of the sub-operations of the second tensor operation to a third computing engine; assigning a second sub-operation of the sub-operations of the second tensor operation to a fourth computing engine; and generating instructions for executing the first sub-operation of the sub-operations of the second tensor operation by the third computing engine and executing the second sub-operation of the sub-operations of the second tensor operation by the fourth computing engine in parallel with the first sub-operation of the first tensor operation and the second sub-operation of the first tensor operation.

[0236] Example 11 includes a non-transitory computer-readable medium having instructions stored therein that, when executed by one or more processors, cause the one or more processors to perform operations including: receiving a neural network model including a tensor operation; dividing the tensor operation into sub-operations, the sub-operations including two sub-operations having no data dependency between the two sub-operations; assigning a first sub-operation of the two sub-operations to a first computing engine; assigning a second sub-operation of the two sub-operations to a second computing engine; and generating instructions for executing the first sub-operation by the first computing engine and executing the second sub-operation by the second computing engine in parallel.

[0237] Example 12 includes the non-transitory computer-readable medium of Example 11, wherein the instructions further cause the one or more processors to perform operations comprising: generating instructions for performing inference based on a result of the first sub-operation, a result of the second sub-operation, or both, wherein the first computing engine and the second computing engine are in the same integrated circuit device or in two different integrated circuit devices.

[0238] Example 13 includes the non-transitory computer-readable medium of any one of Examples 11 to 12, wherein: the first tensor operation uses a set of input feature maps and a set of filters to generate an output tensor comprising a set of output feature maps; and the first sub-operation generates a portion of the set of output feature maps.

[0239] Example 14 includes the non-transitory computer-readable medium of Example 13, wherein the portion of the set of output feature maps comprises at least one of: a portion of a total number of output feature maps in the set of output feature maps; a portion of each output feature map in at least a subset of the set of output feature maps; or a portion of each output feature map in a portion of the total number of output feature maps in the set of output feature maps.

[0240] In Example 15, the first sub-operation of the non-transitory computer-readable medium of any one of Examples 13 to 14 generates the portion of the set of output feature maps using at least one of: a portion of the set of input feature maps and a total number of filters in the set of filters; a portion of the set of filters and each input feature map in at least a subset of the set of input feature maps; or a portion of the total number of filters in the set of filters and each input feature map in a subset of the set of input feature maps.

[0241] Example 16 includes the non-transitory computer-readable medium of any one of Examples 13 to 15, wherein: the set of input feature maps includes C input feature maps for each of N input data sets; the set of filters includes M filters each including C filter matrices; and the set of output feature maps includes M output feature maps for each of the N input data sets, wherein each of the M output feature maps for a certain input data set is a result of a convolution between the C input feature maps of the input data set and the C filter matrices in a certain filter of the M filters.

[0242] Example 17 includes the non-transitory computer-readable medium of Example 16, wherein the first computing engine includes a two-dimensional processing element array, characterized in that: the total number of columns of the two-dimensional processing element array is less than M; or the total number of rows of the two-dimensional processing element array is less than C.

[0243] In Example 18, the first tensor operation of the non-transitory computer-readable medium of any one of Examples 11 to 17 comprises a multilayer perceptron operation.

[0244] Example 19 includes a system comprising: a processor and a computer-readable medium comprising instructions that, when executed by the processor, cause the processor to perform operations comprising: receiving a neural network model comprising a first tensor operation; dividing the first tensor operation into sub-operations of the first tensor operation, the sub-operations comprising two sub-operations with no data dependency between the two sub-operations; assigning a first of the two sub-operations to a first computing engine; assigning a second of the two sub-operations to a second computing engine; and generating instructions for executing the first sub-operation by the first computing engine and the second sub-operation by the second computing engine in parallel.

[0245] In Example 20, the first tensor operation of the system of Example 19 includes a multilayer perceptron operation.

[0246] In Example 21, the first tensor operation of the system of any one of Examples 19 to 20 uses a set of input feature maps and a set of filters to generate an output tensor comprising a set of output feature maps.

[0247] Example 22 includes the system of Example 21, wherein: the set of input feature maps includes C input feature maps for each of the N input data sets; the set of filters includes M filters, each including C filter matrices; and the set of output feature maps includes M output feature maps for each of the N input data sets, wherein each of the M output feature maps for a certain input data set is the result of a convolution between the C input feature maps of the input data set and the C filter matrices in a certain filter of the M filters.

[0248] In Example 23, the first computing engine of the system of Example 22 includes a two-dimensional processing element array, characterized in that: the total number of columns of the two-dimensional processing element array is less than M; or the total number of rows of the two-dimensional processing element array is less than C.

[0249] In Example 24, the first sub-operation of the system of any one of Examples 21 to 23 generates a portion of the set of output feature maps.

[0250] Example 25 includes the system of Example 24, wherein the portion of the set of output feature maps comprises at least one of: a portion of a total number of output feature maps in the set of output feature maps; a portion of each output feature map in at least a subset of the set of output feature maps; or a portion of each output feature map in a portion of the total number of output feature maps in the set of output feature maps.

[0251] Example 26 includes the system of any of Examples 24 to 25, wherein the first sub-operation generates the portion of the set of output feature maps using at least one of: a portion of the set of input feature maps and a total number of filters in the set of filters; a portion of the set of filters and each input feature map in at least a subset of the set of input feature maps; or a portion of the total number of filters in the set of filters and a portion of each input feature map in a subset of the set of input feature maps.

[0252] In Example 27, the operation of the system of any one of Examples 19 to 26 further includes generating instructions for performing inference based on a result of the first sub-operation, a result of the second sub-operation, or both.

[0253] In Example 28, the operation of the system of any one of Examples 19 to 27 further includes: identifying a second tensor operation in the neural network model, the second tensor operation and the first tensor operation having no data dependency; dividing the second tensor operation into sub-operations of the second tensor operation, the sub-operations of the second tensor operation having no data dependency; assigning a first sub-operation of the sub-operations of the second tensor operation to a third computing engine; assigning a second sub-operation of the sub-operations of the second tensor operation to a fourth computing engine; and generating instructions for executing the first sub-operation of the sub-operations of the second tensor operation by the third computing engine and executing the second sub-operation of the sub-operations of the second tensor operation by the fourth computing engine in parallel with the first sub-operation of the first tensor operation and the second sub-operation of the first tensor operation.

Claims

1. A computer-implemented method, comprising: receiving a neural network model including a first tensor operation; dividing the first tensor operation into sub-operations of the first tensor operation, the sub-operations comprising two sub-operations, with no data dependency between the two sub-operations; assigning a first of the two sub-operations to a first accelerator, the first accelerator comprising a first array of processing elements; assigning a second of the two sub-operations to a second accelerator, the second accelerator comprising a second array of processing elements; as well as Instructions are generated for executing the first sub-operation by the first accelerator and executing the second sub-operation by the second accelerator in parallel.

2. The computer-implemented method of claim 1 , wherein the first tensor operation comprises a multilayer perceptron operation.

3. The computer-implemented method of any one of claims 1 to 2, wherein the first tensor operation uses a set of input feature maps and a set of filters to generate an output tensor comprising a set of output feature maps.

4. The computer-implemented method of claim 3, wherein: The set of input feature maps includes C input feature maps for each of the N input data sets; The set of filters comprises M filters each comprising C filter matrices; and The set of output feature maps includes M output feature maps for each of the N input data sets, wherein each of the M output feature maps for a certain input data set is the result of a convolution between the C input feature maps of the input data set and the C filter matrices in a certain filter among the M filters.

5. The computer-implemented method of claim 4 , wherein the first accelerator comprises a two-dimensional array of processing elements, wherein: The total number of columns of the two-dimensional processing element array is less than M; or The total number of rows of the two-dimensional processing element array is less than C.

6. The computer-implemented method of claim 3, wherein the first sub-operation generates a portion of the set of output feature maps.

7. The computer-implemented method of claim 6 , wherein the portion of the set of output feature maps comprises at least one of: A portion of the total number of output feature maps in the set of output feature maps; a portion of each output feature map in at least a subset of the set of output feature maps; or A portion of each output feature map in a portion of a total number of output feature maps in the set of output feature maps.

8. The computer-implemented method of claim 6 , wherein the first sub-operation generates the portion of the set of output feature maps using at least one of: the set of input feature maps and a portion of the total number of filters in the set of filters; the set of filters and a portion of each input feature map in at least a subset of the set of input feature maps; or A portion of the total number of filters in the set of filters and a portion of each input feature map in a subset of the set of input feature maps.

9. The computer-implemented method of any one of claims 1 to 2, further comprising: Instructions are generated for performing inference based on a result of the first sub-operation, a result of the second sub-operation, or both.

10. The computer-implemented method of any one of claims 1 to 2, further comprising: identifying a second tensor operation in the neural network model, the second tensor operation having no data dependency with the first tensor operation; dividing the second tensor operation into sub-operations of the second tensor operation, the sub-operations of the second tensor operation having no data dependencies; assigning a first sub-operation of the sub-operations of the second tensor operation to a third accelerator; assigning a second sub-operation of the sub-operations of the second tensor operation to a fourth accelerator; and Instructions are generated for performing, by the third accelerator, the first of the sub-operations of the second tensor operation and by the fourth accelerator, the second of the sub-operations of the second tensor operation in parallel with the first sub-operation of the first tensor operation and the second sub-operation of the first tensor operation.

11. A non-transitory computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a neural network model including tensor operations; Dividing the tensor operation into sub-operations, the sub-operations comprising two sub-operations, with no data dependency between the two sub-operations; assigning a first of the two sub-operations to a first accelerator, the first accelerator comprising a first array of processing elements; assigning a second of the two sub-operations to a second accelerator, the second accelerator comprising a second array of processing elements; as well as Instructions are generated for executing the first sub-operation by the first accelerator and executing the second sub-operation by the second accelerator in parallel.

12. The non-transitory computer-readable medium of claim 11, wherein the instructions further cause the one or more processors to perform operations comprising: generating instructions for performing inference based on a result of the first sub-operation, a result of the second sub-operation, or both, The first accelerator and the second accelerator are in the same integrated circuit device or in two different integrated circuit devices.

13. The non-transitory computer-readable medium of claim 11 or 12, wherein: The tensor operation uses a set of input feature maps and a set of filters to generate an output tensor comprising a set of output feature maps; and The first sub-operation generates a portion of the set of output feature maps.

14. The non-transitory computer-readable medium of claim 13 , wherein the portion of the set of output feature maps comprises at least one of: A portion of the total number of output feature maps in the set of output feature maps; a portion of each output feature map in at least a subset of the set of output feature maps; or A portion of each output feature map in a portion of a total number of output feature maps in the set of output feature maps.

15. A system comprising: processor; and A computer-readable medium comprising instructions that, when executed by the processor, cause the processor to perform operations comprising: receiving a neural network model including a first tensor operation; dividing the first tensor operation into sub-operations of the first tensor operation, the sub-operations comprising two sub-operations, with no data dependency between the two sub-operations; assigning a first of the two sub-operations to a first accelerator, the first accelerator comprising a first array of processing elements; assigning a second of the two sub-operations to a second accelerator, the second accelerator comprising a second array of processing elements; as well as Instructions are generated for executing the first sub-operation by the first accelerator and executing the second sub-operation by the second accelerator in parallel.

Images