Efficient utilization of processing element arrays

Abstract

A computer-implemented method includes receiving a neural network model implemented using an array of processing elements, where the neural network model includes a convolution operation on a set of input feature maps and a set of filters. The method further includes determining, based on the neural network model, that the convolution operation uses less than a threshold number of rows in the array of processing elements to apply a set of filter elements to the set of input feature maps, where the set of filter elements includes one filter element in each filter of the set of filters. The method further includes generating, for the convolution operation and based on the neural network model, a first instruction and a second instruction for execution by respective rows in the array of processing elements, where the first instruction and the second instruction use different filter elements of a filter in the set of filters.

Description

[0001] Incorporate by reference

[0002] A request form is filed concurrently with this specification as part of this application. Each application identified in the concurrently filed request form that claims its benefit or priority is incorporated herein by reference in its entirety and for all purposes. Background Technology

[0003] Artificial neural networks (ANNs) are computational systems with an architecture based on biological neural networks. They can be trained using training data to learn how to perform a task, such as recognizing or classifying physical objects, activities, people, etc., from images or videos. An ANN can comprise multiple layers of processing nodes. Each processing node in a layer can perform computations on the input data generated by the processing nodes in the previous layer to produce output data. For example, a processing node can perform a set of arithmetic operations (such as multiplication and addition) to generate intermediate outputs or perform post-processing operations on intermediate outputs. The size of the data used in each layer (such as the size of the input data per input channel, the number of input channels, the number of weights to be applied to the input data, etc.) can vary from layer to layer. Therefore, the number of operations (e.g., matrix multiplication) and the size of the data used for each operation performed in each layer can vary from layer to layer. Attached Figure Description

[0004] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0005] Figure 1 An example of a multilayer artificial neural network is shown;

[0006] Figure 2 An example of a convolutional neural network (CNN) is shown;

[0007] Figure 3A and Figure 3B The example shows a convolution operation performed on an input pixel array through a convolutional layer in a convolutional neural network;

[0008] Figures 4A to 4E Examples of convolution, non-linear activation, and pooling operations performed on sample input pixel data are shown;

[0009] Figure 5 An example model of a convolutional layer in a convolutional neural network is shown;

[0010] Figure 6 An example of a convolution operation involving a batch (N=1)C input data channels and M groups including C filters is shown;

[0011] Figure 7This is a simplified block diagram illustrating an example of an integrated circuit device for performing neural network operations according to certain embodiments;

[0012] Figure 8 A simplified example of weighted fixed convolution is shown, according to some implementations, using a computational engine that includes an array of processing elements.

[0013] Figure 9A and Figure 9B An example is shown of loading multiple filter elements into a processing element array according to certain embodiments to utilize the processing element array more efficiently and to share input data between rows of the processing element array for processing using the loaded filter elements;

[0014] Figure 10 An example is shown of copying an input feature map read from memory and sending it to multiple rows of a processing element array, according to certain embodiments;

[0015] Figure 11 A simplified block diagram of an example of an input selector circuit, according to certain embodiments, for selecting input data for parallel processing by a processing element array using multiple filter elements;

[0016] Figure 12 Another example is shown of replicating input data according to certain implementations for parallel processing of a processing element array using multiple filter elements;

[0017] Figure 13 An example of a padded input feature map for a convolution operation is shown according to some implementation schemes;

[0018] Figure 14 This illustrates loading a first set of filter elements into a processing element array and sharing them between rows of the processing element array, according to certain embodiments. Figure 13 The first set of data in the input feature map is used as an example for parallel processing using loaded filter elements;

[0019] Figure 15 This illustrates loading a second set of filter elements into a processing element array and sharing them between rows of the processing element array, according to certain embodiments. Figure 13 The second set of data in the input feature map is used as an example of parallel processing using loaded filter elements;

[0020] Figure 16 The following are shown according to certain implementation schemes: Figure 13 An example of dividing the data in the input feature map into multiple smaller feature maps to perform smaller-sized matrix multiplications using an array of processing elements;

[0021] Figure 17A and Figure 17B An example is shown of loading multiple filter elements in a processing element array and sharing input data between rows of the processing element array to perform parallel processing using the loaded filter elements, according to certain embodiments.

[0022] Figure 18 This is a flowchart illustrating an example of a method for more efficiently utilizing an array of processing elements while reducing data transfer from memory, according to certain embodiments;

[0023] Figure 19 Includes a block diagram of an example host system; and

[0024] Figure 20 A block diagram including an example of an acceleration engine. Detailed Implementation

[0025] The techniques disclosed herein generally relate to artificial neural networks, and more specifically, to implementing artificial neural networks more efficiently using arrays of processing elements while reducing the amount of data transferred from memory. Artificial neural networks typically include multiple processing nodes arranged across two or more layers, where processing nodes in one layer can connect to processing nodes in another layer. Each processing node in a layer can receive a stream of input data elements, multiply each input data element by a weight, compute a weighted sum of the input data elements, and forward the weighted sum to the next layer. The amount of data used in each layer (such as the size of the input data per channel, the number of channels, the number of filters to be applied to the input data, the size of each filter, etc.) can vary from layer to layer. For example, in many neural networks, as the network becomes deeper, the number of channels may increase, while the size of each channel may decrease. Therefore, the number of arithmetic operations performed (e.g., matrix multiplication) and the amount of data used for each arithmetic operation in each layer can vary from layer to layer. The underlying hardware used to implement the neural network (such as a graphics processing unit (GPU) or array of processing elements) can typically have a certain number of processing elements (e.g., a pre-configured number of columns and / or rows) and limited memory space and / or bandwidth. Therefore, for some layers, it may not be possible to fully utilize the same underlying hardware to perform arithmetic operations efficiently. For example, the number of input channels in the first layer of a ResNet-50 network might be 3, while the number of rows in the processing element array might be much larger, such as, for example, 128. Therefore, for example, the utilization of the processing element array might be less than 3%.

[0026] According to some implementations, a compiler can compile a neural network model to generate instructions for more efficiently utilizing arrays of processing elements (PEs) for convolution operations using a small number of input channels. The compiler can generate instructions to load multiple filter elements of a filter into multiple rows of the PE array and copy data from the input feature map for use by multiple rows to simultaneously apply multiple filter elements to the input feature map. In some implementations, compilation can be performed at both the graph level and the tensor level. At the graph level, the compiler can identify convolution operations that may not efficiently utilize the PE array and add the following operations to the neural network model: padding the input feature map used by the convolution operation, partitioning the padded input feature map into smaller partitions, dividing the convolution operation into multiple smaller convolutions operating on the smaller partitions, and, for example, discarding some padding data based on the stride of the convolution. At the tensor level, the compiler can generate instructions for loading multiple filter elements of a filter into multiple rows of the PE array, copying input data read from memory for use by multiple rows, and discarding results generated using some padding data.

[0027] The techniques disclosed herein can improve the utilization of hardware systems used to implement neural networks, which may include convolutional operations using a small number of input channels. The techniques also reduce memory space and memory bandwidth required to store or transmit input data used by multiple rows of a PE array. Furthermore, the techniques disclosed herein can automatically identify operations that may be underutilizing the hardware system (e.g., the PE array) based on the neural network model and hardware system, divide such operations into multiple sub-operations that can be executed in parallel by the PE array, partition the input data into partitions for use by the sub-operations, and generate instructions for efficient execution by the hardware system to implement the neural network.

[0028] In the following description, various examples will be described. Specific configurations and details will be set forth for illustrative purposes to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the examples can be practiced even without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments. The accompanying drawings and descriptions are not intended to be limiting. The terms and expressions used in this disclosure are used as descriptive rather than limiting terms, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof. The word “example” is used herein to mean “served as an example, instance, or illustration.” Any embodiment or design described herein as an “example” is not necessarily to be construed as superior or better than other embodiments or designs.

[0029] Artificial neural networks (also known as "neural networks") are used in machine learning research and industrial applications, achieving many breakthroughs in areas such as image recognition, speech recognition, computer vision, and text processing. An artificial neural network can include multiple processing nodes arranged in two or more layers, where processing nodes in one layer can connect to processing nodes in another layer. Processing nodes can be divided into layers, for example, including an input layer, many intermediate layers (also called hidden layers), and an output layer. Each processing node in a layer (e.g., an input layer, intermediate layers, etc.) can receive a sequence of input data elements, multiply each input data element by a weight, calculate a weighted sum of the input data elements, and forward the weighted sum to the next layer. Artificial neural networks (such as convolutional neural networks) can include thousands or more processing nodes and millions or more weights and input data elements.

[0030] Figure 1 An example of a multilayer neural network 100 is shown. The multilayer neural network 100 may include an input layer 110, hidden (or intermediate) layers 120, and an output layer 130. In many implementations, the multilayer neural network 100 may include two or more hidden layers and may be referred to as a deep neural network. A neural network with a single hidden layer is often sufficient to model any continuous function. However, such a network may require an exponentially larger number of nodes compared to a neural network with multiple hidden layers. It has been shown that deeper neural networks can be trained to perform significantly better than relatively shallower networks.

[0031] Input layer 110 may include multiple input nodes (e.g., nodes 112, 114, and 116) that can provide information (e.g., input data) to the network from the outside. Input nodes can pass information to the next layer and may not perform any computation. Hidden layer 120 may include multiple nodes, such as nodes 122, 124, and 126. Nodes in hidden layers may not have a direct connection to the outside world (hence the name "hidden"). They can perform computations and pass information from input nodes to the next layer (e.g., another hidden layer or output layer 130). While a feedforward neural network may have a single input layer and a single output layer, it may have zero or more hidden layers. Output layer 130 may include multiple output nodes that are responsible for computationally processing information and passing it from the network to the outside world, such as recognizing certain objects or activities, or determining conditions or actions.

[0032] like Figure 1 As shown, in a feedforward neural network, a node (excluding bias nodes, if any) may be connected to all nodes (excluding bias nodes, if any) in the immediately preceding and immediately following layers. Therefore, such a layer can be called a fully connected layer. All connections between nodes can have weights associated with them, even if... Figure 1 This only shows some of these weights. For complex networks, there may be hundreds or thousands of nodes and thousands or millions of connections between them.

[0033] As mentioned above, a feedforward neural network can include zero (called a single-layer perceptron) or one or more hidden layers (called a multilayer perceptron (MLP)). Although Figure 1 Only a single hidden layer in a multilayer perceptron is shown, but a multilayer perceptron can include one or more hidden layers (plus an input layer and an output layer). Feedforward neural networks with many hidden layers can be called deep neural networks. While a single-layer perceptron can only learn linear functions, a multilayer perceptron can learn non-linear functions.

[0034] exist Figure 1 In the example shown, node 112 can be a bias node with a value of 1 or a regular input node. Nodes 114 and 116 can take external inputs X1 and X2, which can be numerical values ​​depending on the input dataset. As mentioned above, no computation is performed on input layer 110, so the outputs from nodes 112, 114, and 116 on input layer 110 are 1, X1, and X2, respectively, which are fed into hidden layer 120.

[0035] exist Figure 1 In the example shown, node 122 can be a bias node with a value of 1 or a regular network node. The outputs of nodes 124 and 126 in hidden layer 120 may depend on the output from input layer 110 (e.g., 1, X1, X2, etc.) and the weights associated with connection 115. For example, node 124 may take numeric inputs X1 and X2 and may have weights w1 and w2 associated with those inputs. Additionally, node 124 may have another input (called a bias), such as 1, with an associated weight w0. The primary function of the bias is to provide a trainable constant value for each node (along with the normal input the node receives). The bias value can allow the activation function to be shifted left or right. It should be noted that although... Figure 1 Only three inputs to node 124 are shown in the figure, but in various implementations, nodes can include dozens, hundreds, thousands or more inputs and associated weights.

[0036] The output Y from node 124 can be calculated in the following way:

[0037] Y = f(w1×X1 + w2×X2 + w0×bias), (1)

[0038] The function f can be a non-linear function, often referred to as an activation function. When a node has K inputs, the output from the node can be computed as follows:

[0039]

[0040] Therefore, the computation on each neural network layer can be described as the multiplication of the input matrix and the weight matrix, and the activation function applied to the product of the matrix multiplication. The output from the nodes in the intermediate layers can then be fed into the nodes in the next layer, such as the output layer 130.

[0041] Activation functions introduce non-linearity into the output of neural network nodes. An example of an activation function is the sigmoid function σ(x), which takes a real-valued input and transforms it to a value between 0 and 1. Another example is the tanh function, which takes a real-valued input and transforms it to a value in the range [-1, 1]. A third example is the rectified linear unit (ReLU) function, which takes a real-valued input and sets its threshold to be greater than 0 (e.g., replacing negative values ​​with 0). Another example activation function is the leaky ReLU function.

[0042] exist Figure 1 The output layer 130 in the example shown may include nodes 132 and 134, which can take input from the hidden layer 120 and perform computations similar to those of the hidden nodes using weights associated with connection 125. The computation results (Y1 and Y2) are the outputs of the multilayer perceptron. In some implementations, in MLPs used for classification, the Softmax function can be used as the activation function in the output layer. The Softmax function can take a vector of real-valued scores and map it to a vector of values ​​between 0 and 1 that sum to 1, for example, for object classification.

[0043] As described above, connections between nodes in adjacent layers of an artificial neural network have associated weights, which determine what the output vector is for a given input vector. The learning or training process assigns appropriate weights to these connections. In some implementations, the initial values ​​of the weights can be randomly assigned. For each input in the training dataset, the output of the artificial neural network can be observed and compared with the expected output, and the error between the expected output and the observed output can be propagated back to the previous layer. The weights can be adjusted accordingly based on the error. This process is repeated until the output error is below a predetermined threshold.

[0044] In many cases, such as due to the size of the input data and the number of weights to be trained and applied, using feedforward neural networks as described above for real-world applications like image classification may be impractical. One way to overcome these problems is to use convolutional neural networks, which perform convolutions using smaller convolutional filters instead of the large matrix multiplications described above. When performing convolutions, the same filters can be used at many locations on the image. Learning a set of convolutional filters (e.g., a 7×7 matrix) is likely much easier and faster than learning the large weight matrix of a fully connected layer.

[0045] For example, a convolutional neural network (ConvNet or CNN) can perform operations including: (1) convolution; (2) nonlinear (or activation) functions (e.g., ReLU); (3) pooling or quadratic sampling; and (4) classification. Different CNNs may have different combinations of these four main operations, as well as other additional operations. For example, a ResNet-50 network may include network layers that mainly consist of convolutional layers and several pooling layers, and may also perform residual addition operations for residual learning.

[0046] Figure 2 An example of a convolutional neural network (CNN) 200 for classifying images or other objects is shown. As described above, the CNN 200 can perform four types of operations, including convolution, non-linear (or activation) functions (e.g., ReLU), pooling or quadratic sampling, and classification (fully connected layers). The object 210 to be classified, such as one or more input images or other input datasets (called input feature maps), can be represented by a matrix of pixel values. For example, object 210 can include multiple channels (e.g., multiple input feature maps), each channel representing a component of object 210. For example, an image from a digital camera can have at least red, green, and blue channels, where each channel can be represented by a 2-D pixel matrix with pixel values, for example, ranging from 0 to 255 (i.e., 8 bits). A grayscale image may have only one channel. The following description describes the processing of a single image channel using the CNN 200. Other channels can be processed in a similar manner.

[0047] like Figure 2As shown, object 210 (e.g., input image) can first be processed by a first convolutional layer 215 using a first set of filters, wherein the first convolutional layer 215 can perform convolution between a matrix representing the input image and a matrix representing each filter in the first set of filters. Convolution can include multiple matrix multiplications. The first convolutional layer 215 can also perform a non-linear activation function (e.g., ReLU). The output matrix 220 from the first convolutional layer 215 can have a smaller size than the input image. The first convolutional layer 215 can perform convolution on the input image using the first set of filters to generate multiple output matrices 220, which can be referred to as output feature maps of the first convolutional layer 215. The number of filters used can be referred to as the depth of the convolutional layer. Figure 2 In the example shown, the depth of the first convolutional layer 215 can be 3. Each output matrix 220 (e.g., an output feature map) can be passed to a pooling layer 225, where each output matrix 220 can be subsampled or downsampled to generate matrix 230.

[0048] Each matrix 230 can be processed by the second convolutional layer 235 using a second set of filters. As described above, a non-linear activation function (e.g., ReLU) can also be performed by the second convolutional layer 235. The output matrix 240 (e.g., output feature map) from the second convolutional layer 235 can have a smaller size than matrix 230. The second convolutional layer 235 can perform convolution on matrix 230 using the second set of filters to generate multiple output matrices 240. Figure 2 In the example shown, the depth of the second convolutional layer 235 can be 6. Each output matrix 240 can be passed to the pooling layer 245, where each output matrix 240 can be double-sampled or downsampled to generate an output matrix 250.

[0049] The output matrix 250 from pooling layer 245 can be flattened into a vector by flattening layer 255 and passed through fully connected layer 260 (e.g., multilayer perceptron (MLP)). Fully connected layer 260 can include input layer 270 that obtains a 2-D output vector from flattening layer 255. Fully connected layer 260 can also include hidden layers and output layer 290. Fully connected layer 260 can use feature maps or output matrix 250 and, for example, a softmax function to classify objects in the input image into one of several categories. Operations on fully connected layers can be represented using matrix multiplication. For example, if there are M nodes on input layer 270 and N nodes on hidden layer 280, and the weights of the connections between the M nodes on input layer 270 and the N nodes on hidden layer 280 can be represented by a matrix W with M×N elements, then the output Y of hidden layer 280 can be determined by Y = X×W.

[0050] Convolution operations in CNNs can be used to extract features from input images. Convolution operations preserve the spatial relationships between pixels by extracting image features using small regions of the input image. In convolution, matrices (called filters, kernels, or feature detectors) may slide across the input image (or feature map) with a certain stride (called stride). For each location (or step), element-wise multiplications between the filter matrix and the overlapping matrix in the input image can be computed and summed to generate a final value representing a single element of the output matrix (e.g., a feature map). Filters can be used to detect certain features in the original input image.

[0051] A convolution using one filter (or a filter bank) on an input pixel array can generate a feature map, and a convolution using another filter (or another filter bank) on the same input pixel array can generate different feature maps. In practice, CNNs may learn the filter weights themselves during training based on user-specified parameters (called hyperparameters) such as the number of filters, filter size, and network architecture. The more filters used, the more image features can be extracted, and the better the network's ability to recognize patterns in new images is likely to be.

[0052] The size of the output feature map can be determined based on parameters such as depth, stride, and zero-padding. As mentioned above, depth can correspond to the number of filters (or filter banks) used for the convolution operation. For example, in Figure 2 In the CNN200 shown, three different filters are used in the first convolutional layer 215 to perform convolution operations on the input image, thereby generating three different output matrices (or feature maps) 220. The stride is the number of pixels the filter matrix slides across the input pixel array. For example, with a stride of 1, the filter matrix moves one pixel at a time. With a stride of 2, the filter matrix moves two pixels at a time. A larger stride may result in smaller feature maps. In some implementations, the input matrix can be padded with zeros around the boundaries so that the filter matrix can be applied to the boundary elements of the input pixel array. Zero padding allows control over the size of the feature maps.

[0053] like Figure 2 As shown, an additional non-linear operation using an activation function (e.g., ReLU) can be applied after each convolutional operation. ReLU is an element-wise operation that replaces all negative pixel values ​​in the feature map with 0. The purpose of the ReLU operation is to introduce non-linearity into the CNN. Other non-linear functions such as tanh or sigmoid can also be used, but ReLU has been found to perform better in many cases.

[0054] Spatial pooling (also known as quadratic sampling or downsampling) reduces the size of each feature map while preserving the most important information. In particular, pooling makes feature sizes smaller and more manageable, and reduces the number of parameters and computations in the network. Spatial pooling can be performed in different ways, such as max pooling, average pooling, sum pooling, etc. In max pooling, the spatial neighborhood is represented using the maximum element in each spatial neighborhood (e.g., a 2×2 window). Instead of taking the maximum element, the spatial neighborhood can be represented using the average (for average pooling) or sum (for sum pooling) of all elements in each window. In many applications, max pooling may perform better than other pooling techniques.

[0055] exist Figure 2 The example shown uses two sets of convolutional and pooling layers. It should be noted that these operations can be repeated any number of times within a single CNN. Furthermore, a pooling layer may not be used after every convolutional layer. For example, in some implementations, the CNN may perform multiple convolutional and ReLU operations before the pooling operation.

[0056] The training process for a convolutional neural network (such as CNN 200) can be similar to that of any feedforward neural network. First, all parameters and weights (including weights in filters and weights in fully connected layers) can be initialized with random values ​​(or with known parameters of the neural network). Second, the convolutional neural network takes training samples (e.g., training images) as input, performs forward propagation steps (including convolution, non-linear activation and pooling operations, and forward propagation operations in fully connected layers), and determines the output probability for each possible class. Since the parameters of the convolutional neural network (such as weights) are randomly assigned to the training samples, the output probabilities may also be randomized.

[0057] At the end of the training process, all the weights and parameters of the CNN have likely been optimized to correctly classify the training samples from the training dataset. When unseen samples (e.g., test samples or new samples) are fed into the CNN, the CNN can use the trained weights and parameters to complete the forward propagation step and output the probability of each class; this can be called the inference (or prediction) process compared to the training process. If the training dataset is sufficient, the trained network can classify unseen samples into the correct category.

[0058] Figure 3A and Figure 3B The diagram illustrates a convolutional operation performed by a convolutional layer in a convolutional neural network on an input pixel array 320 using filter 310. The input pixel array 320 may include an input image, channels of the input image, or feature maps generated by another convolutional or pooling layer. Figure 3AThe convolution operation performed on the first region 322 of the input pixel array 320 at the first step is shown. Figure 3B The diagram illustrates the convolution operation performed on the second region 324 of the input pixel array 320 at the second step after the filter 310 is slid by one step.

[0059] Filter 310 may comprise a two-dimensional matrix, where each element of the 2-D matrix represents a weight. The weights in filter 310 can be designed or trained to detect or extract certain features from the spatial distribution of pixel values ​​in an image. The extracted features may or may not be meaningful to the human eye. Different filters can be used to detect or extract different features from the input pixel array. For example, some filters can be used to detect edges in an image, or to sharpen or blur the image. Filter 310 may have R rows (height) and S columns (width), and is typically smaller than the input pixel array 320, which may have a height of H pixels and a width of W pixels. Each weight in filter 310 can be mapped to a pixel in a region of R rows and S columns within the input pixel array 320. For example, as... Figure 3A As shown, a convolutional layer (e.g., a first convolutional layer 215 or a second convolutional layer 235) or a processing node of a convolutional layer can receive pixel values ​​from region 322 (including R×S pixels) of the input pixel array 320, perform element-wise multiplication between the filter 310 and corresponding elements in region 322, and sum the products of the element-wise multiplications to generate a convolutional output value 332. In other words, according to That is, the dot product between the matrix W of filter 310 and the matrix X representing the pixel values ​​of region 322, and the convolution output value 332 can be the sum of the multiplication results between the weights in filter 310 and the corresponding pixels in region 322.

[0060] Similarly, such as Figure 3B As shown, the convolutional layer (e.g., another processing node of the convolutional layer) can receive pixel values ​​from region 324 (including R×S pixels) of the input pixel array 320, perform element-wise multiplication between the filter 310 and corresponding elements in region 324, and sum the products of the element-wise multiplications to generate a convolutional output value 334. Figure 3B As shown, convolution operations can be performed with a predetermined stride D using a sliding window approach. The stride is the number of pixels the filter matrix slides across the input pixel array. For example, in... Figure 3B In the example shown, region 324 can be located at a distance D (in pixels) from region 322, and the next region for the next convolution operation can be located at the same distance D from region 324. The stride D can be less than or greater than the width S of filter 310.

[0061] The output of the convolution operation can form a convolution output matrix 330 with a row height of E and a column width of F. As mentioned above, matrix 330 can be referred to as a feature map. The size of matrix 330 can be smaller than the input pixel array 320 and can be determined based on the size of the input pixel array 320, the size of the filter 310, and the stride D. As mentioned above, in some implementations, the input pixel array 320 can be padded with zeros around its boundaries so that the filter 310 can be applied to the boundary elements of the input pixel array 320. Zero padding allows control over the size of the feature map (e.g., matrix 330). When the padding size on each side of the 2-D input pixel array 320 is P, the height E of matrix 330 is... And the width F of matrix 330 is For example, if the stride D is equal to one pixel in both the horizontal and vertical directions, then E can be equal to H-R+2P+1, and F can be equal to W-S+2P+1. A larger stride D may generate a smaller feature map.

[0062] Figures 4A to 4E Examples of convolution, non-linear activation, and pooling operations performed on sample input pixel data are shown. For example, the input pixel data could represent a digital image, channels of a digital image, or a feature map generated by a previous layer in a convolutional neural network. Figure 4A An example input matrix 410 including example input pixel data is shown. The input matrix 410 may include a 6×6 pixel array, wherein each element of the pixel array may include a real number, such as an integer or a floating-point number. Figure 4B An example filter 420 is shown. Filter 420 may include a 3×3 matrix, where each element of the matrix represents a filter weight. Filter 420 can be used to extract certain features from the input matrix 410. For example, Figure 4B The example filter 420 shown can be a filter used to detect edges in an image.

[0063] The input matrix 410 and the filter 420 can be convolved to generate, for example... Figure 4CThe output matrix 430 is shown. Each element in the output matrix 430 can be the sum of element-wise multiplications (e.g., dot products) between corresponding elements in the overlapping region 412 of the filter 420 and the input matrix 410, and can be determined in each step of sliding a window with the same size as the filter 420 (e.g., 3×3) over the input matrix 410 in a certain stride (e.g., one element in the horizontal and / or vertical direction). For example, the value of element 432 in the first row and third column of the output matrix 430 can be the dot product between the matrix representing the filter 420 and the matrix representing the region 412 of the input matrix 410, where 2×0+1×1+0×0+5×1+3×(-4)+2×1+2×0+1×1+1×0=1+5-12+2+1=-3. Similarly, the value of element 434 in the 4th row and 1st column of output matrix 430 can be the dot product between the matrix representing filter 420 and the matrix representing region 414 of input matrix 410, where 0×0+2×1+1×0+0×1+0×(-4)+1×1+5×0+3×1+2×0=2+1+3=6. For input matrix 410 with a 6×6 pixel array and filter 420 represented by a 3×3 matrix, output matrix 430 can be a 4×4 matrix when the stride used is one element or pixel.

[0064] Nonlinear activation functions (e.g., ReLU, sigmoid, tanh, etc.) can then be applied to the output matrix 430 to generate, for example, ... Figure 4D The matrix shown is 440. Figure 4D In the example shown, the ReLU function is used, so all negative values ​​in output matrix 430 are replaced with 0s in matrix 440. Pooling operations (e.g., max, average, or sum pooling operations) can be applied to matrix 440 to subsample or downsample the data in matrix 440. Figure 4D and Figure 4E In the example shown, max pooling can be applied to matrix 440, where the 4×4 matrix 440 can be divided into four 2×2 regions 442, 444, 446, and 448. The maximum value of each region can be chosen as the subsample representing each region. For example, the maximum value 9 is selected from region 442, the maximum value 2 from region 444, the maximum value 5 from region 446, and the maximum value 6 from region 448. Therefore, after convolution, non-linear activation, and pooling operations, a feature map 450 with four elements 9, 2, 6, and 5 can be generated from the 6×6 input matrix 410.

[0065] Figure 5An example model 500 of a convolutional layer in a convolutional neural network, for example, for image processing, is shown. As shown in the example, the convolutional layer can have multiple (e.g., N) 3-D inputs 520-1, ..., and 520-N. Each 3-D input can include C channels of a 2-D input feature map (of size H×W). For example, for the first convolutional layer in a CNN (such as ResNet-50), the 3-D input can include three channels of a 2-D image, such as red, green, and blue channels. Multiple (e.g., M) 3-D filters 510-1, ..., and 510-M (each with C 2-D filters of size R×S) can be convolved with N 3-D inputs 520-1, ..., and 520-N (e.g., N batches of C input feature maps of size H×W) to generate multiple (e.g., N) 3-D outputs 530-1, ..., and 530-N, where each of the 3-D outputs 530-1, ..., and 530-N can include M output feature maps (also called output channels). Each 3-D filter 510-1, ..., or 510-M (of size C×R×S) can be applied to the 3-D inputs 520-1, ..., or 520-N (of size C×H×W) to generate output feature maps (of size E×F, as described above) in the 3-D outputs 530-1, ..., or 530-N, which include M output feature maps. Figure 3A and Figure 3B Therefore, M 3D filters can be used to generate M output feature maps in 3D outputs 530-1, ..., or 530-N for 3D inputs 520-1, ..., or 520-N. For example, a 3D filter 510-1 can be applied to a 3D input 520-1 to generate output feature maps 530-1-1, ..., and a 3D filter 510-M can be applied to a 3D input 520-1 to generate output feature map 530-1-M. The same M 3D filters 510-1, ..., and 510-M can be applied to each 3D input 520-1, ..., or 520-N to generate each corresponding 3D output 530-1, ..., or 530-N comprising M output feature maps. For example, a 3-D filter 510-1 can be applied to the 3-D input 520-N to generate an output feature map 530-N-1, and a 3-D filter 510-M can be applied to the 3-D input 520-N to generate an output feature map 530-NM. Therefore, there are N 3-D inputs and N 3-D outputs, where each 3-D output includes M output feature maps.

[0066] More specifically, such as Figure 5As shown, for 3-D inputs 520-1, ..., or 520-N and 3-D filters 510-1, ..., or 510-M, the C 2-D filters (each filter having a size of R×S) in the 3-D filters 510-m can correspond to the C channels of the 2-D input feature maps (each input feature map having a size of H×W) in the 3-D inputs, and a convolution operation can be performed between each of the C 2-D filters and the corresponding channel in the C channels of the 2-D input feature map. The convolution results of C pairs of 2-D filters and their corresponding 2-D input feature maps can be summed to generate a convolutional output (e.g., a pixel) O on the output feature map with index m in the M output feature maps of the 3-D outputs 530-1, ..., or 530-N. e，f m As shown below:

[0067]

[0068] Where m corresponds to the index of the output feature map and the index of the 3D filter among the M 3D filters. X c eD+r，fD+s W is the value of the pixel with horizontal pixel coordinates eD+r and vertical pixel coordinates fD+s in the C channels of the 2D input feature map of the 3D input. D is the sliding window stride distance. e and f are the coordinates of the output pixels in the corresponding output feature maps of the M output feature maps, and may correspond to a specific sliding window. r and s correspond to specific positions (e.g., pixels or elements) within the sliding window or 2D filter. c，m r，s These are the weights corresponding to the pixels at position (r, s) of the 2D filter with index C in the 3D filter with index m. Equation (3) shows that in order to compute the weights of each convolutional output (e.g., pixel) at position (e, f) on the output feature map m, e，f m Each pixel X within a sliding window of the input feature map indexed C can be... c eD+r，fD+s With corresponding weight W c，m r，s Multiplying to generate a product allows us to calculate the partial sum of the products of pixels within each sliding window in the input feature maps with index C. Subsequently, we can calculate the sum of the partial sums of all C input feature maps to determine the pixel O at position (e, f) in the corresponding output feature map with index m in the M output feature maps. e，f m The value of .

[0069] In one example, for a 3-D filter 510-1 and a 3-D input 520-1, each of the C 2-D filters 512 in the 3-D filter 510-1 can correspond to a corresponding input feature map 522 in the 3-D input 520-1 and can be used to convolve (e.g., filter) with the corresponding input feature map 522, wherein each pixel in the sliding window 524 in the input feature map 522 can be multiplied with the corresponding pixel in the 2-D filter 512 to generate a product, and the products of all pixels in the sliding window 524 can be summed to generate a partial sum. The partial sums of the C 2-D filters 512 (and their corresponding input feature maps 522) can be added together to generate an output pixel 532 at position (e, f) on the output feature map 530-1-1 in the 3-D output 530-1. The sliding window 524 can be shifted across all C input feature maps 522 in the 3-D input 520-1 based on a stride D in both dimensions to generate another output pixel 532 at a different location on the output feature map 530-1-1 in the 3-D output 530-1. The sliding window 524 can be repeatedly shifted together across all C input feature maps 522 until all output pixels 532 on the output feature map 530-1-1 in the 3-D output 530-1 are generated.

[0070] Each 3-D filter 510-2, ..., or 510-M can be convolved with the 3-D input 520-1, as described above regarding 3-D filter 510-1, to generate each corresponding output feature map 530-1-2, ..., or 530-1-M in the 3-D output 530-1. Similarly, each 3-D filter 510-1, ..., or 510-M can be convolved with the 3-D input 520-N, as described above regarding 3-D filter 510-1 and 3-D input 520-1, to generate each corresponding output feature map 530-N-1, ..., or 530-NM in the 3-D output 530-N.

[0071] Figure 6 An example of a convolution operation involving a batch (N=1)C input data with 620 channels (C=3) and M groups (M=2)C filters (C=3) is shown. Figure 6 The example shown could be about Figure 5A specific example of model 500 is described, where the batch number N is 1. As shown in the figure, input data 620 includes three input feature maps 622, 622, and 624 (e.g., input channels), each corresponding to an input channel. The filters include a first set of filters 610-1 and a second set of filters 610-2, wherein the first set of filters 610-1 may include three 2-D filters 612-1, 614-1, and 616-1, and the second set of filters 610-2 may include three 2-D filters 612-2, 614-2, and 616-2.

[0072] Each 2-D filter 612-1, 614-1, or 616-1 in the first set of filters 610-1 can be convolved with the corresponding input feature maps 622, 622, or 624, and the results of the convolutions of the three input feature maps can be summed to generate output feature map 630-1 in output feature map 630. For example, a pixel in filter 612-1 can be multiplied with the corresponding pixel in window 622-1 on input feature map 622, and the products can be summed to generate a first partial sum. A pixel in filter 614-1 can be multiplied with the corresponding pixel in window 624-1 on input feature map 624, and the products can be summed to generate a second partial sum. A pixel in filter 616-1 can be multiplied with the corresponding pixel in window 626-1 on input feature map 626, and the products can be summed to generate a third partial sum. The first, second, and third partial sums can be added together to generate output pixel 632-1 on output feature map 630-1. Other output pixels on the output feature map 630-1 can be generated in the same way by shifting the window or filter together on the input feature map.

[0073] Similarly, each 2-D filter 612-2, 614-2, or 616-2 in the second set of filters 610-2 can be convolved with the corresponding input feature maps 622, 622, or 624, and the results of the convolutions of the three input feature maps can be summed to generate output feature map 630-2 in output feature map 630. For example, a pixel in filter 612-2 can be multiplied with the corresponding pixel in window 622-1 on input feature map 622, and the products can be summed to generate a first partial sum. A pixel in filter 614-2 can be multiplied with the corresponding pixel in window 624-1 on input feature map 624, and the products can be summed to generate a second partial sum. A pixel in filter 616-2 can be multiplied with the corresponding pixel in window 626-1 on input feature map 626, and the products can be summed to generate a third partial sum. The first, second, and third partial sums can be added together to generate output pixel 632-2 on output feature map 630-2. Other output pixels on the output feature map 630-2 can be generated in the same way by shifting the window or filter together on the input feature map.

[0074] As illustrated in the models discussed above, the operation of a neural network (e.g., performing inference) typically involves acquiring input data or input activations, performing multiplication and accumulation operations in parallel for each node in a layer, and providing output activations. When the hardware architecture is capable of highly parallel computation, optimal performance of the neural network, measured in response time, can be achieved. Dedicated or domain-specific neural network processors can achieve better performance than both CPUs and GPUs when executing neural networks. Neural network processors can employ a spatial architecture including arrays of processing elements (PEs), where processing elements can form processing chains and data can be directly passed from one processing element to another. This can significantly reduce the number of memory transactions. In some examples, weights or inputs can be preloaded into the array of processing elements. In some examples, the neural network processor can also include on-chip buffers that can store values ​​read from processor memory and can distribute values ​​to multiple computation engines within the processor. Computation engines can also include small local register files (e.g., small memories) for storing intermediate results. Having an on-chip memory hierarchy can improve the operational efficiency of neural networks by reducing memory latency.

[0075] Figure 7 This is a block diagram illustrating an example of an integrated circuit device for performing neural network operations (such as tensor operations) according to certain embodiments. Figure 7The example shown includes accelerator 702. In various examples, accelerator 702 may use processing element array 710, activation engine 716, and / or pooling engine 718 to perform computations on a set of input data (e.g., input data 750). In some examples, accelerator 702 may be an integrated circuit component of a processor, such as a neural network processor. The processor may have other integrated circuit components, including additional accelerator engines.

[0076] In some implementations, accelerator 702 may include a memory subsystem 704 (e.g., a status buffer) comprising a plurality of memory banks 714. Each memory bank 714 can be accessed independently, such that a read of one memory bank is independent of a read of another. Similarly, a write to one memory bank may not affect or limit writes to different memory banks. In some cases, each memory bank can be read and written simultaneously. Various techniques can be used to achieve independently accessible memory banks 714. For example, each memory bank may be a physically separate memory component whose address space is separate from and independent of the address spaces of each other memory bank. In this example, each memory bank may have at least one read channel and may have at least one separate write channel that can be used simultaneously. In these examples, memory subsystem 704 may allow simultaneous access to the read or write channels of multiple memory banks. Furthermore, memory subsystem 704 may include arbitration logic such that arbitration, for example, between the outputs of the plurality of memory banks 714, may result in the use of the outputs of more than one memory bank. In these and other examples, although globally managed by the memory subsystem 704, each memory bank can operate independently of any other memory bank.

[0077] Making memory bank 714 independently accessible can improve the efficiency of accelerator 702. For example, values ​​can be read and provided simultaneously for each row of processing element array 710, allowing the entire processing element array 710 to be used within a single clock cycle. Similarly, memory bank 714 can be read while the results of computations performed by processing element array 710 are written to memory subsystem 704. In contrast, a single memory may only be able to serve one read or write at a time. For example, with a single memory, multiple clock cycles might be required to read input data for each row of processing element array 710 before it can be started.

[0078] In various implementations, the memory subsystem 704 can be configured to serve multiple clients simultaneously, including the processing element array 710, the activation engine 716, the pooling engine 718, and any external clients accessing the memory subsystem 704 via the communication texture 720. In some implementations, the ability to serve multiple clients may mean that the memory subsystem 704 has at least as many memory banks as the number of clients. In some cases, each row of the processing element array 710 can be considered a separate client. In some cases, each column of the processing element array 710 can output results, making each column a separate write client. In some cases, output from the processing element array 710 can be written to memory bank 714, which can then provide input data to the processing element array 710. Furthermore, the activation engine 716 and the pooling engine 718 may include multiple execution channels, each of which can be a separate memory client. For example, memory bank 714 can be implemented using static random access memory (SRAM).

[0079] In various implementations, the memory subsystem 704 may include control logic. For example, the control logic may track the address space of each of the memory banks 714, identify which memory bank 714 to read from or write to, and / or move data between memory banks 714. In some implementations, the memory banks 714 may be hardwired to specific clients. For example, a set of memory banks 714 may be hardwired to provide values ​​to rows of the processing element array 710, with one memory bank serving each row. Alternatively, a set of memory banks may be hardwired to receive values ​​from columns of the processing element array 710, with one memory bank receiving data from each column.

[0080] According to some implementations, accelerator 702 may include input selector circuitry 730. Input selector circuitry 730 can be used to determine the data to be sent to processing element array 710 in any given clock cycle. In some examples, input selector circuitry 730 can control the data input to each row of processing element array 710. In some examples, input selector circuitry 730 can control the data input to a subset of rows. In various examples, for a given row, input selector circuitry 730 can select between data output from memory subsystem 704 and data that has already been selected for input to a different row. For example, input selector circuitry 730 can determine to input data from memory subsystem 704 to row 0 of processing element array 710, while for row 1 of processing element array 710, input selector circuitry 730 can determine to use the data input to row 0 (e.g., after a delay) instead of reading data from memory subsystem 704 again. In other words, the same data read from memory subsystem 704 can be provided to more than one row of processing element array 710. In some implementations, the input selector circuit 730 may be configured such that it can be bypassed or that data copying is not performed, and thus each row of the processing element array 710 may receive data from the memory subsystem 704.

[0081] The processing element array 710 is the computation matrix of the accelerator 702. For example, the processing element array 710 can perform parallel integration, convolution, correlation, and / or matrix multiplication, etc. The processing element array 710 may include multiple processing elements 711 arranged in rows and columns, such that the output of one processing element 711 can be directly input into another processing element 711. Processing elements 711 not located on the outer edge of the processing element array 710 can therefore receive data from other processing elements 711 instead of from the memory subsystem 704 for operation.

[0082] In various examples, the processing element array 710 uses pulsation execution, where data arrives at each processing element 711 from different directions at regular intervals. In some examples, input data can flow into the processing element array 710 from the left and weight values ​​can be loaded at the top. In some examples, weights and input data can flow from the left, while partial sums can flow from top to bottom. In these and other examples, multiplication and accumulation operations move diagonally across the processing element array 710 as wavefronts, where data moves right and down across the array. Control signals can be input simultaneously with the weights on the left and can flow laterally and downwards with the computation.

[0083] In various implementations, the number of columns and rows in the processing element array 710 determines its computational power. For example, the number of rows in the processing element array 710 determines the number of input feature maps that can be processed in parallel, and the number of columns in the processing element array 710 determines the number of filter banks that can be applied to the input data in parallel. The number of rows in the processing element array 710 also determines the memory bandwidth required to achieve maximum utilization of the processing element array 710. For example, the processing element array 710 may have 64 columns and 128 rows, or some other number of columns and rows.

[0084] An example of processing element 711 is shown in Figure 7 As illustrated in the illustration, processing element 711 may include multiplier-accumulator circuitry, as shown in this example. For example, the input on the left may include input data i and weight values ​​w, where the input data is a set of values ​​taken from a set of input data or a set of intermediate results, and the weight values ​​are a set of weights that connect one layer of a neural network to the next. For example, the set of input data may be an image submitted for recognition or object recognition, an audio clip provided for speech recognition, a text string used for natural language processing or machine translation, or the current state of a game that needs to be analyzed to determine the next action. In some examples, the input data and weight values ​​are output to the right side for input to the next processing element 711.

[0085] In the example shown, the input from above can include a partial sum p_in provided from another processing element 711 or from a previous round of computation in the processing element array 710. When starting to compute a new set of input data, the top row of the processing element array 710 can receive a fixed value for p_in, such as 0. As shown in this example, i and w are multiplied, and the result is summed with p_in to generate a new partial sum p_out, which can be input to another processing element 711. Various other implementations of the processing element 711 are also possible.

[0086] The output from the last row in the processing element array 710 can be temporarily stored in the result buffer 712 (e.g., a partial sum (PSUM) buffer). The result can be an intermediate result, which can be written to memory 714 to be provided to the processing element array 710 for additional computation. Alternatively, the result can be a final result, which, once written to memory 714, can be read from memory subsystem 704 via communication texture 720 for system output.

[0087] In some implementations, accelerator 702 includes activation engine 716. In these implementations, activation engine 716 can combine results from processing element array 710 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other examples, it may be necessary to accumulate results from one or more columns in processing element array 710 to produce an output activation for a single node in the neural network. In some examples, activation engine 716 can be bypassed.

[0088] In various examples, the activation engine 716 may include multiple individual execution channels. In these examples, the execution channels may correspond to columns of the processing element array 710 and can perform operations on the outputs of the columns, the results of which may be stored in the memory subsystem 704. In these examples, the activation engine 716 may be able to perform 1 to N parallel computations, where N equals the number of columns in the processing element array 710. In some cases, one or more computations may be performed simultaneously. Examples of computations that each execution channel can perform include exponentiation, square, square root, identity, binary step, bipolar step, sigmoidal, and ramp, among others.

[0089] In some implementations, accelerator 702 may include a pooling engine 718. Pooling is the combination of the outputs of columns from processing element array 710. For example, combination may include calculating maximum, minimum, average, median, summation, multiplication, or other logical or mathematical combinations. In various examples, pooling engine 718 may include multiple execution channels capable of operating on values ​​from corresponding columns of processing element array 710. In these examples, pooling engine 718 may be able to perform 1 to N parallel computations, where N equals the number of columns in processing element array 710. In various examples, the execution channels of pooling engine 718 may operate in parallel and / or simultaneously. In some examples, pooling engine 718 may be bypassed.

[0090] In this paper, the activation engine 716 and the pooling engine 718 can be collectively referred to as the execution engine. The processing element array 710 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine, which may be located outside the accelerator 702.

[0091] Input data 750 can be reached via communication texture 720. Communication texture 720 can connect accelerator 702 to other components of the processor, such as a DMA engine that can obtain input data 750 from input / output (I / O) devices, memory drives, or network interfaces. For example, input data 750 can be one-dimensional data (such as strings or sequences of numbers) or two-dimensional data (such as an array of pixel values ​​of an image or frequency and amplitude values ​​of an audio signal that change over time). In some examples, input data 750 can be three-dimensional, for example, contextual information used by self-driving cars or virtual reality data. In some implementations, memory subsystem 704 may include a separate buffer for input data 750. In some implementations, input data 750 can be stored in memory bank 714 when accelerator 702 receives it.

[0092] In some examples, accelerator 702 can implement a neural network processing engine. In these examples, given a set of input data 750, accelerator 702 can execute a neural network to perform a task, for which the neural network is trained. Executing a neural network on a set of input data can be referred to as inference or performing inference.

[0093] The weights of the neural network can be stored in the memory subsystem 704 along with the input data 750 that the neural network will operate on. The neural network may also include instructions that can program the processing element array 710 to perform various calculations on the weights and input data. These instructions may also be stored in the memory subsystem 704, memory bank 714, or a separate instruction buffer. The processing element array 710 can output intermediate results representing the outputs of individual layers of the neural network. In some cases, the activation engine 716 and / or pooling engine 718 can be enabled for computations required by certain layers of the neural network. The accelerator 702 can store the intermediate results in the memory subsystem 704 for use as input to the processing element array 710 to compute the results of the next layer of the neural network. The processing element array 710 can also output the final result from the last layer of the neural network. The final result can be stored in the memory subsystem 704 and subsequently copied to the host processor memory or another location.

[0094] In some implementation schemes, the above regarding Figure 5 and Figure 6The tensor operations described in equation (3) mapped to the PE array (e.g., PE array 710) for execution may include mapping each of the M 3-D filters to a corresponding column of the PE array, and mapping each of the C input feature maps (e.g., C channels) of the 3-D input to a corresponding row of the PE array. For example, H×W pixels in each 2-D input feature map can be flattened to form a one-dimensional vector and mapped to a row of the PE array. C×R×S weights or pixels in each 3-D filter can be flattened to form a one-dimensional vector and mapped to a column of the PE array. Partial sums can be vertically accumulated in each column. In the case of processing batches containing N 3-D inputs (each containing C channels), each row of the PE array can be mapped to N 2-D input feature maps.

[0095] As described above, moving data (such as input pixels, filter weights, and partial sums to be accumulated) between PEs can reduce access to the state buffer or off-chip memory. In some implementations, the input feature map can be fixed and the filter weights can be shifted; this can be referred to as an "image-fixed" model. In some implementations, a "weight-fixed" model can be used, where the filter weights are fixed (preloaded from the state buffer into registers in the PE array) and the image is moved (loaded from the state buffer during computation) to minimize the cost of weight movement. In some implementations, the output of the PE can be stored in registers at the PE and kept fixed to minimize the cost of moving the partial sum, where the input feature map and weights can be moved through the PE array and the state buffer.

[0096] Figure 8 A simplified example of a weighted convolution operation using a computational engine including a processing element array 810, according to certain embodiments, is shown. The processing element array 810 may include a large number of processing elements arranged, for example, a 64×64 array, a 64×128 array, a 128×128 array, a 256×256 array, etc. Figure 8 In the example shown, the processing element array 810 includes a four-row, four-column processing element array 812. The inputs 820 to the processing element array 810 may include four (corresponding to C) input channels 822, 824, 826, and 828. Each input channel may correspond to an input feature map or one of each of the N inputs (N=1 in this example) as described above. Each input feature map in the example may include an 8×8 matrix and may be flattened into a one-dimensional vector with 64 elements. The PE array 810 can generate four (corresponding to M) output feature maps, one output feature map derived from each column of the PE array 810.

[0097] During the convolution operation, the weights of each of the four 2D filters (R×S) out of each of the four 3D filters (C×R×S) can be preloaded into the PE array 810. For example, as Figure 8 As shown, for each of the four 2D filters of the first output feature map (corresponding to the first 3D filter or m=0), the first element (r=0, s=0) can be loaded into the corresponding PE 812 of the four PEs in the first column of the PE array 810; for each of the four 2D filters of the second output feature map (corresponding to the second 3D filter or m=1), the first element (r=0, s=0) can be loaded into the corresponding PE 812 of the four PEs in the second column of the PE array 810; for each of the four 2D filters of the third output feature map (corresponding to the third 3D filter or m=2), the first element (r=0, s=0) can be loaded into the corresponding PE 812 of the four PEs in the third column of the PE array 810; and for each of the four 2D filters of the fourth output feature map (corresponding to the fourth 3D filter or m=3), the first element (r=0, s=0) can be loaded into the corresponding PE 812 of the four PEs in the fourth column of the PE array 810. Therefore, the 16 values ​​representing the first elements of the 16 2D filters out of the four 3D filters are loaded into the PE array 810. Then, the elements in the one-dimensional vector for each input feature map can be shifted from the state buffer into the PE array 810, and multiplied with the preloaded weights in the PE array 810. The multiplications for each column of the four channels 822, 824, 826, and 828 can be accumulated to generate four partial sum values. As the elements in the one-dimensional vector for each input feature map are shifted into the PE array 810, a first partial sum vector PSUM can be generated. 0,0 (830), which may include four parts and subvectors of four output feature maps. In some implementations, the shifting of elements in the input feature maps may be based on the desired stride of the convolution, such that each part and subvector of the output feature maps may include a desired number of elements (e.g., E×F, as described above).

[0098] The sixteen (16) values ​​representing the second elements (e.g., r = 0, s = 1) of the sixteen 2-D filters out of the four 3-D filters can then be loaded into the PE array 810. Elements in the one-dimensional vector used for each input feature map can be shifted into the PE array 810 and multiplied with the preloaded weights in the PE array 810. The multiplications in each column can be summed to generate the second part sum vector PSUM. 0,1(832), which comprises four parts and sub-vectors of the four output feature maps. Each element of the 16 2-D filters can be loaded into the PE array 810 and multiplied by the elements of a one-dimensional vector to generate a part sum vector comprising the four parts and sub-values ​​of the four output feature maps, up to generating the part sum vector PSUM. R-1,S-1 (834), which corresponds to the element (R-1, S-1) in each 2-D filter and includes four parts and sub-vectors of the four output feature maps. The parts and sub-vectors can be PSUM. 0,0 (830), PSUM 0,1 (832), ..., and PSUM R-1,S-1 In (834), the partial and sub-vectors corresponding to each corresponding output feature map are accumulated to generate corresponding vectors 840, 842, 844 or 846 that can correspond to the flattened output feature map.

[0099] like Figure 5 , Figure 6 and Figure 8 As shown in Equation 3, the processing element array can perform parallel computations using different columns, which can correspond to different filters or different filter banks. The processing element array can also use columns and rows to perform fused multiplication and addition operations to reduce data in dimensions such as input channels, filter height, and filter width. In many convolutional neural networks, the number of channels can increase with depth. Many convolutional layers can have more than 128 input channels, which may be large enough to utilize all rows of the processing element array. However, the number of input channels in the first or early layers of a CNN (such as CNNs used for image recognition or classification) typically includes 3 channels for the R, G, and B components of a color image. For example, the first convolutional layer of ResNet-50 can use three input feature maps (e.g., each 224×224 in size) and can use small filters (e.g., each 7×7 in size) and a stride of 2×2. The number of input channels or input feature maps (e.g., 3) may be too small compared to the number of rows in a PE array (e.g., 128). Therefore, when using only one filter element from each filter simultaneously, the utilization of the PE array can be very low, at least for the first layer of a CNN. Low utilization of the processing element array used in the first or early layers can significantly reduce the overall utilization of the processing element array for much inference.

[0100] According to some implementations, convolutional operations in neural network layers with a small number of input channels can be performed using multiple weighted convolutional operations. This allows the input feature map, or a portion thereof, to be sequentially input into multiple rows, and multiple filter elements of the same filter to be simultaneously loaded into multiple rows of the processing element array for application to the same input channel map or the same portion of the input feature map, thereby improving the utilization of the processing element array. To avoid having more than one copy of the input feature map in memory (e.g., memory subsystem 704) and / or to reduce the data transfer bandwidth required to move the input feature map from memory (to the processing element array (e.g., processing element array 710)), multiple rows of the processing element array can share input data. For example, input data can be read once from memory and copied by input selector circuitry 730 to multiple rows of the processing element array 710, improving the utilization of the processing element array 710 while reducing memory bandwidth usage for data transfer.

[0101] In some implementations, for example, when the stride of the convolution is greater than 1, the input data can be divided into multiple blocks to be loaded into different rows. In some implementations, the compiler can identify neural network layers (which may utilize a small portion of the array of processing elements in an accelerator), divide the convolution operations in each of these network layers into multiple convolution operations, prepare (e.g., pad and divide) the input data, generate a new data flow graph, generate data sharing instructions, and generate instructions for operations to be performed on the convolution results, such as discarding certain output data.

[0102] Figure 9A and Figure 9B Examples are shown, according to certain embodiments, of loading multiple filter elements into a processing element array to utilize the array more efficiently and sharing input data between rows of the array for processing using the loaded filter elements. Figure 9AAs shown, the convolution operation performed by the accelerator 910, which includes the PE array 920, can utilize C channels of a 2-D input feature map 930 (each input feature map being H×W) and a filter 940 comprising M sets of filters, where each set of filters can include C 2-D filters (each filter being R×S). Each input feature map 930 can be flattened into a vector comprising H×W pixel values, where each pixel value in the input feature map can be mapped to a corresponding row in the PE array 920. The filter 940 can be flattened into M channels, each channel comprising C×R×S weight values, where each of the M channels can be mapped to a corresponding column in the PE array 920. During the convolution operation, one filter element from each of the C 2-D filters in the set can be loaded into the row of the corresponding column in the M columns, and the pixel values ​​from each input feature map can be sequentially sent to the processing elements in the corresponding rows of the PE array 920 to perform the fusion multiplication and addition operations as described above. Therefore, C rows of the PE array 920 can be used simultaneously during the same time period. Then, another filter element from each of the C 2D filters can be loaded into the corresponding row of the M columns, and data from each input feature map can be sequentially sent to the processing element in the corresponding row of the PE array 920. Thus, the same C rows of the PE array 920 can be used simultaneously during the same time period. In this way, each filter element can be loaded into the PE array 920 and applied to the input feature map to generate M output feature maps 950. In the example shown in Figure 9, C may be much smaller than the number of rows in the PE array 920. Therefore, the convolution operation can utilize only a small portion of the PE array 920 at any given time.

[0103] according to Figure 9BIn some embodiments shown, multiple (e.g., 4) filter elements from each of a set of C-channel 2-D filters can be loaded into multiple (e.g., 4) rows of corresponding columns in M ​​columns. Pixel values ​​in each input feature map can be copied and sequentially sent to processing elements in multiple (e.g., 4) corresponding rows in the PE array 920 to perform fused multiplication and addition operations. Therefore, 4×C rows of the PE array 920 can be used simultaneously. Subsequently, another multiple (e.g., 4) filter elements from each of the C-channel 2-D filters can be loaded into multiple (e.g., 4) rows of corresponding columns in M ​​columns. Pixel values ​​in each input feature map can be copied and sequentially sent to processing elements in multiple corresponding rows in the PE array 920 to perform fused multiplication and addition operations. Therefore, 4×C rows of the PE array 920 can be used simultaneously. In this way, each filter element in the C channels of the 2-D filter can be loaded into the PE array 920 and applied to the input feature map to generate M output feature maps 950. Therefore, multiple rows of the PE array 920 can be used simultaneously to perform convolution operations more quickly.

[0104] Figure 10 This includes a diagram 1000 illustrating an example of copying each input feature map read from memory and sending it to multiple rows of a processing element array according to certain embodiments. Figure 10 Examples of input data for rows in the PE array 920 can be shown. For clarity, only input data for a few rows of the PE array is shown; input data for other rows can be copied and sent in a similar or different manner and from similar or different sources (e.g., memory devices or storage banks). Figure 10 As an example, the input matrix 410 in Figure 4 is used as an example of input data for layers in a neural network. Figure 1000 shows the data obtained for three input channels (e.g., the R, G, and B channels of the input image). Different input channels (e.g., represented by input feature maps) in... Figure 10 Different crosshairs are used to indicate this. Figure 10 In this format, time is indicated from right to left, so that data that enters the array of processing elements earlier can be in the right column, and data that enters the array of processing elements later can be in the left column.

[0105] exist Figure 10In the example diagram 1000 shown, the first column indicates the row number k (1011) of the PE array, and the remaining columns indicate the elements (e.g., pixels) of the corresponding rows in the input feature map to be supplied to the PE array. In this example, the elements of the input feature map (e.g., input matrix 410) are numbered from left to right and from top to bottom in raster scan order. For example, the element in the upper left corner of input matrix 410 is element [0], the element in the upper right corner is element [5], the element in the lower left corner is element

[30] , and the element in the lower right corner is element

[35] . Also in this example, the elements in the input feature map are shifted from left to right into the PE array such that the elements in the right column enter the processing element array before the elements in the left column. In some examples, each column may correspond to a clock cycle and may indicate data input to the processing element array at different clock cycles. Figure 10 In the example shown, the numbers in parentheses indicate channel indices, such as channel 1, channel 2, or channel 3. The first three rows of the PE array can be loaded with filter elements [0,0] for each of the three filters, where each filter can correspond to an input channel. Rows [3] through [5] can be loaded with filter elements [0,1] for each filter corresponding to the corresponding input channel among the three input channels. Rows [6] through [8] can be loaded with filter elements [0,2] for each filter corresponding to the corresponding input channel among the three input channels.

[0106] Line [0] can receive data from a first input feature map and apply filter elements [0,0] of a first filter for the first input feature map to the received data. Line [1] can receive data from a second input feature map and apply filter elements [0,0] of a second filter for the second input feature map to the received data. Line [2] can receive data from a third input feature map and apply filter elements [0,0] of a third filter for the third input feature map to the received data. In one example, lines [0], [1], and [2] can receive data from a local buffer memory (such as memory subsystem 704).

[0107] Furthermore, in this example, row [3] can also receive data from the first input feature map and can apply filter elements [0,1] of the first filter to the received data. For this row, instead of obtaining the data from the first input feature map from the local buffer memory, the input selector circuitry described above and below can be configured to route the data provided to row [0] to row [3]. Figure 10As shown in the example, filter elements [0,1] are not applied to each element [0] of the input feature map, so the input selector circuit can copy the input data only when reading the element [1] of the first input feature map from the buffer memory. In addition, in order to properly accumulate the data into the output feature map, the data sent to row [3] may be delayed (e.g., delayed by two clock cycles) compared to the data sent to row [0].

[0108] Row [4] can similarly receive data from the second input feature map, starting from element [1] and delayed compared to the data sent to row [1]. For example, the input selector circuit can similarly obtain the data in row [4] by routing the data input to row [1] to row [4] with a two-clock-cycle delay. The data input to row [4] can be multiplied by the filter elements [0,1] of the second filter. Row [5] can similarly receive data from the third input feature map, starting from element [1] and delayed compared to the data sent to row [2]. For example, the input selector circuit can obtain the data in row [5] from the data input to row [2] with a two-clock-cycle delay. The data input to row [5] can be multiplied by the filter elements [0,1] of the third filter.

[0109] Row [6] may also receive data from the first input feature map. In row [6], the third filter element [0,2] of the first filter may be applied to the first input feature map, starting from element [2] of the first input feature map. For row [6], the input selector circuit may obtain input data from the data sent to row [0] or from the data sent to row [3]. For example, when the input selector circuit is able to review at least six rows from row [6], the input selector circuit may use the data sent to row [0] or row [3]. When the input selector circuit is able to review up to three rows, the input selector circuit may use the data sent to row [3]. Similarly, row [7] may be supplied with data obtained from the data sent to row [1] or row [4] from the second input feature map, and the filter element [0,2] of the second filter may be applied to the supplied data. Row [8] may be supplied with data obtained from the data sent to row [2] or row [5] from the third input feature map, and the filter element [0,2] of the third filter may be applied to the supplied data.

[0110] For example, when performing convolution computations on the first layer of a neural network, the example shown in Figure 1000 can be used, where the stride is 1 in both the width and height directions. As mentioned above, the input to the first layer can include three or more input feature maps, and applying filters to these input feature maps may only occupy three rows of the processing element array. By copying each of the three input feature maps to multiple rows of the processing element array and filtering them through multiple filter elements in each corresponding filter, more processing elements can be used when computing the result of the first layer. Subsequent layers may have more input feature maps. For these layers, the input selector circuitry may copy each input feature map fewer times, or it may disable data copying and select the output from the buffer memory. In some examples, for the various layers of the neural network, the input selector circuitry may select the output from the buffer memory for some rows and the copied data for other rows.

[0111] Figure 11 A simplified block diagram of an example of an input selector circuit 1100 (such as the input selector circuit 730 described above) for selecting input data for parallel processing by a processing element array using multiple filter elements, according to certain embodiments. In the illustrated example, the input selector circuit 1100 may include a first multiplexer 1132 and a second multiplexer 1136, the first multiplexer being configured to select input data for a row of the processing element array 1110, and the second multiplexer being configured to select an appropriate delay for inputting the selected data into a row of the processing element array 1110. The row selection 1142 sent to the input selector circuit 1130 may determine the data x'[k](1144) ultimately selected for said row. For clarity, only the multiplexer for a row (e.g., row[k](1111)) is shown, where k represents the row number. It should be noted that, with Figure 11 The similar circuitry shown can be used to select input data for other rows of the processing element array 1110. In some cases, other rows of the processing element array 1110 may each have similar input selector circuitry.

[0112] In various examples, the neural network processing engine can be configured such that a memory bank from the neural network processing engine's memory subsystem can provide input data to a row of the processing element array 1110. This arrangement allows data from an input feature map to be loaded into a memory bank, and subsequently, the input feature map data can be input from said memory bank into a row of the processing element array 1110. Figure 11In the example, the data output by the storage bank for row [k] (1111) is represented as x[k] (1106). For the storage bank output x[k] (1106) to be input into row [k] (1111), row selection 1142 can configure the first multiplexer 1132 to select the storage bank output x[k] (1106) as the output of the first multiplexer 1132.

[0113] In various examples, the first multiplexer 1132 can also select from other data 1140 selected for input to other rows of the processing element array 1110. For example, the other data 1140 may come from other selector circuits ( Figure 11 (Not explicitly shown) The outputs of other multiplexers in the array. For example, other data 1140 may include data selected for input to line [k-1] (e.g., the line preceding line [k](1111)). Figure 11 In the example, this is represented as x'[k-1]. Alternatively, other data 1140 may include data selected for input to row [k-2] (e.g., the two rows preceding row [k](1111)), denoted as x'[k-2]. Similarly, other data 1140 may include data selected for input to row [kn], denoted as x'[kn]. In some implementations, the input to the first multiplexer 1132 for row [k] may include data selected for input to rows [k-3], [k-6], [k-9], ..., and [k-3×n] when using three input channels (such as R, G, and B channels) in a convolution operation.

[0114] In some examples, it may be desirable to delay the data selected by the first multiplexer 1132 by a certain number of clock cycles. Because the processing element array 1110 can be a systolic array, the timing of loading data into the processing element array 1110 may affect the results calculated by the processing element array 1110. Therefore, in various implementations, the input selector circuit 1130 may include one or more delay circuits 1134a-1134n (such as flip-flops) that can delay the output of the first multiplexer 1132 by 1, 2, ..., and n clock cycles. The delay circuits 1134a-1134n can be chained together so that the delay circuit can further delay the output from the immediately preceding delay circuit. For example, the first delay circuit 1134a can delay the data by one clock cycle, the second delay circuit 1134b can further delay the data by one clock cycle, and so on. In some embodiments, the delay circuits 1134a-1134n may each have a different delay. For example, the first delay circuit 1134a can delay the data by one clock cycle, the second delay circuit 1134b can delay the data by two clock cycles, the third delay circuit can delay the data by four clock cycles, and so on. In some implementations, any of the delay circuits 1134a-1134n can be bypassed to achieve the desired total delay via a chain. The input selector circuit 1130 may include a second multiplexer 1136, which can be configured to select the desired delay at the output of the delay circuit. The output of the second multiplexer 1136, denoted as x'[k](1144), can be output by the input selector circuit 1130 as the input to row[k](1111).

[0115] In various examples, x'[k](1144) can also be input from other lines. This is in Figure 11 The example is shown by the dashed line 1150 from the output of the second multiplexer 1136. For example, x'[k](1144) can be the input to a multiplexer that selects the input data of row [k+1] and / or a multiplexer that selects the input data of row [k+2]. In these and other examples, x'[k](1144) can be input to more than one row of the processing element array 1110. In some examples, for example, x'[k](1144) can be selected to be input to up to eight rows below row [k](1111).

[0116] In various examples, the delay selected by the second multiplexer 1136 can be determined based on the data selected by the first multiplexer 1132. For example, as... Figure 10As shown in the example, when row selection 1142 selects x'[k-1], the second multiplexer 1136 can be configured to delay the data by one clock cycle. Similarly, when row selection 1142 selects x'[k-2], the second multiplexer 1136 can be configured to delay the data by two clock cycles. In these and other examples, row selection 1142 can be used to configure or control both the first multiplexer 1132 and the second multiplexer 1136. In some examples, row selection 1142 can be part of instructions for implementing a neural network. For example, row selection 1142 can be stored with filters and loaded into the memory subsystem of the processing engine along with filter elements (e.g., weights). In some examples, row selection 1142 can be determined by software operating the neural network processing engine. In some examples, row selection 1142 can vary depending on the different layers of the neural network. For example, for the first layer of the neural network, row selection 1142 can configure the first multiplexer 1132 to select from other data 1140 to copy previously read and used data, while for subsequent layers, row selection 1142 can select data from the memory output x[k](1106).

[0117] As previously described, the arrangement of multiplexers, delay circuits, and signals in the input selector circuit can be similar for the other rows of the processing element array 1110. In some examples, the input selector circuit 1130 may have different circuitry for some rows. For example, for row [0], the input selector circuit 1130 may only be used to select a memory output as an input to row [0]. In this example, the input selector circuit 1130 may only have delay circuitry between the memory output and the input to the processing element array 1110. As another example, for row [1], the input selector circuit 1130 may only have a memory output and the data selected for row [0] as possible selections for the input to row [1]. In this example, the circuitry for row [1] may include multiplexers with fewer inputs and fewer delay circuitry. In some examples, some rows may have fewer selections as possible inputs compared to other rows. For example, some rows may be configured to receive data provided to either of the first two rows, while other rows may be configured to receive data provided to either of the first eight rows. In some examples, the input selector circuit 1130 can be configured to select a portion of the input to another row of the processing element array 1110.

[0118] As mentioned above, in convolution operations, filters can slide across the input feature map with certain strides. For example, in... Figures 4A to 4C , Figure 8 and Figure 10In the example shown, the stride is 1, where filter element [0,0] can be multiplied by input feature map elements [0,0], [0,1], [0,2], ..., while filter element [0,1] can be multiplied by input feature map elements [0,1], [0,2], [0,3], ..., etc. In the first layer of ResNet-50, filters (e.g., each filter is 7×7 in size) might slide on the input feature map (e.g., each input feature map is 224×224 in size) with a stride of 2×2. Therefore, filter element [0,0] can be multiplied by input feature map elements [0,0], [0,2], [0,4], ..., while filter element [0,1] can be multiplied by input feature map elements [0,1], [0,3], [0,5], ..., etc. Thus, different subsets of the input feature map (e.g., at different starting points) can be sent to rows of PE arrays loaded with different filter elements.

[0119] Figure 12 Figure 1200 includes another example illustrating, according to certain implementations, the replication of input data for parallel processing by a processing element array using multiple filter elements. Figure 12 In the example shown, the step size in the width direction can be 2. For clarity, only data from rows [0] to

[11] is shown in this example, and it should be noted that the processing element array can have more rows (e.g., 64, 128, 256, or more). In Figure 1200, the first column indicates the row number k (1211), and the remaining columns indicate the elements of the corresponding rows in the input feature map to be supplied to the PE array. The elements of the input feature map are numbered from left to right and top to bottom in raster scan order. Figure 1200 shows the data obtained for the three input feature maps. The numbers in parentheses indicate the input channel numbers. Figure 12 Different crosshairs are also used to indicate different input feature maps. For example... Figure 10 As shown, the time is indicated from right to left, so that data that enters the array of processing elements earlier can be in the right column, while data that enters the array of processing elements later can be in the left column.

[0120] exist Figure 12In the example shown, both rows [0] and [3] receive data from the first input feature map, where row [0] receives even-numbered elements and row [3] receives odd-numbered elements. A first filter element [0,0] of the filter for the first input feature map is applied to the even-numbered elements, and a second filter element [0,1] of the filter for the first input feature map is applied to the odd-numbered elements. For example, at the first sliding window position, the first filter element [0,0] can be multiplied by the first element [0] of the first input feature map, while the second filter element [0,1] can be multiplied by the second element [1] of the first input feature map. At the second sliding window position, the first filter element [0,0] can be multiplied by the third element [2] of the first input feature map, while the second filter element [0,1] can be multiplied by the fourth element [3] of the first input feature map.

[0121] According to some implementations, in order to provide even-numbered elements and odd-numbered elements to different rows, software (e.g., operations in a compiler or neural network model) can partition and write data from the input feature map into two different banks of local buffer memory. An input selector circuit (e.g., input selector circuit 1130) as described above can select from the outputs of the two banks as inputs to rows [0] and [3] respectively. Through a mechanism of reading data from the buffer memory and / or by the input selector circuit, the input of data for row [3] (e.g., odd-numbered elements) can be delayed by several clock cycles relative to the data for row [0] (e.g., even-numbered elements).

[0122] The data from the second and third input feature maps can be similarly partitioned and written into two separate storage units to be sent to the corresponding two rows. Figure 12 In the example shown, even-numbered elements of the second input feature map can be input into row [1], and odd-numbered elements of the second input feature map can be input into row [4]. Even-numbered elements of the third input feature map can be input into row [2], and odd-numbered elements of the third input feature map can be input into row [5]. The first filter element [0,0] of the filter for the second and third input feature maps can be loaded into rows [1] and [2] respectively, and the second filter element [0,1] of the filter for the second and third input feature maps can be loaded into rows [4] and [5] respectively. Similar to rows [0] and [3], the data of rows [1], [2], [4] and [5] can be obtained directly from the memory, and the data entering these rows can be staggered in time by instructions read from the memory and / or read by the input selector circuit.

[0123] like Figure 12As shown, applying filter elements [0,0] and [0,1] to the three input feature maps may only use six rows of the processing element array. To apply more filter elements to the three input feature maps, the input selector circuit can use data read from memory and input into rows [0]-[5] to copy data for other rows. For example, even-numbered elements of the first input feature map can be copied from the data input into row [0] and input into row [6], and odd-numbered elements can be copied from the data input into row [3] and input into row [9]. A third filter element [0,2] for the filter of the first input feature map can be loaded into row [6], and a fourth filter element [0,3] for the filter of the first input feature map can be loaded into row [9]. To enable data from the first input feature map to be input into rows [6] and [9], the input selector circuit can select data to be input into rows [0] and [3] respectively for these rows, and apply appropriate delays before sending the data to rows [6] and [9]. For example, for row [6], since the third filter element [0,2] is first applied to the input element [2] of the first input feature map, the input selector circuit can activate the data routing from the data provided to row [0] when the input element [2] is available. Furthermore, for example, the input selector circuit can use the aforementioned delay circuit to delay the input of data into the higher-numbered row.

[0124] The data from the second and third input feature maps can be similarly copied from the data in rows [1], [2], [4], and [5] and input into additional rows such as rows [7], [8],

[10] , and

[11] . In the example shown in Figure 1200, even-numbered elements from the second feature map can be copied from the data in row [1] and input into row [7], and odd-numbered elements can be copied from the data in row [4] and input into row

[10] . For the third input feature map, even-numbered elements can be copied from the data in row [2] and input into row [8], and odd-numbered elements can be copied from the data in row [5] and input into row

[11] . Rows [7] and [8] can be loaded with third filter elements [0,2] for the filters of the second and third input feature maps, respectively, to be applied to the input elements in the second and third input feature maps. Similarly, rows

[10] and

[11] can be loaded with third filter elements [0,3] for the filters of the second and third input feature maps, respectively, to be applied to the input elements in the second and third input feature maps. Copying data from the input to the data in the lower-numbered rows, rather than copying data directly from memory, can significantly reduce the overhead of reading data from memory, especially when the data in the input feature maps is input into many (such as dozens or more) rows in the array of processing elements.

[0125] In some implementations, due to specific hardware configurations, each row of the PE array can use the same size of data (e.g., the same number of input elements, such as pixels) for each convolution operation. However, as described above, convolutions using sliding windows and data duplication indicate that input shifting may occur between rows, where different rows may use input data at different start positions and therefore different end positions. Furthermore, in some implementations, for efficient memory storage, direct memory access (DMA) operations, memory reads, etc., it may be desirable to have the loaded data of a specific length, such as a multiple of 8 bytes. Therefore, the input data (e.g., input feature maps) can be padded with padding data before the convolution operation, and some computation results computed using the padding data can be discarded after computation (e.g., multiplication).

[0126] Figure 13 An example of a padded input feature map 1300 for convolution operations according to certain embodiments is shown. The input feature map 1300 may include the original input feature map 1310 and padded input feature elements. Figure 13 In the example shown, the original input feature map 1310 can be an input channel of ResNet-50, such as the R, G, or B channels of an input color image, and can include, for example, 224×224 input elements (e.g., pixels). For example, the original input feature map 1310 can be convolved using a 7×7 filter or kernel. As described above, additional padding input feature elements can be added to the input feature map 1300 and outside the original input feature map 1310 due to specific hardware configurations, filter sizes, strides, memory alignment conditions, data copying to reduce memory bandwidth usage, etc., as described above and in more detail below. Figure 13 The input elements in the input feature diagram 1300 shown are numbered in raster scan order as described above.

[0127] In the first window 1320 (7×7) of the input feature map 1300, input elements can be multiplied by corresponding filter elements in the 7×7 filter, and the products can be summed. For example, filter elements [0,0], [0,1], [0,2], [0,3], [0,4], [0,5], and [0,6] can be multiplied by input elements [0], [1], [2], [3], [4], [5], and [6], respectively. Filter elements [6,0], [6,1], [6,2], [6,3], [6,4], [6,5], and [6,6] can be multiplied by input elements

[1392] ,

[1393] ,

[1394] ,

[1395] ,

[1396] ,

[1397] , and

[1398] , respectively.

[0128] In different windows 1330 (7×7) of the input feature map 1300, input elements can be multiplied by corresponding filter elements in the 7×7 filter, and the products can be summed. For example, filter element [0,0] can be multiplied by input element

[222] , filter element [0,1] can be multiplied by input element

[223] , ..., filter element [0,6] can be multiplied by input element

[228] , and so on. Filter element [6,0] can be multiplied by input element

[1602] , filter element [6,1] can be multiplied by input element

[1603] , ..., and filter element [6,6] can be multiplied by input element

[1608] .

[0129] In the input feature map 1300, padding input elements can be added so that each row of the PE array can receive the same number of input elements for the input wavelet. For example, input elements [0], [2], and [4] can be used by filter elements [0,0], [0,2], and [0,4], but not by filter element [0,6], while input element

[228] cannot be used by filter elements [0,0], [0,2], and [0,4], but can be used by filter element [0,6]. Padding input elements can be added so that the data input to each row can be loaded and stored in memory more efficiently and / or read from memory more efficiently. For example, it may be desirable that the size of the input elements input to each row of the PE array in the input wavelet is an integer multiple of 8 bytes. Input elements [0], [2], [4], [6], ..., and

[228] can include 115 input elements or 230 bytes, which is not an integer multiple of 8 bytes. Input element 230 can be added to and input into row [0] of the PE array, such that the input elements of row [0] can include 116 input elements or 232 bytes, which is an integer multiple of 8 bytes.

[0130] Figure 14 Figure 1400 illustrates loading a first set of filter elements into a processing element array and sharing them between rows of the processing element array, according to certain embodiments. Figure 13 An example of parallel processing of the first set of data in the input feature map 1300 (e.g., data in some even-numbered rows) using loaded filter elements (e.g., weights). Figure 14 The example shown illustrates filter elements loaded into rows of a PE array and input elements shifted into rows of the PE array during a fixed-weight convolution operation. Figure 14 In the example shown, the input to the neural network layer can include three input channels, such as the R, G, and B channels of a color image. Figure 14The example shown could be an example of the first layer in a ResNet-50 network, where the input to the first layer consists of three 224×224 images (input feature maps). The filter for each image could be a 7×7 matrix, and the filter could slide across the corresponding input image with a 2×2 stride (e.g., skipping every other row and every other column). The original 224×224 input feature maps can be padded to generate, for example... Figure 13 The input feature map is shown in Figure 1300.

[0131] As shown in the figure, the filter element [0,0] for the three filters (e.g., the kernel) of channels 1-3 can be loaded into rows [0]-[2] respectively; the filter element [0,1] for the three filters of channels 1-3 can be loaded into rows [3]-[5] respectively; ...; and the filter element [0,6] for the three filters of channels 1-3 can be loaded into rows

[18] -

[20] respectively. Figure 14 In the first subwavelet shown, the input elements [0], [2], [4], ...,

[230] in each input image can be shifted to rows [0], [1], or [2] of the PE array and multiplied by the corresponding filter element [0,0]; the input elements [2], [4], [6], ...,

[230] , and

[464] in each input image can be shifted to rows [6], [7], or [8] of the PE array and multiplied by the corresponding filter element [0,2]; the input elements in each input image [4], [6], [8], ...,

[230] ,

[464] and

[466] can be shifted to rows

[12] ,

[13] or

[14] of the PE array and multiplied with the corresponding filter element [0,4]; and the input elements [6], [8],

[10] , ...,

[230] ,

[464] ,

[466] and

[468] in each input image can be shifted to rows

[18] ,

[19] or

[20] of the PE array and multiplied with the corresponding filter element [0,6]. Similarly, the input elements [1], [3], [5], ...,

[231] in each input image can be shifted to rows [3], [4], or [5] of the PE array and multiplied by the corresponding filter element [0,1]; the input elements [3], [5], [7], ...,

[231] , and

[465] in each input image can be shifted to rows [9],

[10] , or

[11] of the PE array and multiplied by the corresponding filter element [0,3]; and the input elements [5], [7], [9], ...,

[231] ,

[465] , and

[467] in each input image can be shifted to rows

[15] ,

[16] , or

[17] of the PE array and multiplied by the corresponding filter element [0,5].

[0132] Therefore, rows of the PE array can receive input elements starting from different indices, but can share many common input elements. For example, rows [0], [6],

[12] , and

[18] can share many even-numbered input elements in the first input feature map, while rows [3], [9], and

[15] can share many odd-numbered input elements in the first input feature map. In this way, the input elements in the first input feature map of the filter elements in the first row of the applied filter can be divided into two blocks and stored in two memory banks, wherein some even-numbered input elements in the first input feature map can be stored in memory bank 0 and input into rows [0], [6],

[12] , and

[18] , and some odd-numbered input elements in the first input feature map can be stored in memory bank 1 and input into rows [3], [9], and

[15] . As described above, data in memory bank 0 can be read and input into row [0] of the PE array, and can be copied and appropriately delayed, for example, by the input selector circuit described above, and then input into rows [6],

[12] , and

[18] . Similarly, data in storage 1 can be read and input into row [3] of the PE array, and can be copied and appropriately delayed by input selector circuitry before being input into rows [9] and

[15] .

[0133] In Figure 1400, the input element in box 1410 may not need to be multiplied with the corresponding filter element, such as... Figure 13As shown, the product of these input elements can therefore be discarded (e.g., not added to the sum). As described above, the input elements in the second to fourth columns 1414, 1416, and 1418 of box 1410 can be entered into the corresponding rows so that they can be copied for use in subsequent rows, and the inputs for different rows can have the same size. For example, input element

[224] may not need to be multiplied by filter element [0,0], but can be entered into row [0] so that it can be copied and used by rows [6],

[12] , or

[18] to calculate the partial sum; input element

[228] may not need to be multiplied by filter elements [0,0], [0,2], or [0,4] in rows [0], [2], or [4], but can be entered into row [0] so that it can be copied and used by row

[18] to calculate the partial sum. The input elements in the first column 1412 of box 1410 may not be used by any row, but are added so that the input elements in each row of each subwave can include an integer multiple of 8 bytes for more efficient memory storage and access (e.g., writing or reading). For example, input element 230 may not be used by any row [0]-

[20] , but can be included in the input elements of row [0], such that the input elements read from memory to be sent to row [0] in the subwave can include 116 (instead of 115) elements and 232 (instead of 230) bytes, where 232 is an integer multiple of 8.

[0134] Due to the 2×2 stride, some input elements of the input feature map 1300 (e.g., input elements in some odd-numbered rows) may not be multiplied by some filter elements in the filter (e.g., filter elements in some even-numbered rows), and some input elements of the input feature map 1300 (e.g., input elements in some even-numbered rows) may not be multiplied by some other filter elements in the filter (e.g., filter elements in some odd-numbered rows). For example, it may not be necessary to apply the filter elements in row [0] of the filter to the input elements in rows [1], [3], [5], etc. of the input feature map 1300, and it may not be necessary to apply the filter elements in row [1] of the filter to the input elements in rows [0], [2], [4], etc. of the input feature map 1300. Therefore, the additional partitions of the input feature map 1300 can be used to send input data to the rows of the PE array to apply some filter elements.

[0135] Figure 15 Figure 1500 illustrates loading a second set of filter elements into a processing element array and sharing them between rows of the processing element array, according to certain embodiments. Figure 13The second set of data in the input feature graph 1300 (e.g., data in some odd-numbered rows) is used as an example for parallel processing using loaded filter elements. Figure 15 The example shown illustrates filter elements loaded into rows of a PE array and input elements shifted into rows of the PE array during a fixed-weight convolution operation. Figure 15 In the example shown, the input to a layer of a neural network can include three input channels, such as the R, G, and B channels of a color image. Figure 15 The example shown could be an example of the first layer in a ResNet-50 network, where the input to the first layer consists of three 224×224 images (input feature maps). The filter for each image could be a 7×7 matrix, and the filter could slide across the corresponding input image with a 2×2 stride (e.g., skipping every other row and every other column). The original 224×224 input feature maps can be padded to generate, for example... Figure 13 The input feature map is shown in input feature map 1300. Depending on the number of rows in the PE array, the second set of filter elements can be loaded into rows [0]-

[20] after the first set of filter elements is applied to the input feature map, or the second set of filter elements can be loaded into rows

[21] -

[41] when the first set of filter elements is loaded into rows [0]-

[20] (if the PE array has 42 or more rows). In some implementations, some, but not all, of the second set of filter elements can be loaded into the PE array when the first set of filter elements is loaded into rows [0]-

[20] (if the PE array has more than 21 but less than 42 rows). Figure 15 In the example shown, the PE array may include 64, 128 or more rows, so the second set of filter elements can be loaded into rows

[21] -

[41] while the first set of filter elements is loaded into rows [0]-

[20] .

[0136] As shown in the figure, the filter elements [1,0] for the three filters (e.g., kernels) of channels 1-3 can be loaded into rows

[21] -

[23] respectively; the filter elements [1,1] for the three filters of channels 1-3 can be loaded into rows

[24] -

[26] respectively; ... and the filter elements [1,6] for the three filters of channels 1-3 can be loaded into rows

[39] -

[41] respectively. Figure 15In the first subwavelet shown, the input elements

[232] ,

[234] ,

[236] , ...,

[462] in each input image can be shifted to rows 21, 22, or 23 of the PE array and multiplied by the corresponding filter element [1,0]; the input elements

[234] ,

[236] ,

[238] , ...,

[462] , and

[696] in each input image can be shifted to rows 27, 28, or 29 of the PE array and multiplied by the corresponding filter element [1,2 ... Input elements

[236] ,

[238] ,

[240] , ...,

[462] ,

[696] , and

[698] can be shifted to rows 33, 34, or 35 of the PE array and multiplied by the corresponding filter element [1,4]; and input elements

[238] ,

[240] ,

[242] , ...,

[462] ,

[696] ,

[698] , and

[700] in each input image can be shifted to rows 38, 40, or 41 of the PE array and multiplied by the corresponding filter element [1,6]. Similarly, the input elements

[233] ,

[235] ,

[237] , ...,

[463] in each input image can be shifted to rows 24, 25, or 26 of the PE array and multiplied by the corresponding filter element [1,1]; the input elements

[235] ,

[237] ,

[239] , ...,

[463] , and

[697] in each input image can be shifted to rows 30, 31, or 32 of the PE array and multiplied by the corresponding filter element [1,3]; and the input elements

[237] ,

[239] ,

[241] , ...,

[463] ,

[697] , and

[699] in each input image can be shifted to rows 36, 37, or 38 of the PE array and multiplied by the corresponding filter element [1,5].

[0137] Therefore, the input data for the rows of the PE array can start from different input elements, but can share many common input elements. For example, rows

[21] ,

[27] ,

[33] and

[39] can share many even-numbered input elements in the first input feature map, while rows

[24] ,

[30] and

[36] can share many odd-numbered input elements in the first input feature map. In this way, the input elements in the first input feature map of the filter elements in the second row of the applied filter can be divided into two blocks and stored in two storage banks, where some even-numbered input elements in the first input feature map can be stored in storage bank 2 and input into rows

[21] ,

[27] ,

[33] and

[39] , and some odd-numbered input elements in the first input feature map can be stored in storage bank 3 and input into rows

[24] ,

[30] and

[36] . As described above, data in memory 2 can be read and input into line

[21] , and can also be copied and appropriately delayed by the input selector circuit described above before being input into lines

[27] ,

[33] and

[39] . Similarly, data in memory 3 can be read and input into line

[24] , and can also be copied and appropriately delayed by the input selector circuit before being input into lines

[30] and

[36] .

[0138] In Figure 1500, the input elements in box 1510 may be input elements that do not need to be multiplied with the corresponding filter elements and therefore the products of these input elements can be discarded (e.g., not added to the sum). As described above, the input elements in the second to fourth columns 1514, 1516, and 1518 in box 1510 can be entered into the corresponding rows so that they can be copied for use in subsequent rows, and the data entered into each row of the PE array can have the same size. For example, input element

[456] may not need to be multiplied with filter element [1,0], but is instead entered into row

[21] so that it can be copied and used by rows

[27] ,

[33] , or

[39] to calculate the partial sum; input element

[460] may not need to be multiplied with filter elements [1,0], [1,2], or [1,4] in rows

[21] ,

[27] , or

[33] , but is instead entered into row

[21] so that it can be copied and used by row

[39] to calculate the partial sum. The input elements in the first column 1512 of box 1510 may not be used by any row, but are added as described above for memory alignment.

[0139] Figure 16 Including Figure 1600, which illustrates the implementation of certain schemes. Figure 13 The data in the input feature map 1300 is divided into multiple smaller feature maps for smaller data processing using an array of processing elements. (See the example above regarding...) Figures 13 to 15 As shown, for example, Figure 16 The example shown can be used for the first layer of a ResNet-50 network, where the input to the first layer consists of three 224×224 images (input feature maps). The filter for each image can be a 7×7 matrix, and the filter can slide across the corresponding input image with a 2×2 stride (e.g., skipping every other row and every other column). The original 224×224 input feature maps can be padded to generate a result such as... Figure 13 The input feature map is shown as a 229×232 pixel image (image 1300). The input feature map can then be divided into four blocks and stored in four memory banks, as described above. Figure 14 and Figure 15 As described.

[0140] exist Figure 16 In the example shown, storage bank 0 may include input elements from even rows and even columns of the input feature map 1300, and may be shifted into rows of the PE array containing filter elements from even rows and even columns of the filter map. Storage bank 1 may include input elements from even rows and odd columns of the input feature map 1300, and may be loaded into rows of the PE array containing filter elements from even rows and odd columns of the filter map. Storage bank 2 may include input elements from odd rows and even columns of the input feature map 1300, and may be loaded into rows of the PE array containing filter elements from odd rows and even columns of the filter map. Storage bank 4 may include input elements from odd rows and odd columns of the input feature map 1300, and may be loaded into rows of the PE array containing filter elements from odd rows and odd columns of the filter map. Figures 14 to 16 As shown, each storage cell can include input elements for multiple wavelets.

[0141] Figure 17A and Figure 17B An example is shown, according to certain embodiments, of loading multiple filter elements into an array of processing elements and sharing input data between rows of the array of processing elements to perform parallel processing using the loaded filter elements. Figure 17AAs shown, the convolution operation performed by the accelerator 1710, which includes the PE array 1720, can utilize C channels of a 2-D input feature map 1730 (each input feature map being H×W) and a filter 1740 comprising M sets of filters, where each set of filters can include C 2-D filters (each filter being R×S), and each 2-D filter can be used to filter the corresponding 2-D input feature map 1730. Each input feature map 1730 can be flattened into a vector comprising H×W pixel values, where each pixel value in the input feature map 1730 can be mapped to a row in the PE array 1720. The filter 1740 can be flattened into M channels, each channel comprising C×R×S weight values, where each of the M channels can be mapped to a column in the PE array 1720. During the convolution operation, one element from each of the C 2D filters (a total of R×S elements) can be loaded into the corresponding row of the M columns, and the pixel values ​​from each input feature map can be sequentially sent to the processing element in the corresponding row of the PE array 1720 to perform the fusion multiplication and addition operations as described above. Therefore, C rows of the PE array 1720 can be used simultaneously. Afterwards, another element from each of the C channels of the 2D filters can be loaded into the corresponding row of the M columns, and the pixel values ​​from each input feature map can be sequentially sent to the processing element in the corresponding row of the PE array 1720. Therefore, the same C rows of the PE array 1720 can be used simultaneously. In this way, each filter element can be loaded into the PE array 1720 and applied to the input feature map to generate M output feature maps 1750. In the example shown in Figure 17, C may be much smaller than the number of rows in the PE array 1720. Therefore, the convolution operation may only utilize a small portion of the PE array 1720.

[0142] According to such Figure 17B Some implementations shown can populate pixel values ​​in each input feature map (e.g., original input feature map 1310) to generate an input feature map (e.g., input feature map 1300). For example, the input elements in the input feature map can be divided into multiple (e.g., N) partitions based on the stride used in the convolution. (See above regarding...) Figures 14 to 16 In one example, the stride size is 2×2, and each filled input feature map can be divided into four partitions 1732 and can be stored in four storage volumes.

[0143] During the convolution operation, multiple filter elements (e.g., K filter elements in one or more rows) from each of a set of C 2D filters can be loaded into multiple (e.g., K) rows in the corresponding column of M columns. Data can be read from one or more partitions 1732 in memory and input into the corresponding rows in the PE array 1720. For example, as Figure 14 As shown, data in the first partition 1732 can be read from memory (e.g., memory bank 0) and shifted to row [0] of PE array 1720 to apply filter element [0,0], and data in the second partition 1732 can be read from memory (e.g., memory bank 1) and shifted to row [3] of PE array 1720 to apply filter element [0,1]. For example, data input to row [0] of PE array 1720 can be copied by the input selector circuit described above and sent sequentially to multiple corresponding rows (e.g., rows [6],

[12] and

[18] , as shown) in PE array 1720. Figure 14 The processing elements in (shown) are used to perform fused multiplication and addition operations. For example, data input to row [3] of PE array 1720 can be copied by input selector circuitry and sent sequentially to multiple corresponding rows in PE array 1720 (e.g., rows [9] and

[15] , as shown). Figure 14 The processing elements (shown) perform fused multiplication and addition operations. Similarly, as... Figure 15 As shown, data in the third partition 1732 can be read from memory (e.g., memory bank 2) and shifted to rows (e.g., row

[21] ) of the PE array 1720 to apply filter elements (e.g., filter elements [1,0]), and data in the fourth partition 1732 can be read from memory (e.g., memory bank 3) and shifted to rows (e.g., row

[24] ) of the PE array 1720 to apply filter elements (e.g., filter elements [1,1]). For example, data input to row

[21] of the PE array 1720 can be copied by an input selector circuit and sent sequentially to multiple corresponding rows in the PE array 1720 (e.g., rows

[27] ,

[33] , and

[39] , as shown). Figure 15 The processing elements in (shown) are used to perform fused multiplication and addition operations. For example, data input to row

[24] of PE array 1720 can be copied by input selector circuitry and sent sequentially to multiple corresponding rows in PE array 1720 (e.g., rows

[30] and

[36] , such as...). Figure 15 The processing element (shown) performs fused multiplication and addition operations.

[0144] Therefore, K rows of the PE array 1720 can be used simultaneously, where the input data in each partition 1732 can be read from memory once and filtered using multiple (e.g., K) filter elements. Subsequently, multiple (e.g., K) other filter elements from each of a set of C 2D filters can be loaded into multiple (e.g., K) rows in corresponding columns of M columns, and data from one or more partitions 1732 can be read, copied, and sequentially sent to processing elements in multiple corresponding rows of the PE array 1720 to perform fusion multiplication and addition operations to generate M output feature maps 1750. Therefore, more rows of the PE array 1720 can be used at any given time to perform convolution operations faster.

[0145] The Accelerator 1710 can execute compiler-generated instructions using neural network models such as the ResNet-50 model. The neural network model can be represented by a data flow graph, where each node (e.g., a vertex) in the graph represents an operation, and connections between nodes (e.g., edges) represent data flows or data dependencies. For example, the compiler can perform shape reasoning on the neural network model to determine the size of the data used for each operation. The compiler can then traverse the data flow graph to identify operations that may not efficiently utilize the hardware system's computational engines (e.g., accelerators, or more specifically, arrays of processing elements) to implement the neural network. For example, the compiler can identify operations that use a small number of input channels, such as operations that, when applying a weight to each input channel, use no more than, for example, half the total number of rows in the PE array.

[0146] For the identified operations, the compiler can add operations to the neural network model to fill each input channel of the input feature map based on the parameters of the convolution operation (such as the size of the original input feature map, the size of the filter (e.g., the kernel), the stride used for convolution, memory alignment, and the size of the array of processing elements), as mentioned above regarding, for example... Figure 13 As described. Optionally, the compiler may add operations to the neural network model for partitioning the padded input feature map into multiple partitions and dividing the convolution operation into multiple sub-operations, where the sub-operations can use different partitions of the multiple partitions. In some implementations, the compiler may add operations for discarding some of the padded data or using the results generated with some of the padded data.

[0147] The compiler can then map convolution operations (e.g., suboperations) to hardware systems, such as arrays of processing elements in an accelerator. For example, the compiler can detect operations that use shifted data read from memory and share some data with other operations by analyzing computation loop boundaries. For detected operations, the compiler can add instructions, which can be executed by, for example, the input selector circuit described above, to copy some data to certain rows of the processing element array. The compiler can also determine the amount of data to be discarded due to padding and generate instructions for execution, for example, by an accelerator, to discard some padding data or use the result generated with some padding data.

[0148] Figure 18 This is a flowchart illustrating an example of a method for more efficiently utilizing an array of processing elements while reducing data transfer from memory, according to certain embodiments. For example, the operations described in flowchart 1800 can be performed by a compiler running on a host system (such as those described below). Figure 19 The compiler and host system described above are used for execution. Although flowchart 1800 can describe the operations as a sequential process, in various implementations, many operations can be executed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. The process may have additional steps not shown in the figure. Furthermore, implementations of the method can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments used to perform the associated tasks can be stored in a computer-readable medium such as a storage medium.

[0149] At box 1810, the host system may receive a neural network model comprising convolutional operations in different layers of the neural network. The neural network model may need to be implemented using a computational engine that may include an array of two-dimensional processing elements. The neural network model may be represented by a data flow graph as described above. The convolutional operation can be used to generate an output tensor comprising a set of output feature maps using a set of input feature maps and a set of filters. As described above, in some embodiments, the set of input feature maps may include C input feature maps, the set of filters may include M filters, each filter comprising C filter matrices, and the set of output feature maps may include M output feature maps, where each of the M output feature maps may be the sum of the convolution results between each of the C input feature maps and the corresponding filter matrix in the C filter matrices of one of the M filters. In some embodiments, C may be much less than the total number of rows in the two-dimensional processing element array in the computational engine. In one example, C may be 3, and the three input feature maps may include the red, green, and blue channels of a color image. For example, the processing element array may include 64, 128, or more rows.

[0150] At box 1820, the compiler executing on the host system can detect convolution operations that do not efficiently utilize the computation engine. For example, the compiler can traverse the data flow graph of a neural network model to identify convolution operations that may not fully utilize the computation engine to implement the neural network. The identified operations can use a small number of input channels, such as applying one filter element to each of the few input channels, not exceeding half the total number of rows in the PE array. When the number of input channels does not exceed half the total number of rows in the PE array, the convolution operation can utilize no more than half the rows in the processing element array to apply a set of filter elements to a set of input feature maps, which includes one filter element from each of a set of filters. For example, the number of rows used can be equal to the number of input channels.

[0151] At box 1830, the compiler can add operations to the neural network model, such as adding nodes to the data flow graph representing the neural network. For example, the operations could include: stride-based operations on convolution (e.g., 2×2), as described above regarding... Figures 13 to 15 The above describes filling the input feature map with padding data; as mentioned above. Figures 14 to 16 The process involves partitioning the input feature map into partitions; dividing the convolution operation into smaller sub-operations for operation on the partitions; and / or discarding certain padding data, such as data used for memory alignment. In some embodiments, the added operation may include writing the partitions into different corresponding banks of memory in a memory device. A neural network model with the added operation can be compiled to generate instructions for execution by a computation engine, which includes a two-dimensional processing element array and, for example, in... Figure 7 Other circuits described herein include the input selector circuit 730.

[0152] At box 1840, the compiler can detect instructions that shift data into the PE array, where the data may be shared at least partially between the instructions. For example, the compiler can recognize two instructions that may include shifting the same portion of an input feature map into the PE array for multiplication with different filter elements. The first instruction may include shifting a first portion of the input feature map into the PE array for multiplication with a first filter element of the filter, and the second instruction may include shifting a second portion of the input feature map into the PE array for multiplication with a second filter element of the filter, and there may be some overlap between the first and second portions of the input feature map. The first and second instructions may potentially be executed simultaneously by two rows of the PE array, with one instruction using data used by the other instruction.

[0153] At block 1850, the compiler can generate instructions for copying data that can be shared between instructions. For example, data used by a first instruction can be read from memory and shifted into a first row of the PE array loaded with the first filter elements, while data used by a second instruction can be copied by an input selector circuit based on data read from memory and used by the first instruction. In some embodiments, the input selector circuit can delay the copied data for a number of clock cycles before sending the copied data to a second row of the PE array loaded with the second filter elements. In some embodiments, the instructions for copying data may include instructions for activating the input selector circuit after the data read from memory has been input into the first row.

[0154] Optionally, at box 1860, the compiler may generate instructions to discard the results of calculations performed using some padding data. For example, some data shifted into the first row of the PE array may not be needed by the first row but may be copied for use in the second row. Therefore, the results of calculations performed in the first row using padding data may be discarded (e.g., not added to the sum of products).

[0155] Figure 19 This includes a block diagram illustrating an example of a host system 1900 on which the compiler 1930 described herein can run. The host system 1900 shown is an example of a computing device and includes a processor 1902, processor memory 1904, at least one storage device 1906, various input / output (I / O) devices 1908, and at least one network interface 1910. Figure 19 In the examples, host system 1900 also includes acceleration engine 1912, which is an integrated circuit device that can accelerate certain operations or calculations performed by host system 1900. In various examples, host system 1900 can be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smartphone, among other examples. In some examples, operations or components discussed below that are performed or included in host system 1900 may be performed or included in other computer devices. For example, compiler 1930 may execute on host system 1900, while acceleration engine 1912 resides on a different host system.

[0156] Processor 1902 is an integrated circuit device capable of executing program code in the form of instructions. The program code can be used in various software applications or tools, such as operating system 1920 or compiler 1930. While processor 1902 is executing a program, the program's instructions can be stored in processor memory 1904. Instructions can also be stored elsewhere, such as on storage device 1906, and can be loaded into processor memory 1904 when needed by processor 1902. Processor 1902 can also use processor memory 1904 to temporarily store other data that processor 1902 is operating on. In various examples, processor memory 1904 is a volatile memory type, such as random access memory, but non-volatile memory types can be used alternatively or additionally for processor memory 1904.

[0157] Storage device 1906 is an example of a device that may include non-volatile memory. For example, storage device 1906 may be a disk drive, a solid-state drive, or an optical disk drive, among other examples. Storage device 1906 may also be non-transitory, such that program code and other data stored on storage device 1906 persist even when storage device 1906 is not powered on.

[0158] Storage device 1906 is an example of a peripheral device, which is a component that can be coupled to host system 1900 to add functionality to host system 1900. Other examples of peripheral devices include input / output device 1908 and network interface 1910. Input / output device 1908 may include user input and output devices such as keyboard, mouse, touchscreen, microphone, display, speaker, printer, and scanner, among others. Network interface 1910, which can be implemented using a network interface card, can provide access to one or more networks. For example, network interface 1910 may include a physical port for connecting a network cable and / or a wireless antenna for communicating with Wi-Fi and / or cellular networks. Network interface 1910 may also be described as an I / O device.

[0159] Acceleration Engine 1912 is another type of peripheral device or I / O device. Acceleration Engine 1912 is a device specifically built to perform certain operations that can be performed by Processor 1902 but can be performed faster by Acceleration Engine 1912. For example, Acceleration Engine 1912 could be a neural network accelerator, thus potentially enabling it to perform massively parallel computations of neural networks more efficiently than when Processor 1902 performs computations. As another example, Acceleration Engine 1912 could be a graphics processing unit (GPU) and could be optimized to perform the computations required for graphics rendering. Other examples of devices that can be implemented by Acceleration Engine 1912 include cryptographic accelerators, compression and decompression accelerators, 3D accelerators, regular expression accelerators, security accelerators, and so on.

[0160] In various examples, the acceleration engine 1912 can execute program code to perform certain operations. For example, when the acceleration engine 1912 is a neural network accelerator, it can be programmed to execute a specific neural network, such as a neural network for image recognition or a neural network for machine translation. As another example, to support the execution of a neural network, the acceleration engine 1912 can be programmed to perform operations such as copying data for the neural network from processor memory 1904 (e.g.) to the acceleration engine 1912, copying input data for the neural network from processor memory 1904 to the acceleration engine 1912, and / or copying results from the acceleration engine 1912 to processor memory 1904, among other examples.

[0161] To generate program code for the acceleration engine 1912, in various examples, the host system 1900 could execute compiler 1930. Typically, a compiler is a software program that translates program code written in a human-readable language into a format (e.g., machine instructions) that can be read and processed by integrated circuit devices. Figure 19 In the example, acceleration engine 1912 is a neural network accelerator, and compiler 1930 is used to compile the neural network description into instructions to be executed by acceleration engine 1912. When acceleration engine 1912 implements a different type of accelerator, another compiler may be used. As described above, in some implementations, compiler 1930 may be used to divide tensor operations into suboperations and may schedule the execution of suboperations on different acceleration engines and / or different accelerators within the acceleration engine.

[0162] For example, when the operating system 1920 receives input from the input / output device 1908 via a keyboard, mouse, touchscreen, voice command, or other input, the compiler 1930 can be activated. Input may also include parameters for the compiler 1930, such as input code 1942 for options to compile and configure the compilation process. Once the compiler 1930 is activated, the processor 1902 can load instructions for the compiler 1930 into the processor memory 1904 and execute the instructions.

[0163] exist Figure 19 In the example, compiler 1930 includes a first stage 1932, a second stage 1936, and a third stage 1940, each performing different operations to produce compiled code 1944. In other examples, compiler 1930 may combine the operations of the first stage 1932, the second stage 1936, and / or the third stage 1940 into fewer stages, or it may divide the operation of one of the stages into multiple stages.

[0164] The first stage 1932 can receive and process input code 1942. Input code 1942 can describe programs using high-level programming languages ​​such as Java, C++, or Tensorflow, and many other examples. For example, input code 1942 can describe steps to perform image recognition, speech recognition, machine translation, or other operations. For example, input code 1942 can be obtained from storage device 1906. Alternatively, although not shown here, input code 1942 may reside in processor memory 1904 or be obtained from a network location using network interface 1910. Processing of input code 1942 can include classifying the operations described in input code 1942 into layers, where the output of one layer provides input to the next layer. Processing can also include identifying steps to be performed by processor 1902 rather than by acceleration engine 1912. For example, processor 1902 may need to perform steps such as configuring direct memory access (DMA) descriptors to move data into or out of acceleration engine 1912, and other examples, through the execution of driver 1922.

[0165] For example, the output 1934 of the first stage 1932 can be organized into layers, nodes, and connections between nodes in a neural network. The second stage 1936 can perform intermediate processing on the output 1934. For example, the operations performed at any layer or any node in a layer may be too many for the acceleration engine 1912 to perform simultaneously. For example, the acceleration engine 1912 may have a limited amount of local storage space for the data required for computation, or the computation may exceed what the acceleration engine 1912 can perform at one time. In this example, the first stage 1932 can decompose the operations of a layer or node into smaller operations that can fit within the local memory of the acceleration engine and / or the computational power of the acceleration engine 1912. Processing of the output 1934 of the first stage 1932 may include other steps, such as scheduling or determining the order in which the acceleration engine 1912 and / or the processor 1902 will perform operations, and other examples.

[0166] In various examples, the output 1938 of the second stage 1936 includes the individual steps to be performed by the components of the acceleration engine 1912 in the order of the steps to be performed. For example, the output 1938 can be represented as a data flow graph, where the nodes in the graph represent memory operations, computations, and other operations, and the edges or connections between nodes represent dependencies between nodes, such as data dependencies, memory dependencies, or operation dependencies, and other examples.

[0167] The third stage 1940 can operate on the output 1938 of the second stage 1936 and perform various steps before generating instructions to be executed by the acceleration engine 1912. These steps may include, for example, removing redundant dependencies; resolving or handling dependencies between nodes by inserting synchronization instructions into the code; identifying possible optimizations for memory usage or memory bandwidth usage; and other operations. In some examples, the third stage 1940 may include a data scheduler 1950.

[0168] The output of the third stage 1940 is compiled code 1944, which may include machine instructions in binary format. In some examples, compiled code 1944 may be stored in processor memory 1904. Alternatively or additionally, compiled code 1944 may be copied to storage device 1906 or a network location. As mentioned above, the acceleration engine 1912 may reside on a different host system, in which case compiled code 1944 can be sent to other host systems via network interface 1910.

[0169] exist Figure 19 In the example, host system 1900 can execute driver 1922, which manages acceleration engine 1912; this driver may also be referred to as a device driver or runtime driver. Driver 1922 can provide an interface between an application running on host system 1900 (or another host system) and acceleration engine 1912. For example, driver 1922 can provide an application programming interface (API) that defines functions for feeding input data to acceleration engine 1912 and defining operations performed on the input data. In this and other examples, driver 1922 can configure acceleration engine 1912 to perform operations. For example, driver 1922 can identify the neural network that acceleration engine 1912 wants to execute, and the location of compiled code 1944 for the neural network in processor memory 1904 or on storage device 1906. Driver 1922 can also load into or cause acceleration engine 1912 to load compiled code 1944, load or cause acceleration engine 1912 to load input data for the neural network to operate on, and / or cause acceleration engine 1912 to execute on the input data. Once the acceleration engine 1912 has completed, it can notify the driver 1922, and the driver 1922 can send the result back to the application that requested the result.

[0170] Figure 20 Includes a block diagram illustrating an example of an acceleration engine 2000. The acceleration engine 2000 is an example of an integrated circuit that may include one or more accelerators 2002a-2002n, which may be similar to those described above for example... Figure 7The accelerators described above. For example, in one embodiment, the acceleration engine 2000 may include four accelerators that can be used to perform the four sub-operations of the convolution operation in parallel, as described above.

[0171] exist Figure 20 In the examples, the acceleration engine 2000 includes multiple accelerators 2002a-2002n, each capable of performing a set of operations. In various examples, accelerators 2002a-2002n are used for specific types of operations so that they can perform the operations faster than a general-purpose processor would when performing similar operations. In various examples, to perform a set of operations, the input data to which the operations are to be performed must first be moved into accelerators 2002a-2002n. Furthermore, in some cases, program code is also moved into accelerators 2002a-2002n, which program the operations that accelerators 2002a-2002n will perform on the data. In the illustrated example, the acceleration engine 2000 includes N accelerators 2002a-2002n. Examples of accelerators that can be included in the acceleration engine 2000 include graphics accelerators, floating-point accelerators, neural network accelerators, etc. In various examples, accelerators 2002a-2002n may each be the same (e.g., each of them is a graphics accelerator) or may be different (e.g., accelerators 2002a-2002n include graphics accelerators, floating-point accelerators, and neural network accelerators).

[0172] The example acceleration engine 2000 also includes DRAM controllers 2042a-2042k for communicating with external memory. In this example, the external memory is implemented using DRAM 2030. In the illustrated example, the acceleration engine 2000 includes K DRAM controllers 2042a-2042k, each of which may be able to communicate with a separate set of DRAM banks. In other examples, other types of RAM technologies may be used for the external memory. The DRAM controllers 2042a-2042k may also be referred to as memory controllers.

[0173] In various examples, the input data and / or program code for accelerators 2002a-2002n can be stored in DRAM 2030. Different programs can cause accelerators 2002a-2002n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program can configure the neural network accelerator to perform speech recognition, while another program can configure the neural network accelerator to perform image recognition. In various examples, different programs can be used to program different accelerators 2002a-2002n so that each accelerator performs a different set of operations. In various examples, processors 2048a-2048s can manage the movement of program code from DRAM 2030 to accelerators 2002a-2002n.

[0174] The example acceleration engine 2000 also includes I / O controllers 2044a-2044p for communicating with I / O devices 2032 in the system. For example, the acceleration engine 2000 can communicate with the I / O devices via a processor bus. In some examples, the processor bus may be implemented using Peripheral Component Interconnect (PCI) and / or a variant of the PCI bus protocol. The processor bus can connect the acceleration engine 2000 to I / O devices such as, for example, input and output devices, memory controllers, storage devices, and / or network interface cards. In some examples, the I / O controllers 2044a-2044p enable the acceleration engine 2000 to act as an I / O device for the host processor. For example, the acceleration engine 2000 can be a receiver of input data from the host processor and commands instructing the operation to be performed on the input data (e.g., a specific calculation or analysis). In the illustrated example, the acceleration engine 2000 includes p I / O controllers 2044a-2044p, each of which may include a separate root complex and can communicate with a separate set of I / O devices 2032. In other examples, other standardized bus protocols, such as Hyperpath Interconnect (UPI), can be used for the host bus. In other examples, proprietary bus protocols can be used.

[0175] Data movement within the acceleration engine 2000 can be managed by one or more processors 2048a-2048s, which may also be referred to as data management processors. Figure 20In one example, the acceleration engine 2000 includes S processors 2048a-2048s incorporated in the device (e.g., on the same silicon chip). In other examples, the processors 2048a-2048s may be external to the acceleration engine 2000 (e.g., on a different chip and / or in a different package). In some examples, the processors 2048a-2048s may manage the movement of data from I / O device 2032 to accelerators 2002a-2002n or DRAM 2030. For example, input data may reside in I / O device 2032 or processor memory, and the processors 2048a-2048s may move input from I / O device 2032 or processor memory to the accelerator or DRAM 2030. Furthermore, program code for accelerators 2002a-2002n may reside on I / O device 2032 or in processor memory.

[0176] The example acceleration engine 2000 also includes DMA engines 2046a-2046d, which can move data between accelerators 2002a-2002n, DRAM controllers 2042a-2042k, and I / O controllers 2044a-2044p. In the illustrated example, the acceleration engine 2000 includes D DMA engines 2046a-2046d. In some implementations, the DMA engines 2046a-2046d can be assigned to specific tasks, such as moving data from DRAM controllers 2042a-2042d to accelerators 2002a-2002n, or moving data between I / O controllers 2044a-2044p and accelerators 2002a-2002n. For example, these tasks can be assigned by queuing descriptors using DMA engines 2046a-2046d, where the descriptor identifies the address of a data block and the operation to be performed (e.g., read or write). For example, a descriptor can instruct the DMA engine to tell the DMA controller to read data blocks from DRAM 2030. As another example, a descriptor can instruct the DMA engine to write data read by the DMA controller to the accelerator. Additional descriptors can be used to move data from the accelerator to DRAM 2030.

[0177] In various examples, each of processors 2048a-2048s can be responsible for managing data movement to different accelerators. In some examples, a processor can manage data movement to more than one accelerator. Similarly, in various examples, each of processors 2048a-2048s can be assigned to one or more DMA engines 2046a-2046d. In these and other examples, the relationships between processors 2048a-2048s, accelerators 2002a-2002n, and DMA engines 2046a-2046d are determined by the program code executed by each corresponding processor.

[0178] In the example acceleration engine 2000, various components can communicate via the chip interconnect 2020. The chip interconnect 2020 primarily includes wiring for routing data between components of the acceleration engine 2000. In some cases, the chip interconnect 2020 may include a minimal amount of logic, such as multiplexers for controlling data direction, flip-flops for handling clock domain crossovers, and timing logic.

[0179] Examples of implementation methods can be described in the following numbered clauses:

[0180] Clause 1. A computer-implemented method comprising:

[0181] Receive a neural network model implemented using a neural network accelerator including a first number of processing element rows, the neural network model including network layers, the network layers including convolution operations for generating output feature maps using a second number of input feature maps and a set of filters;

[0182] Determine that the second quantity is equal to or less than half of the first quantity;

[0183] Add operations to the neural network model, the operations including:

[0184] The second number of input feature maps are populated with padding data to generate a padded input feature map;

[0185] Divide each of the filled input feature maps into partitions; and

[0186] The convolution operation is divided into sub-operations based on the partition;

[0187] Instructions are generated based on the neural network model and executed by the neural network accelerator to implement the convolution operation;

[0188] According to the instructions, a first instruction and a second instruction are detected in the first partition of the partition of the input feature map, both using the filled input feature map, wherein the first instruction and the second instruction use different elements of the filters in the group of filters; and

[0189] Generate instructions for copying the first partition, which was read from memory and used by the first instruction, for use by the second instruction.

[0190] Clause 2. The computer-implemented method according to Clause 1 further includes generating instructions for discarding the result generated using the padding data and the first instructions.

[0191] Clause 3. A computer-implemented method according to any one of Clauses 1-2, wherein:

[0192] The padding data includes padding data for memory alignment; and

[0193] The operation also includes discarding the padding data used for memory alignment in the sub-operation.

[0194] Clause 4. The computer-implemented method according to any one of Clauses 1-3, further comprising generating instructions for mapping the first instruction and the second instruction to different rows in the first number of processing element rows for simultaneous execution.

[0195] Clause 5. A method comprising using a computer system to:

[0196] Receive a neural network model implemented using an array of processing elements, the neural network model comprising convolution operations on a set of input feature maps and a set of filters;

[0197] Based on the neural network model, it is determined that the convolution operation uses fewer than a threshold number of rows in the processing element array to apply a set of filter elements to the group input feature map, the group filter elements including one filter element from each filter in the group filter; and

[0198] The convolution operation is performed and a first instruction and a second instruction are generated based on the neural network model to be executed by the corresponding row in the processing element array, wherein the first instruction and the second instruction use different filter elements of the filters in the group of filters.

[0199] Clause 6. The method according to Clause 5 further includes adding operations to the neural network model for padding the input feature map with padding data to generate a padded input feature map.

[0200] Clause 7. The method described in accordance with Clause 6, wherein:

[0201] The padding data includes padding data for memory alignment; and

[0202] The method also includes adding an operation to the neural network model to discard the padding data used for memory alignment when the convolution operation is performed using the processing element array.

[0203] Clause 8. The method according to any one of Clauses 6-7 further includes adding an operation to the neural network model for partitioning each of the filled input feature maps into partitions, wherein the first instruction and the second instruction use the same partition among the partitions of the filled input feature maps in the filled input feature maps.

[0204] Clause 9. The method according to Clause 8 further includes adding an operation to the neural network model for dividing the convolution operation into sub-operations, wherein each sub-operation uses a partition in the partition.

[0205] Clause 10. The method according to any one of Clauses 8-9 further includes adding operations to the neural network model for writing the partition into different corresponding memory banks in the memory device.

[0206] Clause 11. The method according to any one of Clauses 6-10 further includes generating instructions for discarding the results generated using the padding data and the first instructions.

[0207] Clause 12. The method according to any one of Clauses 5-11, wherein the convolution operation is characterized by a stride greater than 1.

[0208] Clause 13. The method according to any one of Clauses 5-12, wherein:

[0209] The first instruction and the second instruction use the same portion of the input feature map in the group of input feature maps; and

[0210] The method further includes generation instructions, the instructions being used to:

[0211] The data input to the first row of the processing element array to execute the first instruction is copied via the input selector circuit; and

[0212] The copied data is sent to the second row of the processing element array to execute the second instruction.

[0213] Clause 14. The method according to any one of Clauses 5-13, wherein the number of thresholds is equal to or less than half the total number of rows in the processing element array.

[0214] Clause 15. The method according to any one of Clauses 5-14 further includes:

[0215] Instructions are generated based on the neural network model for execution by a computing engine, the computing engine including the array of processing elements for implementing the convolution operation, the instructions including the first instruction and the second instruction; and

[0216] According to the instructions, it was detected that the first instruction and the second instruction used the same part of the input feature map in the group input feature map.

[0217] Clause 16. The method according to any one of Clauses 5-15, wherein the neural network model includes a data flow graph, the data flow graph including nodes representing neural network operations.

[0218] Clause 17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including:

[0219] Receive a neural network model implemented using an array of processing elements, the neural network model comprising convolution operations on a set of input feature maps and a set of filters;

[0220] The convolution operation is determined to apply a set of filter elements to the group input feature map using fewer than a threshold number of rows in the processing element array, the group filter elements comprising one filter element from each filter in the group filter; and

[0221] The convolution operation is performed and a first instruction and a second instruction are generated based on the neural network model to be executed by the corresponding row in the processing element array, wherein the first instruction and the second instruction use different filter elements of the filters in the group of filters.

[0222] Clause 18. The non-transitory computer-readable medium as described in Clause 17, wherein said operation further includes adding operations to the neural network model for padding the input feature map with padding data to generate a padded input feature map.

[0223] Clause 19. The non-transitory computer-readable medium as described in Clause 18, wherein said operation further comprises:

[0224] Add an operation to the neural network model to partition each of the padded input feature maps into partitions; and

[0225] Add an operation to the neural network model to divide the convolution operation into sub-operations, wherein each sub-operation uses a partition of the partition of the filled input feature map.

[0226] Clause 20. A non-transitory computer-readable medium according to any one of Clauses 17-19, wherein said operation further comprises generating instructions for:

[0227] The data input to the first row of the processing element array to execute the first instruction is copied via the input selector circuit; and

[0228] The copied data is sent to the second row of the processing element array to execute the second instruction.

[0229] Therefore, this specification and drawings should be interpreted in an illustrative rather than restrictive sense. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the claims.

[0230] Other variations are within the spirit of this disclosure. Therefore, while the disclosed technology allows for various modifications and alternative constructions, specific examples have been shown in the drawings and described in detail above. However, it should be understood that this disclosure is not intended to be limited to the one or more specific forms disclosed, but rather is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined by the appended claims.

[0231] Any of the methods described herein can be implemented as a computer-readable medium or computer program product including instructions that, when executed by one or more computers, cause the one or more computers to perform the steps of the method. Such a computer program product can be transmitted in a data carrier signal carrying the computer program product via a wired or wireless network.

[0232] In the context of describing the disclosed examples (especially in the context of the following claims), the terms “a (a, an)” and “the”, and similar designations, shall be construed as encompassing both the singular and plural, unless otherwise indicated herein or obviously contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” shall be construed as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” shall be construed as being partially or wholly incorporated, attached to, or combined with, even in the presence of an intervening element. Unless otherwise stated herein, the enumeration of value ranges herein is merely intended as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into this specification as if individually stated herein. Unless otherwise stated herein or obviously contradicted by the context, all methods described herein may be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is merely intended to better illustrate examples of this disclosure and, unless otherwise required, does not limit the scope of this disclosure. The language used in this specification should not be construed as indicating any non-claimed element as necessary for practicing this disclosure.

[0233] Unless otherwise specified, separating language such as the phrase “at least one of X, Y or Z” is intended, within the context, to be understood as the case of common use to indicate that an item, term, etc., can be X, Y or Z or any combination thereof (e.g., X, Y and / or Z). Therefore, such separating language is generally not intended and should not imply that certain examples require the presence of at least one of X, at least one of Y or at least one of Z respectively.

[0234] Various examples of this disclosure are described herein, including the best mode known to the inventors for carrying out this disclosure. Variations of these examples will become apparent to those skilled in the art upon reading the foregoing description. The inventors intend that those skilled in the art adopt such variations as appropriate, and the inventors intend to practice this disclosure in ways different from those specifically described herein. Therefore, this disclosure includes all modifications and equivalents of the subject matter set forth in the appended claims, as long as permitted by law. Furthermore, unless otherwise indicated herein or clearly contradicted by the context, this disclosure covers any combination of the foregoing elements in all its possible variations.

Claims

1. A computer-implemented method comprising using a computer system to: Receive a neural network model implemented using an array of processing elements, the neural network model comprising convolution operations on a set of input feature maps and a set of filters; Based on the neural network model, it is determined that the convolution operation uses fewer than a threshold number of rows in the array of processing elements to apply a set of filter elements to the set of input feature maps, the set of filter elements including one filter element in each of the set of filters; An operation is added to the neural network model to fill the input feature map with padding data to generate a filled input feature map, wherein the padding data is added for memory alignment such that each row of the processing element array can receive the same number of input elements for the input wavelet; For the convolution operation, a first instruction and a second instruction are generated based on the neural network model to be executed by the corresponding row in the processing element array, wherein the first instruction and the second instruction use different filter elements of the filters in the set of filters and use the same part of the input feature map in the set of input feature maps; Generate instructions for the following operations: copy data input to the first row of the processing element array via an input selector circuit to execute the first instruction; and send the copied data to the second row of the processing element array to execute the second instruction; as well as Add an operation to the neural network model to discard the padding data used for memory alignment when performing the convolution operation using the processing element array.

2. The method of claim 1, further comprising adding an operation to the neural network model for partitioning each of the filled input feature maps into partitions, wherein the first instruction and the second instruction use the same partition among the partitions of the filled input feature maps.

3. The method of claim 2, further comprising adding an operation to the neural network model for dividing the convolution operation into sub-operations, wherein each sub-operation uses a partition in the partition.

4. The method of claim 2 or 3, further comprising adding operations to the neural network model for writing the partition into different corresponding memory banks in the memory.

5. The method according to any one of claims 1-3, further comprising generating instructions for discarding the result generated using the padding data and the first instruction.

6. The method according to any one of claims 1-3, wherein the convolution operation is characterized by a stride greater than 1.

7. The method according to any one of claims 1-3, wherein the number of thresholds is equal to or less than half the total number of rows in the processing element array.

8. The method according to any one of claims 1-3, further comprising: Instructions are generated based on the neural network model for execution by a computing engine, the computing engine including the array of processing elements for implementing the convolution operation, and the instructions including the first instruction and the second instruction; as well as According to the instructions, it was detected that the first instruction and the second instruction used the same part of the input feature map in the set of input feature maps.

9. A computer program product storing instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including: Receive a neural network model implemented using an array of processing elements, the neural network model comprising convolution operations on a set of input feature maps and a set of filters; The convolution operation is determined to apply a set of filter elements to the set of input feature maps using fewer than a threshold number of rows in the array of processing elements, the set of filter elements including one filter element in each of the set of filters; An operation is added to the neural network model to fill the input feature map with padding data to generate a filled input feature map, wherein the padding data is added for memory alignment such that each row of the processing element array can receive the same number of input elements for the input wavelet; For the convolution operation, a first instruction and a second instruction are generated based on the neural network model to be executed by the corresponding row in the processing element array, wherein the first instruction and the second instruction use different filter elements of the filters in the set of filters and use the same part of the input feature map in the set of input feature maps; Generate instructions for the following operations: copy data input to the first row of the processing element array via an input selector circuit to execute the first instruction; and send the copied data to the second row of the processing element array to execute the second instruction; as well as Add an operation to the neural network model to discard the padding data used for memory alignment when performing the convolution operation using the processing element array.

10. The computer program product according to claim 9, wherein the operation further comprises: Add an operation to the neural network model to divide each of the filled input feature maps into partitions; as well as Add an operation to the neural network model to divide the convolution operation into sub-operations, wherein each sub-operation uses a partition of the partition of the filled input feature map.

11. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including: Receive a neural network model implemented using an array of processing elements, the neural network model comprising convolution operations on a set of input feature maps and a set of filters; Based on the neural network model, it is determined that the convolution operation uses fewer than a threshold number of rows in the array of processing elements to apply a set of filter elements to the set of input feature maps, the set of filter elements including one filter element in each of the set of filters; An operation is added to the neural network model to fill the input feature map with padding data to generate a filled input feature map, wherein the padding data is added for memory alignment such that each row of the processing element array can receive the same number of input elements for the input wavelet; For the convolution operation, a first instruction and a second instruction are generated based on the neural network model to be executed by the corresponding row in the processing element array, wherein the first instruction and the second instruction use different filter elements of the filters in the set of filters and use the same part of the input feature map in the set of input feature maps; Generate instructions for the following operations: copy data input to the first row of the processing element array via an input selector circuit to execute the first instruction; and send the copied data to the second row of the processing element array to execute the second instruction; as well as Add an operation to the neural network model to discard the padding data used for memory alignment when performing the convolution operation using the processing element array.

Images