Method and system for workload-balanced convolution with activation sparsity

Abstract

Methods, systems, and apparatuses for workload-leveled convolution with activation sparsity are described, including computer programs encoded on a computer storage medium. An exemplary method includes: allocating input tensors and weight tensors in a convolutional layer to multiple processors to perform multiply-accumulate (MAC) operations in parallel based on the input tensors and weight tensors, obtaining multiple output values ​​based on results of the MAC operations, constructing one or more banks of output values ​​based on the multiple output values, for each of the banks, performing a top-K sort on the one or more output values ​​in the bank to obtain K output values, pruning each of the banks by setting one or more output values ​​other than the obtained K output values ​​in each bank to zero, and constructing an output tensor of the convolutional layer based on the pruned banks.

Description

[Technical Field]

[0001] The present disclosure relates generally to artificial intelligence, and more particularly to workload-leveled convolutional processing with activation sparsity. [Background technology]

[0002] Neural networks (NNs) are currently the foundation of many modern artificial intelligence (AI) applications, including image and video recognition, recommender systems, classification, medical image analysis, and natural language processing. NNs are used in a variety of usage scenarios, from self-driving cars and cancer detection to playing complex games. A typical NN can comprise a series of convolutional layers, where intensive, and therefore expensive (in terms of computation and energy), convolution operations are performed. A typical convolutional layer can contain one or more activation (or input) tensors and one or more weight tensors.

[0003] In recent years, various approaches have been developed to improve the computational efficiency of neural networks by introducing sparsity into the convolution process, such as pruning the weight tensor to reduce the number of non-zero weights. Some existing hardware architectures completely ignore activation sparsity or assume unstructured activation sparsity, which can lead to workload imbalance when performing convolution operations using parallel processing frameworks. The imbalanced workload can directly impact the system scalability and performance of the architecture. Therefore, it is desirable to build a new system architecture that provides and exploits workload-balanced activation sparsity to improve the efficiency of convolution processing. Summary of the Invention

[0004] Various embodiments herein may include systems, methods, and non-transitory computer-readable media for workload-leveled convolution with activation sparsity.

[0005] According to one aspect, a method for convolution with activation sparsity with workload leveling includes: assigning input tensors and weight tensors in a convolution layer to multiple processors to perform multiply-accumulate (MAC) operations in parallel based on the input tensors and weight tensors; obtaining multiple output values ​​based on results of the MAC operations; constructing one or more banks of output values ​​based on the multiple output values, each bank comprising one or more output values ​​from the multiple output values; for each of the banks, performing a top-K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer; pruning each of the banks by setting one or more output values ​​other than the K obtained output values ​​in each bank to zero; and constructing an output tensor of the convolution layer based on the pruned banks.

[0006] In some embodiments, assigning an input tensor and a weight tensor in a convolutional layer to a plurality of processors comprises obtaining an input tensor and a plurality of weight tensors in a convolutional layer in a neural network; segmenting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to the plurality of processors; and segmenting the plurality of weight tensors into a plurality of weight groups and assigning the plurality of weight groups to the plurality of processors.

[0007] In some embodiments, allocating the plurality of sub-tensors and the plurality of weight groups to the plurality of processors comprises, for each of the plurality of processors, storing one or more sub-tensors of the plurality of sub-tensors and one or more weight groups of the plurality of weight groups in an internal buffer of the processor.

[0008] In some embodiments, assigning the plurality of weight groups to the plurality of processors comprises assigning one or more weight groups of the plurality of weight groups to the plurality of processors during each of a plurality of parallel processing cycles, such that the plurality of processors perform a MAC operation in parallel based on the plurality of sub-tensors and the one or more weight groups of the plurality of weight groups.

[0009] In some embodiments, each of the one or more banks corresponds to an output value generated based on one weight group of the plurality of weight groups.

[0010] In some embodiments, obtaining a plurality of output values ​​based on results of the MAC operation comprises obtaining a plurality of partial sums as a result of the MAC operation performed by the plurality of processors during each of the plurality of parallel processing cycles; providing the plurality of partial sums to an adder tree device to obtain one or more output values; and storing the one or more output values ​​in an accumulation buffer during one or more parallel processing cycles of the plurality of parallel processing cycles to obtain the plurality of output values.

[0011] In some embodiments, performing a top K sort on the output values ​​in the bank to obtain the K output values ​​comprises performing a top K sort on magnitude of the output values ​​in the bank to obtain the K output values.

[0012] In some embodiments, each of the banks has a height dimension greater than zero, a width dimension greater than zero, and a channel dimension greater than one.

[0013] In some embodiments, the method may further comprise representing the K non-zero output values ​​in each of the banks as K multi-dimensional vectors, each of the K multi-dimensional vectors comprising the non-zero output value and a corresponding channel dimension identifier of the non-zero output value in the bank.

[0014] In some embodiments, constructing an output tensor of the convolutional layer based on the pruned banks comprises assembling the K multidimensional vectors of each of the banks to form the output tensor.

[0015] In some embodiments, each of the pruned banks is associated with a bank identifier, and constructing the output tensor of the convolutional layer based on the pruned banks comprises assembling the K multidimensional vectors of each of the banks to form the output tensor based on the corresponding bank identifier.

[0016] In some embodiments, the output tensor is a sparse tensor.

[0017] According to another aspect, a system for workload-leveled convolution with activation sparsity is provided. The system may include one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured to cause the system to perform operations with instructions executable by the one or more processors, including: assigning input tensors and weight tensors in a convolutional layer to multiple processors to perform multiply-accumulate (MAC) operations in parallel based on the input tensors and weight tensors, obtaining multiple output values ​​based on results of the MAC operations, constructing one or more banks of output values ​​based on the multiple output values, each bank comprising one or more output values ​​from the multiple output values, for each bank, performing a top-K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer, pruning each of the banks by setting one or more output values ​​other than the K obtained output values ​​in each bank to zero, and constructing an output tensor of the convolutional layer based on the pruned banks.

[0018] According to yet another aspect, a non-transitory computer-readable storage medium for workload-leveled convolution with activation sparsity is provided. The medium can be configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: assigning input tensors and weight tensors in a convolution layer to multiple processors to perform multiply-accumulate (MAC) operations in parallel based on the input tensors and weight tensors, obtaining multiple output values ​​based on results of the MAC operations, constructing one or more banks of output values ​​based on the multiple output values, each bank comprising one or more output values ​​from the multiple output values, for each bank, performing a top-K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer, pruning each of the banks by setting one or more output values ​​other than the K obtained output values ​​in each bank to zero, and constructing an output tensor of the convolution layer based on the pruned banks.

[0019] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the method of operation and function of the associated elements of structure, combination of parts, and economies of manufacture, will become more apparent from a consideration of the following description and appended claims, with reference to the accompanying drawings, all of which form a part of this specification and in which like reference numerals indicate corresponding parts in the various views. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an exemplary convolutional neural network (CNN), according to various embodiments. [Figure 2]FIG. 2 is a diagram illustrating an exemplary convolution process, according to various embodiments. [Figure 3] FIG. 3 is an exemplary diagram of neural network layers, as well as an exemplary diagram of weight pruning and activation pruning, according to various embodiments. [Figure 4] FIG. 4 is an exemplary diagram for workload-leveled activation pruning, according to various embodiments. [Figure 5] FIG. 5 is an exemplary system diagram for workload-leveled convolution with activation sparsity, in accordance with various embodiments. [Figure 6A] FIG. 6A illustrates an exemplary system architecture for workload-leveled convolution with activation sparsity, in accordance with various embodiments. [Figure 6B] FIG. 6B is an exemplary flowchart for generating an output tensor with workload-leveled activation sparsity, according to various embodiments. [Figure 7] FIG. 7 illustrates an exemplary method for convolution with workload-leveled activation sparsity, in accordance with various embodiments. [Figure 8] FIG. 8 illustrates an exemplary computer system capable of implementing any of the embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0021] The embodiments described herein provide methods, systems, and devices for workload-leveled activation sparsity convolution in neural networks. Convolution can refer to a basic but computationally expensive operation for extracting features from input data. Convolution typically occurs in a convolutional layer within a neural network. Input data to a convolutional layer may include feature maps derived from images or audio waves or activation tensors received from a previous layer, which can be referred to as input tensors. The output of a convolutional layer can be referred to as an output tensor. In some embodiments, both the input tensor and the output tensor can be referred to as activations. The input tensor can comprise a multi-channel feature map. For example, an RGB image can be represented as an input tensor with three channels (red, green, and blue), with each channel comprising a feature map (e.g., a grayscale image of the same size as the original RGB image, but consisting of only one color: red, green, or blue). To extract features from an input tensor, one or more feature extractors (also called weight tensors or filters) can be applied to the input tensor in a convolutional (e.g., sliding or shifting) manner. A multiply-accumulate (MAC) operation between the feature detectors (weight tensors or filters) and the input tensors can be performed during each convolution step. The outputs generated from the convolution steps can then be assembled to form the output tensor of the convolutional layer. The output tensor can become the input tensor for the next layer in the neural network. In some cases, multiple input tensors can be processed by a single convolutional layer.

[0022] During convolution processing, input tensors and / or weight tensors can be pruned or sparsified by introducing a large number of zeros to reduce computational costs and improve inference speed. For example, more than 50% of the values ​​in the input tensors can be set to zero. Because convolution processing is typically performed in a parallel processing framework (a cluster of processors, cores, or another type of processing entity), unstructured or unbalanced pruning of input tensors can result in workload imbalance among processing entities (PEs), which can lead to reduced system scalability and performance. Therefore, an important goal of pruning input tensors includes balancing the workload among processing units during parallel convolution.

[0023] In some embodiments, an input tensor can be first segmented into multiple sub-tensors according to the number of PEs available to participate in the convolution. Each sub-tensor can be assigned to one of the multiple PEs and remain there across multiple convolutions (corresponding to multiple convolution layers in a neural network) without being swapped out from the PE's local memory / buffer. After each convolution, the sub-tensor assigned to the PE as input can be updated and expanded into an output that can be used as input for the next convolution. In some embodiments, the updated sub-tensor can be temporarily swapped out of the PE for memory management purposes and then swapped back in for the next iteration.

[0024] In some embodiments, each weight tensor can be segmented into multiple weight groups, each of which can be stored in one of the multiple PEs and a local MAC operation can be performed using the assigned sub-tensor (portion of the input tensor). In some embodiments, multiple weight groups can be rotated among multiple PEs for convolution with other sub-tensors until the convolution process is complete. Output values ​​from the multiple PEs are accumulated along the way and ultimately assembled as the output tensor of the convolution process.

[0025] The segmentation and parallel processing scheme described above is also highly scalable. In some embodiments, each round (also called an iteration) of parallel processing can generate multiple partial sums, and multiple PEs can perform parallel processing on multiple rounds, accumulating the partial sums generated in each round and assembling the accumulated results as an output tensor. In some embodiments, the partial sums generated by each PE over multiple rounds can be accumulated directly (within each PE) to generate a portion of the output tensor, without having to exchange data with other PEs.

[0026] In the following description, certain non-limiting embodiments of the present invention are described with reference to the drawings. Specific features and aspects of any embodiment disclosed herein can be used with and / or combined with specific features and aspects of any other embodiment disclosed herein. It should also be understood that such embodiments are exemplary and merely describe a few embodiments within the scope of the present invention. Various changes and modifications apparent to those skilled in the art to which the present invention pertains are deemed to be within the spirit, scope, and contemplation of the present invention, as further defined by the appended claims.

[0027] FIG. 1 illustrates an exemplary convolutional neural network (CNN) according to various embodiments. CNNs are widely used in a variety of applications, including image understanding, speech recognition, gameplay, and robotics, and may employ deep layers, including convolutional layers, where convolutional operations are performed. Note that CNNs are used for illustrative purposes only, and the embodiments disclosed herein may be applied to other neural networks involving convolutional operations.

[0028] The exemplary neural network 100 shown in FIG. 1 includes multiple layers, such as a first convolutional layer 120, a first ReLU (Ramp Function) layer, a first pooling layer 130, a second convolutional layer 140, a second ReLU layer, a second pooling layer 150, a flattening layer 160, and a fully connected (FC) layer 170. This exemplary neural network 100 can be trained to match the alphabet in a given image with multiple known alphabet classes. As shown in FIG. 1, a text-encoded image 110 is input into the neural network 100 and transformed through multiple layers. The final layer (FC layer 170) ultimately generates multiple scores representing the similarity between the characters in the input image 110 and each of the known alphabet classes.

[0029] In some embodiments, the input image 110 may first be converted into an input tensor. As an example, if the input image 110 includes 32*32 pixels, each pixel having three color channels (red, green, and blue), then its corresponding input tensor may have a size of 32*32*3, with a height of 32, a width of 32, and a depth (e.g., number of channels) of 3. For ease of explanation, the three-dimensional size may be referred to as the HWC format, where H refers to the height of the input tensor (e.g., 32 in the previous image example), W refers to the width of the input tensor (e.g., 32 in the previous image example), and C refers to the number of channels in the input tensor (e.g., 3 in the previous image example). In some cases, when a convolutional layer has multiple input tensors (e.g., when a convolutional layer has multiple images input, or multiple input activation tensors received from a previous layer), each input tensor can be represented in the NHWC format, where N refers to the index of the input tensor within the batch of input tensors. In the following description, N may be omitted for simplicity (e.g., assuming there is only one input tensor) unless otherwise specified. It will be clear to one skilled in the art to extend the embodiments to cover the case where N>1.

[0030] In some embodiments, a CNN may include multiple convolutional layers (e.g., the first convolutional layer 120 and the second convolutional layer 140 in FIG. 1 ). A convolutional layer reduces an input tensor (e.g., an original input image or an input tensor from a previous layer) to a more manageable form without losing important features for good prediction / classification. One or more feature detectors, such as an edge detector or a curve detector in image processing, may be involved in the convolutional processing in the convolutional layer. These feature detectors may be referred to as filters, weight tensors, or kernels, which are used interchangeably in this description. Each of the filters may have the same number of channels as the input tensor. For ease of explanation, this specification uses the term “sharing multiple channels” to mean that each filter and input tensor have the same number of channels. For example, the input tensor may be a 32*32*3 matrix, and an exemplary filter may be a 3*3*3 matrix. For a detailed description of the convolutional processing, please refer to the description of FIG. 2.

[0031] In some embodiments, convolutional layers in a CNN can be followed by a nonlinear activation function to introduce nonlinearity into the CNN. Exemplary nonlinear activation functions include sigmoid, hyperbolic tangent, and ramp function (ReLU). As shown in FIG. 1, a ReLU function (which may also be referred to as a ReLU layer) follows each of convolutional layers 120 and 140. The ReLU function can apply an element-wise activation function to filter out some of the outputs (activations) generated by convolutional layer 120 or 140. For example, a max(0,x) function can be used to filter out all negative activations in the convolutional output and feed only positive activations to the next layer. The ReLU function cannot change the size of the output activations, but it limits the number of active neurons because all negative activations are zeroed out to improve computational efficiency in subsequent layers.

[0032] In some embodiments, a CNN may also include one or more pooling layers to provide various computations that reduce the dimensionality of the input tensor. In Figure 1, first pooling layer 130 and second pooling layer 150 of neural network 100 may each perform downsampling operations on the input tensor along the spatial dimensions (height and width), but typically do not change the depth dimension (e.g., number of channels).

[0033] In some embodiments, a fully connected (FC) layer in a neural network can learn a nonlinear combination of high-level features represented by the output from a previous layer (e.g., pooling layer 150 in FIG. 1, or a convolutional layer) and calculate class scores accordingly. As shown with reference to FIG. 1, the output from pooling layer 150 can first be flattened into a vector (e.g., in flattening layer 160), from which a unique untrained function in that space can be learned by FC layer 170 to calculate final scores for multiple classes. Each score can represent a quantified similarity between the alphabet in the input image and the corresponding class.

[0034] FIG. 2 illustrates an exemplary convolution process according to various embodiments. A typical convolution process in a neural network can begin by receiving (or obtaining) one or more input tensors and multiple filters and performing a convolution process to generate one or more output tensors. For simplicity, the exemplary convolution process 220 in FIG. 2 includes one input tensor 222, one filter 224, and one output tensor 225. The input tensor 222 can be represented by its height (H), width (W), and number of channels (C). As shown in FIG. 2, the input tensor 222 can be a 3*3*3 matrix with a height of 3, a width of 3, and a number of channels of 3. The height and width of the input tensor 222 for each channel (e.g., 3(H)*3(W)) can be referred to as a 2D input feature map. A tuple of {h,w,c} can refer to one element of the 2D input feature map for channel c. For example, {2,1,1} can refer to an element of the 2D input feature map in channel 1, whose position is defined by {2,1} (e.g., the height index is 2 and the width index is 1). The filter 224 can be represented by its height (R), width (S), and the number of channels (C). In FIG. 2, the filter 224 can be a 1*1*3 matrix. The height and width of the filter 224 in each channel (e.g., 1(R)*1(S)) can be referred to as a kernel (the filter 224 has three kernels for each of the three channels).

[0035] In some scenarios, the convolution process can include stride and padding. For example, if the stride is 1, the filter convolves (e.g., moves, slides) one pixel at a time, and if the stride is 2, the filter convolves (e.g., slides) two pixels at a time. A larger stride produces a spatially smaller output tensor (smaller H*W in the output tensor). As another example, if the spatial size (height*width) of the filter is greater than 1*1 (e.g., 3*3 or 5*5), the input tensor can be padded with zeros around the border to preserve the spatial size of the input tensor, for example, so that the height and width of the input and output are the same, after applying the filter, in order to control the spatial size of the output tensor. In FIG. 2, no padding is performed on the input tensor 222, and the stride is assumed to be 1.

[0036] During the convolution process 220 shown in FIG. 2, a MAC operation is performed on the filter 224 and each depth slice of the input tensor, such as the first depth slice 223, to generate a dot product, such as dot product 228. For example, the first depth slice 223 of the input tensor 222 is a 1*1*3 tensor (three gray cubes) located at the top left of the input tensor 222. Both the first depth slice 223 and the filter 224 have a size of 1*1*3. After the MAC operation, the generated dot product 228 can be assembled as part of the output tensor 225. Thus, the output tensor 225 can be determined after the filter 224 has convolved (e.g., moved) through all depth slices in the input tensor 222 (9 slices in FIG. 2). The number of channels in the output tensor 225 is equal to the number of filters applied during the convolution. Because the convolution process 220 uses only one filter 224, the corresponding output tensor 228 has only one channel.

[0037] In comparison, convolution process 260 includes two filters 264, 265. By convolving filter 264 through input tensor 262, the value of output tensor 268 in a first channel (e.g., a 2D output feature map in channel 1) can be determined. By convolving filter 265 through input tensor 262, the value of output tensor 268 in a second channel (e.g., a 2D output feature map in channel 2) can be determined. Thus, the resulting output tensor 268 comprises two channels of 2D output feature maps.

[0038] Note that some operations during the convolution process can be parallelized. For example, different filters generate 2D output feature maps for different output channels, so the MAC operation performed by one filter is independent of the MAC operation of another filter. As another example, depth slices in an input tensor generate values ​​for different locations in each 2D output feature map, so they are independent of each other. If all these operations could be parallelized across multiple processing entities (e.g., processors, cores, or threads), theoretically ideal parallel processing could be achieved. However, because real-world applications may have a large number of filters and a large number of input tensors, pursuing theoretically ideal parallel processing may require hardware with unachievable computing capabilities (e.g., a huge number of processing entities and memory exchange mechanisms). The embodiments disclosed in the following description provide a method (and corresponding system, storage medium) for achieving ideal parallel processing by fully utilizing the parallel processing capabilities of given hardware, which is a more scalable and practical approach.

[0039] FIG. 3 illustrates an example diagram 300 of neural network layers and an example diagram 340 of weight pruning and activation pruning, according to various embodiments. Diagram 300 illustrates an abstract view of neural network layers, specifically convolutional layers. A convolutional layer in FIG. 3 includes (1) multiple input activations 312 (e.g., input values ​​in an input tensor) denoted as x1, x2, and x3, which may refer to input values ​​of one or more input tensors, and (2) filter / weight tensor / feature extractor weights, W 1,1 (3) optionally, one or more biases 316 for adjusting the activations 312 (along with the weights 314); (4) an activation function 318 for determining whether the neuron should be activated; and (5) a plurality of output activations 320 (e.g., output values ​​in an output tensor) shown as y1, y2, and y3. The diagram 300 shows only one weight 314 for illustrative purposes; more weights may be included. Exemplary activation functions 318 may include sigmoid, hyperbolic tangent (TanH), ReLu, or another suitable activation function.

[0040] Drawing 340 in FIG. 3 illustrates an example neural network with weight pruning 342 and activation pruning 344. Weight pruning 342 and activation pruning 344 can introduce sparsity into the convolution process, reducing energy and achieving higher throughput. As shown, weight pruning 342 can reduce the number of non-zero weights in the filter so that fewer convolution operations need to be performed between the input activations (e.g., x1, x2, x3) and the weights. In some embodiments, activation pruning 344 can include applying masks (e.g., t1, t2, t3) to the output activations (e.g., y1, y2, y3). The masks may set some of the output activations to zero (e.g., deactivate the outputs). In FIG. 3, mask t2 sets output activation y2 to zero, while masks t1 and t3 leave the corresponding output activations y1 and y3 unchanged. In this way, the resulting output activations are pruned to contain some zero values, which can help reduce the computational cost of the next layer.

[0041] However, the mask-based method illustrated in Figure 3 may fail to balance (e.g., evenly distribute) non-zero output activations. Because convolutional processing is typically performed in parallel processing architectures using multiple processors, imbalances in output activations can lead to workload imbalances between processors, limiting the scalability and throughput of the architecture. In the following description, workload-balanced activation pruning methods and architectures are described using exemplary embodiments.

[0042] FIG. 4 shows an example diagram of workload-leveled activation pruning in accordance with various embodiments. Exemplary neural network layer 400 may refer to another perspective of exemplary neural network layer 300 or neural network layer 340 in FIG. 3 . Exemplary neural network layer 400 may begin with a convolution process between an input tensor X and multiple filter / weight tensors and may optionally include a bias layer that conditions the convolution result before generating output activations. Taking one filter, W1, as an example, the convolution process between input tensor X and filter W1 (with a corresponding bias tensor B1) may generate a portion of output tensor Y1.

[0043] In some embodiments, a portion of the output tensor Y1 can be a dense activation matrix represented as a table 410. The matrix 410 is “dense” because it may contain a large number of non-zero values. In some embodiments, the dense matrix 410 can be pruned into a smoothed sparse activation matrix 420 by constructing one or more banks of output values, each containing the same number of non-zero output values. As shown in FIG. 4, two banks of output values ​​can be constructed based on each row of the dense matrix 410. Each row of the dense matrix 410 can contain eight non-zero output values, and the two banks can evenly divide the row so that each bank contains four non-zero output values.

[0044] In some embodiments, to prune activations in a leveled manner, the same number of output values ​​in each bank can be retained as non-zero, and other output values ​​can be pruned to zero. In some embodiments, the output values ​​retained as non-zero in each bank can be determined by a top K sorting operation. The top K sorting operation in each bank can select the K non-zero values ​​with the largest magnitudes and retain those values, while other non-zero values ​​can be set to zero. As shown in FIG. 4, a first bank can be constructed using the top-left four non-zero values ​​(0.2, 0.1, −0.4, and 0.8) in the dense matrix 410. The top two sorts in the first bank (e.g., K=2) can find the two non-zero values ​​with the largest magnitudes (e.g., absolute values), in this case −0.4 and 0.8. Subsequently, −0.4 and 0.8 in the first bank are retained, while the other values, 0.2 and 0.1, are pruned to zero. The same process can be performed in parallel on other banks to generate a smoothed sparse activation matrix 420 .

[0045] The activation process described above can be customized in various ways to achieve desired results. For example, the value of K can be set to a large number if more non-zero activations are preferred, or to a small number if greater sparsity is preferred. As another example, the size of the bank can be configured to determine how the non-zero activations are distributed in the output tensor.

[0046] For clarity, the output activations of layer 400 can be represented as an H*W*C output tensor 430, where H is the height dimension, W is the width dimension, and C is the channel dimension. In some embodiments, each dimension of output tensor 430 can be segmented into groups. For example, the H dimension of output tensor 430 can be divided into three groups, each represented as an Hg (height group). Similarly, the W dimension can be divided into three Wg (width groups), and the C dimension can be divided into three Cg (channel groups). In FIG. 4, the Y1 activation is assumed to correspond to the rightmost face 432 of the output tensor. Therefore, the rightmost face 432 can be represented as the dense activation matrix 410 before pruning or the smoothed sparse activation matrix 420 after pruning. FIG. 4 illustrates one exemplary configuration of banks, with each Hg*Wg*Cg sub-tensor forming a bank. For example, the shaded Hg*Wg*Cg portion of surface 432 may be referred to as a bank, which may be represented as the top-left bank of flattened sparse matrix 420. Depending on the implementation, the bank may have a different shape by setting Hg, Wg, and Cg to different values. In some embodiments, Cg may be configured to be an integer greater than 1 (at least 2), Hg may be configured to be an integer greater than 0, and Wg may be configured to be an integer greater than 0.

[0047] FIG. 5 illustrates an exemplary system diagram for workload-leveled convolution with activation sparsity, according to various embodiments. The architecture illustrated in FIG. 5 includes a computer system designed to perform inference on a neural network as it is loaded and executed. In some embodiments, such a computer system is implemented as a hardware accelerator 500. Some components in the hardware accelerator 500 can work together in a specific manner to facilitate parallel convolution computations across multiple processing entities, thereby improving the throughput and energy efficiency of the neural network. The accelerator 500 is merely exemplary and may include more, fewer, or alternative components. The hardware accelerator 500 can be designed as a reconfigurable device, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0048] As shown in FIG. 5, the hardware accelerator 500 may include a scheduler 570 for controlling the workflow within the accelerator 500 and its interaction with off-chip components, such as a host CPU 510 and a double data rate (DDR) memory 520. For example, the accelerator 500 may interact with the host CPU 510 through a peripheral component interconnect express (PCIe) physical layer (PHY) controller 512 and with the off-chip DDR memory 520 through a DDR interface 530. The accelerator 500 may fetch data from the off-chip DDR memory 520 through a direct memory access (DMA) controller 540, which communicates with the off-chip DDR memory 520 through the DDR interface 530. The fetched data may be stored in an on-chip buffer called a global buffer 550 in preparation for parallel convolution computations. The global buffer 550 may be logically divided into multiple sections, such as an input buffer 552, a weight buffer 554, and an output buffer 556. The input buffer 552 can store data associated with input tensors, the weight buffer 554 can store data associated with filters, and the output buffer 556 can store data associated with the results of convolution computations (e.g., output tensors). The global buffer 550 can exchange data with an array 560 of sparse processing units (SPUs) 562 to perform the convolution computations. The "sparse" in the SPU name indicates that the SPU array 560 is specifically designed to efficiently perform convolution computations involving sparsity (e.g., sparse input tensors and / or sparse filters). Each of the SPUs 562 can include multiple processing entities (PEs) 580, each capable of processing a unit amount of convolution computations.

[0049] In some embodiments, each of the SPUs 562 may further include a module for pruning output values ​​of the convolution calculations performed by the multiple PEs 580 therein. For example, the “Dense2Sparsity” module 564 in FIG. 5 may prune output values ​​from the multiple PEs 580 using the following method: constructing one or more banks of output values ​​based on the multiple output values ​​(generated by the multiple PEs 580), each bank comprising one or more multiple output values; for each bank, performing a top K sort on the one or more output values ​​in the bank to obtain K non-zero output values, where K is a positive integer greater than zero; and pruning each bank by setting one or more output values ​​other than the K non-zero output values ​​obtained in the bank to zero. In some embodiments, one K may correspond to one layer in a neural network. That is, different layers in a neural network may have different K values. The K values ​​for a layer may be learned through machine learning.

[0050] In some embodiments, one or more banks may be equal in size (e.g., each bank contains the same number of output values) and, after pruning, may contain the same number of non-zero output values. The positions of the non-zero output values ​​in each bank may be random, but the total number of non-zero output values ​​in each bank (e.g., an integer K greater than 1) may be the same.

[0051] FIG. 6A illustrates an exemplary system architecture 600 for generating an output tensor with workload-leveled activation sparsity, according to various embodiments. The exemplary system architecture 600 may include a plurality of PEs 630 configured to perform convolution computations in parallel; an adder tree 640 configured to perform additions on outputs (e.g., partial sums) generated by the plurality of PEs 630; an accumulation buffer 650 configured to accumulate outputs (e.g., output values ​​to be assembled into an output tensor) generated by the adder tree 640; and a top-K sorting device 660 configured to prune the outputs accumulated by the accumulation buffer 650. The pruning process performed by the top-K sorting device 660 may generate sparse output activations 680 with non-zero activations distributed in a leveled manner. In some embodiments, the top-K sorting device 660 may correspond to the “Dense2Sparsity” module 564 in FIG. 5. In some embodiments, the adder tree 640 may refer to part of a device or a standalone device.

[0052] In the following description, an exemplary convolution process in a convolution layer is used to explain how an output tensor with workload-leveled activation sparsity is generated for the convolution layer. In some embodiments, the exemplary convolution process can begin by assigning an input tensor 610 and one or more weight tensors 620 to multiple processors (PEs 630) in the convolution layer to perform multiply-accumulate (MAC) operations based on the input tensor 610 and the weight tensor 620 in parallel.

[0053] In some embodiments, the input tensor 610 may be segmented into multiple sub-tensors and assigned to multiple PEs 630, and the weight tensor 620 may be segmented into multiple weight groups and similarly assigned to multiple PEs 630. In some embodiments, the "assignment" of the sub-tensors and weight groups may refer to storing the sub-tensors and weight groups in an internal buffer of the PE 630.

[0054] In some embodiments, after receiving the assigned sub-tensors and weight groups, the multiple PEs 630 may perform MAC operations in parallel to obtain multiple partial sums. These partial sums may be provided to an adder tree 640 to obtain one or more output values, which may be accumulated and stored in an accumulation buffer 650. When the output values ​​in the accumulation buffer 650 are sufficient to form a full bank, the output values ​​may be provided to the top-K sorting devices 660 for pruning. In some embodiments, the size of the bank is predetermined. An exemplary configuration of the bank may refer to the description in FIG. 4. In some embodiments, a portion of the output values ​​in the accumulation buffer 650 may be fed back to the adder tree 640 to be added to the newly generated partial sum by the PE 630. In some embodiments, when a full bank of output values ​​is provided to the top-K sorting devices 660, a controller or scheduler (e.g., scheduler 570 in FIG. 5) may send a corresponding index of the bank 670 to the top-K sorting devices 660. If multiple banks of output values ​​are provided to the top-K sorting devices 660, the controller or scheduler may send an array of bank indexes 670 to the top-K sorting devices 660.

[0055] Depending on the granularity of the segmentation of the input tensor 610 / weight tensor 620 and the size of the bank, the accumulation buffer 650 may generate full banks of output values ​​at different rates. In some embodiments, multiple PEs 630 may need to perform parallel convolution processing over multiple cycles (also referred to as parallel processing cycles) for the accumulation buffer 650 to accumulate one full bank of output values. In some embodiments, the accumulation buffer 650 may accumulate one or more full banks of output values ​​based on the partial sums generated by the multiple PEs 630 during each parallel processing cycle.

[0056] To simplify the discussion of the relationship between segmentation granularity and full bank generation rate, it is assumed that there is only one input tensor, with a bank size denoted by K. For example, if a PE handles one weight tensor and corresponding input values ​​in parallel during each parallel processing cycle, one full bank of output values ​​can be accumulated every K parallel processing cycles. As another example, if a PE handles K weight tensors and corresponding input values ​​in parallel during each parallel processing cycle, one full bank of output values ​​can be generated every parallel processing cycle. As yet another example, if a PE handles N*K weight tensors and corresponding input values ​​in parallel during each parallel processing cycle, N full banks of output values ​​can be generated for each parallel processing cycle.

[0057] In some embodiments, the top-K sorting device 680 performs top-K sorting logic on the magnitudes of the output values ​​in the banks to obtain the K output values ​​with the largest magnitudes. The top-K sorting logic may be implemented by a hardware-based bitonic sorter combined with at least one of the following software sorting algorithms: a heap sorting algorithm, a merge sorting algorithm, a quick sorting algorithm, or another suitable sorting algorithm. In some embodiments, K output values ​​may be kept as is (e.g., as their original non-zero values), and the other output values ​​may be pruned as zeros. Assuming each bank has N output values, the pruning process described above may leave each bank with the K non-zero values ​​with the largest magnitudes and (N) zero values. By incorporating sparsity into the banks and distributing the non-zero output values ​​(non-zero activations) in a leveled manner, the output activations assembled from the banks may provide an input tensor with workload-leveled activation sparsity for the next convolutional layer.

[0058] In some embodiments, the pruned banks can be assembled to construct the output tensor of a convolutional layer, which is sparsity and smoothed (e.g., non-zero activations are distributed in a smoothed manner across banks), benefiting parallel convolution processing in the next convolutional layer by allowing multiple PEs to distribute the workload evenly.

[0059] FIG. 6B illustrates an exemplary flowchart for generating an output tensor with workload-leveled activation sparsity, according to various embodiments. The flowchart in FIG. 6 describes a simplified version of the convolution process in a convolutional layer during one parallel processing cycle. For simplicity and clarity, assume that H*W*C input tensors 610 have already been segmented into four Hg*Wg*Cg input sub-tensors (e.g., the shaded regions of the input tensors 610 in FIG. 6 ), that a weight group 620 contains four weight tensors that share the same dimensions as the input sub-tensors, and that there are four PEs 630 available to perform the MAC operation in parallel. These assumptions may be relaxed or modified depending on the actual use case, and necessary adjustments to the workflow can be made accordingly.

[0060] In some embodiments, during one parallel processing cycle, each of the PEs 630 can handle one input sub-tensor and one weight tensor in a weight group 620. Here, "handling" can refer to performing a MAC operation between the input values ​​in the input sub-tensor and the weights in the weight tensor. The PEs 630 can perform local MAC operations in parallel. In some embodiments, the PEs 630 can generate multiple partial sums after the MAC operation. These partial sums can be provided to an adder tree device 632 to perform the necessary additions, such as adding the multiple partial sums together to obtain an output value. In some embodiments, the output values ​​generated by the adder tree device 632 can be provided to an accumulation buffer 634 for accumulation. The accumulation buffer 634 can continue to accumulate output values ​​to generate a full bank 640 of output values. In some embodiments, the size of the bank can be predetermined.

[0061] In some embodiments, the system architecture, input / weight tensor segmentation, and workload distribution can be configured in a manner that improves the efficiency of generating a full bank of output values. In the example shown in FIG. 6, each PE 630 handles one combination of input sub-tensor and weight tensor, which generates one output value during a parallel processing cycle. The four output values ​​generated by the four PEs 630 can form one bank 640 during a parallel processing cycle. In this manner, the example setup in FIG. 6 generates one bank 640 of output values ​​after each parallel processing cycle. In some embodiments, the weight tensor and / or input sub-tensor can be rotated among the PEs 630 for multiple parallel processing cycles until the convolution between the input tensor 610 and the weight group 620 is completed. The multiple banks generated by the PEs 630 during multiple parallel processing cycles can be pruned and assembled to form the output tensor of the convolution layer.

[0062] In some embodiments, the bank 640 of output values ​​can be provided to a top K sort component 642 for pruning. The top K sort component 642 can perform top K sorting operations on the output values ​​in the bank 640 to identify the K output values ​​with the largest magnitudes (e.g., absolute values). Depending on the implementation, the top K sort component 642, the adder tree 632, and the accumulation buffer 634 can be implemented as separate devices or as part of an integrated device. These K output values ​​can retain their values, and the other output values ​​in the bank 640 can be pruned to zero. For example, the bank 640 generated from the accumulation buffer 634 in FIG. 6 can include four non-zero output values ​​(shaded), which can be pruned to a sparse bank 650 after a top-two sort is performed. The top-two sort can identify the two with the largest magnitudes of the four output values ​​and set the other two values ​​to zero.

[0063] In some embodiments, after the bank 640 is pruned into the sparse bank 650 by the top K sorting operation, the K nonzero values ​​in the result / sparse bank 650 may not need to be evenly distributed therein. However, because each sparse bank 650 contains the same number of nonzero output values ​​(activations), the output tensor assembled based on all sparse banks is smoothed and sparse. When the sparse output tensor becomes the input tensor for the next convolutional layer, multiple PEs can be assigned a smoothed workload to perform the convolution in parallel (e.g., each PE receives the same number of nonzero input values ​​during each parallel processing cycle).

[0064] FIG. 7 illustrates an exemplary method for workload-leveled convolution with activation sparsity, according to various embodiments. Method 700 may be performed by a device, apparatus, or system for optimizing resource allocation. Method 700 may be performed by one or more modules / components of the environment or system illustrated by FIGS. 1-6, such as hardware accelerator 500 in FIG. 5. The operations of method 700 presented below are for illustrative purposes. Depending on the implementation, method 700 may include additional, fewer, or alternative steps performed in various orders or in parallel.

[0065] Block 710 includes assigning input tensors and weight tensors in the convolutional layer to multiple processors to perform multiply-accumulate (MAC) operations based on the input tensors and weight tensors in parallel. In some embodiments, assigning input tensors and weight tensors in the convolutional layer to multiple processors comprises obtaining an input tensor and multiple weight tensors in a convolutional layer of the neural network, segmenting the input tensor into multiple sub-tensors and assigning the multiple sub-tensors to the multiple processors, and segmenting the multiple weight tensors into multiple weight groups and assigning the multiple weight groups to the multiple processors. In some embodiments, assigning the multiple sub-tensors and multiple weight groups to the multiple processors comprises, for each of the multiple processors, storing one or more sub-tensors of the multiple sub-tensors and one or more weight groups of the multiple weight groups in an internal buffer of the processor.

[0066] In some embodiments, allocating the plurality of weight groups to the plurality of processors comprises allocating one or more weight groups of the plurality of weight groups to the plurality of processors for performing the MAC operation in parallel based on the plurality of sub-tensors and the one or more weight groups of the plurality of weight groups during each of the plurality of parallel processing cycles. In some embodiments, each of the one or more banks corresponds to an output value generated based on one weight group of the plurality of weight groups.

[0067] Block 720 includes obtaining a plurality of output values ​​based on results of the MAC operation. In some embodiments, obtaining a plurality of output values ​​based on results of the MAC operation comprises obtaining a plurality of partial sums as a result of the MAC operation performed by the plurality of processors during each of the plurality of parallel processing cycles, providing the plurality of partial sums to an adder tree device to obtain one or more output values, and storing the one or more output values ​​in an accumulation buffer during one or more parallel processing cycles of the plurality of parallel processing cycles to obtain the plurality of output values.

[0068] Block 730 includes constructing one or more banks of output values ​​based on the plurality of output values, each of the banks comprising one or more output values ​​of the plurality of output values. In some embodiments, each of the banks comprises a height dimension greater than zero, a width dimension greater than zero, and a channel dimension greater than one. In some embodiments, the method further comprises representing the K non-zero output values ​​in each of the banks as K multi-dimensional vectors, each of the K multi-dimensional vectors comprising the non-zero output value and a corresponding channel dimension identifier of the non-zero output value in the bank. In some embodiments, constructing an output tensor of the convolutional layer based on the pruned banks comprises assembling the K multi-dimensional vectors of each of the banks to form an output tensor.

[0069] Block 740 includes, for each of the banks, performing a top K sort on one or more output values ​​in the bank to obtain K output values, where K is a positive integer. In some embodiments, performing a top K sort on the output values ​​in the bank to obtain the K output values ​​comprises performing a top K sort on magnitude of the output values ​​in the bank to obtain the K output values.

[0070] Block 750 includes pruning each of the banks by setting one or more output values ​​other than the K output values ​​obtained in each bank to zero.

[0071] Block 760 includes constructing an output tensor of the convolutional layer based on the pruned banks. In some embodiments, each of the pruned banks is associated with a bank identifier, and constructing the output tensor of the convolutional layer based on the pruned banks comprises assembling the K multidimensional vectors of each of the banks to form an output tensor based on the corresponding bank identifier. In some embodiments, the output tensor is a sparse tensor.

[0072] Figure 8 illustrates an exemplary computing device capable of implementing any of the embodiments described herein. The computing device can be used to implement one or more components of the systems and methods shown in Figures 1-7. The computing device 800 can include a bus 802 or other communication mechanism for communicating information and one or more hardware processors 804 coupled to the bus 802 for processing information. The hardware processors 804 can be, for example, one or more general-purpose microprocessors.

[0073] The computing device 800 may also include a main memory 807, such as a random access memory (RAM), cache, and / or other dynamic storage device, coupled to the bus 802 for storing information and instructions executed by the processor 804. The main memory 807 may also be used to store temporary variables or other intermediate information during execution of instructions executed by the processor 804. When such instructions are stored on a storage medium accessible to the processor 804, the computing device 800 may become a special-purpose machine customized to perform the operations specified in the instructions. The main memory 807 may include non-volatile and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, a floppy disk, flexible disk, hard disk, solid state drive, magnetic tape or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with a pattern of holes, RAM, DRAM, PROM, EPROM, FLASH®-EPROM, NVRAN, any other memory chip or cartridge, or networked versions of these.

[0074] Computing device 800 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, when combined with the computing device, renders computing device 800 a special-purpose machine or that can be programmed. According to one embodiment, the techniques described herein are performed by computing device 800 in response to processor 804 executing one or more sequences of one or more instructions contained in main memory 807. Such instructions may be loaded into main memory 807 from another storage medium, such as storage device 809. Execution of the sequence of instructions contained in main memory 807 causes processor 804 to perform the process steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 807. These instructions, when executed by processor 804, may perform the steps illustrated in the corresponding figures and described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0075] Computing device 800 also includes a communication interface 810 coupled to bus 802. The communication interface 810 may provide a two-way data communication coupling to one or more network links that are connected to one or more networks. As another example, communication interface 810 may be a local area network (LAN) card that provides a data communication connection to a compatible LAN (or a WAN component in communication with the WAN). Wireless links may also be implemented.

[0076] Performance of a particular operation may be distributed among processors, which may reside within a single machine or may be spread across multiple machines. In some exemplary embodiments, a processor, or an engine implemented by a processor, may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other exemplary embodiments, a processor, or an engine implemented by a processor, may be distributed across multiple geographic locations.

[0077] Each of the processes, methods, and algorithms described in the previous sections can be embodied in, and fully or partially automated by, one or more computer systems or computer processors comprising computer hardware. The processes and algorithms can be implemented partially or wholly in application-specific circuitry.

[0078] When the functions disclosed herein are implemented in the form of software functional units and sold or used as an independent product, the functions may be stored in a processor-executable, non-volatile, computer-readable storage medium. The specific technical solutions (in whole or in part) disclosed herein or aspects that contribute to the current art may be embodied in the form of a software product. The software product may be stored in a storage medium that includes many instructions for causing a computing device (which may be a personal computer, a server, a network device, etc.) to perform all or some of the steps of the method of the present application. The storage medium may include a flash drive, a portable hard drive, a ROM, a RAM, a magnetic disk, an optical disk, another medium operable to store program code, or any combination thereof.

[0079] Certain embodiments further provide a system comprising a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in the method of any of the above-disclosed embodiments. Certain embodiments further provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in the method of any of the above-disclosed embodiments.

[0080] The embodiments disclosed herein may be implemented by a cloud platform, a server, or a group of servers (collectively, hereinafter, the "service system") that interacts with a client. The client may be a terminal device or a client registered with the platform by a user. The terminal device may be a mobile terminal, a personal computer (PC), or any device on which a platform application program can be installed.

[0081] The various features and processes described above can be used independently of one another or can be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. In addition, certain method or process blocks may be omitted in some implementations. Also, the methods and processes described herein are not limited to any particular sequence, and the associated blocks or states may be performed in other suitable sequences. For example, the described blocks or states may be performed in an order other than the order specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be performed serially, in parallel, or in other manners. Blocks or states may be added or deleted from the disclosed example embodiments. The example systems and components described herein may be configured differently from that described. For example, elements may be added, deleted, or rearranged compared to the disclosed example embodiments.

[0082] Various operations of the example methods described herein may be performed, at least in part, by algorithms. The algorithms may be included in program code or instructions stored in memory (e.g., the non-transitory computer-readable storage medium described above). Such algorithms may comprise machine learning algorithms. In some embodiments, the machine learning algorithms may not explicitly program a computer to perform a function, but may learn from training data to create a predictive model that performs the function.

[0083] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute a processor-implemented engine that operates to perform one or more operations or functions described herein.

[0084] Similarly, the methods described herein may be implemented at least in part by a processor, with one or more particular processors being examples of hardware. For example, at least some of the operations of a method may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may operate to support performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), and the operations may be accessible over a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application program interfaces (APIs)).

[0085] Performance of a particular operation may be distributed among processors, and may reside within a single machine or may be spread across multiple machines. In some exemplary embodiments, a processor, or an engine implemented by a processor, may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other exemplary embodiments, a processor, or an engine implemented by a processor, may be distributed across multiple geographic locations.

[0086] Throughout this specification, components, operations, or structures described as a single instance may be implemented by multiple instances. While individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and the operations need not be performed in the order illustrated. In example configurations, structures and functions presented as separate components may be implemented as combined structures or components. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter of this specification.

[0087] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or dictated otherwise by context. Thus, herein, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C," unless expressly indicated otherwise or dictated otherwise by context. Furthermore, "and" means both in combination and individually, unless expressly indicated otherwise or dictated otherwise by context. Thus, herein, "A and B" means "A and B, in combination or individually," unless expressly indicated otherwise or dictated otherwise by context. Furthermore, for any resource, operation, or structure described herein as a single instance, multiple instances may be provided. Additionally, boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are contemplated and may be within the scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the exemplary configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements are included within the scope of the embodiments of the present disclosure as expressed by the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense, rather than a restrictive sense.

[0088] The terms "comprises" or "comprises" are used to indicate the presence of subsequently declared features but do not exclude the addition of other features. In particular, conditional language such as "can," "could," "may," or "can," unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to generally convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not generally intended to suggest that features, elements, and / or steps are somehow required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or should be performed in any particular embodiment, with or without user input or prompting.

[0089] Although the subject matter has been generally described with reference to certain exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the disclosed embodiments. Such embodiments of the subject matter may be referred to herein, individually or collectively, by the term "invention" for convenience only, without intending to intentionally limit the scope of the application to any single disclosure or concept when, in fact, multiple disclosures or concepts are disclosed.

[0090] The embodiments set forth herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description is not to be construed in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A computer-implemented method comprising: Allocating input tensors and weight tensors in a convolutional layer to a plurality of processors and performing multiply-accumulate (MAC) operations in parallel based on the input tensors and the weight tensors; obtaining a plurality of output values ​​based on the results of the MAC operation; constructing one or more banks of output values ​​based on the plurality of output values, each of the banks comprising one or more of the plurality of output values; for each of the banks, performing a top K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer; pruning each of the banks by setting the one or more output values ​​other than the obtained K output values ​​in each bank to zero; constructing an output tensor of the convolutional layer based on the pruned bank; and A method comprising:

2. The allocating of input tensors and weight tensors in a convolutional layer to a plurality of processors includes: obtaining an input tensor and a plurality of weight tensors in the convolutional layer of the neural network; Segmenting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; segmenting the plurality of weight tensors into a plurality of weight groups and assigning the plurality of weight groups to the plurality of processors; The method of claim 1 , comprising:

3. 3. The method of claim 2, wherein the assigning the plurality of sub-tensors and the plurality of weight groups to a plurality of processors comprises storing, for each of the plurality of processors, one or more sub-tensors of the plurality of sub-tensors and one or more weight groups of the plurality of weight groups in an internal buffer of the processor.

4. The assigning of the plurality of weight groups to the plurality of processors includes:

3. The method of claim 2, further comprising: during each of a plurality of parallel processing cycles, assigning one or more weight groups of the plurality of weight groups to the plurality of processors, and causing the plurality of processors to perform the MAC operation in parallel based on the plurality of sub-tensors and the one or more weight groups of the plurality of weight groups.

5. The method of claim 4 , wherein each of the one or more banks corresponds to an output value generated based on one weight group of the plurality of weight groups.

6. The obtaining of a plurality of output values ​​based on the result of the MAC operation includes: obtaining a plurality of partial sums as a result of the MAC operations performed by the plurality of processors during each of a plurality of parallel processing cycles; providing the plurality of partial sums to an adder tree device to obtain one or more output values; storing the one or more output values ​​in an accumulation buffer during one or more parallel processing cycles of the plurality of parallel processing cycles to obtain the plurality of output values; The method of claim 1 , comprising:

7. performing a top K sort on the output values ​​in the bank to obtain K output values, 2. The method of claim 1, further comprising performing a top K sort by magnitude of the output values ​​in the bank to obtain the K output values.

8. Each of the banks comprises: a height dimension greater than zero, a width dimension greater than zero; channel dimension greater than 1 and The method of claim 1 , comprising:

9. 2. The method of claim 1, further comprising representing the K non-zero output values ​​in each of the banks as K multi-dimensional vectors, each of the K multi-dimensional vectors comprising a non-zero output value and a corresponding channel dimension identifier for the non-zero output value in the bank.

10. constructing the output tensor of the convolutional layer based on the pruned bank includes:

10. The method of claim 9, comprising assembling the K multidimensional vectors of each of the banks to form the output tensor.

11. each of the pruned banks is associated with a bank identifier; constructing the output tensor of the convolutional layer based on the pruned bank includes:

10. The method of claim 9, comprising assembling the K multidimensional vectors for each of the banks based on the corresponding bank identifiers to form the output tensor.

12. The method of claim 1 , wherein the output tensor is a sparse tensor.

13. 1. A system comprising: one or more processors; and instructions coupled to the one or more processors and executable by the one or more processors, Allocating input tensors and weight tensors in a convolutional layer to a plurality of processors and performing multiply-accumulate (MAC) operations in parallel based on the input tensors and the weight tensors; obtaining a plurality of output values ​​based on the results of the MAC operation; constructing one or more banks of output values ​​based on the plurality of output values, each of the banks comprising one or more of the plurality of output values; for each of the banks, performing a top K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer; pruning each of the banks by setting the one or more output values ​​other than the obtained K output values ​​in each bank to zero; constructing an output tensor of the convolutional layer based on the pruned bank; and and one or more non-transitory computer-readable memories configured to cause the system to perform operations comprising:

14. The allocating of input tensors and weight tensors in a convolutional layer to a plurality of processors includes: obtaining an input tensor and a plurality of weight tensors in the convolutional layer of the neural network; Segmenting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; segmenting the plurality of weight tensors into a plurality of weight groups and assigning the plurality of weight groups to the plurality of processors; The system of claim 13, comprising:

15. The obtaining of a plurality of output values ​​based on the result of the MAC operation includes: obtaining a plurality of partial sums as a result of the MAC operations performed by the plurality of processors during each of a plurality of parallel processing cycles; providing the plurality of partial sums to an adder tree device to obtain one or more output values; storing the one or more output values ​​in an accumulation buffer during one or more parallel processing cycles of the plurality of parallel processing cycles to obtain the plurality of output values; The system of claim 13, comprising:

16. performing a top K sort on the output values ​​in the bank to obtain K output values, 14. The system of claim 13, further comprising performing a top K sort on magnitude of the output values ​​in the bank to obtain the K output values.

17. A non-transitory computer-readable storage medium having instructions executable by one or more processors, Allocating input tensors and weight tensors in a convolutional layer to a plurality of processors and performing multiply-accumulate (MAC) operations in parallel based on the input tensors and the weight tensors; obtaining a plurality of output values ​​based on the results of the MAC operation; constructing one or more banks of output values ​​based on the plurality of output values, each of the banks comprising one or more output values ​​of the plurality of output values; for each of the banks, performing a top K sort on the one or more output values ​​in the bank to obtain K output values, where K is a positive integer; pruning each of the banks by setting the one or more output values ​​other than the obtained K output values ​​in each bank to zero; constructing an output tensor of the convolutional layer based on the pruned bank; and 1. A non-transitory computer-readable storage medium configured to cause one or more processors to perform operations comprising:

18. The allocating of input tensors and weight tensors in a convolutional layer to a plurality of processors includes: obtaining an input tensor and a plurality of weight tensors in the convolutional layer of the neural network; Segmenting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; segmenting the plurality of weight tensors into a plurality of weight groups and assigning the plurality of weight groups to the plurality of processors; 20. The non-transitory computer-readable storage medium of claim 17, comprising:

19. The obtaining of a plurality of output values ​​based on the result of the MAC operation includes: obtaining a plurality of partial sums as a result of the MAC operations performed by the plurality of processors during each of a plurality of parallel processing cycles; providing the plurality of partial sums to an adder tree device to obtain one or more output values; storing the one or more output values ​​in an accumulation buffer during one or more parallel processing cycles of the plurality of parallel processing cycles to obtain the plurality of output values; 20. The non-transitory computer-readable storage medium of claim 17, comprising:

20. performing a top K sort on the output values ​​in the bank to obtain K output values, 20. The non-transitory computer-readable storage medium of claim 17, further comprising performing a top K sort on magnitude of the output values ​​in the bank to obtain the K output values.

Images