Adaptive Tensor Convolution Kernels for Sparse Neural Networks

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, are provided for efficient neural network computation using adaptive tensor computation kernels. First, the adaptive tensor computation kernel can change shape according to different shapes of input tensors / weight tensors to distribute weights and input values ​​to processing element (PE) arrays for parallel processing. Depending on the shape of the tensor computation kernel, additional adders between or within clusters may be required to perform convolution operations. Second, the adaptive tensor computation kernel can support two different tensor computation modes, a 1×1 tensor computation mode and a 3×3 tensor computation mode, to accommodate all kinds of convolution computations. Third, the underlying PE array can configure each PE internal buffer (e.g., register file) differently to accommodate different compression ratios and sparse granularity of sparse neural networks.

Description

[Technical field]

[0001] The present disclosure relates generally to improving the computational efficiency of neural networks, and more particularly to dynamically adjusting native tensor dimensions and operation modes to accommodate different input tensor shapes and operations in sparse neural networks. [Background technology]

[0002] Deep learning PE (processing element) arrays are almost entirely defined by native tensor dimensions and operation modes, and typically rely on the compiler to use different nested loop mapping approaches to accommodate different tensor shapes (e.g., input feature maps and filters) and operations. It is obviously inefficient to use PE arrays to perform computations on tensor shapes and operations that are incompatible with the PE array's native tensor dimensions and operation modes. This incompatibility is exacerbated in sparse neural networks with their sparsity properties, because the inflexible PE array's native tensor shapes and operation modes cannot efficiently represent or process tensors with a large number of zeros. Summary of the Invention

[0003] Various embodiments herein may include systems, methods, and non-transitory computer-readable media for using adaptive tensor computation kernels in neural network computations.

[0004] According to one aspect, a method for using adaptive tensor computation kernels in neural network computation includes receiving a first input feature map (IFM) and one or more first filters at a first layer of a convolutional neural network (CNN) for convolution using a processing element (PE) array, where each PE in the PE array comprises (Y1) multipliers, where the PE array is arranged in (Y2) rows and (X) columns; and determining a native tensor shape based on the first IFM and the one or more first filters, where the native tensor shape has a first outer dimension, an inner dimension, and a second outer dimension. determining a native tensor shape based on the second IFM and the one or more second filters, the native tensor shape including mapping the first IFM and the one or more first filters to the PE array; receiving a second IFM and one or more second filters in a second layer of the CNN that performs convolution using the PE array; and reshaping the native tensor shape based on the second IFM and the one or more second filters, the reshaping including expanding an inner dimension and contracting one of the first outer dimension and the second outer dimension, the reshaping and contracting being performed by a factor F; The method may include providing one or more second filters and a second IFM to the PE array to perform a convolution according to the native tensor, where when the first external dimension is reduced, the convolution sums outputs from a same row of the PEs for F rounds to obtain partial sums, and when the second external dimension is reduced, the convolution sums outputs from all F-fold rows of the PEs to obtain partial sums; and obtaining an output tensor of the convolution in the second layer of the CNN by summing the multiple partial sums, where Y1, Y2, X, and F are all integers greater than 1.

[0005] In some embodiments, a second layer of the CNN follows the first layer of the CNN, and the second IFM has more input channels and a lower resolution than the first IFM.

[0006] In some embodiments, each of the one or more second filters comprises multiple channels of a two-dimensional (2D) kernel, each 2D kernel having dimensions of 1×1 (one by one) or 3×3 (three by three).

[0007] In some embodiments, providing one or more second filters to the PE array according to the transformed native tensor includes converting the one or more second filters into a matrix according to a first outer dimension and an inner dimension of the transformed native tensor, where if each 2D kernel of the one or more second filters has a dimension of 1×1 (one by one), each row of the matrix contains weights from a different input channel of the one or more second filters, and distributing the weights of each row of the matrix to different columns of the PEs such that multiple input channels are processed simultaneously at once.

[0008] In some embodiments, providing one or more second filters to the PE array according to the transformed native tensor includes converting the one or more second filters into a matrix according to a first outer dimension and an inner dimension of the transformed native tensor, where if each 2D kernel of the one or more second filters has dimensions of 3×3 and includes 9 weights, the 9 weights are placed in the same row of the matrix, and distributing the 9 weights from the same row of the matrix to different columns of the PE such that weights from the same channel are processed simultaneously at one time.

[0009] In some embodiments, providing the IFM to the PE array according to the transformed native tensor includes converting the IFM to a matrix according to an inner dimension and a second outer dimension of the transformed native tensor, and providing input values ​​of the IFM corresponding to columns of the matrix to a row buffer of the PE.

[0010] In some embodiments, the method further includes dividing channels of the one or more filters into a plurality of channel groups, each channel group including a constant number of channels, the channel group being an integer greater than 1, and pruning each of the one or more filters such that only some channels in each of the plurality of channel groups include non-zero input values ​​and other channels in each channel group include all zeros. After pruning, each of the plurality of channel groups includes the same proportion of non-zero weights.

[0011] In some embodiments, the method may further include determining a depth of a buffer associated with each PE in the PE array, and configuring the buffer as a private memory of each PE if the buffer depth is greater than a constant, and combining the PE's buffer with one or more buffers of adjacent PEs as a shared memory if the buffer depth is less than the constant.

[0012] In some embodiments, the private memory of each PE stores input values ​​that can be obtained by the (Y1) multipliers within the PE.

[0013] In some embodiments, the shared memory stores input values ​​obtainable by the (Y1) multipliers in the PE and one or more adjacent PEs.

[0014] In some embodiments, each row of PEs is coupled with (Y1) adder trees corresponding to the (Y1) multipliers in each PE, and each multiplier in each PE sends product outputs to a corresponding adder tree for summation.

[0015] In some embodiments, each of the one or more second filters includes a plurality of non-zero weights, and providing the one or more second filters to the PE array for convolution includes providing each non-zero weight to a multiplier of the corresponding PE as an index-value pair including the non-zero weight and a corresponding index, and the convolution includes obtaining an input value from a buffer of the corresponding PE according to the index, sending the obtained value and the non-zero weight to the multiplier to obtain an output, and sending the output to the corresponding adder tree for summing with outputs generated by other multipliers of other PEs in the same row as the corresponding PE.

[0016] In some embodiments, the (Y1) multipliers in each PE process data in parallel, and the PEs in the PE array process data in parallel.

[0017] According to yet another aspect, a system may include one or more processors; and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform any of the methods described herein.

[0018] According to yet another aspect, a non-transitory computer-readable storage medium may be configured with instructions executable by one or more processors to cause the one or more processors to perform any of the methods described herein.

[0019] These and other features of the systems, methods, and non-transitory computer readable media disclosed herein, as well as the manner and function of operation of the associated elements of structure, and some combinations and economies of manufacture, will become more apparent from a consideration of the following description and the appended claims in conjunction with the accompanying drawings, all of which form a part hereof, and in which like reference numerals refer to corresponding parts in the various views, it being fully understood, however, that the drawings are for purposes of illustration and example only, and are not intended to limit the invention. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an exemplary system diagram for neural network computation in a PE array according to various embodiments. [Diagram 2] FIG. 2 illustrates an exemplary architectural diagram of a PE array in accordance with various embodiments. [Diagram 3] FIG. 3 illustrates an exemplary neural network computation in a PE array using native tensor geometry according to various embodiments. [Figure 4A] FIG. 4A illustrates an example neural network computation in a PE array using adaptive tensor shapes in accordance with various embodiments. [Figure 4B] FIG. 4B illustrates an exemplary PE array with inter-cluster adders for neural network computations using adaptive tensor shapes in accordance with various embodiments. [Figure 5A] FIG. 5A illustrates another exemplary neural network computation in a PE array using adaptive tensor shapes in accordance with various embodiments. [Figure 5B] FIG. 5B illustrates another exemplary PE array with intra-cluster adders for neural network computations using adaptive tensor shapes in accordance with various embodiments. [Figure 6A] FIG. 6A illustrates an example neural network computation with a 1×1 tensor computation mode in accordance with various embodiments. [Figure 6B] FIG. 6B illustrates an exemplary neural network calculation in a 3×3 tensor operation mode according to various embodiments. [Figure 7] FIG. 7 is a diagram illustrating an exemplary method of neural network computation using adaptive tensor shapes and a 3×3 tensor operation mode in a PE array in accordance with various embodiments. [Figure 8] FIG. 8 illustrates an exemplary method of neural network computation using adaptive tensor shapes in accordance with various embodiments. [Figure 9] FIG. 9 illustrates an example computer system in which any of the embodiments described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The embodiments described herein provide a method, system, and apparatus for neural network computation in a PE array using adaptive tensor shapes and operation modes. In the following description, an adaptive tensor computation kernel is described as having multiple native tensor dimensions and operation modes to handle different shapes of input feature maps (IMFs) and weight tensors (e.g., filters). According to the shapes and operation modes of the input and output tensors, the dimensions and operation modes of the adaptive tensor computation kernel (also referred to as adaptive native tensors) can be dynamically adjusted to fully utilize the underlying hardware resources of the PE array for parallel processing.

[0022] These adaptive tensor computation kernels address the technical challenges in neural network computation (described in the background section) by providing three technical solutions. First, adaptive tensor computation kernels can adapt and adjust their shapes to various shapes of input tensors / weight tensors. Various shapes of input tensors / weight tensors may exist not only across various neural networks but also within the same neural network pipeline. For example, when processing the first few layers of a neural network, tensors are typically configured to have high resolution (large height and width) but fewer input and output channels, and when processing the last few layers of a neural network, tensors are typically configured to have low resolution (small height and width) but more input and output channels. This is believed to be because the first few layers of a neural network focus on feature extraction from input feature maps, whereas the last few layers of a neural network focus on learning the underlying correlations between the extracted features.

[0023] Second, the adaptive tensor computation kernel can support two different tensor computation modes: 1×1 tensor computation mode and 3×3 tensor computation mode. In those neural network layers that involve matrix multiplication, convolutions with 1×1 kernels (e.g., each kernel in the weight tensor has a shape of 1×1) can be mapped to the 1×1 tensor computation mode, and other convolutions (3×3, 5×5, 7×7, etc.) can be mapped to the 3×3 tensor computation mode. These different tensor computation modes can be dynamically determined at runtime based on the shape of the weight tensor.

[0024] Third, the underlying PE array can configure each PE internal buffer (e.g., register file) differently to support different compression ratios and sparsity granularities of the sparse neural network. If the sparse neural network is pruned at a finer granularity (e.g., select one or more nonzero input channels from a small number of input channels, and the small number is smaller than a threshold), the register file in each PE can be configured as a private memory (e.g., used only by the corresponding PE). If the sparse neural network is pruned at a coarser granularity (e.g., select one or more nonzero input channels from a large number of input channels, and the large number is larger than a threshold), multiple register files in adjacent PEs can be configured as multi-ported memories shared by adjacent PEs.

[0025] In the following description, certain non-limiting embodiments of the present invention will be described with reference to the drawings. The specific features and aspects of any embodiment disclosed herein can be used in conjunction with and / or combined with the specific features and aspects of any other embodiment disclosed herein. It should also be understood that such embodiments are exemplary and merely exemplify a small number of embodiments within the scope of the present invention. Various changes and modifications in the technical field to which the present invention pertains that are obvious to those skilled in the art are deemed to be within the spirit, scope and spirit of the present invention, as further defined in the appended claims.

[0026] 1 shows an example system diagram for neural network computation processing in a PE array according to various embodiments. The diagram in FIG. 1 shows a workflow of a typical neural network computation executed in a pipeline using a PE array. The embodiments described in this disclosure may be implemented as part of the neural network computation in FIG. 1 or in other suitable environments.

[0027] At a given (e.g., convolutional) layer in a neural network (e.g., CNN), one or more input feature maps (IFMs) 120 may be obtained from an input source (e.g., an input image, for example) or a previous layer (e.g., a tensor output from a previous layer, for example), and one or more weight tensors 110 may be used to convolve the IFMs to extract various features. The convolution process may be performed in parallel in an array of processing elements (PEs), called a PE array 160. Each PE may refer to a processor having processing power and storage capacity (e.g., a buffer or cache). The PEs may be arranged in a specific manner in the PE array with interconnect wiring and may not be dynamically rearranged at runtime. The PE array may be reused and participate in operations at different layers of the neural network or across different neural networks and different use cases. The incompatibility between the fixed internal PE arrangement in the PE array and the potentially infinite variety of tensor shapes (in the IFMs and / or weight tensors) typically leads to inefficient resource utilization and suboptimal parallel processing.

[0028] Referring to FIG. 1, in some embodiments, IFMs 120 may be stored in an IFM cache 140 and weight tensor 110 may be stored in a weight cache 130 for computation in a PE array 160. The PE array 160 may contain a matrix of PEs (e.g., X×Y), and each PE may contain multiple multipliers for parallel processing. In some embodiments, each IFM from the IFM cache 140 may pass through a matrix transformation layer 150 to facilitate computation in the PE array 160. The matrix transformation may include a Toeplitz matrix transformation using the im2col tool to transform the IFMs from their original HWC format (H stands for height, W stands for weight, and C stands for channel) to an RSC format (R stands for row, S stands for column, and C stands for channel), where the RSC format is determined based on the shape of the weight tensor. Here, the transformation refers to duplicating and arranging the input values ​​in the IFM to form a transformed IFM, so that the matrix multiplication between the transformed IFM and the weight tensors is performed in the PE array 160 in parallel while minimizing dependencies between PEs in the PE array 160. In some embodiments, each round of parallel convolution in the PE array 160 may generate multiple partial sums, which may be summed in the accumulation buffer 170 to generate one or more output values. The output values ​​may eventually become part of the output tensor generated by the current layer.

[0029] 2 shows an example architecture diagram of a PE array according to various embodiments. The arrangement of PEs in the PE array of FIG. 2 is exemplary and may be implemented in other ways depending on the use case.

[0030] As shown in the left portion of FIG. 2, the PE array 200 may include a matrix of PEs. As shown in the right portion of FIG. 2, each PE 240 may include multiple multipliers (MUL gates). The multipliers in each PE 240 may operate in parallel, and the PEs in the PE array 220 may operate in parallel. For ease of reference, the following description will denote the number of columns 220 of PEs in the PE array 200 as X, the number of rows 210 of PEs in the PE array 200 as Y2, and the number of multipliers in each PE 240 as Y1. Each row of PEs 210 is referred to as a PE cluster, and each PE cluster may be coupled to Y1 adder trees (Y1 adder trees) 230 for summing partial sums generated by the multipliers in the PE cluster. That is, the first multiplier in each PE 240 in the PE cluster is coupled to the first adder tree 230 for summation, the second multiplier in each PE 240 in the PE cluster is coupled to the second adder tree 230 for summation, and so on. The results of summing adder trees 230 across all PE clusters (a total of Y1 x Y2 adder trees) may be provided to adder 250 for summation. Adder 250 may refer to a digital circuit that performs addition of numeric values ​​that is part of a network-on-chip (NOC) subsystem.

[0031] In some embodiments, the PE array 200 may broadcast weights to the PEs. In the case of a sparse neural network, most of the weights are zero, and therefore all weights broadcast to the PEs are non-zero weights. Since non-zero weights may come from any position in the weight tensor, each broadcasted weight may include not only the weight value, but also an index indicating the position information of the weight value, i.e., an index-value pair such as (index, weight value). Based on the index, each PE 240 can obtain a corresponding input value from the IFM and perform multiplication with the weight value. The multiplication result can be provided to a corresponding adder tree. As shown in FIG. 1, the first multiplier MUL1 receives a weight in the form of (index 1, value 1), obtains an input value IFM1 from the IBUF 260 (which stores the IFM) based on index 1, performs multiplication based on the input value IFM1 and value 1, and sends the result to the adder tree 1 (e.g., the first adder tree of the Y1 adder tree 230 of the PE cluster in which the PE is located) for summation.

[0032] Figure 3 illustrates an exemplary neural network computation in a PE array using native tensor shapes according to various embodiments. The exemplary computation in Figure 3 involves a matrix multiplication between a transformed weight tensor A (310) and a transformed IFM tensor B (320), which produces an output feature map (OFM) tensor C (330). The matrix multiplication uses the native tensor shape, which corresponds to a PE array 340 having dimensions X and Y.

[0033] In some embodiments, the transformed weight tensor A (310) is obtained by combining all weight tensors in RSC format (3-dimensional) into a 2-dimensional matrix denoted as m'*k' (e.g., weights from different channels are rearranged to the same channel), where m' is the number of output channels determined by the number of weight tensors (usually denoted as K), and k' is the product of the R, S, and C dimensions of each weight tensor (R and S refer to the dimensions of each kernel in the weight tensor, and C refers to the number of input channels).

[0034] In some embodiments, the transformed IFM tensor B (320) can be obtained by integrating all IFMs in HWC format (3-dimensional) based on the RSC format into a 2-dimensional matrix denoted as k'*n', where k' is still the product of the R, S, and C dimensions of each weight tensor, and n' is the product of the H and W dimensions of the IFM (H refers to height and W refers to width). A matrix product of matrix m'*k' (weight tensor A (310)) and matrix k'*n' (IFMB (320)) can generate the OFM tensor C (330) as a matrix of m'*n'.

[0035] With the above transformation, the transformed weight tensor A (310) and the transformed IFM tensor B (320) may be mapped to PEs in the PE array 340 for parallel processing. Assuming that the PE array 340 includes Y2 row PEs and X column PEs, with each PE including Y1 multipliers, tensors A and B may be mapped to the PE array 340 as follows: the inner dimension k' of tensor A (310) and tensor B (320) is mapped to the X (row) dimension of the PE array 340, i.e., X=k'=R*S*C, and the multiplication of the outer dimension m'*=n' of tensor A and tensor B is mapped to the Y (column) dimension of the PE array 340. Because each column of PEs includes Y1*Y2 multipliers, with the above mapping, the multiplication of Y=m'*n'=K*H*W will be processed in parallel by the Y1*Y2 multipliers. For example, within each PE, one multiplier processes weights corresponding to the same output channel (e.g., weights from the same position across all weight tensors), i.e., Y1 = K = m', and each column of PEs processes H*W weights in parallel, i.e., Y2 = H*W = n'.

[0036] In the above description, a native tensor shape m'*k'*n' is determined, where X=k', Y1*Y2=m'*n', to map a workload (e.g., a pair of weights and corresponding input values ​​for multiplication) to a PE in the PE array 340. That is, the native tensor shape is determined based on the layout of the PEs in the PE array. Once the layout of the PE array is determined, the shape of the native tensor is also determined. All input tensors (e.g., IFM and filter / weight tensors) must be transformed according to the determined native tensor shape. However, input tensors in real applications have various shapes, and performing transformations based on the shape of the input tensor, rather than the layout of the PEs in the PE array 340, allows optimal parallel processing. In many cases, transformations using the determined native tensor shape determined based on the PE layout may cause some sequential dependencies between certain PEs (e.g., one PE waits for the output of another PE) even if the workload is mapped to the PEs. In the following discussion, we describe a transformation that uses adaptive native tensor shapes that are determined based on the IFM and filter dimensions, while simultaneously mapping the workload to PEs to maximize parallelism.

[0037] FIG. 4A illustrates an exemplary neural network computation in a PE array using adaptive tensor shapes according to various embodiments. As mentioned above (FIG. 3), if an input tensor (IFM) and weight tensor can be converted to matrix A (410) and matrix B (420) using a fixed native tensor shape m'*k'*n' (i.e., covering matrices A and B), the converted tensors can be distributed to the corresponding PE array. However, in real applications, IFMs and weight tensors for multiplication (e.g., tensors that have undergone different levels of sparsification in different layers of a CNN) may have various shapes that cannot be perfectly mapped to the PE array. Forcing the tensors to be converted using a fixed native tensor shape may cause some PEs to be idle or introduce sequential dependencies during computation. For example, even within the same convolutional neural network (CNN), the tensors in the first few layers of the CNN may have high resolution (e.g., H*W=64) and a small number of input channels (C=16), while the tensors in the last few layers of the CNN may have low resolution (e.g., H*W=16) and a large number of input channels (C=64). Here, "small" and "large" are determined based on a threshold. That is, even within the same CNN, tensors of different shapes may appear.

[0038] In some embodiments, the shape of the native tensor may be dynamically deformed to accommodate the changing shapes of the input tensor and weight tensor. For example, if the input tensor changes from a high resolution (high pixel count) with a low number of input channels (e.g., in the first few CNN layers) to a low resolution with a high number of input channels (e.g., in the last few CNN layers), the native tensor shape can be deformed accordingly. In some embodiments, the native tensor shape has three dimensions, denoted as the first external dimension, the internal dimension, and the second external dimension. The first two dimensions (the first external dimension and the internal dimension) can be used to convert the weight tensor to a matrix, and the last two dimensions (the internal dimension and the second external dimension) can be used to convert the IFM to a matrix. The transformed matrix can provide guidance on how the weights and input values ​​are mapped to the PE array (e.g., how to distribute the weights and input values ​​to achieve optimal parallelism).

[0039] In some embodiments, if the previous tensor uses a native tensor shape m'*k'*n' for mapping and transformation, and the input tensor has more input channels at a lower resolution compared to the previous tensor, the three dimensions of the native tensor shape can be transformed to m'*(F*k')*(n' / F), where F is an integer greater than 1 and represents a scaling factor, the first two dimensions (i.e., the first outer dimension m' and the inner dimension F*k') represent the weight tensor matrix 420, and the next two dimensions (i.e., the inner dimension F*k' and the second outer dimension n' / F) represent the IFM tensor matrix 422 for convolution. That is, the native tensor shape can expand its inner dimension by a factor F and shrink the second outer dimension (corresponding to the IFM tensor matrix) by a factor F. This transformation method may be referred to as k'&n' transformation in the following description. In some embodiments, F is one of 2, 4, 8, etc.

[0040] In some embodiments, the internal dimension F*k' of the transformed tensor shape is shared (e.g., has the same internal dimension) by weight tensor matrix 420 and IFM tensor matrix 422 and corresponds to the number of columns of PEs in the PE array, the first external dimension (e.g., the external dimension m' of weight tensor matrix 420) corresponds to the number of multipliers in each PE, and the second external dimension (e.g., the external dimension n' / F of IFM tensor matrix 422) corresponds to the number of rows of PEs in the PE array. Here, "corresponding" refers to a mapping relationship that dictates how the weights and input values ​​of the transformed tensor matrix are distributed across the PE array. For example, the weights in each external dimension (e.g., each column) of weight tensor matrix 420 may be distributed to multipliers in a single PE for parallel processing, and the input values ​​in each external dimension (e.g., each column) of IFM tensor matrix 422 may be distributed across rows of PEs in the PE array.

[0041] 4A, with this modified native tensor shape, weight tensor matrix 410 is modified to expand its internal dimensions by a factor of F and keep its external dimensions m' the same, thereby forming new weight tensor matrix 420. That is, the internal dimensions of the weight tensor matrix change from k'=R*S*C (e.g., matrix A in 410) to F*k'=R*S*(F*C) (e.g., matrix A in 420), such that new matrix 420 can support more input channels (C to F*C). Similarly, IFM matrix 412 reduces its external dimensions by a factor F and expands its internal dimensions in the same manner as the reduced internal dimensions of weight tensor matrix 420, thereby forming new IFM tensor matrix 422. That is, the internal dimension of the IFM tensor matrix changes from k'=R*S*C (e.g., matrix B in 412) to F*k'=R*S*(F*C) (e.g., matrix B in 422), and the external dimension of the IFM tensor matrix changes from n' (e.g., matrix B in 412) to n' / F (e.g., matrix B in 422), and thus the new matrix 422 may support fewer pixels. Thus, the new matrices 420 and 422 are more suitable to represent tensors from the last few CNN layers with fewer input channels and lower resolution. In some embodiments, the "first few CNN layers" and the "last few CNN layers" may refer to a first number of CNN layers counting from the beginning of the CNN structure and a second number of CNN layers counting from the end of the CNN structure, respectively.

[0042] As an example, the tensors of the first new CNN layer may have a high resolution H*W=64 and a small number of input channels C=16, where "small number" refers to a number smaller than a threshold, and the threshold can be determined by a compiler configured according to the underlying PE array. The native tensor shape of these tensors from the first few CNN layers may have a shape of m'=K=16, k'=1*1*16, n'=64, assuming that the convolution is based on a 1*1 kernel. As the convolution proceeds to the last few CNN layers, the tensors may have a low resolution H*W=16, and as the number of input channels C=64 (e.g., larger than a threshold) increases, the native tensor shape may be transformed to m'=K=16, k'=1*1*64, n'=16.

[0043] After transforming the tensor using the k'&n' transformations above, the transformed tensor matrices 420 and 422 can be distributed to a PE array for parallel processing. FIG. 4B illustrates an example PE array with inter-cluster adders for neural network computations using adaptive tensor shapes based on k'&n' transformations according to various embodiments. The parallel processing scheme using the PE array illustrated in FIG. 4B may correspond to the k'&n' transformation of native tensors described in FIG. 4A. For consistency, assume that the PE array has Y2 rows and X columns of PEs, each with Y1 multipliers.

[0044] By transforming k'&n', the internal dimensions of the weight tensor matrix and the IFM tensor matrix are expanded by a factor of F, and the external dimensions of the IFM tensor matrix are reduced by a factor of F. The distribution of weights and input values ​​to the PE array may be such that weights from the same row (i.e., along the expanded internal dimension / row) of the weight tensor matrix and input values ​​from the same column (i.e., along the also expanded internal dimension / column) of the IFM tensor matrix are assigned to the PEs row by row. This means that these weight and input value pairs may be distributed across the F rows of PEs. Thus, the PE array has inter-cluster (i.e., between PE clusters or between rows) adders 400 that add the outputs generated by each row of PEs to obtain partial sums of the convolution process. Each inter-cluster adder 400 may sum the outputs from the Y1 adder trees of the F rows of PEs as a Y1 partial sum. These partial sums are summed to construct an output tensor as a result of the convolution. In this process, the sum of the partial sums is Y1*(Y2 / F), which means that the number of output channels (i.e. the number of channels in the output tensor of the convolution process) is Y1, and the number of output pixels is Y2 / F=H*W / F.

[0045] FIG. 5A illustrates another exemplary neural network computation in a PE array using adaptive tensor shapes according to various embodiments. Compared with the above k'&n' transformation, the native tensor shape can also be dynamically transformed based on the sparsity of the weight tensor. In many practical applications, the weight tensor in the convolution process can be pruned or sparsified to improve computational efficiency and reduce the footprint of the neural network. A carefully pruned weight tensor can improve the convolution speed without sacrificing the accuracy of feature extraction by introducing zero-valued weights, thereby reducing the total number of computations (e.g., zero-valued weights are skipped). In some embodiments, pruning the weight tensor may include splitting the channels (also called filters) of the weight tensor into multiple channel groups, such that all channel groups have the same number of channels, and keeping only a few channels of each channel group as non-zero input channels (e.g., non-zero weights) and zeroing all other channels in the channel group (e.g., all weights are zero). After the pruning process, each channel group contains the same proportion of non-zero weights. In some embodiments, the size of the channel groups for pruning (e.g., the number of channels in each channel group) may be determined based on the number of weight tensors (filters), i.e., the number of output channels. In general, pruning of weight tensors may be classified into two levels: high weight sparsity, where the number of output channels (e.g., the number of weight tensors) is greater than a first threshold and the number of non-zero input channels is less than a second threshold; and low weight sparsity, where the number of output channels (e.g., the number of weight tensors) is less than a first threshold and the number of non-zero input channels is greater than a second threshold.For example, the native tensor shape for high weight sparsity (16x) is m'=K=16 (e.g., 16 weight tensors or filters), k'=3*3*4 (e.g., 3*3 for each kernel, 4 non-zero channels in a filter), n'=64, while the native tensor shape for low weight sparsity (4x) is m'=K=4 (e.g., 4 weight tensors or filters), k'=3*3*16 (e.g., 3*3 for each kernel, 16 non-zero channels in a filter), n'=64.

[0046] In some embodiments, when the sparsity of weights changes from high to low, the shape of the native tensor (denoted as first_outer_dimension*inner_dimension*second_outer_dimension) may be deformed by expanding the inner dimension (shared by the weight tensor matrix and the IFM tensor matrix) by a factor F and shrinking the first outer dimension (corresponding to the weight tensor matrix) by a factor F. As shown in FIG. 5A , the original native tensor shape m'*k'*n' becomes (m' / F)*(F*k')*n', the weight tensor matrix 510 changes from m'*k' to a deformed tensor matrix 520 of dimension (m' / F)*(F*k'), and the IFM tensor matrix 512 changes from k'*n' to a deformed IFM matrix 522 with dimension (F*k')*n'. ​​This deformation method may be referred to as k'&m' deformation in the following description. The expanded inner dimensionality by a factor F indicates that we support more input channels (C to F*C), and the reduced outer dimensionality of the weight tensor matrix indicates that we support fewer output channels (K to K / F).

[0047] After transforming the tensors using the k'&m' transformations described above, the transformed tensor matrices 520 and 522 can be distributed to a PE array for parallel processing. Figure 5B shows another exemplary PE array with inter-cluster adders for neural network computations using adaptive tensor shapes based on the k'&m' transformations according to various embodiments. The parallel processing scheme using the PE array shown in Figure 5B may correspond to the k'&m' transformation of native tensors described in Figure 5A.

[0048] For consistency, assume that the PE array has Y2 rows and X columns of PEs, with each PE having Y1 multipliers. Further, weight matrix 520 and IFM matrix 522 have identical internal dimensions corresponding to the (X) columns of PEs in the PE array, the external dimensions of weight matrix 520 corresponding to the (Y1) multipliers in each PE in the PE array, and the external dimensions of IFM matrix 522 corresponding to the (Y2) rows of PEs in the PE array.

[0049] Since the transformed native tensor has a first external dimension (corresponding to the external dimension of the weight matrix 520) as m' / F, the weight of each column of the weight tensor matrix can be fed to a Y1 / F multiplier in each PE. To obtain partial sums from the PE array, an intra-cluster adder 500 may be implemented that stores and sums the outputs from the Y1 adder tree for F rounds. Here, a "round" refers to a cycle of performing multiplications using the multipliers in the PE. During each round, the output of the Y1 / F multipliers may be temporarily stored in one intra-cluster adder 500. After F rounds, the intra-cluster adder 500 may store F*Y1 / F=Y1 partial sums collected from the Y1 adder tree. These partial sums may be summed to construct an output tensor as a result of the convolution. The sum of the partial sums in this process is (*Y1 / F)*Y2, which means that the number of output channels (e.g., the number of channels of the output tensor of the convolution process) is Y1 / F and the number of output pixels is Y2=H*W.

[0050] In the field of convolutional neural networks, a weight tensor may be referred to as a 3D filter that includes multiple 2D kernels. The number of 2D kernels in each 3D filter may be referred to as the number of channels in the filter, and each 2D kernel may be a 1×1 or 3×3 matrix. Figure 6A shows an exemplary neural network calculation according to a 1×1 (one by one) tensor operation mode (i.e., using a 1×1 (one by one) kernel) in accordance with various embodiments, and Figure 6B shows an exemplary neural network calculation according to a 3×3 tensor operation mode (i.e., using a 3×3 kernel) in accordance with various embodiments.

[0051] In some embodiments, general matrix multiplication (GEMM) and 1×1 convolution operations may be mapped to 1×1 operation modes (e.g., using 1×1 kernels). As shown in FIG. 6A, 2D kernels (i.e., weights) from different input channels (or channel groups for sparsified input tensors) may be placed in different columns of PEs so that multiple input channels are processed simultaneously at once, and 2D kernels from the same input channel may be distributed to multiple multipliers in one PE so that the multipliers can process multiple output channels simultaneously at once. For example, weight number Y1 from channel 1 (C=1) and 1-Y1 kernels from the filters (i.e., weights from the same input channel of multiple filters) may be fed to a first PE, and weight number Y1 from channel 2 (C=2) and 1-Y1 kernels from the filters may be fed to a second PE. In this way, 2D kernels from different input channels are distributed to columns of PEs.

[0052] In some embodiments involving sparsified input tensors, each weight can be represented as an index-value pair, where the value of the index-value pair is the value of the non-zero weight, and the index of the index-value pair is the index of the non-zero weight, which can be used to identify the corresponding input value for multiplication in one multiplier. In some embodiments, if the number of channels is less than the number of PEs in each PE cluster (each row), the remaining PEs can be used for other vector operations.

[0053] In some embodiments, convolution operations other than the above-mentioned 1×1 convolution operation may be decomposed into one or more 3×3 convolutions and mapped to a 3×3 native operation mode (e.g., using a 3×3 kernel). As shown in FIG. 6B, each 2D 3×3 kernel has 9 weights ((0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,0), (2,1), (2,2)) from the same input channel, which may be distributed to the same row (different columns) of the PE for simultaneous processing. The 9 weights from different input channels may be distributed to different rows of the PE.

[0054] FIG. 7 illustrates an exemplary architecture diagram of an internal buffer in an array of PEs according to various embodiments. In some embodiments, each PE in the PE array is coupled with an input buffer (IBUF) 722 for storing input values. These input values ​​can be retrieved by the PE based on a predefined weight index to find the corresponding input value. The retrieved input value can be multiplied with the weight value in a multiplier in the PE. In practical implementations, the depth of the IBUF is usually limited, and one IBUF 722 can only store input values ​​from a constant number of input channels. This design works well for sparsified input tensors, since the number of nonzero input channels is also limited. However, it often happens that the number of nonzero input channels exceeds the depth of the IBUF 722. In such cases, the IBUF 722 needs to perform cache replacement to read the required input values ​​from external memory, which is costly and inefficient.

[0055] In some embodiments, the IBUF of each PE can be configured as a private memory or a shared memory depending on the degree of sparsification of the weight tensor (filter). For example, sparsifying one or more weight tensors may include dividing the input channels of the one or more weight tensors into a number of channel groups, each channel group including a constant number of channels, the number being an integer greater than one, and pruning the one or more weight tensors, respectively, such that in each of the plurality of channel groups, only a small number of channels in each of the plurality of channel groups include non-zero input values, and the other channels in each channel group all include zeros. That is, after the pruning process, each channel group includes the same proportion of non-zero weights. The granularity of sparsification can be classified as fine-grained 710 and coarse-grained 750. Fine-grained 710 sparsity occurs when non-zero input channels are selected from a number of channels smaller than a constant, and coarse-grained 750 sparsity occurs when non-zero input channels are selected from a number of channels larger than a constant. For example, if the sparsity of the weights is 15 / 16 (one non-zero input channel out of 16 channels), selecting one non-zero input channel out of every 16 input channels (e.g., one channel group contains 16 input channels) may be determined as fine-grained 710 sparsity, while selecting four non-zero input channels out of every 64 input channels (e.g., one channel group contains 64 input channels) may be determined as coarse-grained 750 sparsity.

[0056] In some embodiments, IBUF722 can be configured as a private memory for sparse weight tensors with fine grain or as a shared memory for sparse tensors with coarse grain. That is, the depth of IBUF722 can be compared to a constant used to classify fine-grained and coarse-grained sparsity. If the depth of IBUF722 is greater than the constant, it means that IBUF722 is sufficient to store the required input values. In this way, the dedicated private memory optimizes data acquisition performance. If the depth of IBUF722 is less than the constant, multiple adjacent PEs share their IBUFs, denoted as shared IBUFs, to store the input values ​​that can be acquired by them. In this way, duplicate input values ​​are reduced and overall storage efficiency is improved.

[0057] FIG. 8 illustrates an exemplary method 800 for neural network computation using adaptive tensor shapes in accordance with various embodiments. Method 800 may be performed by a device, apparatus, or system described in FIGS. 1-7. The operations of method 800 presented below are intended to be exemplary. Depending on the implementation, method 800 may include additional, fewer, or alternative steps performed in various orders or in parallel.

[0058] Block 810 includes receiving a first input feature map (IFM) and one or more first filters in a first layer of a convolutional neural network (CNN) for convolution using a processing element (PE) array, where each PE in the PE array includes (Y1) multipliers, and the PE array is arranged in (Y2) rows and (X) columns. In some embodiments, each row of PEs is coupled with (Y1) adder trees corresponding to the (Y1) multipliers in each PE, and each multiplier in each PE sends multiplication output to the corresponding adder tree for summation. The (Y1) multipliers in each PE process processing data in parallel, and the PEs in the PE array process processing data in parallel.

[0059] Block 820 includes determining a native tensor shape based on the first IFM and the one or more first filters, where the shape of the native tensor includes a first outer dimension, an inner dimension, and a second outer dimension, where the native tensor shape maps the first IFM and the one or more first filters to the PE array.

[0060] Block 830 includes receiving a second IFM and one or more second filters at a second layer of the CNN that performs convolution using the PE array. In some embodiments, the second layer of the CNN is after the first layer of the CNN, and the second IFM has more input channels than the first IFM and a lower resolution than the first IFM. In some embodiments, each of the one or more second filters comprises a plurality of two-dimensional (2D) kernel channels, each 2D kernel having a dimension of one by one or three by three.

[0061] Block 840 includes transforming a shape of the native tensor based on the second IFM and the one or more second filters, the transformation including expanding an inner dimension and contracting one of the first outer dimension and the second outer dimension, the expansion and contraction being performed by a factor F.

[0062] Block 850 includes providing one or more second filters and a second IFM to the PE array for convolution according to the transformed native tensor, where if the first external dimension is reduced, the convolution includes summing outputs from the same row of the PE for F rounds to obtain partial sums, and if the second external dimension is reduced, the convolution includes summing outputs from all F times the rows of the PE to obtain partial sums. In some embodiments, providing one or more second filters to the PE array according to the transformed native tensor includes converting the one or more second filters to a matrix according to the first external and internal dimensions of the transformed native tensor, where each row of the matrix includes weights from a different input channel of the one or more second filters according to each 2D kernel of the one or more second filters having a dimension of one by one, respectively, and distributing the weights of each row of the matrix to different columns of the PE such that multiple input channels are simultaneously processed at one time. In some embodiments, providing one or more second filters to the PE array according to the transformed native tensor includes converting the one or more second filters to a matrix according to a first outer dimension and an inner dimension of the transformed native tensor, where if each 2D kernel of the one or more second filters has a dimension of 3×3 and includes 9 weights, the 9 weights are placed in the same row of the matrix, and distributing the 9 weights from the same row of the matrix to different columns of the PE such that weights from the same channel are processed simultaneously at once. In some embodiments, providing the IFM to the PE array according to the transformed native tensor includes converting the IFM to a matrix according to the inner dimension and a second outer dimension of the transformed native tensor, and providing input values ​​of the IFM corresponding to the columns of the matrix to a buffer of rows of the PE.

[0063] Block 860 includes obtaining an output tensor of the convolution in the second layer of the CNN by summing the multiple partial sums.

[0064] In the above description, Y1, Y2, X, and F are all integers greater than 1.

[0065] In some embodiments, the method 800 further includes dividing the channels of the one or more filters into a plurality of channel groups, each channel group including a constant number of channels, the constant being an integer greater than one, and pruning each of the one or more filters such that only one of a small number of channels in each group of the plurality of channel groups includes a non-zero input value, and other channels of each channel group include all zeros. In some embodiments, the method 800 may further include determining a depth of a buffer associated with each PE in the PE array, configuring the buffer as a private memory of each PE if the buffer depth is greater than the constant, and combining the buffer of the PE and one or more buffers of an adjacent PE as a shared memory if the buffer depth is less than the constant. In some embodiments, the private memory of each PE stores input values ​​obtainable by the (Y1) multipliers in the PE, and the shared memory stores input values ​​obtainable by the (Y1) multipliers in the PE and one or more adjacent PEs.

[0066] In some embodiments, each of the one or more second filters includes a plurality of non-zero weights, and providing the one or more second filters to the PE array for convolution includes providing each non-zero weight to a multiplier of a corresponding PE as an index-value pair including the non-zero weight and a corresponding index, and the convolution includes obtaining an input value from a buffer of the corresponding PE according to the index, sending the obtained value and the non-zero weight to the multiplier to obtain an output, and sending the output to a corresponding adder tree for summing with outputs generated by other multipliers of other PEs in the same row as the corresponding PE.

[0067] Figure 9 illustrates an exemplary computing device capable of performing any of the embodiments described herein. The computing device can be used to implement one or more components of the systems and methods illustrated in Figures 1-8. The computing device 900 can include a bus 902 or other communication mechanism for communicating information and one or more hardware processors 704 coupled to the bus 902 for processing information. The hardware processor(s) 704 can be, for example, one or more general-purpose microprocessors.

[0068] The computing device 900 may also include a main memory 907, such as a random access memory (RAM), cache, and / or other storage device coupled to the bus 902 for storing information and instructions executed by the processor(s) 904. The main memory 907 may also be used to store temporary variables or other intermediate information during execution of instructions executed by the processor(s) 904. Such instructions, when stored in a storage medium accessible by the processor(s) 904, may make the computing device 900 a specialized machine customized to perform the operations specified in the instructions. The main memory 907 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks. Volatile media may include dynamic memory. Common forms of media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tapes, other magnetic data storage media, CD-ROMs, other optical data storage media, physical media with perforated patterns, RAM, DRAM, PROM, EPROM, Flash EPROM, NVRAM, other memory chips or cartridges, or networked versions thereof.

[0069] The computing device 900 may use customized hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic that can be combined with the computing device to make the computing device 900 a special-purpose machine to implement the techniques described herein. According to one embodiment, the techniques described herein are performed by the computing device 900 in response to the processor 904 executing one or more sequences of one or more instructions contained in the main memory 907. Such instructions may be read into the main memory 907 from another storage medium, such as the storage device 909. Execution of the sequence of instructions contained in the main memory 907 may cause the processor 904 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be performed by computer program instructions stored in the main memory 907. When these instructions are executed by the processor(s) 904, steps illustrated in the corresponding figures and described above may be performed. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0070] The computing device 900 also includes a communication interface 910 coupled to the bus 902. The communication interface 910 can provide a two-way data communication coupled to one or more network links that are connected to one or more networks. As another example, the communication interface 910 can be a local area network (LAN) card that provides a data communication connection to a compatible LAN (or a WAN component in communication with a WAN). Wireless links can also be implemented.

[0071] Execution of certain operations may be distributed among processors spread across multiple machines as well as within a single machine. In some exemplary embodiments, the processor or processor-implemented engine may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other exemplary embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0072] The processes, methods, and algorithms described in the preceding sections may be implemented in code modules executed by one or more computer systems or computer processors comprising computer hardware, and may be fully or partially automated. The processes and algorithms may be implemented partially or fully in application-specific circuitry.

[0073] When the functions disclosed herein are implemented in software functional units and sold or used as an independent product, they can be stored in a non-volatile computer-readable storage medium executable by a processor. The specific technical solutions (in whole or in part) disclosed herein or an aspect contributing to the current art can be embodied in the form of a software product. The software product may include a plurality of instructions stored in a storage medium and causing a computing device (such as a personal computer, a server, a network device, etc.) to execute all or some steps of the method of the embodiment of the present invention. The storage medium may include a flash drive, a portable hard drive, a ROM, a RAM, a magnetic disk, an optical disk, other media operable to store program code, or a combination thereof.

[0074] Certain embodiments further comprise a processor and a non-transitory computer-readable storage medium storing processor-executable instructions, whereby the system is configured to perform operations corresponding to the steps of the method of any of the above-mentioned embodiments. Certain embodiments further comprise a non-transitory computer-readable storage medium including one or more processor-executable instructions, whereby the one or more processors are configured to perform operations corresponding to the steps of the method of any of the above-mentioned embodiments.

[0075] The embodiments disclosed herein may be implemented through a cloud platform, a server, or a group of servers (collectively referred to as a "service system") that interacts with a client. The client may be a terminal device or a client registered by a user on the platform, and the terminal device may be a mobile terminal, a personal computer (PC), or any device on which a platform application program can be installed.

[0076] The various functions and processes described above may be used independently of each other or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Furthermore, in some implementations, certain method or process blocks may be omitted. Also, the methods and processes described herein are not limited to a particular order, and the blocks or states associated therewith may be performed in other orders as appropriate. For example, the described blocks or states may be performed in an order other than the order specifically disclosed, or multiple blocks or states may be combined into a single block or state. The example blocks or states may be performed in serial, parallel, or in some other manner. Blocks or states may be added to or deleted from the disclosed example embodiments. The example systems and components described herein may be configured in different ways than described. For example, elements may be added, deleted, or rearranged as compared to the disclosed example embodiments.

[0077] Various operations of the example methods described herein may be performed, at least in part, by an algorithm. The algorithm may include program code or instructions stored in a memory (e.g., a non-transitory computer-readable storage medium as described above). Such algorithms may include machine learning algorithms. In some embodiments, the machine learning algorithms may learn from training samples to create predictive models that perform functions, rather than explicitly programming a computer to perform the functions.

[0078] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute a processor-implemented engine that is operatively configured to perform one or more operations or functions described herein.

[0079] Similarly, the methods described herein may be at least partially performed by a processor, with a particular processor or group of processors being an example of hardware. For example, at least some operations of the methods may be performed by one or more processors or processor-implemented engines. Furthermore, one or more processors may operate to support the execution of related operations within a "cloud computing" environment or as a "software as a service" (SaaS). For example, some operations in this embodiment are performed by a group of computers, which are an example of a machine that includes a processor, and these operations are accessible via a network (e.g., the Internet) and one or more appropriate interfaces (e.g., application program interfaces (APIs)).

[0080] Execution of certain operations may be distributed among processors spread across multiple machines as well as within a single machine. In some exemplary embodiments, a processor or processor-implemented engine may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other exemplary embodiments, a processor or processor-implemented engine may be distributed across multiple geographic locations.

[0081] Throughout this specification, multiple examples may implement a component, operation, or structure that is described as a single example. Although individual operations of one or more methods are illustrated and described as separate operations, one or more individual operations may be performed simultaneously and need not be performed in the order shown. Structures and functions shown as separate components in an example configuration may be implemented as an integrated structure or component. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the present invention.

[0082] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context clearly dictates otherwise. Thus, in this specification, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C," unless expressly indicated otherwise or the context clearly dictates otherwise. Furthermore, "and" is inclusive and particular, unless expressly indicated otherwise or the context clearly dictates otherwise. Thus, in this specification, "A and B" means "both or either A and B," unless expressly indicated otherwise or the context clearly dictates otherwise. Furthermore, for any resource, operation, or structure described herein as a single example, multiple examples may be provided. Moreover, boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary, and particular operations are described in the context of specific illustrative configurations. Other allocations of functionality are contemplated and may be within the scope of various embodiments of the present disclosure. In general, structures and functions presented as separate resources in the exemplary configurations may be implemented as integrated structures or resources. Similarly, structures and functions presented as a single resource may be implemented as separate resources. These variations, modifications, additions, and improvements are within the scope of the embodiments of the present disclosure as set forth in the appended claims. Accordingly, the specification and drawings should be interpreted in an illustrative, rather than a restrictive, sense.

[0083] The terms "including" or "comprising" are used to indicate the presence of subsequently stated features, but do not exclude the addition of other features. Conditional statements such as "can," "could," "may," "may," and the like generally mean that certain features, elements, and / or steps are included in certain embodiments, but not in other embodiments, unless otherwise specified or otherwise interpreted by context. Thus, such conditional language does not imply that the features, elements, and / or steps are in any way required by one or more embodiments, nor does it imply that one or more embodiments necessarily include logic for determining whether or not those features, elements, and / or steps are included in or should be performed in a particular embodiment, with or without user input or prompting.

[0084] Although the invention has been described with reference to certain exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the disclosed embodiments. Such embodiments of the invention may be referred to herein, individually or collectively, for convenience only by the term "invention," but are not intended to autonomously limit the scope of this application to any single disclosure or concept, even if in fact multiple disclosures or concepts are disclosed.

[0085] The embodiments described herein are described in sufficient detail to enable those skilled in the art to practice the disclosed invention. Other embodiments may be utilized and derived therefrom, and structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Therefore, the detailed description is not to be construed in a limiting sense, and the scope of the various embodiments is defined solely by the appended claims, together with all equivalents encompassed by such claims.

Claims

1. 1. A computer-implemented method comprising: Receiving a first input feature map (IrM) and one or more first filters in a first layer of a convolutional neural network (CNN) for performing convolution using a processing element (PE) array, wherein each PE in the PE array comprises (Y1) multipliers, and the PE array is arranged in (Y2) columns and (X) rows; determining a native tensor shape based on the first IFM and the one or more first filters, the native tensor shape including a first outer dimension, an inner dimension, and a second outer dimension, the native tensor shape mapping the first IFM and the one or more first filters to the PE array; receiving a second IFM and one or more second filters at a second layer of the CNN that performs convolution using the PE array; transforming the native tensor shape based on the second IFM and the one or more second filters, the transformation including expanding the inner dimension and contracting one of the first outer dimension and the second outer dimension, the expansion being performed by a factor F and the contraction being performed by a factor 1 / F; providing the one or more second filters and the second IFM according to the transformed native tensor to the PE array for convolution; summing outputs from the same row of PEs F times to obtain partial sums when the first external dimension is reduced; If the second external dimension is reduced, the convolution includes summing the outputs from all F-multiplied rows of PEs to obtain partial sums. To supply and Obtaining an output tensor of the convolution in the second layer of the CNN by summing a plurality of the partial sums, where Y1, Y2, X, and F are all integers greater than 1; A method comprising:

2. 2. The method of claim 1, wherein the second layer of the CNN is after the first layer of the CNN, and the second IFM has more input channels and a lower resolution than the first IFM.

3. The method of claim 1 , wherein each of the one or more second filters comprises multiple channels of a two-dimensional (2D) kernel, each 2D kernel having dimensions of 1×1 or 3×3.

4. providing the one or more second filters to the PE array according to the transformed native tensor; transforming the one or more second filters into a matrix according to the first outer dimension and the inner dimension of the transformed native tensor, where when each 2D kernel in the one or more second filters has the 1×1 dimension, each row of the matrix contains weights from a different input channel of the one or more second filters; Distributing the weights of each row of said matrix to different PEs so that multiple input channels are processed simultaneously at once; The method of claim 3, comprising:

5. providing the one or more second filters to the PE array according to the transformed native tensor; transforming the one or more second filters into a matrix according to the first outer dimension and the inner dimension of the transformed native tensor, where if each 2D kernel in the one or more second filters has the 3x3 dimension and includes 9 weights, the 9 weights are arranged in the same row of the matrix; Distributing the nine weights from the same row of the matrix to different PEs such that the weights from the same channel are processed simultaneously in a single operation; The method of claim 3, comprising:

6. providing the IFM to the PE array according to the transformed native tensor; transforming the IFM into a matrix according to the inner dimension and the second outer dimension of the transformed native tensor; providing input values ​​of the IFM corresponding to rows of the matrix to a row buffer of the PE; The method of claim 5 , comprising:

7. dividing the channels of the one or more filters into a plurality of channel groups, each channel group including a constant number of channels, the number being an integer greater than one; pruning each of the plurality of channel groups so that a fixed proportion of weights within each of the channel groups are non-zero; The method of claim 1 further comprising:

8. determining a depth of a buffer associated with each PE in the PE array; If the depth of the buffer is greater than the constant, configure the buffer as private memory of each PE; If the depth of the buffer is less than the constant, combining the buffer of the processing element and one or more buffers of adjacent processing elements as a shared memory. The method of claim 7.

9. 9. The method of claim 8, wherein the private memory of each PE stores input values ​​obtainable by the (Y1) multipliers within the PE.

10. 9. The method of claim 8, wherein the shared memory stores input values ​​obtainable by the (Y1) multipliers in the PE and the one or more neighboring PEs.

11. 2. The method of claim 1, wherein each row of PEs is coupled with (Y1) adder trees corresponding to the (Y1) multipliers in each PE, and each multiplier in each PE sends product outputs to a corresponding adder tree for summation.

12. each of the one or more second filters includes a plurality of non-zero weights, and providing the one or more second filters to the PE array for convolution includes: providing each non-zero weight to a multiplier of a corresponding PE as an index-value pair including the non-zero weight and a corresponding index, wherein the convolution comprises: Obtaining an input value from a buffer of the corresponding processing element according to the index; sending the obtained value and the non-zero weight to the multiplier to obtain an output; sending said output to a corresponding adder tree for summing with outputs generated by other multipliers of other PEs in the same row as said corresponding PE; The method of claim 1 , comprising:

13. 2. The method of claim 1, wherein the (Y1) multipliers in each PE process data in parallel, and the PEs in the PE array process data in parallel.

14. 1. A system for neural network computing comprising: one or more processors; and one or more non-transitory computer-readable memories coupled to the one or more processors and comprising instructions executable by the one or more processors and configured to cause the system to perform operations, Receiving a first input feature map (IFM) and one or more first filters in a first layer of a convolutional neural network (CNN) for performing convolution using a processing element (PE) array, wherein each PE in the PE array comprises (Y1) multipliers, and the PE array is arranged in (Y2) columns and (X) rows; determining a native tensor shape based on the first IFM and the one or more first filters, the native tensor shape including a first outer dimension, an inner dimension, and a second outer dimension, the native tensor shape mapping the first IFM and the one or more first filters to the PE array; receiving a second IFM and one or more second filters at a second layer of the CNN that performs convolution using the PE array; transforming the native tensor shape based on the second IFM and the one or more second filters, the transformation including expanding the inner dimension and contracting one of the first and second outer dimensions, the expansion being performed by a factor F and the contraction being performed by a factor 1 / F; providing the one or more second filters and the second IFM according to the transformed native tensor to the PE array for convolution; If the first external dimension is reduced, the convolution includes summing outputs from the same row of processing elements F times to obtain partial sums; If the second external dimension is reduced, the convolution includes summing the outputs from all F-multiplied rows of PEs to obtain partial sums. To supply and Obtaining an output tensor of the convolution in the second layer of the CNN by summing a plurality of the partial sums, where Y1, Y2, X, and F are all integers greater than 1; Including, the system.

15. 15. The system of claim 14, wherein the second layer of the CNN is after the first layer of the CNN, and the second IFM has more input channels and a lower resolution than the first IFM.

16. The calculation further comprises: dividing the channels of the one or more filters into a plurality of channel groups, each channel group including a constant number of channels, the number being an integer greater than one; pruning each channel of the one or more filters such that in each of the plurality of channel groups, only one channel contains a non-zero input value and all other channels in each channel group contain zeros; The system of claim 14 , comprising:

17. The calculation is determining a buffer depth associated with each PE in the PE array; If the depth of the buffer is greater than the constant, configure the buffer as private memory of each PE; If the depth of the buffer is less than the constant, combining the buffer of the processing element and one or more buffers of adjacent processing elements as a shared memory.

17. The system of claim 16.

18. 15. The system of claim 14, wherein each of the one or more second filters includes multiple channels of a two-dimensional (2D) kernel, each 2D kernel having dimensions of 1x1 or 3x3.

19. providing the one or more second filters to the PE array according to the transformed native tensor; transforming the one or more second filters into a matrix according to the first outer dimension and the inner dimension of the transformed native tensor, where when each 2D kernel in the one or more second filters has the 1×1 dimension, each row of the matrix contains weights from a different input channel of the one or more second filters; Distributing the weights of each row of said matrix to different PEs so that multiple input channels are processed simultaneously at once; 20. The system of claim 18, comprising:

20. 1. A non-transitory computer-readable storage medium configured to have instructions executable by one or more processors, the one or more processors comprising: Receiving a first input feature map (IFM) and one or more first filters in a first layer of a convolutional neural network (CNN) for convolution using a processing element (PE) array, wherein each PE in the PE array comprises (Y1) multipliers, and the PE array is arranged in (Y2) columns and (X) rows; determining a native tensor shape based on the first IFM and the one or more first filters, the native tensor shape including a first outer dimension, an inner dimension, and a second outer dimension, the native tensor shape mapping the first IFM and the one or more first filters to the PE array; receiving a second IFM and one or more second filters at a second layer of the CNN that performs convolution using the PE array; transforming the native tensor shape based on the second IFM and the one or more second filters, the transformation including expanding the inner dimension and contracting one of the first and second outer dimensions, the expansion being performed by a factor F and the contraction being performed by a factor 1 / F; providing the one or more second filters and the second IFM to the PE array for convolution according to the transformed native tensor; If the first external dimension is reduced, the convolution includes summing outputs from the same row of processing elements F times to obtain partial sums; If the second external dimension is reduced, the convolution includes summing the outputs from all F-multiplied rows of PEs to obtain partial sums. To supply and Obtaining an output tensor of the convolution in the second layer of the CNN by summing a plurality of the partial sums, where Y1, Y2, X, and F are all integers greater than 1; A non-transitory computer-readable storage medium that causes operations including:

Images