Method and system for hierarchical weighted sparse convolution processing

Abstract

To provide a method, a system, and a device containing a computer program coded on a computer storage medium and used for hierarchically weighted sparse convolution process.SOLUTION: A method includes: acquiring an input tensor and a plurality of filters in a convolution layer in a neural network; dividing the input tensor into a plurality of sub-tensors; assigning the plurality of sub-tensors to a plurality of processors; generating, for each of the plurality of filters, hierarchical bit expression having a plurality of key value pairs corresponding to a plurality of bits indicating whether the sub-filter includes at least one non-zero weight and to the plurality of non-zero weights in the filters, of the plurality of non-zero weights in the filters; and identifying, in order to executing a multiply accumulation (MAC), a corresponding input value form one or more of the plurality of non-zero weights and the assigned sub-tensors on the basis of the hierarchical bit expression.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure relates generally to artificial intelligence, and more particularly to hierarchical convolution with sparse weights (hereinafter referred to as "hierarchical weight sparse convolution"). [Background technology]

[0002] Neural networks (NNs) are now the foundation for many modern artificial intelligence (AI) applications, such as image and video recognition, recommender systems, classification, medical image analysis, and natural language processing. NNs are employed in a variety of usage scenarios, from self-driving cars and cancer detection to playing complex games. A typical NN can comprise a series of convolutional layers in which intensive, and therefore expensive (in terms of computation and energy), convolutional operations are performed.

[0003] In recent years, various approaches have been developed to improve the computational efficiency of neural networks by introducing sparsity into the convolution process, such as by pruning filters to reduce the number of non-zero weights. However, existing hardware architectures rely on general-purpose computing engines (e.g., GPUs) that are not optimized for convolutions involving filters with sparse weights (e.g., a limited number of non-zero weights). These architectures tend not to store sparse filters efficiently in memory or do not fully exploit sparsity to achieve high levels of massive parallelism. Therefore, it is desirable to build a new system architecture for efficient sparse convolution. Summary of the Invention

[0004] Various embodiments herein may include systems, methods, and non-transitory computer-readable media for hierarchical weight sparse convolution processing and parallelization.

[0005] According to one aspect, a method for hierarchical weight sparse convolution includes obtaining an input tensor and a plurality of filters in a convolutional layer of a neural network; dividing the input tensor into a plurality of subtensors and assigning the plurality of subtensors to a plurality of processors; and generating, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer and a second layer, the first layer corresponding to a plurality of subfilters in the filter, the corresponding subfilter comprising a plurality of bits indicating whether it has at least one non-zero weight, and the second layer comprising a plurality of bits indicating whether it has at least one non-zero weight in the filter. the plurality of non-zero weights and the corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input values ​​to obtain partial sums; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of the convolutional layer of the neural network.

[0006] In some embodiments, allocating the plurality of subtensors to the plurality of processors comprises, for each of the plurality of processors, storing one or more of the plurality of subtensors in an internal buffer of the processor.

[0007] In some embodiments, each of the plurality of subtensors and each of the plurality of subfilters have the same number of channels.

[0008] In some embodiments, identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation comprises, for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on a first tier of the hierarchical bit representation; determining an offset of the non-zero weight within the filter channel group based on a second tier of the hierarchical bit representation; and identifying the input value from the assigned subtensor based on the channel group index and offset.

[0009] In some embodiments, the hierarchical bit representation further comprises a third layer between the first layer and the second layer, the third layer comprising, for each of the plurality of sub-filters, a plurality of bit vectors respectively corresponding to a plurality of horizontal planes of the sub-filter, each of the plurality of bit vectors indicating whether the corresponding horizontal plane has at least one of a plurality of non-zero weights.

[0010] In some embodiments, identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation comprises, for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on a first tier of the hierarchical bit representation; determining a height index and a width index for the non-zero weight based on a third tier of the hierarchical bit representation; determining an offset of the non-zero weight within the filter channel group based on a second tier of the hierarchical bit representation; and identifying the input value from the assigned subtensor based on the channel group index, the height index, the width index, and the offset.

[0011] In some embodiments, the multiple filters are truncated so that each of the multiple sub-filters has either all zero weights or a predetermined number of non-zero weights.

[0012] In some embodiments, the processors are electrically arranged in a matrix having a plurality of rows and columns, with each processor having one or more columnar neighbors.

[0013] In some embodiments, each of the multiple processors is enabled to read one or more key-value pairs from the internal buffer of one of its column-wise neighboring processors.

[0014] In some embodiments, the method may further include assigning one or more of a plurality of subtensors from a channel group having a plurality of channels to rows of processors, and each of the rows of processors completing a multiply-accumulate operation between one subtensor and a corresponding sub-filter based on the hierarchical bit representation to obtain a partial sum, and then rotating the partial sum among the rows of processors.

[0015] In some embodiments, the method may further include assigning one or more of the plurality of sub-filters from the same height index and the same weight index to each of the plurality of processors; and each of the plurality of processors completing a multiply-accumulate operation between one or more of the plurality of sub-filters and a corresponding sub-filter based on the hierarchical bit representation to obtain partial sums, and then accumulating the partial sums to construct an output tensor.

[0016] According to another aspect, a system for hierarchical weighted sparse convolution is provided. The system may include one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions to cause the system to obtain an input tensor and a plurality of filters in a convolutional layer of a neural network, divide the input tensor into a plurality of subtensors and assign the plurality of subtensors to a plurality of processors, and generate, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation includes a first layer and a second layer, the first layer corresponding to a plurality of subfilters in the filter, and a plurality of bits indicating whether the corresponding subfilter has at least one non-zero weight. the second layer comprises a plurality of key-value pairs corresponding to a plurality of non-zero weights in the filter, each key-value pair comprising one of the plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key; and is executable by the one or more processors to perform operations comprising: identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and a corresponding input value from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input value to obtain partial sums; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of the convolutional layer of the neural network.

[0017] According to yet another aspect, a non-transitory computer-readable storage medium for hierarchical weight sparse convolution is provided. The medium can be configured with instructions to cause one or more processors to obtain an input tensor and a plurality of filters in a convolutional layer of a neural network, divide the input tensor into a plurality of subtensors and assign the plurality of subtensors to a plurality of processors, and generate, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer and a second layer, the first layer corresponding to a plurality of subfilters in the filter, the corresponding subfilter comprising a plurality of bits indicating whether it has at least one non-zero weight, and the second layer corresponding to the plurality of non-zero weights in the filter. The neural network is executable by one or more processors to perform operations comprising: a plurality of key-value pairs, each key-value pair having one of a plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input values ​​to obtain partial sums; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of a convolutional layer of the neural network.

[0018] These and other features of the systems, methods, and non-transitory computer-readable medium disclosed herein, together with the method of operation, function of associated elements of construction, combination of parts, and economies of manufacture, will become apparent from a consideration of the following description and the appended claims, taken in conjunction with the drawings, which form a part hereof in their entirety and in which like reference numerals indicate corresponding parts in the various views. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example convolutional neural network (CNN), according to various embodiments. [Figure 2] FIG. 2 illustrates an example convolution process, according to various embodiments. [Figure 3] FIG. 1 illustrates an example method for splitting an input tensor and a filter, according to various embodiments. [Figure 4] FIG. 2 illustrates an example hierarchical memory layout for storing non-zero weights, according to various embodiments. [Figure 5] FIG. 1 illustrates an example system for hierarchical weight sparse convolution, in accordance with various embodiments. [Figure 6A] FIG. 1 illustrates an example system architecture for hierarchical weight sparse convolution, in accordance with various embodiments. [Figure 6B] FIG. 1 illustrates another example system architecture for hierarchical weight sparse convolution, according to various embodiments. [Figure 7] FIG. 1 illustrates an example method for hierarchical weight sparse convolution, according to various embodiments. [Figure 8] FIG. 1 illustrates an example computer system in which any of the embodiments described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0020] The embodiments described herein provide methods, systems, and devices for hierarchical sparse-weight convolution in neural networks. The convolution process can refer to a basic but computationally expensive operation for extracting features of input data from previous layers, such as feature maps derived from images or audio waves, or activations output from previous layers in a neural network. The input data to the convolution process can be referred to as an input tensor, and the output of the convolution process can be referred to as an output tensor. The input tensor can comprise multiple channels of feature maps. For example, an RGB image can be represented as an input tensor with three channels (red, green, and blue), with each channel comprising a feature map (e.g., a grayscale image of the same size as the original RGB image but generated by only one of the red, green, or blue colors). To extract features from the input tensor, one or more feature extractors (also called filters) can be applied to the input tensor in a convolutional (e.g., sliding or moving) manner. A multiply-accumulate (MAC) operation between the feature detector (filter) and the input tensor can be performed during each convolution step. The outputs generated from the convolution steps can be subsequently assembled to form the output tensor of the convolution process. The output tensor can become the input tensor for the next layer in the neural network. In some cases, multiple input tensors can be included in a convolution layer.

[0021] During the convolution process, to reduce computational cost and improve inference speed, the filter can be truncated, or sparsed, by introducing a large number of zeros. For example, more than 50% of the values ​​in the filter can be set to zero. Some embodiments herein take advantage of the sparsity of the filter by efficiently storing the sparsified filter in a memory-efficient layout and parallelizing the convolution process on multiple processing entities (PEs) (e.g., processors).

[0022] In some embodiments, an input tensor can first be divided into multiple subtensors according to the number of PEs available to participate in the convolution process. Each of the subtensors can be assigned to one of multiple PEs and can remain there throughout multiple iterations of the convolution process (corresponding to multiple convolution layers in a NN) without being swapped out from the PE's local memory / buffer. After each iteration of the convolution process, the subtensor assigned to the PE as input can be updated and evolved into an input that can be used as input for the next iteration of the convolution process. In some embodiments, the updated subtensor can be temporarily swapped out of the PE for memory management purposes and swapped back in for the next iteration.

[0023] In some embodiments, each sparse filter can be divided into multiple sub-filters that can be stored in a memory-efficient layout. For example, after being truncated and divided, each sparse filter can be divided so that each sub-filter has either all zero weights or a predefined number of non-zero weights, denoted as W. The number of non-zero weights in each sub-filter can be affected by the size of the sub-filter (e.g., a larger size of the sub-filter can include more non-zero weights) and the desired feature extraction accuracy (e.g., too few non-zero weights can cause feature loss). In some embodiments, based on knowledge of the number of non-zero weights in each sub-filter (either 0 or W), each sparse filter can be represented in a hierarchical bit representation (e.g., memory layout shape) with multiple multiplication layers. For example, the top layer can include multiple bits to represent whether multiple blocks in the filter contain non-zero weights, the second layer can similarly include multiple bits to represent whether multiple sections within each of the blocks contain non-zero weights, and the bottom layer can store the actual non-zero weights. The number of layers in each hierarchical bit representation can be determined based on, for example, how the filter is divided.

[0024] Because the sparse filter is represented as a hierarchical bit representation, the corresponding convolution process can be referred to as hierarchical weight sparse convolution. The hierarchical bit representation of the filter can enable each of the PEs to store and iterate non-zero weights more efficiently. For example, the hierarchical bit representation can significantly reduce the memory footprint of the filter (e.g., each PE can load less data from a global buffer into its local buffer to perform local convolution), making the solution suitable for devices with limited memory resources. For example, the embodiments described herein can be applied to edge devices in edge computing systems. See FIG. 4 for a detailed description of the hierarchical bit representation. In some embodiments, subfilters and subtensors can be assigned to multiple PEs in a manner that ensures that subtensor-subfilter pairs assigned to different PEs are independent of each other, allowing multiple PEs to perform local operations in parallel, improving performance. See FIGS. 6A and 6B for a detailed description of the assignment.

[0025] The partitioning and parallel processing scheme described above is also highly scalable. In some embodiments, each round (also called an iteration) of parallel processing can generate multiple partial sums, and multiple PEs can perform parallel processing on multiple rounds to accumulate the partial sums generated in each round and assemble the accumulated results as an output tensor. In some embodiments, the partial sums generated by each PE over multiple rounds can be accumulated directly (within each PE) to generate a portion of the output tensor without having to exchange data with other PEs.

[0026] In the following description, specific, non-limiting embodiments of the present invention are described with reference to the figures. The particular features and aspects of any embodiment disclosed herein can be used with and / or combined with the particular features and aspects of any other embodiment disclosed herein. It should also be understood that such embodiments are examples and represent only a small number of embodiments within the scope of the present invention. Various changes and modifications obvious to those skilled in the art to which the invention pertains are deemed to be within the spirit, scope, and intent of the invention, as further defined in the appended claims.

[0027] 1 illustrates an example convolutional neural network (CNN) according to various embodiments. CNNs are widely used in a variety of applications, including image understanding, speech recognition, gameplay, and robotics, and can employ deep layers, including convolutional layers, where convolutional operations occur. It should be noted that CNNs are used for exemplary purposes only, and that the embodiments disclosed herein can be applied to other neural networks that include convolutional operations.

[0028] The example neural network 100 shown in FIG. 1 includes multiple layers, such as a first convolutional layer 120, a first rectified linear unit (ReLU) layer, a first pooling layer 130, a second convolutional layer 140, a second rectified linear unit (ReLU) layer, a second pooling layer 150, a flattening layer 160, and a fully connected (FC) layer 170. The example neural network 100 can be trained to match the alphabet in a given image with multiple known alphabet classes. As shown in FIG. 1, an image 110 with characters is input to the neural network 100 and transformed through multiple layers. The final layer (FC layer 170) results in multiple scores representing the similarity between the characters in the input image 100 and each of the known alphabet classes.

[0029] In some embodiments, the input image 110 can first be converted to an input tensor. As an example, if the input image 110 includes 32*32 pixels, each with three color channels (red, green, and blue), then its corresponding input tensor can have a size of 32*32*3, i.e., a height of 32, a width of 32, and a depth (e.g., number of channels) of 3. For ease of description, the three-dimensional size can be referred to as HWC format, where H refers to the height of the input tensor (e.g., 32 in the example image above), W refers to the width of the input tensor (e.g., 32 in the example image above), and C refers to the number of channels in the input tensor (e.g., 3 in the example image above). In some cases, when there are multiple input tensors for a convolutional layer (e.g., when there are multiple images input to a convolutional layer or when there are multiple input activation tensors received from a previous layer), each input tensor can be represented in NHWC format, where N refers to the index of the input tensor within the collection of input tensors. In the following description, unless explicitly specified otherwise, N can be omitted for brevity (e.g., by assuming there is only one input tensor). It will be obvious to those skilled in the art to extend the embodiments to include the case where N>1.

[0030] In some embodiments, a CNN may include multiple convolutional layers (e.g., the first convolutional layer 120 and the second convolutional layer 140 in FIG. 1 ). The convolutional layers convert an input tensor (e.g., an original input image or an input tensor from a previous layer) into a form that is easier to process without losing important features for good prediction / classification. One or more feature detectors in image processing, such as an edge detector or a curve detector, may be involved in the convolutional processing in the convolutional layers. These feature detectors may be referred to as filters. Each of the filters may have the same number of channels as the input tensor. For ease of description, the term "sharing multiple channels" is used herein to mean that each filter and the input tensor have the same number of channels. For example, the input tensor may be a 32*32*3 matrix, and the exemplary filter may be a 3*3*3 matrix. For a detailed description of the convolutional processing, please refer to the description of FIG. 2.

[0031] Convolutional layers in CNNs can be followed by nonlinear activation functions to introduce nonlinearity into the CNN. Exemplary nonlinear activation functions include sigmoid, hyperbolic tangent, and rectified linear unit (ReLU). As shown in FIG. 1, a ReLU function (which can also be referred to as a ReLU layer) follows each of the convolutional layers 120 and 140. The ReLU function can apply a component-wise activation function to remove some of the outputs (activations) generated by the convolutional layer 120 or 140. For example, a max(0,x) function can be employed to remove all negative activations in the convolutional output and feed only positive activations to the next layer. The ReLU function cannot change the size of the output activations, but because all negative activations have been removed, it can limit the number of active neurons in subsequent layers to improve computational efficiency.

[0032] A CNN may also include one or more pooling layers to provide various operations for reducing the dimensionality of an input tensor. In Figure 1, the first pooling layer 130 and the second pooling layer 150 of neural network 100 may each perform a downsampling operation on the input tensor along the spatial dimensions (height and width), but typically leave the depth dimension (e.g., number of channels) unchanged.

[0033] A fully connected (FC) layer in a neural network can learn nonlinear combinations of high-level features as represented by the output from the previous layer (e.g., pooling layer 150 in FIG. 1, or a convolutional layer) and compute class scores accordingly. Referring to FIG. 1, the output from pooling layer 150 can first be flattened into a vector (e.g., in flattening layer 160), from which a unique untrained function in that space can be learned by FC layer 170 to compute final scores for multiple classes. Each score can represent a quantified similarity between the alphabet in the input image and the corresponding class.

[0034] FIG. 2 illustrates an example convolution process according to various embodiments. A typical convolution process in a neural network can begin by receiving (or obtaining) one or more input tensors and multiple filters and executing a convolution process to generate one or more output tensors. For simplicity, the example convolution process 220 in FIG. 2 includes one input tensor 222, one filter 224, and one output tensor 225. The input tensor 222 can be described by its height (H), width (W), and number of channels (C). As shown in FIG. 2, the input tensor 222 can be a 3*3*3 matrix with a height of 3, a width of 3, and a number of channels of 3. The height and width of the input tensor 222 for each channel (e.g., 3(H)*3(W)) can be referred to as a two-dimensional input feature map. A tuple of {h,w,c} can refer to one element of the two-dimensional input feature map for channel c. For example, {2,1,1} can refer to the element of the two-dimensional input feature map in channel 1, where {2,1} (e.g., the height index is 2 and the width index is 1) is defined. The filter 224 can be described by its height (R), width (S), and number of channels (C). In FIG. 2, the filter 224 can be a 1*1*3 matrix. The height and width of the filter 224 in each channel (e.g., 1(R)*1(S)) can be referred to as a kernel (the filter 224 has three kernels in each of the three channels).

[0035] In some scenarios, the convolution process can include a stride (the width the filter moves) and padding (squeezing pixels). For example, when the stride is 1, the filter convolves (e.g., moves, slides) one pixel at a time, and when the stride is 2, the filter convolves two pixels at a time as it slides. A larger stride produces a spatially smaller output tensor (a smaller H*W in the output tensor). As another example, when the spatial size (height*width) of the filter is greater than 1*1 (e.g., 3*3 or 5*5), the input tensor can be filtered and padded around its boundary with zeros to control the spatial size of the output tensor, for example, to keep the spatial size of the input tensor so that the input and output heights and widths are the same. In FIG. 2, no padding is performed on input tensor 222, and a stride of 1 is assumed.

[0036] During the convolution process 220 shown in FIG. 2, a multiply-and-accumulate operation is performed on the filter 224 and each depth slice, such as the first depth slice 223, of the input tensor to generate a dot product, such as dot product 228. For example, the first depth slice 223 of the input tensor 222 is the 1*1*3 tensor (three gray cubes) at the top left of the input tensor 222. Both the first depth slice 223 and the filter 224 have a size of 1*1*3. After the multiply-and-accumulate operation, the generated dot product 228 can be assembled as part of the output tensor 225. Thus, the output tensor 225 can be determined after the filter 224 has convolved (e.g., translated) all depth slices (9 slices in FIG. 2) in the input tensor 222. The number of channels in the output tensor 225 is equal to the number of filters applied during the convolution. Because the convolution process 220 only uses one filter 224, the corresponding output tensor 228 only has one channel.

[0037] In comparison, convolution process 260 includes two filters 264 and 265. Filter 264 can be convolved with input tensor 262 to determine the values ​​of output tensor 268 in a first channel (e.g., a two-dimensional output feature map in channel 1). Filter 265 can be convolved with input tensor 262 to determine the values ​​of output tensor 268 in a second channel (e.g., a two-dimensional output feature map in channel 2). Thus, the resulting output tensor 268 comprises two channels of two-dimensional output feature maps.

[0038] It is noteworthy that some operations during the convolution process can be parallelized. For example, a multiply-and-accumulate operation performed on a filter is independent of the multiply-and-accumulate operation on another filter because different filters generate two-dimensional output feature maps for different output channels. As another example, depth slices on an input tensor are independent of each other because they generate values ​​for different locations on the respective two-dimensional output feature maps. Theoretically ideal parallelization can be achieved if all these operations can be parallelized across multiple processing entities (e.g., processors, cores, or threads). However, because real-world applications may have a large number of filters and a large number of input tensors, pursuing theoretically ideal parallelization may require hardware with unachievable computing power (e.g., hardware with a huge number of processing entities and memory exchange mechanisms). The embodiments disclosed in the following description provide methods (and corresponding systems, storage media) for achieving ideal parallelization by fully utilizing the parallel processing capabilities of given hardware, which is a more scalable and practical approach.

[0039] FIG. 3 illustrates an example method for splitting input tensors and filters, according to various embodiments. During each convolution process, each input tensor 322 and each filter 342 and 343 share multiple channels (i.e., have the same number of channels). Without loss of generality, each input tensor can be represented as a three-dimensional tensor having a height dimension (H), a width dimension (W), and a channel dimension (C). Each filter can similarly be represented as a three-dimensional tensor having a height dimension (R), a width dimension (S), and a channel dimension (C). Each dimension can include multiple corresponding dimensional units. For example, the input tensor 323 in FIG. 3 can be represented as 3(H)*3(W)*6(C), where the H dimension includes three height units, the W dimension includes three width units, and the C dimension includes six channels. In some embodiments, multiple input tensors can be included in the convolution process, and the H*W*C representation of each of the multiple input tensors can be extended with an additional dimension N (e.g., representing an input tensor identifier).

[0040] In some embodiments, the input tensor 322 can be partitioned into multiple subtensors 323. For example, the partitioning can ensure that the multiple subtensors can be evenly distributed among the available PEs so that the workload on the PEs is evenly distributed. In some embodiments, the size of each subtensor can be expressed as Hg*Wg*Cg, where Hg refers to the height dimension of the subtensor (e.g., "g" represents group, and "Hg" represents a group of height units within the subtensor), Wg refers to the width dimension of the subtensor, and Cg refers to the channel dimension of the subtensor, or a channel group 324 that includes one or more continuous channels. Taking the input tensor 322 in FIG. 3 as an example, the subtensor 323 (shaded region) can have a size of 1(Hg)*1(Wg)*2(Cg). In an actual implementation, Hg and Wg may be equal to or greater than 1, and Cg may be a large positive number such as 8, 16, or 32. In some embodiments, a subtensor such as 323 may be retrieved in part at a time by a single PE rather than retrieved entirely at once (e.g., due to limited size of the PE's internal buffer).

[0041] In some embodiments, the filter 342 can be divided into multiple subfilters, and each of the multiple subtensors and each of the multiple subfilters can have the same number of channels; for example, they can have a channel group Cg 324 in the channel dimension. The division of the filter 342 can be implemented in various ways and with various granularities. For example, the filter 342 can be divided into filter blocks 345, each of which has a size of R*S*Cg and is treated as a subfilter. As another example, each of the filter blocks 345 can be further divided into multiple horizontal planes, each of which has a size of 1*S*Cg and is treated as a subfilter 346. Assuming the filter 342 has a size of 3(R)*3(S)*C, the subfilter 346 can have a size of 1(R)*3(C)*Cg, and the number of channels C is divided into multiple channel groups, each of which has Cg channels.

[0042] In some embodiments, filters 342-344 can be pruned or sparsed to limit the number of non-zero weights therein. Sparsified filters can significantly increase computational efficiency without significantly losing accuracy in feature extraction from the input tensor. In some embodiments, filters 342-344 can be pruned so that each of the subfilters (e.g., 345 or 346) has either all zero weights or a predetermined number of non-zero weights. The predetermined number of non-zero weights in each subfilter are used to capture desired features from the input tensor 322, and all-zero subfilters are useful in reducing the storage / memory footprint and computational cost of the filter. In some embodiments, the pruning process can be performed in multiple steps. For example, a first step is to truncate each of the filter blocks 345 so that they contain the same number of non-zero weights (although the locations of the non-zero weights may be different within each filter block 345), then within each filter block 345, the sub-filters 346 therein may be truncate so that the sub-filters 346 have the same number of non-zero weights, and finally, some of the sub-filters 346 may have non-zero weights removed to create all-zero sub-filters.

[0043] In some embodiments, multiple filters can be present in a single convolutional layer to capture features in the input tensor 322 from different angles. For example, FIG. 3 shows that filters 342-344 can be used to convolve with the input tensor. In some embodiments, multiple filters 342-344 (denoted as K filters) can be divided into filter groups (denoted as Kg), each filter group having a fixed number of filters. Within each filter group, subfilters from the same channel group can form a subfilter group, which can be provided to a processing element (PE) to perform local convolution with the corresponding subtensor 323. For example, assuming filters 342 and 343 are the same filter group, subfilter 346 from the first channel group of filter 342 and the subfilter from the first channel group of filter 343 can form a subfilter group, which can be provided to a processing element (PE) to perform convolution with the subtensor 323 from the same first channel group.

[0044] FIG. 4 illustrates an example hierarchical memory layout for storing non-zero weights, according to various embodiments. The example filter 420 in FIG. 4 is assumed to be a 3(R)*3(S)*32(C) tensor for illustrative purposes and can refer to one of the filters shown in FIG. 3. We also assume that the filter 420 is divided into multiple channel groups, each with multiple channels (Cg), and that within each channel group, the filter block R*S*Cg is further divided into multiple sub-filters, such as multiple horizontal planes, each denoted as 1(R)*3(S)*Cg. As shown in FIG. 4, a value of 8 is assumed for Cg. These assumptions are merely for simplicity and clarity and can be easily modified or relaxed to meet different methods of filter division or actual needs of convolution.

[0045] As shown in FIG. 4, filter 420 is divided into multiple filter blocks, such as the first filter block (shaded cube) 3 (R) * 3 (S) * 8 (Cg). The first filter block can be further divided into three horizontal planes 432, 433, and 434, corresponding to R=0, R=1, and R=2, respectively. Each of horizontal planes 432, 433, and 434 can be represented as a two-dimensional table with a width of 3 (corresponding to S) and a height of 8 (corresponding to Cg). Note that each of the sub-filters (horizontal planes) is already truncated to contain either all zeros (such as 434) or a predetermined number of non-zero weights (four non-zero weights in each of 432 and 433). For example, table 432 corresponding to plane R=0 can be configured with four non-zero weights W0 through W3, with all other values ​​set to zero. Among the non-zero weights, W0 and W1 are located in column S=0, and W2 and W3 are located in column 1. Similarly, table 433 corresponding to plane R=1 also has four non-zero weights, with W4 and W5 located in column S=0, and W6 and W7 located in column S=2. Table 434 corresponding to plane R=2 has no non-zero weights.

[0046] 4, the filter 420 can be stored in the memory of the PE as a hierarchical bit representation to represent the non-zero weights in the filter 420. The hierarchical bit representation of the filter 420 can be generated in various forms depending on the structure of the filter 420. In some embodiments, if the filter has a size of 1(R)*1(S)*C and is divided into multiple filter blocks, each of which is 1(R)*1(S)*Cg, the hierarchical bit representation of the filter can have a first layer and a second layer. The first layer can have multiple bits corresponding to the multiple filter blocks in the filter and indicating whether the corresponding filter block has at least one non-zero weight. The second layer can have multiple key-value pairs corresponding to the multiple non-zero weights in the filter, each key-value pair having one of the multiple non-zero weights as a value and a channel identifier of the non-zero weight as a key.

[0047] In some embodiments, the number of layers in the hierarchical bit representation of filter 420 can be increased to three if the width and height dimensions of filter 420 are greater than one. For example, a first layer can include a plurality of bits corresponding to a plurality of filter blocks in the filter and indicating whether the corresponding filter block has at least one non-zero weight. A second layer can include a plurality of bit vectors for each of the plurality of filter blocks, each bit vector corresponding to a plurality of horizontal planes in the filter block, each bit vector indicating whether the corresponding horizontal plane has at least one non-zero weight. A third layer can include a plurality of key-value pairs corresponding to a plurality of non-zero weights in the filter, each key-value pair having one of the plurality of non-zero weights as a value and a channel identifier of the non-zero weight as a key.

[0048] For example, assuming filter 420 has a first filter block (shaded cube) with some non-zero weights and a second filter block with all zero weights, hierarchical bit representation 440 for filter 420 can be constructed as shown in FIG. 4. The first layer of hierarchical bit representation 440 includes a number of bits indicating whether the corresponding filter block includes at least one non-zero weight. In this example, the first bit corresponding to the first filter block is 1, and the second bit corresponding to the second filter block is 0 (indicating an all-zero filter block). The second layer of hierarchical bit representation 440 includes a number of bit vectors corresponding to the horizontal planes of the filter block that include at least one non-zero weight. In this example, the first filter block has three horizontal planes, represented as tables 432, 433, and 434, where table 432 includes non-zero weights in columns 0 and 1, table 433 includes non-zero weights in columns 0 and 2, and table 434 includes only zeros. Thus, the first bit vector in the second layer of the hierarchical bit representation 440 has a value (1,1,0), indicating that the first horizontal plane (R=0) has non-zero weights in the first column (S=0) and the second column (S=1). The second bit vector in the second layer of the hierarchical bit representation 440 has a value (1,0,1), indicating that the second horizontal plane (R=1) has non-zero weights in the first column (S=0) and the third column (S=2). And the third bit vector in the second layer of the hierarchical bit representation 440 has a value (0,0,0), indicating that there are no non-zero values ​​in the corresponding horizontal plane (R=2). The third layer of the hierarchical bit representation 440 may contain actual non-zero values. 4, non-zero weight W0 is located on channel 2 (the idx value in table 432 refers to the channel index starting from 0), so the key-value pair corresponding to W0 in tier 3 is (2,W0). Similarly, non-zero weight W1 is represented as (7,W1), indicating that non-zero weight W1 is located on channel 7.

[0049] The hierarchical bit representation 440 efficiently stores the necessary information of the non-zero weights in the filter 420. In some embodiments, the hierarchical bit representation 440 can provide the precise location of each non-zero weight by determining, based on a first layer of the hierarchical bit representation 440, a channel group index corresponding to the filter channel group in which the non-zero weight is located, by determining, based on a second layer of the hierarchical bit representation, a height index and a width index for the non-zero weight, and by determining, based on a third layer of the hierarchical bit representation 440, the offset of the non-zero weight within the filter channel group (i.e., channel index).

[0050] During the convolution process, each of multiple processing entities (PEs) can be assigned one or more filters (e.g., filter groups) or one or more filter blocks (e.g., from the same channel group) and one or more corresponding subtensors (e.g., portions of the input tensor from the same channel group as the filter / filter block) to perform local product-accumulation (MAC) operations to generate partial sums.

[0051] To perform the multiply-and-accumulate operation for each PE, the input values ​​(in the subtensor) corresponding to the assigned non-zero weights (in the filter or filter block) can be identified based on the hierarchical bit representation of one or more filters or filter blocks. The following description uses the hierarchical bit representation of the filter 420 as an example to illustrate how the corresponding input values ​​are identified. In FIG. 4, it is assumed that the filter 420 is assigned to a PE. Each of the non-zero weights in the filter 420 can be enumerated and multiplied by the corresponding input value in the input tensor. For example, the PE can start from the first layer by identifying the first non-zero value indicating a filter block with a non-zero weight. The position of the non-zero value can indicate which channel group it corresponds to. In FIG. 4, the first bit in the first layer is 1, which means that the filter block of the first channel group has at least one non-zero weight. The PE can then check the second layer by locating a fixed number of bit vectors corresponding to the filter blocks, where the fixed number refers to the number of horizontal planes in each filter block (e.g., the three horizontal planes 432, 433, and 434 in the first filter block in FIG. 4). Each of these bit vectors can contain a fixed number of bits, each corresponding to the number of columns in the S dimension of the filter 420 (e.g., the three columns S=0, 1, 2 in each horizontal plane 432, 433, or 434). Based on the location of the bit vector in the second layer, an index in the R dimension can be determined, and based on the bit values ​​in the bit vector, an index in the S dimension can be determined. Finally, the third layer contains the channel identifier and the actual nonzero weights. By knowing the three indices in the R, S, and channel dimensions, the PE can locate the corresponding input values ​​from the input tensor and perform a multiply-accumulate operation based on the nonzero weights and input values ​​to obtain partial sums. The partial sums obtained by multiple PEs can be accumulated to construct the output tensor of the convolutional layer.

[0052] In some embodiments, each layer in the hierarchical bit representation 440 can maintain a pointer to a previously processed bit, bit vector, or key-value pair. During enumeration of non-zero weights, the pointer can assist the PE in determining the next filter block, bit vector, or key-value pair to process. In some embodiments, the hierarchical bit representation 440 can be stored as a tree with a root node corresponding to the first layer. Each non-zero bit in the root node can have a subtree containing one or more child nodes storing bit vectors belonging to the filter block corresponding to the non-zero bit, where each bit vector corresponds to a horizontal plane in the filter block. Each non-zero bit in each bit vector can have one or more grandchild nodes corresponding to one or more columns in each horizontal plane of the filter block and storing key-value pairs with non-zero weights in one or more columns.

[0053] The hierarchical bit representation 440 provides an efficient way to store non-zero weights along with all the information needed to locate the corresponding input value to perform the convolution. For example, if a bit in the first layer is 0, the entire corresponding filter block can be skipped. In this case, the entire filter block only takes up one bit of storage space in the PE.

[0054] FIG. 5 illustrates an example system diagram for hierarchical weight sparse convolution processing, according to various embodiments. The architecture illustrated in FIG. 5 includes a computer system designed to perform inference when a neural network is loaded and executed. In some embodiments, such a computer system is implemented as a hardware accelerator 500. Several components in the hardware accelerator 500 can cooperate in a specific manner to improve the throughput and energy efficiency of the neural network by facilitating parallel convolution operations across multiple processing entities. The accelerator 500 is exemplary only and may include more, fewer, or alternative components. The hardware accelerator 500 can be designed as a reconfigurable device, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0055] As shown in FIG. 5 , the hardware accelerator 500 may include a scheduler 570 to control the workflow within the accelerator 500 and the interaction between a host CPU 510 and off-chip components such as a double data rate (DDR) memory 520. For example, the accelerator 500 may interact with the host CPU 510 through a peripheral component interconnect express (CPIe) physical layer (PHY) controller 512 and with the off-chip DDR memory 520 through a DDR interface 530. The accelerator 500 may retrieve data from the off-chip DDR memory 520 through a direct memory access (DMA) controller 540, which communicates with the off-chip DDR memory 520 via the DDR interface 530. The retrieved data may be stored in an on-chip buffer called a global buffer 550 in preparation for parallel convolution operations. The global buffer 550 may be logically divided into multiple sections, such as an input buffer 552, a weight buffer 554, and an output buffer 556. The input buffer 552 can store data associated with input tensors, the weight buffer 554 can store data associated with filters, and the output buffer 556 can store data associated with the results of the convolution operations (e.g., output tensors). The global buffer 550 can exchange data with an array 560 of sparse processing units (SPUs) 562 to perform the convolution operations. The "sparse" in the SPU's name indicates that the SPU array 560 is specifically designed to efficiently perform convolution operations involving sparsity (e.g., sparse input tensors and / or sparse filters). Each of the SPUs 562 can include multiple processing entities (PEs) 580, each capable of handling a unit amount of convolution operations.

[0056] 6A shows an example diagram for hierarchical weight sparse convolution, according to various embodiments. For illustrative purposes, FIG. 6A shows how subtensors (shaded portions of input tensor 600) and sub-filter groups (shaded portions of filter 610) are assigned to multiple PEs to perform the convolution in parallel.

[0057] In some embodiments, the PEs may be arranged in a matrix having multiple rows and columns of PEs, and each of the PEs may be capable of exchanging data with its neighboring PEs (e.g., reading a non-zero weight from the internal buffer of one of its column-wise neighboring PEs, or rotating partial sum results between its row-wise neighboring PEs).

[0058] In some embodiments, the input tensor 600 (H*W*C tensor) can first be divided into multiple subtensors, e.g., multiple W*Cg subtensors (e.g., multiple horizontal planes in the input tensor 600), where Cg refers to a channel group containing multiple channels. These subtensors can then be assigned to multiple PEs or stored in internal buffers of multiple PEs. As shown in FIG. 6A , each row of PEs (e.g., PE0, PE32, ..., PE224 in the first row 630) can be assigned a portion of the W*Cg subtensor. Each PE can then individually go through the horizontal planes in the filter (e.g., S*Cg) to enumerate all non-zero weights. For each identified non-zero weight, the channel dimension index, R dimension index, and S dimension index of the non-zero weight can be easily deduced from the corresponding hierarchical bit representation, and the PE can easily locate the corresponding input value from the assigned subtensor based on the three indices. For non-zero weights and corresponding input values, a multiply-accumulate operation can be performed to generate partial sums. A PE can skip an entire filter block or an individual channel group if the corresponding bit is 0. In some embodiments, each PE can read more non-zero weights from its neighboring PEs, if necessary, through the weight exchange connection 624 between every two PEs in the same column 640. In some embodiments, when the slowest PE in the same row 630 completes enumerating all assigned non-zero weights, the partial sums generated by each PE in the same row 630 can be rotated through the rotate connection 622 for the next round of processing (e.g., these partial sums can be further accumulated with new partial sums generated in the next round). Here, the "slowest PE" can refer to the PE with the least computing power or the PE with the heaviest workload (e.g., most of the non-zero weights to process). In some embodiments, these partial sums can be accumulated to build the output tensor of the convolutional layer.

[0059] The example assignment of filters and input tensors shown in FIG. 6A can potentially provide a computation time benefit if any slowest PE in a row can reduce its computation time for the convolution process in a neural network, as well as a significant power benefit by skipping memory accesses and multiplications (by avoiding the all-zero portion of the filter).

[0060] FIG. 6B shows another example diagram for hierarchical weight-sparse convolution, according to various embodiments. In FIG. 6B, the input tensor 600 and filter 610 are partitioned and assigned to multiple PEs in the same manner as the assignment shown in FIG. 6A. For example, each PE can read more non-zero weights from its column-wise neighboring PEs through weight exchange connections 624, if needed. The difference between these two diagrams in FIGS. 6A and 6B is that each PE continues to go through all S*Cg of the filter 610 until the final result for the corresponding output channel is calculated. In this way, the partial sums do not need to be rotated among PEs in the same row. In some embodiments, these partial sums can be accumulated to build the output tensor of the convolutional layer.

[0061] The example assignment of filters and input tensors shown in FIG. 6B can provide the following benefits to the convolution process in a neural network: one PE can skip operations for an entire output channel computation, thereby saving computation time, and one PE can skip operations for a whole output channel computation, thereby saving power, due to skipping memory accesses and multiplications.

[0062] FIG. 7 illustrates an example method for hierarchical weight sparse convolution, according to various embodiments. Method 700 may be performed by a device, apparatus, or system for optimizing resource allocation. Method 700 may be performed by one or more modules / components of the environments or systems illustrated by FIGS. 1-6, such as hardware accelerator 500 in FIG. 5. The operations of method 700 presented below are intended to be examples. Depending on the implementation, method 700 may include additional, fewer, or alternative steps performed in various orders or in parallel.

[0063] Block 710 includes obtaining an input tensor and a plurality of filters in a convolutional layer of a neural network. In some embodiments, the plurality of filters are pruned such that each of a plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights.

[0064] Block 720 includes dividing the input tensor into a plurality of subtensors and assigning the plurality of subtensors to a plurality of processors. In some embodiments, assigning the plurality of subtensors to the plurality of processors comprises, for each of the plurality of processors, storing one or more of the plurality of subtensors in an internal buffer of the processor.

[0065] Block 730 includes, for each of the plurality of filters, generating a hierarchical bit representation of a plurality of non-zero weights in the filter, where the hierarchical bit representation comprises a first layer and a second layer, the first layer comprising a plurality of bits corresponding to a plurality of subfilters in the filter and indicating whether the corresponding subfilter has at least one non-zero weight, and the second layer comprising a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair comprising one of the plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key. In some embodiments, each of the plurality of subtensors and each of the plurality of subfilters have the same number of channels.

[0066] Block 740 includes identifying, by each of the plurality of processors based on the hierarchical bit representation, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor. In some embodiments, identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation comprises, for each of the plurality of non-zero weights, determining, based on a first tier of the hierarchical bit representation, a channel group index corresponding to a filter channel group in which the non-zero weight is located, determining, based on a second tier of the hierarchical bit representation, an offset of the non-zero weight within the filter channel group, and identifying the input value from the assigned subtensor based on the channel group index and offset.

[0067] In some embodiments, the hierarchical bit representation further comprises a third layer between the first layer and the second layer, the third layer comprising, for each of the plurality of subfilters, a plurality of bit vectors each corresponding to a plurality of horizontal planes of the subfilter, wherein each of the plurality of bit vectors indicates whether the corresponding horizontal plane comprises at least one of a plurality of non-zero weights. In some embodiments, identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation comprises, for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on the first layer of the hierarchical bit representation, determining a height index and a width index for the non-zero weight based on the third layer of the hierarchical bit representation, determining an offset of the non-zero weight within the filter channel group based on the second layer of the hierarchical bit representation, and identifying the input value from the assigned subtensor based on the channel group index, height index, width index, and offset.

[0068] Block 750 involves performing a multiply-accumulate (MAC) operation on one or more identified non-zero weights and corresponding input values ​​to obtain partial sums.

[0069] Block 760 involves accumulating the partial sums obtained by each of the multiple processors to construct an output tensor of the convolutional layer of the neural network.

[0070] In some embodiments, the multiple processors can be electrically arranged in a matrix having multiple rows and columns, with each of the multiple processors having one or more column-wise neighboring processors. In some embodiments, each of the multiple processors can be enabled to read one or more key-value pairs from an internal buffer of one of its column-wise neighboring processors. In some embodiments, method 700 can further include assigning one or more of a plurality of subtensors from a channel group having multiple channels to a row of processors, and after each row of processors completes a multiply-accumulate operation between one subtensor and a corresponding subfilter based on the hierarchical bit representation to obtain a partial sum, rotating the partial sums among the row of processors. In some embodiments, method 700 can further include assigning one or more of a plurality of subfilters from the same height index and the same weight index to each of the multiple processors, and after each of the multiple processors completes a multiply-accumulate operation between one or more of the multiple subfilters and a corresponding subfilter based on the hierarchical bit representation to obtain a partial sum, accumulating the partial sums to construct an output tensor.

[0071] Figure 8 illustrates an example computing device capable of implementing any of the embodiments described herein. The computing device can be used to implement one or more components of the systems and methods illustrated in Figures 1-7. The computing device 800 can include a bus 802 or other communication mechanism for communicating information, and one or more hardware processors 804 coupled to the bus 802 for processing information. The hardware processors 804 can be, for example, one or more general-purpose microprocessors.

[0072] The computing device 800 may also include a main memory 807 coupled to the bus 802 for storing information and instructions executed by the processor 804, such as a random access memory (RAM), cache, and / or other dynamic storage device. The main memory 807 may also be used to store temporary variables or other intermediate information during execution of instructions executed by the processor 804. When such instructions are stored on a storage medium accessible by the processor 804, they may transform the computing device 800 into a special-purpose machine customized to perform the operations specified in the instructions. The main memory 807 may include non-volatile and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks. Volatile media may include dynamic memory. Common forms of media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical memory with a pattern of holes, a RAM, a DRAM, a PROM, an EPROM, a flash EPROM, a NVRAM, any other memory chip or cartridge, or a network version thereof.

[0073] The computing device 800 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computing device, cause the computing device 800 to be, or can be programmed to be, a special-purpose machine. According to one embodiment, the techniques herein are performed by the computing device 800 in response to the processor 804 executing one or more sequences of one or more instructions contained in the main memory 807. Such instructions may be read into the main memory 807 from another storage medium, such as the storage device 809. Execution of the sequences of instructions contained in the main memory 807 causes the processor 804 to perform the process steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in the main memory 807. When executed by the processor 804, these instructions may perform the steps illustrated in the corresponding figures and described above. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0074] Computing device 800 also includes a communication interface 810 coupled to bus 802. Communication interface 810 may provide a two-way data communication coupling to one or more network links that are connected to one or more networks. As another example, communication interface 810 may be a local area network (LAN) card that provides a data communication connection to a compatible LAN (or a WAN component that communicates with a WAN). Wireless links may also be implemented.

[0075] The performance of certain operations may be distributed among processors and may reside within a single machine or may be spread across multiple machines. In some example embodiments, the processor or the engine implemented by the processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or the engine implemented by the processor may be distributed across multiple geographic locations.

[0076] Each of the processes, methods, and algorithms described in the preceding sections can be embodied in, and fully or partially automated by, one or more computer systems or computer processors comprising computer hardware. The processors and algorithms can be implemented partially or entirely in application-specific circuitry.

[0077] When the functions disclosed herein are implemented in the form of software functional units and sold or used as an independent product, the functions can be stored in a non-volatile computer-readable storage medium executable by a processor. The specific technical solutions disclosed herein (in whole or in part), or aspects that contribute to the current art, can be embodied in the form of a software product. The software product can be stored in a storage medium, with a number of instructions that cause a computing device (which may be a personal computer, a server, a network device, etc.) to perform all or some steps of the method of the embodiments of the present application. The storage medium can include a flash drive, a portable hard drive, a ROM, a RAM, a magnetic disk, an optical disk, other media operable to store program code, or any combination thereof.

[0078] Particular embodiments further provide a system comprising a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in the method of any of the above-disclosed embodiments.Particular embodiments further provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in the method of any of the above-disclosed embodiments.

[0079] The embodiments disclosed herein can be realized through a cloud platform, a server, or a group of servers (hereinafter collectively referred to as a "server system") that interact with a client. The client may be a terminal device or a client registered by a user in the platform, and the terminal device may be a mobile terminal, a personal computer (PC), or any device capable of installing a platform application program.

[0080] The various features and processors described above can be used independently of each other or can be combined in various ways. All possible combinations and subcombinations are intended to be within the scope of the present disclosure. Additionally, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states associated therewith may be performed in other sequences as appropriate. For example, the described blocks or states may be performed in an order other than the order specifically disclosed, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be performed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the example embodiments disclosed. The example systems and components described herein may be configured differently from the systems and components described. For example, elements may be added to, removed from, or rearranged relative to the example embodiments disclosed.

[0081] Various operations of the example methods described herein may be performed, at least in part, by an algorithm. The algorithm may comprise program code or instructions stored in a memory (e.g., a non-transitory computer-readable storage medium as described above). Such an algorithm may comprise a machine learning algorithm. In some embodiments, a machine learning algorithm may not explicitly program a computer to perform a function, but may learn from training data to generate a predictive model that performs that function.

[0082] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the operations. Whether temporarily or permanently configured, such processors may constitute a processor-implemented engine that operates to perform one or more operations or functions described herein.

[0083] Similarly, the methods described herein may be implemented at least in part by a processor, with a particular processor being an example of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented engines. Furthermore, one or more processors may also be operable to support execution of the operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as an example of a machine that includes a processor), and these operations may be accessible over a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application program interfaces (APIs)).

[0084] The performance of certain operations may be distributed among processors and may reside on a single machine or may be spread across multiple machines. In some example embodiments, the processor or the engine implemented by the processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or the engine implemented by the processor may be distributed across multiple geographic locations.

[0085] Throughout this specification, multiple examples may implement components, operations, or structures that are described as a single example. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and the operations are not required to be performed in the order illustrated. In example configurations, structures and functions presented as separate components may be implemented as combined structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter herein.

[0086] As used herein, "or" is inclusive and not exclusive unless expressly indicated otherwise or indicated otherwise by context. Thus, herein, "A, B, or C" means "A, B, C, A and B, A and C, B and C, or A, B, and C," unless expressly indicated otherwise or indicated otherwise by context. Further, "and" means both and each, unless expressly indicated otherwise or indicated otherwise by context. Thus, herein, "A and B" means both A and B, or each, unless expressly indicated otherwise or indicated otherwise by context. Furthermore, multiple examples may be provided for resources, operations, or structures described herein as a single example. Additionally, boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific example configurations. Other allocations of functionality are contemplated and may be within the scope of various embodiments of the present disclosure. In general, structures and functions presented as separate resources in example configurations may be implemented as a combined structure or resource. Similarly, structures and functions presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements are within the scope of the disclosed embodiments as expressed by the appended claims. Accordingly, the specification and drawings should be regarded as illustrative and not restrictive.

[0087] The terms "comprise" or "comprises" are used to indicate the presence of subsequently declared features but do not exclude the addition of other features. Conditional language, particularly conditional language such as "may," "could," "could," or "may," is generally intended to convey that one embodiment includes certain features, elements, and / or steps that other embodiments do not include, unless specifically stated otherwise or understood within the context in which it is used. As such, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without user input or prompting, whether those features, elements, and / or steps are included in or performed in any particular embodiment.

[0088] Although the summary of the subject matter has been described with reference to certain example embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of the disclosed embodiments. Such embodiments of the subject matter may be referred to herein individually or collectively by the term "invention" merely for convenience and without any intention to arbitrarily limit the scope of the present application to any single disclosure or concept when in fact more than one is disclosed.

[0089] The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments can be utilized and derived therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of the disclosure. Therefore, the Detailed Description should not be construed in a limiting sense, and the scope of various embodiments is defined solely by the appended claims, along with the full scope of equivalents to which such claims are entitled. Some aspects of the present invention are described below. [Aspect 1] 1. A computer-implemented method comprising: Obtaining an input tensor and a plurality of filters in a convolutional layer of a neural network; dividing the input tensor into a plurality of subtensors and allocating the plurality of subtensors to a plurality of processors; generating, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter; the hierarchical bit representation comprises a first layer and a second layer; the first layer corresponds to a plurality of sub-filters in the filter, the corresponding sub-filters comprising a plurality of bits indicating whether they have at least one non-zero weight; the second layer comprising a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair comprising one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input values ​​to obtain partial sums; accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of the convolutional layer of the neural network. [Aspect 2] The assigning of the plurality of subtensors to a plurality of processors comprises: 2. The method of aspect 1, comprising, for each of the plurality of processors, storing one or more of the plurality of subtensors in an internal buffer of the processor. [Aspect 3] 2. The method of claim 1, wherein each of the plurality of subtensors and each of the plurality of subfilters have the same number of channels. [Aspect 4] Identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on the first tier of the hierarchical bit representation; determining offsets of the non-zero weights within the filter channel group based on the second layer of the hierarchical bit representation; 2. The method of aspect 1, comprising identifying an input value from the assigned subtensor based on the channel group index and the offset. [Aspect 5] the hierarchical bit representation further comprises a third layer between the first layer and the second layer; the third layer includes, for each of the plurality of sub-filters, a plurality of bit vectors corresponding to a plurality of horizontal planes of the sub-filter, 2. The method of claim 1, wherein each of the plurality of bit vectors indicates whether the corresponding horizontal plane has at least one of the plurality of non-zero weights. [Aspect 6] Identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on the first tier of the hierarchical bit representation; determining the non-zero weight height and width indices based on the third layer of a hierarchical bit representation; determining offsets of the non-zero weights within the filter channel group based on the second layer of the hierarchical bit representation; 6. The method of embodiment 5, comprising identifying an input value from the assigned subtensor based on the channel group index, the height index, the width index, and the offset. [Aspect 7] 2. The method of embodiment 1, wherein the plurality of filters are truncated such that each of the plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights. [Aspect 8] The method of aspect 1, wherein the plurality of processors are electrically arranged in a matrix having a plurality of rows and columns, and each of the plurality of processors has one or more columnar neighboring processors. [Aspect 9] 9. The method of claim 8, wherein each of the plurality of processors is enabled to read one or more key-value pairs from an internal buffer of one of its column-wise neighboring processors. [Aspect 10] assigning one or more of the plurality of subtensors from a channel group comprising a plurality of channels to a row of a processor; 9. The method of claim 8, further comprising: after each of the rows of processors completes the multiply-accumulate operation between one subtensor and a corresponding subfilter based on the hierarchical bit representation to obtain a partial sum, rotating the partial sum among the rows of processors. [Aspect 11] assigning one or more of the plurality of sub-filters from the same height index and the same weight index to each of the plurality of processors; 9. The method of aspect 8, further comprising: each of the plurality of processors completing the multiply-and-accumulate operation between the one or more of the plurality of subfilters and a corresponding subfilter based on the hierarchical bit representation to obtain partial sums, and then accumulating the partial sums to construct the output tensor. [Aspect 12] 1. A system for hierarchical weight sparse convolution processing, comprising: one or more processors; and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions, the instructions providing the system with: Obtaining an input tensor and a plurality of filters in a convolutional layer of a neural network; dividing the input tensor into a plurality of subtensors and allocating the plurality of subtensors to a plurality of processors; generating, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter; the hierarchical bit representation comprises a first layer and a second layer; the first layer corresponds to a plurality of sub-filters in the filter, the corresponding sub-filters comprising a plurality of bits indicating whether they have at least one non-zero weight; the second layer comprises a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair comprising one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input values ​​to obtain partial sums; accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of the convolutional layer of the neural network. [Aspect 13] The assigning of the plurality of subtensors to a plurality of processors comprises: 13. The system of aspect 12, comprising, for each of the plurality of processors, storing one or more of the plurality of subtensors in an internal buffer of the processor. [Aspect 14] the hierarchical bit representation further comprises a third layer between the first layer and the second layer; the third layer includes, for each of the plurality of sub-filters, a plurality of bit vectors corresponding to a plurality of horizontal planes of the sub-filter, 13. The system of aspect 12, wherein each of the plurality of bit vectors indicates whether the corresponding horizontal plane has at least one of the plurality of non-zero weights. [Aspect 15] 13. The system of embodiment 12, wherein the plurality of filters are truncated such that each of the plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights. [Aspect 16] Identifying one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index corresponding to a filter channel group in which the non-zero weight is located based on the first tier of the hierarchical bit representation; determining the non-zero weight height and width indices based on the third layer of a hierarchical bit representation; determining offsets of the non-zero weights within the filter channel group based on the second layer of the hierarchical bit representation; 16. The system of embodiment 15, further comprising identifying an input value from the assigned subtensor based on the channel group index, the height index, the width index, and the offset. [Aspect 17] 1. A non-transitory computer-readable storage medium for hierarchical weight sparse convolution processing, configured with instructions that, when executed by one or more processors, cause the one or more processors to: Obtaining an input tensor and a plurality of filters in a convolutional layer of a neural network; dividing the input tensor into a plurality of subtensors and allocating the plurality of subtensors to a plurality of processors; generating, for each of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter; the hierarchical bit representation comprises a first layer and a second layer; the first layer corresponds to a plurality of sub-filters in the filter, the corresponding sub-filters comprising a plurality of bits indicating whether they have at least one non-zero weight; the second layer comprises a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair comprising one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned subtensor based on the hierarchical bit representation; performing a multiply-accumulate (MAC) operation on the one or more identified non-zero weights and the corresponding input values ​​to obtain partial sums; accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of the convolutional layer of the neural network. [Aspect 18] The assigning of the plurality of subtensors to a plurality of processors comprises: 18. The storage medium of aspect 17, comprising, for each of the plurality of processors, storing one or more of the plurality of subtensors in an internal buffer of the processor. [Aspect 19] the hierarchical bit representation further comprises a third layer between the first layer and the second layer; the third layer includes, for each of the plurality of sub-filters, a plurality of bit vectors corresponding to a plurality of horizontal planes of the sub-filter, 18. The storage medium of claim 17, wherein each of the plurality of bit vectors indicates whether the corresponding horizontal plane has at least one of the plurality of non-zero weights. [Aspect 20] 18. The storage medium of claim 17, wherein the plurality of filters are truncated such that each of the plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights.

Claims

1. 1. A computer-implemented method comprising: Obtaining input tensors and filters in a convolutional layer of a neural network; dividing the filter into a plurality of sub-filters; generating a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer, the first layer comprising a plurality of bits corresponding to each of the plurality of sub-filters in the filter, each of the plurality of bits indicating whether the corresponding sub-filter has at least one non-zero weight; performing a multiply-accumulate (MAC) operation based on the hierarchical bit representation of the filter and the input tensor; Equipped with the hierarchical bit representation further comprises a third layer, the third layer comprising a plurality of bit vectors respectively corresponding to a plurality of horizontal planes in each of the plurality of sub-filters, the plurality of bit vectors indicating whether the corresponding horizontal plane comprises at least one of the plurality of non-zero weights.

2. performing the MAC operation identifying one or more of a plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; and performing a MAC operation on the identified one or more non-zero weights and the corresponding input values.

3. performing the MAC operation determining a channel group index based on the first tier of the hierarchical bit representation for each of the plurality of non-zero weights, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining offsets of the non-zero weights within the filter channel group based on a second layer of the hierarchical bit representation; identifying an input value within the input tensor based on the channel group index and the offset; performing a MAC operation based on the identified input values ​​and the non-zero weights.

4. The method of claim 1 , further comprising pruning the filter so that each of the plurality of sub-filters has all zero weights or a predetermined number of non-zero weights.

5. obtaining a plurality of partial sums from the MAC operation; constructing an output tensor of the convolutional layer of the neural network based on the plurality of partial sums; The method of claim 1 further comprising:

6. The method of claim 1 , wherein the MAC operations are performed in parallel across multiple processors.

7. segmenting the input tensor into a plurality of subtensors, wherein each of the plurality of subtensors and each of the plurality of sub-filters has the same number of channels; The method of claim 1 , wherein performing the MAC operation comprises assigning the plurality of subtensors and the plurality of subfilters to a plurality of processors to perform the MAC operation in parallel.

8. 1. A system for hierarchical weight sparse convolution processing, comprising: one or more processors; and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions, the instructions providing the system with: Obtaining input tensors and filters in a convolutional layer of a neural network; dividing the filter into a plurality of sub-filters; generating a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer, the first layer comprising a plurality of bits corresponding to each of the plurality of sub-filters in the filter, each of the plurality of bits indicating whether the corresponding sub-filter has at least one non-zero weight; performing a multiply-accumulate (MAC) operation based on the hierarchical bit representation of the filter and the input tensor; and The system, wherein the hierarchical bit representation further comprises a third layer, the third layer comprising a plurality of bit vectors respectively corresponding to a plurality of horizontal planes in each of the plurality of sub-filters, the plurality of bit vectors indicating whether the corresponding horizontal plane comprises at least one of the plurality of non-zero weights.

9. The MAC operation to be performed is identifying one or more of a plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; performing a MAC operation on the identified one or more non-zero weights and the corresponding input values; The system of claim 8 , comprising:

10. performing the MAC operation determining a channel group index based on the first tier of the hierarchical bit representation for each of the plurality of non-zero weights, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining offsets of the non-zero weights within the filter channel group based on a second layer of the hierarchical bit representation; identifying an input value within the input tensor based on the channel group index and the offset; and performing a MAC operation based on the identified input values ​​and the non-zero weights.

11. 1. A non-transitory computer-readable storage medium for hierarchical weight sparse convolution processing, configured with instructions that, when executed by one or more processors, cause the one or more processors to: Obtaining input tensors and filters in a convolutional layer of a neural network; dividing the filter into a plurality of sub-filters; generating a hierarchical bit representation of a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer, the first layer comprising a plurality of bits corresponding to each of the plurality of sub-filters in the filter, each of the plurality of bits indicating whether the corresponding sub-filter has at least one non-zero weight; performing a multiply-accumulate (MAC) operation based on the hierarchical bit representation of the filter and the input tensor; performing an operation comprising: The storage medium is characterized in that the hierarchical bit representation further comprises a third layer, the third layer comprising a plurality of bit vectors respectively corresponding to a plurality of horizontal planes in each of the plurality of sub-filters, the plurality of bit vectors indicating whether the corresponding horizontal plane comprises at least one of the plurality of non-zero weights.

12. The MAC operation to be performed is identifying one or more of a plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; performing a MAC operation on the identified one or more non-zero weights and the corresponding input values; The storage medium of claim 11 .

13. performing the MAC operation determining a channel group index based on the first tier of the hierarchical bit representation for each of the plurality of non-zero weights, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining offsets of the non-zero weights within the filter channel group based on a second layer of the hierarchical bit representation; identifying an input value in the input tensor based on the channel group index and the offset; performing a MAC operation based on the identified input values ​​and the non-zero weights; The storage medium of claim 11 .

Images