Method and system for hierarchical weight sparse convolution processing

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are described for hierarchical weighted sparse convolution processing. An exemplary method includes obtaining an input tensor and a plurality of filters at a convolutional layer of a neural network; segmenting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; generating, for each filter of the plurality of filters, a hierarchical bit representation of a plurality of non-zero weights in the filter, where the hierarchical bit representation includes a plurality of bits indicating whether a sub-filter has at least one non-zero weight and a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter; one or more of a plurality of non-zero weights and corresponding input values from the assigned sub-tensors are identified based on the hierarchical bit representations to perform multiplication and accumulation (MAC) operations.

Description

[0001] This application is a divisional application of the invention patent application with application date of August 23, 2021, application number 202180028638.4, and invention name “Method and system for hierarchical weighted sparse convolution processing”. Technical Field

[0002] The present disclosure relates generally to artificial intelligence and more particularly to hierarchical weighted sparse convolution processing. Background Art

[0003] Currently, neural networks (NNs) are the basis of many modern artificial intelligence (AI) applications, such as image and video recognition, recommendation systems, classification, medical image analysis, and natural language processing. NNs are used in a variety of use cases ranging from self-driving cars and detecting cancer to playing complex games. A typical NN may include a series of convolutional layers in which intensive and expensive (in terms of computation and energy) convolution operations are performed.

[0004] In recent years, various methods have been developed to improve the computational efficiency of NNs by introducing sparsity into the convolution process in NNs (e.g., pruning filters to reduce the number of non-zero weights). However, existing hardware architectures rely on general-purpose computing engines (e.g., GPUs) that are not optimized for convolutions involving filters with sparse weights (e.g., a limited number of non-zero weights). These architectures are often unable to efficiently store sparse filters in memory or fully exploit sparsity to achieve higher levels of parallelism on a large scale. Therefore, it is desirable to build novel system architectures for efficient sparse convolutions. Summary of the invention

[0005] Various embodiments of the present specification may include systems, methods, and non-transitory computer-readable media for layered-weight sparse convolution processing and parallelization.

[0006] According to one aspect, a method for hierarchical weight sparse convolution includes: obtaining an input tensor and a plurality of filters at a convolutional layer of a neural network; partitioning the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; for each of the plurality of filters, generating a hierarchical bit representation of the plurality of non-zero weights in the filter, wherein the hierarchical bit representation includes a first layer and a second layer, the first layer including a plurality of bits corresponding to the plurality of sub-filters in the filter and indicating whether the corresponding sub-filter has at least one non-zero weight, and the second layer including a plurality of bits corresponding to the plurality of sub-filters in the filter. A plurality of key-value pairs of non-zero weights, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors based on a hierarchical bit representation, one or more of the plurality of non-zero weights and a corresponding input value from an assigned sub-tensor; performing a multiplication and accumulation (MAC) operation on the identified one or more non-zero weights and the corresponding input value to obtain a partial sum; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of a convolutional layer of a neural network.

[0007] In some embodiments, assigning the plurality of sub-tensors to the plurality of processors includes, for each processor of the plurality of processors, storing one or more of the plurality of sub-tensors in an internal buffer of the processor.

[0008] In some embodiments, each sub-tensor in the plurality of sub-tensors and each sub-filter in the plurality of sub-filters has the same number of channels.

[0009] In some embodiments, identifying one or more of a plurality of non-zero weights and a corresponding input value from an assigned sub-tensor based on a hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index based on a first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining an offset of the non-zero weight within the filter channel group based on a second layer of the hierarchical bit representation; and identifying an input value from the assigned sub-tensor based on the channel group index and the offset.

[0010] In some embodiments, the layered bit representation further includes a third layer between the first layer and the second layer, wherein, for each subfilter in the plurality of subfilters, the third layer includes a plurality of bit vectors corresponding to the plurality of horizontal planes of the subfilter, respectively, and each of the plurality of bit vectors indicates whether the corresponding horizontal plane includes at least one of the plurality of non-zero weights.

[0011] In some embodiments, identifying one or more of a plurality of non-zero weights and corresponding input values ​​from an assigned sub-tensor based on a hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index based on a first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining a height index and a width index of the non-zero weight based on a third layer of the hierarchical bit representation; determining an offset of the non-zero weight within the filter channel group based on a second layer of the hierarchical bit representation; and identifying an input value from the designated sub-tensor based on the channel group index, the height index, the width index, and the offset.

[0012] In some embodiments, the plurality of filters are pruned such that each sub-filter in the plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights.

[0013] In some embodiments, the plurality of processors are electrically arranged in a matrix comprising a plurality of columns and rows, and each processor of the plurality of processors has one or more adjacent processors in the column direction.

[0014] In some embodiments, each processor of the plurality of processors is allowed to read one or more key-value pairs from an internal buffer of one of its column-wise neighboring processors.

[0015] In some embodiments, the method may further include: assigning one or more of the plurality of sub-tensors from a channel group to a row of processors, wherein the channel group includes a plurality of channels; and after each processor in the row of processors completes a MAC operation between a sub-tensor and a corresponding sub-filter based on the hierarchical bit representation and obtains a partial sum, rotating the partial sum among the row of processors.

[0016] In some embodiments, the method may further include: assigning one or more of the plurality of sub-filters from the same height index and the same width index to each of the plurality of processors; and after each of the plurality of processors completes a MAC operation between one or more of the plurality of sub-filters and the corresponding sub-filter based on the hierarchical bit representation and obtains partial sums, accumulating the partial sums to construct an output tensor.

[0017] According to another aspect, a system for hierarchical weight sparse convolution is provided. The system may include one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions, which instructions can be executed by the one or more processors to cause the system to perform operations including the following: obtaining an input tensor and a plurality of filters at a convolutional layer of a neural network; dividing the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; for each filter in the plurality of filters, generating a hierarchical bit representation of the plurality of non-zero weights in the filter, wherein the hierarchical bit representation includes a first layer and a second layer, the first layer includes a plurality of sub-filters corresponding to the filter and indicating whether the corresponding sub-filter has The invention relates to a method for constructing a convolutional layer of a neural network comprising: a plurality of bits of at least one non-zero weight, and a second layer comprising a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair comprising a channel identifier of one of the plurality of non-zero weights as a value and one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors based on the hierarchical bit representation, one or more of the plurality of non-zero weights and a corresponding input value from the assigned sub-tensor; performing a multiplication and accumulation (MAC) operation on the identified one or more non-zero weights and the corresponding input value to obtain a partial sum; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of a convolutional layer of the neural network.

[0018] According to another aspect, a non-transitory computer-readable storage medium for hierarchical weight sparse convolution is provided. The medium may be configured with instructions that can be executed by one or more processors to cause the one or more processors to perform operations including the following: obtaining an input tensor and a plurality of filters at a convolutional layer of a neural network; dividing the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors; for each filter in the plurality of filters, generating a hierarchical bit representation of the plurality of non-zero weights in the filter, wherein the hierarchical bit representation includes a first layer and a second layer, the first layer includes a plurality of bits corresponding to the plurality of sub-filters in the filter and indicating whether the corresponding sub-filter has at least one non-zero weight, and the second layer includes The invention relates to a method for constructing a convolutional layer of a neural network, comprising: comprising a plurality of key-value pairs corresponding to a plurality of non-zero weights in the filter, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key; identifying, by each of the plurality of processors based on a hierarchical bit representation, one or more of the plurality of non-zero weights and a corresponding input value from an assigned sub-tensor; performing a multiplication and accumulation (MAC) operation on the identified one or more non-zero weights and the corresponding input value to obtain a partial sum; and accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of a convolutional layer of the neural network.

[0019] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the methods of operation and functions and economies of manufacture of the related structural elements and combinations of parts, will become more apparent after considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification and in which like reference numerals designate corresponding parts in the various figures. It should be expressly understood, however, that the drawings are for illustration and description purposes only and are not intended to define the limitations of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An exemplary convolutional neural network (CNN) is depicted in accordance with various embodiments.

[0021] Figure 2 An exemplary convolution process is depicted in accordance with various embodiments.

[0022] Figure 3 Depicted are exemplary methods for partitioning input tensors and filters according to various embodiments.

[0023] Figure 4 An exemplary hierarchical memory layout for storing non-zero weights is shown in accordance with various embodiments.

[0024] Figure 5 An illustrative system diagram for hierarchical weight sparse convolutions according to various embodiments is shown.

[0025] Fig. 6A An exemplary system architecture for hierarchical weight sparse convolution is depicted in accordance with various embodiments.

[0026] Figure 6B Another exemplary system architecture for hierarchical weight sparse convolutions in accordance with various embodiments is shown.

[0027] Figure 7 Depicted is an example method for hierarchical weight sparse convolution in accordance with various embodiments.

[0028] Figure 8 An example computer system is depicted in which any of the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0029] Embodiments described herein provide methods, systems, and devices for hierarchical weight sparse convolution in neural networks. The convolution process may refer to a basic but computationally expensive operation for extracting features of input data from a previous layer (such as a feature map derived from an image or audio wave or an activation output from a previous layer in a neural network). The input data of the convolution process may be referred to as an input tensor, and the output of the convolution process may be referred to as an output tensor. The input tensor may include feature maps of multiple channels. For example, an RGB image may be represented as an input tensor with three channels (a red channel, a green channel, and a blue channel), each of which includes a feature map (e.g., a grayscale image of the same size as the original RGB image but made of only one of red, green, or blue). In order to extract features from the input tensor, one or more feature extractors (also referred to as filters) may be applied to the input tensor by convolution (e.g., sliding or moving). A multiplication and accumulation (MAC) operation between a feature detector (filter) and an input tensor may be performed during each convolution step. The outputs generated from the convolution steps may then be assembled to form an output tensor of the convolution process. The output tensor may become the input tensor of the next layer in the neural network. In some cases, multiple input tensors may be involved in a convolutional layer.

[0030] During the convolution process, filters may be pruned or sparse by introducing a large number of zeros to reduce computational cost and increase inference speed. For example, more than 50% of the values ​​in a filter may be set to zero. Some embodiments of the present specification exploit filter sparsity by efficiently storing the sparse filters in a memory efficient layout and parallelizing convolution processing on multiple processing entities (PEs) (e.g., processors).

[0031] In some embodiments, the input tensor may first be split into a plurality of sub-tensors according to the number of PEs available to participate in the convolution process. Each sub-tensor may be assigned to one of the plurality of PEs and maintained within the PE without being substituted out of the local memory / buffer of the PE for multiple iterations of the convolution process (corresponding to multiple convolution layers in the NN). After each iteration of the convolution process, the sub-tensor assigned to the PE as input may be updated and evolved into an output that may be used as input for the next iteration of the convolution process. In some embodiments, the updated sub-tensor may be temporarily substituted out of the PE for memory management purposes and substituted back into the PE for the next iteration.

[0032] In some embodiments, each sparse filter may be partitioned into a plurality of sub-filters that may be stored in a memory efficient layout. For example, after being pruned and partitioned, each sparse filter may be partitioned in such a way that each sub-filter includes either all zero weights or a predetermined number of non-zero weights, denoted as W. The number of non-zero weights within each sub-filter may be affected by the size of the sub-filter (e.g., larger sized sub-filters may include more non-zero weights) and the desired feature extraction accuracy (e.g., too few non-zero weights may cause feature loss). In some embodiments, based on the knowledge of the number of non-zero weights (0 or W) within each sub-filter, each sparse filter may be represented in a hierarchical bit representation (e.g., a form of memory layout) having multiple layers. For example, a top layer may include a plurality of bits for indicating whether a plurality of blocks within the filter contain non-zero weights, a second layer may similarly include a plurality of bits for indicating whether a plurality of segments within each block contain non-zero weights, and a bottom layer may store the actual non-zero weights. The number of layers in each hierarchical bit representation may be determined based on, for example, how the filter is partitioned.

[0033] Because sparse filters are represented as hierarchical bit representations, the corresponding convolution process can be called hierarchical weight sparse convolution. The hierarchical bit representation of the filter allows each PE to store and iterate non-zero weights more efficiently. For example, the hierarchical bit representation can significantly reduce the memory footprint of the filter (e.g., each PE can read less data from the global buffer into its local buffer to perform local convolution) and make the solution suitable for devices with limited memory resources. For example, the embodiments disclosed herein can be applied to edge devices in edge computing systems. For a detailed description of the hierarchical bit representation, please refer to Figure 4 In some embodiments, sub-filters and sub-tensors may be assigned to multiple PEs in a manner that ensures that sub-tensors and sub-filters assigned to different PEs are independent of each other so that multiple PEs can perform local operations in parallel to improve performance. For a detailed description of the assignment, refer to Fig. 6A and Figure 6B .

[0034] The above partitioning and parallel processing scheme is also highly scalable. In some embodiments, each round (also referred to as iteration) of parallel processing may generate a plurality of partial sums, and a plurality of PEs may perform multiple rounds of parallel processing to accumulate the partial sums generated in each round and assemble the accumulated results into an output tensor. In some embodiments, the partial sums generated by each PE in multiple rounds may be directly accumulated (within each PE) to generate a portion of the output tensor without exchanging data with other PEs.

[0035] In the following description, specific non-limiting embodiments of the present invention will be described with reference to the drawings. The specific features and aspects of any embodiment disclosed herein may be used and / or combined with the specific features and aspects of any other embodiment disclosed herein. It should also be understood that these embodiments are given by way of example and only illustrate a few embodiments within the scope of the present invention. Various changes and modifications that are obvious to those skilled in the art to which the present invention relates are considered to fall within the spirit, scope, and consideration of the present invention as further defined by the appended claims.

[0036] Figure 1 An exemplary convolutional neural network (CNN) is depicted in accordance with various embodiments. CNNs are widely used in various applications including image understanding, speech recognition, game playing, robotics, and may employ a deep hierarchy of layers including convolutional layers (in which convolution operations are performed). It should be noted that CNNs are used for illustration purposes only and that the embodiments disclosed herein may be applied to other neural networks involving convolution operations.

[0037] Figure 1 The exemplary neural network 100 shown in FIG. 1 includes a plurality of layers, such as a first convolutional layer 120 and a first ReLU (rectified linear unit) layer, a first pooling layer 130, a second convolutional layer 140 and a second ReLU layer, a second pooling layer 150, a flattening layer 160, and a fully connected (FC) layer 170. This exemplary neural network 100 can be trained to match the alphabet in a given image with a plurality of known alphabet classes. Figure 1 As shown in , an image 110 with letters is fed into a neural network 100 and transformed through a plurality of layers. The last layer (FC layer 170) ultimately produces a plurality of scores representing the similarity between the letters in the input image 110 and each known alphabet category.

[0038] In some embodiments, the input image 110 may first be transformed into an input tensor. As an example, if the input image 110 contains 32*32 pixels and each pixel has three color channels (red, green, and blue), its corresponding input tensor may have a size of 32*32*3, where the height is 32, the width is 32, and the depth (e.g., the number of channels) is 3. For ease of description, the three-dimensional size may be referred to as an HWC format, where H refers to the height of the input tensor (e.g., 32 in the previous image example), W refers to the width of the input tensor (e.g., 32 in the previous image example), and C refers to the number of channels in the input tensor (e.g., 3 in the previous image example). In some cases, if there are multiple input tensors for a convolutional layer (e.g., when there are multiple images input to the convolutional layer or there are multiple input activation tensors received from a previous layer), each input tensor may be represented in NHWC format, where N refers to the index of the input tensor within the batch of input tensors. In the following description, unless otherwise explicitly stated, N may be omitted for simplicity (eg, assuming there is only one input tensor). It is obvious to those skilled in the art to extend the embodiments to cover the case where N>1.

[0039] In some embodiments, a CNN may include multiple convolutional layers (e.g., Figure 1 The first convolution layer 120 and the second convolution layer 140 in the convolution layer 120). The convolution layer simplifies the input tensor (for example, the original input image or the input tensor from the previous layer) into a format that is easier to process without losing features that are critical to obtaining good predictions / classification. One or more feature detectors may be involved in the convolution process in the convolution layer, for example, edge detectors and curve detectors in image processing. These feature detectors may be called filters. Each filter may have the same number of channels as the input tensor. For ease of description, this specification uses the term "shared multiple channels" to express that each filter and the input tensor have the same number of channels. For example, the input tensor is a 32*32*3 matrix, and the example filter may be a 3*3*3 matrix. For a detailed explanation of the convolution process, please refer to Figure 2 Description.

[0040] The convolutional layer in a CNN may be followed by a nonlinear activation function to introduce nonlinearity into the CNN. Exemplary nonlinear activation functions include the sigmoid function, the hyperbolic tangent function, and the rectified linear unit (ReLU). Figure 1As shown in FIG. 1 , a ReLU function (also referred to as a ReLU layer) follows each of the convolutional layers 120 and 140. The ReLU function may apply an element-wise activation function to filter out some of the outputs (activations) produced by the convolutional layer 120 or 140. For example, a max(0, x) function may be employed to filter out all negative activations in the convolution output and feed only positive activations to the next layer. The ReLU function may not change the size of the output activations but may limit the number of active neurons because all negative activations are cleared to zero to improve computational efficiency of subsequent layers.

[0041] CNNs may also contain one or more pooling layers to provide various computations that reduce the dimensionality of the input tensor. Figure 1 In the embodiment of the present invention, the first pooling layer 130 and the second pooling layer 150 of the neural network 100 may respectively perform downsampling operations on the input tensor along the spatial dimensions (height and width), but typically do not change the depth dimension (eg, the number of channels).

[0042] A fully connected (FC) layer in a neural network learns to combine information from previous layers (e.g. Figure 1 The output of the pooling layer 150 or convolutional layer in the network represents a nonlinear combination of high-level features and calculates the category scores accordingly. Figure 1 , the output from the pooling layer 150 may first be flattened into a vector (e.g., in the flattening layer 160), from which the FC layer 170 may learn an intrinsic non-learning function in the space to calculate the final scores for the plurality of categories. Each score may represent a quantized similarity between the alphabet in the input image and the corresponding category.

[0043] Figure 2 An exemplary convolution process according to various embodiments is shown. A typical convolution process in a neural network may begin by receiving (or obtaining) one or more input tensors and a plurality of filters and performing a convolution process to produce one or more output tensors. For simplicity, Figure 2 The exemplary convolution process 220 in FIG. 2 involves an input tensor 222, a filter 224, and an output tensor 225. The input tensor 222 can be represented by its height (H), width (W), and number of channels (C). Figure 2 As shown in , the input tensor 222 may be a 3*3*3 matrix, where the height is 3, the width is 3, and the number of channels is 3. The height and width of the input tensor 222 in each channel (e.g., 3(H)*3(W)) may be referred to as a 2D input feature map. The tuple {h,w,c} may point to an element of a 2D input feature map in channel c. For example, {2,1,1} may point to an element of a 2D input feature map in channel 1, where the position is defined by {2,1} (e.g., the height direction index is 2, and the width direction index is 1). The filter 224 may be represented by its height (R), width (S), and the number of channels (C). In Figure 2 In FIG. 1 , the filter 224 may be a 1*1*3 matrix. The height and width (eg, 1(R)*1(S)) of the filter 224 in each channel may be referred to as a kernel (the filter 224 has three kernels in three channels, respectively).

[0044] In some cases, the convolution process may involve stride and padding. For example, when the stride is 1, the filter convolves (e.g., moves, slides) one pixel at a time; when the stride is 2, the filter convolves two pixels at a time as it slides around it. A larger stride produces a spatially smaller output tensor (smaller H*W in the output tensor). As another example, when the spatial size (height*width) of the filter is greater than 1*1 (e.g., 3*3 or 5*5), the input tensor may be padded with zeros around its boundaries in order to apply the filter and control the spatial size of the output tensor, thereby (for example) maintaining the spatial size of the input tensor so that the input and output height and width are the same. In Figure 2 , it is assumed that no padding is performed on the input tensor 222 and the stride is 1.

[0045] exist Figure 2 During the convolution process 220 shown in FIG. 2 , a MAC operation is performed on each depth-wise slice (such as the first depth-wise slice 223) of the filter 224 and the input tensor to generate a dot product (such as the dot product 228). For example, the first depth-wise slice 223 of the input tensor 222 is a 1*1*3 tensor (three gray cubes) at the top left of the input tensor 222. Both the first depth-wise slice 223 and the filter 224 have a size of 1*1*3. After the MAC operation, the generated dot product 228 can be assembled as part of the output tensor 225. Thus, the filter 224 can be convolved (e.g., moved) through all depth-wise slices ( Figure 2 224) and then determine the output tensor 225. The number of channels in the output tensor 225 is equal to the number of filters that have been applied during the convolution. Because the convolution process 220 only uses one filter 224, the corresponding output tensor 228 has only one channel.

[0046] In contrast, convolution process 260 involves two filters 264 and 265. By convolving filter 264 through input tensor 262, the value of output tensor 268 in the first channel (e.g., the 2D output feature map in channel 1) may be determined. By convolving filter 265 through input tensor 262, the value of output tensor 268 in the second channel (e.g., the 2D output feature map in channel 2) may be determined. Thus, the resulting output tensor 268 includes 2D output feature maps for two channels.

[0047] It should be noted that some operations during convolution processing can be parallelized. For example, the MAC operation performed on one filter is independent of the MAC operation of another filter because different filters produce 2D output feature maps of different output channels. As another example, the depth-wise slices in the input tensor are independent of each other because they produce values ​​at different positions on each 2D output feature map. If all such operations can be parallelized between multiple processing entities (e.g., processors, cores, or threads), theoretical ideal parallelism can be achieved. However, because real-world applications may have a large number of filters and a huge number of input tensors, pursuing theoretical ideal parallelism requires hardware with unattainable computing power (e.g., having a huge number of processing entities and memory exchange mechanisms). The embodiments disclosed in the following description provide a method (and corresponding system, storage medium) for achieving ideal parallelism by fully utilizing the parallel processing capabilities of a given hardware, which is a more scalable and practical method.

[0048] Figure 3 An exemplary method for partitioning input tensors and filters according to various embodiments is illustrated. During each convolution process, each input tensor 322 and each filter 342 and 343 share a plurality of channels (i.e., have the same number of channels). Without loss of generality, each input tensor can be represented as a three-dimensional tensor having a height dimension (H), a width dimension (W), and a channel dimension (C); each filter can be similarly represented as a three-dimensional tensor having a height dimension (R), a width dimension (S), and a channel dimension (C). Each dimension can include a plurality of corresponding dimensional units. For example, Figure 3 The input tensor 323 in can be represented as 3(H)*3(W)*6(C), where the H dimension includes 3 height units, the W dimension includes 3 width units, and the C dimension includes 6 channels. In some embodiments, multiple input tensors can be involved in the convolution process, and an additional dimension N (e.g., representing an input tensor identifier) ​​can be used to expand the H*W*C representation of each input tensor in the multiple input tensors.

[0049] In some embodiments, the input tensor 322 can be partitioned into a plurality of sub-tensors 323. For example, the partitioning can ensure that the plurality of sub-tensors can be evenly distributed among the available PEs, so that the workload on the PEs is evenly distributed. In some embodiments, the size of each sub-tensor can be expressed as Hg*Wg*Cg, where Hg refers to the height dimension of the sub-tensor (for example, "g" represents a group, and "Hg" represents a group of height units within a sub-tensor), Wg refers to the width dimension of the sub-tensor, and Cg refers to the channel dimension of the sub-tensor or a channel group 324 containing one or more consecutive channels. Figure 3For example, the sub-tensor 323 (shaded area) may have a size of 1(Hg)*1(Wg)*2(Cg). In practical implementations, Hg and Wg may be equal and greater than 1, and Cg may be a large positive number such as 8, 16, or 32. In some embodiments, a sub-tensor (such as 323) may be fetched a portion at a time by one PE, rather than fetching the entire sub-tensor 323 at once (e.g., due to the limited size of the internal buffer of the PE).

[0050] In some embodiments, the filter 342 may be partitioned into a plurality of sub-filters, and each sub-tensor in the plurality of sub-tensors and each sub-filter in the plurality of sub-filters may have the same number of channels, for example, they may have a channel group Cg 324 in their channel direction dimension. The partitioning of the filter 342 may be implemented in various ways and have various granularities. For example, the filter 342 may be partitioned into filter blocks 345, wherein each filter block 345 has a size of R*S*Cg and is treated as a sub-filter. As another example, each filter block 345 may be further partitioned into a plurality of horizontal planes, wherein each horizontal plane has a size of 1*S*Cg and is treated as a sub-filter 346. Assuming that the filter 342 has a size of 3(R)*3(S)*C, the sub-filter 346 may have a size of 1(R)*3(S)*Cg, wherein the number of channels C is divided into a plurality of channel groups, wherein each channel group has Cg channels.

[0051] In some embodiments, filters 342 to 344 may be pruned or thinned to limit the number of non-zero weights therein. Thinned filters can greatly improve computational efficiency without significantly losing the accuracy of extracting features from the input tensor. In some embodiments, filters 342 to 344 may be pruned so that each subfilter (e.g., 345 or 346) has all zero weights or a predetermined number of non-zero weights. The predetermined number of non-zero weights in each subfilter is used to capture the desired features from the input tensor 322, and the all-zero subfilter helps to reduce the memory / storage device footprint and computational cost of the filter. In some embodiments, the pruning process may be performed in multiple steps. For example, the first step is to prune each filter block 345 so that they contain the same number of non-zero weights (the position of the non-zero weights may be different in each filter block 345); then, in each filter block 345, the subfilters 346 therein may be pruned so that they contain the same number of non-zero weights; finally, the non-zero weights in some subfilters 346 may be removed to produce all-zero subfilters.

[0052] In some embodiments, multiple filters may be present in a convolutional layer to capture features in the input tensor 322 from different perspectives. For example, Figure 3Filters 342 to 344 are shown to be available for convolution with the input tensor. In some embodiments, a plurality of filters 342 to 344 (denoted as K filters) may be grouped into filter groups (denoted as Kg), where each filter group includes a fixed number of filters. Within each filter group, subfilters from the same channel group may form a subfilter group that may be fed into a PE to perform local convolution with the corresponding subtensor 323. For example, assuming that filters 342 and 343 are in the same filter group, a subfilter 346 from the first channel group of filter 342 and a subfilter from the first channel group of filter 343 may form a subfilter group that may be fed into a PE to perform convolution with the subtensor 323 from the (same) first channel group.

[0053] Figure 4 An exemplary hierarchical memory layout for storing non-zero weights is shown in accordance with various embodiments. Figure 4 The exemplary filter 420 in is assumed to be a 3(R)*3(S)*32(C) tensor for illustration and may be referred to as Figure 3 It is also assumed that the filter 420 is divided into a plurality of channel groups, wherein each channel group includes a plurality of channels (Cg); and within each channel group, the filter block R*S*Cg is further divided into a plurality of sub-filters (such as a plurality of horizontal planes), wherein each sub-filter is represented as 1(R)*3(S)*Cg. Figure 4 In the figure, the value of Cg is assumed to be 8. These assumptions are only for simplicity and clarity and can be easily modified or relaxed to adapt to different filter partitioning methods or actual needs of convolution.

[0054] like Figure 4As shown in , filter 420 is divided into a plurality of filter blocks, such as a first filter block (shaded cube) 3(R)*3(S)*8(Cg). The first filter block can be further divided into three horizontal planes 432, 433, and 434 corresponding to R=0, R=1, and R=2, respectively. Each of the horizontal planes 432, 433, and 434 can be represented as a two-dimensional table with a width of 3 (corresponding to S) and a height of 8 (corresponding to Cg). It should be noted that each sub-filter (horizontal plane) has been pruned to include all zeros (such as 434) or a fixed number of non-zero weights (four non-zero weights in each of 432 and 433). For example, table 432 corresponding to plane R=0 may have four non-zero weights W0 to W3, and all other values ​​are zero. Among the non-zero weights, W0 and W1 are located in column S=0, and W2 and W3 are located in column S=1. Similarly, table 433 corresponding to plane R = 1 also has four non-zero weights, with W4 and W5 located in column S = 0, and W6 and W7 located in column S = 2. Table 434 corresponding to plane R = 2 has no non-zero weights.

[0055] exist Figure 4 In the embodiment of the present invention, filter 420 may be stored in the memory of PE as a hierarchical bit representation to represent the non-zero weights in filter 420. Depending on the structure of filter 420, the hierarchical bit representation of filter 420 may be generated in various forms. In some embodiments, if the filter has a size of 1(R)*1(S)*C and is divided into a plurality of filter blocks (each filter block is 1(R)*1(S)*Cg), the hierarchical bit representation of the filter may include a first layer and a second layer. The first layer may include a plurality of bits corresponding to the plurality of filter blocks in the filter and indicating whether the corresponding filter block has at least one non-zero weight, and the second layer may include a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of the one non-zero weight as a key.

[0056] In some embodiments, if the width and height dimensions of filter 420 are greater than 1, the number of layers in the hierarchical bit representation of filter 420 may be increased to 3. For example, a first layer may include a plurality of bits corresponding to a plurality of filter blocks in the filter and indicating whether the corresponding filter block has at least one non-zero weight; a second layer may include, for each of the plurality of filter blocks, a plurality of bit vectors corresponding to a plurality of horizontal planes within the filter block, respectively, wherein each bit vector indicates whether the corresponding horizontal plane has at least one non-zero weight; and a third layer may include a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of the one non-zero weight as a key.

[0057] For example, assuming filter 420 has a first filter block (the shaded cube) containing some non-zero weights and a second filter block containing all zero weights, the hierarchical bit representation 440 of filter 420 may be as follows: Figure 4 434. The first layer of hierarchical bit representation 440 includes a plurality of bits indicating whether the corresponding filter block contains at least one non-zero weight. In this example, the first bit corresponding to the first filter block is 1, and the second bit corresponding to the second filter block is 0 (indicating an all-zero filter block). The second layer of hierarchical bit representation 440 includes a plurality of bit vectors corresponding to the horizontal planes of the filter blocks having at least one non-zero weight. In this example, the first filter block has three horizontal planes represented as tables 432, 433, and 434, where table 432 contains non-zero weights in columns 0 and 1, table 433 contains non-zero weights in columns 0 and 2, and table 434 contains all zeros. Thus, the first bit vector in the second layer of the layered bit representation 440 has values ​​(1, 1, 0) indicating that the first horizontal plane (R=0) has non-zero weights in the first column (S=0) and the second column (S=1); the second bit vector in the second layer of the layered bit representation 440 has values ​​(1, 0, 1) indicating that the second horizontal plane (R=1) has non-zero weights in the first column (S=0) and the third column (S=2); and the third bit vector in the second layer of the layered bit representation 440 has values ​​(0, 0, 0) indicating that there are no non-zero values ​​in the corresponding horizontal plane (R=2). The third layer of the layered bit representation 440 may contain actual non-zero values. Figure 4 As shown in , the non-zero weight W0 is located in channel 2 (the idx value in table 432 refers to the index of the channel and starts from 0), so the key-value pair corresponding to W0 in the third layer is (2, W0). Similarly, the non-zero weight W1 is represented as (7, W1), indicating that the non-zero weight W1 is located in channel 7.

[0058] The hierarchical bit representation 440 stores the necessary information of the non-zero weights in the filter 420 in an efficient manner. In some embodiments, the hierarchical bit representation 440 can provide the accurate location of each non-zero weight by: determining a channel group index based on a first layer of the hierarchical bit representation 440, wherein the channel group index corresponds to the filter channel group in which the non-zero weight is located; determining a height index and a width index of the non-zero weight based on a second layer of the hierarchical bit representation; and determining an offset (i.e., a channel index) of the non-zero weight within the filter channel group based on a third layer of the hierarchical bit representation 440.

[0059] During a convolution process, each of a plurality of processing entities (PEs) may be assigned one or more filters (e.g., filter groups) or one or more filter blocks (e.g., filter blocks from the same channel group) and one or more corresponding sub-tensors (e.g., portions of an input tensor from the same channel group as the filter / filter block) to perform a local multiplication and accumulation (MAC) operation to produce a partial sum.

[0060] To perform a MAC operation on each PE, an input value (input value in a sub-tensor) corresponding to an assigned non-zero weight (non-zero weight in a filter or filter block) may be identified based on a hierarchical bit representation of one or more filters or filter blocks. The following description uses an exemplary hierarchical bit representation of filter 420 as an example to show how to identify the corresponding input value. Figure 4 Assume that filter 420 is assigned to a PE. Each non-zero weight in filter 420 can be enumerated and multiplied with the corresponding input value in the input tensor. For example, the PE can start from the first layer by identifying the first non-zero value bit indicating a filter block with a non-zero weight. The position of the non-zero value bit can indicate which channel group it corresponds to. Figure 4 , the first bit of the first layer is 1, which means that the filter block of the first channel group has at least one non-zero weight. Then, the PE can check the second layer by locating a fixed number of bit vectors corresponding to the filter block, where the fixed number refers to the number of horizontal planes in each filter block (e.g., Figure 4 432, 433, and 434 in the first filter block in ). Each of these bit vectors may include a fixed number of bits corresponding to the number of columns in the S dimension of the filter 420 (e.g., 3 columns S=0, 1, 2 in each horizontal plane 432, 433, or 434). Based on the position of the bit vector in the second layer, the index in the R dimension can be determined; based on the bit value in the bit vector, the index in the S dimension can be determined. Finally, the third layer includes the channel identifier and the actual non-zero weight. By knowing the three indices in the R dimension, the S dimension, and the channel dimension, the PE can locate the corresponding input value from the input tensor and perform a MAC operation based on the non-zero weight and the input value to obtain a partial sum. The partial sums obtained by multiple PEs can be accumulated to construct the output tensor of the convolution layer.

[0061] In some embodiments, each layer in the hierarchical bit representation 440 may maintain a pointer to a previously processed bit, bit vector, or key-value pair. During the enumeration of non-zero weights, the pointer may help the PE determine the next filter block, bit vector, or key-value pair to be processed. In some embodiments, the hierarchical bit representation 440 may be stored as a tree with a root node corresponding to the first layer. Each non-zero bit in the root node may have a subtree containing one or more child nodes, which store bit vectors belonging to the filter block corresponding to the non-zero bit, wherein each bit vector corresponds to a horizontal plane in the filter block. Each non-zero bit in each bit vector may have one or more grandchild nodes, which correspond to one or more columns in each horizontal plane of the filter block and store key-value pairs of non-zero weights in one or more columns.

[0062] The hierarchical bit representation 440 provides an efficient way to store all the information needed to store non-zero weights and locate corresponding input values ​​to perform convolution. For example, if a bit in the first layer is 0, the entire corresponding filter block can be skipped. In this case, the entire filter block only occupies 1 bit of storage space in the PE.

[0063] Figure 5 An illustrative system diagram for hierarchical weight sparse convolution processing is depicted in accordance with various embodiments. Figure 5 The architecture shown in 500 includes a computer system designed to perform inference when a neural network is loaded and running. In some embodiments, this computer system is implemented as a hardware accelerator 500. Some components in the hardware accelerator 500 can cooperate in a particular way to improve the throughput and energy efficiency of the neural network by facilitating parallel convolution calculations on multiple processing entities. Accelerator 500 is merely illustrative and may include more, fewer, or alternative components. Hardware accelerator 500 can be designed as a reconfigurable device, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0064] like Figure 5, the hardware accelerator 500 may include a scheduler 570 for controlling the workflow within the accelerator 500 and the interaction with off-chip components such as a host CPU 510 and a double data rate (DDR) memory 520. For example, the accelerator 500 may interact with the host CPU 510 through a peripheral component interconnect express (PCIe) physical layer (PHY) controller 512 and interact with the off-chip DDR memory 520 through a DDR interface 530. The accelerator 500 may extract data from the off-chip DDR memory 520 through a direct memory access (DMA) controller 540 (which communicates with the off-chip DDR memory 520 via the DDR interface 530). The extracted data may be stored in an on-chip buffer (referred to as a global buffer 550) in preparation for parallel convolution computations. The global buffer 550 may be logically divided into multiple sections such as an input buffer 552, a weight buffer 554, and an output buffer 556. The input buffer 552 may store data associated with the input tensor, the weight buffer 554 may store data associated with the filter, and the output buffer 556 may store data associated with the result of the convolution calculation (e.g., the output tensor). The global buffer 550 may exchange data with an array 560 of sparse processing units (SPUs) 562 to perform convolution calculations. The "sparse" in the SPU name indicates that the SPU array 560 is specifically designed to efficiently perform convolution calculations involving sparsity (e.g., sparse input tensors and / or sparse filters). Each SPU 562 may include a plurality of processing entities (PEs) 580, where each PE 580 can handle a unit number of convolution calculations.

[0065] Fig. 6A An example diagram of a hierarchical weighted sparse convolution according to various embodiments is shown. For illustration, Fig. 6A It shows how sub-tensors (shaded portions in input tensor 600) and sub-filter groups (shaded portions in filter 610) are assigned to multiple PEs to perform convolution in parallel.

[0066] In some embodiments, the plurality of PEs may be arranged into a matrix having a plurality of columns and rows of PEs. Each of the plurality of PEs may be allowed to exchange data with its neighboring PEs (e.g., read a non-zero weight from an internal buffer of one of its neighboring PEs in the column direction or rotate a partial sum between its neighboring PEs in the row direction).

[0067] In some embodiments, the input tensor 600 (H*W*C tensor) may first be split into a plurality of sub-tensors, for example, a plurality of W*Cg sub-tensors (e.g., a plurality of horizontal planes in the input tensor 600), where Cg refers to a channel group including a plurality of channels. These sub-tensors may then be assigned to a plurality of PEs or stored in internal buffers of a plurality of PEs. Fig. 6AAs shown in , each row of PEs (e.g., PE0, PE32, ..., PE224 of the first row 630) can be assigned a portion of the W*Cg sub-tensor. Then, each PE can pass through the horizontal plane (e.g., S*Cg) in the filter alone to list all non-zero weights. For each non-zero weight identified, the channel dimension index, R dimension index, and S dimension index of the non-zero weight can be easily derived from the corresponding hierarchical bit representation, and the PE can then easily locate the corresponding input value from the assigned sub-tensor based on the three indexes. The non-zero weight and the corresponding input value can be multiplied and accumulated to produce a partial sum. If the corresponding bit is 0, the PE can skip the entire filter block or the corresponding channel group. In some embodiments, each PE can read more non-zero weights from its neighboring PEs as needed through the weight exchange connection 624 between every two PEs in the same column 640. In some embodiments, when the slowest PE in the same row 630 completes enumerating all assigned non-zero weights, the partial sums generated by each PE in the same row 630 may be rotated for the next processing round (e.g., these partial sums may be further accumulated with new partial sums generated in the next round) through the rotation connection 622. Here, the "slowest PE" may refer to a PE with the smallest computing power or with the heaviest workload (e.g., with the most non-zero weights to be processed). In some embodiments, these partial sums may be accumulated to construct the output tensor of the convolution layer.

[0068] Fig. 6A The illustrative assignment of filters and input tensors shown in can provide the following benefits for the convolution process in a neural network: potentially saving computation time where any slowest PE in a row can reduce computation time and explicitly saving power by skipping memory accesses and multiplications (avoiding all-zero portions of filters).

[0069] Figure 6B Another example diagram of layered weight sparse convolution according to various embodiments is shown. Figure 6B In, through Fig. 6A The input tensor 600 and the filter 610 are divided and assigned to a plurality of PEs in the same manner as shown in FIG. For example, each PE can read more non-zero weights from its neighboring PEs in the column direction through the weight exchange connection 624 as needed. Fig. 6A and Figure 6B The difference between the two diagrams in is that each PE continues to pass through all S*Cg of the filter 610 until the final result of the corresponding output channel is calculated. In this way, the partial sums do not need to rotate between PEs in the same row. In some embodiments, these partial sums can be accumulated to construct the output tensor of the convolutional layer.

[0070] Figure 6BThe exemplary assignment of filters and input tensors shown in can provide the following benefits for the convolution process in a neural network: saving computation time because one PE can skip the computation of an entire output channel computation and saving power because of skipping memory accesses and multiplications.

[0071] Figure 7 An example method for hierarchical weight sparse convolution according to various embodiments is shown. The method 700 may be performed by an apparatus, device, or system for optimizing resource allocation. The method 700 may be performed by Figure 1 To one or more modules / components of the environment or system depicted in FIG. 6 (such as, Figure 5 The operations of method 700 presented below are intended to be illustrative. Depending on the implementation, method 700 may include additional, fewer, or alternative steps that are performed in various orders or in parallel.

[0072] Block 710 includes obtaining an input tensor and a plurality of filters at a convolutional layer of a neural network. In some embodiments, the plurality of filters are pruned so that each of the plurality of sub-filters has all zero weights or a predetermined number of non-zero weights.

[0073] Block 720 includes splitting the input tensor into a plurality of sub-tensors and assigning the plurality of sub-tensors to a plurality of processors. In some embodiments, assigning the plurality of sub-tensors to the plurality of processors includes: for each processor in the plurality of processors, storing one or more of the plurality of sub-tensors in an internal buffer of the processor.

[0074] Block 730 includes, for each filter in the plurality of filters, generating a hierarchical bit representation of the plurality of non-zero weights in the filter, wherein the hierarchical bit representation includes a first layer and a second layer, the first layer including a plurality of bits corresponding to the plurality of subfilters in the filter and indicating whether the corresponding subfilter has at least one non-zero weight, and the second layer including a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filter, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of one of the plurality of non-zero weights as a key. In some embodiments, each subtensor in the plurality of subtensors and each subfilter in the plurality of subfilters has the same number of channels.

[0075] Block 740 includes identifying, by each of the plurality of processors, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned sub-tensor based on the hierarchical bit representation. In some embodiments, identifying, based on the hierarchical bit representation, one or more of the plurality of non-zero weights and corresponding input values ​​from the assigned sub-tensor includes: for each non-zero weight of the plurality of non-zero weights, determining, based on a first level of the hierarchical bit representation, a channel group index, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is ultimately located; determining, based on a second level of the hierarchical bit representation, an offset of the non-zero weight within the filter channel group; and identifying the input value from the assigned sub-tensor based on the channel group index and the offset.

[0076] In some embodiments, the hierarchical bit representation further includes a third layer between the first layer and the second layer, wherein for each of the plurality of subfilters, the third layer includes a plurality of bit vectors corresponding to the plurality of horizontal planes of the subfilter, respectively, and each of the plurality of bit vectors indicates whether the corresponding horizontal plane includes at least one of the plurality of non-zero weights. In some embodiments, identifying one or more of the plurality of non-zero weights and a corresponding input value from the assigned subtensor based on the hierarchical bit representation includes: for each of the plurality of non-zero weights, determining a channel group index based on the first layer of the hierarchical bit representation, wherein the channel group index corresponds to the filter channel group in which the non-zero weight is located; determining a height index and a width index of the non-zero weight based on the third layer of the hierarchical bit representation; determining an offset of the non-zero weight within the filter channel group based on the second layer of the hierarchical bit representation; and identifying the input value from the assigned subtensor based on the channel group index, the height index, the width index, and the offset.

[0077] Block 750 includes performing a multiply and accumulate (MAC) operation on the identified one or more non-zero weights and corresponding input values ​​to obtain a partial sum.

[0078] Block 760 includes accumulating the partial sums obtained by each of the plurality of processors to construct an output tensor of a convolutional layer of the neural network.

[0079] In some embodiments, a plurality of processors are electrically arranged into a matrix including a plurality of columns and rows, and each of the plurality of processors has one or more adjacent processors in the column direction. In some embodiments, each of the plurality of processors is allowed to read one or more key-value pairs from an internal buffer of one of its adjacent processors in the column direction. In some embodiments, method 700 may further include: assigning one or more of a plurality of sub-tensors from a channel group to a row of processors, wherein the channel group includes a plurality of channels; and after each processor in the row of processors completes a MAC operation between a sub-tensor and a corresponding sub-filter based on a hierarchical bit representation and obtains a partial sum, rotating the partial sum between the processors in the row. In some embodiments, method 700 may further include: assigning one or more sub-filters from the same height index and the same width index in the plurality of sub-filters to each of the plurality of processors; and after each processor in the plurality of processors completes a MAC operation between one or more sub-filters in the plurality of sub-filters and a corresponding sub-filter based on a hierarchical bit representation and obtains a partial sum, accumulating the partial sum to construct an output tensor.

[0080] Figure 8 An example computing device is shown in which any of the embodiments described herein may be implemented. The computing device may be used to implement Figures 1 to 7 The computing device 800 may include a bus 802 or other communication mechanism for transmitting information and one or more hardware processors 804 coupled to the bus 802 for processing information. The hardware processor(s) 804 may be, for example, one or more general-purpose microprocessors.

[0081] The computing device 800 may also include a main memory 807 coupled to the bus 802 for storing information and instructions executed by the (one or more) processors 804, such as a random access memory (RAM), a cache, and / or other dynamic storage device. The main memory 807 may also be used to store temporary variables or other intermediate information during the execution of the instructions executed by the (one or more) processors 804. These instructions, when stored in a storage medium accessible to the (one or more) processors 804, can make the computing device 800 a special-purpose machine customized to perform the operations specified in the instructions. The main memory 807 may include non-volatile media and / or volatile media. Non-volatile media may include (for example) optical disks or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or networked versions thereof.

[0082] The computing device 800 may use customized hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic (which, in combination with the computing device, may cause the computing device 800 to become or program the computing device 800 to be a special-purpose machine) to implement the techniques described herein. According to one embodiment, the techniques herein are performed by the computing device 800 in response to the execution of one or more sequences of one or more instructions contained in the main memory 807 by the processor(s) 804. These instructions may be read into the main memory 807 from another storage medium (such as a storage device 809). Execution of the sequence of instructions contained in the main memory 807 may cause the processor(s) 804 to perform the process steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in the main memory 807. When these instructions are executed by the processor(s) 804, they may perform the steps shown in the corresponding figures and described above. In alternative embodiments, hard-wired circuitry may be used instead of or in conjunction with software instructions.

[0083] The computing device 800 also includes a communication interface 810 coupled to the bus 802. The communication interface 810 can provide a bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, the communication interface 810 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or a WAN component that communicates with a WAN). Wireless links can also be implemented.

[0084] The execution of certain operations may be distributed among processors and not reside only in a single machine, but deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0085] Each of the processes, methods, and algorithms described in the above sections may be embodied in a code module executed by one or more computer systems or computer processors including computer hardware and fully or partially automated by these code modules. The processes and algorithms may be partially or fully implemented in a dedicated circuit system.

[0086] When the functions disclosed herein are implemented in the form of software functional units and sold or used as independent products, they can be stored in non-volatile computer-readable storage media executable by a processor. The specific technical solutions (all or part) disclosed herein or the aspects contributing to the current technology can be embodied in the form of software products. The software product can be stored in a storage medium and includes several instructions that cause a computing device (which can be a personal computer, a server, a network device, etc.) to perform all or some steps of the method of an embodiment of the application. The storage medium may include a flash drive, a portable hard drive, a ROM, a RAM, a disk, an optical disk, other media operable to store program code, or any combination thereof.

[0087] Certain embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions, which instructions can be executed by the processor to cause the system to perform operations corresponding to the steps in any method of the embodiments disclosed above. Certain embodiments further provide a non-transitory computer-readable storage medium configured with instructions, which instructions can be executed by one or more processors to cause the one or more processors to perform operations corresponding to the steps in any method of the embodiments disclosed above.

[0088] The embodiments disclosed herein may be implemented by a cloud platform, a server, or a server group (hereinafter collectively referred to as a "service system") that interacts with a client. The client may be a terminal device or a client registered by a user at the platform, wherein the terminal device may be a mobile terminal, a personal computer (PC), and any device that can be installed with a platform application.

[0089] The various features and processes described above can be used independently of each other or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, some methods or process blocks can be omitted in some embodiments. The methods and processes described herein are also not limited to any particular order, and the blocks or states related thereto can be performed in other appropriate orders. For example, the blocks or states described can be performed in an order other than the order specifically disclosed, or multiple blocks or states can be combined in a single block or state. Example blocks or states can be performed continuously, in parallel, or in some other way. Blocks or states can be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein can be configured differently from the described. For example, components can be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0090] The various operations of the exemplary methods described herein may be performed at least in part by an algorithm. The algorithm may be included in a program code or instruction stored in a memory (e.g., the non-transitory computer-readable storage medium described above). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly teach the computer to perform a function but may learn from training data to generate a predictive model for performing the function.

[0091] The various operations of the illustrative methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors may constitute a processor-implemented engine that operates to perform one or more operations or functions described herein.

[0092] Similarly, the methods described herein may be at least partially processor-implemented, where one or more specific processors are instances of hardware. For example, at least some of the operations of the methods may be performed by one or more processors or processor-implemented engines. In addition, one or more processors may also be operable to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some operations may be performed by a computer group (as an example of a machine including a processor), where these operations may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)).

[0093] The execution of certain operations may be distributed among processors and not just reside in a single machine, but deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0094] Throughout this specification, multiple examples may implement components, operations, or structures described as a single example. Although individual operations of one or more methods are depicted and described as separate operations, one or more of the individual operations may be performed simultaneously, and there is no need to perform the operations in the order depicted. The structures and functions presented as separate components in the example configuration may be implemented as combined structures or components. Similarly, the structures and functions presented as a single component may be implemented as separate components. These and other changes, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0095] As used herein, "or" is inclusive and not exclusive, unless otherwise expressly indicated or indicated in the context. Therefore, herein, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B and C", unless otherwise expressly indicated or indicated in the context. In addition, "and" is combined and separately, unless otherwise expressly indicated or indicated in the context. Therefore, herein, "A and B" means "A and B, combined or separately", unless otherwise expressly indicated or indicated in the context. In addition, multiple instances of the resources, operations, or structures described herein may be provided as single instances. In addition, the boundaries between various resources, operations, engines, and data storage devices are slightly arbitrary, and specific operations are depicted in the context of a specific illustrative configuration. Other functional allocations are foreseeable and may fall within the scope of various embodiments of the present disclosure. In general, structures and functions presented as separate resources in the example configuration may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as separate resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as expressed by the appended claims.Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0096] The terms "comprise" or "include" are used to indicate the presence of subsequently claimed features, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context of use, conditional language such as "may," "could," "might," "can" is generally intended to convey that some embodiments include and other embodiments do not include certain features, components, and / or steps. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way or that one or more embodiments must include logic for determining whether these features, components, and / or steps are included or executed in any particular embodiment with or without user input or prompting.

[0097] Although the overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. If in fact more than one embodiment is disclosed, these embodiments of the subject matter may be referred to herein individually or collectively as the term "invention", which is only for convenience and is not intended to automatically limit the scope of the present application to any single disclosure or concept.

[0098] The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the "embodiments" are not intended to be limiting, and the scope of the various embodiments is defined solely by the appended claims and the full range of equivalents to which such claims are entitled.

Claims

1. A computer-implemented method comprising: Get input tensors and filters in the convolutional layer of the neural network; Splitting the filter into a plurality of sub-filters; generating a hierarchical bit representation of the filter representing a plurality of non-zero weights in the filter, wherein the hierarchical bit representation comprises a first layer, the first layer comprising a plurality of bits corresponding to the plurality of sub-filters in the filter, each bit in the plurality of bits indicating whether the corresponding sub-filter comprises at least one non-zero weight; as well as A multiply and accumulate (MAC) operation is performed based on the hierarchical bit representation of the filter and the input tensor.

2. The method of claim 1, wherein: MAC operations include: identifying one or more of the plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; and A MAC operation is performed on the identified one or more non-zero weights and corresponding input values.

3. The method of claim 1, wherein: The layered bit representation further includes a second layer, which includes a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filters, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key.

4. The method of claim 3, wherein: MAC operations include: for each non-zero weight of the plurality of non-zero weights, determining a channel group index based on the first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining an offset of the non-zero weight within the group of filter channels based on the second layer of the hierarchical bit representation; identifying an input value in the input tensor based on the channel group index and the offset; and A MAC operation is performed based on the identified input value and the non-zero weight.

5. The method of claim 3, wherein: The layered bit representation further includes a third layer, the third layer including a plurality of bit vectors corresponding to the plurality of horizontal planes in each of the plurality of sub-filters, respectively, wherein each bit vector of the plurality of bit vectors indicates whether the corresponding horizontal plane includes at least one non-zero weight of the plurality of non-zero weights.

6. The method of claim 5, wherein: MAC operations include: for each non-zero weight of the plurality of non-zero weights, determining a channel group index based on the first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; Determining a height index and a width index of the non-zero weight based on the third layer of the hierarchical bit representation; determining an offset of the non-zero weight within the group of filter channels based on the second layer of the hierarchical bit representation; identifying an input value in the input tensor based on the channel group index, the height index, the width index, and the offset; and A MAC operation is performed based on the identified input value and the non-zero weight.

7. The method of claim 1, further comprising: The filter is pruned so that each of the plurality of sub-filters has either all zero weights or a predetermined number of non-zero weights.

8. The method of claim 1, further comprising: Obtaining a plurality of partial sums from the MAC operation; as well as An output tensor of the convolutional layer of the neural network is constructed based on the plurality of partial sums.

9. The method of claim 1, wherein: The MAC operations are processed in parallel in a plurality of processors.

10. The method of claim 1, further comprising: The input tensor is split into a plurality of sub-tensors, wherein each sub-tensor in the plurality of sub-tensors and each sub-filter in the plurality of sub-filters have the same number of channels, and MAC operations include: The plurality of sub-tensors and the plurality of sub-filters are assigned to a plurality of processors for performing MAC operations in parallel.

11. A system for hierarchical weight sparse convolution processing, comprising one or more processors and one or more non-transitory computer readable memories coupled to the one or more processors and configured with instructions, the instructions executable by the one or more processors to cause the system to perform operations including: Get input tensors and filters in the convolutional layer of the neural network; Splitting the filter into a plurality of sub-filters; generating a hierarchical bit representation of the filter representing a plurality of non-zero weights in the filter, wherein The hierarchical bit representation includes a first layer, the first layer including a plurality of bits corresponding to the plurality of sub-filters in the filter, each bit in the plurality of bits indicating whether the corresponding sub-filter includes at least one non-zero weight; as well as A multiply and accumulate (MAC) operation is performed based on the hierarchical bit representation of the filter and the input tensor.

12. The system of claim 11, wherein: MAC operations include: identifying one or more of the plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; and A MAC operation is performed on the identified one or more non-zero weights and corresponding input values.

13. The system of claim 11, wherein: The layered bit representation further includes a second layer, which includes a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filters, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key.

14. The system of claim 13, wherein: MAC operations include: for each non-zero weight of the plurality of non-zero weights, determining a channel group index based on the first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining an offset of the non-zero weight within the group of filter channels based on the second layer of the hierarchical bit representation; identifying an input value in the input tensor based on the channel group index and the offset; and A MAC operation is performed based on the identified input value and the non-zero weight.

15. The system of claim 13, wherein: The layered bit representation further includes a third layer, the third layer including a plurality of bit vectors corresponding to the plurality of horizontal planes in each of the plurality of sub-filters, respectively, wherein each bit vector of the plurality of bit vectors indicates whether the corresponding horizontal plane includes at least one non-zero weight of the plurality of non-zero weights.

16. A non-transitory computer-readable storage medium for hierarchical weight sparse convolution processing, configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: Get input tensors and filters in the convolutional layer of the neural network; Splitting the filter into a plurality of sub-filters; generating a hierarchical bit representation of the filter representing a plurality of non-zero weights in the filter, wherein The hierarchical bit representation includes a first layer, the first layer including a plurality of bits corresponding to the plurality of sub-filters in the filter, each bit in the plurality of bits indicating whether the corresponding sub-filter includes at least one non-zero weight; as well as A multiply and accumulate (MAC) operation is performed based on the hierarchical bit representation of the filter and the input tensor.

17. The non-transitory computer readable storage medium of claim 16, wherein: MAC operations include: identifying one or more of the plurality of non-zero weights in the filter and corresponding input values ​​in the input tensor based on the hierarchical bit representation; and A MAC operation is performed on the identified one or more non-zero weights and corresponding input values.

18. The non-transitory computer readable storage medium of claim 16, wherein: The layered bit representation further includes a second layer, which includes a plurality of key-value pairs corresponding to the plurality of non-zero weights in the filters, each key-value pair including one of the plurality of non-zero weights as a value and a channel identifier of the one of the plurality of non-zero weights as a key.

19. The non-transitory computer readable storage medium of claim 18, wherein: MAC operations include: for each non-zero weight of the plurality of non-zero weights, determining a channel group index based on the first layer of the hierarchical bit representation, wherein the channel group index corresponds to a filter channel group in which the non-zero weight is located; determining an offset of the non-zero weight within the group of filter channels based on the second layer of the hierarchical bit representation; identifying an input value in the input tensor based on the channel group index and the offset; and A MAC operation is performed based on the identified input value and the non-zero weight.

20. The non-transitory computer readable storage medium of claim 18, wherein: The layered bit representation further includes a third layer, the third layer including a plurality of bit vectors corresponding to the plurality of horizontal planes in each of the plurality of sub-filters, respectively, wherein each bit vector of the plurality of bit vectors indicates whether the corresponding horizontal plane includes at least one non-zero weight of the plurality of non-zero weights.

Images