System and method for balancing sparsity in weights to accelerate deep neural networks
The sparsity balancing system optimizes DNN accelerator efficiency by evenly distributing sparsity in weight and activation vectors, addressing computational inefficiencies and energy consumption issues in deep neural networks.
Patent Information
- Application Number
- JP2023580469
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-24
- Filing Date
- 2022-10-14
- Publication Date
- 2024-12-03
AI Technical Summary
Deep neural networks (DNNs) require significant computational resources and energy due to their high accuracy, which is hindered by unbalanced sparsity in weight and activation distributions, leading to inefficiencies in processing elements (PEs) and synchronization bottlenecks.
A sparsity balancing system that rearranges weight and activation vectors using bitmaps to achieve an even distribution of zero-valued elements, optimizing workload balance across PEs and reducing synchronization requirements.
This approach enhances the efficiency of DNN accelerators by improving computational throughput and reducing energy consumption through balanced sparsity, thereby accelerating deep learning processes.
Smart Images

Figure 2024544447000001_ABST
Abstract
Description
[Technical field]
[0001]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Non-provisional Application No. 17 / 534,976, filed November 24, 2021, entitled “SYSTEM AND METHOD FOR BALANCING SPARSUS IN WEIGHTS TO ACCELERATE DEEP NEURAL NETWORKS,” the entire contents of which are incorporated herein by reference for all purposes.
[0002]
[0002] Technical Field The present disclosure relates generally to neural networks, and more particularly to deep neural networks. neural networks, This is related to accelerating the computational power of deep neural networks (DNNs). [Background technology]
[0003]
[0003] Deep neural networks are widely used for various artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, that high accuracy comes at the cost of significant computational cost. DNNs have extremely high computational demands, as each inference may require hundreds of millions of filter weights to be stored for classification or detection, as well as hundreds of millions of MAC (multiply-accumulate) operations. Thus, techniques to improve the efficiency of DNNs are needed. [Brief description of the drawings]
[0004]
[0004] The embodiments will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which:
[0004] To facilitate this description, like reference numerals may denote like structural elements.
[0004] The embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which: [Figure 1]
[0005] FIG. 1 illustrates an exemplary DNN architecture in accordance with various embodiments. [Diagram 2]
[0006] FIG. 2 illustrates a hardware architecture for a layer of a DNN according to various embodiments. [Diagram 3]
[0007] FIG. 3 is a block diagram of a processing element (PE) in accordance with various embodiments. [Figure 4]
[0008] FIG. 4 illustrates a process for generating a bitmap according to various embodiments. [Diagram 5]
[0009] FIG. 5 illustrates PEs with unbalanced workloads in accordance with various embodiments. [Figure 6]
[0010] FIG. 6 illustrates a PE with a balanced workload in accordance with various embodiments. [Figure 7]
[0011] FIG. 7 illustrates a process for balancing sparsity in a weight vector according to various embodiments. [Figure 8]
[0012] FIG. 8 illustrates two adjacent DNN layers with rearranged weight vectors in accordance with various embodiments. [Figure 9]
[0013] FIG. 9 illustrates a sparsity balancing system architecture for combined sparsity balancing in accordance with various embodiments. [Figure 10]
[0014] FIG. 10 illustrates a deep learning environment in accordance with various embodiments. [Figure 11]
[0015] FIG. 11 is a block diagram of a DNN system according to various embodiments. [Figure 12]
[0016] FIG. 12 is a flow chart illustrating a method for balancing sparsity in a weight vector according to various embodiments. [Figure 13]
[0017] FIG. 13 is a block diagram of an exemplary computing device in accordance with various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0005]
[0018] overview Deep Learning learning, DL models are characterized by their ability to generate valid abstract representations of data using autonomous latent feature extraction. To achieve this, DNNs are substantially more complex than more traditional machine learning techniques and require orders of magnitude larger numbers of parameters to be trained. Processing Units, GPUs and ASICs (application-specific integrated GPU- and ASIC-based DNN accelerators are suitable for executing these types of workloads because they consist of thousands of parallel MAC units that can work simultaneously and generate output in a shorter time. However, these GPU- and ASIC-based execution platforms usually have very high power consumption, which makes them unsuitable for deployment in high-energy constrained systems where power and area budgets are severely limited.
[0006]
[0019] An efficient technique to improve the performance and energy consumption of DNN accelerators is by exploiting the property of sparsity, which is abundantly present in the network. Sparsity refers to the presence of zeros in the weights and activations in a DNN. Zeroed activations in a DNN come from processing the layers with an activation function, while zeroed weights are usually pruned by filter pruning. These zeroed activations and weights do not contribute to the results during MAC operations in the convolutional and fully connected layers, and therefore can be skipped in terms of both computation and storage. To this end, machine learning accelerators can exploit this sparsity available in activations and weights to achieve significant speedups during computation, which translates into power savings, since the same work can be accomplished using less energy, in addition to reducing the storage requirements for weights (and activations) through efficient compression schemes. Both reducing the total amount of data transfer across the memory hierarchy and reducing the overall computation time are important to improve energy efficiency in machine learning accelerators.
[0007]
[0020] Some DNN accelerators can improve throughput by accelerating sparsity, but the processing elements of the DNN accelerator elements, Many situations can arise where the random distribution of sparse data in the PEs is negligible or even zero, which can lead to sparsity-induced speedups. Most sparse DNN accelerators: assume DNN data flow, drain phase Synchronization barrier during the extraction of output points from the arithmetic units to the upper memory hierarchy Due to a variety of factors, including overhead associated with splitting work into multiple smaller tasks among multiple PEs, and the overhead associated with skipping computations on sparse data, we rarely achieve the maximum speedup that can be obtained from skipping computations on sparse data. The synchronization requirement during the drain operation mainly stems from requiring that the final output points (those corresponding to the output channels) be extracted simultaneously so that they can be compressed or packed together for storage before forming the activations for the next DNN layer.
[0008]
[0021] An embodiment of the present invention relates to a sparsity balancing system capable of balancing sparsity in weights to improve the efficiency of a DNN accelerator. The sparsity balancing system uses a bitmap to rearrange the weights to achieve an even (or near-even) distribution of sparse data (i.e., zero values) in the weights, thereby allowing the processing PEs to have a balanced workload. The sparsity balancing system is also capable of combined sparsity balancing, including both sparsity balancing in weights and sparsity balancing in activations.
[0009]
[0022] In some embodiments, the sparsity balancing system generates a bitmap based on a weight vector. The weight vector includes a sequence of weights. The weight vector may be derived from one or more filters of a DNN layer. The weights may have non-zero or zero values. The bitmap includes a sequence of bitmap elements, each of which has a value determined based on the values of the different weights. For example, a bitmap element for a non-zero value weight has a value of one, while a bitmap element for a zero value weight has a value of zero. An index of a bitmap element in the bitmap matches an index of a corresponding weight in the weight vector, i.e., a first bitmap element in the bitmap corresponds to a first weight in the weight vector, a second bitmap element in the bitmap corresponds to a second weight in the weight vector, and so on. The sparsity balancing system may generate a bitmap based on both weights and activations. Such a bitmap may be referred to as a combined bitmap. The sparsity balancing system then rearranges the bitmap and generates a new bitmap.
[0010]
[0023] In the process of rearranging the bitmap, the sparsity balancing system changes the indexes of at least some of the bitmap elements to achieve an even (or nearly even) distribution of non-zeroed (or zeroed) bitmap elements in the new bitmap. For example, the sparsity balancing system determines a number of intervals (such as a constant number of intervals) based on a division operation on the non-zero (or zero-valued) bitmap elements and the total number of bitmap elements in the bitmap. The number of intervals may be an integer. The sparsity balancing system rearranges the bitmap so that for every interval number of bitmap elements, there is a non-zero (or zero-valued) bitmap element. In some embodiments, the sparsity balancing system compares the number of non-zero valued bitmap elements to the number of zero valued bitmap elements and selects the smaller number to perform the division operation. The sparsity balancing system also targets an even distribution of non-zero valued bitmap elements when there are fewer non-zero valued bitmap elements than zero valued bitmap elements, and vice versa. In this way, the rearrangement process may be more efficient. An even distribution of the non-zero valued bitmap elements may result in an even distribution of the zero valued bitmap elements. Similarly, an even distribution of the zero valued bitmap elements may result in an even distribution of the non-zero valued bitmap elements.
[0011]
[0024] After the new bitmap is generated, the sparsity balancing system can rearrange the weight vector based on the new bitmap. For example, if a bitmap element is moved during the rearrangement process, the corresponding weight is similarly moved. Thus, the rearranged weight vector has an even (or nearly even) distribution of non-zero value weights. In some embodiments, for example, in embodiments where the bitmap is a combined bitmap, the activation vector can also be arranged based on the new bitmap. Furthermore, the sparsity balancing system divides the weight vector into subsets and assigns each subset to a different PE. Since the rearranged weight vector (or both the rearranged weight vector and the rearranged activation vector) have an even (or nearly even) distribution of non-zero value weights, the PEs will have the same or similar workload. Thus, there will be good synchronization between the MAC operations of the PEs. Thus, the present invention reduces or even eliminates sparsity bottlenecks resulting from synchronization requirements between multiple PEs in a DNN accelerator.
[0012]
[0025] For purposes of explanation, specific numbers, materials, and configurations are set forth to provide a thorough understanding of example implementations. However, it will be apparent to one skilled in the art that the disclosure may be practiced without the specific details and / or that the disclosure may be practiced with only some of the described aspects. In other instances, well-known features have been omitted or simplified so as not to obscure example implementations.
[0013]
[0026] Additionally, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration embodiments which may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.
[0014]
[0027] Various operations may be described in sequence as multiple separate operations or processes in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order presented. The operations described may be performed in a different order than in the described embodiment. Various additional operations may be performed or the operations described may be omitted in additional embodiments.
[0015]
[0028] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in reference to a range of measurements, the term "between" includes both ends of the range of measurements.
[0016]
[0029] The description uses the phrases "in an embodiment" or "in an embodiment," each of which may refer to one or more of the same or different embodiments. Terms such as "comprising," "including," and "having," as used with respect to the embodiments of the present disclosure, are synonymous. Although the present disclosure may use perspective descriptions such as "upper," "lower," "top," "bottom," and "side" to describe various features of the drawings, these terms are merely for ease of discussion and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects is merely to indicate that different instances of similar objects are being referred to and is not intended to imply that the objects so described must be in a given order, whether temporally, spatially, ranking, or in any other manner.
[0017]
[0030] In the following detailed description, various aspects of exemplary implementations are described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.
[0018]
[0031] The terms "substantially," "close," "approximately," "near," and "about" generally refer to within + / - 20% of a target value, based on the context of the particular value, as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as "coplanar," "vertical," "orthogonal," "parallel," or any other angle between elements, generally refer to within + / - 5-20% of a target value, based on the context of the particular value, as described herein or known in the art.
[0019]
[0032] It is noted that the terms "comprises," "comprising," "including," "including," "having," "having," or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a method, process, device, or sparsity balancing system that includes a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or that are essential to such method, process, device, or sparsity balancing system. Also, the term "or" refers to an inclusive "or" and not an exclusive "or."
[0020]
[0033] The sparsity balancing systems, methods, and devices of the present disclosure each have several inventive aspects, no single aspect of which is solely responsible for all of the desirable attributes disclosed herein. Details of one or more implementations of the subject matter described herein are set forth in the following description and the accompanying drawings.
[0021]
[0034] DNN architecture example FIG. 1 illustrates an exemplary DNN in accordance with various embodiments. 100. For illustrative purposes, the DNN in FIG. 100 is VGG (Visual Geometry Group-based convolutional networks neural In another embodiment, the DNN 100 may also be other types of DNNs. 100 is trained to receive images and output classifications of objects in the images. The DNN 100 receives an input image 105 containing object objects 115, 125, and 135. The DNN 100 includes a series of layers (sequence layers) including a number of convolutional layers 110 (individually referred to as "convolutional layers 110"), a number of pooling layers 120 (individually referred to as "pooling layers 120"), and a number of fully connected layers 130 (individually referred to as "fully connected layers 130"). In another embodiment, the DNN 100 may include fewer, more, or different layers.
[0022]
[0035] The convolutional layer 110 summarizes the presence of features in the input image 105. The convolutional layer 110 acts as a feature extractor. The first layer of 100 is a convolutional layer 110. In one example, the convolutional layer 110 performs the IFM by using a weight matrix 150. Perform convolution on 140 and extract OFM from the convolution. Generated 160, OFM Pass 160 to the next layer in the sequence. IFM 140 may contain multiple IFM matrices. 160 may contain multiple OFM matrices. Regarding the first convolutional layer 110, which is also the first layer of 100, 140 is the input image 105. For other convolution layers, IFM 140 can be the output of another convolutional layer 110 or the output of the pooling layer 120. The convolution is performed using a weight matrix 150 and an IFM It is a linear operation involving multiplication by 140. The filter can be a two-dimensional array of weights. The filter weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the filter weights is determined by the IFM The filter may indicate the importance of the weight matrix 150 in extracting features from the IFM It is possible that it is smaller than 140.
[0023]
[0036] IFM The multiplication applied between the patch and the filter with a filter size of 140 could be a dot product. The dot product is the IFM IFM is an element-wise multiplication between patches of filter size 140 and the corresponding filters, which are then summed and always result in a single value. Since it results in a single value, the operation is sometimes referred to as a "scalar product". Using a filter smaller than 140 is intentional because the same filter (set of weights) can be used in IFM IFM at 140 different points 140. In particular, the filter is an IFM For each overlapping portion or patch of filter size 140, the IFM filter is systematically applied from left to right and top to bottom. The result of multiplying by 141 once is a single value. The filter is IFM 140 multiple times, the multiplication result is IFM The output array from this operation is a two-dimensional array of output values that represents the filtering of 140. Thus, the two-dimensional output array from this operation is called a "feature map." It is called the “map”.
[0024]
[0037] In some embodiments, the OFM 160 is passed through an activation function. One example of an activation function is the rectified linear activation function. linear activation The ReLU function (ReLU) is a calculation that either directly returns the value provided as input, or returns the value 0 if the input is less than or equal to 0. The convolutional layer 110 is capable of receiving several images as input and computing the convolution of each of them with each of the filters. This process can be repeated several times. For example, the OFM 160 is a convolutional layer 110 (i.e., OFM in the sequence) The new filter is then passed to the convolutional layer 110 that produces the OFM 160. The subsequent convolutional layer 110 then uses the new filter to generate the OFM 160. 160 to generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be filtered again by further subsequent convolutional layers 110, and so on.
[0025]
[0038] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of filters, the size F of the filters (e.g., the filters are F×F×D dimensional pixels), the step S by which the window corresponding to the filters is dragged over the image (e.g., a step of 1 means to move the window one pixel at a time), and zero padding P (e.g., adding a black contour P pixels thick to the input image of the convolutional layer 110). The convolutional layer 110 can perform 2D convolution, dilated or atrous convolution, spatial separation, and other convolutional operations. separable convolution, depth-wise separable It is possible to perform different types of convolutions, such as separable convolutions, transposed convolutions, etc. 100 includes 16 convolutional layers 110. In another embodiment, the DNN 100 may contain a different number of convolutional layers.
[0026]
[0039] The pooling layer 120 downsamples the feature map generated by the convolutional layers, e.g., by summarizing the presence of features within patches of the feature map. The pooling layer 120 is located between two convolutional layers 110: the preceding convolutional layer 110 (the convolutional layer 110 that precedes the pooling layer 120 in the sequence of layers) and the succeeding convolutional layer 110 (the convolutional layer 110 that follows the pooling layer 120 in the sequence of layers). In some embodiments, the pooling layer 120 is located after the convolutional layer 110, e.g., with an activation function (e.g., ReLU) that is OFM. This is added after the 160 is applied.
[0027]
[0040] The pooling layer 120 receives the feature map generated by the preceding convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while preserving important properties of the feature map. Thus, the pooling operation improves the efficiency of the DNN and avoids overfitting. The pooling layer 120 may perform the pooling operation by average pooling (which calculates the average value for each patch on the feature map), max pooling (which calculates the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of 2 pixels, so that the pooling operation reduces the size of the feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to a quarter of its size. In one example, the pooling layer 120 applied to a 6x6 feature map results in an output pooled feature map of 3x3. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of the same number of pooled feature maps.
[0028]
[0041] fully connected layer connected The fully connected layers 130 are the last layers of the DNN. The fully connected layers 130 may be convolutional or not. They receive an input vector. The input vector defines the output of the convolutional layers 110 and the pooling layers 120 and contains the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input vector to generate an output vector. The output vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1, and the sum of all equals 1. These probabilities are calculated by the last fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as the activation function.
[0029]
[0042] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns a vector of size N, where N is the number of classes in the image classification problem. In the embodiment of FIG. 1, N is equal to 3 because there are three objects 115, 125, and 135 in the input image. Each element of the vector indicates the probability that the input image 105 belongs to a class. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, takes the sum, and then applies an activation function (e.g., logistic if N=2, softmax if N>2). This is equivalent to multiplying the input vector by a matrix that includes the weights. In one example, the output vector includes three probabilities: a first probability indicating that the object 115 is a tree, a second probability indicating that the object 125 is a car, and a third probability indicating that the object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the output vectors may be different.
[0030]
[0043] Example hardware architecture for DNN layer FIG. 2 illustrates a hardware architecture 200 for a layer of a DNN in accordance with various embodiments. The hardware architecture 200 includes multiple PEs. 210 (individually "PE 210”) and column buffers 220 (individually referred to as “column buffers 220”). In other embodiments, hardware architecture 200 includes other components such as static random access memory (SRAM) for storing the inputs and outputs of the layers. Hardware architecture 200 may also include a distribution unit for distributing data stored in the SRAM to the column buffers 220.
[0031]
[0044] PE 210 performs multiply-accumulate (MAC) operations. 210 may also be referred to as a neuron in a DNN. 210 receives the input and the filter of the layer, and generates the output of the layer through a multiply-and-accumulate operation. 210 has two input signals 250 and 260 and an output signal 270. The input signal 250 is, for example, part of the input to the layer (e.g., an input feature map). The input signal 260 is part of the weights of the filters in the layer. The weights can have non-zero and zero values. The values of the weights are determined during the process of training the DNN. The weights can be divided and assigned to the PEs based on a bitmap. Further details regarding the bitmap are provided below in connection with FIG. 4.
[0032]
[0045] Each PE 210 performs a multiplication operation on input signals 250 and 260. 210 are connected to each other as shown by the dashed arrows in FIG. The output signal of 210 is PEs 210 through the interconnection between many other PEs as input signals 210 (and possibly itself receiving the return value). The output signal 270 of 210 is Through 210 accumulation operations, one or more other PEs There is a possibility to incorporate the output signal of 210. Further details about 210 are discussed below in conjunction with FIG.
[0033]
[0046] In the embodiment of FIG. 210 are arranged in columns 205 (individually referred to as “columns 205”). The inputs and filters of the layers are assigned to the PEs based on the columns 205. For example, each column 205 may be associated with an input channel and an output channel of a layer. 210 uses filters to transform the input channels into output channels. In other embodiments, the inputs and filters of the layers are processed by the PEs in different ways. 210. Each column 205 has a column buffer 220. The column buffer 220 stores the PEs. Data provided to 210 and PEs 210. Each column buffer 220 is associated with a load 230 and a drain 240. The data supplied to 210 is transmitted to the column buffer 220 via the load 230. The data generated by 210 is removed from the column buffer 220 via a drain 240. In some embodiments, the data extracted from the column buffer 220 is sent to a higher memory hierarchy, for example, SRAM, via a drain operation.
[0034]
[0047] The drain operation is performed by draining all PEs in the column. However, different PEs 210 may take different amounts of time to complete their multiply-accumulate operations due to the uneven sparsity in the filter weights. For example, given the sparsity of the filter weights, 210 is the same as other PEs in the same column 205 It is possible that the PE receives a filter with a lower sparsity level than the filter sent to 210. A lower sparsity level means that there are more non-zero weights. If 210 receives a filter with a lower sparsity level, The multiplication and accumulation of 210 is performed by another PE It will be slower than the multiply-and-accumulate operation of 210. Therefore, 210 bottlenecks the drain operation of column 205. Similarly, column 205 may be slower than other columns and bottleneck the drain operation of the entire layer. Therefore, improving sparsity balance in the filter weights can improve the efficiency of the layer. Further details regarding the impact of sparsity balance on the efficiency of the layer are described below in conjunction with Figures 5 and 6.
[0035]
[0048] FIG. 3 illustrates a PE according to various embodiments. 210 in FIG. 210 includes an input register file 310, a weight register file 320, an output register file 330, and a MAC unit 340. 210 may include fewer, more, or different components.
[0036]
[0049] The input register file 310 is The input signal received by the PE 210 is temporarily stored. The input feature data is then sent to other PEs. 210 and the output signal from the PE The output register file 310 temporarily stores the weight received by the PE 210 temporarily stores the generated output signal. For the sake of explanation and simplicity, the PE in FIG. 210 includes one input register file 310, one weight register file 320, and one output register file 330. In another embodiment, the PE 210 may include multiple register files for each type of data.
[0037]
[0050] The MAC unit 340 performs a MAC operation on the data in the input register file 310 and the weight register file 320. The MAC unit 340 includes a multiplication unit 350 and an accumulation unit 360. The multiplication unit 350 performs a multiplication operation on the input feature data in the input register file 310 and the weights in the weight register file 330. The length of time required by the multiplication unit 350 for the multiplication operation depends on the sparsity level of the weights used in the multiplication operation. If the weights are denser (i.e., the sparsity level is lower), the multiplication unit 350 requires more time to perform the multiplication operation. The accumulation unit 360 performs an accumulation operation on the output of the multiplication unit 350, the output signals from other PEs. The output of the accumulation unit 360 is used to accumulate the weights in the PE. This is the output signal of 210.
[0038]
[0051] Example process for generating a bitmap 4 illustrates a process for generating a bitmap 410 according to various embodiments. The bitmap 410 is generated based on a weight vector 420 and an activation vector 440. For purposes of illustration and simplicity, the weight vector 420 and the activation vector 440 each contain eight elements. In other embodiments, the weight vector 420 or the activation vector 440 may contain a different number of elements.
[0039]
[0052] The weight vector 420 may be a vector in the weight matrix of a DNN layer. The weight matrix may be part or all of the filters in the DNN layer. The weight vector 420 includes a sequence of eight elements, i.e., eight weights. Each weight has a value. The value may be zero, a non-zero integer, or a fraction. For example, the first weight in the vector has a value of 2.1, the second weight in the vector has a value of 4, and so on. The fourth, fifth, and seventh weights have a value of zero. The weight bitmap 430 is generated from the weight vector 420. In some embodiments, the weight bitmap 430 is generated by converting each weight in the weight vector 420 into a weight bitmap element. The index of the weight bitmap element in the weight bitmap 430 corresponds to the index of the corresponding weight in the weight vector 420. Weight bitmap elements have values that are determined based on the values of the weights: weight bitmap elements for non-zero weights have a value of 1, while weight bitmap elements for zero weights have a value of 0. For example, the first weight has a value of 2.1, so the first weight bitmap element has a value of 1. The seventh weight has a value of 0, so the seventh weight bitmap element has a value of 0.
[0040]
[0053] Similarly, an activation bitmap 450 is generated based on the activation vector 440. The activation vector 440 is a vector associated with the activations in the DNN layer. The activation bitmap 450 includes a sequence of eight activation bitmap elements, whose indexes and values are determined based on the elements of the activation vector 440.
[0041]
[0054] Further, bitmap 410 is generated by combining weight bitmap 430 and activation bitmap 450. In FIG. 4, a multiplication operation 460 is performed on weight bitmap 430 and activation bitmap 450. Multiplication operation 460 outputs bitmap 450. Multiplication operation 460 includes multiplying each weight bitmap element in weight bitmap 430 by a corresponding activation bitmap element in activation bitmap 450. The indices of the activation bitmap elements in activation bitmap 450 match the indices of the weight bitmap elements in weight bitmap 430. For example, a first weight bitmap element in weight bitmap 430 is multiplied by a first activation bitmap element in activation bitmap 450, a second weight bitmap element in weight bitmap 430 is multiplied by a second activation bitmap element in activation bitmap 450, and so on. Thus, a bitmap element in bitmap 410 has a value of zero as long as one of the corresponding weight and activation bitmap elements has a value of zero.
[0042]
[0055] 4 is generated based on both the weight vector 420 and the activation vector 440. Thus, the bitmap 410 represents sparsity in both the weight vector 420 and the activation vector 440. In other embodiments, the bitmap 410 may be generated based on the weight vector 420 but not the activation vector 440, and thus may represent sparsity in the weight vector 420 but not the activation vector 440.
[0043]
[0056] Example of a PE with an unbalanced workload FIG. 5 illustrates a PE with an unbalanced workload in accordance with various embodiments. FIG. 5 shows five bitmaps 510 and 520A-E for five different weight vectors (not shown in FIG. 5). FIG. 5 shows a set of filters 510 (collectively referred to as “bitmaps 510” or “bitmaps 510”) from the DNN layers. The weight vectors may be extracted from a single or multiple filters in the DNN layers. The bitmaps 510 are generated based on the weight vectors. In some embodiments, the bitmaps 510 may also be generated based on a combination of the weight vectors and the activation vectors. Each bitmap 510 includes a sequence of bitmap elements having values of 1 and 0. The number of bitmap elements having a value of 0 in a bitmap 510 indicates the sparsity level of the corresponding weight vector. As shown in FIG. 5, bitmap 510E has the lowest sparsity level since all bitmap elements have a value of 1. Bitmap 510A has the second lowest sparsity level with three 0 bitmap elements. Bitmap 510D has the third lowest sparsity level with four 0 bitmap elements. Bitmap 510B-C has the highest sparsity level because six of the bitmap elements have a value of 0, more than any of the other bitmaps 510.
[0044]
[0057] The five weight vectors are assigned to five PEs to perform the MAC operation. 520A-E(“PEs 520” or “PE Each PE is assigned a 520 performs a MAC operation on the different weight vectors. The amount of time for the MAC operation correlates with the sparsity level of the corresponding weight vector. Assuming that the PEs 520 have the same computing power, it takes longer to multiply a weight vector with more non-zero values. Since the sparsity of the weight vectors is not balanced (i.e., the sparsity levels are different), the PEs The workload of the 520 is not balanced.
[0045]
[0058] Figure 5 shows each PE PE 520 includes a clock 530 to indicate the amount of time it takes to perform a MAC operation on the weight vector assigned to that PE. The 520A takes 5 cycles to perform its MAC operation, while the PE 520B-C is 2 cycle, PE 520D: 3 cycles, PE It takes 7 cycles on the 520E. 520 is after they have completed their respective MAC calculations. 520E will be inactive for a period before it finishes its MAC operation. The drain operation continues until the PEs 520 finish their MAC operations. Since it is not possible to extract the output signal from the 520, 520E is PE This is the bottleneck in the efficiency of the 520, This slows down the entire MAC calculation process of the 510.
[0046]
[0059] Example of a PE with a balanced workload FIG. 6 illustrates a PE with a balanced workload in accordance with various embodiments. FIG. 6 shows five bitmaps 610A-E (collectively referred to as “bitmaps 610” or “bitmaps 610”) for five different weight vectors (not shown in FIG. 6). The weight vectors may be extracted from one or more filters of the DNN layers. The bitmaps 610 are generated based on the weight vectors. In some embodiments, the bitmaps 610 may also be generated based on a combination of the weight vectors and the activation vectors. Each bitmap 610 includes a sequence of bitmap elements having values of 1 and 0. Unlike the bitmaps 510 of FIG. 5, which have different sparsity levels, the bitmaps 610 of FIG. 6 have the same sparsity level. All bitmaps 610 include three zero elements.
[0047]
[0060] The five weight vectors are assigned to five PEs to perform the MAC operation. 620A-E(“PEs 620” or “PE Each PE is assigned a 620 performs MAC operations on different weight vectors. Since the weight vectors have the same sparsity level, the PEs 620 has a workload. Assuming that 620 have the same computing power, PEs 6, each PE 620 takes the same amount of time to complete their MAC operations. 630 takes 4 cycles as indicated by clock 630 to complete the MAC operation. The extract operation can begin at the end of the 4th cycle, which is faster than the MAC operation of FIG. 6 as compared to the embodiment of FIG. 5, where the drain operation cannot begin until the end of the 6th cycle.
[0048]
[0061] Bitmap 610 can be generated by rearranging bitmap 510, i.e., by changing the indices of some bitmap elements in the sequence. The weight vectors in Figure 5 can be rearranged based on the rearranged bitmap, i.e., bitmap 610, to accelerate the DNN layer.
[0049]
[0062] Example of the Bitmap Repositioning Process FIG. 7 illustrates a process for rearranging weight vectors according to various embodiments. The rearrangement of the weight vectors is based on a bitmap 710 of the weight vectors. The bitmap 710 may be generated from the weight vectors or a combination of the weight vectors and activation vectors. For simplicity and illustrative purposes, the bitmap 710 includes 16 elements arranged in a sequence. Each element has a different index within the sequence. In other embodiments, the bitmap 710 may include a different number of elements.
[0050]
[0063] The rearrangement process begins by determining how many elements have a value of 1 and how many elements have a value of 0. In Figure 7, bitmap 710 includes four elements with a value of 1 and 12 other elements with a value of 0. Because the number of elements with a value of 1 is less than the number of elements with a value of 0, the elements with a value of 1 are rearranged, reducing the resources required by the rearrangement process.
[0051]
[0064] Next, a division operation is performed: the number of all elements in the bitmap 710, i.e. 16, is divided by the number of elements with a value of 1, i.e. 4. The result of the division is 4 (16 / 4=4). The result of the division may not be an integer. In some embodiments, a floor operation is performed on the result of the division. The operation is performed and can return the largest integer less than or equal to the division result. 4 is the interval number. number).
[0052]
[0065] Additionally, at least some of the elements with a value of 1 are rearranged so that for every four (i.e., the number of intervals) elements, there is one element with a value of 1. Figure 7 shows four steps for rearranging the elements with a value of 1. Step 1 identifies the third element in the sequence as the first element with a value of 1 and moves the third element to the beginning of the sequence. The element that was at the beginning of the sequence is moved to the third index of the sequence. In other words, the indexes / positions of the two elements are swapped.
[0053]
[0066] Similarly, step 2 identifies the fourth element as the second element having a value of 1 and swaps the index of the fourth element with the fifth element, so that the difference between the indexes of both of the two nearby elements having a value of 1 is the interval number, 4. Step 3 identifies the seventh element as the third element having a value of 1 and swaps the indexes of the seventh and ninth elements. Step 4 identifies the fifteenth element as the fourth element having a value of 1 and swaps the indexes of the fifteenth and thirteenth elements.
[0054]
[0067] After the four steps, a new bitmap 720 is generated. The new bitmap 720 contains all 16 elements in the bitmap 710, but the indices of the elements have changed. In the new bitmap 720, for every fourth element, there is an element with a value of 1. The new bitmap 720 can be used to rearrange the weight vector to generate a new weight vector that contains the same weights as the original weight vector, but with different indices of the weights. The sparsity in the new weight vector is balanced. The new weight vector can be divided into four sections, corresponding to the four sections 725A-D of the new bitmap 720. Each section can be sent to a different PE that performs a MAC operation on the section. In this way, all four PEs have the same workload.
[0055]
[0068] Example DNN Layer with Rearranged Weight Vectors FIG. 8 illustrates two adjacent DNN layers 810 and 820 with rearranged weight vectors, according to various embodiments. Layer 810 is adjacent to and precedes layer 820 in the DNN. Layers 810 and 820 may be convolutional layers in the DNN. An activation function 830 is between layer 810 and layer 820. In some embodiments, activation function 830 may be implemented as part of layer 820.
[0056]
[0069] Layer 810 includes a load 813, a PE array 815, and a drain 817. The load 813 loads the input feature map and filters of layer 810 into the PE array 815. The PE array 815 performs a MAC operation. An embodiment of the PE array is the PE array 200 of FIG. 3. The drain 817 extracts the output of the PE array 815, which is the output feature map of layer 810. The output feature map of layer 810 is transmitted to layer 820 via an activation function 830. The activation function 830 takes the output feature map of layer 810 and converts it into the input feature map of layer 820.
[0057]
[0070] The layer 820 includes a load 823, a PE array 825, and a drain 827. The load 823 loads the input feature map and filters of the layer 820 into the PE array 825. An embodiment of the PE array 825 is the PE array 200 of FIG. 2, which performs a MAC operation on the input feature map, the filters, and the activations 830. The filters include one or more weight vectors that are rearranged to balance sparsity. In some embodiments, all weights in a filter are rearranged to balance sparsity. Each filter may be used as a unit in the sparsity balancing process. For example, the weight matrix of the filter is converted into a weight vector, e.g., weight vector 710. Alternatively, a portion of the filter is used as a unit.
[0058]
[0071] If the order of the weights is changed, the order of the elements in the input feature map may also need to be changed. The reason for this is that the input feature map and the weights come to the DNN layer as a pair, so if the index of the weights is changed, the same change needs to be made to the elements in the input feature map. The change to the order of the elements in the input feature map can be made by the preceding layer, i.e., layer 810, to generate the input feature map of layer 820 in an order that matches the rearranged weight vector. The ordering of the input feature maps and the output feature maps in the DNN layer can be independent, and thus the input feature maps and the output feature maps can be ordered in different ways. This decoupling allows the change to the order of the output feature map of layer 810 (i.e., the input feature map of layer 820) to match the rearranged weight vector in layer 820.
[0059]
[0072] In some embodiments, the activation vectors in the activation function 830 are rearranged based on the bitmap of the rearranged weight vectors, so that the input feature map of layer 820 generated by the activation function 830 is aligned with the rearranged filter of layer 820. The weight vectors in the filter of layer 810 may be rearranged based on the bitmap of the rearranged weight vectors to offset the rearrangement of the weight vectors of layer 820. When the weight vectors of layer 810 are rearranged, the output feature map of layer 810 and the input feature map of layer 820 will be rearranged accordingly. Thus, during the MAC operation of the PE array 815, the effect of the rearrangement on the output feature map of layer 820 is eliminated, so that the output feature map of layer 820 is still aligned with the MAC operation in the next layer, i.e., the layer following layer 820. In embodiments where layer 810 is the first layer of the DNN, the weights and activations can be rearranged offline, for example by a compiler, before being loaded into the PE array 815.
[0060]
[0073] Sparsity Balancing System Example FIG. 9 illustrates a combined sparsity balancing scheme in accordance with various embodiments. sparsity 4 shows an architecture of a sparsity balancing system 900 for joint sparsity balancing. Joint sparsity balancing includes balancing sparsity in weights and balancing sparsity in activations. The sparsity balancing system 900 balances sparsity in both weights and activations based on a joint bitmap 980. The joint bitmap 980 is generated based on both weights and activations. An embodiment of the combined bitmap is the bitmap 410 of FIG. 4. The sparsity balancing system 900 includes an activation store 910, an activation rearranger 920, an activation distributor 925, a weight store 930, a weight rearranger 940, a weight distributor 945, and multiplexers, MUXes 950, 960, 970. In other embodiments, the sparsity balancing system 900 may include fewer, more, or different components. The sparsity balancing system 900 may be implemented within a DNN layer, before the layer's PE array.
[0061]
[0074] The activation storage unit 910 stores the activation functions and the output feature maps received from the previous layer. The activation rearrangement unit 920 rearranges the activation vectors of the activation functions based on the combined bitmap 980 to balance the sparsity of the activation vectors. The rearranged activation vectors have a better balanced sparsity than the original activation vectors. MUX MUX 960 receives the rearranged activation vector from the activation rearranger and receives one or more other activation vectors from the activation store. The other activation vectors are not rearranged. 960 is a combinational logic circuit designed to switch one of several input lines to a single common output line by application of a control signal. MUX 960 combines the rearranged activation vector with other activation vectors, for example, into a combined activation vector. The combined activation vector has a better balanced sparsity of the rearranged activation vector. The combined activation vector is sent to the activation distribution unit 925. The activation distribution unit 925 distributes the combined activation vector to the PEs. For example, the activation distribution unit 925 divides the combined activation vector into multiple sections and sends each section to a different PE. Since the combined activation vector has a better balanced sparsity, the different PEs have a better balanced workload, which can accelerate the deep learning process.
[0062]
[0075] MUX 950 receives the original weight vector (i.e., the unsorted weight vector) and the sorted weight vector (i.e., the weight vector sorted, for example, by software). The sorted weight vector has better balanced sparsity than the original weight vector. MUX 950 combines two weight vectors into one weight vector. MUX The weight vector output from 950 is stored in the weight storage unit 930.
[0063]
[0076] The weight rearrangement unit 940 rearranges the weight vectors stored in the weight storage unit 930 based on, for example, the combined bitmap 980, which further improves the sparsity balance in the weight vectors. The rearranged weight vectors and the weight vectors stored in the weight storage unit 930 but not rearranged by the weight rearrangement unit 940 are fed to the MUX 910. Input to 970. MUX 970 combines the two weight vectors into a combined weight vector. The combined weight vector is sent to the weight distributor 945. The weight distributor 945 distributes the combined weight vector to the PEs. For example, the weight distributor 945 splits the combined weight vector into multiple sections and sends each section to a different PE. Because the combined weight vector has improved sparsity balance, different PEs will have better balanced workloads, which accelerates the deep learning process.
[0064]
[0077] DL environment example 10 illustrates a DL environment 1000 according to various embodiments. The DL environment 1000 includes a DL server 1010 and a number of client devices 1020 (individually referred to as client devices 1020). The DL server 1010 is connected to the client devices 1020 via a network 1040. In other embodiments, the DL environment 1000 may include fewer, more, or different components.
[0065]
[0078] The DL Server 1010 trains DL models using neural networks. Neural networks are structured like the human brain and consist of artificial neurons, also known as nodes. These nodes are stacked next to each other in three types of layers: input layer, hidden layer(s), and output layer. Data provides information to each node in the form of inputs. The nodes multiply the inputs by random weights, calculate them, and add biases. Finally, a nonlinear function, also known as an activation function, is applied to determine which neurons to fire. The DL Server 1010 trains DL models using DNNs, RNNs (Recurrent Neural Networks), and other neural networks. Neural Network), GAN (Generative Adversarial Network), LSTMN (Long Short Term Memory Various forms of neural networks can be utilized, such as neural networks (DLNs), neural networks that use unknown elements in the input distribution to extract features, group objects, and discover useful data patterns during the process of training a DL model. DL models can be used to solve a variety of problems, such as making predictions, classifying images, etc. The DL server 1010 can build DL models that are specific to the particular type of problem that needs to be solved. DL models are trained to receive inputs and output solutions to a particular problem.
[0066]
[0079] In FIG. 10, DL server 1010 includes a DNN system 1050, a database 1060, and a distributor 1070. DNN system 1050 trains a DNN. The DNN can be used to process images, such as images captured by an autonomous vehicle, a medical device, a satellite, etc. In an embodiment, the DNN receives an input image and outputs a classification of an object in the input image. An example of a DNN is the DNN described above in connection with FIG. 100. The DNN system 1050 also compresses the trained DNN to reduce the size of the trained DNN. Since the compressed DNN has a smaller size, the application of the compressed DNN requires less time and computational resources (e.g., memory, processor, etc.) compared to the uncompressed DNN. The compressed DNN may be used in low memory sparsity balancing systems such as mobile phones, IOT edge devices, etc. The DNN system 1050 may also rearrange the weight vectors and activation vectors in the trained or compressed DNN to balance sparsity in the weight vectors and activation vectors. Further details regarding the DNN system 1050 are described below in conjunction with FIG. 11.
[0067]
[0080] The database 1060 stores data received, used, generated, or otherwise associated with the DL server 1010. For example, the database 1060 stores a training dataset that the DNN system 1050 uses to train a DNN. In an embodiment, the training dataset is an image gallery that can be used to train a DNN to classify images. The training dataset can include data received from the client device 1020. As another example, the database 1060 stores hyperparameters of a neural network constructed by the DL server 1010.
[0068]
[0081] The distributor 1070 distributes the DL model generated by the DL server 1010 to the client device 1020. In some embodiments, the distributor 1070 receives a request for a DNN from the client device 1020 over the network 1040. The request may include a description of a problem that the client device 1020 needs to solve. The request may also include information of the client device 1020, such as information describing computing resources available at the client device. The information describing computing resources available at the client device 1020 may be information indicative of network bandwidth, information indicative of available memory size, information indicative of processing capabilities of the client device 1020, etc. In embodiments, the distributor may instruct the DNN system 1050 to generate a DNN according to the request. The DNN system 1050 may generate a DNN based on the problem description. Alternatively or additionally, the DNN system 1050 may compress the DNN based on information describing computing resources available at the client device.
[0069]
[0082] In another embodiment, the distributor 1070 may select a DNN from among a group of existing DNNs based on the request. The distributor 1070 may select a DNN for a particular client device 1030 based on the size of the DNN and the available resources of the client device 1030. In an embodiment in which the distributor 1070 determines that the client device 1030 has limited memory or processing capabilities, the distributor 1070 may select a compressed DNN for the client device 1030 rather than an uncompressed DNN having a larger size. The distributor 1070 then communicates the generated or selected DNN for the client device 1020 to the client device 1020.
[0070]
[0083] In some embodiments, the distributor 1070 may receive feedback from the client device 1020. For example, the distributor 1070 may receive new training data from the client device 1020 and send the new training data to the DNN system 1050 for further training the DNN. As another example, the feedback includes an update of the computational resources available at the client device 1020. The distributor 1070 may send a different DNN to the client device 1020 based on the update. For example, after receiving feedback indicating that the computational resources of the client device 1020 have decreased, the distributor 1070 sends a smaller sized DNN to the client device 1020.
[0071]
[0084] The client device 1020 receives the DNN from the distributor 1070 and applies the DNN to solve a problem, such as to classify objects in an image. In various embodiments, the client device 1020 inputs images into the DNN and applies it to various applications, such as visual reconstruction. The output of the DNN is used for image reconstruction, augmented reality, robot localization and navigation, medical diagnosis, weather prediction, etc. The client device 1020 may be one or more computing devices capable of receiving user input and transmitting and / or receiving data over the network 1040. In one embodiment, the client device 1020 is a conventional computing device such as a desktop or laptop computer. Alternatively, the client device 1020 may be a device having computing capabilities such as a personal digital assistant (PDA), a mobile phone, a smartphone, an autonomous vehicle, or other suitable device. The client device 1020 is configured to communicate over the network 1040. In one embodiment, the client device 1020 executes an application that allows a user of the client device 1020 to interact with the DL server 1010 (e.g., a distributor 1070 of the DL server 1010). The client device 1020 may request a DNN or send feedback to the distributor 1070 via the application. For example, the client device 1020 executes a browser application to enable interaction between the client device 1020 and the DL server 1010 over the network 1040. In another embodiment, the client device 1020 interacts with the DL server 1010 via an application programming interface (API) that operates on a native operating sparsity balancing system of the client device 1020, such as IOS® or ANDROID®.
[0072]
[0085] In an embodiment, the client device 1020 is an integrated computing device that operates as a standalone network-enabled device. For example, the client device 1020 includes a display, a speaker, a microphone, a camera, and input devices. In another embodiment, the client device 1020 is a computing device that couples to an external media device, such as a television or other external display and / or audio output sparsity balancing system. In this embodiment, the client device 1020 may couple to the external media device via a wireless interface or a wired interface (e.g., an HDMI cable) and may utilize various features of the external media device, such as its display, speaker, microphone, camera, and input devices. Here, the client device 1020 may be configured to be compatible with a generic external media device that does not have dedicated software, firmware, or hardware specifically for interacting with the client device 1020.
[0073]
[0086] The network 1040 supports communication between the DL server 1010 and the client devices 1020. The network 1040 may include any combination of local area and / or wide area networks using both wired and / or wireless communication sparsity balancing systems. In one embodiment, the network 1040 may use standard communication technologies and / or protocols. For example, the network 1040 may use any of the following communication technologies and / or protocols: Ethernet, 8010.11, WiMAX (Worldwide Interconnect), and the like. Interoperability for microwave access), 3G, 4G, CDMA (code Division multiple Network 1040 may include communication links using technologies such as TLS, TLS, Digital Subscriber Line, and the like. Examples of networking protocols used to communicate over network 1040 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data communicated over network 1040 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 1040 may be encrypted using any suitable one or more techniques.
[0074]
[0087] Example of a DNN system FIG. 11 is a block diagram of a DNN system 1050 according to various embodiments. The DNN system 1050 trains a DNN. The DNN system 1050 can train a DNN that can be used to solve various problems such as image classification, learning relationships between biological cells (e.g., DNA, proteins, etc.), controlling behavior of devices (e.g., robots, machines, etc.), etc. The DNN system 1050 includes an interface module 1110, a training module 1120, a compression module 1130, a validation module 1140, a sparsity balancing module 1150, and an application module 1160. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 1050. Furthermore, functionality derived from components of the DNN system 1050 may be achieved by different components or different sparsity balancing systems included in the DNN system 1050.
[0075]
[0088] The interface module 1110 facilitates communication of the DNN system 1050 with other systems. For example, the interface module 1110 establishes communication between the DNN system 1050 and an external database to receive data that can be used to train or input the DNN to perform a task. As another example, the interface module 1110 supports the DNN system 1050 to distribute the DNN to other systems (e.g., computing devices configured to apply the DNN to perform a task).
[0076]
[0089] The training module 1120 trains the DNN with a training dataset. The training module 1120 forms a training dataset. In embodiments where the training module 1120 trains the DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in the training image. In some embodiments, a portion of the training dataset may be used to initially train the DNN, and the remaining training dataset may be retained as a tuning subset used by the compression module 1130 to tune the compressed DNN or as a validation subset used by the validation module 1140 to validate the performance of the trained or compressed DNN. The portions of the training dataset that do not include the tuning and validation subsets may be used to train the DNN.
[0077]
[0090] The training module 1120 also determines hyperparameters for training the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., filter weights). In some embodiments, hyperparameters include variables that determine the architecture of the DNN, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. The batch size determines the number of training samples to process before updating the parameters of the DNN. The batch size is equal to or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backward through the network. The number of epochs defines how many times the deep learning algorithm runs through the entire training dataset. An epoch means that each training sample in the training dataset has had a chance to update the parameters inside the DNN. An epoch can include one or more batches. The number of epochs may be 10, 100, 500, 1000, or more.
[0078]
[0091] The training module 1120 defines the architecture of the DNN, for example, based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN may include tensors (e.g., multidimensional arrays) that specify attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits that specify the color of the pixels in the input image). The output layer includes labels of the objects in the input layer. The hidden layer is a layer between the input layer and the output layer. The hidden layer includes one or more convolutional layers and one or more other types of layers, such as a rectified linear unit (ReLU) layer, a pooling layer, a fully connected layer, a normalization layer, a softmax or logistic layer, etc. The convolutional layer of the DNN abstracts the input image into feature maps, which are represented by tensors that specify the feature map height, the feature map width, and the feature map channels (e.g., a red, green, and blue image includes three channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. It is used between two convolutional layers. A fully connected layer contains weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories through training.
[0079]
[0092] The training module 1120 inputs the training dataset into the DNN and modifies parameters within the DNN to minimize the error between the generated labels and the training labels for the objects in the training images. The parameters include the weights of the filters in the convolutional layers of the DNN. In some embodiments, the training module 210 uses a cost function to minimize the error. After the training module 1120 finishes a predetermined number of epochs, the training module 1120 may stop updating the parameters within the DNN. The DNN with the updated parameters is referred to as a trained (or learned) DNN.
[0080]
[0093] The compression module 1130 compresses the trained DNN to reduce the complexity of the trained DNN at the expense of a small loss of model accuracy. The compression module 1130 converts all or a portion of the convolutional tensors in the trained DNN into reduced tensors that reduce the dimensionality of the corresponding convolutional tensors. The compression module 1130 then integrates the reduced tensors into the trained DNN to reduce the complexity of the trained DNN. In some embodiments, the compression module 1130 prunes a subset of filters in the convolutional layer to generate sparse tensors, and then decomposes the sparse tensors to generate reduced tensors for the convolutional layer. The compression module 1130 compresses the trained DNN by removing the convolutional tensors from the network and placing the reduced tensors into the network. After all or a portion of the convolutional tensors in the trained DNN are removed and the reduced tensors are integrated, a compressed DNN is generated. The compression module 1130 can fine-tune the compressed DNN. For example, the compression module 1130 trains the compressed DNN using a training dataset or a subset of the training dataset. Because the compressed DNN is transformed from a pre-trained DNN, the fine-tuning process is a retraining (or relearning) process.
[0081]
[0094] The validation module 1140 validates the accuracy of the trained or compacted DNN. In some embodiments, the validation module 1140 inputs samples in a validation dataset into the DNN and uses the output of the DNN to determine model accuracy. In some embodiments, the validation dataset may be formed from all or a portion of the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples other than the samples in the training set. In some embodiments, the validation module 1140 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 1140 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN) Here, precision may indicate how many of the total predictions (TP+FP or false positives) the reference classification model correctly predicts (TP or true positives), and recall may indicate how many of the total objects with the characteristic in question (TP+FN or false negatives) the reference classification model correctly predicts (TP). F-score = 2*PR / (P+R)) combines precision and recall into a single measure.
[0082]
[0095] The validation module 1140 may compare the accuracy score to a threshold score. In instances where the validation module 1140 determines that the accuracy score of the DNN (model) is lower than a threshold score, the validation module 1140 instructs the training module 1120 or the compression module 1130 to retrain the DNN. In one embodiment, the training module 1120 or the compression module 1130 may iteratively retrain the DNN until a stopping condition occurs, such as an accuracy measure indication that the DNN may be sufficiently accurate or the number of training rounds performed.
[0083]
[0096] In some embodiments, the validation module 1140 instructs the compression module 1130 to compress the DNN. For example, the validation module 1140 may determine whether the accuracy score of the compressed DNN exceeds a threshold score. In response to determining that the accuracy score of the compressed DNN exceeds the threshold score, the validation module 1140 instructs the compression module 1130 to further compress the DNN, for example, by compressing uncompressed convolutional layers in the DNN. In an embodiment, the validation module 1140 may determine a compression ratio based on the accuracy score and instructs the compression module 1130 to further compress the DNN based on the compression ratio. The compression ratio is, for example, a percentage indicating a reduced size from the compression of the DNN.
[0084]
[0097] The sparsity balancing module 1150 balances sparsity in the filters of a DNN layer. An embodiment of the sparsity balancing module 1150 is the sparsity balancing system 900 in FIG. 9. In some embodiments, the sparsity balancing module 1150 accesses a weight vector of a layer of a sequence of layers in the DNN. The weight vector includes a first sequence of weights. The weights have various values. For example, some weights have non-zero values, while some weights have zero values. In an embodiment, the weight vector represents a filter in the layer. A filter may be a matrix of weights that includes X rows and Y columns, where X and Y are integers. The weight vector may include Y sections, where each section is a row in the matrix. For example, the first row is a first section in the weight vector, the second row is a second section in the weight vector, and so on. In other embodiments, the weight vector may represent a portion of a filter, or may represent multiple filters of a layer. A layer may also contain activations, which may be represented by an activation vector that contains a sequence of activations.
[0085]
[0098] The sparsity balancing module 1150 generates a bitmap based at least on the weight vector. The bitmap includes a second sequence of bitmap elements. Each of the bitmap elements in the second sequence corresponds to a different weight in the first sequence. The value of the bitmap element is determined based at least on the value of the corresponding weight. Also, an index of the bitmap element in the second sequence matches an index of the corresponding weight in the first sequence. For example, for each weight having a non-zero value, the sparsity balancing module 1150 may select a bitmap element that is set to a value of one. valued bitmap For each weight that has a zero value, the sparsity balancing module 1150 generates a zero-valued bitmap element. valued bitmap element).
[0086]
[0099] In some embodiments, the sparsity balancing module 1150 generates a bitmap using both the weight vector and the activation vector. The sparsity balancing module 1150 generates a weight bitmap based on the weight vector. The weight bitmap includes a sequence of weight bitmap elements. The sparsity balancing module 1150 generates an activation bitmap based on the activation vector. The activation bitmap includes a sequence of activation bitmap elements. Each respective bitmap element in the bitmap corresponds to a different weight bitmap element in the weight bitmap and a different activation bitmap element in the activation bitmap. A bitmap element has a value equal to the product of the value of the corresponding weight bitmap element and the value of the corresponding activation bitmap element. In an embodiment, if a weight bitmap element has a value of one but an activation bitmap element has a value of zero, then the bitmap element has a value of zero.
[0087]
[0100] The sparsity balancing module 1150 generates a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements. The sparsity balancing module 1150 swaps the indices of the bitmap elements that are set to a value of 1 with the indices of the bitmap elements that are set to a value of 0. In some embodiments, the sparsity balancing module 1150 determines a first number of bitmap elements that are set to a value of 1. The sparsity balancing module 1150 also determines a second number of zero-valued bitmap elements. The sparsity balancing module 1150 determines the sum of the two numbers and identifies the smaller number of the two numbers. In instances where the first number is smaller than the second number (i.e., there are more bitmap elements that have a value of 1 than there are bitmap elements that have a value of 0), the sparsity balancing module 1150 divides the sum by the first number. The sparsity balancing module 1150 may also perform a floor operation to determine the largest integer that is less than or equal to the result of the division. The integer returned by the floor operation is used as the interval number. The sparsity balancing module 1150 rearranges the bitmap elements in the second sequence so that every interval number in the new bitmap contains at least one bitmap element with a value of one. In another example where the second number is less than the first number, the sparsity balancing module 1150 divides the sum by the second number and rearranges the bitmap elements in the second sequence so that every interval number in the new bitmap contains at least one bitmap element with a value of zero.
[0088]
[0101] The sparsity balancing module 1150 then assigns weight vectors to processing elements in the DNN network based on the new bitmap. Each processing element is configured to perform a multiply-add operation using a different subset of the weight vector. The sparsity balancing module 1150 may rearrange weights in the weight vector based on the new bitmap. For example, the sparsity balancing module 1150 moves weights based on an index of a bitmap element that corresponds to the weight in the new bitmap. The sparsity balancing module 1150 divides the reconstructed weight vector into subsets based on the interval number. Each subset may include at least the interval number of bitmap elements. Since a bitmap element with a value of 1 represents a non-zero value weight, the number of non-zero value weights in the subsets is the same or similar. Thus, the workload of the PEs is the same or similar.
[0089]
[0102] The sparsity balancing module 1150 can also rearrange the activation vectors based on the new bitmap so that the order of activations is consistent with the new order of weights. In some embodiments, the sparsity balancing module 1150 also rearranges the weight vectors of a previous layer, i.e., a different layer that precedes the layer in the DNN, to offset the rearrangement of the weight vectors of the layer. When the weight vectors of the previous layer are rearranged, the output of the previous layer that is the input to the layer will be changed. The changed input to the layer offsets the rearrangement of the weight vectors in the layer, so that the output of the layer can still be used as an input to the next layer in the DNN without modifying the next layer.
[0090]
[0103] The application module 1160 applies the trained or compressed DNN to perform a task. For example, the application module 1160 inputs an image into the DNN. The DNN outputs a classification of objects in the image. As an example, the DNN may be prepared in a security setting to detect malicious or dangerous objects in images captured by a security camera. As another example, the DNN may be prepared to detect objects (e.g., road signs, hazards, humans, pets, etc.) in images captured by a camera of an autonomous vehicle. The input to the DNN may be formatted according to a predefined input structure that reflects how the training dataset was provided to the DNN. The DNN may generate an output structure that may be, for example, a classification of the image, a list of detected objects, a boundary of the detected objects, etc. In some embodiments, the application module 1160 distributes the DNN to a computing device in communication with other sparsity balancing systems, such as the DNN system 1050, such that other sparsity balancing systems may apply the DNN to perform tasks.
[0104] Example of how to balance sparsity FIG. 12 is a flow chart illustrating a method 1200 for balancing sparsity in a weight vector according to various embodiments. The method 1200 may be performed by the sparsity balancing module 1150 of FIG. 11. Although the method 1200 is described with reference to the flow chart shown in FIG. 12, numerous other methods for balancing sparsity may alternatively be used. For example, the order of execution of the steps in FIG. 12 may be changed. As another example, some steps may be modified, eliminated, or combined.
[0091]
[0105] The sparsity balancing module 1150 obtains (1210) a weight vector that represents at least a portion of a weight matrix of a layer in a sequence of layers in the DNN. In some embodiments, the weight vector is a vector in a weight matrix of the DNN layer, or a combination of multiple vectors in a weight matrix. The weight matrix may be a portion or an entirety of a filter in the DNN layer. The weight vector includes a first sequence of weights. The weights have different values.
[0092]
[0106] The sparsity balancing module 1150 generates 1220 a bitmap based at least on the weight vector. The bitmap includes a second sequence of bitmap elements. Each of the bitmap elements in the second sequence corresponds to a different weight in the first sequence. The value of the bitmap element is determined based at least on the value of the corresponding weight. Also, an index of the bitmap element in the second sequence matches an index of the corresponding weight in the first sequence. For example, for each weight having a non-zero value, the sparsity balancing module 1150 generates a bitmap element having a value of one, and for zero-valued weights, generates a bitmap element having a value of zero. In some embodiments, the sparsity balancing module 1150 generates the bitmap using both the weight vector and the activation vector.
[0093]
[0107] The sparsity balancing module 1150 generates 1230 a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements. The sparsity balancing module 1150 reorders the bit elements in the bitmap to obtain balanced sparsity, i.e., an even (or nearly even) distribution of zero-value bit elements in the bitmap. The sparsity balancing module 1150 may rearrange the bitmap by using the process illustrated in FIG. 7.
[0094]
[0108] The sparsity balancing module 1150 then assigns (1240) weight vectors to a number of processing elements in a layer of the DNN based on the new bitmap. Each processing element is configured to perform a multiply-accumulate operation using a different subset of the weight vector. The sparsity balancing module 1150 may rearrange the weight vectors based on the order of the bitmaps in the new bitmap. For example, the sparsity balancing module 1150 moves weights based on the index of the bitmap element that corresponds to the weight in the new bitmap. Because the new bitmap has balanced sparsity, the rearranged weight vectors will also have balanced sparsity.
[0095]
[0109] The sparsity balancing module 1150 may also rearrange the activation vectors based on the new bitmap. In some embodiments, the sparsity balancing module 1150 also rearranges the weight vectors of a previous layer, i.e., a different layer that precedes a layer in the DNN, to offset the rearrangement of the weight vectors of a layer. When the weight vectors of a previous layer are rearranged, the output of the previous layer that is the input to the layer changes. The changed input to the layer offsets the rearrangement of the weight vectors in the layer, so that the output of the layer can still be used as an input to the next layer in the DNN without modifying the next layer.
[0096]
[0110] Example of computing device FIG. 13 is a block diagram of an exemplary computation sparsity balancing system for use as a DNN system 1050 according to various embodiments. Although numerous components are shown in FIG. 13 as being included in the computation sparsity balancing system 1300, any one or more of these components may be omitted or duplicated as appropriate depending on the application. In some embodiments, all or some of the components included in the computation sparsity balancing system 1300 may be mounted on one or more motherboards. In some embodiments, all or some of these components are fabricated on a single sparsity balancing system-on-chip (SoC) die. Additionally, in various embodiments, the computation sparsity balancing system 1300 may not include one or more of the components shown in FIG. 13, but the computation sparsity balancing system 1300 may include interface circuitry for coupling to one or more components. For example, the computational sparsity balancing system 1300 may not include a display device 1306, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which the display device 1306 may be coupled. In another set of examples, the computational sparsity balancing system 1300 may not include an audio input device 1318 or an audio output device 1308, but may include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which the audio input device 1318 or the audio output device 1308 may be coupled.
[0097]
[0111] The computational sparsity balancing system 1300 may include a processing device 1302 (e.g., one or more processing devices). As used herein, the term “processing device” or “processor” may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that may be stored in registers and / or memory. The processing device 1302 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices. The computational sparsity balancing system 1300 may include a memory 1304, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), flash memory, solid state memory, and / or hard drives. In some embodiments, memory 1304 may include memory sharing a die with processing device 1302. In some embodiments, memory 1304 includes one or more non-transitory computer-readable media that store instructions executable to perform operations to balance sparsity in a weight vector, such as operations performed by method 1200 described above in connection with FIG. 12 or sparsity balancing module 1150 described above in connection with FIG. 11. The instructions stored in the one or more non-transitory computer-readable media may be executed by processing device 1302.
[0098]
[0112] In some embodiments, the computational sparsity balancing system 1300 may include a communications chip 1312 (e.g., one or more communications chips). For example, the communications chip 1312 may be configured to manage wireless communications for the transfer of data to and from the computational sparsity balancing system 1300. The term "wireless" and its derivatives refer to a nonsolid medium. The term may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., capable of communicating data by using the use of modulated electromagnetic radiation over a wireless (wireless) medium. The term does not imply that the associated devices do not include any wiring, although in some embodiments they may not.
[0099]
[0113] The communication chip 1312 is a Wi-Fi (IEEE Institute of Electrical and Electronics Engineers (IEEE) standards, including the IEEE 802.13 family 802.16 standards (e.g., IEEE IEEE 802.16-2005 (amendment), the Long Term Evolution (LTE) project, together with any amendments, updates, and / or revisions (e.g., the Advanced LTE project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). 802.16-enabled Broadband Wireless Access (BWA) networks are commonly referred to as WiMAX networks, which are a “Worldwide Interoperability for Microwave Access” and IEEE This is a certification mark for products that have passed the 802.16 standard compliance and interoperability test. Sparsity balancing system for Mobile Communication), GPRS (General Packet Radio Service), UMTS (Universal Mobile Telecommunications Sparsity balancing system), HSPA (High Speed Packet The communication chip 1312 may operate according to the Enhanced Access (EDGE) or LTE network. Data for GSM Evolution), UTRAN (Universal Terrestrial Radio Access Network), E-UTRAN (Evolved The communications chip 1312 may operate according to CDMA, TDMA (Time Division Multiple Access) or the like. Division Multiple Access ), DECT (Digital Enhanced Cordless Telecommunications), EV-DO(Evolution-Data The communications chip 1312 may operate according to the IEEE 802.11b, IEEE 802.11b, IEEE 802.11b, IEEE 802.11b Optimized) and derivatives thereof, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. The communications chip 1312 may operate according to other wireless protocols in other embodiments. The computational sparsity balancing system 1300 may include an antenna 1322 for facilitating wireless communications and / or for receiving other wireless communications (such as AM or FM radio transmissions).
[0100]
[0114] In some embodiments, the communications chip 1312 may manage wired communications, such as electrical, optical, or any other suitable communications protocol (e.g., Ethernet). As noted above, the communications chip 1312 may include multiple communications chips. For example, a first communications chip 1312 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, while a second communications chip 1312 may be dedicated to global positioning system (GPS), etc. indexing sparsity balancing The first communications chip 1312 may be dedicated to long-range wireless communications such as IEEE 802.11b, IEEE 802.11b, EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communications chip 1312 may be dedicated to wireless communications and the second communications chip 1312 may be dedicated to wired communications.
[0101]
[0115] The computational sparsity balancing system 1300 may include a battery / power circuitry 1314. The battery / power circuitry 1314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computational sparsity balancing system 1300 to an energy source separate from the computational sparsity balancing system 1300 (e.g., AC line power).
[0102]
[0116] The computational sparsity balancing system 1300 may include a display device 1306 (or corresponding interface circuitry as described above). The display device 1306 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0103]
[0117] The computational sparsity balancing system 1300 may include an audio output device 1308 (or corresponding interface circuitry as described above), which may include any device that generates an audible indicator, such as, for example, a speaker, a headset, or earphones.
[0104]
[0118] The computational sparsity balancing system 1300 may include an audio input device 1318 (or corresponding interface circuitry as described above), which may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital device (e.g., a device having a Musical Instrument Digital Interface (MIDI) output).
[0105]
[0119] The computational sparsity balancing system 1300 may include a GPS device 1316 (or corresponding interface circuitry, as described above), which may be in communication with a satellite-based sparsity balancing system and may receive the position of the computational sparsity balancing system 1300, as known in the art.
[0106]
[0120] The computational sparsity balancing system 1300 may include other output devices 1313 (or corresponding interface circuitry as described above). Examples of other information providing devices 1313 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.
[0107]
[0121] The computational sparsity balancing system 1300 may include other input devices 1320 (or corresponding interface circuitry as described above). Examples of other input devices 1320 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.
[0108]
[0122] The computation sparsity balancing system 1300 may have any desired form factor, such as a handheld or mobile computation sparsity balancing system (e.g., a cellular phone, a smartphone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultra-mobile personal computer, etc.), a desktop computation sparsity balancing system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computation sparsity balancing system. In some embodiments, the computation sparsity balancing system 1300 may be any other electronic device that processes data.
[0109]
[0123] Selected Examples The following paragraphs provide various examples of the embodiments disclosed herein.
[0110]
[0124] Example 1 provides a method for balancing sparsity in deep neural networks (DNNs), the method comprising: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vectors.
[0111]
[0125] Example 2 provides the method of example 1, wherein the layer is associated with an activation vector including a third sequence of activations, the method further comprising: rearranging activations within the third sequence based on the new bitmap.
[0112]
[0126] Example 3 provides the method of example 1, further comprising: generating a bitmap based on at least the weight vector; generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap;
[0113]
[0127] Example 4 provides a method in example 3, wherein each bitmap element in the bitmap corresponds to a different weight bitmap element in the weight bitmap and a different activation bitmap element in the activation bitmap, and each bitmap element has a value equal to the product of a value of the corresponding weight bitmap element and a value of the corresponding activation bitmap element.
[0114]
[0128] Example 5 provides the method of example 1, wherein the DNN further includes a preceding layer preceding the layer in the sequence, the preceding layer being associated with a third sequence of weights, and the method further comprises: rearranging weights in the third sequence based on the new bitmap.
[0115]
[0129] Example 6 provides the method of example 1, wherein the first sequence of weights includes non-zeroed weights and zeroed weights; Bitmap elements corresponding to said non-zeroed weights have a value of 1; Bitmap elements that correspond to the zeroed weights have a value of 0.
[0116]
[0130] Example 7 provides the method of example 6, further comprising the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements: toggling between indices of bitmap elements that have been set to a value of 1 and indices of bitmap elements that have been set to a value of 0.
[0117]
[0131] Example 8 provides the method of example 6, further comprising the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being greater than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence so that every interval number in the new bitmap includes at least one bitmap element having a value of 1;
[0118]
[0132] Example 9 provides a method in Example 6, further comprising the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being less than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence such that every interval number in the new bitmap includes at least one bitmap element having a value of zero.
[0119]
[0133] Example 10 provides the method of example 1, further comprising: assigning the weight vector to a plurality of processing elements in a layer of the DNN based on the new bitmap: generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap.
[0120]
[0134] Example 11 provides one or more non-transitory computer-readable storage media storing instructions executable to perform operations for balancing sparsity in a deep neural network (DNN), the operations including: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vectors.
[0121]
[0135] Example 12 provides the one or more non-transitory computer-readable medium of Example 11, wherein the layer is associated with an activation vector including a third sequence of activations, and the operations further include: rearranging activations within the third sequence based on the new bitmap.
[0122]
[0136] Example 13 provides one or more non-transitory computer-readable mediums according to example 11, further comprising: generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap;
[0123]
[0137] Example 14 provides one or more non-transitory computer-readable media according to Example 13, wherein each bitmap element in the bitmap corresponds to a different weight bitmap element in the weight bitmap and a different activation bitmap element in the activation bitmap, and each bitmap element has a value equal to the product of a value of the corresponding weight bitmap element and a value of the corresponding activation bitmap element.
[0124]
[0138] Example 15 provides one or more non-transitory computer-readable media for the one or more embodiments of the present invention, wherein the DNN further includes a preceding layer preceding the layer in the sequence, the preceding layer being associated with a third sequence of weights, and the operations further include: rearranging weights in the third sequence based on the new bitmap.
[0125]
[0139] Example 16 provides one or more non-transitory computer-readable media for implementing the method of example 11, wherein the first sequence of weights includes non-zeroed weights and zeroed weights; Bitmap elements corresponding to said non-zeroed weights have a value of 1; Bitmap elements that correspond to the zeroed weights have a value of 0.
[0126]
[0140] Example 17 provides one or more non-transitory computer-readable mediums according to example 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements includes: toggling between indices of bitmap elements that have been set to a value of 1 and indices of bitmap elements that have been set to a value of 0.
[0127]
[0141] Example 18 provides the one or more non-transitory computer-readable medium of example 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements includes: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being greater than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence such that every interval number bitmap element in the new bitmap includes at least one bitmap element having a value of 1.
[0128]
[0142] Example 19 provides the one or more non-transitory computer-readable medium of example 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements includes: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being less than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence such that all interval number bitmap elements in the new bitmap include at least one bitmap element having a value of zero.
[0129]
[0143] Example 20 provides the one or more non-transitory computer-readable medium of example 11, further comprising: assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap; generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap.
[0130]
[0144] Example 21 provides an apparatus for balancing sparsity in a deep neural network (DNN), the apparatus comprising: a computer processor that executes computer program instructions; and a non-transitory computer readable memory that stores computer program instructions executable by the computer processor to perform operations; the operations including: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vectors.
[0131]
[0145] Example 22 provides the apparatus of example 21, wherein the layer is associated with an activation vector including a third sequence of activations, and the operations further include: rearranging activations within the third sequence based on the new bitmap.
[0132]
[0146] Example 23 provides the apparatus of example 21, further comprising: generating a bitmap based on at least the weight vector; generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap;
[0133]
[0147] Example 24 provides the apparatus of example 21, wherein the DNN further includes a preceding layer preceding the layer in the sequence, the preceding layer associated with a third sequence of weights, and the operations further include: rearranging weights in the third sequence based on the new bitmap.
[0134]
[0148] Example 25 provides the apparatus of example 21, further comprising: assigning the weight vector to a plurality of processing elements in a layer of the DNN based on the new bitmap; generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap.
[0135]
[0149] The above description of exemplary implementations of the disclosure, including those described in the Abstract, are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples relating to, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the art will recognize. These modifications may be made to the disclosure in light of the above detailed description.
Claims
1. A method for balancing sparsity in a deep neural network (DNN), comprising: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, and an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vector; The method includes:
2. 10. The method of claim 1 , wherein the layer is associated with an activation vector comprising a third sequence of activations, the method further comprising: rearranging activations within the third sequence based on the new bitmap; The method includes:
3. 2. The method of claim 1, wherein generating a bitmap based on at least the weight vector comprises: generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap; A method comprising:
4. 4. The method of claim 3, wherein each bitmap element in the bitmap corresponds to a different weight bitmap element in the weight bitmap and a different activation bitmap element in the activation bitmap, and each bitmap element has a value equal to the product of a value of the corresponding weight bitmap element and a value of the corresponding activation bitmap element.
5. 10. The method of claim 1, wherein the DNN further includes a previous layer preceding the layer in the sequence, the previous layer being associated with a third sequence of weights, the method further comprising: rearranging weights in the third sequence based on the new bitmap; The method includes:
6. 13. The method of claim 1 : the first sequence of weights includes non-zero value weights and zero value weights; a bitmap element corresponding to the non-zero value weight has a value of 1; A method according to claim 1, wherein bitmap elements corresponding to said zero-value weights have a value of 0.
7. 7. The method of claim 6, wherein the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: swapping the indices of the bitmap elements that have been set to a value of 1 with the indices of the bitmap elements that have been set to a value of 0; A method comprising:
8. 7. The method of claim 6, wherein the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being greater than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence such that all interval number bitmap elements in the new bitmap include bitmap elements that have been set to a value of 1; A method comprising:
9. 7. The method of claim 6, wherein the step of rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being less than the first number; determining a number of intervals based on the first number and the second number; and rearrange the bitmap elements in the second sequence such that all interval number bitmap elements in the new bitmap include bitmap elements that have been zeroed out; A method comprising:
10. 10. The method according to claim 1, wherein the step of assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap comprises: generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap; A method comprising:
11. One or more non-transitory computer-readable storage media storing instructions executable to perform operations for balancing sparsity in a deep neural network (DNN), the operations including: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, and an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vector; A storage medium including
12. 12. The one or more non-transitory computer-readable storage media of claim 11, wherein the layer is associated with an activation vector that includes a third sequence of activations, and the operations further include: rearranging activations within the third sequence based on the new bitmap; A storage medium including
13. 12. The one or more non-transitory computer-readable storage media of claim 11, wherein generating a bitmap based on at least the weight vector comprises: generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap; A storage medium including
14. 14. The non-transitory computer-readable storage medium of claim 13, wherein each bitmap element in the bitmap corresponds to a different weight bitmap element in the weight bitmap and a different activation bitmap element in the activation bitmap, and each bitmap element has a value equal to a product of a value of a corresponding weight bitmap element and a value of a corresponding activation bitmap element.
15. 12. The one or more non-transitory computer-readable storage media of claim 11, wherein the DNN further includes a previous layer that precedes the layer in the sequence, the previous layer being associated with a third sequence of weights, and the operations further include: rearranging weights in the third sequence based on the new bitmap; A storage medium including
16. 12. The one or more non-transitory computer-readable storage media of claim 11, the first sequence of weights includes non-zero value weights and zero value weights; a bitmap element corresponding to the non-zero value weight has a value of 1; A storage medium, wherein bitmap elements corresponding to the zero-value weights have a value of 0.
17. 17. The one or more non-transitory computer-readable storage media of claim 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: swapping the indices of the bitmap elements that have been set to a value of 1 with the indices of the bitmap elements that have been set to a value of 0; A storage medium including
18. 17. The one or more non-transitory computer-readable storage media of claim 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being greater than the first number; determining a number of intervals based on the first number and the second number; and rearranging the bitmap elements in the second sequence such that every interval number bitmap element in the new bitmap includes at least one bitmap element having a value of 1; A storage medium including
19. 17. The one or more non-transitory computer-readable storage media of claim 16, wherein rearranging the bitmap elements in the second sequence based on the values of the bitmap elements comprises: determining a first number of bitmap elements that have been set to a value of one; determining a second number of zeroed bitmap elements, the second number being less than the first number; determining a number of intervals based on the first number and the second number; and rearranging the bitmap elements in the second sequence such that every interval number bitmap element in the new bitmap includes at least one bitmap element having a value of zero; A storage medium including
20. 20. The one or more non-transitory computer-readable storage media of claim 11, wherein the step of assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap comprises: generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap; A storage medium including
21. An apparatus for balancing sparsity in a deep neural network (DNN), comprising: a computer processor for executing computer program instructions; and a non-transitory computer readable memory storing computer program instructions executable by said computer processor to perform operations; the operations including: obtaining a weight vector representing at least a portion of a weight matrix of a layer in a layer sequence in the DNN, the weight vector including a first sequence of weights having distinct values; generating a bitmap based at least on the weight vector, the bitmap comprising a second sequence of bitmap elements, each of the bitmap elements in the second sequence corresponding to a different weight in the first sequence, a value of a bitmap element being determined based at least on a value of its corresponding weight, and an index of a bitmap element in the second sequence corresponding to an index of a corresponding weight in the first sequence; generating a new bitmap from the bitmap by rearranging the bitmap elements in the second sequence based on the values of the bitmap elements; and assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap, each processing element configured to perform a multiply-accumulate operation using a different subset of the weight vector; 13. An apparatus comprising:
22. 22. The apparatus of claim 21, wherein the layer is associated with an activation vector that includes a third sequence of activations, and the operations further include: rearranging activations within the third sequence based on the new bitmap; 13. An apparatus comprising:
23. 22. The apparatus of claim 21, wherein generating a bitmap based on at least the weight vector comprises: generating a weight bitmap based on the weight vector, the weight bitmap comprising a sequence of weight bitmap elements; generating an activation bitmap based on activation associated with the layer, the activation bitmap comprising a sequence of activation bitmap elements; and generating the bitmap based on the weight bitmap and the activation bitmap; 13. An apparatus comprising:
24. 22. The apparatus of claim 21, wherein the DNN further includes a previous layer preceding the layer in the sequence, the previous layer being associated with a third sequence of weights, and the operations further include: rearranging weights in the third sequence based on the new bitmap; 13. An apparatus comprising:
25. 25. The apparatus according to claim 21, wherein the step of assigning the weight vectors to a plurality of processing elements in a layer of the DNN based on the new bitmap comprises: generating a new weight vector by rearranging one or more weights in the first sequence such that an index of each weight in the new weight vector matches an index of a corresponding bitmap element in the new bitmap; 13. An apparatus comprising:
Citation Information
Patent Citations
Exploiting input data sparsity in neural network computation units
JP2020500365A
