Activation and weighting using programmable threshold pruning neural networks

Abstract

Activation (e.g., output activation) or weights of an intermediate layer of a deep neural network (DNN) can be pruned to increase sparsity and reduce the amount of computation required for performing computations in a layer or subsequent layer. The pruning threshold may be determined, for example, by an iterative process, and the activation or weight whose absolute value is below the pruning threshold is changed to zero. The output tensor or kernel of the layer may be pruned using a first pruning threshold. A loss in accuracy of DNN due to pruning can be determined. A second pruning threshold may be determined based on the first pruning threshold and the loss of accuracy. The deep neural network may be modified by adding a pruning operation to the layers. The pruning operation can prune the output tensor or kernel of the layer based on a second pruning threshold.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 453,715, filed August 22, 2023, entitled “PRUNING ACTIVATIONS AND WEIGHTSOF NEURAL NETWORKS WITH PROGRAMMABLE THRESHOLDS”, which in turn claims priority to U.S. Provisional Patent Application No. 63 / 515,903, filed July 27, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to deep neural networks (DNNs), and more specifically, to the activation and weights of DNNs using programmable threshold pruning. Background Technology

[0004] Deep neural networks (DNNs) are widely used in various artificial intelligence applications, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of enormous computational costs. DNNs have extremely high computational requirements, as each inference requires hundreds of millions of MAC (multiply-add) operations as well as massive amounts of data read and write operations. DNN inference also requires the computation of activation functions. Therefore, techniques to improve the efficiency of DNNs are needed. Attached Figure Description

[0005] Various embodiments will be readily understood from the following detailed description and in conjunction with the accompanying drawings. For ease of description, similar reference numerals denote similar structural elements. In the accompanying figures, embodiments are illustrated by way of example rather than limitation.

[0006] Figure 1 The illustration shows an example DNN according to various embodiments.

[0007] Figure 2 The illustration shows example convolutions according to various embodiments.

[0008] Figure 3 This is a block diagram of a DNN system according to various embodiments.

[0009] Figure 4 The illustration shows pruning operations performed by a sparse encoder according to various embodiments.

[0010] Figure 5 The illustration shows sparsity acceleration performed by the processing element (PE) in MAC operations according to various embodiments.

[0011] Figure 6 This is a block diagram of a DNN module according to various embodiments.

[0012] Figure 7 This is a block diagram of a compression module according to various embodiments.

[0013] Figure 8 The illustration shows the selection of the optimal threshold for pruning activation according to various embodiments.

[0014] Figure 9 The illustration shows the selection of the optimal threshold for pruning weights according to various embodiments.

[0015] Figure 10 The illustration shows the selection of optimal thresholds for pruning activation and pruning weights according to various embodiments.

[0016] Figure 11 An example PE array according to various embodiments is illustrated.

[0017] Figure 12 This is a block diagram of a PE according to various embodiments.

[0018] Figure 13 This is a flowchart illustrating a method for modifying a DNN according to various embodiments.

[0019] Figure 14 This is a flowchart illustrating another method for modifying a DNN according to various embodiments.

[0020] Figure 15 This is a block diagram of an example computing device according to various embodiments. Detailed Implementation

[0021] Overview

[0022] The past decade has witnessed the rapid development of data processing based on artificial intelligence (AI), particularly deep neural networks (DNNs). DNNs are widely used in computer vision, speech recognition, image and video processing primarily due to their ability to achieve accuracy surpassing human levels. Significant improvements in DNN model size and accuracy, along with the rapid growth in the computing power of execution platforms, have enabled DNN applications to be used even on resource-constrained and energy-limited mobile and edge devices.

[0023] A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. Deep learning operations in a DNN can be performed on one or more intrinsic parameters (e.g., weights) and one or more activations of the DNN, which are determined during the training phase. Activations can be data points (also known as "data elements" or "elements"). The activations or weights of a DNN layer can be elements of a DNN layer tensor. A tensor is a data structure with multiple elements in one or more dimensions. Example tensors include vectors and matrices; vectors are one-dimensional tensors, and matrices are two-dimensional tensors. Three-dimensional tensors, and even higher-dimensional tensors, are also possible. A DNN layer can have an input tensor (also known as an "input feature map (IFM)") that includes one or more input activations and a weight tensor that includes one or more weights. Weights are elements in the weight tensor. The weight tensor for convolution can be a kernel, a filter, or a set of filters. The output data of a DNN layer can be an output tensor (also known as an "input feature map (OFM)") that includes one or more output activations (also known as "output elements").

[0024] DNNs exhibit sparsity in the form of activations and weights because many of these data elements can have zero values. These zero values ​​do not participate in the accumulation of partial sums during MAC operations. They lead to sparsity in the activations of subsequent layers after passing through nonlinear activation functions such as the Modified Linear Activation Function (ReLU). Leveraging sparsity in DNN accelerators is crucial for achieving efficient and scalable AI systems. By utilizing sparsity, DNN accelerators can reduce the computational and memory accesses required for a given task, resulting in faster and more energy-efficient inference. Sparsity also enables the deployment of larger and more accurate models without requiring more expensive hardware.

[0025] Pruning is one of the techniques currently available for introducing sparsity into DNNs. Regularization techniques are used to prune weights. For example, L1 regularization (also known as Lasso regularization) penalizes the sum of the absolute values ​​of weights, which causes some weights to become zero, resulting in a sparse model. Another weight pruning method reduces the size of the DNN by removing some connections or neurons that are less important to network performance. One example is amplitude-based pruning, where connections with smaller absolute values ​​are removed. Furthermore, various methods for pruning activations have been explored, including random activation pruning, converting it to a finite field vector through nonlinear dimensionality reduction, and combining L1-regularization and Hoyer regularization to enhance activation sparsity. Additionally, thresholding techniques have been introduced to further enhance sparsity.

[0026] However, currently available sparse injection techniques have some drawbacks. For example, these techniques can lead to reduced accuracy, especially when properly retrained with optimal hyperparameter settings using the original training dataset. This is because connections between neurons in the network are removed, resulting in information loss and performance degradation. Sparse injection also increases network complexity, as additional computation is required to determine which connections to remove and which to retain. This leads to longer training times and more complex code. Furthermore, these sparse injection techniques can be difficult to implement. Emerging neural networks such as Transformers impose limitations on these methods due to their inherent network architecture and non-ReLU-based activations (such as Gaussian Error Linear Unit (GELU), SoftMax, Swish, and Sigmoid).

[0027] Embodiments of this disclosure provide methods and systems for programmable threshold-based pruning of activations and weights. Example DNNs can be modified after training by adding pruning operations to one or more layers in the DNN. These layers can be intermediate layers. These layers can be selected based on one or more properties related to the computational complexity of the layers, such as the number of intrinsic parameters (e.g., weights) in the layer, the number of operations (e.g., MAC operations) in the layer, the layer type, etc. Pruning operations can introduce additional sparsity on top of the existing sparsity of the layers. This additional sparsity can reduce the computational complexity of these layers.

[0028] In various embodiments of this disclosure, the pruning operation can be an activation pruning operation or a weight pruning operation. Activation pruning can prune activations, such as those generated by a layer. Weight pruning can prune the weights of a layer. In a pruning operation, one or more activations or weights with an absolute value less than a threshold can be modified to zero, while activations or weights with an absolute value greater than or equal to the threshold are left unchanged. The threshold can be programmable. The threshold can be determined before inference execution, for example, during the compilation phase. Programmable thresholds can be specific to a particular layer or a particular type of layer.

[0029] In some embodiments, a layer may have more than one activation threshold or more than one weight threshold. For example, a layer may have one activation threshold for pruning activations with positive values ​​and a different activation threshold for pruning activations with negative values. The activation threshold for negative activations can be either positive or negative. The activation pruning operation compares the absolute value of the activation threshold with the absolute value of the activations to which that activation threshold applies. Similarly, a layer may have one weight threshold for pruning weights with positive values ​​and a different weight threshold for pruning weights with negative values. The weight pruning operation compares the absolute value of the weight threshold with the absolute value of the weights to which that weight threshold applies.

[0030] The optimal threshold for pruning operations can be searched through an iterative process involving multiple rounds of search. In each iteration, the dataset is fed into the DNN, and different thresholds are applied to the pruning operations added to one or more layers of the DNN. The accuracy loss of the DNN caused by the pruning operations can be measured and compared with an accuracy loss constraint. When the accuracy loss is lower than the accuracy loss constraint, the search can continue, and a higher threshold can be used in the next round. When the accuracy loss is not lower than the accuracy loss constraint, the threshold used in the previous round can be selected as the optimal threshold. The DNN can be modified by pruning operations using the optimal threshold. The modified DNN can then be deployed to perform AI tasks.

[0031] Pruning operations can be performed during the compilation phase or using existing sparsity hardware in DNN accelerators with limited or no additional overhead in terms of area and power. Pruning introduces sparsity into the input activation or weight data before it is stored in the DNN accelerator's memory, thereby reducing memory usage and computational cost, and improving the efficiency of the DNN accelerator.

[0032] This disclosure provides a method to enhance the sparsity of various network layers without incurring costly retraining processes. It can target non-linear activation functions (including ReLU) as well as activation functions widely used in convolutional neural networks and Transformers (including Sigmoid, GELU, and SoftMax). This method optimizes sparsity after normalization layers (e.g., LayerNorm) in Transformers, thereby improving efficiency in multilayer perceptrons (MLPs) and multi-head attention layers. Since the programmable threshold is determined based on accuracy loss constraints, it minimizes the impact on DNN accuracy while maximizing energy savings.

[0033] For ease of explanation, specific figures, materials, and configurations are listed to provide a thorough understanding of the exemplary embodiments. However, those skilled in the art will understand that this disclosure may be practiced without specific details, or may be practiced using only some of the described aspects. In other instances, well-known features have been omitted or simplified to avoid obscuring the exemplary embodiments.

[0034] Furthermore, reference has been made to the accompanying drawings, which form part of this document, in which illustrative embodiments that may be implemented are shown. It should be understood that other embodiments may be employed, and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be considered limiting.

[0035] Various operations can be described as a series of discrete actions or operations performed sequentially in a manner most conducive to understanding the claimed subject matter. However, the order in which they are described should not be construed as implying that these operations are necessarily sequential. Specifically, these operations may not be performed in the order presented. The described operations may be performed in a different order than the described embodiments. In other embodiments, various other operations may be performed, or the described operations may be omitted.

[0036] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" refers to (A), (B), or (A and B). For the purposes of this disclosure, "A, B, or C" or the phrase "A, B, and / or C" refers to (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used to describe a measurement range, the term "between" includes both ends of the measurement range.

[0037] The specification uses the phrases "in one embodiment" or "in multiple embodiments," which can refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," etc., used in the embodiments of this disclosure are synonyms. This disclosure may use view-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are merely for ease of discussion and do not imply desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise stated, the ordinal numbers "first," "second," "third," etc., are used to describe the same object only to indicate different instances of the same kind of object and are not intended to imply that the described objects must be arranged in a given order in time, space, hierarchy, or any other respect.

[0038] In the following detailed description, terms commonly used by those skilled in the art will be used to describe various aspects of the exemplary embodiments in order to convey the substance of the work to others skilled in the art.

[0039] The terms “substantially,” “near,” “approximately,” “around,” and “about” generally refer to a target value within ±20% of the value described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to a target value within ±5–20% of the value described herein or known in the art.

[0040] Furthermore, the terms “comprising,” “including,” “containing,” “having,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements and may also include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Additionally, the term “or” refers to an inclusive “or” rather than an exclusive “or.”

[0041] The systems, methods, and apparatus disclosed herein have several innovative aspects, but no single aspect can alone achieve all the desirable characteristics disclosed herein. The following description and accompanying drawings illustrate details of one or more specific embodiments of the subject matter described herein.

[0042] Example DNN

[0043] Figure 1 An example DNN 100 according to various embodiments is illustrated. For ease of illustration, Figure 1 DNN 100 in this example is a Convolutional Neural Network (CNN). In other embodiments, DNN 100 can be other types of DNNs. DNN 100 is trained to receive images and output classifications of objects in the images. Figure 1 In one embodiment, DNN 100 receives an input image 105, which includes objects 115, 125, and 135. DNN 100 includes a series of layers, including multiple convolutional layers 110 (collectively referred to as "convolutional layer 110"), multiple pooling layers 120 (collectively referred to as "pooling layer 120"), and multiple fully connected layers 130 (collectively referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. During inference in DNN 100, the layers of DNN 100 perform tensor computations including a variety of tensor operations, such as convolution (e.g., multiply-add (MAC) operations, pooling operations, and element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combinations thereof.

[0044] Convolutional layer 110 summarizes the presence of features in the input image 105. Convolutional layer 110 acts as a feature extractor. The first layer of DNN 100 is convolutional layer 110. In the example, convolutional layer 110 performs convolution on the input tensor 140 (also called IFM 140) and filter 150. Figure 1As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes three input channels, each represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also called input points) in each row and 7 input elements in each column. The filter 150 is represented by a 3×3×3 three-dimensional matrix. The filter 150 includes three kernels, each corresponding to a different input channel of the IFM 140. The kernel is a two-dimensional weight matrix, where the weights are arranged by column and row. The kernel can be smaller than the IFM. Figure 1 In the illustrated embodiment, each kernel is represented by a 3×3 two-dimensional matrix. The 3×3 kernel includes 3 weights in each row and 3 weights in each column. The weights can be initialized and updated using backpropagation with gradient descent. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0045] Convolution involves performing a MAC operation on the input elements in IFM 140 and the weights in filter 150. The convolution can be either a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides over IFM 140. All input channels are combined to generate an output tensor 160 (also called an output feature map (OFM) 160). OFM 160 is represented by a 5×5 two-dimensional matrix. The 5×5 two-dimensional matrix includes 5 output elements (also called output points) in each row and 5 output elements in each column. For ease of illustration, Figure 1 The standard convolution in the embodiments includes a filter. In embodiments with multiple filters, the standard convolution can produce multiple output channels in OFM 160.

[0046] Multiplication between kernel-size patches of IFM 140 and the kernel can be a dot product. A dot product is an element-wise multiplication of an IFM 140 kernel-size patch with its corresponding kernel, followed by summing the results, always yielding a single value. Because the result is a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (weight set) to be multiplied by IFM 140 multiple times at different points in the IFM 140. Specifically, the kernel is systematically applied from left to right and top to bottom to each overlapping portion or kernel-size patch of IFM 140. The result of multiplying a kernel by IFM 140 once is a single value. Because the kernel is applied to IFM 140 multiple times, the result of the multiplication is a two-dimensional output element matrix. Therefore, the two-dimensional output matrix from a standard convolution 163 (i.e., OFM 160) is called OFM.

[0047] In depthwise convolution 183, the input channels are not combined. Instead, the MAC operation is performed on a single input channel and a single kernel, producing an output channel. For example... Figure 1 As shown, depthwise convolution 183 produces a depth output tensor 180. The depthwise output tensor 180 is represented by a 5×5×3 three-dimensional matrix. The depthwise output tensor 180 includes three output channels, each represented by a 5×5 two-dimensional matrix. The 5×5 two-dimensional matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is the result of a MAC operation between the input channel of IFM 140 and the kernel of filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation between the first input channel (dot pattern) and the first kernel (dot pattern), the second output channel (horizontal stripe pattern) is the result of a MAC operation between the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern), and the third output channel (diagonal stripe pattern) is the result of a MAC operation between the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In this depthwise convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input channel and the output channel are collectively referred to as the depth-wise channel. After the depth-wise convolution, pointwise convolution 193 is performed on the depth-wise output tensor 180 and the 1×1×3 tensor 190 to generate OFM 160.

[0048] Then, OFM 160 is passed to the next layer in the sequence. In some embodiments, OFM 160 is passed through an activation function. An example activation function is ReLU. ReLU is calculated as follows: it directly returns the value provided as input, or returns zero if the input is zero or less. Convolutional layer 110 can receive multiple images as input and computes a convolution of each image with each kernel. This process can be repeated multiple times. For example, OFM 160 is passed to subsequent convolutional layers 110 (i.e., convolutional layer 110 after the one that generates OFM 160 in the sequence). Subsequent convolutional layers 110 perform convolution on OFM 160 with new kernels and generate new feature maps. The new feature maps can also be normalized and resized. The new feature maps can be kernelized again by further subsequent convolutional layers 110, and so on.

[0049] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size F (e.g., the kernel size is F×F×D pixels), the stride S of dragging the window corresponding to the kernel across the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a P-pixel-thick black outline to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolutions, such as 2D convolution, dilated convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include different numbers of convolutional layers.

[0050] Pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing features present in the patch of the feature map. Pooling layer 120 is located between two convolutional layers 110: a preceding convolutional layer 110 (the convolutional layer 110 preceding pooling layer 120 in the layer sequence) and a subsequent convolutional layer 110 (the convolutional layer 110 following pooling layer 120 in the layer sequence). In some embodiments, pooling layer 120 is added after convolutional layer 110, for example, after applying an activation function (e.g., ReLU, etc.) to OFM 160.

[0051] Pooling layer 120 receives the feature map generated by the previous convolutional layer 110 and applies a pooling operation to the feature map. Pooling reduces the size of the feature map while preserving its important characteristics. Therefore, pooling improves the efficiency of the DNN and avoids overlearning. Pooling layer 120 can perform pooling operations using average pooling (calculating the average value of each patch on the feature map), max pooling (calculating the maximum value of each patch on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied with a stride of 2 pixels (2×2 pixels), such that the pooling operation reduces the size of the feature map by a factor of 2, for example, reducing the number of pixels or values ​​in the feature map to one-quarter of its original size. For example, applying pooling layer 120 to a 6×6 feature map yields a 3×3 output pooled feature map. The output of pooling layer 120 is fed into subsequent convolutional layers 110 for further feature extraction. In some embodiments, pooling layer 120 operates on each feature map individually to create a set of pooled feature maps of the same number.

[0052] Fully connected layer 130 is the last layer of the DNN. Fully connected layer 130 may or may not be a convolutional layer. Fully connected layer 130 receives input operands. The input operands define the outputs of convolutional layer 110 and pooling layer 120, and include the values ​​of the last feature map generated by the last pooling layer 120 in the sequence. Fully connected layer 130 applies a linear combination and activation function to the input operands and generates a vector. The vector can contain as many elements as the class: element i represents the probability that the image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the last fully connected layer 130 using either a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function.

[0053] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1In this embodiment, N equals 3 because there are 3 objects 115, 125, and 135 in the input image. Each element of the operand represents the probability that the input image 105 belongs to a certain class. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, sums them, and then applies an activation function (e.g., a logistic function if N = 2; a SoftMax function if N > 2). This is equivalent to multiplying the input operand by a matrix containing the weights. In this example, the vector includes 3 probabilities: a first probability that object 115 is a tree, a second probability that object 125 is a car, and a third probability that object 135 is a person. In other embodiments, the values ​​may differ when the input image 105 includes different objects or different numbers of objects.

[0054] Example Convolution

[0055] Figure 2 The illustration shows example convolutions performed according to various embodiments. Convolution can be a deep learning operation within the convolutional layers of a DNN, such as... Figure 1 The convolutional layer 110 is used. Convolution can be performed on the input tensor 210 and the filter 220 (referred to separately as "filter 220"). The result of the convolution is the output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. Examples of DNN accelerators could be... Figure 3 The DNN accelerator 302 in the middle.

[0056] exist Figure 2 In this embodiment, the input tensor 210 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. Input elements are data points in the input tensor 210. The spatial size of the input tensor 210 is H. in ×W in ×C in H in W is the height of the 3D matrix (i.e., the length along the Y-axis, which represents the number of activations in each column of the 3D matrix for each input channel). in It is the width of the 3D matrix (i.e., the length along the X-axis, which represents the number of activations per row in the 2D matrix for each input channel), and C in This is the depth of the 3D matrix (i.e., the length along the Z-axis, which represents the number of input channels). For simplicity and illustration, the spatial size of input tensor 210 is 7×7×3, meaning input tensor 210 includes three input channels, each with a 7×7 two-dimensional matrix. Each input element in input tensor 210 can be represented by (X,Y,Z) coordinates. In other embodiments, the height, width, or depth of input tensor 210 may be different.

[0057] Each filter 220 includes weights arranged in a 3D matrix. The values ​​of the weights can be determined by training a DNN. The spatial size of the filter 220 is H. f ×W f ×C f H f It is the height of the filter (i.e., the length along the Y-axis, which represents the weight of each column in each kernel), W f It is the width of the filter (i.e., the length along the X-axis, which represents the number of weights per row in each kernel), C f This is the filter depth (i.e., the length along the Z-axis, which represents the number of channels). In some embodiments, C f Equals C in For the sake of simplicity and explanation, Figure 2 Each filter 220 in the input tensor has a spatial size of 2×3×3, meaning that filter 220 includes two convolutional kernels with a spatial size of 2×3. In other embodiments, the height, width, or depth of filter 220 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0058] An activation or weight can occupy one or more bytes of memory. The number of bytes used for activation or weight can depend on the data format. For example, when the activation or weight is in INT8 format, the activation occupies one byte. When the activation or weight is in FP16 format, the activation or weight occupies two bytes. Activation or weight can use other data formats.

[0059] In the convolution, each filter 220 slides on the input tensor 210 and generates a 2D matrix for the output channel in the output tensor 230. Figure 2 In this embodiment, the spatial size of the 2D matrix is ​​5×5. The output tensor 230 includes activations (also referred to as "output activations," "elements," or "output elements") arranged in a 3D matrix. Output activations are data points in the output tensor 230. The spatial size of the output tensor 230 is H. out ×W out ×C out H out W is the height of the 3D matrix (i.e., the length along the Y-axis, which represents the number of output activations per column in the 2D matrix for each output channel). out It is the width of the 3D matrix (i.e., the length along the X-axis, which represents the number of output activations per row in the 2D matrix for each output channel), C out It represents the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of output channels). C out This can be equal to the number of filters 220 in the convolution. H out and W outIt can depend on the height and weights of the input tensor 210 and each filter 220.

[0060] As part of the convolution, the input tensor 210 and the 2×3×3 sub-tensor 215 in each filter 220 (in... Figure 2 (Highlighted by dashed lines) The MAC operation is performed. The result of the MAC operation on subtensor 215 and filter 220 is the output activation. In some embodiments (e.g., embodiments where the convolution is integral convolution), the output activation may include 8 bits, for example, one byte. In other embodiments (e.g., embodiments where the convolution is floating-point convolution), the output activation may include more than one byte. For example, the output element may include two bytes.

[0061] After completing the MAC operation on subtensor 215 and all filters 220, vector 235 is generated. Vector 235 is then... Figure 2 Highlighted by a diagonal line. Vector 235 comprises a series of output activations arranged along the Z-axis. The output activations in vector 235 have the same (X,Y) coordinates, but each output activation corresponds to a different output channel and has a different Z-coordinate. The dimension of vector 235 along the Z-axis can be equal to the total number of output channels in output tensor 230. After generating vector 235, further MAC operations are performed to generate other vectors until output tensor 230 is generated.

[0062] In some embodiments, the MAC operation performed on the 2×3×3 subtensor (e.g., subtensor 215) and filter 220 can be performed by multiple PEs. One or more PEs can receive input operands (e.g., Figure 2 The input operands 217 and weight operands (e.g., shown) are shown. Figure 2 The weight operand 227 is shown. Input operand 217 includes a series of activations with the same (x,y) coordinates but different z-coordinates. Input operand 217 includes activations from each input channel in input tensor 210. Weight operand 227 includes a series of weights with the same (x,y) coordinates but different z-coordinates. Weight operand 227 includes weights from each channel in filter 220. The activations in input operand 217 and the weights in weight operand 227 can be sequentially input into the PE. The PE can receive one activation and one weight (“activation-weight pair”) at a time and multiply the activations and weights. The positions of the activations in input operand 217 can be matched with the positions of the weights in weight operand 227. Activations and weights can correspond to the same channel.

[0063] The activation or weight can be a floating-point number. Floating-point numbers can have various data formats, such as FP32, FP16, BF16, etc. Floating-point numbers can be positive or negative with a decimal point. A floating-point number can be represented by a series of bits, including one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing the exponent of the floating-point number, and bits representing the mantissa of the floating-point number. The mantissa is the part of the floating-point number that represents the significant digits of the number. Multiplying the mantissa by the exponent of the base gives the actual value of the floating-point number.

[0064] In some embodiments, the output activations in output tensor 230 may be further processed based on one or more activation functions before being stored or input to the next layer of the DNN. Processing based on one or more activation functions may be at least part of convolutional post-processing. In some embodiments, post-processing may include one or more other computations, such as offset computation, bias computation, etc. The result of post-processing may be stored in the local memory of the computation block and may be used as input to the next DNN layer. In some embodiments, the input activations in input tensor 210 may be the result of post-processing of a previous DNN layer.

[0065] Example DNN system

[0066] Figure 3 This is a block diagram of a DNN system 300 designed according to various embodiments. The entire DNN system 300 or a portion thereof can be implemented in one or more computing devices, such as... Figure 15 The computing device 1500 in the middle. The DNN system 300 can generate and execute DNNs, such as Figure 1 DNN 100 in the example. Figure 3 As shown, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, the DNN system 300 may include alternative configurations, different or additional components. For example, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality of the components of the DNN system 300 may be implemented by different components or different systems included in the DNN system 300. In some embodiments, the DNN module 301 and the DNN accelerator 302 may include different types of processing units. The DNN module 301 and the DNN accelerator 302 may be implemented on the same chip or on separate chips.

[0067] DNN module 301 facilitates the generation and deployment of DNNs. In some embodiments, DNN module 301 can generate and train DNNs. For example, DNN module 301 can define the hierarchical architecture of the DNN. DNN module 301 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 301 can also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is the sparsity rate, which defines the sparsity of one or more deep learning tensors used for the DNN.

[0068] DNN module 301 can also compress the DNN, for example, after it has been trained. DNN module 301 can reduce the size of the DNN by adding pruning operations to one or more layers of the DNN. DNN module 301 can select one or more layers based on the computational complexity of the layers. Pruning operations can, for example, prune activations generated by layers or prune weights in layers by modifying non-zero value activations or weights to zero. These zeros can be skipped from memory storage and computation so that DNN inference will consume less memory usage, power, and time. In some embodiments, DNN module 301 can determine a threshold for the pruning operation. During the pruning operation, the absolute value of a data element (e.g., activation or weight) can be compared to the threshold; data elements with an absolute value less than the threshold can be changed to zero, while data elements with an absolute value greater than or equal to the threshold can remain unchanged. Pruning operations can be performed, for example, by DNN accelerator 302 during inference execution. Alternatively, pruning operations (e.g., weight pruning operations) can be performed before inference execution. For example, weight pruning operations can be performed during the compilation phase after the DNN has been trained.

[0069] DNN module 301 can deploy trained, compressed, or validated DNNs for deep learning applications. In some embodiments, DNN module 301 can distribute trained, compressed, or validated DNNs to devices or systems that can use the DNNs to perform tasks trained on the DNNs (e.g., image classification, motion planning, etc.). In other embodiments, DNN module 301 can use DNN accelerator 302 to facilitate DNN deployment. For example, DNN module 301 can receive data from a device or system coupled to DNN system 300 and input the received data (or data generated by DNN module 301, for example, based on the received data) into the DNN. DNN module 301 can generate instructions (e.g., configuration files) to control the operation of DNN accelerator 302 during DNN inference. DNN module 301 can receive the output of the DNN from DNN accelerator 302. DNN module 301 can transmit the output of the DNN (or the result of processing the DNN output by DNN module 301) to a device or system. Figure 6 Provides certain aspects of the DNN module 301.

[0070] DNN accelerator 302 executes the DNN provided by DNN module 301. For example, DNN accelerator 302 can perform DNN inference, for example, by running deep learning operations in the DNN to train the DNN, or by using a trained / compressed / validated DNN to perform a task. Figure 3 As shown, the DNN accelerator 302 includes a memory 310, a direct memory access (DMA) engine 320, and a compute block 330 (referred to separately as "compute block 330"). In other embodiments, the DNN accelerator 302 may include alternative configurations, different, or additional components. For example, the DNN accelerator 302 may include more than one memory 310 or DMA engine 320. As another example, the DNN accelerator 302 may include a single compute block 330. Furthermore, the functionality of the components of the DNN accelerator 302 may be implemented by different components or different systems included in the DNN accelerator 302. The components of the DNN accelerator 302 may be implemented in hardware, software, firmware, or a combination thereof.

[0071] Memory 310 stores data associated with deep learning operations performed by the DNN accelerator. In some embodiments, memory 310 may store data to be used by computation block 330 for DNN inference. For example, memory 310 may store weights, such as the weights of convolutional layers, which are determined by training the DNN. Memory 310 may also store weight thresholds used for pruning weights. For example, memory 310 may store activation thresholds, such as activation thresholds used to prune activations generated by intermediate layers of the DNN. Memory 310 may also store data generated by computation block 330 performing deep learning operations in the DNN. Example deep learning operations include convolution (also known as a “convolution operation”), pooling operations, element-wise operations, activation functions, other types of deep learning operations, or some combination thereof. Memory 310 may be the main memory of the DNN accelerator 302. In some embodiments, memory 310 includes one or more dynamic random access memories (DRAM).

[0072] DMA engine 320 facilitates data transfer between memory 310 and the local memory of computation block 330. For example, DMA engine 320 can read data from memory 310 and write the data to the local memory of computation block 330. As another example, DMA engine 320 can read data from the local memory of computation block 330 and write the data to memory 310. DMA engine 320 provides DMA functionality that allows computation block 330 to initiate data transfers between memory 310 and its local memory, and to perform other operations while the data transfers are in progress. In some embodiments, DMA engine 320 can read a tensor from memory 310, modify the tensor in a manner optimized for computation block 330, and then write the tensor to the local memory of computation block 330.

[0073] Computation block 330 can perform deep learning operations within a DNN. For example, computation block 330 can run deep learning operations in a DNN layer at once, or run a portion of such deep learning operations. Computation block 330 can be capable of running various types of deep learning operations, such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. In the example, computation block 330 can perform convolution, such as standard convolution or depthwise convolution. In some embodiments, computation block 330 receives an input tensor and one or more convolution kernels, and performs convolution using the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by computation block 330 or another computation block 330. In some embodiments, the operations of a DNN layer can be run in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 can each perform a portion of the convolution workload. Data can be shared between computation blocks 330. Computation block 330 can also be referred to as a computation tile. In some embodiments, each computation block 330 can be a processing unit.

[0074] exist Figure 3 In one embodiment, each computation block 330 includes a local memory 340, a PE array 350, a sparsity accelerator 360, a post-processing unit 370, and a sparsity encoder 380. Some or all of the components of the computation block 330 can be implemented on the same chip. In other embodiments, the computation block 330 may include alternative configurations, different or additional components. Furthermore, the functionality of the components of the computation block 330 can be implemented by different components included in the computation block 330, different computation blocks 330, another component of the DNN accelerator 302, or different systems. The components of the computation block 330 can be implemented in hardware, software, firmware, or a combination thereof.

[0075] Local memory 340 is the local memory of the corresponding computation block 330. Figure 3In one embodiment, local memory 340 is internal to computation block 330. In other embodiments, local memory 340 may be external to computation block 330. Local memory 340 may store data received, used, or generated by PE array 350 and post-processing unit 370. Examples of data may include input activations, weights, output activations, sparsity bitmaps, etc. Local memory 340 may also include activation thresholds or weight thresholds for pruning activations or weights. Data in local memory 340 may be transferred to or from memory 310, for example, via DMA engine 320. In some embodiments, data in local memory 340 may be transferred to or from the local memory of another computation block 330.

[0076] In some embodiments, local memory 340 includes one or more static random access memories (SRAMs). Local memory 340 may be byte-addressable, with each memory address identifying a single byte (eight bits) of storage space. In some embodiments, local memory 340 may include a bank of memory. The number of banks of memory in local memory 340 may be 16, 64, 128, 356, 512, 1024, 3048, or other numbers. A bank of memory may include multiple memory cells. In an example, a bank of memory may include 8, 16, 64, or other numbers of memory cells. A bank of memory or a memory cell within a bank may have a memory address. In an example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses (i.e., adjacent memory cells). For example, a memory cell may store an integer in INT8 format, while storing a number in FP16 or BF16 format (16 bits) may require two memory cells. In some embodiments, 16 bits may be transferred from local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 340 over multiple read cycles (such as two cycles).

[0077] PE array 350 may include PEs arranged in columns, or PEs arranged in both columns and rows. Each PE may perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more accumulators (“adders”) for performing accumulation. A column of PEs is called a PE column. A PE column may be associated with one or more MAC channels. A MAC channel is a path used to load data into a MAC column. A MAC channel may also be referred to as a data transfer channel or a data loading channel. A PE column may have multiple MAC channels. The loading bandwidth of a MAC column is the sum of the loading bandwidths of all MAC channels associated with that MAC column. Data can be simultaneously input into the same number of independent PEs through a certain number of MAC channels. In some embodiments, when a MAC column has four MAC channels for inputting activations or weights into that MAC column, and each MAC channel has a bandwidth of 16 bytes, the four MAC channels may have a total loading bandwidth of 64 bytes.

[0078] In some embodiments, the PE array 350 can perform depthwise convolution, standard convolution, or a combination of both. In depthwise convolution, the PE can perform a MAC operation, which includes a series of multiplications over input operands and weight operands. Each multiplication in the sequence (also called a period) multiplies different activations in the input operands with different weights in the weight operands. Activations and weights within the same period may correspond to the same channel. This series of multiplications produces a product operand comprising a series of products. The MAC operation can also include accumulation, where multiple product operands are accumulated to generate the output operand of the PE. The PE array 350 can output multiple output operands at once, each generated by a different PE. In standard convolution, the MAC operation can include accumulation across channels. For example, instead of generating output operands, the PE can accumulate products across different channels to generate a single output point.

[0079] In some embodiments, the PE array 350 can perform MAC operations in quantized inference, such as MAC operations in quantized convolution. In some embodiments, the PEs in the PE array 350 can receive quantized activations and quantized weights, and compute a quantized MAC result. The quantized MAC result can be a quantized value in integer format and can be the output of the PE. In some embodiments, the PE can also include a quantization multiplier that can multiply the quantization scale by the quantized MAC result, and the output of the PE can be a real value in floating-point format. The PE may not include a quantization subtractor because MAC operations in quantized inference do not require zero-point offset.

[0080] The sparsity accelerator 360 accelerates computation in the PE array 350 based on the sparsity of activations or weights. In some embodiments (e.g., an embodiment where computation block 330 performs convolutional layers), computation in the PE can be a MAC operation on input operands and weight operands. Input operands can include one or more activations in the input tensor of the convolution. Different activations can be in different input channels. Weight operands can include one or more weights in the filters of the convolution. The values ​​of the weights are determined by training a DNN. The weights in the weight operands can be in different input channels.

[0081] In some embodiments, the input operand is associated with an activation bitmap, which may be stored in local memory 340. The activation bitmap may be generated by a sparse encoder 380. The activation bitmap may indicate the location of non-zero activations in the input operand. The activation bitmap may include multiple bits, each corresponding to a respective activation in the input operand. The position of a bit in the activation bitmap may match the position of the corresponding activation in the input operand. The bits in the activation bitmap may be 0 or 1. A 0 value bit indicates that the value of the corresponding activation is 0, and a 1 value bit indicates that the value of the corresponding activation is not 0. In some embodiments, the activation bitmap may be generated during the execution of another DNN layer (e.g., a layer preceding a convolutional layer in the DNN).

[0082] In some embodiments, weight operands are associated with a weight bitmap, which may be stored in local memory 340. The weight bitmap may be generated by a sparsity encoder 380. The weight bitmap may indicate the positions of non-zero weights in the weight operands. The weight bitmap may include multiple bits, each corresponding to a respective weight in the weight operand. The positions of bits in the weight bitmap may match the positions of the corresponding weights in the weight operands. Bits in the weight bitmap may be 0 or 1. A 0 value bit indicates that the value of the corresponding weight is 0, and a 1 value bit indicates that the value of the corresponding weight is not 0.

[0083] In some embodiments, the sparsity accelerator 360 may receive an activation bitmap and a weight bitmap and generate a combined sparse bitmap for a MAC operation to be performed by the PE. In some embodiments, the sparsity accelerator 360 generates the combined sparse bitmap 735 by performing one or more AND operations on the activation bitmap and the weight bitmap. Each bit in the combined sparse bitmap is the result of an AND operation between a bit in the activation bitmap and a bit in the weight bitmap, i.e., the product of a bit in the activation bitmap and a bit in the weight bitmap. The positions of the bits in the combined sparse bitmap match the positions of the bits in the activation bitmap and the positions of the bits in the weight bitmap. The bits in the combined bitmap correspond to a pair of activations and weights (activation-weight pairs). A 0 bit in the combined sparse bitmap indicates that at least one of the activations and weights in the pair is 0. A 1 bit in the combined sparse bitmap indicates that neither the activation nor the weight in the pair is 0. The combined sparse bitmap may be stored in local memory 340.

[0084] The sparsity accelerator 360 can provide activations and weights to the PE based on a combined sparse bitmap. For example, the sparsity accelerator 360 can identify one or more non-zero activation-weight pairs from local memory 340 based on the combined sparse bitmap. Local memory 340 can store input operands and weight operands in a compressed format to store non-zero activations and non-zero weights, but not zero activations and zero weights. The non-zero activations(s) of the input operands can constitute the compressed input operands. The non-zero weights(s) of the weight operands can constitute the compressed weight operands. For a non-zero activation-weight pair, the sparsity accelerator 360 can determine the position of the activation in the compressed input operand and the position of the weight in the compressed weight operand based on the activation bitmap, weight bitmap, and combined bitmap. The activations and weights can be read from local memory 340 based on the positions determined by the sparsity accelerator 360.

[0085] In some embodiments, the sparsity accelerator 360 includes sparsity acceleration logic capable of computing a position bitmap based on an activation bitmap and a weight bitmap. The sparsity accelerator 360 can determine the position indices of activations and weights based on the position bitmap. In an example, the position index of an activation in a compressed input operand can be equal to the number of 1s in the activation position bitmap generated by the sparsity accelerator 360, and the position index of a weight in a compressed weight operand can be equal to the number of 1s in the weight position bitmap generated by the sparsity accelerator 360. The position index of an activation or weight indicates the position of the activation or weight in the compressed input operand or compressed weight operand. The sparsity accelerator 360 can read activations and weights from one or more memories based on their position indices.

[0086] The sparsity accelerator 360 can forward identified non-zero activation-weight pairs to the PE. The sparsity accelerator 360 can skip other activations and weights because they do not affect the result of the MAC operation. In some embodiments, local memory 340 can store non-zero activations and weights, but not zero-value activations or weights. Non-zero activations and weights can be loaded into one or more register files of the PE, from which the sparsity accelerator 360 can retrieve activations and weights corresponding to the activations and weights in the combined sparsity bitmap. In some embodiments, the total number of 1s in the combined sparsity bitmap is equal to the total number of activation-weight pairs to be computed by the PE, and the PE does not compute other activation-weight pairs. By skipping activation-weight pairs corresponding to zero bits in the combined sparsity bitmap, the PE's computation is faster compared to computeing all activation-weight pairs in the input operands and weight operands.

[0087] The sparse accelerator 360 can be implemented in hardware, software, firmware, or a combination thereof. In some embodiments, at least a portion of the sparse accelerator 360 may be located inside the PE. Although Figure 3 A single sparsity accelerator 360 is shown, but computation block 330 may include multiple sparsity accelerators 360. In some embodiments, each PE in PE array 350 is implemented using a sparsity accelerator 360 to accelerate computation and reduce the power consumption of a single PE. In other embodiments, a subset of PE array 350 (e.g., one or more columns of PEs in PE array 350) may be implemented using a sparsity accelerator 360 to accelerate computation within that subset of PEs. Further details regarding sparsity acceleration will be provided below. Figure 5 supply.

[0088] Post-processing unit 370 processes the output of PE array 350. In some embodiments, post-processing unit 370 computes activation functions. Post-processing unit 370 may receive the output of PE array 350 as input to the activation functions. Post-processing unit 370 may transfer the output of the activation functions to local memory 340. PE array 350 may later retrieve the output of the activation functions from local memory 340 for further computation. For example, post-processing unit 370 may receive the output tensor of a DNN layer from PE array 350 and compute one or more activation functions on that output tensor. The result computed by post-processing unit 370 may be stored in local memory 340 and subsequently used as the input tensor of the next DNN layer. In addition to or in lieu of activation functions, post-processing unit 370 may perform other types of post-processing on the output of PE array 350. For example, post-processing unit 370 may apply a bias to the output of PE array 350.

[0089] The sparse encoder 380 transforms dense data into compressed data based on the sparsity of the dense data. The sparse encoder 380 can perform pruning operations, such as activation pruning or weight pruning. The sparse encoder 380 can also generate sparse bitmaps based on the pruning operations, including activation bitmaps and weight bitmaps.

[0090] In some embodiments, the sparse encoder 380 may receive the output tensor of the layer (e.g., Figure 2 The output tensor 230 in the output tensor. The sparsity encoder 380 can generate a compressed version of the output tensor. In some embodiments, the sparsity encoder 380 can compress the output tensor based on an activation threshold. The sparsity encoder 380 can compare each activation to the activation threshold. When the absolute value of an activation is less than the activation threshold, the sparsity encoder 380 can output zero; but when the absolute value of an activation is greater than or equal to the activation threshold, the sparsity encoder 380 can output the value of the activation. Non-zero value outputs of the sparsity encoder 380 (i.e., activations with absolute values ​​greater than or equal to the activation threshold) are stored in memory, such as local memory 340, and can be used in operations at the next layer. Other activations may not be stored. The sparsity encoder 380 can also generate one or more sparse bitmaps of the output tensor. The sparse bitmaps can correspond to vectors in the output tensor (e.g., ...). Figure 2 (vector 235 in the vector). The sparsity graph can include bits, each bit corresponding to a different activation in the vector and indicating whether the corresponding activation is zero.

[0091] In some embodiments, the sparse encoder 380 can compress the weight tensor. For example, the sparse encoder 380 can be based on the kernel of the weight threshold pruning layer. The sparse encoder 380 can compare each weight to a weight threshold. When the absolute value of a weight is less than the weight threshold, the sparse encoder 380 can output zero; but when the absolute value of a weight is greater than or equal to the weight threshold, the sparse encoder 380 can output the value of the weight. The non-zero value outputs of the sparse encoder 380 (i.e., weights whose absolute values ​​are greater than or equal to the weight threshold) are stored in memory, such as local memory 340, and can be used in the next layer of operation. Other weights may not be stored. The sparse encoder 380 can also generate one or more sparse bitmaps of the weight tensor. The sparse bitmaps can correspond to the weight operands in the weight tensor (e.g., ...). Figure 2 The sparse graph can include bits, each corresponding to a different weight in the weight operand and indicating whether the corresponding weight is zero.

[0092] In some embodiments, local memory 340 is associated with a load path, and data can be transferred within compute block 330 using an emission path. For example, data can be transferred from local memory 340 to PE array 350 via the load path. Data can also be transferred from PE array 350 to local memory 340 via an emission path. A sparsity encoder 380 may be arranged on the emission path to compress data before it is written to local memory 340.

[0093] Figure 4 Pruning operations performed by a sparse encoder 400 according to various embodiments are illustrated. The sparse encoder 400 may be... Figure 3 An embodiment of the sparse encoder 380 in [the example]. Figure 4 As shown, the sparse encoder 400 includes a comparator 410 (referred to solely as "comparator 410"), a threshold register 420, and a compression packer 430. In other embodiments, the sparse encoder 400 may include different, fewer, or more components. Furthermore, the functionality of the components of the sparse encoder 400 can be implemented by the different components of the sparse encoder 400.

[0094] For ease of explanation, Figure 4 The pruning operation shown is performed on a vector comprising 16 data elements P0-P15. An example of this vector could be computed within a layer of a DNN and could be the result of deep learning operations within that layer. Another example of this vector could be a portion of the weight tensor of a DNN layer. Each comparator 410 receives a data element and compares its absolute value to a threshold stored in a threshold register 420. The threshold can be predetermined by the DNN module 301. In some embodiments, the threshold can be zero. In other embodiments, the threshold can be a positive number. Comparator 410 outputs 16 data elements O0-O15 and 16 bits B0-B14. When the absolute value of a data element received by comparator 410 is below the threshold, comparator 410 can output a zero-value data element 0 and a 0-bit value. When the absolute value of a data element received by comparator 410 is not below the threshold, comparator 410 can output its received data element and a 1-bit value.

[0095] The compression packer 430 receives data elements O0-O15 and bits B0-B14 from the comparator 410. The compression packer 430 generates a new vector that includes the non-zero value output of the comparator 410. This new vector is a compressed version of the vector received by the sparse encoder 400. For ease of illustration, Figure 4 The new vector in the data structure consists of five data elements: P1, P5, P8, P12, and P14. Since it is not necessary to store and compute all 16 data elements, but only five data elements in the new vector need to be stored and computed, memory usage and power consumption can be reduced.

[0096] The compression packer 430 also generates a sparse bitmap consisting of 16 bits, B0-B14. Each bit in the sparse bitmap corresponds to one of 16 data elements P0-P15. For example, B0 corresponds to P0, B1 to P1, B2 to P2, and so on. Each bit can indicate whether the corresponding data element should be provided to the PE for computation or skipped from the computation. Figure 4 In this embodiment, a 0-bit value indicates that the corresponding data element is not provided to any PE for computation, while a 1-bit value indicates that the corresponding data element is to be provided to one or more PEs for computation. For example, bit B0 indicates that data element P0 is to be skipped from computation, while bit B1 indicates that data element P1 is to be provided to one or more PEs for computation. The new vector and sparsity bitmap can be stored in memory, such as local memory 340.

[0097] Figure 5 The illustration shows sparsity acceleration in MAC operations performed by PE 500 according to various embodiments. PE 500 can be a PE in PE array 440. Figure 5 In one embodiment, PE 500 includes an input register file 510, a weight register file 520, a multiplier 530, an accumulator 540, and an output register file 550. In other embodiments, PE 500 may include fewer, more, or different components. PE 500 is associated with a sparsity accelerator 560. The sparsity accelerator 560 may be... Figure 3 An embodiment of the medium sparsity accelerator 360.

[0098] Input register file 510 stores at least a portion of the activation operand. The activation operand comprises a series of input elements, also referred to as activations. The activation operand may be a portion of an input tensor (e.g., the input tensor of a convolutional layer). The activation operand is associated with activation bitmap 515. Activation bitmap 515 may be stored in input register file 510, in local memory including the computation block of PE500, or both. Activation bitmap 515 may indicate the location of non-zero activations in the activation operand. Activation bitmap 515 comprises a series of bits, each corresponding to a respective activation in the activation operand. In some embodiments, the position of a bit in activation bitmap 515 matches the position of a corresponding activation in the activation operand. For ease of illustration, activation bitmap 515 comprises 8 bits, and the activation operand comprises 8 activations. In other embodiments, activation bitmap 515 may comprise fewer or more bits. Figure 5 As shown, in the activation bitmap 515, 4 out of 8 bits are 0 values, and the remaining 4 bits are 1 values. A 0-value bit indicates that the corresponding activation value is 0, and a 1-value bit indicates that the corresponding activation value is not 0. Therefore, the activation operands include 4 zero-value activations and 4 non-zero-value activations.

[0099] Weight register file 520 stores at least a portion of weight operands. Weight operands comprise a series of weights. Weight operands may be part of a filter (e.g., a filter in a convolutional layer). Weight operands are associated with weight bitmap 525. Weight bitmap 525 may be stored in weight register file 520, in local memory including the computation block of PE 500, or both. Weight bitmap 525 may indicate the positions of non-zero weights in the weight operands. Weight bitmap 525 includes a series of bits, each corresponding to a respective weight in the weight operand. In some embodiments, the positions of the bits in weight bitmap 525 match the positions of the corresponding weights in the weight operand. For ease of illustration, weight bitmap 525 comprises 8 bits, and the weight operand comprises 8 weights. In other embodiments, weight bitmap 525 may include fewer or more bits. Figure 5 As shown, in the weight bitmap 525, 4 out of 8 bits are 0 values, and the remaining 4 bits are 1 values. A 0-value bit indicates that the corresponding weight has a value of 0, and a 1-value bit indicates that the corresponding weight has a non-zero value. Therefore, the weight operand includes 4 zero-value weights and 4 non-zero-value weights. The weight bitmap 525 can indicate the position of the non-zero-value weights in the weight operand.

[0100] The sparsity accelerator 560 generates a combined sparse bitmap 535 based on the activation bitmap 515 and the weight bitmap 525. The sparsity accelerator 560 may receive the activation bitmap 515 from the input register file 510 or from the local memory of the computation block including PE 500. The sparsity accelerator 560 may receive the weight bitmap 525 from the weight register file 520 or from the local memory of the computation block. In some embodiments, the sparsity accelerator 560 is an AND operator. The sparsity accelerator 560 can generate the combined sparse bitmap 535 by performing one or more AND operations on the activation bitmap 515 and the weight bitmap 525. Each bit in the combined sparse bitmap 535 is the result of an AND operation between a bit in the activation bitmap 515 and a bit in the weight bitmap 525. The positions of the bits in the combined sparse bitmap 535 match the positions of the bits in the activation bitmap 515 and the weight bitmap 525. For example, the first bit in the combined sparse bitmap 535 is the result of an AND operation between the first bit of the activation bitmap 515 and the first bit of the weight bitmap 525; the second bit in the combined sparse bitmap 535 is the result of an AND operation between the second bit of the activation bitmap 515 and the second bit of the weight bitmap 525; the third bit in the combined sparse bitmap 535 is the result of an AND operation between the third bit of the activation bitmap 515 and the third bit of the weight bitmap 525, and so on.

[0101] When the values ​​of the corresponding bits in the activation bitmap 515 and the corresponding bits in the weight bitmap 525 are both 1, the value of the bit in the combined sparse bitmap 535 is 1. When the value of at least one of the corresponding bits in the activation bitmap 515 and the corresponding bits in the weight bitmap 525 is 0, the value of the bit in the combined sparse bitmap 535 is 0. Figure 5 As shown, the combined sparsity bitmap 535 includes 6 zeros and 2 ones.

[0102] The total number of 1s in the combined sparse bitmap 535 is equal to the total number of non-zero activation-weight pairs that will be computed by the PE 500 to calculate the non-zero partial sum. Other activation-weight pairs are zero-value activation-weight pairs and can be skipped in computation without affecting output accuracy, as these pairs would result in a zero-value partial sum. Therefore, the workload of the PE 500 in this computation round can be determined based on the total number of 1s in the combined sparse bitmap 535. The computation time can also be estimated based on the total number of 1s in the combined sparse bitmap 535. The more 1s in the combined sparse bitmap 535, the higher the workload of the PE 500, and the longer the computation time of the PE 500.

[0103] In some embodiments, input register file 510 or weight register file 520 stores dense data points, such as non-zero activations or non-zero weights. Sparse data points (such as zero activations or zero weights) are not stored in input register file 510 or weight register file 520. Dense data points can be compressed and kept adjacent to each other in input register file 510 or weight register file 520. Dense data points of activation operand(s) constitute compressed activation operand(s). Dense data points of weight operand(s) constitute compressed weight operand(s). The position of 1 in combined sparsity bitmap 535 does not indicate the position of activation in compressed activation operand(s) or the position of weight in compressed weight operand(s). Sparsity accelerator 560 can perform sparsity computation to determine the position of activation in compressed activation operand(s) and the position of weight in compressed weight operand(s). Sparsity accelerator 560 can perform one round of sparsity computation for each of two non-zero activation-weight pairs. In each round of sparsity computation, the sparsity accelerator 560 can compute activation position bitmaps and weight position bitmaps based on activation bitmap 515, weight bitmap 525, and combined sparsity bitmap 535. The position of activations in compressed activation operands can be indicated by the number of 1s in the activation position bitmap, and the position of weights in compressed weight operands can be indicated by the number of 1s in the weight position bitmap. In the first round of sparsity computation, an intermediate bitmap can be determined, and this intermediate bitmap can be used in the second round to identify the next non-zero activation-weight pair.

[0104] The sparsity accelerator 560 can read the activations and weights of non-zero activation-weight pairs from the input register file 510 and the weight register file 520 based on the positions determined by sparsity computation, and provide the activations and weights to the multiplier 530. The multiplier 530 performs multiplication on the activations and weights. For example, the multiplier 530 performs multiplication on the activations and weights in each individual non-zero activation-weight pair and outputs a partial sum, i.e., the product of the activations and weights. Since there are two activation-weight pairs, the multiplier 530 can perform two multiplications sequentially, for example, based on the positions of 1s in the combined sparsity bitmap 535. Without sparsity acceleration, the multiplier 530 would require eight multiplications. By reducing the number of multiplications from eight to two, the MAC operation speed in the PE 500 is increased. Since DNN accelerators typically perform a large number of MAC operations when executing DNNs, sparsity acceleration can significantly improve the efficiency and performance of DNN accelerators.

[0105] Accumulator 540 receives two partial sums from multiplier 530 and accumulates these two partial sums. The result of the accumulation is an internal partial sum at the PE level. The PE-level internal partial sum can be stored in output register file 550. In some embodiments, accumulator 540 receives one or more PE-level internal partial sums from one or more other PEs. Accumulator 540 can accumulate one or more PE-level internal partial sums with the PE-level internal partial sum of PE 500 and store the accumulated result (i.e., multi-PE internal partial sum) in output register file 550. One or more other PEs can be located in the same column as PE 500 in the PE array. The multi-PE internal partial sum can be a column-level internal partial sum. In some embodiments, the PE-level internal partial sum or the multi-PE internal partial sum of PE 500 can be sent to one or more other PEs for further accumulation.

[0106] although Figure 5 A single multiplier 530 is shown, but PE 500 may include multiple multipliers capable of performing multiple multiplication operations simultaneously. These multipliers can be coupled to internal adder components, such as... Figure 12 The internal adder component 1240 in the middle.

[0107] Figure 6 This is a block diagram of a DNN module 600 according to various embodiments. The DNN module 600 can be... Figure 3 An example of the DNN module 301. (See example...) Figure 6As shown, the DNN module 600 includes an interface module 610, a training module 620, a compression module 630, a validation module 640, and a data storage 650. In other embodiments, the DNN module 600 may include alternative configurations, different or additional components. Furthermore, the functionality of the components of the DNN module 600 can be implemented by different components, modules, or systems included in the DNN module 600.

[0108] Interface module 610 facilitates communication between DNN module 600 and other modules or systems. For example, interface module 610 establishes communication between DNN module 600 and an external database to receive data that can be used to train the DNN or input into the DNN to perform tasks. As another example, interface module 610 supports DNN module 600 in distributing the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0109] Training module 620 trains the DNN using a training dataset. Training module 620 forms the training dataset. In embodiments where training module 620 trains the DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe the true classification of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a portion of the training dataset can be used for initial training of the DNN, while the remainder of the training dataset can be reserved as a validation subset for validation module 640 to validate the performance of the trained DNN. The portion of the training dataset excluding the tuning and validation subsets can be used to train the DNN.

[0110] The training module 620 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters differ from parameters internal to the DNN (e.g., filter weights). In some embodiments, hyperparameters include variables that determine the DNN architecture, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. Batch size defines the number of training samples to be processed before updating the DNN parameters. Batch size is equal to or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is forward and backward propagated throughout the network. The number of epochs defines how many times the deep learning algorithm works on the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the parameters internal to the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0111] The training module 620 defines the architecture of the DNN based on some hyperparameters. The DNN architecture includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN can include tensors specifying attributes of the input image (e.g., a multidimensional array), such as the height, width, and depth of the input image (e.g., specifying the number of bits in the pixel colors of the input image). The output layer includes labels for the objects in the input layer. Hidden layers are layers located between the input and output layers. Hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, SoftMax layers, or logical layers, etc. The convolutional layers of the DNN abstract the input image into a feature map, which is represented by a tensor specifying the feature map height, width, and channels (e.g., red, green, and blue images include 3 channels). Pooling layers are used to reduce the spatial volume of the input image after convolution. They are used between two convolutional layers. Fully connected layers involve weights, biases, and neurons. They connect neurons in one layer to neurons in another layer. It is used to classify images of different categories through training.

[0112] In defining the DNN architecture, the training module 620 also adds activation functions to the hidden or output layers. The activation function of a layer is a weighted sum of the layer's inputs, which is then converted into the layer's output. The activation function can be, for example, a modified linear unit activation function, a tangent activation function, or other types of activation functions.

[0113] After defining the architecture of the DNN in training module 620, training module 620 inputs a training dataset into the DNN. The training dataset includes multiple training samples. Examples of training samples include objects in images and the ground truth labels of those objects. Training module 620 modifies the parameters internal to the DNN (“DNN internal parameters”) to minimize the error between the training object labels generated by the DNN and the ground truth labels of the objects. Internal parameters include the weights of the filters in the convolutional layers of the DNN. In some embodiments, training module 620 uses a cost function to minimize the error.

[0114] The training module 620 can train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines how many times the deep learning algorithm will operate on the entire training dataset. An epoch means that each sample in the training dataset has an opportunity to update the internal parameters of the DNN. After the training module 620 completes the predetermined number of epochs, it can stop updating the parameters in the DNN. At this point, the DNN with updated parameters is called the trained DNN.

[0115] Compression module 630 compresses the DNN. For example, compression module 630 can add pruning operations to the DNN layers to reduce computational complexity or memory usage. The pruning operation can be an activation pruning operation or a weight pruning operation. Activation pruning can prune the output tensor of the DNN layer. Weight pruning can prune the weight tensor of the DNN layer. In some embodiments, compression module 630 can select one or more computationally complex layers in the DNN and modify each selected layer using activation pruning or weight pruning operations.

[0116] For pruning operations on layers or a class of layers, the compression module 630 can determine that the accuracy loss of the DNN will not exceed the activation threshold or weight threshold of the accuracy loss constraint. Activation pruning operations can modify the output activations of a layer whose absolute value is below the activation threshold to zero, while keeping the other output activations of that layer unchanged. Reducing output data can reduce memory traffic because zero-value activations can be skipped from memory storage. This also reduces the number of operations in the next layer, because zero-value activations can also be skipped from computation.

[0117] Weight pruning can modify weights with absolute values ​​greater than a weight threshold to zero, while leaving other weights unchanged. Weight pruning reduces memory storage because zero-value weights are not stored. Furthermore, because the calculation of zero-value weights can be skipped without affecting the layer's output, the number of operations within a layer can be reduced. In some embodiments, the compression module 700 can also measure the energy savings, final DNN accuracy, or layer-by-layer sparsity resulting from pruning operations. The following is in conjunction with... Figure 7 Provides certain aspects of the compression module 630.

[0118] After compressing the DNN, the compression module 630 can fine-tune the DNN, for example, through a retraining process. The compression module 630 can fine-tune the DNN after weight pruning. In some embodiments, the fine-tuning process is a retraining process or a further training process. For example, after the weights in the DNN are pruned, the compression module 630 can further train the DNN by inputting a training dataset into the DNN. The values ​​of the unpruned weights in the DNN can be modified based on the DNN's output and the true labels of the training samples in the training dataset. In some embodiments, the values ​​of the pruned weights (i.e., zero) do not change during the fine-tuning process. For example, the compression module 630 can place a mask on the pruned weight blocks to prevent the values ​​in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values ​​of all weights (including the pruned weights) can be changed during the fine-tuning process. After one or more cycles of retraining and weight changes by the compression module 630, the compression module 630 can perform a new pruning process, for example, by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process can be repeated multiple times before the fine-tuning process is complete.

[0119] In some embodiments, the number of epochs in the fine-tuning process may differ from the number of epochs in the training process, in which the pre-pruning values ​​for the weights are determined. For example, the fine-tuning process may have fewer epochs than the training process. In the examples, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, etc.

[0120] The validation module 640 validates the accuracy of the trained or compressed DNN. In some embodiments, the validation module 640 feeds samples from the validation dataset into the trained DNN and uses the output of the DNN to determine model accuracy. In some embodiments, the validation dataset may be formed from some or all of the samples in the training dataset. Alternatively or additionally, the validation dataset includes samples other than those in the training set. In some embodiments, the validation module 640 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 640 may use the following metrics to determine the accuracy score: Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where precision may be the proportion of the number of correctly predicted objects (TP or true positives) by the reference classification model to its total number of predictions (TP+FP or false positives), and recall may be the proportion of the number of correctly predicted objects (TP) by the reference classification model to the total number of objects with the attribute in question (TP+FN or false negatives). The F-score (F-score = 2*PR / (P+R)) combines precision and recall into a single metric.

[0121] The verification module 640 can compare the accuracy score with a threshold score. In an example where the verification module 640 determines that the accuracy score of the augmented model is below the threshold score, the verification module 640 instructs the training module 620 to retrain the DNN. In one embodiment, the training module 620 can iteratively retrain the DNN until a stopping condition is met, such as an accuracy measurement indicating that the accuracy of the DNN is sufficient, or that the required number of training epochs has been reached.

[0122] Data storage 650 stores data received, generated, used, or otherwise associated with DNN module 600. For example, data storage 650 stores datasets used by training module 620 and validation module 640. Data storage 650 may also store data generated by training module 620 and validation module 640, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Figure 6In one embodiment, data storage 650 is a component of DNN module 600. In other embodiments, data storage 650 may be located external to DNN module 600 and communicate with DNN module 600 via a network.

[0123] Figure 7 This is a block diagram of a compression module 700 according to various embodiments. The compression module 700 may be... Figure 6 An embodiment of the compression module 630 in [the text]. For example... Figure 7 As shown, the compression module 700 includes a layer selection module 710, a graph generation module 720, a modification module 730, an activation threshold module 740, and a weight threshold module 750. In other embodiments, the compression module 700 may include alternative configurations, different or additional components. Furthermore, the functionality of the components in the compression module 700 can be implemented by different components included in the compression module 700, the DNN module 600, or different modules or systems.

[0124] The layer selection module 710 analyzes the DNN and selects layers from the DNN for modification through pruning operations. In some embodiments, the layer selection module 710 may select one or more computationally complex layers from the DNN. Computationally complex layers may be computationally intensive or memory-intensive. Pruning the activation or weights of a layer can reduce the computational or memory resources required to execute that layer.

[0125] In some embodiments, the layer selection module 710 can select the layers to which activation pruning operations are to be added, and select different layers to which weight pruning operations are to be added. The layer selection module 710 can also select layers to which both activation pruning and weight pruning operations are to be added. In some embodiments, the layer selection module 710 can exclude the last few layers of the DNN from being selected for modification; for example, the last layer that outputs the final class probability after the SoftMax operation and has low computational cost can be excluded.

[0126] In some embodiments, the layer selection module 710 can identify computationally complex layers in the DNN and select these layers as layers to be modified. The layer selection module 710 can determine that a layer is computationally complex based on one or more attributes of the layer, such as the size of the layer, the number of operations (e.g., MAC operations) in the layer, the type of the layer, special operations that exist before the layer, or other attributes of the layer.

[0127] The layer selection module 710 can determine the layer size based on the number of the layer's internal parameters (e.g., weights). The layer selection module 710 can determine the number of operations in the layer based on the layer's hyperparameters, such as the layer's tensor size (e.g., input tensor, weight tensor, output tensor, etc.), padding size, stride size, etc. The layer selection module 710 can determine that a particular type of layer is computationally complex. Examples of such layers can include convolutional layers, nonlinear attention layers (e.g., ReLU, GELU, etc.), activation layers, normalization layers, element-wise operation layers (addition, subtraction, multiplication, division, etc.), and feedforward layers. The layer selection module 710 can select layers that have one or more operations preceding them. Examples of such special operations include transpose, reshape, splice, etc.

[0128] Graph generation module 720 generates a graph representing the DNN. A graph is a data structure that includes a set of nodes and one or more edges. Nodes are entities in the graph, and edges are connections between two nodes. A graph can be associated with one or more embeddings. For example, a graph can have graph embeddings for encoding one or more properties of the graph; nodes in the graph can have node embeddings for encoding one or more properties of the nodes; edges in the graph can have edge embeddings for encoding one or more properties of the edges. Embeddings can be vectors, also known as embedding vectors.

[0129] Nodes in a graph representing a DNN can represent deep learning operations or layers of the DNN. A node can connect to another node representing a layer of another deep learning operation or DNN. Edges between two nodes can encode the data flow between the two layers. In some embodiments, edges can encode tensors, such as the output tensor of a first layer or the input tensor of a second layer. Nodes or edges can also encode specific operations performed before or after the corresponding layer, such as transpose, reshaping, concatenation, etc. In some embodiments, nodes in the graph can be arranged based on the forward execution pass of the DNN. For example, nodes can be arranged in an order that matches the order of the layers represented by the nodes in the DNN. This graph facilitates model operation and allows for easier statistical analysis of intermediate activations without accessing the model's source code.

[0130] In some embodiments, the graph generation module 720 can also optimize the graph for inference. For example, multiple layers (e.g., convolutional layers) can be fused together. As another example, normalization layers can be batched. Optimizations can be performed to simulate inference execution in a DNN accelerator, such as DNN accelerator 302. The graph generation module 720 may also eliminate dropout layers, as these layers may be invalid during inference. The graph generation module 720 can perform other types of optimizations. Optimizations can reduce latency, improve the performance of the DNN accelerator, or reduce memory usage.

[0131] The modification module 730 modifies the layers selected by the layer selection module 710 through pruning operations. In some embodiments, layer modification can be accomplished using a graph representing the DNN generated by the graph generation module 720. The modification module 730 can identify the location of the selected layer in the DNN based on the graph representing the DNN. After identifying the location of the selected layer, the modification module 730 can add pruning operations to the identified location or to locations before or after the identified location.

[0132] Modification module 730 can add at least one pruning operation to each selected layer. For example, modification module 730 can add an activation pruning operation to the selected layer so that the activation pruning operation can prune the output activation of that layer. Modification module 730 can place the activation pruning operation after the selected layer. For example, modification module 730 can add a weight pruning operation to the selected layer so that the weight pruning operation can prune the weights of that layer. As yet another example, modification module 730 can add both an activation pruning operation and a weight pruning operation to the selected layer.

[0133] Activating the pruning operation can be represented as: Where l represents the layer where the pruning operation is placed, act l This indicates that the output of this layer is activated, act′ l λ represents the output activation determined by the activation pruning operation. l This represents the activation threshold used by the activation pruning operation to prune activations. The activation pruning operation does not change the values ​​of activations whose absolute values ​​are greater than the activation threshold, but it will change the values ​​of activations whose absolute values ​​are less than the activation threshold to zero.

[0134] Weighted pruning can be represented as: Where l represents the layer where the pruning operation is activated, and w l w′ represents the weight of this layer. l λ represents the pruned weights output by the activation pruning operation. l This represents the weight threshold used by the weight pruning operation to prune weights. The weight pruning operation does not change the values ​​of weights whose absolute values ​​are greater than the weight threshold, but it changes the values ​​of weights whose absolute values ​​are less than or equal to the weight threshold to zero. Pruning operations (activation pruning or weight pruning) can be performed using one or more sparse encoders (e.g., ...). Figure 3 (Sparse encoder 380 in the DNN). In some embodiments, weight pruning can be performed during the compilation phase because the weight values ​​are predetermined by training the DNN. Before loading the weights into the DNN accelerator, weight thresholding can be used to prune the weights, thus eliminating the area and power overhead that weight pruning incurs on the DNN accelerator.

[0135] Activation threshold module 740 determines an activation threshold for activating the pruning operation. In some embodiments, activation threshold module 740 may select the optimal activation threshold for the pruning operation (e.g., from multiple candidate activation thresholds) through an iterative process based on an accuracy loss constraint. The optimal activation threshold may be the highest candidate activation threshold that will not cause the DNN accuracy loss to exceed the accuracy loss constraint. The accuracy loss constraint may be a maximum threshold for accuracy loss. The accuracy loss may be the difference between the accuracy of the DNN after activation pruning and the baseline accuracy of the DNN. The baseline accuracy of the DNN may be the accuracy of the DNN without activation pruning.

[0136] In some embodiments, the accuracy loss can be expressed as: ΔA = (Acc) baseline -Acc threshold ) < A b Acc baseline This represents the baseline accuracy of the DNN, specifically the Top l Acc. threshold A represents the accuracy of the DNN after activation pruning, specifically activation threshold pruning. b The baseline accuracy and accuracy loss constraint can be predetermined. In some embodiments, Top-1 accuracy can be used. Top-1 accuracy can represent the proportion (e.g., percentage) of inputs whose generated labels match the true input labels. b It can be 0.5%, 1%, 1.5%, 2%, 2.5%, etc.

[0137] Different layers can have different tolerances to pruning, meaning that even when using the same activation threshold to prune the activation of different layers, the accuracy loss may vary. The activation threshold module 740 can determine a layer-specific activation threshold. In the example, the activation threshold module 740 can determine an activation threshold specific to a particular type of layer. Different types of layers can have different activation thresholds.

[0138] In some embodiments, the iterative process may include multiple rounds. In each round, the activation of the pruning operation may be performed using different candidate thresholds. The threshold used in the first round may be the smallest threshold, such as zero. The threshold used for each subsequent round may be determined by incrementing the threshold of the previous round. In some embodiments, the increment may be fixed.

[0139] In each round, the activation threshold module 740 can determine (or instruct the validation module 640 to determine) the accuracy of the DNN using activation pruning. To determine the accuracy of the DNN, a dataset can be input into the DNN. This dataset can be a training dataset, a validation dataset, or other datasets. Inference of the DNN is performed. The output of the DNN can be compared with the true output of the DNN to determine the accuracy of the DNN. An accuracy loss is determined and compared with an accuracy loss constraint. In embodiments where the accuracy loss is lower than the accuracy loss constraint, the next round will be performed using a higher activation threshold. In embodiments where the accuracy loss is greater than or equal to the accuracy loss constraint, the iteration process stops, and the threshold used in the previous round is selected as the optimal activation threshold for that layer. The optimal activation threshold can be stored, for example, in an internal register integrated with one or more sparse encoders (e.g., sparse encoder 380) during the inference run. The sparse encoder can use the optimal activation threshold to prune the activations generated by the layer and provide the pruned activations to the next layer.

[0140] The weight threshold module 750 determines the weight threshold for the weight pruning operation. In some embodiments, the weight threshold module 750 may determine the optimal weight threshold for the weight pruning operation based on an accuracy loss constraint, a minimum constraint, and a maximum constraint (e.g., from a plurality of candidate weight thresholds). The accuracy loss constraint may be the accuracy loss constraint described above. The minimum constraint and the maximum constraint may constitute the constraints on the optimal weight threshold; for example, the optimal weight threshold may be greater than the minimum constraint and less than the maximum constraint.

[0141] In some embodiments, the weight threshold module 750 determines the minimum value of the absolute weights. The absolute weights can be the absolute values ​​of the weights in the kernel of the layer. The minimum value can be the smallest absolute value of the weights. In embodiments where the minimum value is higher than the minimum constraint, the weight threshold module 750 sets zero as the optimal weight threshold for that layer to avoid introducing sparsity.

[0142] In embodiments where the minimum value is less than or equal to the minimum constraint, the weight threshold module 750 can, for example, determine the optimal weight threshold for non-zero values ​​for each subsequent incremental iteration, where the maximum constraint can be determined. The increment of the maximum constraint can be a predetermined fixed value. The weight threshold module 750 can calculate one or more quantiles based on the distribution of absolute weights. Quantiles determine how many values ​​in the distribution are above or below a certain limit. In the example, the weight threshold module 750 can calculate 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 quantiles, which can be equivalent to the 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80th, and 90th percentiles. In each round of the iteration process, the weight threshold module 750 can identify the highest quantile below the maximum constraint.

[0143] The weight thresholding module 750 can further compare the identified quantiles with the minimum constraint. The weight thresholding module 750 can select the higher of the identified quantiles and the minimum constraint as the weight threshold for this round. The weight thresholding module 750 can further determine (or instruct the validation module 640 to determine) the accuracy of the DNN after the pruning operation of using the weight threshold to prune the weights. To determine the accuracy of the DNN, a dataset can be input into the DNN. This dataset can be a training dataset, a validation dataset, or other datasets. Inference of the DNN is performed. The output of the DNN can be compared with the true output of the DNN to determine the accuracy of the DNN. The accuracy loss is determined and compared with the accuracy loss constraint. In embodiments where the accuracy loss is greater than or equal to the accuracy loss constraint, the iteration process stops, and the threshold used in the previous round is selected and stored as the optimal activation threshold for that layer.

[0144] In embodiments where the accuracy loss is lower than the accuracy loss constraint, the next round will be performed using a higher maximum constraint. Using the higher maximum constraint, the weight threshold module 750 can identify a new highest quantile for the absolute weight that is lower than the higher maximum constraint. The weight threshold module 750 can repeat the step of comparing the highest quantile with the minimum constraint to select which one to use as the weight threshold for this round, and repeat the step of checking the accuracy loss to determine whether to stop the iteration process or execute the next round.

[0145] In some embodiments (e.g., in embodiments where the layer is modified using both activation pruning and weight pruning operations), the activation thresholding module 740 and the weight thresholding module 750 may operate sequentially. In one embodiment, the weight thresholding module 750 may determine the optimal weight threshold for the layer before the activation thresholding module 740 determines the optimal activation threshold. The baseline DNN accuracy used by the activation thresholding module 740 may be the accuracy of the DNN after weight pruning using the optimal weight threshold found by the weight thresholding module 750. In another embodiment, the activation thresholding module 740 determines the optimal activation threshold before the weight thresholding module 750 can determine the optimal weight threshold for the layer. The baseline DNN accuracy used by the weight thresholding module 750 may be the accuracy of the DNN after activation pruning using the optimal activation threshold found by the activation thresholding module 740.

[0146] Example process of selecting the optimal threshold

[0147] Figure 8 The illustrations depict the selection of an optimal threshold for pruning activation according to various embodiments. The optimal threshold can be selected through... Figure 7 The activation threshold module 740 in the middle is used for execution. For example... Figure 8In step 810 shown, a threshold λ is initialized for the first round (i.e., the round with round index i = 0). The threshold λ can be zero or a positive number.

[0148] In step 820, model accuracy is measured using a dataset. The dataset may be generated by training module 620 or validation module 640. The dataset may include one or more inputs to the DNN, each input potentially associated with one or more ground truth labels. Model accuracy can be determined by comparing the labels of the inputs generated by the DNN with the ground truth labels of the inputs. In some embodiments, model accuracy may be a percentage indicating how many labels generated by the DNN match the corresponding ground truth labels. Model accuracy may be Top-1 accuracy.

[0149] In step 830, the accuracy loss ΔA(i) is compared with the accuracy loss constraint A. b A comparison is then made. The accuracy loss ΔA(i) can be equal to the baseline model accuracy of the DNN minus the model accuracy measured in step 830. Accuracy loss constraint A b It can be predetermined and stored in memory.

[0150] When the accuracy loss ΔA(i) is not less than the accuracy loss constraint A b Then, step 840 is executed. In step 840, the optimal threshold is set to the threshold used in the previous round i-1. The optimal threshold can be stored in memory, such as local memory 340, threshold register 420, etc. The optimal threshold can be used in the activation pruning operation in a layer to prune the activations generated by that layer. In some embodiments, the optimal threshold can be used in the activation pruning operation of multiple layers. These layers can be of the same type.

[0151] When the accuracy loss ΔA(i) is lower than the accuracy loss constraint A b Then, step 850 is executed. In step 850, the threshold λ is incremented by a predetermined value Δλ. The incremented threshold λ is used in the next round i+1. In the next round, steps 820 and 830 are executed again. Alternatively, steps 840 or 850 can be executed. This process can continue until the optimal threshold is found.

[0152] Figure 9 The illustration shows the selection of the optimal threshold for pruning weights according to various embodiments. The optimal threshold can be selected by... Figure 7 The weighted threshold module 750 in the middle completes this. Figure 9 In step 910 shown, the minimum absolute weight w is found by searching for the absolute value of the DNN layer weight w. min The values ​​of the weights w can be determined by training the DNN.

[0153] In step 920, the minimum absolute weight |w| is... min With minimum constraint θ low Compare them. When the minimum absolute weight |w| is reached... min Not less than the minimum constraint θ low At that time, step 923 is executed, and the optimal threshold is set to zero, which means that no additional sparsity is introduced into the weight w of this layer.

[0154] When the minimum absolute weight |w| min Below the minimum constraint θ low Then, proceed to step 925. In step 925, find the quantile q of the absolute weight |w|. k (|w|). In Figure 9 In the example, nine quantiles were found. A quantile can be a decimal.

[0155] In step 930, the largest quantile q is found. max (|w|). Maximum quantile q max (|w|) is a value with a larger constraint θ high The quantile of the smallest maximum value, the maximum constraint θ high It can be predetermined and stored in memory.

[0156] In step 940, the largest quantile q is... max (|w|) and minimum constraint θ low Compare them. When the largest quantile q max (|w|) is below the minimum constraint θ low Then, execute step 943 and set the weight threshold λ(i) of the current round to the maximum quantile q. max (|w|). When the largest quantile q max (|w|) is not less than the minimum constraint θ low At that time, execute step 945 and set the weight threshold λ(i) of the current round to the minimum constraint θ. low .

[0157] In step 950, the weight pruning operator is applied with the current weight threshold λ(i) (i.e., the weight threshold set in step 943 or 945). Weights with an absolute value less than λ(i) are changed to zero, while weights with an absolute value greater than or equal to λ(i) remain unchanged.

[0158] In step 960, model accuracy is measured using a dataset. The dataset may be generated by training module 620 or validation module 640. The dataset may include one or more inputs to the DNN, each input potentially associated with one or more ground truth labels. Model accuracy can be determined by comparing the labels of the inputs generated by the DNN with pruned weights to the ground truth labels of the inputs. In some embodiments, model accuracy may be a percentage indicating how many labels generated by the DNN match the corresponding ground truth labels. Model accuracy may be Top-1 accuracy.

[0159] In step 970, the accuracy loss ΔA(i) is compared with the accuracy loss constraint A. b A comparison is then made. The accuracy loss ΔA(i) can be equal to the baseline model accuracy of the DNN minus the model accuracy measured in step 960. Accuracy loss constraint A b It can be predetermined and stored in memory.

[0160] When the accuracy loss ΔA(i) is not less than the accuracy loss constraint A b Then, step 980 is executed. In step 980, the optimal threshold is set to the threshold used in the previous round i-1. The optimal threshold can be stored in memory, such as local memory 340, threshold register 420, etc. The optimal threshold can be used in weight pruning operations within a layer to prune the weights generated by that layer. In some embodiments, the optimal threshold can be used in weight pruning operations across multiple layers. These layers can be of the same type.

[0161] When the accuracy loss ΔA(i) is lower than the accuracy loss constraint A b Then, proceed to step 990. In step 990, the maximum constraint θ... high The incremented predetermined value Δθ high In the next round i+1, the incremented maximum constraint θ will be used. high In the next round, repeat steps 930, 940, 943 or 945, 950, 960, and 970. Additionally, steps 980 or 990 can be performed. This process can continue until the optimal threshold is found.

[0162] Figure 10 The illustrations depict the selection of optimal thresholds for pruning activation and pruning weights according to various embodiments. The optimal thresholds can be selected through... Figure 7 The compression module 700 in the code is used for execution. (In a similar context...) Figure 10 In step 1010 shown, a weight pruning operator is applied. The weight pruning operator can be used for weight pruning operations using programmable weight thresholds.

[0163] In step 1020, model accuracy is measured using a dataset. The dataset may be generated by training module 620 or validation module 640. The dataset may include one or more inputs to the DNN, each input potentially associated with one or more ground truth labels. Model accuracy can be determined by comparing the labels of the inputs generated by the DNN with pruned weights to the ground truth labels of the inputs. In some embodiments, model accuracy may be a percentage indicating how many labels generated by the DNN match the corresponding ground truth labels. Model accuracy may be Top-1 accuracy.

[0164] In step 1030, the accuracy loss ΔA(i) is compared with the accuracy loss constraint A. b A comparison is then made. The accuracy loss ΔA(i) can be equal to the baseline model accuracy of the DNN minus the model accuracy measured in step 1020. Accuracy loss constraint A b It can be predetermined and stored in memory.

[0165] When the accuracy loss ΔA(i) is lower than the accuracy loss constraint A b If the weight threshold is increased, proceed to step 1035 and increase the weight threshold. Then, proceed to step 1010 again using the increased weight threshold.

[0166] When the accuracy loss ΔA(i) is not less than the accuracy loss constraint A b At that time, the optimal weight threshold can be determined. For example, the optimal weight threshold can be the weight threshold determined in the previous round.

[0167] Then, step 1040 is executed. In step 1040, an activation pruning operator is applied to perform an activation pruning operation with a programmable activation threshold.

[0168] In step 1020, the model accuracy is measured using the dataset. In step 1050, the accuracy loss ΔA(i) is compared with the accuracy loss constraint A. b The comparison is then performed. The accuracy loss ΔA(i) can be equal to the baseline model accuracy of the DNN minus the model accuracy measured in step 1030. The baseline model accuracy of the DNN can be the accuracy of the DNN after weight pruning using the optimal weight threshold.

[0169] When the accuracy loss ΔA(i) is lower than the accuracy loss constraint A b When the accuracy loss ΔA(i) is not less than the accuracy loss constraint A, proceed to step 1055 and increase the activation threshold. Then, proceed to step 1040 again using the increased activation threshold. b At that time, the optimal activation threshold can be determined. For example, the optimal activation threshold may be the weight threshold determined in the previous round.

[0170] In step 1070, optimal weight thresholds and optimal activation thresholds are set. These thresholds can be stored in memory, such as local memory 340, threshold register 420, etc. The optimal weight and activation thresholds can be used to prune the weights and activations of one or more layers. Figure 10 In one embodiment, the optimal weight threshold is determined before the optimal activation threshold is determined. In other embodiments, the optimal weight threshold may be determined after the optimal activation threshold is determined.

[0171] Example PE array

[0172] Figure 11 The illustration shows an example PE array according to various embodiments. PE array 1100 may be... Figure 3 An embodiment of the PE array 350 is described. The PE array 1100 includes a plurality of PEs 1110 (collectively referred to as "PE 1110"). PE 1110 is capable of performing MAC operations, including MAC operations in quantized inference. PE 1110 can also be referred to as neurons in a DNN. Each PE 1110 has two input signals 1150 and 1160 and one output signal 1170. Input signal 1150 is at least part of the IFM used for the layer. Input signal 1160 is at least part of the filter of the layer. In some embodiments, the input signal 1150 of PE 1110 includes one or more input operands, and the input signal 1160 includes one or more weight operands.

[0173] Each PE 1110 performs a MAC operation on input signals 1150 and 1160 and outputs an output signal 1170, which is the result of the MAC operation. Some or all of the input signals 1150 and 1160 and the output signal 1170 can be in integer format (such as INT8) or floating-point format (such as FP16 or BF16). For simplicity and illustrative purposes, all PE 1110 input and output signals have the same reference numerals, but PE 1110 can receive different input signals and output different output signals. Furthermore, PE 1110 can differ from another PE 1110, for example, by including more, fewer, or different components.

[0174] like Figure 11 As shown, PE 1110s are connected to each other, as follows: Figure 11As shown by the dashed arrow in the diagram. The output signal 1170 of PE 1110 can be sent as an input signal to many other PE 1110s (and possibly back to itself) through the interconnection between PE 1110s. In some embodiments, the output signal 1170 of PE 1110 can be combined with the output signals of one or more other PE 1110s through the accumulation operation of PE 1110s, and generate the internal part sum of the PE array.

[0175] exist Figure 11 In one embodiment, PEs 1110 are arranged in columns 1105 (referred to as "column 1105"). Layer inputs and weights can be distributed to PEs 1110 based on columns 1105. Each column 1105 has a column buffer 1120. Column buffer 1120 provides data storage for columns 1105 of PEs 1110 for a short period. Column buffer 1120 can also store the output data of the last PE 1110 in column 1105. The output of the last PE 1110 can be the sum of the MAC operations of all PEs 1110 in column 1105, which is the column-level internal partial sum of the PE array 1100. In other embodiments, inputs and weights can be distributed to PEs 1110 based on rows in the PE array 1100. The PE array 1100 can include row buffers instead of column buffers 1120. Row buffers can store the input signals of the PEs in the corresponding rows and can also store the row-level internal partial sums of the PE array 1100.

[0176] In some embodiments, column buffer 1120 may be Figure 3 Part of local memory 340. Column buffer 1120 can be associated with the upper memory hierarchy, for example... Figure 3 The memory 310 is in the upper memory hierarchy. Data in the column buffer 1120 can be sent to the upper memory hierarchy. The column buffer 1120 can receive data from the upper memory hierarchy.

[0177] Figure 12 This is a block diagram of PE 1200 according to various embodiments. PE 1200 can be... Figure 11 Examples of PE 1110, or Figure 3 An embodiment of the PE in the PE array 350. PE 1200 can perform MAC operations, for example, performing MAC operations using data in integer format. Figure 12As shown, PE 1200 includes an input register file 1210 (referred to individually as "Input Register File 1210"), a weight register file 1220 (referred to individually as "Weight Register File 1220"), a multiplier 1230 (referred to individually as "Multiplier 1230"), an internal adder component 1240, and an output register file 1250. In other embodiments, PE 1200 may include fewer, more, or different components. For example, PE 1200 may include multiple output register files 1250. As another example, PE 1200 may include a single input register file 1210, a weight register file 1220, or a multiplier 1230. As yet another example, PE 1200 may include an adder instead of the internal adder component 1240.

[0178] Input register file 1210 temporarily stores input operands for MAC operations performed by PE 1200. In some embodiments, input register file 1210 may store a single input operand at a time. In other embodiments, input register file 1210 may store multiple input operands or a portion of an input operand at a time. Input operands comprise multiple input elements (i.e., input components) in an input tensor. The input elements of the input operands may be stored sequentially in input register file 1210 to allow sequential processing of the input elements. In some embodiments, each input element in the input operand may come from a different input channel of the input tensor. The input operand may include input elements from each input channel of the input tensor, and the number of input elements in the input operand may be equal to the number of input channels. The input elements in the input operand may have the same (X,Y) coordinates, which can be used as the (X,Y) coordinates of the input operand. For example, all input elements of the input operand may be X0Y0, X0Y1, X1Y1, etc.

[0179] Weight register file 1220 temporarily stores weight operands used for MAC operations performed by PE 1200. Weight operands include weights from filters in the DNN layer. In some embodiments, weight register file 1220 may store a single weight operand at a time. In other embodiments, weight register file 1220 may store multiple weight operands or a portion of a single weight operand at a time. A weight operand may include multiple weights. The weights of the weight operands may be stored sequentially in weight register file 1220 to allow for sequential processing of weights. In some embodiments, for multiplication operations involving weight operands and input operands, each weight in the weight operand may correspond to an input element of the input operand. The number of weights in the weight operand may be equal to the number of input elements in the input operand.

[0180] In some embodiments, the weight register file 1220 may be the same as or similar to the input register file 1210, for example, having the same size. PE 1200 may include multiple register files, some of which are designated as input register file 1210 for storing input operands; some of which are designated as weight register file 1220 for storing weight operands; and some of which are designated as output register file 1250 for storing output operands. In other embodiments, the register files in PE 1200 may be designated for other purposes, such as storing scaling operands used in element-wise addition operations.

[0181] Multiplier 1230 performs multiplication operations on input operands and weight operands. Multiplier 1230 can perform a series of multiplication operations on a single input operand and a single weight operand, generating a product operand comprising a series of products. Each multiplication operation in the sequence involves multiplying an input element in the input operand by a weight in the weight operand. In some embodiments, the position (or index) of the input element in the input operand matches the position (or index) of the weight in the weight operand. For example, a first multiplication operation multiplies a first input element in the input operand by a first weight in the weight operand, a second multiplication operation multiplies a second input element in the input operand by a second weight in the weight operand, a third multiplication operation multiplies a third input element in the input operand by a third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation can correspond to the same depth-wise channel, and their products can also correspond to the same depth-wise channel.

[0182] Multiple multipliers 1230 can perform multiplication operations simultaneously. These multiplication operations can be referred to as a round of multiplication. In a round of multiplication by multipliers 1230, each multiplier 1230 can use different input operands and different weight operands. Different input operands or weight operands can be stored in different register files of PE 1200. For example, the first multiplier 1230 uses (e.g., stored in the first input register file 1210) a first input operand and (e.g., stored in the first weight register file 1220) a first weight operand, while the second multiplier 1230 uses (e.g., stored in the second input register file 1210) a second input operand and (e.g., stored in the second weight register file 1220) a second weight operand, the third multiplier 1230 uses (e.g., stored in the third input register file 1210) a third input operand and (e.g., stored in the third weight register file 1220) a third weight operand, and so on. For a single multiplier 1230, the multiplication cycle can include multiple cycles. One cycle includes multiplying the input elements and weights.

[0183] Multiplier 1230 can perform multiple rounds of multiplication. Multiplier 1230 can use the same weighted operands but different input operands in different rounds. For example, multiplier 1230 performs a series of multiplications on a first input operand stored in a first input register file in the first round, and a series of multiplications on a second input operand stored in a second input register file in the second round. In the second round, a different multiplier 1230 can use the first input operand and different weighted operands to perform another sequence of multiplications. Thus, the first input operand can be reused in the second round. The first input operand can also be reused again in additional rounds, for example, by an additional multiplier 1230.

[0184] The internal adder component 1240 includes one or more adders within the PE 1200, i.e., internal adders. The internal adder component 1240 can perform accumulation operations on two or more product operands from the multiplier 1230 and produce output operands for the PE 1200. In some embodiments, the internal adders are arranged in a tier order. A tier includes one or more internal incrementers. For a first tier of the internal adder component 1240, the internal adder may receive product operands from two or more multipliers 1230 and generate a sum operand through a series of accumulation operations. Each accumulation operation produces a sum of two or more products, each product originating from a different multiplier 1230. The sum operand comprises a series of sums, each sum being the result of an accumulation operation and corresponding to a depth-wise channel. For the other tiers(s) of the internal adder component 1240, the internal adders within a tier receive sum operands from the previous tier in the sequence. Each of these numbers can be generated by a different internal adder in the preceding level. The ratio of the number of internal adders in a certain level to the number of internal adders in subsequent levels can be 2:1. In some embodiments, the final level of the internal adder assembly 1240 may include a single internal adder that produces the output operand of PE 1200.

[0185] Output register file 1250 stores the output operands of PE 1200. In some embodiments, output register file 1250 may store one output operand at a time. In other embodiments, output register file 1250 may store multiple output operands or a portion of an output operand at a time. The output operands include multiple output elements in the IFM. The output elements of the output operands may be stored sequentially in output register file 1250 so that the output elements can be processed sequentially. In some embodiments, each output element in the output operand corresponds to a different depth-wise channel and is an element of a different output channel of the depth-wise convolution. The number of output elements in the output operand may be equal to the number of depth-wise channels of the depth-wise convolution.

[0186] Modify the example method of DNN

[0187] Figure 13 This is a flowchart illustrating a method 1300 for modifying a DNN according to various embodiments. Method 1300 can be... Figure 7 The compression module 700 is executed. Although method 1300 is a reference... Figure 13 The flowchart shown illustrates the process, but alternatively, many other methods can be used to modify the DNN. For example, one could change... Figure 13 The execution order of the steps. As another example, some steps can be changed, deleted, or merged.

[0188] Compression module 700 inputs the dataset 1310 into the neural network. The neural network comprises multiple layers. Compression module 700 selects 1320 layers from the multiple layers. The selected layers in the neural network generate tensors based on the dataset. In some embodiments, compression module 700 selects layers based on the number of internal parameters of the layer (e.g., the number of weights, etc.), the amount of computation in the layer (e.g., the number of MAC operations, etc.), the type of the layer, or some combination thereof.

[0189] The compression module 700 prunes the 1330 tensor based on a first activation threshold by modifying the absolute value of the activation in the tensor to zero. The absolute value of the activation is lower than the first activation threshold.

[0190] Compression module 700 determines the accuracy of neural network 1340 based on the output of the neural network. The neural network generates output based on pruned tensors.

[0191] Compression module 700 determines a second activation threshold 1350 based on a first activation threshold and the accuracy of the neural network. The second activation threshold has a different value than the first activation threshold. In some embodiments, compression module 700 determines the accuracy loss caused by pruning tensors based on the accuracy of the neural network. Compression module 700 determines whether the accuracy loss exceeds a threshold. In response to determining that the accuracy loss is below the threshold, compression module 700 determines the second activation threshold by increasing the first activation threshold. In response to determining that the accuracy loss exceeds the threshold, compression module 700 determines the second activation threshold by decreasing the first activation threshold.

[0192] In some embodiments, the compression module 700 determines a first activation threshold based on a third activation threshold. The third activation threshold has a value different from the first activation threshold and the second activation threshold. The third activation threshold may be lower than the first activation threshold. In some embodiments, the compression module 700 inputs an additional dataset into the neural network. Layers in the neural network generate additional tensors based on the additional dataset. The compression module 700 prunes the additional tensors based on the third activation threshold. The compression module 700 determines the additional accuracy of the neural network based on the additional output generated by the neural network based on the pruned additional tensors. The compression module 700 determines the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.

[0193] The compression module 700 modifies the 1360 neural network by adding activation pruning operations to the layers. The activation pruning operation prunes one or more tensors to be generated by the layer based on a second activation threshold. In some embodiments, the compression module 700 further modifies the neural network by adding weight pruning operations to the layers. The activation pruning operation prunes the kernel of the layer by modifying the value of the weight in the kernel of the layer to zero, based on a weight threshold. The value of the weight is below the weight threshold. In some embodiments, the neural network is trained to determine the value of the weight. In some embodiments, the compression module 700 determines the second activation threshold after adding weight pruning operations to the layers.

[0194] Figure 14 This is a flowchart illustrating another method 1400 for modifying a DNN according to various embodiments. Method 1400 can be derived from... Figure 7 The compression module 700 is executed. Although method 1400 is a reference... Figure 14 The flowchart shown illustrates the process, but alternatively, many other methods can be used to modify the DNN. For example, one could change... Figure 14 The execution order of steps. For example, some steps can be changed, deleted, or merged.

[0195] Compression module 700 inputs the dataset 1410 into the neural network. The neural network includes layers. Each layer has a weight tensor. The weight tensor can be a filter, kernel, vector in the filter, etc. In some embodiments, compression module 700 selects a layer from multiple layers of the neural network based on the number of internal parameters of the layer, the number of operations in the layer, the type of the layer, or some combination thereof.

[0196] The compression module 700 prunes the weight tensor 1420 by modifying the absolute values ​​of the weights in the weight tensor to zero based on a first weight threshold. The absolute value of this weight is lower than the first weight threshold. In some embodiments, the compression module 700 determines the first weight threshold based on the absolute values ​​of multiple weights in the weight tensor, a first threshold (e.g., a maximum constraint), and a second threshold (e.g., a minimum constraint). The first weight threshold is lower than the first threshold but greater than the second threshold. In some embodiments, the compression module 700 determines that the minimum absolute value of the multiple weights in the weight tensor is lower than the second threshold.

[0197] In some embodiments, the compression module 700 determines one or more quantiles based on the absolute values ​​of multiple weights. The compression module 700 selects a quantile from the one or more quantiles based on a first threshold. The compression module 700 selects the larger of the selected quantile and a second threshold as the first weight threshold. In some embodiments, the value of the selected quantile is greater than the value of another quantile of the one or more quantiles, but less than the first threshold.

[0198] In some embodiments, the compression module 700 determines a first weight threshold. The compression module 700 inputs an additional dataset into the neural network. The compression module 700 prunes the weight tensor based on a third weight threshold lower than the first weight threshold. The compression module 700 determines the additional accuracy of the neural network based on the additional output generated by the neural network using the weight tensor pruned based on the third weight threshold. The compression module 700 determines the first weight threshold based on the additional accuracy of the neural network.

[0199] The compression module 700 determines the accuracy of the neural network 1430 based on the output of the neural network. This output is generated by the neural network based on the pruned weight tensor.

[0200] Compression module 700 determines a second weight threshold 1440 based on a first weight threshold and the accuracy of the neural network. The second weight threshold has a different value than the first weight threshold. In some embodiments, compression module 700 determines the accuracy loss caused by the pruned weight tensor based on the accuracy of the neural network. Compression module 700 determines whether the accuracy loss is less than a threshold. In response to determining that the accuracy loss is less than the threshold, compression module 700 determines the second weight threshold by increasing the second threshold. In response to determining that the accuracy loss is greater than or equal to the threshold, compression module 700 determines the second weight threshold based on a predetermined weight threshold. The predetermined weight threshold is lower than the first weight threshold.

[0201] The compression module 700 modifies the 1450 neural network by adding weight pruning operations to the layers. The weight pruning operation prunes the weight tensors based on a second weight threshold. In some embodiments, the compression module 700 further modifies the neural network by adding activation pruning operations to the layers, which prune the tensors generated by the layer using the weight tensors after the weight tensors have been pruned by the weight pruning operation.

[0202] Example computing device

[0203] Figure 15 This is a block diagram of an example computing device 1500 according to various embodiments. In some embodiments, the computing device 1500 can be used as at least a part of a DNN system 300. Figure 15The diagram illustrates multiple components included in computing device 1500; however, any one or more of these components may be omitted or repeated to suit the application. In some embodiments, some or all of the components included in computing device 1500 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system-on-a-chip (SoC) die. Furthermore, in various embodiments, computing device 1500 may not include... Figure 15 One or more of the components shown may be included, but computing device 1500 may include interface circuitry for coupling to said one or more components. For example, computing device 1500 may not include display device 1506, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1506 may be coupled. In another set of examples, computing device 1500 may not include audio input device 1518 or audio output device 1508, but may include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which audio input device 1518 or audio output device 1508 may be coupled.

[0204] Computing device 1500 may include processing device 1502 (e.g., one or more processing devices). Processing device 1502 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 1500 may include memory 1504, which itself may include one or more storage devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. In some embodiments, memory 1504 may include memory sharing a die with processing device 1502. In some embodiments, memory 1504 includes one or more non-transitory computer-readable media storing executable instructions to perform operations for pruning weights in a DNN, for example, combining... Figure 13 The method 1300 described above, combined with Figure 14 The method 1400 described above, or by combination Figure 6 The described DNN module 600 (e.g., compression module 630, etc.) performs some operations. Instructions stored in one or more non-transitory computer-readable media can be executed by processing device 1502.

[0205] In some embodiments, computing device 1500 may include communication chip 1512 (e.g., one or more communication chips). For example, communication chip 1512 may be configured to manage wireless communication to transmit data to or from computing device 1500. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., which can transmit data in a non-solid medium by using modulated electromagnetic radiation. This term does not mean that the associated devices do not contain any wires, although they may be wire-free in some embodiments.

[0206] The 1512 communication chip can implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, such as Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 revision), Long Term Evolution (LTE) projects, and any revisions, updates, and / or modifications (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compliant with the IEEE 802.16 standard are often referred to as WiMAX networks. WiMAX stands for Global Microwave Access Interoperability, a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The 1512 communication chip can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication chip 1512 can operate according to Enhanced GSM Evolved Data (EDGE), GSM Electrified Data Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication chip 1512 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Communication (DECT), Evolved Data Optimization (EV-DO) and its derivatives, and any other wireless protocol designated as 3G, 4G, 5G, or higher. In other embodiments, communication chip 1512 can operate according to other wireless protocols. Computing device 1500 may include antenna 1522 to facilitate wireless communication and / or receive other wireless communications (such as AM or FM radio transmissions).

[0207] In some embodiments, the communication chip 1512 can manage wired communications, such as electrical communications, optical communications, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1512 may include multiple communication chips. For example, a first communication chip 1512 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, while a second communication chip 1512 may be dedicated to long-range wireless communications, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication chip 1512 may be dedicated to wireless communications, and the second communication chip 1512 may be dedicated to wired communications.

[0208] The computing device 1500 may include a battery / power circuit 1514. The battery / power circuit 1514 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1500 to a power source (e.g., AC power) that is separate from the computing device 1500.

[0209] The computing device 1500 may include a display device 1506 (or a corresponding interface circuit as described above). The display device 1506 may include any visual indicator, such as, for example, a head-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0210] The computing device 1500 may include an audio output device 1508 (or a corresponding interface circuit as described above). The audio output device 1508 may include any device that generates sound indications, such as, for example, a speaker, headphones, or earphones.

[0211] The computing device 1500 may include an audio input device 1518 (or a corresponding interface circuit as described above). The audio input device 1518 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0212] The computing device 1500 may include a GPS device 1516 (or a corresponding interface circuit as described above). The GPS device 1516 may communicate with a satellite-based system and may receive the location of the computing device 1500 as known in the art.

[0213] The computing device 1500 may include another output device 1510 (or a corresponding interface circuit as described above). Examples of other output devices 1510 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0214] The computing device 1500 may include another input device 1520 (or a corresponding interface circuit as described above). Examples of other input devices 1520 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse), a stylus, a touchpad, a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0215] The computing device 1500 can have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook, ultrabook, personal digital assistant (PDA), ultra-mobile PC, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1500 can be any other electronic device that processes data.

[0216] Select Example

[0217] The following paragraphs provide examples of the various embodiments disclosed herein.

[0218] Example 1 provides a method for modifying a neural network, comprising: inputting a dataset into the neural network, the neural network including multiple layers; selecting one layer from the multiple layers of the neural network, the selected layer generating a tensor based on the dataset; pruning the tensor based on a first activation threshold by modifying the absolute value of activations in the tensor to zero, wherein the absolute value of activations is lower than the first activation threshold; determining the accuracy of the neural network based on the output of the neural network, the neural network generating the output based on the pruned tensor; determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a value different from the first activation threshold; and modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on the second activation threshold.

[0219] Example 2 provides the method of Example 1, wherein determining the second activation threshold includes: determining the accuracy loss caused by the pruning tensor based on the accuracy of the neural network; determining whether the accuracy loss exceeds the threshold; and determining the second activation threshold by increasing the first activation threshold in response to determining that the accuracy loss is below the threshold.

[0220] Example 3 provides the method of Example 2, wherein determining the second activation threshold further includes: determining the second activation threshold by decreasing the first activation threshold in response to determining that the accuracy loss exceeds the threshold.

[0221] Example 3 provides a method from any of Examples 1-3, further including: determining a first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.

[0222] Example 5 provides the method of Example 4, wherein determining the first activation threshold includes: inputting an additional dataset into a neural network, wherein selected layers in the neural network generate additional tensors based on the additional dataset; pruning the additional tensors based on a third activation threshold; determining the additional accuracy of the neural network based on the additional output generated by the neural network based on the pruned additional tensors; and determining the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.

[0223] Example 6 provides a method for any of Examples 1-5, further including selecting another layer in the neural network; and modifying the neural network by adding another activation pruning operation to the other layer, the other activation pruning operation being used to prune one or more tensors to be generated by the other layer based on another activation threshold.

[0224] Example 7 provides a method for any of Examples 1-6, where selecting a layer involves selecting a layer based on the amount of internal parameters of the layer, the amount of computation in the layer, the type of the layer, or a combination thereof.

[0225] Example 8 provides a method from any of Examples 1-7, further including modifying the neural network by adding a weight pruning operation to the selected layer, activating the pruning operation to prune the kernel of the selected layer based on a weight threshold by modifying the absolute value of the weights in the kernel of the selected layer to zero, where the absolute value of the weights is below the weight threshold.

[0226] Example 9 provides the method of Example 8, where the neural network is trained to determine the absolute values ​​of the weights.

[0227] Example 10 provides the method of Example 8 or 9, where determining the second activation threshold includes determining the second activation threshold after adding a weight pruning operation to the layer.

[0228] Example 11 provides a non-transitory computer-readable medium storing one or more instructions executable to perform operations for modifying a neural network, the operations including: inputting a dataset into the neural network, the neural network including multiple layers; selecting one layer from the multiple layers of the neural network, the selected layer generating a tensor based on the dataset; pruning the tensor based on a first activation threshold by modifying the absolute value of activations in the tensor to zero, wherein the absolute value of activations is below the first activation threshold; determining the accuracy of the neural network based on the output of the neural network, the neural network generating the output based on the pruned tensor; determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a value different from the first activation threshold; and modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on the second activation threshold.

[0229] Example 12 provides one or more non-transitory computer-readable media as in Example 11, wherein determining the second activation threshold includes: determining the accuracy loss caused by the pruning tensor based on the accuracy of the neural network; determining whether the accuracy loss exceeds the threshold; and determining the second activation threshold by increasing the first activation threshold in response to determining that the accuracy loss is below the threshold.

[0230] Example 13 provides one or more non-transitory computer-readable media as in Example 12, wherein determining the second activation threshold further includes determining the second activation threshold by decreasing the first activation threshold in response to determining that the accuracy loss exceeds the threshold.

[0231] Example 14 provides one or more non-transitory computer-readable media of any of Examples 11-13, wherein the operation further includes determining a first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.

[0232] Example 15 provides one or more non-transitory computer-readable media as in Example 14, wherein determining a first activation threshold includes: inputting an additional dataset into a neural network, wherein selected layers in the neural network generate additional tensors based on the additional dataset; pruning the additional tensors based on a third activation threshold; determining additional accuracy of the neural network based on additional output generated by the neural network based on the pruned additional tensors; and determining the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.

[0233] Example 16 provides one or more non-transitory computer-readable media of any of Examples 11-15, wherein the operation further includes selecting another layer in a neural network; and modifying the neural network by adding another activation pruning operation to the other layer, the other activation pruning operation being used to prune one or more tensors to be generated by the other layer based on another activation threshold.

[0234] Example 17 provides one or more non-transitory computer-readable media of any of Examples 11-16, wherein selecting a layer includes selecting a layer based on the amount of internal parameters of the layer, the amount of computation in the layer, the type of the layer, or a combination thereof.

[0235] Example 18 provides one or more non-transitory computer-readable media of any one of Examples 11-17, wherein the operation further includes modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation being used to prune the kernel of the selected layer based on a weight threshold by modifying the absolute value of the weights in the kernel of the selected layer to zero, wherein the absolute value of the weights is below the weight threshold.

[0236] Example 19 provides one or more non-transitory computer-readable media as in Example 18, wherein a neural network is trained to determine the absolute values ​​of the weights.

[0237] Example 20 provides one or more non-transitory computer-readable media as in Example 18 or 19, wherein determining the second activation threshold includes determining the second activation threshold after a weight pruning operation is performed on the layer.

[0238] Example 21 provides an apparatus including a computer processor for executing computer program instructions; and a non-transitory computer-readable storage memory storing computer program instructions executable by the computer processor to perform operations for modifying a neural network, the operations including: inputting a dataset into the neural network, the neural network including a plurality of layers; selecting one layer from the plurality of layers in the neural network, the selected layer generating a tensor based on the dataset; pruning the tensor based on a first activation threshold by modifying the absolute value of activations in the tensor to zero, wherein the absolute value of activations is below the first activation threshold; determining the accuracy of the neural network based on the output of the neural network, the neural network generating the output based on the pruned tensor; determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a value different from the first activation threshold; and modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on the second activation threshold.

[0239] Example 22 provides an apparatus of Example 21, wherein determining the second activation threshold includes: determining an accuracy loss caused by a pruning tensor based on the accuracy of the neural network; determining whether the accuracy loss exceeds a threshold; determining the second activation threshold by increasing a first activation threshold in response to determining that the accuracy loss is below the threshold; and determining the second activation threshold by decreasing the first activation threshold in response to determining that the accuracy loss exceeds the threshold.

[0240] Example 23 provides an apparatus of Example 21 or 22, wherein the operation further includes determining a first activation threshold based on a third activation threshold by: inputting an additional dataset into a neural network, wherein the layers in the neural network generate additional tensors based on the additional dataset; pruning the additional tensors based on the third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold; determining the additional accuracy of the neural network based on the additional output generated by the neural network based on the pruned additional tensors; and determining the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.

[0241] Example 24 provides an apparatus for any of Examples 21-23, wherein the operation further includes modifying the neural network by adding a weight pruning operation to the layer, and activating the pruning operation to prune the kernel of the layer to zero based on a weight threshold by modifying the absolute value of the weights in the kernel of the layer to a weight threshold.

[0242] Example 25 provides an apparatus for any of Examples 21-24, wherein the operation further includes selecting another layer in the neural network; and modifying the neural network by adding another activation pruning operation to the other layer, the other activation pruning operation being used to prune one or more tensors to be generated by the other layer based on another activation threshold.

[0243] Other Featured Examples

[0244] The following paragraphs provide examples of the various embodiments disclosed herein.

[0245] Example 1 provides a method for modifying a neural network, comprising: inputting a dataset into the neural network, the neural network including layers having weight tensors; pruning the weight tensors based on a first weight threshold by modifying the absolute values ​​of the weights in the weight tensors to zero, wherein the absolute values ​​of the weights are less than the first weight threshold; determining the accuracy of the neural network based on the output of the neural network, the output being generated by the neural network based on the pruned weight tensors; determining a second weight threshold based on the first weight threshold and the accuracy of the neural network, the second weight threshold having a value different from the first weight threshold; and modifying the neural network by adding a weight pruning operation to the layer, the weight pruning operation being used to prune one or more weight tensors based on the second weight threshold.

[0246] Example 2 provides the method of Example 1, further including: determining a first weight threshold based on the absolute values ​​of multiple weights in a weight tensor, a first threshold, and a second threshold, wherein the first weight threshold is lower than the first threshold and higher than the second threshold.

[0247] Example 3 provides the method of Example 2, wherein determining the first weight threshold includes: determining one or more quantiles based on the absolute values ​​of multiple weights; selecting a quantile from the one or more quantiles based on the first threshold; and selecting the larger of the selected quantile and the second threshold as the first weight threshold.

[0248] Example 4 provides the method of Example 3, wherein the value of the selected quantile is greater than the value of another quantile of one or more quantiles, and less than a first threshold.

[0249] Example 5 provides a method for any of Examples 2-4, wherein determining the second weight threshold includes: determining the accuracy loss caused by the pruned weight tensor based on the accuracy of the neural network; determining whether the accuracy loss is less than a threshold; and determining the second weight threshold by increasing the second threshold in response to determining that the accuracy loss is less than the threshold.

[0250] Example 6 provides the method of Example 5, wherein determining the second weight threshold further includes: in response to determining that the accuracy loss is greater than or equal to a threshold, determining the second weight threshold based on a predetermined weight threshold, wherein the predetermined weight threshold is lower than the first weight threshold.

[0251] Example 7 provides a method from any of Examples 2-6, further including determining that the minimum absolute value of a plurality of weights in the weight tensor is below a second threshold.

[0252] Example 8 provides a method from any of Examples 1-7, further comprising determining a first weight threshold by: inputting an additional dataset into the neural network; pruning the weight tensor based on a third weight threshold below the first weight threshold; determining the additional accuracy of the neural network based on the additional output generated by the neural network using the weight tensor pruned based on the third weight threshold; and determining a first activation threshold based on the additional accuracy of the neural network.

[0253] Example 9 provides a method from any of Examples 1-8, further including: selecting a layer from multiple layers of a neural network based on the number of internal parameters in the layer, the number of operations in the layer, the type of the layer, or some combination thereof.

[0254] Example 10 provides a method from any of Examples 1-9, further including: modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation being used to prune the tensor generated by the layer using the weight tensor after the weight tensor has been pruned by a weight pruning operation.

[0255] Example 11 provides a non-transitory computer-readable medium storing one or more instructions executable to perform operations for modifying a neural network, the operations including: inputting a dataset into the neural network, the neural network including layers having weight tensors; pruning the weight tensors based on a first weight threshold by modifying the absolute values ​​of the weights in the weight tensors to zero, wherein the absolute values ​​of the weights are less than the first weight threshold; determining the accuracy of the neural network based on the output of the neural network, the output being generated by the neural network based on the pruned weight tensors; determining a second weight threshold based on the first weight threshold and the accuracy of the neural network, the second weight threshold having a value different from the first weight threshold; and modifying the neural network by adding a weight pruning operation to the layer, the weight pruning operation being used to prune one or more weight tensors based on the second weight threshold.

[0256] Example 12 provides one or more non-transitory computer-readable media as in Example 11, wherein the operation further includes: determining a first weight threshold based on the absolute values ​​of a plurality of weights in a weight tensor, a first threshold, and a second threshold, wherein the first weight threshold is lower than the first threshold and higher than the second threshold.

[0257] Example 13 provides one or more non-transitory computer-readable media as in Example 12, wherein determining a first weight threshold includes: determining one or more quantiles based on the absolute values ​​of a plurality of weights; selecting a quantile from the one or more quantiles based on the first threshold; and selecting the larger of the selected quantile and a second threshold as the first weight threshold.

[0258] Example 14 provides one or more non-transitory computer-readable media as in Example 12 or 13, wherein determining a second weight threshold includes: determining an accuracy loss caused by a pruned weight tensor based on the accuracy of a neural network; determining whether the accuracy loss is less than a threshold; determining a second weight threshold by increasing a second threshold in response to determining that the accuracy loss is less than the threshold; and determining a second weight threshold based on a predetermined weight threshold in response to determining that the accuracy loss is greater than or equal to the threshold, wherein the predetermined weight threshold is lower than a first weight threshold.

[0259] Example 15 provides one or more non-transitory computer-readable media of any one of Examples 11-14, wherein the operation further includes determining a first weight threshold by: inputting an additional dataset into a neural network; pruning a weight tensor based on a third weight threshold below the first weight threshold; determining additional accuracy of the neural network based on additional output generated by the neural network using the weight tensor pruned based on the third weight threshold; and determining a first activation threshold based on the additional accuracy of the neural network.

[0260] Example 16 provides one or more non-transitory computer-readable media of any one of Examples 11-15, wherein the operation further includes: further modifying the neural network by adding activation pruning operations to the layer, the activation pruning operations being used to prune the tensors generated by the layer using the weight tensors after the weight tensors have been pruned by weight pruning operations.

[0261] Example 17 provides an apparatus including a computer processor for executing computer program instructions; and a non-transitory computer-readable storage memory storing computer program instructions executable by the computer processor to perform operations for modifying a neural network, the operations including: inputting a dataset into the neural network, the neural network including layers having weight tensors; pruning the weight tensors based on a first weight threshold by modifying the absolute values ​​of the weights in the weight tensors to zero, wherein the absolute values ​​of the weights are less than the first weight threshold; determining the accuracy of the neural network based on the output of the neural network, the output being generated by the neural network based on the pruned weight tensors; determining a second weight threshold based on the first weight threshold and the accuracy of the neural network, the second weight threshold having a value different from the first weight threshold; and modifying the neural network by adding a weight pruning operation to the layer, the weight pruning operation being used to prune one or more weight tensors based on the second weight threshold.

[0262] Example 18 provides the apparatus of Example 17, wherein the operation further includes determining a first weight threshold based on the absolute values ​​of a plurality of weights in a weight tensor, a first threshold, and a second threshold, wherein the first weight threshold is lower than the first threshold and higher than the second threshold.

[0263] Example 19 provides the apparatus of Example 18, wherein determining a first weight threshold includes: determining one or more quantiles based on the absolute values ​​of a plurality of weights; selecting a quantile from the one or more quantiles based on the first threshold; and selecting the larger of the selected quantile and a second threshold as the first weight threshold.

[0264] Example 20 provides an apparatus of Example 18 or 19, wherein determining the second weight threshold includes: determining the accuracy loss caused by the pruned weight tensor based on the accuracy of the neural network; determining whether the accuracy loss is less than a threshold; and determining the second weight threshold by increasing the second threshold in response to determining that the accuracy loss is less than the threshold.

[0265] The foregoing description of illustrative embodiments of this disclosure, including those described in the abstract, is not intended to be exhaustive of all embodiments, nor is it intended to limit this disclosure to the precise form disclosed. While specific embodiments and examples of this disclosure have been described herein for ease of explanation, those skilled in the art will understand that various equivalent modifications can be made within the scope of this disclosure. These modifications can be made based on the foregoing detailed description.

Claims

1. A method for modifying a neural network, comprising: The dataset is input into the neural network, which includes multiple layers; One layer is selected from the plurality of layers in the neural network, and the selected layer in the neural network generates a tensor based on the dataset; Based on a first activation threshold, the tensor is pruned by modifying the absolute value of activation in the tensor to zero, wherein the absolute value of activation is lower than the first activation threshold. The accuracy of the neural network is determined based on its output, and the neural network generates the output based on the pruned tensor. A second activation threshold is determined based on the first activation threshold and the accuracy of the neural network, wherein the second activation threshold has a value different from the first activation threshold; as well as The neural network is modified by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on a second activation threshold.

2. The method of claim 1, wherein determining the second activation threshold comprises: The accuracy loss caused by pruning the tensor is determined based on the accuracy of the neural network. Determine whether the accuracy loss exceeds a threshold; as well as In response to determining that the accuracy loss is below the threshold, the second activation threshold is determined by increasing the first activation threshold.

3. The method of claim 2, wherein determining the second activation threshold further comprises: In response to determining that the accuracy loss exceeds the threshold, the second activation threshold is determined by decreasing the first activation threshold.

4. The method according to any one of claims 1-3, further comprising: The first activation threshold is determined based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.

5. The method of claim 4, wherein determining the first activation threshold comprises: An additional dataset is input into the neural network, and the selected layers in the neural network generate additional tensors based on the additional dataset; The additional tensor is pruned based on the third activation threshold; The additional accuracy of the neural network is determined based on the additional output generated by the neural network based on the additional tensor after pruning. as well as The first activation threshold is determined based on the third activation threshold and the additional accuracy of the neural network.

6. The method according to any one of claims 1-5, further comprising: Select another layer in the neural network; as well as The neural network is modified by adding another activation pruning operation to the other layer, which prunes one or more tensors to be generated by the other layer based on another activation threshold.

7. The method according to any one of claims 1-6, wherein selecting the layer comprises: The layer is selected based on the amount of its internal parameters, the amount of computation in the layer, the type of the layer, or a combination thereof.

8. The method according to any one of claims 1-7, further comprising: The neural network is further modified by adding a weight pruning operation to the selected layer. This activation pruning operation is used to prune the kernel of the selected layer based on a weight threshold by setting the absolute value of the weights in the kernel of the selected layer to zero. The absolute value of the weight is lower than the weight threshold.

9. The method of claim 8, wherein the neural network is trained to determine the absolute values ​​of the weights.

10. The method of claim 8 or 9, wherein determining the second activation threshold comprises: The second activation threshold is determined after the weight pruning operation is performed on the layer.

11. A non-transitory computer-readable medium storing one or more instructions, said instructions being executable to perform operations for modifying a neural network, said operations including: The dataset is input into the neural network, which includes multiple layers; One layer is selected from the plurality of layers in the neural network, and the selected layer in the neural network generates a tensor based on the dataset; Based on a first activation threshold, the tensor is pruned by modifying the absolute value of activation in the tensor to zero, wherein the absolute value of activation is lower than the first activation threshold. The accuracy of the neural network is determined based on its output, and the neural network generates the output based on the pruned tensor. A second activation threshold is determined based on the first activation threshold and the accuracy of the neural network, wherein the second activation threshold has a value different from the first activation threshold; as well as The neural network is modified by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on a second activation threshold.

12. The one or more non-transitory computer-readable media of claim 11, wherein determining the second activation threshold comprises: The accuracy loss caused by pruning the tensor is determined based on the accuracy of the neural network. Determine whether the accuracy loss exceeds a threshold; as well as In response to determining that the accuracy loss is below the threshold, the second activation threshold is determined by increasing the first activation threshold.

13. The one or more non-transitory computer-readable media of claim 12, wherein determining the second activation threshold further comprises: In response to determining that the accuracy loss exceeds the threshold, the second activation threshold is determined by decreasing the first activation threshold.

14. One or more non-transitory computer-readable media according to any one of claims 11-13, wherein the operation further comprises: The first activation threshold is determined based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.

15. One or more non-transitory computer-readable media according to claim 14, wherein determining the first activation threshold comprises: An additional dataset is input into the neural network, and the selected layers in the neural network generate additional tensors based on the additional dataset; The additional tensor is pruned based on the third activation threshold; The additional accuracy of the neural network is determined based on the additional output generated by the neural network based on the additional tensor after pruning. as well as The first activation threshold is determined based on the third activation threshold and the additional accuracy of the neural network.

16. One or more non-transitory computer-readable media according to any one of claims 11-15, wherein the operation further comprises: Select another layer in the neural network; as well as The neural network is modified by adding another activation pruning operation to the other layer, which prunes one or more tensors to be generated by the other layer based on another activation threshold.

17. One or more non-transitory computer-readable media according to any one of claims 11-16, wherein selecting the layer comprises: The layer is selected based on the amount of its internal parameters, the amount of computation in the layer, the type of the layer, or a combination thereof.

18. One or more non-transitory computer-readable media according to any one of claims 11-17, wherein the operation further comprises: The neural network is further modified by adding a weight pruning operation to the selected layer. This activation pruning operation is used to prune the kernel of the selected layer based on a weight threshold by setting the absolute value of the weights in the kernel of the selected layer to zero. The absolute value of the weight is lower than the weight threshold.

19. One or more non-transitory computer-readable media according to claim 18, wherein the neural network is trained to determine the absolute values ​​of the weights.

20. One or more non-transitory computer-readable media according to claim 18 or 19, wherein determining the second activation threshold comprises: The second activation threshold is determined after the weight pruning operation is performed on the layer.

21. An apparatus comprising: Computer processor, the computer processor being used to execute computer program instructions; as well as A non-transitory computer-readable storage device stores computer program instructions that can be executed by the computer processor to perform operations for modifying a neural network, the operations including: The dataset is input into the neural network, which includes multiple layers; One layer is selected from the plurality of layers in the neural network, and the selected layer in the neural network generates a tensor based on the dataset; Based on a first activation threshold, the tensor is pruned by modifying the absolute value of activation in the tensor to zero, wherein the absolute value of activation is lower than the first activation threshold. The accuracy of the neural network is determined based on its output, and the neural network generates the output based on the pruned tensor. A second activation threshold is determined based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a value different from the first activation threshold; and The neural network is modified by adding an activation pruning operation to the layer, the activation pruning operation being used to prune one or more tensors to be generated by the layer based on a second activation threshold.

22. The apparatus of claim 21, wherein determining the second activation threshold comprises: The accuracy loss caused by pruning the tensor is determined based on the accuracy of the neural network. Determine whether the accuracy loss exceeds a threshold; In response to determining that the accuracy loss is below the threshold, the second activation threshold is determined by increasing the first activation threshold; as well as In response to determining that the accuracy loss exceeds the threshold, the second activation threshold is determined by decreasing the first activation threshold.

23. The apparatus of claim 21 or 22, wherein the operation further comprises determining the first activation threshold based on a third activation threshold in such a manner as follows: An additional dataset is input into the neural network, and the layers in the neural network generate additional tensors based on the additional dataset; The additional tensor is pruned based on the third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold; The additional accuracy of the neural network is determined based on the additional output generated by the neural network based on the additional tensor after pruning. as well as The first activation threshold is determined based on the third activation threshold and the additional accuracy of the neural network.

24. The apparatus according to any one of claims 21-23, wherein the operation further comprises: The neural network is further modified by adding a weight pruning operation to the layer. This activation pruning operation is used to prune the kernel of the layer based on a weight threshold by setting the absolute value of the weights in the kernel to zero. The absolute value of the weight is lower than the weight threshold.

25. The apparatus according to any one of claims 21-24, wherein the operation further comprises: Select another layer in the neural network; as well as The neural network is modified by adding another activation pruning operation to the other layer, which prunes one or more tensors to be generated by the other layer based on another activation threshold.

Images