Hybrid multiply-accumulate operations with compressed weights

Abstract

The compute block may perform a hybrid multiply-accumulate (MAC) operation. The compute block may include a weight compression module and an array of processing elements (PEs). A weight compression module may select a first set of one or more weights and a second set of one or more weights from weight tensors of a DNN (deep neural network) layer. The weights in the first group are quantized as a power of two. The weights in the second group are quantized as integers. An integer and an exponent of a power of two may be stored in a memory in place of the original value of the weight. A PE in the PE array includes: a shifter configured to shift an activation value of a layer by an exponent of a power of two; and a multiplier configured to multiply the integer by another activation value of the layer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 184,101, filed on Mar. 15, 2023, entitled “HYBRID MULTIPY-ACCUMULATION OPERATION WITH COMPRESSED WEIGHTS,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to neural networks and, more particularly, to hybrid MAC (multiply-accumulate) operations with compressed weights in deep neural networks (DNNs). Background Art

[0004] Because DNNs can achieve high accuracy, they are widely used in a variety of AI applications, from computer vision to speech recognition and natural language processing. However, this high accuracy comes at a significant computational cost. DNNs are computationally demanding, as each inference can require hundreds of millions of MAC operations and extensive data reading and writing. Therefore, technologies that improve DNN efficiency are needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Each embodiment will be readily understood by the following detailed description in conjunction with the accompanying drawings. For ease of description, like reference numerals denote like structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not limitation.

[0006] Figure 1 An example DNN according to various embodiments is shown.

[0007] Figure 2 Example convolutions according to various embodiments are shown.

[0008] Figure 3 is a block diagram of a DNN accelerator according to various embodiments.

[0009] Figure 4 is a block diagram of computational blocks according to various embodiments.

[0010] Figure 5 Hybrid compression of weight operands according to various embodiments is shown.

[0011] Figures 6A-6D Different processes for partitioning the weight tensor according to various embodiments are shown.

[0012] Figure 7 Hybrid MAC operations according to various embodiments are shown.

[0013] Figure 8 An array of processing elements (PEs) is shown in accordance with various embodiments.

[0014] Figure 9 is a block diagram of a PE capable of performing hybrid MAC operations according to various embodiments.

[0015] Figure 10 An example PE capable of performing hybrid MAC operations according to various embodiments is shown.

[0016] Figure 11 Another example PE capable of performing hybrid MAC operations according to various embodiments is shown.

[0017] Figure 12 A configurable PE capable of performing hybrid MAC operations according to various embodiments is shown.

[0018] Figure 13 A PE with a compressor and adder tree is shown according to various embodiments.

[0019] Figure 14 is a flow chart illustrating a method of performing a hybrid MAC operation according to various embodiments.

[0020] Figure 15 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0021] Overview

[0022] The past decade has witnessed the rapid rise of artificial intelligence (AI)-based data processing technologies, particularly those based on deep neural networks (DNNs). DNNs have been widely used in computer vision, speech recognition, image, and video processing, primarily due to their ability to achieve superhuman accuracy. Significant improvements in DNN model size and accuracy, coupled with the rapid increase in the computational power of execution platforms, have enabled DNN applications to be used even in resource-constrained mobile and edge devices with limited energy supplies.

[0023] Deep learning operations in DNNs are becoming increasingly important in data center and edge applications. Examples of deep learning operations in DNNs include convolution (e.g., standard convolution, depthwise convolution, pointwise convolution, group convolution, etc.), matrix multiplication (e.g., matrix multiplication in transformer networks, etc.), deconvolution, pooling operations, element-by-element operations (e.g., element-by-element addition, element-by-element multiplication, etc.), linear operations, nonlinear operations, other types of deep learning operations, or combinations of these operations. One of the main challenges is the substantial increase in computational and memory bandwidth required for these operations. Many deep learning operations (e.g., many convolutions and large matrix multiplications) are performed on large datasets. In addition, although the accuracy of these operations has improved over time, these improvements typically result in a significant increase in model parameter size and the number of operations.

[0024] To reduce the computational and memory bandwidth requirements for executing DNNs, some approaches focus on efficient deep learning network architectures. Other approaches attempt to reduce the computational cost of convolution and matrix multiplication operations. These methods include: pruning weights and skipping MAC operations for pruned weights (whose values are zero); quantizing weights to lower precision values and using lower-precision but lower-cost multipliers; reducing complexity by replacing multiplication operations with shift operations by quantizing weights or activation values to powers of two; and so on.

[0025] For pruning-based and quantization methods, retraining or fine-tuning is often required to restore the performance of the DNN, especially for low bit width or very sparse weights. However, the typical retraining / fine-tuning process has drawbacks. For example, retraining or fine-tuning often requires software infrastructure to enable sparsity. In addition, retraining or fine-tuning can be a time-consuming process, especially for large transformer-based networks. To achieve satisfactory accuracy and acceptable convergence speed, hyperparameters often need to be tuned. Moreover, datasets may not always be available from customers. Therefore, improved techniques are needed to reduce the computational and memory bandwidth requirements for executing DNNs.

[0026] Embodiments of the present disclosure can ameliorate at least some of the challenges and issues described above by compressing weights in a hybrid manner that can facilitate hybrid MAC operations that require fewer computational and memory resources than currently available MAC operations. The hybrid MAC operations can be performed by a combination of multipliers and shifters (e.g., arithmetic shifters).

[0027] In various embodiments of the present disclosure, a DNN accelerator can be used to execute layers in a DNN. A DNN layer (e.g., a convolutional layer) can have an input tensor (also referred to as an "input feature map (IFM)"), a weight tensor including one or more data points (also referred to as "input elements," "input activation values," or "activation values"), a weight tensor including one or more weights, and an output tensor (also referred to as an "output feature map (OFM)"), wherein the input tensor includes one or more data points (also referred to as "input elements," "input activation values," or "activation values"), the weight tensor includes one or more weights, and the output tensor includes one or more data points (also referred to as "output elements," "output activation values," or "activation values"). The output tensor is calculated by performing one or more deep learning operations on the input tensor and the weight tensor. A tensor is a data structure having multiple elements in one or more dimensions. Example tensors include vectors (which are one-dimensional tensors) and matrices (which are two-dimensional tensors). There can also be three-dimensional tensors and even higher-dimensional tensors.

[0028] The DNN accelerator may divide the weight tensor of the DNN layer into sub-tensors. Each sub-tensor includes a subset of the weights in the weight tensor. In some embodiments, the weight tensor may be a four-dimensional tensor. For example, the weight tensor may include filters, each of which is a three-dimensional tensor. The fourth dimension of the weight tensor may be the number of filters in the DNN layer. The weight sub-tensor may have fewer dimensions than the weight tensor. In some embodiments, the weight tensor of the DNN layer may be referred to as the entire weight tensor of the DNN layer, and the weight sub-tensor is referred to as a weight tensor, which is a subset of the entire weight tensor of the DNN layer.

[0029] The DNN accelerator may compress the weight sub-tensors in a hybrid manner. For example, the DNN accelerator selects a first group of one or more weights and a second group of one or more weights from the weight sub-tensors. The DNN accelerator may quantize each weight in the first group to an integer and quantize each weight in the second group to a power of two value. For example, the DNN accelerator may determine an integer or power of two value of the weight based on the original value of the weight, for example, by minimizing the difference between the original value of the weight and the integer or power of two value. For example, the difference between the original value of the weight and the integer (or power of two value) may be less than the difference between the original value of the weight and any other integer (or any other power of two value).

[0030] The DNN accelerator may select the first group based on a predetermined partitioning parameter. The partitioning parameter may indicate a ratio between the number of weights (one or more) in the first group and the total number of weights in the weight sub-tensor. After hybrid compression, the DNN accelerator may store integer and power-of-two exponents instead of the original values of the weights. Compared to the original values of the weights, integer and power-of-two exponents have a smaller storage size because they have fewer bits. Therefore, hybrid compression can reduce memory storage and bandwidth requirements.

[0031] The DNN accelerator includes a PE that can perform hybrid MAC operations using compressed weights. The MAC operations include: multiplication, each multiplication is a multiplication of an activation value and a weight; and accumulation of products calculated from the multiplications. The PE that performs the hybrid MAC operation includes one or more multipliers, one or more shifters, and one or more accumulators. The multiplier can calculate the product between the activation value and the weight quantized to an integer, which is achieved by multiplying the activation value by the integer. The shifter can calculate the product between the activation value and the weight quantized to a power of two value, which is achieved by shifting the activation value by an exponent of the power of two value. The shifter can be an arithmetic shifter. The accumulator can accumulate the outputs of multiple multipliers, the outputs of multiple shifters, or the outputs of at least one multiplier and at least one shifter. Because the DNN accelerator uses one or more shifters instead of multipliers, less area and power are required to execute the DNN layer.

[0032] The shift operation performed by the shifter may be faster than the multiplication operation performed by the multiplier. In some embodiments, the DNN accelerator includes one or more adders (e.g., ripple-carry adders) to accumulate the output of the shifter. These adders are smaller and consume less power. By using such adders, the area or power can be further reduced. Although these adders may be slower, the performance of the DNN accelerator is not affected because the shifter can be faster than the multiplier.

[0033] The present disclosure can reduce inference power and memory bandwidth while maintaining good classification accuracy. Unlike existing quantization and sparsification techniques that require retraining or fine-tuning, the present disclosure can use static calibration to achieve good classification accuracy. In addition, the present disclosure can reduce weight memory bandwidth because the number of bits of the weights is reduced and the weights can be stored in a compressed format. Replacing multipliers with arithmetic shifters can reduce the power (average power and peak power) and area consumed by DNN accelerators. Therefore, compared with current technologies that can be used to execute DNNs, the present disclosure provides a technology that requires fewer computing resources and memory resources.

[0034] For illustrative purposes, specific numbers, materials, and configurations are set forth to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details, or / and may be practiced with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative embodiments.

[0035] In addition, reference is made to the accompanying drawings which form a part thereof, and in which are shown by way of illustration embodiments that may be practiced. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be construed in a limiting sense.

[0036] The various operations may be described as multiple discrete actions or operations in a manner that best facilitates understanding of the claimed subject matter. However, the order of description should not be interpreted as implying that the operations are necessarily order-dependent. In particular, the operations may not be performed in the order presented. The described operations may be performed in an order different from that of the described embodiments. Various additional operations may be performed, or described operations may be omitted in additional embodiments.

[0037] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0038] This description uses the phrases "in one embodiment" or "in an embodiment," each of which may refer to one or more of the same or different embodiments. The terms "including," "comprising," "having," and the like as used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are merely for ease of discussion and do not imply desired or required directions. The drawings are not necessarily drawn to scale. Unless otherwise noted, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects merely indicates different instances of the similar objects referred to, and is not intended to imply that the objects so described must be arranged in a given order, whether in time, space, ranking, or otherwise.

[0039] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0040] The terms "substantially," "close," "approximately," "near," and "about" generally refer to input operands being within + / - 20% of target values based on specific values described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to input operands being within + / - 5-20% of target values based on specific values described herein or known in the art.

[0041] Furthermore, the terms "comprises," "includes," "has," or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Furthermore, the term "or" refers to an inclusive or rather than an exclusive or.

[0042] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0043] Example DNN

[0044] Figure 1 An example DNN 100 is shown in accordance with various embodiments. For illustration purposes, Figure 1 The DNN 100 in FIG is a CNN. In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In one embodiment, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. During inference of DNN 100, the layers of the DNN perform tensor computations including a number of tensor operations, such as convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination of these operations.

[0045] The convolution layer 110 summarizes the presence of features in the input image 105. The convolution layer 110 acts as a feature extractor. The first layer of the DNN 100 is the convolution layer 110. In the example, the convolution layer 110 performs a convolution operation on the input tensor 140 (also referred to as IFM 140) and the filter 150. Figure 1 As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, and each channel is represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also called input points) per row and 7 input elements per column. The filter 150 is represented by a 3×3×3 3D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. The kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. The kernel may be smaller than the IFM. Figure 1 In the embodiment, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel includes 3 weights per row and 3 weights per column. The weights can be initialized and updated using gradient descent via backpropagation. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0046] Convolution involves performing a MAC operation on the input elements in the IFM 140 and the weights in the filter 150. The convolution can be a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides over the IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also called output points) per row and 5 output elements per column. For illustration, in Figure 1 In the embodiment of FIG, the standard convolution includes one filter. In the embodiment where multiple filters are present, the standard convolution can produce multiple output channels in OFM 160.

[0047] The multiplication performed between the kernel-sized local patch of IFM 140 and the kernel can be a dot product. A dot product is an element-by-element multiplication between the kernel-sized local patch of IFM 140 and the corresponding kernel, followed by addition, always producing a single value. Because it produces a single value, this operation is often referred to as a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times with IFM 140 at different points on IFM 140. Specifically, the kernel is systematically applied from left to right and from top to bottom to each overlapping portion or kernel-sized local patch of IFM 140. The result of multiplying the kernel once with IFM 140 is a single value. Since the kernel is applied multiple times to IFM 140, the multiplication result is an output element of a 2D matrix. Therefore, the 2D output matrix from standard convolution 163 (i.e., OFM 160) is referred to as OFM.

[0048] In depthwise convolution 183, the input channels are not combined. Instead, MAC operations are performed on individual input channels and individual kernels, and output channels are generated. Figure 1 As shown, depthwise convolution 183 produces a depthwise output tensor 180. Depthwise output tensor 180 is represented by a 5×5×3 3D matrix. Depthwise output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains five output elements, and each column also contains five output elements. Each output channel is the result of a MAC operation between the input channels of IFM 140 and the kernel of filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation between the first input channel (dot pattern) and the first kernel (dot pattern), the second output channel (horizontal stripe pattern) is the result of a MAC operation between the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern), and the third output channel (diagonal stripe pattern) is the result of a MAC operation between the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, pointwise convolution 193 is performed on the depthwise output tensor 180 and the 1×1×3 tensor 190 to produce the OFM 160 .

[0049] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is a rectified linear unit (ReLU). A ReLU is a calculation that directly returns the value provided as input, or returns a value of 0 if the input is 0 or less. The convolutional layer 110 can receive several images as input and calculate the convolution of each of them with each kernel. This process can be repeated several times. For example, the OFM 160 is passed to a subsequent convolutional layer 110 (i.e., the convolutional layer 110 after the convolutional layer 110 in the sequence that generated the OFM 160). The subsequent convolutional layer 110 performs convolution on the OFM 160 using the new kernel and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be kernelized again by a further subsequent convolutional layer 110, and so on.

[0050] In some embodiments, convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size (e.g., the kernel size is F×F×D pixels), the stride S for dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolutions, such as two-dimensional convolution, dilated or atrous convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. DNN 100 includes 16 convolutional layers 110. In other embodiments, DNN 100 may include a different number of convolutional layers.

[0051] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in local patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the front convolutional layer 110 (the convolutional layer 110 located before the pooling layer 120 in the layer sequence) and the back convolutional layer 110 (the convolutional layer 110 located after the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after the activation function (e.g., ReLU) has been applied to the OFM 160.

[0052] The pooling layer 120 receives the feature maps generated by the preceding convolutional layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the DNN and prevents overlearning. The pooling layer 120 can perform the pooling operation through average pooling (calculating the average value of each local block on the feature map), maximum pooling (calculating the maximum value of each local block on the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied to 2×2 pixels with a step size of 2 pixels, thereby reducing the size of the feature map by a factor of 2, for example, reducing the number of pixels or values in the feature map to one-quarter of its original size. In one example, the pooling layer 120 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps with the same number of pooled feature maps.

[0053] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can be implemented as a convolutional layer or not. The fully connected layer 130 receives input operands. The input operands define the output of the convolutional layer 110 and the pooling layer 120 and include the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operands and generates a vector. This vector can contain as many elements as there are categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the final fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as the activation function.

[0054] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1 In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probability, the fully connected layer 130 multiplies each input element by a weight, sums the values, and then applies an activation function (e.g., logistic if N=2, softmax if N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In one example, the vector includes three probabilities: a first probability indicating that object 115 is a tree, a second probability indicating that object 125 is a car, and a third probability indicating that object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual values may be different.

[0055] Example Convolution

[0056] Figure 2 An example convolution according to various embodiments is shown. The convolution can be a convolutional layer in a DNN (e.g., Figure 1 The convolution may be performed on an input tensor 210 and a filter 220 (individually referred to as "filter 220"). The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator comprising one or more computational blocks. An example of a DNN accelerator may be Figure 3 The DNN accelerator 300 in

[15] is shown in Figure 1. An example computation block can be Figure 4 4. The calculation block 400 in FIG.

[0057] exist Figure 2 In the embodiment of the present invention, the input tensor 210 is a 3D tensor. The input tensor 210 includes activation values (also referred to as "input activation values," "elements," or "input elements") arranged in a 3D matrix. The activation values in the input tensor 210 are data points in the input tensor 210. The input tensor 210 has a spatial size H in ×W in ×C in , where H in is the height of the 3D matrix (i.e., the length along the Y axis, representing the number of activation values in the columns of the 2D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of activation values in a row of the 2D matrix for each input channel), C in is the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of input channels). For simplicity and illustration, the 2D matrix for each input channel has a spatial size of 7×7. In other embodiments, the 2D matrix may have a different spatial size. C in It can be an integer ranging from a small number (e.g., 1, 3, 5, etc.) to a large number (e.g., 100, 500, 1000, or even larger). Each input element in input tensor 210 can be represented by an (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of input tensor 210 can be different.

[0058] Each filter 220 is a 3D tensor. Each filter includes weights arranged in a 3D matrix. The value of the weight can be determined by training the DNN. The filter 220 has a spatial size H f ×W f ×C f , where H f is the height of the filter (i.e., the length along the Y-axis, representing the number of weights in the column for each kernel), Wf is the width of the filter (i.e., the length along the X-axis, representing the number of weights in the row for each kernel), C f is the depth of the filter (i.e., the length along the Z axis, representing the number of channels). In some embodiments, C f Equal to C in For simplicity and illustration, Figure 2 Each kernel in has a spatial size of 3×3. In other embodiments, the height or width of the kernel can be different. The spatial size of the convolution kernel can be smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0059] The number of filters 220 in the convolution may be equal to C out , which is the number of output channels described below. C out It can be an integer ranging from a small number (e.g., 2, 3, 5, etc.) to a large number (e.g., 100, 500, 1000, or even larger). All filters 220 can constitute the weight tensor of convolution 200. The weight tensor is a four-dimensional tensor with H f ×W f ×C in ×C out Although the space size. Figure 2 Multiple filters 220 are shown, but convolution 200 may include a single filter 220 and C out = 1, in which case the weight tensor of convolution 200 is a three-dimensional tensor. In some embodiments, H f and W f can be equal to zero, and the weight tensor of convolution 200 is a two-dimensional tensor.

[0060] The activation value or weight may occupy one or more bytes in memory. The number of bytes of the activation value or weight may depend on the data format. For example, when the activation value or weight has an integer format (e.g., INT8), the activation value occupies one byte. When the activation value or weight has a floating point format (e.g., FP16 or BF16), the activation value or weight occupies two bytes. Other data formats may be used for the activation value or weight. The activation value or weight may be compressed to save memory and computational resources, such as power consumed by processing the activation value or weight, memory storage, data transmission bandwidth, etc.

[0061] In some embodiments, weights may be quantized to integers or powers of two. For example, some weights may be quantized to integers, while other weights may be quantized to powers of two. The exponents of the powers of two or integers may be stored in place of the original values of the weights, thereby saving memory storage and data transfer bandwidth. Additionally, hybrid MAC operations may be performed to compute the output tensor 230. The hybrid MAC operations include integer multiplications, which are generated by quantizing the weights and activation values. Integer multiplications may be faster and consume less energy than floating-point multiplications. The hybrid MAC operations also include shift operations or weights quantized to powers of two. The shift operations may be performed by a shifter, which shifts the corresponding activation values by the exponent of the power of two value. Shifters may be faster or consume less energy than multipliers. More details on quantized weights will be discussed below in conjunction with Figure 4 - Figure 6. More details about the hybrid MAC operation will be described below in conjunction with Figure 4 、 Figure 7 ,as well as Figure 10-13 Provide a description.

[0062] In convolution, each filter 220 slides over the input tensor 210 and produces a 2D matrix for the output channels in the output tensor 230. Figure 2 In the embodiment of FIG, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activation values (also referred to as “output activation values,” “elements,” or “output elements”) arranged in a 3D matrix. The activation values in the output tensor 230 are the data points in the output tensor 230. The output tensor 230 has a spatial size H out ×W out ×C out , where H out is the height of the 3D matrix (i.e., the length along the Y axis, representing the number of output activation values in the column of the 2D matrix for each output channel), W out is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of output activation values in the row of the 2D matrix for each output channel), C out H is the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of output channels). out and W out Can be based on H in 、W in 、H f , and W f . C out It can be equal to the number of filters 220 in the convolution. out and W out It may depend on the height and width of the input tensor 210 and each filter 220 .

[0063] As part of the convolution, the 3×3×C in Subtensor 215 (in Figure 2 MAC operations are performed on the sub-tensor 215 and each filter 220 (highlighted with a dot pattern in FIG). The result of the MAC operation on the sub-tensor 215 and one filter 220 is an output activation value. In some embodiments (e.g., embodiments in which the convolution is an integer convolution), the output activation value may include 8 bits, such as one byte. In other embodiments (e.g., embodiments in which the convolution is a floating-point convolution), the output activation value may include more than one byte. For example, an output element may include two bytes.

[0064] After completing the MAC operation on the sub-tensor 215 and all filters 220, a vector 235 is generated. The vector 235 is Figure 2 . Vector 235 includes a sequence of output activation values arranged along the Z-axis. The output activation values in vector 235 have the same (x, y) coordinates, but these output activation values correspond to different output channels and have different Z-coordinates. The dimension of vector 235 along the Z-axis can be equal to the total number of output channels in output tensor 230.

[0065] After vector 235 is generated, further MAC operations are performed to generate additional vectors until output tensor 230 is generated. For example, filter 220 can be moved along the X-axis or Y-axis on input tensor 210, and a MAC operation can be performed on filter 220 and another sub-tensor in input tensor 210 (the sub-tensor has the same size as filter 220). The amount by which filter 220 moves on input tensor 210 in different computational rounds is called the stride size of the convolution. The stride size can be 1 (i.e., the amount by which filter 220 moves is one activation value), 2 (i.e., the amount by which filter 220 moves is two activation values), and so on. The height and width of output tensor 230 can be determined based on the stride size.

[0066] In some embodiments, a plurality of PEs (e.g., Figure 8 PE 810, Figure 9 PE 900, Figure 10 PE 1000, Figure 11 PE 1100, Figure 12 PE 1200, or Figure 13 PE 1300) for 3×3×C in The sub-tensors (e.g., sub-tensor 215) and filters 220 perform MAC operations. One or more PEs may receive input operands (e.g., Figure 2 ) and weight operands (e.g., Figure 2227). The input operand 217 includes a sequence of activation values having the same (X, Y) coordinates but different Z coordinates. The weight operand 227 includes a sequence of weights having the same (X, Y) coordinates but different Z coordinates. The length of the input operand 217 is the same as the length of the weight operand 227. The activation values in the input operand 217 and the weights in the weight operand 227 can be input to the PE in sequence. The PE can receive a pair of activation values and weights at a time and multiply the activation value by the weight. The position of the activation values in the input operand 217 can match the position of the weights in the weight operand 227.

[0067] Example DNN Accelerator

[0068] Figure 3 is a block diagram of a DNN accelerator 300 according to various embodiments. The DNN accelerator 300 can run a DNN, such as Figure 1 100 in the DNN 100. The DNN accelerator 300 includes a memory 310, a DMA (direct memory access) engine 320, and a computation block 330. In other embodiments, the DNN accelerator 300 may include alternative configurations, different, or additional components. For example, the DNN accelerator 300 may include multiple memories 310 or multiple DMA engines 320. For another example, the DNN accelerator 300 may include a single computation block 330. In addition, the functions implemented by the components of the DNN accelerator 300 may be implemented by other components or other systems included in the DNN accelerator 300.

[0069] Memory 310 stores data that the computation block 330 uses to perform deep learning operations in the DNN model. Example deep learning operations include convolution (also known as "convolution operation"), matrix multiplication (e.g., matrix multiplication in a converter network), pooling operations, element-by-element operations, other types of deep learning operations, or combinations of these operations. Memory 310 can be the main memory of the DNN accelerator 300. In some embodiments, memory 310 includes one or more DRAMs (dynamic random access memories). For example, memory 310 can store input tensors, convolution kernels, or output tensors of a convolution in a convolution layer (e.g., convolution layer 30) of a DNN. The output tensor can be transferred from the local memory of the computation block 330 to the memory 310 via the DMA engine 320.

[0070] The DMA engine 320 facilitates data transfers between the memory 310 and the local memory of the compute block 330. For example, the DMA engine 320 can read data from the memory 310 and write data to the local memory of the compute block 330. For another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to the memory 310. The DMA engine 320 provides DMA functionality, allowing the compute block 330 to initiate data transfers between the memory 310 and the local memory of the compute block 330 and perform other operations while the data transfer is in progress. In some embodiments, the DMA engine 320 can read tensors from the memory 310 and modify the tensors in a manner optimized for the compute block 330 before writing the tensors to the local memory of the compute block 330.

[0071] The computation block 330 performs computations to perform deep learning operations. The computation block 330 may run one or more deep learning operations in a DNN layer, or a portion of a deep learning operation in a DNN layer. The computation block 330 may perform convolutions, such as standard convolutions (e.g., Figure 1 Standard convolution in 163), depthwise convolution (e.g., Figure 1 ), and point-wise convolution (e.g., Figure 1 In some embodiments, the computation block 330 receives an input tensor and one or more convolution kernels and performs convolution on the input tensor and the convolution kernels. The result of the convolution can be an output tensor that can be further computed, for example, by the computation block 330 or another computation block.

[0072] Compute block 330 can facilitate hybrid MAC operations. Hybrid MAC operations include one or more multiplications and one or more shift operations. Compute block 330 can compress the weights of the DNN layer in a hybrid manner. For example, some weights can be compressed into integers, while other weights can be compressed into power-of-two values. In the hybrid MAC operation, the integers can be processed by multipliers, while the exponentials of the power-of-two values can be processed by shifters.

[0073] The computation blocks 330 may also perform other types of deep learning operations, such as matrix multiplication (e.g., matrix multiplication in a transformer network, etc.), pooling operations, element-wise operations, deconvolution, linear operations, nonlinear operations, and the like. The computation blocks 330 may execute one or more DNN layers. In some embodiments, a DNN layer may be executed in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 may each execute a portion of a convolutional workload. Data may be shared between computation blocks 330. Certain aspects of the computation blocks will be discussed below in conjunction with Figure 4 Provide a description.

[0074] Figure 4is a block diagram of a computation block 400 according to various embodiments. The computation block 400 may perform deep learning operations in a DNN during training, inference, or both phases of the DNN. The computation block 400 may be Figure 3 An embodiment of the calculation block 330 in FIG. Figure 4 As shown, the computation block 400 includes a local memory 410, a weight compression module 420, a data storage device 430, and a PE array 440. In other embodiments, the computation block 400 may include alternative configurations, different or additional components. For example, the computation block 400 may include multiple local memories 410, weight compression modules 420, data storage devices 430, and PE arrays 440. In addition, the functions implemented by the components of the computation block 400 may be implemented by other components included in the computation block 400, another component in the DNN accelerator 300, or other systems.

[0075] Local memory 410 is local to the computation block 400. Figure 4 In some embodiments, local memory 410 is located within computing block 400. In other embodiments, local memory 410 may be located outside computing block 400. Local memory 410 and computing block 400 may be implemented on the same chip. In some embodiments, local memory 410 includes one or more SRAMs (static random access memories). Local memory 410 may be byte-addressable, with each memory address identifying a single byte (eight bits) of storage space. In some embodiments, local memory 410 may include memory banks, each of which may have a fixed number of bytes (e.g., 22, 64, etc.).

[0076] The local memory 410 can store input data (e.g., input tensors, filters, etc.) and output data (e.g., output tensors, etc.) of the deep learning operations executed by the computation block 400. A tensor can include elements arranged in a vector, a 2D matrix, a 3D matrix, or a 4D matrix. The data stored in the local memory 410 can be in a compressed format. For example, for a tensor including one or more non-zero-valued elements and one or more zero-valued elements, the local memory 410 can store the one or more non-zero-valued elements without storing the one or more zero-valued elements.

[0077] The local memory 410 can store weights in a hybrid compressed format. For example, the local memory 410 can store integers and power-of-two values generated by quantizing weights in a weight tensor. The local memory 410 can also store other data associated with the deep learning operations executed by the computation block 400, such as a compressed bitmap for hybrid MAC operations. The compressed bitmap can include a plurality of bits, each of which can correspond to a weight in the weight tensor and indicate whether the weight is quantized to an integer or a power-of-two value.

[0078] The weight compression module 420 compresses the weight tensor in a hybrid manner. For example, the weight compression module 420 compresses some weights in the weight tensor by quantizing them to integers, while compressing other weights in the weight tensor by quantizing them to powers of two. The weight compression module 420 can also generate a compression bitmap that indicates which weights are quantized to integers and which weights are quantized to powers of two.

[0079] like Figure 4 As shown, the weight compression module 420 includes a partitioning module 450, a quantization module 460, and a bitmap generator 470. In other embodiments, the weight compression module 420 may include alternative configurations, more or different components. For example, the weight compression module 420 may include multiple partitioning modules 450, quantization modules 460, or bitmap generators 470. In addition, the functions implemented by the components of the weight compression module 420 may be performed by other components included in the weight compression module 420, another component in the computation block 400, another component in the DNN accelerator 300, or other systems.

[0080] The partitioning module 450 partitions the weight tensor of the DNN layer into sub-tensors. In some embodiments, the weight tensor can be a four-dimensional tensor. For example, the weight tensor can include filters, each of which has H f ×W f ×C in A three-dimensional tensor of spatial dimensions, where H f is the height, W f is the width, C in is the depth, which is equal to the number of input channels in the IFM of the DNN layer. The number of filters in the weight tensor can be equal to C out , which is the number of output channels in the output feature map of the DNN layer.

[0081] In some embodiments, the partitioning module 450 may partition the weight tensor into a plurality of weight sub-tensors, each of which may have a 1×1×C in ×C out The weight sub-tensor is a two-dimensional tensor with a width of C in , height C out The number of weights in a row in the weight subtensor is C in , the number of weights in the columns of the weight sub-tensor is C out In other embodiments, the weight tensor may be divided into sub-tensors with different spatial sizes or different dimensions.

[0082] The partitioning module 450 may further partition the weight sub-tensor into a first group and a second group, each group including one or more weights in the weight tensor. Each corresponding weight in the first group may be quantized to an integer, and each corresponding weight in the second group may be quantized to a power of two value. The integers may fall within a range from a small number (e.g., 0, 1, 2, 3, etc.) to a large number (e.g., 100, 500, 1000, etc.). The power of two value may be represented as 2 e , where e is the exponent of the power of two, and the exponent can fall within a range from a smaller number to a larger number. In some embodiments, the weights in the weight sub-tensor are either in the first group or in the second group, rather than being included in both groups at the same time. In some embodiments, the partitioning module 450 can partition the weight sub-tensor based on a predetermined partitioning parameter. The partitioning parameter can indicate the ratio of the number of (one or more) weights in the first group to the total number of weights in the weight sub-tensor. In some embodiments, the partitioning parameter can be a percentage. In one embodiment, the partitioning parameter is represented as p, the total number of weights in the weight sub-tensor is represented as N, and the partitioning module 450 can select p×N weights as the first group and select (1-p)×N weights as the second group.

[0083] In some embodiments, the partitioning module 450 may use the same partitioning parameters to partition multiple weight sub-tensors. In other embodiments, the partitioning module 450 may use different partitioning parameters for different weight sub-tensors. For example, the partitioning module 450 may partition a weight sub-tensor into a first group comprising half of the weights and a second group comprising the other half of the weights, while partitioning another weight sub-tensor into a first group comprising a quarter of the weights and a second group comprising the remaining three quarters of the weights.

[0084] In some embodiments, the weight sub-tensor can be a two-dimensional tensor with weights arranged in rows and columns. In embodiments where the weight sub-tensor has a height H and a width W, N can be equal to H×W. The partitioning module 450 can partition the columns in the weight sub-tensor separately. For example, for each corresponding column, the partitioning module 450 can select one or more weights to include in the first group or select one or more other weights to include in the second group. In some embodiments, the partitioning module 450 can partition the columns by minimizing the Euclidean norm (i.e., L 2 norm) to divide the column, L 2 The norm can be expressed as:

[0085]

[0086] Where i is the index of the weight in the column, n is the total number of weights in the column, and w i is the original value of the weight (i.e. the value of the weight before hybrid compression), w i' is an integer or a power of two value calculated by quantizing the weight (ie, the value of the weight after hybrid compression).

[0087] In some embodiments, the partitioning module 450 may select the same number of weights from each corresponding row of the weight sub-tensor as the first group, for example, for the purpose of balancing the computational workload between computational pipelines. In one example, the weight sub-tensor may include W rows, and the partitioning module 450 may select P weights as the first group by selecting P / W weights from each row of the weight sub-tensor. More details about partitioning the weight sub-tensors will be described below in conjunction with Figures 6A-6D Provide a description.

[0088] Quantization module 460 quantizes each weight in the first group to an integer and quantizes each weight in the second group to a power of two. In some embodiments, for the weights in the first group, quantization module 460 can determine the integer of the weight based on the original value of the weight, for example, by minimizing the difference between the original value of the weight and the integer. For example, the difference between the original value of the weight and the integer (or power of two) may be less than the difference between the original value of the weight and any other integer. The integer may have the same sign (positive or negative) as the original value of the weight. For the weights in the second group, quantization module 460 can determine the power of two of the weight based on the original value of the weight, for example, by minimizing the difference between the original value of the weight and the power of two. For example, the difference between the original value of the weight and the power of two may be less than the difference between the original value of the weight and any other power of two. The power of two may have the same sign (positive or negative) as the original value of the weight.

[0089] The exponents of the integers and powers of two values may be stored in the local memory 410 or the data storage device 430. In some embodiments, the quantization module 460 may receive the original values of the weights from the local memory 410 or the memory 310. The quantization module 460 may store the exponents of the integers and powers of two values generated by quantizing the original values of the weights in the local memory 410 or the data storage device 430. Compared to the original values of the weights, the exponents of the integers and powers of two values have a smaller storage size because they have fewer bits. Therefore, hybrid compression can reduce memory storage and bandwidth requirements.

[0090] The bitmap generator 470 generates a compressed bitmap for the weight sub-tensor compressed by the weight compression module 420. In some embodiments, the bitmap generator 470 generates a compressed bitmap for the weight sub-tensor. The compressed bitmap includes a plurality of bits, each bit corresponding to a corresponding weight in the sub-tensor. The bit indicates whether the corresponding weight is in the first group or the second group, that is, whether the corresponding weight is quantized as an integer or a power of two. In one example, a zero-valued bit indicates that the corresponding weight is quantized as a power of two, and a one-valued bit indicates that the corresponding weight is quantized as an integer. The bits in the compressed bitmap can be arranged in order. The position of the bit in the compressed bitmap can match the position of the corresponding weight in the weight sub-tensor.

[0091] The compressed bitmap generated by the bitmap generator 470 can be stored in the local memory 410 or the data storage device 430. In some embodiments, the weight sub-tensor and its compressed bitmap can be stored as a single data packet. For example, the compressed bitmap can be a header of the data packet. Although the number of bits in the compressed bitmap increases, the total storage size can still be smaller than the storage size of the weight sub-tensor before hybrid compression. Therefore, memory space and bandwidth can still be saved. More details about the compressed bitmap will be discussed below in conjunction with Figure 5 Provide a description.

[0092] The data storage device 430 stores data to be used by the PE array 440 to perform deep learning operations. The data storage device 430 can serve as one or more buffers between the local memory 410 and the PE array 440. Data in the data storage device 430 can be loaded from the local memory 410 and can be transferred to the PE array 440 for calculation. In some embodiments, the data storage device 430 includes one or more data banks. A data bank can include a sequence of memory cells. A memory cell can store a portion of the data in the data bank. In some embodiments, a memory cell can have a fixed storage size, such as 32, 64, or 126 bytes. The number of memory cells in the data storage device 430 can be 8, 16, 32, 64, or the like.

[0093] A memory cell may be a buffer for a PE at a given moment. Data in the memory cell may be fed to one or more PEs during their computation cycles. A memory cell may be a buffer for different PEs during different computation cycles. Data in the memory cell may be fed to the PE array 440 via a MAC lane. A MAC lane is a path used to load data into the PE array 440 or a portion of the PE array 440 (e.g., a PE column in the PE array 440). A MAC lane may also be referred to as a data transmission lane or a data loading lane. The PE array 440 (or PE column) may have multiple MAC lanes. The loading bandwidth of the PE array 440 (or PE column) is the sum of the loading bandwidths of all MAC lanes associated with the PE array 440 (or PE column). In one example, the PE array 440 (or PE column in the PE array 440) may have four MAC lanes, each of which may have a bandwidth of 16 bytes, resulting in a total loading bandwidth of 64 bytes for these four MAC lanes. When there are N MAC lanes (where N is an integer), data can be fed to N PEs simultaneously. In some embodiments (e.g., embodiments where each PE column has a separate MAC path), data in a memory cell can be broadcast to multiple PE columns via the MAC paths of these PE columns. In embodiments where each PE column has multiple separate MAC paths, data in multiple memory cells can be broadcast to multiple PE columns. In one example, where each PE column has four MAC paths, data in four memory cells can be broadcast to multiple PE columns.

[0094] In some embodiments, the data storage device 430 may store at least a portion of an input tensor (e.g., input tensor 210), at least a portion of a weight tensor (e.g., a weight tensor including filter 220), at least a portion of an output tensor (e.g., output tensor 230), or a combination thereof. The storage unit may store at least a portion of an operand (e.g., an input operand or a weight operand). The operand may be a sub-tensor of the input tensor or weight tensor (e.g., a vector, a two-dimensional matrix, or a three-dimensional matrix). The storage unit may also store a compressed bitmap of the weight sub-tensor. In some embodiments (e.g., embodiments in which the local memory 410 stores input data in a compressed format), the input data in the data storage device 430 is in a compressed format. For example, the data storage device 430 stores non-zero activation values or weights, but zero activation values or weights are not stored in the data storage device 430. As another example, for a weight tensor or sub-tensor, the data storage device 430 stores exponents of integers and powers of two generated by quantizing the weights in the weight tensor or sub-tensor.

[0095] The PE array 440 performs MAC operations (including hybrid MAC operations) in convolution. The PE array 440 may also perform other deep learning operations. The PE array 440 may include PEs arranged in columns or in rows and columns. Each PE may perform MAC operations. In some embodiments, the PE includes one or more multipliers for performing multiplication. The PE may also include one or more adders for performing accumulation. A column of PEs is referred to as a PE column. A PE column may be associated with one or more MAC paths. A MAC path is a path for loading data into the PE column. A MAC path may also be referred to as a data transmission path or a data loading path. A PE column may have multiple MAC paths. The loading bandwidth of a PE column is the sum of the loading bandwidths of all MAC paths associated with the PE column. Using a specific number of MAC paths, data can be input into the same number of independent PEs simultaneously. In some embodiments, a PE column has four MAC paths for inputting activation values or weights into the PE column. Each MAC path may have a bandwidth of 16 bytes, and the four MAC paths may have a total loading bandwidth of 64 bytes.

[0096] In some embodiments, the PE array 440 may be capable of performing standard convolution, depthwise convolution, pointwise convolution, other types of convolution, or a combination of these. In depthwise convolution, the PE may perform a MAC operation that includes a sequence of multiplications of an input operand (e.g., input operand 217) and a weight operand (e.g., weight operand 227). Each multiplication in the sequence is a multiplication of a different activation value in the input operand and a different weight in the weight operand. The activation value and weight in the same cycle may correspond to the same channel. The multiplication sequence generates a product operand that includes a product sequence. The MAC operation may also include accumulation, wherein multiple product operands are accumulated to generate an output operand of the PE. The PE array 440 may output multiple output operands simultaneously, each output operand generated by a different PE. In standard convolution, the MAC operation may include accumulation across channels. For example, instead of generating output operands, the PE may accumulate products across different channels to generate a single output point.

[0097] In some embodiments, a PE can perform multiple rounds of MAC operations for convolution. Data (activations, weights, or both) can be reused within a single round (e.g., between different multipliers in a PE) or between different rounds of MAC operations. More details about the PE array will be provided below in conjunction with Figure 5 6 for description.

[0098] In some embodiments (e.g., embodiments in which computation block 400 performs a convolutional layer), the computation in the PE may be a MAC operation performed on an input operand and a weight operand. The input operand may be part of an input tensor for the convolution. The input operand includes a sequence of input elements (i.e., activation values). The activation values may come from different input channels. For example, each activation value comes from an input channel that is different from all other activation values in the input operand. The weight operand may be part of the kernel of the convolution. The weight operand includes a sequence of weights. The values of these weights are determined by training the DNN. The weights in the weight operand may come from different input channels. For example, each weight comes from an input channel that is different from all other weights in the weight operand. The PE may perform a multiplication operation on each activation-weight pair by multiplying the activation value by the corresponding weight. The position of the activation value in the input operand may match (e.g., be identical to) the position of the corresponding weight in the weight operand. The PE may also perform a partial sum of the products of the activation-weight pairs to calculate the MAC operation.

[0099] In some embodiments, the PE may perform a hybrid MAC operation in which weights have been compressed in a hybrid manner. The PE may include one or more multipliers, one or more shifters, and one or more accumulators. The weights may be distributed to the one or more multipliers and one or more shifters based on a compression bitmap associated with the weights. For example, the weights corresponding to the bits in the compression bitmap indicating that the weights are quantized to integers are transferred to the multipliers. The multipliers may multiply the weights with the corresponding activation values. The weights corresponding to the bits in the compression bitmap indicating that the weights are quantized to powers of two are transferred to the shifters. The shifters may shift the corresponding activation values by the exponent of the power of two (e.g., shift to the left). The one or more accumulators may add the outputs of the one or more multipliers and the one or more shifters and generate a partial sum of the hybrid MAC operation. More details about the hybrid MAC operation will be described below in conjunction with Figure 7 and Figure 10-13 More details about PE will be described below. Figure 8 and Figure 9 Provide a description.

[0100] Example Hybrid Compression

[0101] Figure 5 shows a hybrid compression of weight operands 510 according to various embodiments. Hybrid compression can be achieved by Figure 4 The weight operation 510 includes eight weights, each weight in Figure 5denoted by a box in FIG. Weight operand 510 may be a sub-tensor of a weight tensor of a DNN layer. For illustration purposes, weight operand 510 is a vector. In other embodiments, weight operand 510 may be a two-dimensional or three-dimensional tensor. Additionally, weight operand 510 may include other numbers of weights.

[0102] Four weights in weight operand 510 (represented by shaded boxes) are selected to be quantized to powers of two. The other four weights (represented by white boxes) are selected to be quantized to integers. After hybrid compression, compressed weight operand 520 is generated. Compressed weight operand 520 includes the same number of elements as weight operand 510. The weights selected to be quantized to integers have integer values in compressed weight operand 520. For example, in an embodiment where the original value of the weight is an integer, the integer value can be the original value; in an embodiment where the original value is not an integer (e.g., a floating-point value), the integer value is different from the original value and may have fewer bits than the original value. For weights selected to be quantized to powers of two, compressed weight operand 520 has an exponent of a power of two, which has fewer bits than the original value of the weight. Compressed weight operand 520 has fewer bits than weight operand 510 and therefore requires less memory storage space and bandwidth.

[0103] The compressed weight operand 520 is associated with a compressed bitmap 530, which includes eight bits. Each of the eight bits corresponds to a weight in the weight operand 510 and indicates whether the weight is quantized to an integer or a power of two value. Figure 5 In the embodiment of FIG5 , a bit of 1 indicates that the weight is quantized to an integer, while a bit of 0 indicates that the weight is quantized to a power of two value. The compressed bitmap 530 can be stored in a memory (e.g., local memory 410 or data storage device 430) as a header for the compressed weight operand 520. Although this adds eight extra bits to the data, the total number of bits of the compressed bitmap 530 plus the compressed weight operand 520 is still less than the number of bits of the weight operand 510.

[0104] Example weight tensor partitioning

[0105] Figures 6A-6D 610 according to various embodiments. Figure 4 The partitioning module 450 in is executed. Figure 6A A weight tensor 610 having a spatial size of 4×4 is shown, i.e., the weight tensor 610 is a two-dimensional tensor comprising 16 weights arranged in four columns and four rows. Each weight in the weight tensor 610 is Figures 6A-6Ddenoted by a box in the figure. Weight tensor 610 may be a sub-tensor of the entire weight tensor of the DNN layer. In other embodiments, weight tensor 610 may be a vector or a three-dimensional tensor. Additionally, weight tensor 610 may include a different number of weights.

[0106] exist Figures 6B-6D In the embodiment of , the weight tensor 610 is divided based on a division parameter equal to 0.25, which means that the weight tensor 610 is divided into a first group including one quarter of the 16 weights (i.e., four weights) and a second group including three quarters of the 16 weights (i.e., 12 weights). Figure 6B In , the four weights in the first row are selected as the first group. These four weights can be selected by dividing the four columns separately. Figure 6C In , two weights in the first row, one weight in the second row, and one weight in the fourth row are selected as the first group. These four weights can be selected by partitioning the four columns separately, for example, by minimizing the distance norm of the weights before and after hybrid compression.

[0107] exist Figure 6D In , the four weights in the first group come from all four rows, and each corresponding row selects one weight. In some embodiments, in order to balance the workload of the computing pipeline, Figure 6B After the division is executed Figure 6D division. Figure 6B The partitioning of may result in an unbalanced pipeline because the rows in the weight tensor 610 do not have the same number of power-of-two values, whereas Figure 6D The partitioning allows for a balanced pipeline because all four rows have the same number of power-of-two values.

[0108] Example Hybrid MAC Operation

[0109] Figure 7 1 shows a hybrid MAC operation 700 according to various embodiments. The hybrid MAC operation 700 may be performed by a PE (e.g., Figure 4 The hybrid MAC operation 700 has two inputs: a weight operand 710 and an input operand 720. The weight operand 710 includes eight weights, each of which is in Figure 7denoted by a box in . Weight operand 710 can be a sub-tensor of a weight tensor of a DNN layer. For illustration purposes, weight operand 710 is a vector. In other embodiments, weight operand 710 can be a two-dimensional or three-dimensional tensor. Additionally, weight operand 710 can include other numbers of weights. Input operand 720 includes eight activation values, each corresponding activation value corresponding to a different weight in weight operand 710. For example, activation value d1 corresponds to weight w1, activation value d2 corresponds to weight w2, activation value d3 corresponds to weight w3, and so on.

[0110] In a conventional MAC operation, a multiplication operation may be performed on each activation value-weight pair, and the products of the multiplication operations may be accumulated to generate a partial sum. In the hybrid MAC operation 700, a shift operation is also performed. A subset of the weight operands 710 (i.e., Figure 7 The weights w2, w4, w7, and w8 represented by the shaded boxes in FIG are quantized to power-of-two values. The other weights (i.e., w1, w3, w5, and w6) are quantized to integers.

[0111] In four multiplications 730 (referred to as "multiplication 730"), four integers are multiplied by corresponding activation values (i.e., d1, d3, d5, and d6) by four multipliers. The multipliers can be integer multipliers. The other four activation values (i.e., d2, d4, d7, and d8) are shifted by four power-of-two exponents by four shifters in four shift operations 740 (referred to as "shift operation 740"). The outputs of multiplications 730 and shift operations 740 are summed in accumulation 750 to generate a partial sum for the hybrid MAC operation.

[0112] Example PE array

[0113] Figure 8 PE array 800 according to various embodiments is shown. PE array 800 may be Figure 4 8. PE array 800 includes multiple PEs 810 (individually referred to as "PE 810"). PE 810 performs MAC operations. PE 810 can also be referred to as a neuron in a DNN. Each PE 810 has two input signals 850 and 860 and an output signal 870. Input signal 850 is at least a portion of an IFM of a layer. Input signal 860 is at least a portion of a filter of a layer. In some embodiments, input signal 850 of PE 810 includes one or more input operands, and input signal 860 includes one or more weight operands.

[0114] Each PE 810 performs a MAC operation on input signals 850 and 860 and outputs an output signal 870 as a result of the MAC operation. Some or all of the input signals 850 and 860 and the output signal 870 may be in integer format (e.g., INT8) or floating-point format (e.g., FP16 or BF16). For simplicity and illustration, the input signals and output signals of all PEs 810 use the same reference numerals, but PEs 810 may receive different input signals and output different output signals. Furthermore, a PE 810 may differ from another PE 810, for example, including more, fewer, or different components.

[0115] like Figure 8 As shown, PEs 810 are interconnected, as shown in FIG. Figure 8 As shown by the dashed arrows in FIG. PE 810 output signal 870 can be sent as an input signal to many other PEs 810 (and possibly back to itself) via the interconnections between PEs 810. In some embodiments, the output signal 870 of PE 810 can be combined with the output signals of one or more other PEs 810 by an accumulation operation of PE 810 to generate an internal partial sum of the PE array. More details about PE 810 will be discussed below in conjunction with Figure 8 Provide a description.

[0116] exist Figure 8 In an embodiment, the PEs 810 are arranged into columns 805 (individually referred to as "columns 805"). The inputs and weights of a layer can be distributed to the PEs 810 based on the columns 805. Each column 805 has a column buffer 820. The column buffer 820 stores data provided to the PEs 810 in the column 805 for a short period of time. The column buffer 820 can also store data output by the last PE 810 in the column 805. The output of the last PE 810 can be the sum of the MAC operations of all the PEs 810 in the column 805, which is the column-level internal partial sum of the PE array 800. In other embodiments, the inputs and weights can be distributed to the PEs 810 based on the rows in the PE array 800. The PE array 800 can include a row buffer instead of the column buffer 820. The row buffer can store the input signals of the PEs in the corresponding row and can also store the row-level internal partial sum of the PE array 800.

[0117] like Figure 8 As shown, each column buffer 820 is associated with a loader 830 and a drain 840. Data provided to a column 805 is transferred to the column buffer 820 by the loader 830, for example, via an upper memory hierarchy, e.g. Figure 4 The data generated by the column 805 is extracted from the column buffer 820 by the ejector 840. The column buffer 820 can be Figure 4 In some embodiments, the data extracted from the column buffer 820 is sent to the upper memory hierarchy via a drain operation, e.g. Figure 4 In some embodiments, the drain operation does not begin until all PEs 810 in column 805 have completed their MAC operations. Figure 8 Not shown, but one or more columns 805 may be associated with external adder components.

[0118] Example PE

[0119] Figure 9 is a block diagram of a PE 900 according to various embodiments. The PE 900 may be Figure 4 An embodiment of a PE in the PE array 440 or Figure 8 810. PE 900 includes an input register file 910 (individually referred to as "input register file 910"), a weight register file 920 (individually referred to as "weight register file 920"), a multiplier 930 (individually referred to as "multiplier 930"), a shifter 935 (individually referred to as "shifter 935"), a first adder component 940, a second adder component 945, and an output register file 960. In other embodiments, PE 900 may include fewer, more, or different components. For example, PE 900 may include multiple output register files 960. For another example, PE 900 may include a single input register file 910, weight register file 920, multiplier 930, or shifter 935. For another example, PE 900 may not include the first adder component 940 or the second adder component 945.

[0120] The input register file 910 temporarily stores input operands for MAC operations performed by the PE 900. In some embodiments, the input register file 910 can store a single input operand at a time. In other embodiments, the input register file 910 can store multiple input operands or a portion of an input operand at a time. An input operand includes multiple input elements (i.e., input elements) in an input tensor. The input elements in the input operands can be stored sequentially in the input register file 910 so that the input elements can be processed sequentially. In some embodiments, each input element in the input operand can come from a different input channel of the input tensor. The input operand can include input elements from each input channel of the input tensor, and the number of input elements in the input operand can be equal to the number of input channels. The input elements in the input operand can have the same XY coordinates, which can be used as the XY coordinates of the input operand. For example, all input elements of the input operand can be X0Y0, X0Y1, X1Y1, etc. In some embodiments, one or more input register files 910 can store non-zero-valued elements of the input operand, but not zero-valued elements of the input operand.

[0121] The weight register file 920 temporarily stores weight operands for MAC operations performed by the PE 900. Weight operands include weights in filters of the DNN layer. In some embodiments, the weight register file 920 can store a single weight operand at a time. In other embodiments, the weight register file 920 can store multiple weight operands or a portion of a weight operand at a time. The weight operand may include multiple weights. The weights in the weight operands may be stored sequentially in the weight register file 920 so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and input operands, each weight in the weight operand may correspond to an input element in the input operand. The number of weights in the weight operand may be equal to the number of input elements in the input operand.

[0122] In some embodiments, one or more weight register files 920 may store (one or more) non-zero valued weights of weight operands, without storing (one or more) zero valued weights of weight operands. Additionally or alternatively, one or more weight register files 920 may store weights compressed in a hybrid manner. For example, one or more weight register files 920 may store one or more integers and one or more power-of-two exponents of weight operands.

[0123] In some embodiments, the weight register file 920 can be the same as or similar to the input register file 910, for example, having the same size. PE 900 may include multiple register files, some of which are designated as input register files 910 for storing input operands, some as weight register files 920 for storing weight operands, and some as output register files 960 for storing output operands. In other embodiments, register files in PE 900 may be designated for other purposes, such as for storing scaling operands used in element-wise addition operations. The designation of register files may be controlled by control module 340.

[0124] The multiplier 930 performs a multiplication operation on the activation value and the weight. The multiplier 930 can perform a sequence of multiplication operations and generate a sequence of products. Each multiplication operation in the sequence includes multiplying the activation value by the corresponding weight. In some embodiments, the position (or index) of the activation value in the input operand matches the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first activation value in the input operand and the first weight in the weight operand, the second multiplication operation is the multiplication of the second activation value in the input operand and the second weight in the weight operand, the third multiplication operation is the multiplication of the third activation value in the input operand and the third weight in the weight operand, and so on. The activation value and weight in the same multiplication operation can correspond to the same input channel, and their product can also correspond to the same input channel.

[0125] Multiple multipliers 930 can perform multiplication operations simultaneously. These multiplication operations can be referred to as a round of multiplication operations. In a round of multiplication operations performed by these multipliers 930, each multiplier 930 can use a different set of activation values and a different set of weights. The different sets of activation values or weights can be stored in different register files of the PE 900. For example, a first multiplier 930 uses a first set of activation values (e.g., stored in the first input register file 910) and a first set of weights (e.g., stored in the first weight register file 920), a second multiplier 930 uses a second set of activation values (e.g., stored in the second input register file 910) and a second set of weights (e.g., stored in the second weight register file 920), a third multiplier 930 uses a third set of activation values (e.g., stored in the third input register file 910) and a third set of weights (e.g., stored in the third weight register file 920), and so on. For an individual multiplier 930, a round of multiplication operations may include multiple cycles. A cycle includes a multiplication operation on an activation value and a weight.

[0126] The multiplier 930 can perform multiple rounds of multiplication operations. The multiplier 930 can use the same (one or more) weights but different activation values in different rounds. For example, the multiplier 930 performs a sequence of multiplication operations on a first set (one or more) activation values stored in a first input register file in a first round, and performs a sequence of multiplication operations on a second set (one or more) activation values stored in a second input register file in a second round. In the second round, another multiplier 930 can perform another sequence of multiplication operations using the first set (one or more) activation values and another set (one or more) weights. In this way, the first set (one or more) activation values are reused in the second round. The first set (one or more) activation values can be further reused in additional rounds, for example, by additional multipliers 930.

[0127] The shifter 935 performs a shift operation on the activation value and the exponent of the power of two value obtained from the weight quantization. The shifter can be an arithmetic shifter, a logical shifter, a barrel shifter, or other types of shifters. The shifter 935 can be a left shifter. The shifter 935 can perform a sequence of shift operations and generate a sequence of products, each product being the product of the activation value and the corresponding weight. Each shift operation in the sequence includes shifting the activation value left by the exponent of the power of two value obtained by the corresponding weight quantization. The output of the shift operation can be the product of the activation value multiplied by the power of two value. In some embodiments, the position (or index) of the activation value in the input operand matches the position (or index) of the weight in the weight operand. For example, the first shift operation is for the first activation value in the input operand and the first weight in the weight operand, the second shift operation is for the second activation value in the input operand and the second weight in the weight operand, the third shift operation is for the third activation value in the input operand and the third weight in the weight operand, and so on. The activation values and weights in the same shift operation can correspond to the same input channel, and their products can also correspond to the same input channel.

[0128] Multiple shifters 935 can perform shift operations simultaneously. These shift operations can be referred to as a round of shift operations. In a round of shift operations performed by the shifters 935, each shifter 935 can use a different set (one or more) of activation values and a different set (one or more) of weights. These different sets (one or more) of activation values or different sets (one or more) of weights can be stored in different register files of the PE 900. For example, the first shifter 935 uses a first set (one or more) of activation values (e.g., stored in the first input register file 910) and a first set (one or more) of weights (e.g., stored in the first weight register file 920), while the second shifter 935 uses a second set (one or more) of activation values (e.g., stored in the second input register file 910) and a second set (one or more) of weights (e.g., stored in the second weight register file 920), the third shifter 935 uses a third set (one or more) of activation values (e.g., stored in the third input register file 910) and a third set (one or more) of weights (e.g., stored in the third weight register file 920), and so on. For an individual shifter 935, a round of shifting operations may include multiple cycles. A cycle includes shifting operations on activation values and weights.

[0129] Shifter 935 can perform multiple rounds of shift operations. Shifter 935 can use the same (one or more) weights but different activation values in different rounds. For example, shifter 935 performs a sequence of shift operations on a first group (one or more) activation values stored in a first input register file in a first round, and performs a sequence of shift operations on a second group (one or more) activation values stored in a second input register file in a second round. In the second round, a different shifter 935 can perform another sequence of shift operations using the first group (one or more) activation values and a different group (one or more) weights. In this way, the first group (one or more) activation values can be reused in the second round. The first group (one or more) activation values can be further reused in additional rounds, for example, by additional shifters 935.

[0130] The first adder component 940 includes one or more adders (i.e., internal adders) within the PE 900. The first adder component 940 is coupled to the multiplier 930. The first adder component 940 can perform an accumulation operation on two or more product operands from the multiplier 930 and generate a multiplication sum. The first adder component 940 can include one or more compressors (e.g., a 3-2 compressor that receives three inputs and generates two outputs), ripple carry adders, prefix adders, other types of adders, or combinations thereof.

[0131] In some embodiments, the internal adders can be arranged in a tier sequence. The tier includes one or more internal adders. For the first tier of the first adder component 940, the internal adder can receive outputs from two or more multipliers 930 and generate a sum in an individual accumulation cycle. For the other tiers (one or more) of the first adder component 940, the internal adder in the tier can sum two or more outputs of the previous tier in the sequence. Each of these outputs can be generated by a different internal adder in the previous tier. The ratio of the number of internal adders in a tier to the number of internal adders in a subsequent tier can be 2:1. In some embodiments, the last tier of the first adder component 940 can include a single internal adder that generates a multiplication partial sum.

[0132] The second adder component 945 includes one or more other adders within the PE 900, i.e., other internal adders. The second adder component 945 is coupled to the shifter 935. The second adder component 945 can perform an accumulation operation on two or more outputs from the shifter 935 and generate a shifted sum. The second adder component 945 can include one or more compressors (e.g., a 3-2 compressor that receives three inputs and generates two outputs), a ripple carry adder, a prefix adder, other types of adders, or combinations thereof.

[0133] In some embodiments, the internal adders in the second adder component 945 can be arranged into a hierarchical sequence. The hierarchy includes one or more internal adders. For the first hierarchy of the second adder component 945, the internal adder can receive outputs from two or more shifters 935 and generate the sum of these outputs in an individual accumulation cycle. For (one or more) other hierarchies of the second adder component 945, the internal adder in the hierarchy can sum two or more outputs from the previous hierarchy in the sequence. Each of these outputs can be generated by a different internal adder in the previous hierarchy. The ratio of the number of internal adders in the hierarchy to the number of internal adders in the subsequent hierarchy can be 2:1. In some embodiments, the last hierarchy of the second adder component 945 can include a single internal adder that generates a shifted partial sum.

[0134] The accumulator 950 may add the multiplication partial sum and the shifted partial sum to generate the partial sum of the PE 900. In some embodiments, the accumulator 950 may be an adder. Figure 9The internal accumulator 950 is a separate component from the first adder component 940 and the second adder component 945 in the PE 900, but in other embodiments, the accumulator 950 can be implemented in the first adder component 940 or the second internal adder component. For example, the accumulator 950 can be an adder in the last level of the internal adder tree in the first adder component 940 or the second internal adder component.

[0135] Output register file 960 stores one or more output activation values calculated by PE 900. In some embodiments, output register file 960 can store one output activation value at a time. In other embodiments, output register file 960 can store multiple output activation values at a time. The output activation value can be a partial sum of PE 900, which is calculated by accumulator 950. In some embodiments, accumulator 950 can receive one or more partial sums of one or more other PEs and accumulate the partial sum of PE 900 with the one or more partial sums. The sum of the partial sum of PE 900 and the one or more partial sums can be a partial sum of a group of PEs (e.g., a PE column (e.g., PE column 805)).

[0136] Figure 10 An example PE 1000 capable of performing hybrid MAC operations according to various embodiments is shown. The PE 1000 may be Figure 9 1000 includes an input register file 1010, a weight register file 1020, four multipliers 1030 (individually referred to as "multiplier 1030"), four shifters 1040 (individually referred to as "shifter 1040"), an accumulator 1050, and an output register file 1060. In other embodiments, PE 1000 may include fewer, more, or different components. Input register file 1010 may be Figure 9 10. The weight register file 1020 may be an embodiment of the weight register file 920. The multiplier 1030 may be an embodiment of the multiplier 930. The shifter 1040 may be an embodiment of the shifter 935. The accumulator 1050 may be an embodiment of the accumulator 950. The output register file 1060 may be an embodiment of the output register file 960.

[0137] Each multiplier 1030 receives an activation value from the input register file 1010 and an integer-valued weight from the weight register file 1020. The multiplier 1030 can receive different activation value-weight pairs. The integer value of the weight can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The integer value can have a single byte. Each multiplier 1030 multiplies the activation value by the integer value. In some embodiments, the multiplier 1030 can be an integer multiplier. The output of the multiplier 1030 is transmitted to the accumulator 1050.

[0138] Each shifter 1040 receives an activation value from the input register file 1010 and an exponent of a power of two from the weight register file 1020. The power of two value can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The shifters 1040 can receive different activation value-weight pairs. Each shifter 1040 shifts the bits in the activation value left by the exponent. In some embodiments, the shifters 1040 can be arithmetic shifters. The output of the shifters 1040 is transmitted to the accumulator 1050.

[0139] The bits in the compressed bitmap can be used to determine whether to send the activation value-weight pair to the multiplier 1030 or to the shifter 1040. For example, the determination can be based on the value of the bit corresponding to the weight in the activation value-weight pair. In embodiments where the bit value is one, the activation value-weight pair can be sent to the multiplier 1030. In embodiments where the bit value is zero, the activation value-weight pair can be sent to the shifter 1040. In some embodiments, the calculations in the shifter 1040 can be faster than those in the multiplier 1030. The shifter 1040 has a simpler structure and a smaller gate depth than the multiplier 1030. The path length through the shifter 1040 can be shorter than the path length through the multiplier 1030. In some embodiments, the complexity of one or more shifters 1040 can be further reduced by limiting the shift range, for example, where larger shifts are rare.

[0140] Accumulator 1050 accumulates the outputs of multiplier 1030 and shifter 1040 and generates a partial sum for PE 1000. The partial sum may be stored in output register file 1060. In some embodiments, accumulator 1050 may receive one or more partial sums of one or more other PEs and accumulate the partial sum of PE 1000 with the one or more partial sums. The sum of the partial sum of PE 1000 and the one or more partial sums may be a partial sum for a group of PEs (e.g., a PE column (e.g., PE column 805)). The partial sum for the group of PEs may be stored in output register file 1060. In some embodiments, the partial sum for the group of PEs may be further accumulated with one or more other partial sums by PE 1000 or another PE.

[0141] although Figure 10 Four multipliers 1030 and four shifters 1040 are shown, but a PE may include other numbers of multipliers 1030 or other numbers of shifters 1040. The total number of multipliers 1030 and shifters 1040 may be less than or greater than eight. Furthermore, the ratio of the number of multipliers 1030 to the number of shifters 1040 in a PE may vary.

[0142] Figure 11 Another example PE 1100 capable of performing hybrid MAC operations according to various embodiments is shown. The PE 1100 may be Figure 9 1. An embodiment of PE 900 in FIG. PE 1100 includes an input register file 1110, a weight register file 1120, two multipliers 1130 (individually referred to as "multiplier 1130"), six shifters 1140 (individually referred to as "shifter 1140"), an accumulator 1150, and an output register file 1160. In other embodiments, PE 1100 may include fewer, more, or different components. Input register file 1110 may be Figure 9 11. The input register file 910 may be an embodiment of the weight register file 920. The weight register file 1120 may be an embodiment of the weight register file 920. The multiplier 1130 may be an embodiment of the multiplier 930. The shifter 1140 may be an embodiment of the shifter 935. The accumulator 1150 may be an embodiment of the accumulator 950. The output register file 1160 may be an embodiment of the output register file 960.

[0143] Each multiplier 1130 receives an activation value from the input register file 1110 and an integer-valued weight from the weight register file 1120. The multiplier 1130 can receive different activation value-weight pairs. The integer value of the weight can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The integer value can have a single byte. Each multiplier 1130 multiplies the activation value by the integer value. In some embodiments, the multiplier 1130 can be an integer multiplier. The output of the multiplier 1130 is transmitted to the accumulator 1150.

[0144] Each shifter 1140 receives an activation value from the input register file 1110 and an exponent of a power of two from the weight register file 1120. The power of two value can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The shifters 1140 can receive different activation value-weight pairs. Each shifter 1140 shifts the bits in the activation value left by the exponent. In some embodiments, the shifters 1140 can be arithmetic shifters. The output of the shifters 1140 is transmitted to the accumulator 1150.

[0145] The bits in the compressed bitmap can be used to determine whether to send the activation value-weight pair to the multiplier 1130 or to the shifter 1140. For example, the determination can be based on the value of the bit corresponding to the weight in the activation value-weight pair. In embodiments where the bit value is one, the activation value-weight pair can be sent to the multiplier 1130. In embodiments where the bit value is zero, the activation value-weight pair can be sent to the shifter 1140. In some embodiments, the calculations in the shifter 1140 can be faster than the calculations in the multiplier 1130. The shifter 1140 has a simpler structure and a smaller gate depth than the multiplier 1130. The path length through the shifter 1140 can be shorter than the path length through the multiplier 1130. In some embodiments, the complexity of one or more shifters 1140 can be further reduced by limiting the shift range, for example, where larger shifts are rare.

[0146] Accumulator 1150 accumulates the outputs of multiplier 1130 and shifter 1140 and generates a partial sum for PE 1100. The partial sum may be stored in output register file 1160. In some embodiments, accumulator 1150 may receive one or more partial sums of one or more other PEs and accumulate the partial sum of PE 1100 with the one or more partial sums. The sum of the partial sum of PE 1100 and the one or more partial sums may be a partial sum for a group of PEs (e.g., a PE column (e.g., PE column 805)). The partial sum for the group of PEs may be stored in output register file 1160. In some embodiments, the partial sum for the group of PEs may be further accumulated with one or more other partial sums by PE 1100 or another PE.

[0147] The number of multipliers or shifters in a PE is not necessarily fixed. In some embodiments, the number of multipliers or shifters in a PE can be flexible. Figure 12 FIG2 shows a configurable PE 1200 capable of performing hybrid MAC operations according to various embodiments. The PE 1200 may be Figure 9 12. PE 1200 includes an input register file 1210, a weight register file 1220, three multiplier-shifter pairs 1270, three selectors 1280, an accumulator 1250, and an output register file 1260. Each multiplier-shifter pair 1270 includes a multiplier 1230 and a shifter 1240. The multiplier-shifter pairs 1270 are each coupled to a selector 1280. In other embodiments, PE 1200 may include fewer, more, or different components. Input register file 1210 may be Figure 9 10. The weight register file 1220 may be an embodiment of the weight register file 920. The multiplier 1230 may be an embodiment of the multiplier 930, the multiplier 1030, or the multiplier 1130. The shifter 1240 may be an embodiment of the shifter 935, the shifter 1040, or the shifter 1140. The accumulator 1250 may be an embodiment of the accumulator 950. The output register file 1260 may be an embodiment of the output register file 960.

[0148] Each multiplier-shifter pair 1270 receives an activation value from the input register file 1210 and a data element from the weight register file 1220. A selector 1280 receives a bit from the weight register file 1220. This bit is located in a compressed bitmap and corresponds to a data element from the weight register file 1220. Based on the value of this bit, selector 1280 can transfer the activation value and data element to either the multiplier 1230 or the shifter 1240 in the multiplier-shifter pair 1270. In embodiments where this bit is a one, this indicates that the data element is an integer generated by quantizing the weights. The activation value and data element are sent to the multiplier 1230. The multiplier 1230 multiplies the activation value by the data element. In embodiments where this bit is a zero, this indicates that the data element is an exponent of a power of two generated by quantizing the weights. The activation value and data element are sent to the shifter 1240. The shifter 1240 shifts the bits in the activation value left by the exponent.

[0149] Thus, the number of multipliers 1230 or shifters 1240 active during a computation cycle of the PE 1200 may be dynamic. This number may vary based on the number of integer or power-of-two values in the weight subtensors processed by the PE 1200. Figure 12 Three multiplier-shifter pairs 1270 and three selectors 1280 are shown, but the PE may include other numbers (one or more) of multiplier-shifter pairs 1270 or other numbers of selectors 1280. In some embodiments, the PE may be partially configurable. For example, in addition to a single multiplier (e.g., multiplier 1030 or 1130) or a single shifter (e.g., shifter 1040 or 1140), the PE may include one or more multiplier-shifter pairs and one or more selectors.

[0150] Accumulator 1250 accumulates the outputs of multiplier-shifter pair 1270 and generates a partial sum for PE 1200. The partial sum may be stored in output register file 1260. In some embodiments, accumulator 1250 may receive one or more partial sums of one or more other PEs and accumulate the partial sum of PE 1200 with the one or more partial sums. The sum of the partial sum of PE 1200 and the one or more partial sums may be a partial sum for a group of PEs (e.g., a PE column (e.g., PE column 805)). The partial sum for the group of PEs may be stored in output register file 1260. In some embodiments, the partial sum for the group of PEs may be further accumulated with one or more other partial sums by PE 1200 or another PE.

[0151] Figure 1313. PE 1300 is shown with compressor 1370 and adder tree 1375 according to various embodiments. PE 1300 may be Figure 9 13. PE 1300 also includes an input register file 1310, a weight register file 1320, four multipliers 1330 (individually referred to as "multiplier 1330"), four shifters 1340A-1340D (collectively referred to as "shifters 1340"), an accumulator 1350, and an output register file 1360. In other embodiments, PE 1300 may include fewer, more, or different components. Input register file 1310 may be Figure 9 13. The input register file 910 may be an embodiment of the input register file 910. The weight register file 1320 may be an embodiment of the weight register file 920. The multiplier 1330 may be an embodiment of the multiplier 930. The shifter 1340 may be an embodiment of the shifter 935. The accumulator 1350 may be an embodiment of the accumulator 950. The output register file 1360 may be an embodiment of the output register file 960. The compressor 1370 may be an embodiment of the first adder component 940. The adder tree 1375 may be an embodiment of the second adder component 945.

[0152] Each multiplier 1330 receives an activation value from the input register file 1310 and an integer-valued weight from the weight register file 1320. The multiplier 1330 can receive different activation value-weight pairs. The integer value of the weight can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The integer value can have a single byte. Each multiplier 1330 multiplies the activation value by the integer value. In some embodiments, the multiplier 1330 can be an integer multiplier. The output of the multiplier 1330 is transmitted to the accumulator 1350.

[0153] Each shifter 1340 receives an activation value from the input register file 1310 and an exponent of a power of two from the weight register file 1320. The power of two value can be generated by quantizing the original value of the weight, which can be determined by training the DNN. The shifters 1340 can receive different activation value-weight pairs. Each shifter 1340 shifts the bits in the activation value to the left by the exponent. In some embodiments, the shifters 1340 can be arithmetic shifters. The output of the shifters 1340 is transmitted to the accumulator 1350.

[0154] The bits in the compressed bitmap can be used to determine whether to send the activation value-weight pair to the multiplier 1330 or to the shifter 1340. For example, the determination can be based on the value of the bit corresponding to the weight in the activation value-weight pair. In embodiments where the value of the bit is one, the activation value-weight pair can be sent to the multiplier 1330. In embodiments where the value of the bit is zero, the activation value-weight pair can be sent to the shifter 1340. In some embodiments, the calculations in the shifter 1340 can be faster than the calculations in the multiplier 1330. The shifter 1340 has a simpler structure and a smaller gate depth than the multiplier 1330. The path length through the shifter 1340 can be shorter than the path length through the multiplier 1330. In some embodiments, the complexity of one or more shifters 1340 can be further reduced by limiting the shift range, for example, where larger shifts are rare.

[0155] Compressor 1370 receives the output of multiplier 1330. Compressor 1370 may be an adder compressor that compresses N inputs into two outputs, where N is an integer greater than two. In some embodiments, N may be 3, 4, 5, etc. Figure 13 In the embodiment of FIG. 1 , the input to compressor 1370 is the output of multiplier 1330. Compressor 1370 outputs the sum of the outputs of multiplier 1330, which is a partial sum of multiplier 1330. Although Figure 13 One compressor 1370 is shown, but the PE 1300 may include multiple compressors 1370 coupled to the multipliers 1330 to calculate the sum of the outputs of the multipliers 1330. In addition, the compressor 1370 may be coupled to other numbers of multipliers 1330.

[0156] Adder tree 1375 receives the output of shifter 1340. Figure 13 As shown, adder tree 1375 includes two layers of adders: the first layer includes two adders 1380A and 1380B, and the second layer includes one adder 1390. Adder 1380A receives the outputs of shifters 1340A and 1340B and generates a sum of the outputs. Adder 1380B receives the outputs of shifters 1340C and 1340D and generates a sum of the outputs. Adder 1390 sums the outputs of adders 1380A and 1380B and generates a partial sum of shifter 1340. The number of adders in adder tree 1375 can depend on the number of shifters 1340 in PE 1300. Although Figure 13 Four shifters 1340 are shown, but the PE 1300 may include other numbers of shifters 1340 and other numbers of adders in the adder tree 1375. An adder in the adder tree (eg, adder 1380A, 1380B, or 1390) may be a ripple-carry adder.

[0157] The adders in adder tree 1375 may be slower than compressor 1370, i.e., the computational speed of the adders is lower than the computational speed of compressor 1370. Furthermore, the paths through adder tree 1375 may be longer than the paths through compressor 1370. An advantage of adder tree 1375 is that it may be smaller than compressor 1370. For example, adder tree 1375 may have fewer gates per unit function than compressor 1370. Because shifter 1340 may be faster than multiplier 1330, the overall speed of the paths (i.e., the first path including multiplier 1330 and compressor 1370, and the second path including shifter 1340 and adder tree 1375) may be the same or substantially similar. In some cases, adder tree 1375 may be implemented using one or more adders (e.g., ripple carry adders) and one or more compressors. In some cases, one or both of the output of the compressor 1370 and the output of the adder tree 1375 may exist in redundant form, in which case additional adder(s) or compressor(s) may be used to prepare two inputs for the accumulator 1350.

[0158] Accumulator 1350 accumulates the outputs of compressor 1370 and adder tree 1375 and generates a partial sum for PE 1300. The partial sum may be stored in output register file 1360. In some embodiments, accumulator 1350 may receive one or more partial sums of one or more other PEs and accumulate the partial sum of PE 1300 with the one or more partial sums. The sum of the partial sum of PE 1300 and the one or more partial sums may be a partial sum for a group of PEs (e.g., a PE column (e.g., PE column 805)). The partial sum for the group of PEs may be stored in output register file 1360. In some embodiments, the partial sum for the group of PEs may be further accumulated with one or more other partial sums by PE 1300 or another PE.

[0159] Example method for performing a hybrid MAC operation

[0160] Figure 14 is a flow chart illustrating a method 1400 for performing a hybrid MAC operation according to various embodiments. The method 1400 may be performed by Figure 4 Although the method 1400 is a reference to Figure 14 Although the flowchart shown in FIG is described, many other methods for performing hybrid MAC operations can be used. For example, Figure 14 The order of execution of the steps in the embodiment of the present invention may be changed. As another example, some steps may be changed, deleted, or combined.

[0161] The computation block 400 selects (1410) a first set of one or more weights from a weight tensor of a layer of the DNN. The weight tensor includes the first set of one or more weights and the second set of one or more weights. The layer may be a convolutional layer, e.g. Figure 1 One of the convolutional layers 110 in the layer. The weight tensor can be a subtensor of the entire weight tensor of the layer. The entire weight tensor can be a four-dimensional tensor. The weight tensor can be a two-dimensional or three-dimensional tensor. In some embodiments, the weight tensor includes a plurality of weights arranged in one or more rows and one or more columns. The number of weights in the rows of the weight tensor is equal to the number of channels in the IFM of the layer. The number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

[0162] In some embodiments, computation block 400 selects the same number of one or more weights from each corresponding row of the weight tensor. In some embodiments, computation block 400 selects a first set of one or more weights from the weight tensor based on a predetermined partitioning parameter. The partitioning parameter indicates a ratio of the number of weights in the first set to the number of weights in the second set.

[0163] In some embodiments, computation block 400 selects a first set of one or more weights from a weight tensor by minimizing the difference between the weight tensor and a tensor comprising one or more integers and one or more powers of two. The one or more integers are generated by quantizing one or more weights in the first set. The one or more powers of two are generated by quantizing one or more weights in the second set.

[0164] The computation block 400 quantizes (1420) the weights in the first group to a power of two. In some embodiments, the computation block 400 divides the entire weight tensor of the layer into the weight tensor and the additional weight tensor. The computation block 400 selects a third group of one or more weights from the additional weight tensor and quantizes each corresponding weight in the third group to a power of two. The ratio of the number of weights in the first group to the number of weights in the weight tensor is different from the ratio of the number of weights in the third group to the number of weights in the additional weight tensor.

[0165] The computation block 400 quantizes 1430 the weights in the second set into integers. In some embodiments, the computation block 400 stores exponents of powers of two in memory in place of the weights in the first set. The computation block 400 stores the integers in memory in place of the weights in the second set. The memory space required to store the integers and exponents may be less than the memory space required to store the weights.

[0166] The computation block 400 shifts the activation values of the layer by an exponential of a power of two (1440). The computation block 400 may include a shifter that shifts the activation values by the exponential. The computation block 400 may include a plurality of shifters that shift the activation values by an exponential of a power of two generated by quantizing the weights. The shifters may be coupled to an accumulator that accumulates the outputs of the shifters.

[0167] The computation block 400 multiplies the integer by another activation value of the layer (1450). The computation block 400 may include a multiplier that may multiply the other activation value by the integer. The computation block 400 may include a plurality of multipliers that may multiply the activation value by an integer generated by quantizing the weights. The multiplier may be coupled to an accumulator that accumulates the outputs of the multiplier. The accumulator coupled to the multiplier may have a computation speed faster than the accumulator coupled to the shifter.

[0168] In some embodiments, the computation block 400 generates a bitmap for the weight tensor. The bitmap includes a plurality of bits. Each bit corresponds to a weight in the weight tensor and indicates whether the weight is quantized as an integer or a power of two. For example, a bit with a value of zero indicates that the corresponding weight is quantized as a power of two. A bit with a value of one indicates that the corresponding weight is quantized as an integer. In some embodiments, the computation block transfers a first set of one or more weights from a memory to one or more shifters based on the bitmap. In addition, the computation block transfers a second set of one or more weights from a memory to one or more multipliers based on the bitmap.

[0169] Example computing device

[0170] Figure 15 is a block diagram of an example computing device 1500 according to various embodiments. In some embodiments, the computing device 1500 may be used as Figure 3 At least a portion of the DNN accelerator 300 in. Multiple components in Figure 15 1500, but any one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1500 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system on a chip (SoC) die. Additionally, in various embodiments, computing device 1500 may not include Figure 151500 may include one or more of the components shown, but computing device 1500 may include interface circuitry for coupling to the one or more components. For example, computing device 1500 may not include display device 1506, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which display device 1506 may be coupled. In another set of examples, computing device 1500 may not include audio input device 1518 or audio output device 1508, but may include audio input or output device interface circuitry (e.g., a connector and supporting circuitry) to which audio input device 1518 or audio output device 1508 may be coupled.

[0171] The computing device 1500 may include a processing device 1502 (e.g., one or more processing devices). The processing device 1502 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1500 may include a memory 1504, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 1504 may include a memory that shares a die with the processing device 1502. In some embodiments, the memory 1504 includes one or more non-transitory computer-readable media storing instructions that are executable to perform hybrid MAC operations in the DNN, for example, in combination with the above. Figure 14 The method 1400 described above, or a combination thereof Figure 4 Some of the operations performed by the computation block 400 or the weight compression module 420 are described. Instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 1502.

[0172] In some embodiments, computing device 1500 may include a communication chip 1512 (e.g., one or more communication chips). For example, communication chip 1512 may be configured to manage wireless communications for transmitting data to and from computing device 1500. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, and the like that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not imply that the associated devices do not contain any wires, although in some embodiments they may not.

[0173] The communication chip 1512 may implement any of a number of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), the Long Term Evolution (LTE) project and any amendments, updates, and / or revisions (e.g., the LTE-Advanced project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband wireless access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks, which is an abbreviation for Worldwide Interoperability for Microwave Access and is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 1512 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 1512 can operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1512 may operate according to code-division multiple access (CDMA), time division multiple access (TDMA), digital enhanced cordless telecommunication (DECT), evolution-data optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, etc. In other embodiments, the communication chip 1512 may operate according to other wireless protocols.Computing device 1500 may include antenna 1522 to facilitate wireless communications and / or receive other wireless communications (eg, AM or FM radio transmissions).

[0174] In some embodiments, the communication chip 1512 can manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1512 can include multiple communication chips. For example, the first communication chip 1512 can be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and the second communication chip 1512 can be dedicated to long-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1512 can be dedicated to wireless communications, and the second communication chip 1512 can be dedicated to wired communications.

[0175] Computing device 1500 may include battery / power circuitry 1514. Battery / power circuitry 1514 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 1500 to an energy source separate from computing device 1500 (e.g., AC line power).

[0176] Computing device 1500 may include a display device 1506 (or corresponding interface circuitry as described above). For example, display device 1506 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat-panel display.

[0177] Computing device 1500 may include an audio output device 1508 (or corresponding interface circuitry as described above). For example, audio output device 1508 may include any device that generates an audible indicator, such as a speaker, headphones, or earbuds.

[0178] Computing device 1500 may include an audio input device 1518 (or corresponding interface circuitry as described above). Audio input device 1518 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., an instrument with a MIDI (Musical Instrument Digital Interface) output).

[0179] Computing device 1500 may include a GPS device 1516 (or corresponding interface circuitry as described above). As is known in the art, GPS device 1516 may communicate with a satellite-based system and may receive the location of computing device 1500.

[0180] Computing device 1500 may include other output devices 1510 (or corresponding interface circuitry as described above). Examples of other output devices 1510 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0181] Computing device 1500 may include other input devices 1520 (or corresponding interface circuitry as described above). Examples of other input devices 1520 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0182] The computing device 1500 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smartphone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA (personal digital assistant), an ultraportable personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1500 can be any other electronic device that processes data.

[0183] Select Example

[0184] The following paragraphs provide various examples of the embodiments disclosed herein.

[0185] Example 1 provides a method for executing a DNN, comprising: selecting a first group of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising the first group of one or more weights and a second group of one or more weights; quantizing the weights in the first group to power-of-two values; quantizing the weights in the second group to integers; shifting the activation value of the layer by an exponent of the power-of-two value; and multiplying the integer by another activation value of the layer.

[0186] Example 2 provides the method of Example 1, wherein the weight tensor includes multiple weights arranged in one or more rows and one or more columns, the number of weights in the rows of the weight tensor is equal to the number of channels in the IFM of the layer, and the number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

[0187] Example 3 provides the method of Example 2, wherein selecting the first set of one or more weights from the weight tensor includes selecting the same number of one or more weights from each corresponding row of the weight tensor.

[0188] Example 4 provides a method as described in any of the preceding examples, wherein selecting the first group of one or more weights from the weight tensor includes selecting the first group of one or more weights from the weight tensor based on a predetermined partitioning parameter, the partitioning parameter indicating a ratio of the number of weights in the first group to the total number of weights in the weight tensor.

[0189] Example 5 provides a method of any of the preceding examples, wherein selecting the first group of one or more weights from the weight tensor comprises selecting the first group of one or more weights from the weight tensor by minimizing the difference between the weight tensor and a tensor comprising one or more integers and one or more powers of two, wherein the one or more integers are generated by quantizing one or more weights in the first group and the one or more powers of two are generated by quantizing one or more weights in the second group.

[0190] Example 6 provides the method of any of the preceding examples, further comprising: dividing the entire weight tensor of the layer into the weight tensor and an additional weight tensor; selecting a third group of one or more weights from the additional weight tensor; quantizing each corresponding weight in the third group to a power of two, wherein the ratio of the number of weights in the first group to the number of weights in the weight tensor is different from the ratio of the number of weights in the third group to the number of weights in the additional weight tensor.

[0191] Example 7 provides the method described in any of the preceding examples, further comprising: generating a bitmap for the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a weight in the weight tensor and indicating whether the weight is quantized as an integer or a power of two value.

[0192] Example 8 provides the method of Example 7, wherein a bit having a value of zero indicates that the corresponding weight is quantized to a power of two value, and a bit having a value of one indicates that the corresponding weight is quantized to an integer.

[0193] Example 9 provides the method of Example 7 or 8, further comprising: transferring the first set of one or more weights from the memory to one or more shifters based on the bitmap; and transferring the second set of one or more weights from the memory to one or more multipliers based on the bitmap.

[0194] Example 10 provides the method of any of the preceding examples, further comprising: storing the exponent of the power of two value in a memory to replace the weights in the first group; and storing the integer in the memory to replace the weights in the second group.

[0195] Example 11 provides a computational block configured to execute a DNN, the computational block comprising a weight compression module configured to: select a first group of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising a first group of one or more weights and a second group of one or more weights; quantize the weights in the first group to a power of two value, and quantize the weights in the second group to an integer; and a PE comprising: a shifter configured to shift an activation value of the layer by an exponent of the power of two value; and a multiplier configured to multiply the integer by another activation value of the layer.

[0196] Example 12 provides the computation block of Example 11, wherein the PE further comprises one or more other shifters and one or more other multipliers.

[0197] Example 13 provides the computation block described in Example 12, wherein the PE further comprises: a first accumulator configured to accumulate the output of the shifter and one or more other shifters; and a second accumulator configured to accumulate the output of the multiplier and one or more other multipliers.

[0198] Example 14 provides the computational block of Example 13, wherein the first accumulator is configured to accumulate the outputs of the shifter and the one or more other shifters at a first speed, the second accumulator is configured to accumulate the outputs of the multiplier and the one or more other multipliers at a second speed, and the first speed is lower than the second speed.

[0199] Example 15 provides the computation block of Example 14, wherein the first accumulator comprises an additive compressor.

[0200] Example 16 provides the computation block of Example 14 or 15, wherein the second accumulator comprises a ripple carry adder.

[0201] Example 17 provides the computation block of any one of Examples 14-16, wherein the PE further comprises a third accumulator configured to accumulate outputs of the first accumulator and the second accumulator.

[0202] Example 18 provides the computational block of any one of Examples 11-17, wherein the computational block further comprises a memory configured to: store exponents of powers of two to replace weights in the first group; and store integers to replace weights in the second group.

[0203] Example 19 provides the computational block of Example 18, wherein the memory is further configured to store a bitmap of the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a corresponding weight in the weight tensor and indicating whether the weight is quantized as an integer or a power of two value.

[0204] Example 20 provides the computational block of Example 19, wherein the PE further comprises an additional multiplier coupled to the shifter, and the PE is configured to determine, based on the bit in the bitmap corresponding to the weight in the first group, to transfer the exponent of the power of two value to the shifter instead of the additional multiplier.

[0205] Example 21 provides one or more non-transitory computer-readable media storing instructions that are executable to perform operations for executing a layer of a DNN, the operations comprising: selecting a first group of one or more weights from a weight tensor of the layer of the DNN, the weight tensor comprising the first group of one or more weights and a second group of one or more weights; quantizing the weights in the first group to power-of-two values; quantizing the weights in the second group to integers; shifting the activation values of the layer by an exponent of the power-of-two value; and multiplying the integer by another activation value of the layer.

[0206] Example 22 provides one or more non-transitory computer-readable media as described in Example 21, wherein the weight tensor includes a plurality of weights arranged in one or more rows and one or more columns, the number of weights in the rows of the weight tensor is equal to the number of channels in the IFM of the layer, and the number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

[0207] Example 23 provides one or more non-transitory computer-readable media as described in Example 21 or 22, wherein selecting the first group of one or more weights from the weight tensor includes: selecting the first group of one or more weights from the weight tensor based on a predetermined partitioning parameter, the partitioning parameter indicating a ratio of the number of weights in the first group to the total number of weights in the weight tensor.

[0208] Example 24 provides one or more non-transitory computer-readable media of any one of Examples 21 to 23, wherein the operation further comprises generating a bitmap for the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a weight in the weight tensor and indicating whether the weight is quantized to an integer or a power of two value.

[0209] Example 25 provides one or more non-transitory computer-readable media of any one of Examples 21 to 24, wherein the operation further comprises: storing the exponent of the power of two value in a memory to replace the weights in the first group; and storing the integer in the memory to replace the weights in the second group.

[0210] The above description of the illustrated embodiments of the present disclosure, including that described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, various equivalent modifications may be made within the scope of the present disclosure, as will be appreciated by those skilled in the relevant art. These modifications may be made to the present disclosure in light of the above detailed description.

Claims

1. A method for executing a deep neural network (DNN), comprising: selecting a first set of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising the first set of one or more weights and a second set of one or more weights; quantizing the weights in the first group to powers of two; quantizing the weights in the second group to integers; shifting activation values of the layer by the power of two; as well as Multiply the integer by another activation value of the layer.

2. The method according to claim 1, wherein The weight tensor includes multiple weights arranged in one or more rows and one or more columns, the number of weights in the rows of the weight tensor is equal to the number of channels in the input feature map of the layer, and the number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

3. The method according to claim 2, wherein: Selecting the first set of one or more weights from the weight tensor includes: The same number of one or more weights is selected from each corresponding row of the weight tensor.

4. A method according to any one of the preceding claims, wherein Selecting the first set of one or more weights from the weight tensor includes: The first set of one or more weights is selected from the weight tensor based on a predetermined partition parameter indicating a ratio of the number of weights in the first set to the total number of weights in the weight tensor.

5. A method according to any one of the preceding claims, wherein Selecting the first set of one or more weights from the weight tensor includes: selecting the first set of one or more weights from the weight tensor by minimizing a difference between the weight tensor and a tensor comprising one or more integers and one or more powers of two, The one or more integers are generated by quantizing one or more weights in the first group, and the one or more power-of-two values are generated by quantizing one or more weights in the second group.

6. The method according to any one of the preceding claims, further comprising: Splitting an entire weight tensor of the layer into the weight tensor and an additional weight tensor; selecting a third set of one or more weights from the additional weight tensor; quantizing each corresponding weight in the third set to a power-of-two value, wherein a ratio of the number of weights in the first group to the number of weights in the weight tensor is different from a ratio of the number of weights in the third group to the number of weights in the additional weight tensor.

7. The method according to any one of the preceding claims, further comprising: A bitmap is generated for the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a weight in the weight tensor and indicating whether the weight is quantized to an integer or a power of two value.

8. The method according to claim 7, wherein: A bit with a value of zero indicates that the corresponding weight is quantized to a power of two value, and a bit with a value of one indicates that the corresponding weight is quantized to an integer.

9. The method according to claim 7 or 8, further comprising: transferring the first set of one or more weights from a memory to one or more shifters based on the bitmap; as well as The second set of one or more weights is transferred from the memory to one or more multipliers based on the bitmap.

10. The method according to any one of the preceding claims, further comprising: storing the exponent of the power of two value in a memory to replace the weights in the first set; as well as The integer is stored in the memory in place of the weights in the second set.

11. A device comprising: a computer processor for executing computer program instructions; as well as a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: selecting a first set of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising the first set of one or more weights and a second set of one or more weights; quantizing the weights in the first group to powers of two; quantizing the weights in the second group to integers; shifting the activation values of the layer by the power of two; and Multiply the integer by another activation value of the layer.

12. The device according to claim 11, wherein The weight tensor includes multiple weights arranged in one or more rows and one or more columns, the number of weights in the rows of the weight tensor is equal to the number of channels in the input feature map of the layer, and the number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

13. The device according to claim 11 or 12, wherein: Selecting the first set of one or more weights from the weight tensor includes: The first set of one or more weights is selected from the weight tensor based on a predetermined partition parameter indicating a ratio of the number of weights in the first set to the total number of weights in the weight tensor.

14. The device according to any one of claims 11 to 13, wherein The operations further include: A bitmap is generated for the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a weight in the weight tensor and indicating whether the weight is quantized to an integer or a power of two value.

15. The device according to any one of claims 11 to 14, wherein The operations further include: storing the exponent of the power of two value in a memory to replace the weights in the first set; and The integer is stored in the memory in place of the weights in the second set.

16. One or more non-transitory computer-readable media storing instructions executable to perform operations for executing a layer of a deep neural network (DNN), the operations comprising: selecting a first set of one or more weights from a weight tensor of a layer of the DNN, the weight tensor comprising the first set of one or more weights and a second set of one or more weights; quantizing the weights in the first group to powers of two; quantizing the weights in the second group to integers; shifting activation values of the layer by the power of two; as well as Multiply the integer by another activation value of the layer.

17. The one or more non-transitory computer-readable media of claim 16, wherein: The weight tensor includes multiple weights arranged in one or more rows and one or more columns, the number of weights in the rows of the weight tensor is equal to the number of channels in the input feature map of the layer, and the number of weights in the columns of the weight tensor is equal to the number of channels in the output feature map of the layer.

18. The one or more non-transitory computer-readable media of claim 17, wherein: Selecting the first set of one or more weights from the weight tensor includes: The same number of one or more weights is selected from each corresponding row of the weight tensor.

19. One or more non-transitory computer-readable media according to any one of claims 16 to 18, wherein: Selecting the first set of one or more weights from the weight tensor includes: The first set of one or more weights is selected from the weight tensor based on a predetermined partition parameter indicating a ratio of the number of weights in the first set to the total number of weights in the weight tensor.

20. The one or more non-transitory computer-readable media of any one of claims 16 to 19, wherein: Selecting the first set of one or more weights from the weight tensor includes: selecting the first set of one or more weights from the weight tensor by minimizing a difference between the weight tensor and a tensor comprising one or more integers and one or more powers of two, The one or more integers are generated by quantizing one or more weights in the first group, and the one or more power-of-two values are generated by quantizing one or more weights in the second group.

21. One or more non-transitory computer-readable media according to any one of claims 16 to 20, wherein: The operations further include: Splitting an entire weight tensor of the layer into the weight tensor and an additional weight tensor; selecting a third set of one or more weights from the additional weight tensor; quantizing each corresponding weight in the third set to a power-of-two value, wherein a ratio of the number of weights in the first group to the number of weights in the weight tensor is different from a ratio of the number of weights in the third group to the number of weights in the additional weight tensor.

22. One or more non-transitory computer-readable media according to any one of claims 16 to 21, wherein: The operations further include: A bitmap is generated for the weight tensor, the bitmap comprising a plurality of bits, each bit corresponding to a weight in the weight tensor and indicating whether the weight is quantized to an integer or a power of two value.

23. The one or more non-transitory computer-readable media of claim 22, wherein: A bit with a value of zero indicates that the corresponding weight is quantized to a power-of-two value, and a bit with a value of one indicates that the corresponding weight is quantized to an integer.

24. One or more non-transitory computer-readable media according to claim 22 or 23, wherein: The operations further include: transferring the first set of one or more weights from a memory to one or more shifters based on the bitmap; and The second set of one or more weights is transferred from the memory to one or more multipliers based on the bitmap.

25. One or more non-transitory computer-readable media according to any one of claims 16 to 24, wherein: The operations further include: storing the exponent of the power of two value in a memory to replace the weights in the first set; and The integer is stored in the memory in place of the weights in the second set.

Images