Detecting and mitigating failure in sparse computation in deep

Abstract

Computations in a processing element (PE) for executing a deep neural network (DNN) may be accelerated based on sparsity. A compression activation operand and a compression weight operand may be stored. The compressed activation operand includes one or more non-zero activation values in the activation operands. The compressed weight operands include one or more non-zero weights in the weight operands. A sparsity module associated with the PE may generate a bitmap based on an activation sparsity vector of the activation operand and a weight sparsity vector of the weight operand. The sparsity module identifies non-zero activation values (or non-zero weights) from the compressed activation operands (or compressed weight operands) based on the bitmap. The sparsity module may detect a fault in identifying a non-zero activation value or a non-zero weight based on a number of one or more non-zero elements in the bitmap. The sparsity module may also further mitigate the fault.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 304,713, filed April 21, 2023, entitled “DETECTING AND MITIGATING FULT IN SPARSITY COMPUTATION INDEEP NEURAL NETWORK,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to neural networks and, more particularly, to detecting and mitigating failures in sparsity computations in deep neural networks (DNNs). Background Art

[0004] Because DNNs can achieve high accuracy, they are widely used in a variety of AI applications, from computer vision to speech recognition and natural language processing. However, this high accuracy comes at a significant computational cost. DNNs are computationally demanding, as each inference can require hundreds of millions of MAC (multiply-accumulate) operations and extensive data reading and writing. Therefore, technologies are needed to improve DNN efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Each embodiment will be readily understood by the following detailed description in conjunction with the accompanying drawings. For ease of description, like reference numerals denote like structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not limitation.

[0006] Figure 1 An example DNN according to various embodiments is shown.

[0007] Figure 2 Example convolutions according to various embodiments are shown.

[0008] Figure 3 is a block diagram of a DNN accelerator according to various embodiments.

[0009] Figure 4 is a block diagram of computational blocks according to various embodiments.

[0010] Figure 5 An array of processing elements (PEs) is shown in accordance with various embodiments.

[0011] Figure 6 is a block diagram of a PE according to various embodiments.

[0012] Figure 7Sparsity acceleration of PE in MAC operations according to various embodiments is shown.

[0013] Figure 8 Sparsity computation for identifying pairs of non-zero-valued activations and non-zero-valued weights is shown in accordance with various embodiments.

[0014] Figure 9 Detection of failures in sparsity computations according to various embodiments is shown.

[0015] Figure 10 Mitigation of computational errors in sparsity calculations according to various embodiments is shown.

[0016] Figure 11 is a flow chart illustrating a method of detecting failures in sparsity computations, according to various embodiments.

[0017] Figure 12 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0018] Overview

[0019] The past decade has witnessed the rapid rise of artificial intelligence (AI)-based data processing technologies, particularly those based on deep neural networks (DNNs). DNNs have been widely used in computer vision, speech recognition, image, and video processing, primarily due to their ability to achieve superhuman accuracy. Significant improvements in DNN model size and accuracy, coupled with the rapid increase in the computational power of execution platforms, have enabled DNN applications to be used even in resource-constrained mobile and edge devices with limited energy supplies.

[0020] A DNN layer may include one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, nonlinear operations, and the like. Deep learning operations in the DNN may be performed on one or more internal parameters (e.g., weights) and one or more activation values ​​of the DNN, which are determined during the training phase. An activation value may be a data point (also referred to as a "data element" or "element"). The activation values ​​or weights of a DNN layer may be elements of a tensor of the DNN layer. A tensor is a data structure with multiple elements spanning one or more dimensions. Example tensors include vectors (which are one-dimensional tensors) and matrices (which are two-dimensional tensors). Three-dimensional tensors and even higher-dimensional tensors are also possible. A DNN layer may have an input tensor (also referred to as an "input feature map (IFM)"), which includes one or more input activation values ​​(also referred to as "input elements"), and a weight tensor that includes one or more weights. Weights are elements in the weight tensor. The weight tensor of a convolution may be a kernel, a filter, or a filter bank. The combination of one or more input activation values ​​and one or more weights may be referred to as the input data of the DNN layer. The output data of a DNN layer may be an output tensor (also referred to as an “output feature map (OFM)”), which includes one or more output activation values ​​(also referred to as “output elements”).

[0021] An accelerator for a DNN ("DNN accelerator") may include one or more large arrays of PEs that concurrently execute layers in a DNN. The PEs may perform deep learning operations. For example, for a convolution, the PEs may perform MAC operations on activations and weights. Input tensors or weight tensors may include zero-valued elements that do not affect the dot product output. The DNN accelerator may exploit sparsity in these input tensors or weight tensors to accelerate deep learning operations in the DNN, which may increase speed or throughput and reduce energy consumption. For example, the DNN accelerator may include a sparsity acceleration unit that may be used to skip processing of zero-valued activations or zero-valued weights by the PEs.

[0022] However, such sparsity-based DNN accelerators are susceptible to circuit-level hardware faults, such as those caused by aging, temperature, process variations, and soft errors. Hardware faults can lead to erroneous outputs of the DNNs run by the DNN accelerator. Hardware faults in DNN accelerators can be transient or permanent. Transient faults can be caused by high-energy particle impacts, resulting in temporary errors in the data path. Permanent faults can escape the manufacturing screening process and manifest as latent defects due to structural deformations (such as overlapping vias, partial shorts, and opens). The conversion to hard defects during field operation and transistor aging can accelerate the degradation of these latent defects and the manifestation of permanent faults. Circuit-level hardware faults can not only introduce unreasonable misclassifications but also cause control failures in the sparsity acceleration logic of the DNN accelerator. Even though sparse acceleration units may constitute a relatively small portion of the overall area and power consumption of a DNN accelerator, detecting and mitigating faults in sparse acceleration units is crucial, as such faults can have a devastating impact on the overall inference accuracy of the DNN. Faults in sparse acceleration units can sometimes have a greater impact than faults in PEs.

[0023] Software-based fault handling schemes have been proposed. However, these techniques are unable to mitigate the characteristic fault manifestations in hardware accelerators. Furthermore, while the impact of memory faults in DNN accelerators has been addressed, the impact of data path faults has been overlooked. Current techniques for handling faults in DNN accelerators may also suffer from the drawback of significantly increasing area and power consumption. These techniques are not suitable for resource-constrained DNN accelerators, such as those running at the edge. Therefore, improved techniques are needed to handle faults in DNN accelerators.

[0024] Embodiments of the present disclosure can improve at least some of the aforementioned challenges and issues by dynamically detecting and mitigating failures in sparsity computations within DNNs. The DNN accelerator disclosed herein can accelerate deep learning operations within DNN layers by storing and processing compressed input data. The compressed input data for a DNN layer includes data that will affect the output of the DNN layer and excludes data that does not. Taking a convolutional layer as an example, the compressed input data for the MAC operation within the convolution layer can include compressed activation operands and compressed weight operands. A MAC operation has an activation operand and a weight operand. The activation operand includes a sequence of activation values, and the weight operand includes a sequence of weights. The MAC operation comprises a sequence of multiplications, each of which is for an activation-weight pair. The positions of the activation values ​​in the activation operands can be matched with the positions of the weights in the weight operands. The products obtained from the multiplication sequence can be accumulated to generate a single data point. Zero-valued activations or zero-valued weights do not affect the accumulated result because the product of such activations or weights with any number is zero. Compressed activation operands include one or more non-zero activation values ​​in the activation operands for the MAC operation. The compressed weight operand includes one or more non-zero weights in the weight operand for the MAC operation. The compressed input data does not include zero-valued data points. Therefore, the position of the activation value (or weight) in the compressed activation operand (or compressed weight operand) may be different from the position of the activation value (or weight) in the activation operand (or weight operand).

[0025] In various embodiments of the present disclosure, a DNN accelerator includes a sparsity module that facilitates the acceleration of deep learning operations by determining the location of activation values ​​in compressed activation operands and the location of weights in compressed weight operands. The sparsity module can perform a sequence of sparsity computation rounds for MAC operations. Each round can be used to identify activation value-weight pairs. The sparsity module can also detect and mitigate failures in its sparsity computations to prevent these failures from negatively impacting the performance of the DNN accelerator or the accuracy of the DNN.

[0026] In one example, the sparsity module can obtain an activation bitmap and a weight bitmap. The activation bitmap includes a bit sequence indicating whether each activation value in the activation operand is zero or non-zero. The weight bitmap includes a bit sequence indicating whether each weight in the weight operand is zero or non-zero. In each round of sparsity calculation, the sparsity module can generate an activation position bitmap and a weight position bitmap based on the activation bitmap and the weight bitmap. The activation position bitmap indicates the position of the activation value in the compressed activation operand. For example, the number of ones in the activation position bitmap can be the position index of the activation value in the compressed activation operand. Similarly, the weight position bitmap indicates the position of the weight in the compressed weight operand. The sparsity module can use the position index to read the activation value and weight from one or more memories storing the compressed activation operand and the compressed weight operand, and provide the activation value and weight to the PE to perform a MAC operation.

[0027] The sparsity module can detect computational errors (e.g., errors in the activation position bitmap or weight position bitmap) by comparing the number of ones in the activation position bitmap with the number of ones in the activation bitmap, or by comparing the number of ones in the weight position bitmap with the number of ones in the weight bitmap. After detecting a computational error, the sparsity module can mitigate the computational error. The sparsity module can select different mitigation mechanisms based on the available or expected amount of resources (e.g., power, area, time, etc.) for mitigating computational errors. For example, the sparsity module can determine whether the fault is permanent or temporary. For permanent faults, the sparsity module can disable fault detection and maintain fault mitigation until the MAC operation ends, the deep learning operation ends, or even the DNN reasoning process ends. As another example, the sparsity module can use a redundant mechanism for applications that have sufficient resources to facilitate redundant sparsity calculations, which can calculate multiple activation position bitmaps or multiple weight position bitmaps for a single activation value-weight pair. Alternatively, the sparsity module can use a non-redundant mechanism to adapt to resource-limited applications. For example, rather than computing redundant activation position bitmaps or multiple weight position bitmaps, the sparsity module may identify an activation value or weight that immediately follows an activation value or weight identified in a previous round.

[0028] Additionally or alternatively, the sparsity module can detect one or more control faults in a sparsity calculation round based on one or more intermediate bitmaps generated during the sparsity calculation round. An example of an intermediate bitmap is a control bitmap, which is generated based on an initial control bitmap that may have been generated from a previous sparsity calculation round and is used to generate a subsequent control bitmap that can be used in the next sparsity calculation round. Thus, the control bitmap can inherit calculation information from the previous round or carry calculation information to the next round. The sparsity module can determine whether a control fault exists based on the control bitmap of the current round and the control bitmap of the previous round. After detecting a control fault, the sparsity module can mitigate the control fault by replacing the control bitmap with a new control bitmap generated based on the control bitmap of the previous round. The new control bitmap can continue to be used in the current round.

[0029] This disclosure provides a framework for in-situ detection and dynamic mitigation of circuit-level faults in DNN accelerators performing sparsity computations. Such a framework can reduce or even mitigate the impact of these faults on the accuracy of DNNs or the performance of DNN accelerators without incurring significant overhead. The framework can also provide flexibility for resource-constrained applications (e.g., applications running on edge devices).

[0030] For illustrative purposes, specific numbers, materials, and configurations are set forth to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details, or / and may be practiced with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative embodiments.

[0031] In addition, reference is made to the accompanying drawings which form a part thereof, and in which are shown by way of illustration embodiments that may be practiced. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be construed in a limiting sense.

[0032] The various operations may be described as multiple discrete actions or operations in a manner that best facilitates understanding of the claimed subject matter. However, the order of description should not be interpreted as implying that the operations are necessarily order-dependent. In particular, the operations may not be performed in the order presented. The described operations may be performed in an order different from that of the described embodiments. Various additional operations may be performed, or described operations may be omitted in additional embodiments.

[0033] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0034] This description uses the phrases "in one embodiment" or "in an embodiment," each of which may refer to one or more of the same or different embodiments. The terms "including," "comprising," "having," and the like as used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are merely for ease of discussion and do not imply desired or required directions. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects merely indicates different instances of the similar objects referred to, and is not intended to imply that the objects so described must be arranged in a given order, whether in time, space, ranking, or otherwise.

[0035] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0036] The terms "substantially," "close," "approximately," "near," and "about" generally refer to within + / - 20% of a target value based on the specific value described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to within + / - 5-20% of a target value based on the specific value described herein or known in the art.

[0037] Furthermore, the terms "comprises," "includes," "has," or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Furthermore, the term "or" refers to an inclusive or rather than an exclusive or.

[0038] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0039] Example DNN

[0040] Figure 1 An example DNN 100 is shown in accordance with various embodiments. For illustration purposes, Figure 1 The DNN 100 in FIG is a CNN. In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In one embodiment, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. During inference of DNN 100, the layers of the DNN perform tensor computations including a number of tensor operations, such as convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination of these operations.

[0041] The convolution layer 110 summarizes the presence of features in the input image 105. The convolution layer 110 acts as a feature extractor. The first layer of the DNN 100 is the convolution layer 110. In the example, the convolution layer 110 performs a convolution operation on the input tensor 140 (also referred to as IFM 140) and the filter 150. Figure 1 As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, and each channel is represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also called input points) per row and 7 input elements per column. The filter 150 is represented by a 3×3×3 3D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. The kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. The kernel may be smaller than the IFM. Figure 1 In the embodiment, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel includes 3 weights per row and 3 weights per column. The weights can be initialized and updated using gradient descent via backpropagation. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0042] Convolution involves a MAC operation of the input elements in the IFM 140 and the weights in the filter 150. The convolution can be a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides over the IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also called output points) per row and 5 output elements per column. For illustration, in Figure 1 In the embodiment of FIG, the standard convolution includes one filter. In the embodiment where multiple filters are present, the standard convolution can produce multiple output channels in OFM 160.

[0043] The multiplication performed between the kernel-sized local patch of IFM 140 and the kernel can be a dot product. A dot product is an element-by-element multiplication between the kernel-sized local patch of IFM 140 and the corresponding kernel, followed by addition, always producing a single value. Because it produces a single value, this operation is often referred to as a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times with IFM 140 at different points on IFM 140. Specifically, the kernel is systematically applied from left to right and from top to bottom to each overlapping portion or kernel-sized local patch of IFM 140. The result of multiplying the kernel once with IFM 140 is a single value. Since the kernel is applied multiple times to IFM 140, the multiplication result is an output element of a 2D matrix. Therefore, the 2D output matrix from standard convolution 163 (i.e., OFM 160) is referred to as OFM.

[0044] In depthwise convolution 183, the input channels are not combined. Instead, MAC operations are performed on individual input channels and individual kernels, and output channels are generated. Figure 1As shown, depthwise convolution 183 produces a depthwise output tensor 180. Depthwise output tensor 180 is represented by a 5×5×3 3D matrix. Depthwise output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains five output elements, and each column also contains five output elements. Each output channel is the result of a MAC operation between the input channels of IFM 140 and the kernel of filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation between the first input channel (dot pattern) and the first kernel (dot pattern), the second output channel (horizontal stripe pattern) is the result of a MAC operation between the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern), and the third output channel (diagonal stripe pattern) is the result of a MAC operation between the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, pointwise convolution 193 is performed on the depthwise output tensor 180 and the 1×1×3 tensor 190 to produce the OFM 160 .

[0045] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is a rectified linear unit (ReLU). A ReLU is a calculation that directly returns the value provided as input, or returns a value of 0 if the input is 0 or less. The convolutional layer 110 can receive several images as input and calculate the convolution of each of them with each kernel. This process can be repeated several times. For example, the OFM 160 is passed to a subsequent convolutional layer 110 (i.e., the convolutional layer 110 after the convolutional layer 110 in the sequence that generated the OFM 160). The subsequent convolutional layer 110 performs convolution on the OFM 160 using the new kernel and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be kernelized again by a further subsequent convolutional layer 110, and so on.

[0046] In some embodiments, convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size (e.g., the kernel size is F×F×D pixels), the stride S for dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolutions, such as two-dimensional convolution, dilated or atrous convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. DNN 100 includes 16 convolutional layers 110. In other embodiments, DNN 100 may include a different number of convolutional layers.

[0047] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in a block of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the front convolutional layer 110 (the convolutional layer 110 located before the pooling layer 120 in the layer sequence) and the back convolutional layer 110 (the convolutional layer 110 located after the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after the activation function (e.g., ReLU) has been applied to the OFM 160.

[0048] The pooling layer 120 receives the feature maps generated by the preceding convolutional layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the DNN and prevents overlearning. The pooling layer 120 can perform the pooling operation through average pooling (calculating the average value of each local block on the feature map), maximum pooling (calculating the maximum value of each local block on the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied to 2×2 pixels with a step size of 2 pixels, thereby reducing the size of the feature map by a factor of 2, for example, reducing the number of pixels or values ​​in the feature map to one-quarter of its original size. In one example, the pooling layer 120 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps with the same number of pooled feature maps.

[0049] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be convolutional. The fully connected layer 130 receives input operands. The input operands define the outputs of the convolutional layer 110 and the pooling layer 120 and include the values ​​of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operands and generates a vector. This vector can contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the final fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as the activation function.

[0050] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probability, the fully connected layer 130 multiplies each input element by a weight, sums the values, and then applies an activation function (e.g., logistic if N=2, softmax if N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In one example, the vector includes three probabilities: a first probability indicating that object 115 is a tree, a second probability indicating that object 125 is a car, and a third probability indicating that object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual values ​​may be different.

[0051] Example Convolution

[0052] Figure 2 An example convolution according to various embodiments is shown. The convolution can be a convolutional layer in a DNN (e.g., Figure 1 The convolution layer 110 in FIG. 2 may be a front-end layer. The convolution may be performed on an input tensor 210 and a filter 220 (individually referred to as “filter 220”). The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator comprising one or more computational blocks. An example of a DNN accelerator may be Figure 3 The DNN accelerator 300 in

[0054] can be an example of a computational block. Figure 3 Calculation block 325 in .

[0053] exist Figure 2 In the embodiment of FIG, the input tensor 210 includes activation values ​​(also referred to as “input activation values,” “elements,” or “input elements”) arranged in a 3D matrix. The activation values ​​in the input tensor 210 are data points in the input tensor 210. The input tensor 210 has a spatial size H in ×W in ×C in , where H in is the height of the 3D matrix (i.e., the length along the Y axis, representing the number of activation values ​​in the columns of the 2D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of activation values ​​in a row of the 2D matrix for each input channel), C inis the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of input channels). For simplicity and illustration, the input tensor 210 has a spatial size of 7×7×3, i.e., the input tensor 210 includes three input channels, each of which has a 7×7 2D matrix. Each input element in the input tensor 210 can be represented by an (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the input tensor 210 can be different.

[0054] Each filter 220 includes weights arranged in a 3D matrix. The values ​​of the weights can be determined by training the DNN. The filter 220 has a spatial size H f ×W f ×C f , where H f is the height of the filter (i.e., the length along the Y-axis, representing the number of weights in the column for each kernel), W f is the width of the filter (i.e., the length along the X-axis, representing the number of weights in the row for each kernel), C f is the depth of the filter (i.e., the length along the Z axis, representing the number of channels). In some embodiments, C f Equal to C in For simplicity and illustration, Figure 2 Each filter 220 in has a spatial size of 3×3×3, that is, filter 220 includes three convolution kernels with a spatial size of 3×3. In other embodiments, the height, width, or depth of filter 220 can be different. The spatial size of the convolution kernel is smaller than the spatial size of the 2D matrix of each input channel in input tensor 210.

[0055] The activation value or weight may occupy one or more bytes in memory. The number of bytes of the activation value or weight may depend on the data format. For example, when the activation value or weight has an integer format (e.g., INT8), the activation value occupies one byte. When the activation value or weight has a floating point format (e.g., FP16 or BF16), the activation value or weight occupies two bytes. Other data formats may be used for activation values ​​or weights.

[0056] In convolution, each filter 220 slides over the input tensor 210 and produces a 2D matrix for the output channels in the output tensor 230. Figure 2 In the embodiment of FIG, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activation values ​​(also referred to as “output activation values,” “elements,” or “output elements”) arranged in a 3D matrix. The activation values ​​in the output tensor 230 are the data points in the output tensor 230. The output tensor 230 has a spatial size H out ×W out ×C out , where H outis the height of the 3D matrix (i.e., the length along the Y axis, representing the number of output activation values ​​in the column of the 2D matrix for each output channel), W out is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of output activation values ​​in the row of the 2D matrix for each output channel), C out is the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of output channels). out It can be equal to the number of filters 220 in the convolution. out and W out It may depend on the height and width of the input tensor 210 and each filter 220 .

[0057] As part of the convolution, a 3×3×3 sub-tensor 215 of the input tensor 210 (in Figure 2 MAC operations are performed on the sub-tensor 215 and each filter 220 (highlighted with a dot pattern in FIG). The result of the MAC operation on the sub-tensor 215 and one filter 220 is an output activation value. In some embodiments (e.g., embodiments in which the convolution is an integer convolution), the output activation value may include 8 bits, such as one byte. In other embodiments (e.g., embodiments in which the convolution is a floating-point convolution), the output activation value may include more than one byte. For example, an output element may include two bytes.

[0058] After completing the MAC operation on the sub-tensor 215 and all filters 220, a vector 235 is generated. The vector 235 is Figure 2 . Vector 235 includes a sequence of output activation values ​​arranged along the Z-axis. The output activation values ​​in vector 235 have the same (x, y) coordinates, but these output activation values ​​correspond to different output channels and have different Z-coordinates. The dimension of vector 235 along the Z-axis can be equal to the total number of output channels in output tensor 230.

[0059] After vector 235 is generated, further MAC operations are performed to generate additional vectors until output tensor 230 is generated. For example, filter 220 can be moved along the X-axis or Y-axis on input tensor 210, and a MAC operation can be performed on filter 220 and another sub-tensor in input tensor 210 (the sub-tensor has the same size as filter 220). The amount by which filter 220 moves on input tensor 210 in different computational rounds is called the stride size of the convolution. The stride size can be 1 (i.e., the amount by which filter 220 moves is one activation value), 2 (i.e., the amount by which filter 220 moves is two activation values), and so on. The height and width of output tensor 230 can be determined based on the stride size.

[0060] In some embodiments, a plurality of PEs (e.g., Figure 5PE 510, Figure 6 PE 600, or Figure 7 PE 700 in ) performs MAC operations on a 3×3×3 sub-tensor (e.g., sub-tensor 215) and filter 220. One or more MAC units may receive activation operands (e.g., Figure 2 ) and weight operands (e.g., Figure 2 227). The activation operand 217 includes a sequence of activation values ​​having the same (Y, Z) coordinates but different X coordinates. The weight operand 227 includes a sequence of weights having the same (Y, Z) coordinates but different X coordinates. The length of the activation operand 217 is the same as the length of the weight operand 227. The activation values ​​in the activation operand 217 and the weights in the weight operand 227 can be input to the PE in sequence. The PE can receive a pair of activation values ​​and weights at a time and multiply the activation value by the weight. The position of the activation value in the activation operand 217 can match the position of the weight in the weight operand 227.

[0061] Example DNN Accelerator

[0062] Figure 3 is a block diagram of a DNN accelerator 300 according to various embodiments. The DNN accelerator 300 can run a DNN, such as Figure 1 100 in the DNN 100. The DNN accelerator 300 includes a memory 310, a DMA (direct memory access) engine 320, and a computation block 330. In other embodiments, the DNN accelerator 300 may include alternative configurations, different, or additional components. For example, the DNN accelerator 300 may include multiple memories 310 or multiple DMA engines 320. For another example, the DNN accelerator 300 may include a single computation block 330. In addition, the functions implemented by the components of the DNN accelerator 300 may be implemented by other components or other systems included in the DNN accelerator 300.

[0063] Memory 310 stores data that the computation block 330 uses to perform deep learning operations in the DNN model. Example deep learning operations include convolution (also known as "convolution operations"), pooling operations, element-by-element operations, other types of deep learning operations, or combinations of these operations. Memory 310 can be the main memory of the DNN accelerator 300. In some embodiments, memory 310 includes one or more DRAMs (dynamic random access memories). For example, memory 310 can store input tensors, convolution kernels, or output tensors of a convolution in a convolution layer (e.g., convolution layer 30) of a DNN. The output tensor can be transferred from the local memory of the computation block 330 to the memory 310 via the DMA engine 320.

[0064] The DMA engine 320 facilitates data transfers between the memory 310 and the local memory of the compute block 330. For example, the DMA engine 320 can read data from the memory 310 and write data to the local memory of the compute block 330. For another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to the memory 310. The DMA engine 320 provides DMA functionality, allowing the compute block 330 to initiate data transfers between the memory 310 and the local memory of the compute block 330 and perform other operations while the data transfer is in progress. In some embodiments, the DMA engine 320 can read tensors from the memory 310 and modify the tensors in a manner optimized for the compute block 330 before writing the tensors to the local memory of the compute block 330.

[0065] The computation block 330 performs computations for deep learning operations. The computation block 330 may run operations in a DNN layer, or a portion of operations in a DNN layer. The computation block 330 may perform convolutions, such as standard convolutions (e.g., Figure 1 Standard convolution in 163), depthwise convolution (e.g., Figure 1 ), and point-wise convolution (e.g., Figure 1 193 in the point-by-point convolution 193) and so on. In some embodiments, the computation block 330 receives an input tensor and one or more convolution kernels and performs convolution on the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further calculated, for example, by the computation block 330 or another computation block. In some embodiments, the operation of the DNN layer can be run in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 can each perform a portion of the convolution workload. Data can be shared between the computation blocks 330.

[0066] Figure 4 is a block diagram of a computation block 400 according to various embodiments. The computation block 400 may be Figure 3 An example of the calculation block 330 in FIG. Figure 4 As shown, computation block 400 includes local memory 410, PE array 420, and sparsity module 430. In other embodiments, computation block 400 may include alternative configurations, different or additional components. For example, computation block 400 may include multiple local memories 410, PE arrays 420, or sparsity modules 430. In addition, the functions implemented by the components of computation block 400 may be implemented by other components included in computation block 400, another component in DNN accelerator 300, or other systems.

[0067] Local memory 410 is local to the computation block 400. Figure 4In an embodiment of the present invention, the local memory 410 is located within the computing block 400. In other embodiments, the local memory 410 may be located outside the computing block 400. The local memory 410 and the computing block 400 may be implemented on the same chip. The local memory 410 stores data used for convolution or data generated from convolution, such as input activation values, weights, and output activation values. In some embodiments, the local memory 410 includes one or more SRAMs (static random access memories). The local memory 410 may be byte-addressable, with each memory address identifying a single byte (eight bits) of storage space. In some embodiments, the local memory 410 may include memory banks, each of which may have a capacity of a fixed number of bytes (e.g., 32, 64, etc.).

[0068] PE array 420 performs MAC operations in convolution. PE array 420 may also perform other deep learning operations. PE array 420 may include PEs arranged in columns or in rows and columns. Each PE may perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more adders for performing accumulation. A column of PEs is referred to as a PE column. A PE column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into a MAC column. A MAC lane may also be referred to as a data transmission lane or a data loading lane. A PE column may have multiple MAC lanes. The loading bandwidth of a MAC column is the sum of the loading bandwidths of all MAC lanes associated with the MAC column. Using a specific number of MAC lanes, data can be simultaneously input into the same number of independent PEs. In some embodiments, a MAC column has four MAC lanes for inputting activation values ​​or weights into the MAC column. Each MAC lane may have a bandwidth of 16 bytes, and the four MAC lanes may have a total loading bandwidth of 64 bytes.

[0069] In some embodiments, the PE array 420 may be capable of performing standard convolution, depthwise convolution, pointwise convolution, other types of convolution, or a combination of these. In depthwise convolution, the PE may perform a MAC operation that includes a sequence of multiplications for an activation operand (e.g., activation operand 217) and a weight operand (e.g., weight operand 227). Each multiplication in the sequence is a multiplication of a different activation value in the activation operand and a different weight in the weight operand. The activation value and weight in the same cycle may correspond to the same channel. The multiplication sequence generates a product operand that includes a product sequence. The MAC operation may also include accumulation, wherein multiple product operands are accumulated to generate an output operand of the PE. The PE array 420 may output multiple output operands simultaneously, each output operand generated by a different PE. In standard convolution, the MAC operation may include accumulation across channels. For example, instead of generating output operands, the PE may accumulate products across different channels to generate a single output point.

[0070] In some embodiments, a PE can perform multiple rounds of MAC operations for convolution. Data (activations, weights, or both) can be reused within a single round (e.g., between different multipliers in a PE) or between different rounds of MAC operations. More details about the PE array will be provided below in conjunction with Figure 5 and Figure 6 Provide a description.

[0071] The sparsity module 430 accelerates deep learning operations in the DNN based on the sparsity of the input data of the deep learning operations. The sparsity module 430 can have sparsity acceleration logic that can identify non-zero activation value-weight pairs and skip zero-value activation value-weight pairs. Non-zero activation value-weight pairs include non-zero activation values ​​and non-zero weights, while zero-value activation value-weight pairs include zero activation values ​​or zero weights. The sparsity module 430 can also detect and mitigate faults within the sparsity acceleration logic, such as controlling faults or computational errors in the process of identifying non-zero activation value-weight pairs.

[0072] although Figure 4 A single sparsity module 430 is shown, but the computation block 400 may include multiple sparsity modules 430. In some embodiments, each PE in the PE array 420 implements the sparsity module 430 to accelerate computations in the individual PEs. In other embodiments, a subset of the PE array 420 (e.g., one or more PE columns in the PE array 420) may implement the sparsity module 430 to accelerate computations in the subset of PEs. Figure 4As shown, sparsity module 430 includes a sparsity accelerator 440, a fault detector 450, and a fault mitigator 460. In other embodiments, sparsity module 430 may include fewer, more, or different components. In addition, the functionality implemented by one component of sparsity module 430 may be accomplished by other components included in sparsity module 430, other components in computation block 400, or another device different from that shown.

[0073] The sparsity module 430 accelerates computations in the PE array 420 based on the sparsity of the input data being computed. In some embodiments (e.g., embodiments in which the computation block 400 performs a convolutional layer), computations in the PEs may be MAC operations on activation operands and weight operands. The activation operand may be part of the input tensor of the convolution. The activation operand includes a sequence of input elements (i.e., activation values). The activation values ​​may come from different input channels. For example, each activation value comes from an input channel different from all other activation values ​​in the activation operand. The activation operand is associated with an activation bitmap (also called an "activation sparsity vector"), which may be stored in the local memory 410. The activation bitmap may indicate the locations of non-zero activation values ​​in the activation operand. The activation bitmap may include a sequence of bits, with each bit corresponding to a corresponding activation value in the activation operand. The positions of the bits in the activation bitmap may match the positions of the corresponding activation values ​​in the activation operand. The bits in the activation bitmap may be either zero or one. A zero-valued bit indicates that the corresponding activation value is zero, and a one-valued bit indicates that the corresponding activation value is non-zero. In some embodiments, the activation bitmap may be generated during execution of another DNN layer (e.g., a layer preceding a convolutional layer in the DNN).

[0074] The weight operand can be part of the kernel of the convolution. The weight operand includes a sequence of weights. The values ​​of these weights are determined by training the DNN. The weights in the weight operand can come from different input channels. For example, each weight comes from an input channel different from all other weights in the weight operand. The weight operand is associated with a weight bitmap (also called an "activation sparsity vector"), which can be stored in the local memory 410. The weight bitmap can indicate the position of non-zero valued weights in the weight operand. The weight bitmap can include a sequence of bits, each bit corresponding to a corresponding weight in the weight operand. The position of the bit in the weight bitmap can match the position of the corresponding weight in the weight operand. The bits in the weight bitmap can be zero or one. A zero-valued bit indicates that the value of the corresponding weight is zero, and a one-valued bit indicates that the value of the corresponding weight is non-zero.

[0075] The sparsity accelerator 440 can receive the activation bitmap and the weight bitmap and generate a combined sparsity bitmap for the MAC operation to be performed by the PE. In some embodiments, the sparsity accelerator 440 generates the combined sparsity bitmap 735 by performing one or more AND operations on the activation bitmap and the weight bitmap. Each bit in the combined sparsity bitmap is the result of performing an AND operation on a bit in the activation bitmap and a bit in the weight bitmap, that is, the product of a bit in the activation bitmap and a bit in the weight bitmap. The position of the bit in the combined sparsity bitmap matches the position of the bit in the activation bitmap and the position of the bit in the weight bitmap. The bits in the combined bitmap correspond to a pair of activation values ​​and weights (activation value-weight pairs). A zero-valued bit in the combined sparsity bitmap indicates that at least one of the activation value and weight in the pair is zero. A one-valued bit in the combined sparsity bitmap indicates that both the activation value and weight in the pair are non-zero. The combined sparsity bitmap can be stored in the local memory 410.

[0076] The sparsity accelerator 440 can provide activation values ​​and weights to the PE based on the combined sparsity bitmap. For example, the sparsity accelerator 440 can identify one or more non-zero activation value-weight pairs from the local memory 410 based on the combined sparsity bitmap. The local memory 410 can store activation operands and weight operands in a compressed format, so that non-zero activation values ​​and non-zero weights are stored, but zero activation values ​​and zero weights are not stored. The (one or more) non-zero activation values ​​of the activation operands can constitute compressed activation operands. The (one or more) non-zero weights of the weight operands can constitute compressed weight operands. For non-zero activation value-weight pairs, the sparsity accelerator 440 can determine the position of the activation value in the compressed activation operand and the position of the weight in the compressed weight operand based on the activation bitmap, the weight bitmap, and the combined bitmap. The activation values ​​and weights can be read from the local memory 410 based on the position determined by the sparsity accelerator 440.

[0077] In some embodiments, the sparsity accelerator 440 includes sparsity acceleration logic that can calculate a position bitmap based on the activation bitmap and the weight bitmap. The sparsity accelerator 440 can determine the position index of the activation value and the weight based on the position bitmap. For example, the position index of the activation value in the compressed activation operand can be equal to the number of (one or more) ones in the activation position bitmap generated by the sparsity accelerator 440, and the position index of the weight in the compressed weight operand can be equal to the number of (one or more) ones in the weight position bitmap generated by the sparsity accelerator 440. The position index of the activation value or weight indicates the position of the activation value or weight in the compressed activation operand or the compressed weight operand. The sparsity accelerator 440 can read the activation value and weight from one or more memories based on the position index of the activation value and weight. More details about identifying non-zero activation value-weight pairs will be described below in conjunction with Figure 8 supply.

[0078] The sparsity accelerator 440 may forward the identified non-zero activation value-weight pairs to the PE. The sparsity accelerator 440 may skip other activation values ​​and other weights because they do not contribute to the result of the MAC operation. In some embodiments, the local memory 310 may store non-zero activation values ​​and weights without storing zero activation values ​​or weights. The non-zero activation values ​​and weights may be loaded into one or more register files of the PE, from which the sparsity accelerator 440 may retrieve the activation values ​​and weights corresponding to the ones in the combined sparsity bitmap. In some embodiments, the total number of ones in the combined sparsity bitmap is equal to the total number of activation value-weight pairs to be calculated by the PE, while the PE does not calculate other activation value-weight pairs. By skipping the activation value-weight pairs corresponding to the zero bits in the combined sparsity bitmap, the PE's calculation will be faster than if the PE calculates all activation value-weight pairs in the activation operands and weight operands.

[0079] Fault detector 450 detects faults in the sparsity calculations performed by sparsity accelerator 440. In some embodiments, fault detector 450 can detect control faults in a sparsity calculation round based on a control bitmap calculated in the sparsity calculation round. In the process of calculating the position bitmap, sparsity accelerator 440 can calculate one or more intermediate bitmaps based on the activation bitmap and the weight bitmap. One of these one or more intermediate bitmaps can be referred to as a control bitmap. The control bitmap can be generated based on an initial control bitmap generated based on a previous round of sparsity calculations. Additionally or alternatively, the control bitmap can be used to determine a subsequent control bitmap in the current round, which can be used as the initial control bitmap for the next round of sparsity calculations. Thus, the control bitmap can inherit calculation information from the previous round of sparsity calculations or pass calculation information to the next round of sparsity calculations.

[0080] Fault detector 450 may calculate the sum of the number of ones in the control vector and a number such as one. Fault detector 450 may also compare this sum with the control vector used in the previous round (e.g., the round immediately before the current round). In an embodiment, if the sum is equal to the control vector in the previous round, fault detector 450 determines that there is no control fault and sparsity calculation can continue. In an embodiment, if the sum is not equal to the control vector in the previous round, fault detector 450 determines that there is a control fault.

[0081] In some embodiments, the fault detector 450 may also detect computational errors in the sparsity calculations. A computational error may be an error in the position index determined by the sparsity accelerator 440. The fault detector 450 may determine whether the number of ones in the activation position bitmap is greater than the number of ones in the activation bitmap. The fault detector 450 may also determine whether the number of ones in the weight position bitmap is greater than the number of ones in the weight bitmap. In an embodiment, if the number of ones in the activation position bitmap is greater than the number of ones in the activation bitmap, or the number of ones in the weight position bitmap is greater than the number of ones in the weight bitmap, the fault detector 450 determines that a computational error exists. Otherwise, the fault detector 450 determines that there is no computational error.

[0082] After detecting a fault (e.g., a control fault or a computational error), the fault detector 450 may instruct the fault mitigator 460 to mitigate the fault. The fault detector 450 may also instruct the sparsity accelerator 440 to suspend sparsity computation until the fault is mitigated. More details on fault detection will be provided below in conjunction with Figure 9 Provide a description.

[0083] Fault mitigator 460 mitigates faults in the sparsity computations performed by sparsity accelerator 440. In some embodiments, fault mitigator 460 may receive an indication from fault detector 450 to mitigate a detected control fault. Fault mitigator 460 may mitigate the control fault based on a control vector from a previous round. The previous round may not have had a control fault. Fault mitigator 460 may temporarily store the control vector from the previous round, for example, in a temporary register. Fault mitigator 460 may generate a new control vector based on the control vector from the previous round to replace the control vector based on the detected control fault. To generate the new control vector, fault mitigator 460 may change the last one in the control vector from the previous round to a zero. The remaining bits in the control vector from the previous round may not be changed. Fault mitigator 460 may provide the new control vector to sparsity accelerator 440, which may continue the sparsity computation using the new control vector. The sparsity accelerator 440 may also generate subsequent control vectors based on the new control vector, so that the control failure will not be carried into the next round of sparsity calculation.

[0084] The fault mitigator 460 may receive an indication from the fault detector 450 to mitigate the detected computational error. In some embodiments, the fault mitigator 460 may employ a redundant approach to mitigate the computational error. For example, the fault mitigator 460 may provide one or more variants of the sparsity acceleration logic that may run in parallel with the original acceleration logic. Thus, multiple sparsity acceleration logics may run simultaneously to identify a non-zero activation value-weight pair. The sparsity acceleration logics may use the same control vector but calculate different intermediate vectors based on the control vector. Each sparsity acceleration logic may output a position index of the activation value and a position index of the weight. The position indexes from different sparsity acceleration logics may be the same. The fault mitigator 460 may select a position index determined by the majority of the sparsity acceleration logics. For example, if five sparsity acceleration logics are used and three sparsity acceleration logics output the same position index of the activation value or weight, the fault mitigator 460 may select the position index to identify the activation value or weight. More details on the redundant approach to mitigating faults will be provided below in conjunction with Figure 10 Provide a description.

[0085] The redundant approach minimizes time overhead because different sparsity acceleration logic can run in parallel. Furthermore, area overhead can be minimized because the sparsity acceleration logic can be implemented using basic gates. Power overhead is also negligible. In some embodiments (e.g., where computational resources are limited, deep learning operations are not critical to the accuracy of the DNN, or DNN accuracy is not critical to the application), fault mitigator 460 can use other methods to mitigate computation errors to further reduce area or power overhead. In one example, fault mitigator 460 can prevent activation values ​​or weights from being identified upon detecting a computation error, so that the activation values ​​or weights are not calculated by the PE. Instead of using the redundant approach, fault mitigator 460 can identify or instruct sparsity accelerator 440 to ignore the position index determined from the current round of sparsity calculations and identify the next activation value or weight in the compressed activation operand or compressed weight operand. The next activation value or weight in the compressed activation operand or compressed weight operand can be the activation value or weight in the compressed activation operand or compressed weight operand that immediately follows the activation value or weight identified in the previous round of sparsity calculations.

[0086] In some embodiments, the fault mitigator 460 may use a dynamic mitigation mechanism to mitigate the fault. The fault mitigator may determine whether the fault is temporary or permanent. The fault mitigator 460 may count the number of times the fault is detected. In embodiments where the number of times the fault is detected exceeds a threshold number, the fault mitigator may determine that the fault is permanent. For permanent faults, the fault mitigator 460 may deactivate the fault detector 450 to avoid fault detection, thereby saving time and energy. It is assumed that permanent faults will continue to occur, so there is no need to continue detecting them. The fault mitigator 460 may keep the fault mitigation mechanism (e.g., a redundancy mechanism) active until the MAC operation ends, until the deep learning operation ends, or even until the DNN inference process ends. In one example, if detection is disabled during the DNN inference process, the power saved may be expressed as:

[0087]

[0088] Where M is the number of sparsity computation rounds where transient failures occur, and N is the total number of sparsity computation rounds required for the DNN accelerator to complete inference.

[0089] In embodiments where the fault is transient (e.g., the number of times the fault is detected is below a threshold number), fault detection can remain active. A fault mitigator can be activated after a fault is detected. Dynamic mitigation methods can save power without significantly sacrificing the performance of the DNN accelerator or the accuracy of the DNN.

[0090] Figure 5 PE array 500 according to various embodiments is shown. PE array 500 may be Figure 4 5. PE array 500 includes multiple PEs 510 (individually referred to as "PE 510"). PE 510 performs MAC operations. PE 510 may also be referred to as a neuron in a DNN. Each PE 510 has two input signals 550 and 560 and an output signal 570. Input signal 550 is at least a portion of an IFM of a layer. Input signal 560 is at least a portion of a filter of a layer. In some embodiments, input signal 550 of PE 510 may include one or more activation operands, and input signal 560 may include one or more weight operands.

[0091] Each PE 510 performs a MAC operation on input signals 550 and 560 and outputs an output signal 570 as a result of the MAC operation. Some or all of the input signals 550 and 560 and the output signal 570 may be in integer format (e.g., INT8) or floating-point format (e.g., FP16 or BF16). For simplicity and illustration, the input signals and output signals of all PEs 510 use the same reference numerals, but PEs 510 may receive different input signals and output different output signals. Furthermore, a PE 510 may differ from another PE 510, for example, including more, fewer, or different components.

[0092] like Figure 5 As shown, PEs 510 are interconnected, as shown in FIG. Figure 5 The output signal 570 of PE 510 can be sent as an input signal to many other PEs 510 (and possibly back to itself) through the interconnection between PEs 510. In some embodiments, the output signal 570 of PE 510 can be combined with the output signals of one or more other PEs 510 through the accumulation operation of PE 510 to generate an internal partial sum of the PE array. More details about PE 510 will be discussed below in conjunction with Figure 5 Provide a description.

[0093] exist Figure 5 In an embodiment of the present invention, the PEs 510 are arranged into columns 505 (individually referred to as "columns 505"). The inputs and weights of the layers can be distributed to the PEs 510 based on the columns 505. Each column 505 has a column buffer 520. The column buffer 520 stores the data provided to the PEs 510 in the column 505 for a short period of time. The column buffer 520 can also store the data output by the last PE 510 in the column 505. The output of the last PE 510 can be the sum of the MAC operations of all the PEs 510 in the column 505, which is the column-level internal partial sum of the PE array 500. In other embodiments, the inputs and weights can be distributed to the PEs 510 based on the rows in the PE array 500. The PE array 500 can include a row buffer instead of the column buffer 520. The row buffer can store the input signals of the PEs in the corresponding row, and can also store the row-level internal partial sum of the PE array 500.

[0094] like Figure 5 As shown, each column buffer 520 is associated with a loader 530 and a drain 540. Data provided to a column 505 is transferred to the column buffer 520 by the loader 530, for example, via an upper memory hierarchy, e.g. Figure 4The data generated by the column 505 is extracted from the column buffer 520 by the drainer 540. In some embodiments, the data extracted from the column buffer 520 is sent to the upper memory hierarchy through the drain operation, e.g. Figure 4 In some embodiments, the drain operation does not begin until all PEs 510 in column 505 have completed their MAC operations. In some embodiments, the loader 530 or the drainer 540 may be controlled by the control module 340. Although Figure 5 Not shown, but one or more columns 505 may be associated with external adder components.

[0095] Figure 6 is a block diagram of a PE 600 according to various embodiments. The PE 600 may be Figure 5 6. PE 600 includes an input register file 610 (individually referred to as "input register file 610"), a weight register file 620 (individually referred to as "weight register file 620"), a multiplier 630 (individually referred to as "multiplier 630"), an internal adder component 640, and an output register file 650. In other embodiments, PE 600 may include fewer, more, or different components. For example, PE 600 may include multiple output register files 650. For another example, PE 600 may include a single input register file 610, weight register file 620, or multiplier 630. For another example, PE 600 may include an adder instead of internal adder component 640.

[0096] The input register file 610 temporarily stores activation operands for MAC operations performed by the PE 600. In some embodiments, the input register file 610 can store a single activation operand at a time. In other embodiments, the input register file 610 can store multiple activation operands or a portion of an activation operand at a time. The activation operand includes multiple input elements (i.e., input elements) in the input tensor. The input elements in the activation operand can be stored sequentially in the input register file 610 so that the input elements can be processed sequentially. In some embodiments, each input element in the activation operand can come from a different input channel of the input tensor. The activation operand can include input elements for each input channel of the input tensor, and the number of input elements in the activation operand can be equal to the number of input channels. The input elements in the activation operand can have the same XY coordinates, which can be used as the XY coordinates of the activation operand. For example, all input elements of the activation operand can be X0Y0, X0Y1, X1Y1, etc.

[0097] The weight register file 620 temporarily stores weight operands for MAC operations performed by the PE 600. Weight operands include weights in filters of the DNN layer. In some embodiments, the weight register file 620 can store a single weight operand at a time. In other embodiments, the weight register file 620 can store multiple weight operands or a portion of a weight operand at a time. The weight operand may include multiple weights. The weights in the weight operands may be stored sequentially in the weight register file 620 so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and activation operands, each weight in the weight operand may correspond to an input element in the activation operand. The number of weights in the weight operand may be equal to the number of input elements in the activation operand.

[0098] In some embodiments, the weight register file 620 can be the same as or similar to the input register file 610, for example, having the same size. PE 600 may include multiple register files, some of which are designated as input register files 610 for storing activation operands, some as weight register files 620 for storing weight operands, and some as output register files 650 for storing output operands. In other embodiments, register files in PE 600 may be designated for other purposes, such as storing scale operands used in element-by-element addition operations. The designation of register files may be controlled by control module 340.

[0099] The multiplier 630 performs a multiplication operation on the activation operand and the weight operand. The multiplier 630 can perform a multiplication operation sequence on a single activation operand and a single weight operand, and generate a product operand including a product sequence. Each multiplication operation in the sequence includes multiplying the input element in the activation operand with the weight in the weight operand. In some embodiments, the position (or index) of the input element in the activation operand matches the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the activation operand and the first weight in the weight operand, the second multiplication operation is the multiplication of the second input element in the activation operand and the second weight in the weight operand, the third multiplication operation is the multiplication of the third input element in the activation operand and the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation can correspond to the same depth channel, and their product can also correspond to the same depth channel.

[0100] Multiple multipliers 630 can perform multiplication operations simultaneously. These multiplication operations can be referred to as a round of multiplication operations. In a round of multiplication operations performed by these multipliers 630, each multiplier 630 can use different activation operands and different weight operands. Different activation operands or weight operands can be stored in different register files of the PE 600. For example, a first multiplier 630 uses a first activation operand (e.g., stored in the first input register file 610) and a first weight operand (e.g., stored in the first weight register file 620), a second multiplier 630 uses a second activation operand (e.g., stored in the second input register file 610) and a second weight operand (e.g., stored in the second weight register file 620), a third multiplier 630 uses a third activation operand (e.g., stored in the third input register file 610) and a third weight operand (e.g., stored in the third weight register file 620), and so on. For each multiplier 630, a round of multiplication operations can include multiple cycles. A cycle consists of a multiplication operation on the input elements and the weights.

[0101] The multiplier 630 can perform multiple rounds of multiplication operations. The multiplier 630 can use the same weight operand but different activation operands in different rounds. For example, the multiplier 630 performs a sequence of multiplication operations on a first activation operand stored in a first input register file in a first round, and performs a sequence of multiplication operations on a second activation operand stored in a second input register file in a second round. In the second round, another multiplier 630 can perform another sequence of multiplication operations using the first activation operand and a different weight operand. In this way, the first activation operand is reused in the second round. The first activation operand can be further reused in additional rounds, for example, by additional multipliers 630.

[0102] The internal adder component 640 includes one or more adders, i.e., internal adders, inside the PE 600. The internal adder component 640 can perform an accumulation operation on two or more product operands from the multiplier 630 and generate an output operand of the PE 600. In some embodiments, the internal adders are arranged in a hierarchical sequence. The hierarchy includes one or more internal adders. For the first level of the internal adder component 640, the internal adder can receive product operands from two or more multipliers 630 and generate a sum operand through a sequence of accumulation operations. Each accumulation operation generates the sum of two or more products, each product coming from a different multiplier 630. The sum operand includes a sequence of sums, each of which is the result of an accumulation operation and corresponds to a depth channel. For (one or more) other levels of the internal adder component 640, the internal adder in the level receives a sum operand from the previous level in the sequence. Each of these values ​​can be generated by a different internal adder in the previous level. The ratio of the number of internal adders in a level to the number of internal adders in a subsequent level can be 2: 1. In some embodiments, the last level of the internal adder component 640 can include a single internal adder that generates the output operand of the PE 600 .

[0103] The output register file 650 stores the output operands of the PE 600. In some embodiments, the output register file 650 can store one output operand at a time. In other embodiments, the output register file 650 can store multiple output operands or a portion of an output operand at a time. The output operands include multiple output elements in the IFM. The output elements in the output operands can be stored sequentially in the output register file 650 so that the output elements can be processed sequentially. In some embodiments, each output element in the output operands corresponds to a different depth channel, and each output element is an element of a different output channel of the depthwise convolution. The number of output elements in the output operands can be equal to the number of depth channels of the depthwise convolution.

[0104] Sparsity acceleration in example PE

[0105] Figure 7 FIGURE 7 illustrates the sparsity acceleration of PE 700 in MAC operations according to various embodiments. PE 700 may be Figure 5 An example of PE 510 in Figure 7 In the embodiment of the present invention, PE 700 includes an input register file 710, a weight register file 720, a multiplier 730, an accumulator 740, and an output register file 750. In other embodiments, PE 700 may include fewer, more, or different components. PE 700 is associated with a sparsity module 760. Sparsity module 760 may be Figure 4 An embodiment of the medium sparsity module 430.

[0106] The input register file 710 stores at least some activation operands. The activation operands include a sequence of input elements (i.e., activation values). The activation operands may be part of an input tensor (e.g., an input tensor of a convolutional layer). The activation operands are associated with an activation bitmap 715. The activation bitmap 715 may be stored in the input register file 710, in a local memory of a computational block including PE 700, or in both. The activation bitmap 715 may indicate the location of non-zero activation values ​​in the activation operands. The activation bitmap 715 includes a sequence of bits, each bit corresponding to a corresponding activation value in the activation operand. In some embodiments, the positions of the bits in the activation bitmap 715 match the positions of the corresponding activation values ​​in the activation operands. For illustrative purposes, the activation bitmap 715 includes eight bits, and the activation operand includes eight activation values. In other embodiments, the activation bitmap 715 may include fewer or more bits. As Figure 7 As shown, four of the eight bits in activation bitmap 715 are zero-valued bits, and the remaining four are one-valued bits. A zero-valued bit indicates that the corresponding activation value is zero, and a one-valued bit indicates that the corresponding activation value is non-zero. Therefore, the activation operand includes four zero-valued activation values ​​and four non-zero-valued activation values.

[0107] The weight register file 720 stores at least a portion of the weight operands. The weight operands include a sequence of weights. The weight operands may be part of a filter (e.g., a filter of a convolutional layer). The weight operands are associated with a weight bitmap 725. The weight bitmap 725 may be stored in the weight register file 720, in a local memory of a computational block including the PE 700, or in both. The weight bitmap 725 may indicate the location of non-zero valued weights in the weight operands. The weight bitmap 725 includes a sequence of bits, each bit corresponding to a corresponding weight in the weight operand. In some embodiments, the positions of the bits in the weight bitmap 725 match the positions of the corresponding weights in the weight operands. For illustrative purposes, the weight bitmap 725 includes eight bits and the weight operand includes eight weights. In other embodiments, the weight bitmap 725 may include fewer or more bits. As Figure 7 As shown, four of the eight bits in the weight bitmap 725 are zero-valued bits, and the remaining four are one-valued bits. The zero-valued bits indicate that the value of the corresponding weight is zero, and the one-valued bits indicate that the value of the corresponding weight is non-zero. Therefore, the weight operand includes four zero-valued weights and four non-zero-valued weights. The weight bitmap 725 can indicate the positions of the non-zero-valued weights in the weight operand.

[0108] Sparsity module 760 generates a combined sparsity bitmap 735 based on activation bitmap 715 and weight bitmap 725. Sparsity module 760 may receive activation bitmap 715 from input register file 710 or from the local memory of a computation block including PE 700. Sparsity module 760 may receive weight bitmap 725 from weight register file 720 or from the local memory of a computation block. In some embodiments, sparsity module 760 is an AND operator. Sparsity module 760 may generate combined sparsity bitmap 735 by performing one or more AND operations on activation bitmap 715 and weight bitmap 725. Each bit in combined sparsity bitmap 735 is the result of an AND operation on a bit in activation bitmap 715 and a bit in weight bitmap 725. The positions of the bits in combined sparsity bitmap 735 match the positions of the bits in activation bitmap 715 and the positions of the bits in weight bitmap 725. For example, the first bit in the combined sparsity bitmap 735 is the result of an AND operation on the first bit in the activation bitmap 715 and the first bit in the weight bitmap 725, the second bit in the combined sparsity bitmap 735 is the result of an AND operation on the second bit in the activation bitmap 715 and the second bit in the weight bitmap 725, the third bit in the combined sparsity bitmap 735 is the result of an AND operation on the third bit in the activation bitmap 715 and the third bit in the weight bitmap 725, and so on.

[0109] When the value of the corresponding bit in the activation bitmap 715 and the corresponding bit in the weight bitmap 725 are both one, the value of the bit in the combined sparsity bitmap 735 is one. When at least one of the values ​​of the corresponding bit in the activation bitmap 715 and the corresponding bit in the weight bitmap 725 is zero, the value of the bit in the combined sparsity bitmap 735 is zero. Figure 7 As shown, combined sparsity bitmap 735 includes six zeros and two ones.

[0110] The total number of ones in the combined sparsity bitmap 735 is equal to the total number of non-zero activation value-weight pairs that will be calculated by PE 700 to calculate the non-zero partial sum. Other activation value-weight pairs are zero-valued activation value-weight pairs, and calculation of other activation value-weight pairs can be skipped without any impact on output accuracy because these pairs will produce zero-valued partial sums. Therefore, the workload of PE 700 in that calculation round can be determined based on the total number of ones in the combined sparsity bitmap 735. The amount of time for calculation can also be estimated based on the total number of ones in the combined sparsity bitmap 735. The more ones in the combined sparsity bitmap 735, the higher the workload of PE 700 and the longer the calculation time of PE 700.

[0111] In some embodiments, the input register file 710 or the weight register file 720 stores dense data points (e.g., non-zero activation values ​​or non-zero weights). Sparse data points (e.g., zero activation values ​​or zero weights) are not stored in the input register file 710 or the weight register file 720. Dense data points can be compressed and stored adjacent to each other in the input register file 710 or the weight register file 720. The (one or more) dense data points of the activation operand are compressed activation operands. The (one or more) dense data points of the weight operand constitute the compressed weight operand. The position of one in the combined sparsity bitmap 735 cannot indicate the position of the activation value in the compressed activation operand or the position of the weight in the compressed weight operand. The sparsity module 760 can perform a sparsity calculation to determine the position of the activation value in the compressed activation operand and the position of the weight in the compressed weight operand. The sparsity module 760 can perform a sparsity calculation round for each pair of two non-zero activation value-weight pairs. In each sparsity calculation round, the sparsity module 760 can calculate an activation position bitmap and a weight position bitmap based on the activation bitmap 715, the weight bitmap 725, and the combined sparsity bitmap 735. The positions of the activation values ​​in the compressed activation operand can be indicated by the number of ones in the activation position bitmap, and the positions of the weights in the compressed weight operand can be indicated by the number of ones in the weight position bitmap. In the first sparsity calculation round, an intermediate bitmap can be determined, and the intermediate bitmap can be used in the second round to identify the next non-zero activation value-weight pair.

[0112] Sparsity module 760 can read the activation values ​​and weights of non-zero activation value-weight pairs from input register file 710 and weight register file 720 based on the positions determined by the sparsity calculation, and provide the activation values ​​and weights to multiplier 730. Multiplier 730 performs a multiplication operation on the activation values ​​and weights. For example, multiplier 730 performs a multiplication operation on the activation values ​​and weights in each non-zero activation value-weight pair and outputs a partial sum, i.e., the product of the activation value and the weight. Since there are two activation value-weight pairs, multiplier 730 can, for example, perform two multiplication operations sequentially based on the position of a in combined sparsity bitmap 735. Without sparsity acceleration, multiplier 730 would need to perform eight multiplication operations. By reducing the number of multiplication operations from eight to two, the MAC operations in PE 700 are accelerated. Since DNN accelerators typically perform a large number of MAC operations when executing a DNN, sparsity acceleration can significantly improve the efficiency and performance of DNN accelerators.

[0113] The accumulator 740 receives the two partial sums from the multiplier 730 and accumulates the two partial sums. The result of the accumulation is a PE-level internal partial sum. The PE-level internal partial sum can be stored in the output register file 750. In some embodiments, the accumulator 740 receives one or more PE-level internal partial sums from one or more other PEs. The accumulator 740 can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of PE 700 and store the accumulated result (i.e., the multiple PE internal partial sums) in the output register file 750. The one or more other PEs can be in the same column in the PE array as PE 700. The multiple PE internal partial sums can be column-level internal partial sums. In some embodiments, the PE-level internal partial sum or the multiple PE internal partial sum of PE 700 can be sent to one or more other PEs for further accumulation.

[0114] although Figure 7 A single multiplier 730 is shown, but PE 700 may include multiple multipliers that can perform multiple multiplication operations simultaneously. These multipliers may be coupled to an internal adder component, such as internal adder component 640.

[0115] Example sparsity acceleration logic

[0116] Figure 8 FIGURE 1 illustrates a sparsity calculation for identifying pairs of non-zero activation values ​​and non-zero weight values ​​according to various embodiments. The sparsity calculation may be performed by Figure 4 The sparsity module 430 in (eg, the sparsity accelerator 440 in the sparsity module 430) or Figure 7 The sparsity module 760 in FIG. For illustration, non-zero activation values ​​are located in an activation operand comprising eight activation values, and non-zero weights are located in a weight operand comprising eight weights. In other embodiments, the activation operands or weight operands may include other numbers of data points. The activation operands and weight operands may be used, for example, by a PE to perform a MAC operation in a convolution. The activation operand may be part of the input tensor of the convolution. The weight operand may be part of the filter of the convolution.

[0117] The activation operand is associated with an activation sparsity vector 810 comprising eight elements, each element corresponding to a corresponding activation value in the activation operand and indicating whether the activation value is zero or non-zero. The weight operand is associated with a weight sparsity vector 820 comprising eight elements, each element corresponding to a corresponding weight in the weight operand and indicating whether the weight is zero or non-zero. A combined sparsity vector 830 is generated by performing an AND operation on the activation sparsity vector 810 and the weight sparsity vector 820. Each element in the combined sparsity vector 830 is the product of the corresponding element in the activation sparsity vector 810 and the corresponding element in the weight sparsity vector 820. The position of the element in the combined sparsity vector 830 matches the position of the corresponding element in the activation sparsity vector 810 and the position of the corresponding element in the weight sparsity vector 820. The combined sparsity vector 830 shows that there are two non-zero activation value-weight pairs for the MAC operation because there are two ones in the combined sparsity vector 830. Two rounds of sparsity computation may be required to identify the activation and weight in these two non-zero activation-weight pairs.

[0118] An initial control vector 835 is defined for the first sparsity calculation round. The first sparsity calculation round can be used to identify the activation value and weight in the first non-zero activation value-weight pair. Initial control vector 835 includes eight elements. Since this is the first sparsity calculation round, all eight elements in initial control vector 835 are zero. In other rounds, initial control vector 835 can be determined in a previous round. Vector 840 is generated by performing an AND operation on the inverse of initial control vector 835 and combined sparsity vector 830. The inverse of initial control vector 835 includes eight elements, all of which are one. Another vector 845 is generated by subtracting one from vector 840. Furthermore, control vector 850 is generated by performing an AND operation on vector 840 and vector 845. Vector 855 is generated by inverting control vector 850. Vector 860 is generated by performing an AND operation on vector 845 and vector 855.

[0119] An activation position vector 870 is generated by performing an AND operation on activation sparsity vector 810 and vector 860. The number of ones in activation position vector 870 is counted, which indicates the position of the activation value in the compressed activation operand. In addition, a weight position vector 880 is generated by performing an AND operation on weight sparsity vector 820 and vector 860. The number of ones in weight position vector 880 is counted, which indicates the position of the weight in the compressed weight operand. Finally, a subsequent control vector 890 is calculated by performing an XOR operation on vector 840 and vector 845. Subsequent control vector 890 will be used as the initial control vector for the second sparsity calculation round. The second sparsity calculation round can be used to identify the activation value and weight in the second non-zero activation value-weight pair.

[0120] exist Figure 8 In the embodiment shown, there are two non-zero activation value-weight pairs, so the sparsity calculation can be stopped when the second round is completed. In other embodiments, there can be more sparsity calculation rounds to identify more non-zero activation value-weight pairs. In some embodiments, the sparsity calculation can be terminated after determining that all subsequent control vectors 890 in a particular round are ones. Figure 8 The vectors shown in FIG8 (e.g., activation sparsity vector 810, weight sparsity vector 820, combined sparsity vector 830, initial control vector 835, vector 840, vector 845, control vector 845, vector 855, vector 860, activation position vector 870, weight position vector 880, or subsequent control vector 890) can be bitmaps. In embodiments where the vectors are bitmaps, each element of the vectors can be a bit.

[0121] The control vector 850 can be used to detect control faults in a sparse calculation round. The control vector 850 can carry the computation information from the current sparse calculation round to the next round. Although a fault manifested in the control vector 850 may cause a decrease in classification accuracy in the DNN accelerator, it may not cause an actual control fault. This provides an opportunity to mitigate the control fault. Other vectors generated in the sparse calculation round may not have the computation information for that round. Alternatively, other vectors generated in the sparse calculation round may cause an actual control fault to occur before the control fault is actually detected, making it too late to mitigate the control fault before the control fault ultimately occurs in the DNN accelerator.

[0122] Example Fault Detection

[0123] Figure 9 FIGURE 1 shows the detection of failures in sparsity computation according to various embodiments. The detection of failures can be performed by Figure 4 The sparsity module 430 in (e.g., the failure detector 450 in the sparsity module 430) or Figure 7 For illustration purposes, the sparsity calculation can be Figure 8 The sparsity calculation shown in .

[0124] A bit count (popcount) 910 of the control vector 850 in the current sparsity calculation round is determined. Bit count 910 can be equal to the number of ones (one or more) in the control vector 850. Another bit count 920 is also determined. Bit count 920 can correspond to the control vector in the previous sparsity calculation round. Bit count 920 is compared to the sum of bit count 910 plus one. In embodiments where bit count 920 is equal to the sum, no control fault is detected and the current sparsity calculation round can continue. For example, vector 855, vector 860, activation position vector 870, weight position vector 880, and subsequent control vector 890 can be calculated. In embodiments where bit count 920 is not equal to the sum, a control fault is detected.

[0125] After detecting a control fault, the fault mitigator 460 may receive an instruction to mitigate the control fault (e.g., from the fault detector 450). In some embodiments, the control fault is mitigated based on the control vector of the previous round ("previous control vector"). The previous round may be the round immediately preceding the current round. The previous control vector may be stored in a temporary register in the previous round. The fault mitigator 460 may obtain the previous control vector and replace the last "1" in the previous control vector with "0" to generate a new control vector. The new control vector will be used to calculate other vectors in the current round (e.g., vector 855, vector 860, activation position vector 870, weight position vector 880, and subsequent control vector 890). By correcting the control vector in the current round, the control fault can be prevented from propagating in the control vectors of subsequent rounds. The reduced accuracy of the DNN can be restored to a certain extent.

[0126] Computational errors can also be detected. To determine whether there are computational errors in the sparsity calculation round, the bit count 930 of the activation position vector 870 is determined. Bit count 930 can be equal to the number of ones (one or more) in the activation position vector 870. In addition, the bit count 940 of the activation sparsity vector 810 is determined. Bit count 940 can be equal to the number of ones (one or more) in the activation sparsity vector 810. Bit count 930 and bit count 940 are compared. Similarly, the bit count 950 of the weight position vector 880 is determined. Bit count 950 can be equal to the number of ones (one or more) in the weight position vector 880. In addition, the bit count 960 of the weight sparsity vector 820 is determined. Bit count 960 can be equal to the number of ones (one or more) in the weight sparsity vector 820. Bit count 950 and bit count 960 are compared.

[0127] In embodiments where bit count 930 is not greater than bit count 940 and bit count 950 is not greater than bit count 960, no computation error has been detected and the sparsity computation round can continue. For example, a subsequent control vector 890 can be determined. Additionally, the activation values ​​identified by activation position vector 870 and the weights identified by weight position vector 880 can be provided to the PE for MAC operations. In embodiments where bit count 930 is greater than bit count 940 or bit count 950 is greater than bit count 960, a computation error has been detected. The computation error can be mitigated, for example, by fault mitigator 460.

[0128] Example Fault Mitigation

[0129] Figure 10 The present invention illustrates mitigating computational errors in sparsity computations according to various embodiments. Figure 8 The sparsity calculation shown is at least part of the error. The error in the calculation can be expressed as Figure 9 The computational errors detected by the method shown can be mitigated by Figure 4 The fault mitigator 460 in is executed. Figure 10 In the embodiment of the present invention, computation errors are mitigated by using a redundancy method in which three different variants of the sparsity acceleration logic are used to determine the activation position index and the weight position index, respectively. In other embodiments, other numbers of sparsity acceleration logic variants may be used.

[0130] like Figure 10 As shown, the three variants of the sparsity acceleration logic all start from the same activation sparsity vector 810, weight sparsity vector 820, vector 840, vector 845 and control vector 850. The three variants of the sparsity acceleration logic are different in the rest of the sparsity calculation. The first variant is Figure 8 The sparsity calculation shown is performed and generates vector 855, vector 860, activation position vector 870, and weight position vector 880. The position index of the activation value in the compressed activation operand is equal to the number of ones in activation position vector 870. The position index of the weight in the compressed weight operand is equal to the number of ones in weight position vector 880.

[0131] Unlike the first variant, the second variant calculates vector 1010, for example, by performing an OR operation on the inverse of vector 845 and control vector 850. Further, activation position vector 1020 is generated by performing an OR operation on the sum of the inverse of activation sparsity vector 810 and vector 1010. In addition, weight position vector 1030 is generated by performing an OR operation on the sum of the inverse of weight sparsity vector 820 and vector 1010. The position index of the activation value in the compressed activation operand is equal to the number of zeros in activation position vector 1020. The position index of the weight in the compressed weight operand is equal to the number of zeros in weight position vector 1030.

[0132] The third variant is different from the first and second variants. The third variant includes performing an AND operation on the activation sparsity vector 810 and the vector 845 to generate the vector 1040. The vector 1050 is generated by performing an AND operation on the weight sparsity vector 820 and the vector 845. Further, the activation position vector 1060 is generated by performing an XOR operation on the control vector 850 and the vector 1040. In addition, the weight position vector 1070 is generated by performing an XOR operation on the control vector 850 and the vector 1050. The position index of the activation value in the compressed activation operand is equal to the number of zeros (one or more) in the activation position vector 1060. The position index of the weight in the compressed weight operand is equal to the number of zeros (one or more) in the weight position vector 1070.

[0133] These three variants can be performed simultaneously to avoid additional time consumption. The final position of the activation value is determined based on the three position indices of the activation values ​​output by the three variants. In some embodiments, two or all three position indices may match. The position index with the most votes (i.e., the position index output by the most variants) can be selected and used to identify the activation value. Similarly, the final position of the weight is determined based on the three position indices of the weight output by the three variants. In some embodiments, two or all three position indices may match. The position index with the most votes (i.e., the position index output by the most variants) can be selected and used to identify the weight.

[0134] This mitigation can reduce or even eliminate the degradation in accuracy of the DNN run by the DNN accelerator, even in the presence of circuit-level faults. This redundancy approach has minimal overhead in terms of time, power, and area. Adding additional sparsity acceleration logic variants does not add additional computational cycles to the sparsity logic because all variants are executed in parallel. Variants of the sparsity acceleration logic can be implemented using logic gates, which is neither difficult nor expensive in hardware implementation. Therefore, the redundancy approach can provide an effective solution to mitigate computational errors in sparsity computations without significantly affecting the performance of the DNN accelerator.

[0135] Example method for detecting failures in sparsity computations

[0136] Figure 11 is a flow chart illustrating a method 1100 for detecting failures in sparsity computations, according to various embodiments. The method 1100 may be performed by Figure 4 Although the method 1100 is based on the sparsity module 430 of Figure 11 The flowchart shown is described, but many other methods for detecting failures in sparsity computations can also be used. For example, one can change Figure 11 As another example, some steps can be changed, deleted, or combined.

[0137] The sparsity module 430 stores (1110) compressed activation operands. The compressed activation operands include one or more non-zero activation values ​​for activation operands in the deep learning operation. The compressed activation operands can be stored in local memory (e.g., local memory 410) or one or more register files in the PE array. The activation operands also include one or more zero-valued activation values, which may not be stored in local memory or register file(s).

[0138] The sparsity module 430 stores (1120) compressed weight operands. The compressed weight operands include one or more non-zero weights of the weight operands in the deep learning operation. The compressed weight operands can be stored in local memory (e.g., local memory 410) or one or more register files in the PE array. The weight operands also include one or more zero-valued weights that may not be stored in local memory or (one or more) register files.

[0139] The sparsity module 430 generates (1130) a bitmap based on the activation sparsity vector and the weight sparsity vector. The activation sparsity vector indicates one or more non-zero activation values ​​at one or more positions in the activation operand. The weight sparsity vector indicates one or more non-zero weights at one or more positions in the weight operand. In some embodiments, the bitmap is generated based on a previous bitmap. Another non-zero activation value in the compressed activation operand or another non-zero weight in the compressed weight operand is identified based on the previous bitmap.

[0140] The sparsity module 430 identifies (1140) non-zero valued activations in the compressed activation operand or non-zero valued weights in the compressed weight operand based on the bitmap. In some embodiments, the sparsity module 430 determines the location of the non-zero valued activations in the compressed activation operand. The sparsity module 430 determines the location of the non-zero valued weights in the compressed weight operand. In deep learning operations, the non-zero valued activations are multiplied by the non-zero valued weights.

[0141] The sparsity module 430 determines (1150) whether there is a failure in identifying non-zero valued activation values ​​or non-zero valued weights based on the number of one or more non-zero elements in the bitmap. In one embodiment, the sparsity module 430 determines the number of one or more non-zero elements in the previous bitmap. The sparsity module 430 determines whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one. In another embodiment, the sparsity module 430 determines whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued activation values ​​in the activation operand. In yet another embodiment, the sparsity module 430 determines whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued weights in the weight operand.

[0142] In one embodiment, upon determining that there is a failure in identifying non-zero valued activations or non-zero valued weights, the sparsity module 430 generates a new bitmap based on the previous bitmap, for example, by replacing non-zero elements in the previous bitmap with zeros. The sparsity module 430 identifies non-zero valued activations in the compressed activation operand or non-zero valued weights in the compressed weight operand based on the new bitmap.

[0143] In another embodiment, after determining that there is a failure in identifying non-zero valued activation values, the sparsity module 430 generates a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector. The sparsity module 430 determines the position of the non-zero valued activation values ​​in the compressed activation operand based on the bitmap. The sparsity module 430 determines the first position of the non-zero valued activation value in the compressed activation operand based on the first bitmap. The sparsity module 430 determines the second position of the non-zero valued activation value in the compressed activation operand based on the second bitmap. The sparsity module 430 identifies the non-zero valued activation value in the compressed activation operand based on the position, the first position, and the second position.

[0144] In yet another embodiment, after determining that there is a failure in identifying non-zero valued weights, the sparsity module 430 generates a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector. The sparsity module 430 determines the positions of the non-zero valued weights in the compressed weight operand based on the bitmap. The sparsity module 430 determines the first position of the non-zero valued weights in the compressed weight operand based on the first bitmap. The sparsity module 430 determines the second position of the compressed non-zero valued weights in the weight operand based on the second bitmap. The sparsity module 430 identifies the non-zero valued weights in the compressed weight operand based on the position, the first position, and the second position.

[0145] In yet another embodiment, upon determining that there is a failure in identifying a non-zero valued activation value or a non-zero valued weight, the sparsity module 430 generates a new bitmap based on the previous bitmap. The sparsity module 430 identifies another non-zero valued activation value in the compressed activation operand. The other non-zero valued activation value immediately follows the previously identified non-zero valued activation value in the compressed activation operand.

[0146] Example computing device

[0147] Figure 12 is a block diagram of an example computing device 1200 according to various embodiments. In some embodiments, the computing device 1200 may be used as Figure 3 At least a portion of the DNN accelerator 300 in. Multiple components in Figure 12 1200, but any one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system on a chip (SoC) die. Additionally, in various embodiments, computing device 1200 may not include Figure 12 1200 may include one or more of the components shown, but the computing device 1200 may include interface circuitry for coupling to the one or more components. For example, the computing device 1200 may not include the display device 1206, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which the display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include the audio input device 1218 or the audio output device 1208, but may include audio input or output device interface circuitry (e.g., a connector and supporting circuitry) to which the audio input device 1218 or the audio output device 1208 may be coupled.

[0148] The computing device 1200 may include a processing device 1202 (e.g., one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1200 may include a memory 1204, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state state memory, and / or a hard drive. In some embodiments, the memory 1204 may include a memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-transitory computer-readable media storing instructions that are executable to perform operations for detecting and mitigating failures in sparsity computations in DNNs, for example, in combination with the above. Figure 11 The method 1100 described above, or a combination thereof Figure 4 Some of the operations performed by the sparsity module 430 are described. Instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 1202.

[0149] In some embodiments, computing device 1200 may include a communication chip 1212 (e.g., one or more communication chips). For example, communication chip 1212 may be configured to manage wireless communications for transmitting data to and from computing device 1200. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, and the like that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not imply that the associated devices do not contain any wires, although in some embodiments they may not.

[0150] The communication chip 1212 may implement any of a number of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), the Long Term Evolution (LTE) project and any amendments, updates, and / or revisions (e.g., the LTE-Advanced project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband wireless access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks, which is an abbreviation for Worldwide Interoperability for Microwave Access and is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 1212 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 1212 can also operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1212 may operate according to code-division multiple access (CDMA), time division multiple access (TDMA), digital enhanced cordless telecommunication (DECT), evolution-data optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, etc. In other embodiments, the communication chip 1212 may operate according to other wireless protocols.Computing device 1200 may include antenna 1222 to facilitate wireless communications and / or receive other wireless communications (eg, AM or FM radio transmissions).

[0151] In some embodiments, the communication chip 1212 can manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1212 can include multiple communication chips. For example, the first communication chip 1212 can be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and the second communication chip 1212 can be dedicated to long-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1212 can be dedicated to wireless communications, and the second communication chip 1212 can be dedicated to wired communications.

[0152] The computing device 1200 may include a battery / power circuit 1214. The battery / power circuit 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0153] Computing device 1200 may include a display device 1206 (or corresponding interface circuitry as described above). For example, display device 1206 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat-panel display.

[0154] Computing device 1200 may include an audio output device 1208 (or corresponding interface circuitry as described above). For example, audio output device 1208 may include any device that generates an audible indicator, such as a speaker, headphones, or earbuds.

[0155] The computing device 1200 may include an audio input device 1218 (or corresponding interface circuitry as described above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., an instrument with a MIDI (Musical Instrument Digital Interface) output).

[0156] Computing device 1200 may include a GPS device 1216 (or corresponding interface circuitry as described above). As is known in the art, GPS device 1216 may communicate with a satellite-based system and may receive the location of computing device 1200.

[0157] Computing device 1200 may include other output devices 1210 (or corresponding interface circuitry as described above). Examples of other output devices 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0158] Computing device 1200 may include other input devices 1220 (or corresponding interface circuitry as described above). Examples of other input devices 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0159] The computing device 1200 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smartphone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA (personal digital assistant), an ultraportable personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 can be any other electronic device that processes data.

[0160] Select Example

[0161] The following paragraphs provide various examples of the embodiments disclosed herein.

[0162] Example 1 provides a method for deep learning, comprising: storing compressed activation operands and compressed weight operands, wherein the compressed activation operands include one or more non-zero valued activation values ​​in an activation operand in a deep learning operation, and the compressed weight operands include one or more non-zero valued weights in a weight operand in the deep learning operation; generating a bitmap based on an activation sparsity vector and a weight sparsity vector, the activation sparsity vector indicating one or more positions of the one or more non-zero valued activation values ​​in the activation operand, and the weight sparsity vector indicating one or more positions of the one or more non-zero valued weights in the weight operand; identifying the non-zero valued activation values ​​in the compressed activation operands or the non-zero valued weights in the compressed weight operands based on the bitmap; and determining whether there is a failure in identifying the non-zero valued activation values ​​or the non-zero valued weights based on the number of one or more non-zero elements in the bitmap.

[0163] Example 2 provides the method described in Example 1, wherein the bitmap is generated based on a previous bitmap, another non-zero valued activation value in the compression activation operand or another non-zero valued weight in the compression weight operand is identified based on the previous bitmap, and determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: determining the number of one or more non-zero elements in the previous bitmap; and determining whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

[0164] Example 3 provides the method described in Example 2, wherein identifying the non-zero value activation value in the compressed activation operand includes: generating a new bitmap based on the previous bitmap after determining that there is a failure in identifying the non-zero value activation value or the non-zero value weight; and identifying the non-zero value activation value in the compressed activation operand or the non-zero value weight in the compressed weight operand based on the new bitmap.

[0165] Example 4 provides the method of Example 3, wherein generating a new bitmap based on the previous bitmap comprises replacing non-zero elements in the previous bitmap with zeros.

[0166] Example 5 provides a method as described in any of the previous examples, wherein determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: determining whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued activation values ​​in the activation operand.

[0167] Example 6 provides a method as described in any of the previous examples, wherein determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: determining whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued weights in the weight operand.

[0168] Example 7 provides a method as described in any of the previous examples, wherein identifying the non-zero value activation value in the compressed activation operand includes: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero value activation value; determining a position of the non-zero value activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero value activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero value activation value in the compressed activation operand based on the second bitmap; and identifying the non-zero value activation value in the compressed activation operand based on the position, the first position, and the second position.

[0169] Example 8 provides a method of any of the previous examples, wherein identifying the non-zero valued weight in the compressed weight operand includes: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero valued weight; determining a position of the non-zero valued weight in the compressed weight operand based on the bitmap; determining a first position of the non-zero valued weight in the compressed weight operand based on the first bitmap; determining a second position of the non-zero valued weight in the compressed weight operand based on the second bitmap; and identifying the non-zero valued weight in the compressed weight operand based on the position, the first position, and the second position.

[0170] Example 9 provides a method as described in any of the previous examples, wherein identifying the non-zero value activation value in the compressed activation operand includes: after determining that there is a failure in identifying the non-zero value activation value, identifying another non-zero value activation value in the compressed activation operand, wherein the another non-zero value activation value immediately follows the previously identified non-zero value activation value in the compressed activation operand.

[0171] Example 10 provides the method of any of the previous examples, wherein identifying the non-zero valued activation value in the compressed activation operand or the non-zero valued weight in the compressed weight operand comprises: determining a position of the non-zero valued activation value in the compressed activation operand; and determining a position of the non-zero valued weight in the compressed weight operand, wherein, in the deep learning operation, the non-zero valued activation value is multiplied by the non-zero valued weight.

[0172] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for intra-network computation, the operations comprising: storing compressed activation operands and compressed weight operands, wherein the compressed activation operands comprise one or more non-zero-valued activation values ​​in an activation operand in a deep learning operation, and the compressed weight operands comprise one or more non-zero-valued weights in a weight operand in the deep learning operation; generating a bitmap based on an activation sparsity vector and a weight sparsity vector, the activation sparsity vector indicating one or more locations of the one or more non-zero-valued activation values ​​in the activation operands, and the weight sparsity vector indicating one or more locations of the one or more non-zero-valued weights in the weight operands; identifying non-zero-valued activation values ​​in the compressed activation operands or non-zero-valued weights in the compressed weight operands based on the bitmap; and determining whether there is a failure in identifying the non-zero-valued activation values ​​or the non-zero-valued weights based on the number of one or more non-zero elements in the bitmap.

[0173] Example 12 provides one or more non-transitory computer-readable media of Example 11, wherein the bitmap is generated based on a previous bitmap, another non-zero-valued activation value in the compressed activation operand or another non-zero-valued weight in the compressed weight operand is identified based on the previous bitmap, and determining whether a failure exists in identifying the non-zero-valued activation value or the non-zero-valued weight includes: determining the number of one or more non-zero elements in the previous bitmap; and determining whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

[0174] Example 13 provides one or more non-transitory computer-readable media as described in Example 11 or 12, wherein determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: determining whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued activation values ​​in the activation operand.

[0175] Example 14 provides one or more non-transitory computer-readable media of any of Examples 11 to 13, wherein determining whether a failure exists in identifying the non-zero valued activation value or the non-zero valued weight comprises determining whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued weights in the weight operand.

[0176] Example 15 provides one or more non-transitory computer-readable media of any one of Examples 11 to 14, wherein identifying the non-zero valued activation value in the compressed activation operand includes: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero valued activation value; determining a position of the non-zero valued activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero valued activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero valued activation value in the compressed activation operand based on the second bitmap; and identifying the non-zero valued activation value in the compressed activation operand based on the position, the first position, and the second position.

[0177] Example 16 provides one or more non-transitory computer-readable media of any one of Examples 11 to 15, wherein identifying the non-zero value activation value in the compressed activation operand includes: after determining that there is a failure in identifying the non-zero value activation value, identifying another non-zero value activation value in the compressed activation operand, wherein the another non-zero value activation value immediately follows the previously identified non-zero value activation value in the compressed activation operand.

[0178] Example 17 provides an apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions, the computer program instructions executable by the computer processor to perform operations, the operations comprising: storing compressed activation operands and compressed weight operands, wherein the compressed activation operands comprise one or more non-zero valued activation values ​​in an activation operand in a deep learning operation, and the compressed weight operands comprise one or more non-zero valued weights in a weight operand in the deep learning operation; generating a bitmap based on an activation sparsity vector and a weight sparsity vector, the activation sparsity vector indicating one or more locations of the one or more non-zero valued activation values ​​in the activation operands, and the weight sparsity vector indicating one or more locations of the one or more non-zero valued weights in the weight operands; identifying the non-zero valued activation values ​​in the compressed activation operands or the non-zero valued weights in the compressed weight operands based on the bitmap; and determining whether there is a failure in identifying the non-zero valued activation values ​​or the non-zero valued weights based on the number of one or more non-zero elements in the bitmap.

[0179] Example 18 provides the apparatus of Example 17, wherein the bitmap is generated based on a previous bitmap, another non-zero valued activation value in the compressed activation operand or another non-zero valued weight in the compressed weight operand is identified based on the previous bitmap, and determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: determining the number of one or more non-zero elements in the previous bitmap; and determining whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

[0180] Example 19 provides the apparatus of Example 17 or 18, wherein determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight comprises determining whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero valued activation values ​​in the activation operand.

[0181] Example 20 provides an apparatus of any one of Examples 17 to 19, wherein identifying the non-zero value activation value in the compressed activation operand includes: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero value activation value; determining a position of the non-zero value activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero value activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero value activation value in the compressed activation operand based on the second bitmap; and identifying the non-zero value activation value in the compressed activation operand based on the position, the first position, and the second position.

[0182] The above description of the illustrated embodiments of the present disclosure, including that described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, various equivalent modifications may be made within the scope of the present disclosure, as will be appreciated by those skilled in the relevant art. These modifications may be made to the present disclosure in light of the above detailed description.

Claims

1. A method for deep learning, comprising: Storing compressed activation operands and compressed weight operands, wherein the compressed activation operands include one or more non-zero valued activation values ​​in activation operands in a deep learning operation and the compressed weight operands include one or more non-zero valued weights in weight operands in the deep learning operation; generating a bitmap based on an activation sparsity vector indicating one or more locations of the one or more non-zero valued activation values ​​in the activation operand and a weight sparsity vector indicating one or more locations of the one or more non-zero valued weights in the weight operand; identifying a non-zero valued activation value in the compressed activation operand or a non-zero valued weight in the compressed weight operand based on the bitmap; and A determination is made based on a number of one or more non-zero elements in the bitmap whether there is a failure in identifying the non-zero-valued activation value or the non-zero-valued weight.

2. The method according to claim 1, wherein The bitmap is generated based on a previous bitmap, another non-zero-valued activation value in the compressed activation operand or another non-zero-valued weight in the compressed weight operand is identified based on the previous bitmap, and determining whether a failure exists in identifying the non-zero-valued activation value or the non-zero-valued weight comprises: determining the number of one or more non-zero elements in the previous bitmap; and A determination is made as to whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

3. The method according to claim 2, wherein: Identifying the non-zero valued activation value in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap upon determining that there is a failure in identifying the non-zero-valued activation value or the non-zero-valued weight; and The non-zero-valued activation values ​​in the compressed activation operand or the non-zero-valued weights in the compressed weight operand are identified based on the new bitmap.

4. The method according to claim 3, wherein: Generating a new bitmap based on the previous bitmap includes: The non-zero elements in the previous bitmap are replaced with zeros.

5. The method according to any one of claims 1 to 4, wherein Determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: A determination is made as to whether a number of one or more non-zero elements in the bitmap is greater than a number of one or more non-zero valued activation values ​​in the activation operand.

6. The method according to any one of claims 1 to 5, wherein Determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: A determination is made as to whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero-valued weights in the weight operand.

7. The method according to any one of claims 1 to 6, wherein Identifying the non-zero valued activation value in the compressed activation operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero-valued activation value; determining a position of the non-zero activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero activation value in the compressed activation operand based on the second bitmap; and The non-zero valued activation value in the compressed activation operand is identified based on the position, the first position, and the second position.

8. The method according to any one of claims 1 to 7, wherein Identifying the non-zero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero-valued weights; determining a position of the non-zero valued weight in the compressed weight operand based on the bitmap; determining a first position of the non-zero-valued weight in the compressed weight operand based on the first bitmap; determining a second position of the non-zero-valued weight in the compressed weight operand based on the second bitmap; and The non-zero valued weight in the compressed weight operand is identified based on the position, the first position, and the second position.

9. The method according to any one of claims 1 to 8, wherein Identifying the non-zero valued activation value in the compressed activation operand comprises: After determining that a failure exists in identifying a non-zero valued activation value, another non-zero valued activation value is identified in the compressed activation operand, wherein the another non-zero valued activation value immediately follows the previously identified non-zero valued activation value in the compressed activation operand.

10. The method according to any one of claims 1 to 9, wherein Identifying the non-zero valued activation value in the compressed activation operand or the non-zero valued weight in the compressed weight operand comprises: determining a position of the non-zero valued activation value in the compressed activation operand; and determining the position of the non-zero valued weight in the compressed weight operand, Wherein, in the deep learning operation, the non-zero activation value is multiplied by the non-zero weight.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations for intra-network computing, the operations comprising: Storing compressed activation operands and compressed weight operands, wherein the compressed activation operands include one or more non-zero valued activation values ​​in activation operands in a deep learning operation and the compressed weight operands include one or more non-zero valued weights in weight operands in the deep learning operation; generating a bitmap based on an activation sparsity vector indicating one or more locations of the one or more non-zero valued activation values ​​in the activation operand and a weight sparsity vector indicating one or more locations of the one or more non-zero valued weights in the weight operand; identifying a non-zero valued activation value in the compressed activation operand or a non-zero valued weight in the compressed weight operand based on the bitmap; and A determination is made based on a number of one or more non-zero elements in the bitmap whether there is a failure in identifying the non-zero-valued activation value or the non-zero-valued weight.

12. The one or more non-transitory computer-readable media of claim 11, wherein: The bitmap is generated based on a previous bitmap, another non-zero-valued activation value in the compressed activation operand or another non-zero-valued weight in the compressed weight operand is identified based on the previous bitmap, and determining whether a failure exists in identifying the non-zero-valued activation value or the non-zero-valued weight comprises: determining the number of one or more non-zero elements in the previous bitmap; and A determination is made as to whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

13. The one or more non-transitory computer-readable media of claim 12, wherein: Identifying the non-zero valued activation value in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap upon determining that there is a failure in identifying the non-zero-valued activation value or the non-zero-valued weight; and The non-zero-valued activation values ​​in the compressed activation operand or the non-zero-valued weights in the compressed weight operand are identified based on the new bitmap.

14. The one or more non-transitory computer-readable media of claim 13, wherein: Generating a new bitmap based on the previous bitmap includes: The non-zero elements in the previous bitmap are replaced with zeros.

15. One or more non-transitory computer-readable media according to any one of claims 11 to 14, wherein: Determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: A determination is made as to whether a number of one or more non-zero elements in the bitmap is greater than a number of one or more non-zero valued activation values ​​in the activation operand.

16. One or more non-transitory computer-readable media according to any one of claims 11 to 15, wherein: Determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: A determination is made as to whether the number of one or more non-zero elements in the bitmap is greater than the number of one or more non-zero-valued weights in the weight operand.

17. One or more non-transitory computer-readable media according to any one of claims 11 to 16, wherein: Identifying the non-zero valued activation value in the compressed activation operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero-valued activation value; determining a position of the non-zero activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero activation value in the compressed activation operand based on the second bitmap; and The non-zero valued activation value in the compressed activation operand is identified based on the position, the first position, and the second position.

18. One or more non-transitory computer-readable media according to any one of claims 11 to 17, wherein: Identifying the non-zero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero-valued weights; determining a position of the non-zero-valued weight in the compressed weight operand based on the bitmap; determining a first position of the non-zero-valued weight in the compressed weight operand based on the first bitmap; determining a second position of the non-zero-valued weight in the compressed weight operand based on the second bitmap; and The non-zero valued weight in the compressed weight operand is identified based on the position, the first position, and the second position.

19. One or more non-transitory computer-readable media according to any one of claims 11 to 18, wherein: Identifying the non-zero valued activation value in the compressed activation operand comprises: After determining that a failure exists in identifying the non-zero valued activation value, another non-zero valued activation value is identified in the compressed activation operand, wherein the another non-zero valued activation value immediately follows the previously identified non-zero valued activation value in the compressed activation operand.

20. The one or more non-transitory computer-readable media of any one of claims 11 to 19, wherein: Identifying the non-zero valued activation value in the compressed activation operand or the non-zero valued weight in the compressed weight operand comprises: determining a position of the non-zero valued activation value in the compressed activation operand; and determining the position of the non-zero valued weight in the compressed weight operand, Wherein, in the deep learning operation, the non-zero activation value is multiplied by the non-zero weight.

21. An apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: Storing compressed activation operands and compressed weight operands, wherein the compressed activation operands include one or more non-zero valued activation values ​​in activation operands in a deep learning operation and the compressed weight operands include one or more non-zero valued weights in weight operands in the deep learning operation; generating a bitmap based on an activation sparsity vector indicating one or more locations of the one or more non-zero valued activation values ​​in the activation operand and a weight sparsity vector indicating one or more locations of the one or more non-zero valued weights in the weight operand; identifying a non-zero valued activation value in the compressed activation operand or a non-zero valued weight in the compressed weight operand based on the bitmap; and A determination is made based on a number of one or more non-zero elements in the bitmap whether there is a failure in identifying the non-zero-valued activation value or the non-zero-valued weight.

22. The device according to claim 21, wherein The bitmap is generated based on a previous bitmap, another non-zero-valued activation value in the compressed activation operand or another non-zero-valued weight in the compressed weight operand is identified based on the previous bitmap, and determining whether a failure exists in identifying the non-zero-valued activation value or the non-zero-valued weight comprises: determining the number of one or more non-zero elements in the previous bitmap; and A determination is made as to whether the number of one or more non-zero elements in the bitmap is not equal to the sum of the number of one or more non-zero elements in the previous bitmap plus one.

23. The device according to claim 21 or 22, wherein Determining whether there is a failure in identifying the non-zero valued activation value or the non-zero valued weight includes: A determination is made as to whether a number of one or more non-zero elements in the bitmap is greater than a number of one or more non-zero valued activation values ​​in the activation operand.

24. The device according to any one of claims 21 to 23, wherein Identifying the non-zero valued activation value in the compressed activation operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector after determining that there is a failure in identifying the non-zero-valued activation value; determining a position of the non-zero activation value in the compressed activation operand based on the bitmap; determining a first position of the non-zero activation value in the compressed activation operand based on the first bitmap; determining a second position of the non-zero activation value in the compressed activation operand based on the second bitmap; and The non-zero valued activation value in the compressed activation operand is identified based on the position, the first position, and the second position.

25. The device according to any one of claims 21 to 24, wherein Identifying the non-zero valued activation value in the compressed activation operand or the non-zero valued weight in the compressed weight operand comprises: determining a position of the non-zero valued activation value in the compressed activation operand; and determining the position of the non-zero valued weight in the compressed weight operand, Wherein, in the deep learning operation, the non-zero activation value is multiplied by the non-zero weight.

Images