Scheduling computations based on sparsity in deep neural networks

Abstract

Calculations in processing elements (PEs) for executing the deep neural network are scheduled via a calculation scheduler based on sparsity of the calculated input data to reduce voltage droop. Each PE may calculate an input operand and a weight operand at the time of calculation. The compute scheduler may predict a workload for computation by the PE based on a combined sparsity bitmap, which may be generated based on a sparsity bitmap of the input operand and a sparsity bitmap of the weight operand. The computation scheduler may schedule a start time of the computation in the PE based on the predicted workload of the PE. The computation scheduler may instruct the PE with the highest workload to start computation first and instruct the other PEs to start computation later. In some embodiments, calculations in the PE may end at the same clock cycle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 180,415, filed on Mar. 8, 2023, entitled “SCHEDULING COMPUTATIONS IN DEEP NEURAL NETWORK BASED ON SPARSITY,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to neural networks and, more particularly, to scheduling computations based on sparsity in deep neural networks (DNNs). Background Art

[0004] Because DNNs can achieve high accuracy, they are widely used in a variety of AI applications, from computer vision to speech recognition and natural language processing. However, this high accuracy comes at a significant computational cost. DNNs are computationally demanding, as each inference can require hundreds of millions of MAC (multiply-add) operations and extensive data reading and writing. Therefore, technologies are needed to improve DNN efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Each embodiment will be readily understood by the following detailed description in conjunction with the accompanying drawings. For ease of description, like reference numerals denote like structural elements. In the figures of the accompanying drawings, the embodiments are illustrated by way of example and not limitation.

[0006] Figure 1 An example DNN according to various embodiments is shown.

[0007] Figure 2 Example convolutions according to various embodiments are shown.

[0008] Figure 3 is a block diagram of a DNN accelerator according to various embodiments.

[0009] Figure 4 is a block diagram of computational blocks according to various embodiments.

[0010] Figure 5 An array of processing elements (PEs) is shown in accordance with various embodiments.

[0011] Figure 6 is a block diagram of a PE according to various embodiments.

[0012] Figure 7 Sparsity acceleration of PE in MAC operations according to various embodiments is shown.

[0013] Figure 8 A computation schedule for a group of PEs is shown, where computations for these PEs start simultaneously, according to various embodiments.

[0014] Figure 9 A computation schedule for a group of PEs that can reduce voltage droop is shown according to various embodiments.

[0015] Figure 10 Another computation schedule for a group of PEs that can reduce voltage droop is shown according to various embodiments.

[0016] Figure 11 is a flowchart illustrating a method of scheduling computations in a DNN, according to various embodiments.

[0017] Figure 12 is a block diagram of an example computing device in accordance with various embodiments. DETAILED DESCRIPTION

[0018] Overview

[0019] The past decade has witnessed the rapid rise of artificial intelligence (AI)-based data processing technologies, particularly those based on deep neural networks (DNNs). DNNs have been widely used in computer vision, speech recognition, image, and video processing, primarily due to their ability to achieve superhuman accuracy. Significant improvements in DNN model size and accuracy, coupled with the rapid increase in the computational power of execution platforms, have enabled DNN applications to be used even in resource-constrained mobile and edge devices with limited energy supplies.

[0020] An accelerator for a DNN ("DNN accelerator") may include one or more large PE arrays that run concurrently while executing layers in the DNN. The synchronous start of computations (e.g., MAC operations) in the DNN may result in rapid and large activity transitions that may induce large current transients. Such large current transients may cause significant voltage drops, thereby degrading system performance. Voltage drops may also cause functional failures. The synchronous start of computations may occur during every computation cycle. Therefore, voltage drops are a common event in DNN accelerators.

[0021] One currently available design applies a voltage guard band by operating the supply voltage above the minimum voltage. Another currently available design applies a voltage guard band by operating below the maximum clock frequency (F CLK ) CLK To apply F CLK Guard bands are added to ensure proper functionality during voltage sag events. However, these additional guard bands may degrade system performance during operation in some common operating cycles.

[0022] One currently available solution for reducing voltage drops is to interleave the execution of functional units or individual arithmetic components to smooth transient current demands. Taking a DNN accelerator including PEs as an example, the calculations in the PEs can be activated simultaneously, which may result in large simultaneous current demands. This currently available solution can facilitate interleaved calculations in the PEs by delaying the calculations in the PEs by a predetermined time increment. For example, the calculation start time of an individual PE can be delayed by an increment of ΔT so that each PE can start its calculation at a different time. However, the interleaving of calculations is typically performed without considering the input data pattern or the instructions that trigger the calculations. One disadvantage of interleaved calculations is that the time to complete the calculations may be delayed, and therefore the throughput performance of the DNN accelerator may be adversely affected.

[0023] Other solutions for reducing voltage drops apply adaptive circuit techniques by using on-chip monitors to measure supply voltage variations and adjust the F CLK , to reduce the impact of voltage drop on system performance. These reactive technologies require response time to detect voltage drop and adjust F CLK , to avoid critical path timing margin failures. However, although these techniques have low overhead, they are not very effective in mitigating the effects of high-frequency voltage drops. Other adaptive techniques that can address response time are adaptive frequency systems. These adaptive frequency systems can directly modulate the phase-locked loop (PLL) clock output to follow the V DD Adapt to changes CLK . However, the analog circuits used for such adaptive frequency systems can be complex. To avoid complex analog circuits, digital adaptive clock distribution is used. Digital adaptive clock distribution can use an adjustable length delay between the PLL and the global clock distribution to utilize a temporary clock for data path compensation during voltage drops. This can provide acceptable response times without affecting system performance, during which the clock frequency can be adaptively reduced. However, the performance of this system is not uniform across all frequency points, and since many DNN accelerators operate over a very large frequency range and often dynamically scale voltage and frequency (DVFS), it does not perform well across the entire operating range. Some other adaptive designs will V DD and F CLK into a single control loop. Although such a control loop can enable unlimited clock-data compensation, it is difficult to develop a practical and efficient V DD There are challenges with the regulator.

[0024] Another solution for mitigating voltage sags is based on recovery techniques. Resilient timing error detection and recovery circuitry is used to relax response time constraints by detecting timing margin violations caused by voltage sags, isolating the errors from corrupted architectural state, and correcting the errors through recovery techniques. Because architectural state is preserved, error correction can occur within multiple clock cycles. While this technique is effective at high frequencies, the design complexity of implementing error recovery while ensuring coverage of all fault scenarios is a significant obstacle.

[0025] Embodiments of the present disclosure can improve at least some of the challenges and problems described above by reducing voltage drops in DNN accelerators based on the sparsity of input data calculated in the DNN. The input data of a DNN layer (e.g., a convolutional layer, etc.) may include weights and activation values. The activation values or weights of a DNN layer may be arranged in a tensor. A tensor is a data structure with multiple elements in one or more dimensions. Example tensors include vectors (which are one-dimensional tensors) and matrices (which are two-dimensional tensors). There may also be three-dimensional tensors or even higher-dimensional tensors. Weights can be determined by training the DNN. The activation value may be a data element in the input to the DNN (e.g., in an embodiment where the DNN layer is the first layer of the DNN), or it may be a data element generated in a previous layer of the DNN.

[0026] DNN layers may have a large number of zero-valued weights (i.e., weights with a value of zero), which may be generated during the training phase. Zero-valued weights do not contribute to the partial sum accumulation in the MAC operation of the convolution. Sparse weights may cause activation values to become sparse in subsequent layers of the DNN. Network quantization running inference on edge devices may also cause a large number of zeros to appear in weights and activation values. In addition, nonlinear activation functions such as the rectified linear activation function (ReLU) can truncate negative activation values to zero and are prevalent in DNNs. DNN accelerators can significantly speed up computations by skipping zeros in the MAC operation of the convolution. Various embodiments of the present disclosure may further exploit the sparsity present in weights and activation values to reduce voltage drops in DNN accelerators.

[0027] In some embodiments, a scheduler is associated with a group of PEs in a DNN accelerator. The scheduler can schedule the start time of calculations for these PEs based on the sparsity of the activation values and weights to be calculated by the group of PEs. The group of PEs can be an entire PE array, a plurality of PE arrays, a column of PEs in an array, a portion of a column of PEs, and so on. For example, a PE is to perform a MAC operation on an activation operand and a weight operand. The activation operand can include a sequence of activation values, each of which can be a data element in an input tensor (e.g., an input feature map (IFM)) of a DNN layer. The weight operand can include a sequence of weights, each of which can be a data element in a filter of a DNN layer. The activation operand is associated with an activation sparsity bitmap, which includes a sequence of bits. Each bit corresponds to a corresponding activation value in the activation operand and indicates whether the activation value is zero or non-zero. The weight operand is associated with a weight sparsity bitmap, which includes a sequence of bits. Each bit in the weight sparsity bitmap corresponds to a corresponding weight in the weight operand and indicates whether the weight value is zero or non-zero. A combined sparsity bitmap can be generated based on the activation sparsity bitmap and the weight sparsity bitmap. The combined sparsity bitmap includes a sequence of bits, each bit corresponding to a corresponding activation value and weight. For a bit corresponding to a non-zero activation and a non-zero weight, the value of the bit can be one. For a bit corresponding to a zero activation or a zero weight, the value of the bit can be zero.

[0028] The PE can skip the calculation of zero-valued activation values and zero-valued weights based on the combined sparsity bitmap. For example, non-zero-valued activation values and non-zero-valued weights can be identified based on the combined sparsity bitmap and loaded into the PE for calculation. The scheduler can predict the workload of the PE based on the number of non-zero bits in the combined sparsity bitmap. The scheduler can determine the calculation start time of the PE based on the predicted workload so that PEs with different workloads can start calculations at different times, which can avoid large current transients in the DNN accelerator. In addition, since the scheduler knows the workload of the PE, the scheduler can determine the calculation start time of the PE in the following manner: by ensuring that no calculation will end later than the calculation of the PE with the largest workload to avoid sacrificing the overall performance of the DNN accelerator. In some embodiments, the scheduler can cause the calculations of the PEs to end at the same time. In other embodiments, the calculations of the PEs may end at different times.

[0029] The scheduler may determine a workload score for each PE. The workload score of the PE may be equal to the number of non-zero bits in the combined sparsity bitmap of the PE. A down counter may count down in fixed increments (e.g., one) from the highest workload score to a lower value (e.g., the lowest workload score or zero) over a sequence of clock cycles. The down counter has a different value in each corresponding clock cycle. The scheduler may instruct the PE to begin its calculation in a cycle following (e.g., the cycle immediately following) a cycle in which the value in the down counter matches the workload score of the PE.

[0030] Since the scheduler can schedule the calculations in the PEs to start at different times, large current transients and voltage drops in the DNN accelerator can be reduced. The scheduler can also ensure that other PEs complete the calculation no later than the PE with the highest workload completes the calculation, thereby maintaining the benefits of sparsity acceleration in the DNN accelerator. The scheduler can be extended to PEs of one or more columns in the PE array. In addition, the scheduler can be extended to multiple PE arrays. Each scheduler can operate independently on its assigned PE and does not need to communicate with each other. In addition, the compiler that determines how to execute the DNN layer in the DNN accelerator (or across multiple DNN accelerators) does not need to rearrange the activation values or weights in the memory. Compared with currently available technologies, the present disclosure provides more advantageous technologies to reduce voltage drops in DNN accelerators and improve efficiency.

[0031] For illustrative purposes, specific numbers, materials, and configurations are set forth to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details, or / and may be practiced with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative embodiments.

[0032] In addition, reference is made to the accompanying drawings which form a part thereof, and in which are shown by way of illustration embodiments that may be practiced. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be construed in a limiting sense.

[0033] The various operations may be described as multiple discrete actions or operations in a manner that best facilitates understanding of the claimed subject matter. However, the order of description should not be interpreted as implying that the operations are necessarily order-dependent. In particular, the operations may not be performed in the order presented. The described operations may be performed in an order different from that of the described embodiments. Various additional operations may be performed, or described operations may be omitted in additional embodiments.

[0034] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0035] This description uses the phrases "in one embodiment" or "in an embodiment," each of which may refer to one or more of the same or different embodiments. The terms "including," "comprising," "having," and the like as used with respect to the embodiments of the present disclosure are synonymous. The present disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are merely for ease of discussion and do not imply desired or required directions. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects merely indicates different instances of the similar objects referred to, and is not intended to imply that the objects so described must be arranged in a given order, whether in time, space, ranking, or otherwise.

[0036] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0037] The terms "substantially," "close," "approximately," "near," and "about" generally refer to input operands being within + / - 20% of target values based on specific values described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to input operands being within + / - 5-20% of target values based on specific values described herein or known in the art.

[0038] Furthermore, the terms "comprises," "includes," "has," or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Furthermore, the term "or" refers to an inclusive or rather than an exclusive or.

[0039] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0040] Example DNN

[0041] Figure 1 An example DNN 100 is shown in accordance with various embodiments. For illustration purposes, Figure 1 The DNN 100 in FIG is a CNN. In other embodiments, the DNN 100 may be another type of DNN. The DNN 100 is trained to receive an image and output a classification of an object in the image. Figure 1 In one embodiment, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a sequence of layers including a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully connected layers 130 (individually referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. During inference of DNN 100, the layers of the DNN perform tensor computations including a number of tensor operations, such as convolutions (e.g., multiply-add (MAC) operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination of these operations.

[0042] The convolution layer 110 summarizes the presence of features in the input image 105. The convolution layer 110 acts as a feature extractor. The first layer of the DNN 100 is the convolution layer 110. In the example, the convolution layer 110 performs a convolution operation on the input tensor 140 (also referred to as IFM 140) and the filter 150. Figure 1 As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, and each channel is represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also called input points) per row and 7 input elements per column. The filter 150 is represented by a 3×3×3 3D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. The kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. The kernel may be smaller than the IFM. Figure 1 In the embodiment, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel includes 3 weights per row and 3 weights per column. The weights can be initialized and updated using gradient descent via backpropagation. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0043] Convolution involves a MAC operation of the input elements in the IFM 140 and the weights in the filter 150. The convolution can be a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides over the IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also called output points) per row and 5 output elements per column. For illustration, in Figure 1 In the embodiment of FIG, the standard convolution includes one filter. In the embodiment where multiple filters are present, the standard convolution can produce multiple output channels in OFM 160.

[0044] The multiplication performed between the kernel-sized local patch of IFM 140 and the kernel can be a dot product. A dot product is an element-by-element multiplication between the kernel-sized local patch of IFM 140 and the corresponding kernel, followed by addition, always producing a single value. Because it produces a single value, this operation is often referred to as a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times with IFM 140 at different points on IFM 140. Specifically, the kernel is systematically applied from left to right and from top to bottom to each overlapping portion or kernel-sized local patch of IFM 140. The result of multiplying the kernel once with IFM 140 is a single value. Since the kernel is applied multiple times to IFM 140, the multiplication result is an output element of a 2D matrix. Therefore, the 2D output matrix from standard convolution 163 (i.e., OFM 160) is referred to as OFM.

[0045] In depthwise convolution 183, the input channels are not combined. Instead, MAC operations are performed on individual input channels and individual kernels, and output channels are generated. Figure 1As shown, depthwise convolution 183 produces a depthwise output tensor 180. Depthwise output tensor 180 is represented by a 5×5×3 3D matrix. Depthwise output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains five output elements, and each column also contains five output elements. Each output channel is the result of a MAC operation between the input channels of IFM 140 and the kernel of filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation between the first input channel (dot pattern) and the first kernel (dot pattern), the second output channel (horizontal stripe pattern) is the result of a MAC operation between the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern), and the third output channel (diagonal stripe pattern) is the result of a MAC operation between the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, pointwise convolution 193 is performed on the depthwise output tensor 180 and the 1×1×3 tensor 190 to produce the OFM 160 .

[0046] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is a ReLU. A ReLU is a calculation that directly returns the value provided as input, or returns a value of 0 if the input is 0 or less. The convolution layer 110 can receive several images as input and calculate the convolution of each of them with each kernel. This process can be repeated several times. For example, the OFM 160 is passed to a subsequent convolution layer 110 (i.e., the convolution layer 110 after the convolution layer 110 in the sequence that generated the OFM 160). The subsequent convolution layer 110 performs convolution on the OFM 160 using the new kernel and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be kernelized again by a further subsequent convolution layer 110, and so on.

[0047] In some embodiments, convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size (e.g., the kernel size is F×F×D pixels), the stride S for dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thick to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolutions, such as two-dimensional convolution, dilated or atrous convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. DNN 100 includes 16 convolutional layers 110. In other embodiments, DNN 100 may include a different number of convolutional layers.

[0048] The pooling layer 120 downsamples the feature map generated by the convolution layer, for example, by summarizing the presence of features in local patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110: the front convolutional layer 110 (the convolutional layer 110 located before the pooling layer 120 in the layer sequence) and the back convolutional layer 110 (the convolutional layer 110 located after the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after the activation function (e.g., ReLU) has been applied to the OFM 160.

[0049] The pooling layer 120 receives the feature maps generated by the preceding convolutional layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while retaining their important characteristics. Therefore, the pooling operation improves the efficiency of the DNN and prevents overlearning. The pooling layer 120 can perform the pooling operation through average pooling (calculating the average value of each local block on the feature map), maximum pooling (calculating the maximum value of each local block on the feature map), or a combination of the two. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied to 2×2 pixels with a step size of 2 pixels, thereby reducing the size of the feature map by a factor of 2, for example, reducing the number of pixels or values in the feature map to one-quarter of its original size. In one example, the pooling layer 120 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates on each feature map separately to create a new set of pooled feature maps with the same number of pooled feature maps.

[0050] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be convolutional. The fully connected layer 130 receives input operands. The input operands define the outputs of the convolutional layer 110 and the pooling layer 120 and include the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operands and generates a vector. This vector can contain as many elements as there are categories: element i represents the probability that the image belongs to category i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the final fully connected layer 130 using a logistic function (binary classification) or a softmax function (multi-class classification) as the activation function.

[0051] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1In the embodiment of , N is equal to 3 because there are three objects 115, 125 and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probability, the fully connected layer 130 multiplies each input element by a weight, sums the values, and then applies an activation function (e.g., logistic if N=2, softmax if N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In one example, the vector includes three probabilities: a first probability indicating that object 115 is a tree, a second probability indicating that object 125 is a car, and a third probability indicating that object 135 is a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual values may be different.

[0052] Example Convolution

[0053] Figure 2 An example convolution according to various embodiments is shown. The convolution can be a convolutional layer in a DNN (e.g., Figure 1 The convolution layer 110 in FIG. 2 may be a front-end layer. The convolution may be performed on an input tensor 210 and a filter 220 (individually referred to as “filter 220”). The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator comprising one or more computational blocks. An example of a DNN accelerator may be Figure 3 The DNN accelerator 300 in

[0054] can be an example of a computational block. Figure 3 Calculation block 325 in .

[0054] exist Figure 2 In the embodiment of FIG, the input tensor 210 includes activation values (also referred to as “input activation values,” “elements,” or “input elements”) arranged in a 3D matrix. The activation values in the input tensor 210 are data points in the input tensor 210. The input tensor 210 has a spatial size H in ×W in ×C in , where H in is the height of the 3D matrix (i.e., the length along the Y axis, representing the number of activation values in the columns of the 2D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of activation values in a row of the 2D matrix for each input channel), C inis the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of input channels). For simplicity and illustration, the input tensor 210 has a spatial size of 7×7×3, i.e., the input tensor 210 includes three input channels, each of which has a 7×7 2D matrix. Each input element in the input tensor 210 can be represented by an (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the input tensor 210 can be different.

[0055] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights can be determined by training the DNN. The filter 220 has a spatial size H f ×W f ×C f , where H f is the height of the filter (i.e., the length along the Y-axis, representing the number of weights in the column for each kernel), W f is the width of the filter (i.e., the length along the X-axis, representing the number of weights in the row for each kernel), C f is the depth of the filter (i.e., the length along the Z axis, representing the number of channels). In some embodiments, C f Equal to C in For simplicity and illustration, Figure 2 Each filter 220 in has a spatial size of 3×3×3, that is, filter 220 includes three convolution kernels with a spatial size of 3×3. In other embodiments, the height, width, or depth of filter 220 can be different. The spatial size of the convolution kernel is smaller than the spatial size of the 2D matrix of each input channel in input tensor 210.

[0056] The activation value or weight may occupy one or more bytes in memory. The number of bytes of the activation value or weight may depend on the data format. For example, when the activation value or weight has an integer format (e.g., INT8), the activation value occupies one byte. When the activation value or weight has a floating point format (e.g., FP16 or BF16), the activation value or weight occupies two bytes. Other data formats may be used for activation values or weights.

[0057] In convolution, each filter 220 slides over the input tensor 210 and produces a 2D matrix for the output channels in the output tensor 230. Figure 2 In the embodiment of FIG, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activation values (also referred to as “output activation values,” “elements,” or “output elements”) arranged in a 3D matrix. The activation values in the output tensor 230 are the data points in the output tensor 230. The output tensor 230 has a spatial size H out ×W out ×C out , where H outis the height of the 3D matrix (i.e., the length along the Y axis, representing the number of output activation values in the column of the 2D matrix for each output channel), W out is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of output activation values in the row of the 2D matrix for each output channel), C out is the depth of the 3D matrix (i.e., the length along the Z axis, representing the number of output channels). out It can be equal to the number of filters 220 in the convolution. out and W out It may depend on the height and width of the input tensor 210 and each filter 220 .

[0058] As part of the convolution, a 3×3×3 sub-tensor 215 of the input tensor 210 (in Figure 2 MAC operations are performed on the sub-tensor 215 and each filter 220 (highlighted with a dot pattern in FIG). The result of the MAC operation on the sub-tensor 215 and one filter 220 is an output activation value. In some embodiments (e.g., embodiments in which the convolution is an integer convolution), the output activation value may include 8 bits, such as one byte. In other embodiments (e.g., embodiments in which the convolution is a floating-point convolution), the output activation value may include more than one byte. For example, an output element may include two bytes.

[0059] After completing the MAC operation on the sub-tensor 215 and all filters 220, a vector 235 is generated. The vector 235 is Figure 2 . Vector 235 includes a sequence of output activation values arranged along the Z-axis. The output activation values in vector 235 have the same (x, y) coordinates, but these output activation values correspond to different output channels and have different Z-coordinates. The dimension of vector 235 along the Z-axis can be equal to the total number of output channels in output tensor 230.

[0060] After vector 235 is generated, further MAC operations are performed to generate additional vectors until output tensor 230 is generated. For example, filter 220 can be moved along the X-axis or Y-axis on input tensor 210, and a MAC operation can be performed on filter 220 and another sub-tensor in input tensor 210 (the sub-tensor has the same size as filter 220). The amount by which filter 220 moves on input tensor 210 in different computational rounds is called the stride size of the convolution. The stride size can be 1 (i.e., the amount by which filter 220 moves is one activation value), 2 (i.e., the amount by which filter 220 moves is two activation values), and so on. The height and width of output tensor 230 can be determined based on the stride size.

[0061] In some embodiments, a plurality of PEs (e.g., Figure 5PE 510, Figure 6 PE 600, or Figure 7 PE 700 in ) performs MAC operations on a 3×3×3 sub-tensor (e.g., sub-tensor 215) and filter 220. One or more MAC units may receive input operands (e.g., Figure 2 ) and weight operands (e.g., Figure 2 227). The input operand 217 includes a sequence of activation values having the same (Y, Z) coordinates but different X coordinates. The weight operand 227 includes a sequence of weights having the same (Y, Z) coordinates but different X coordinates. The length of the input operand 217 is the same as the length of the weight operand 227. The activation values in the input operand 217 and the weights in the weight operand 227 can be input to the PE in sequence. The PE can receive a pair of activation values and weights at a time and multiply the activation value by the weight. The position of the activation values in the input operand 217 can match the position of the weights in the weight operand 227.

[0062] Example DNN Accelerator

[0063] Figure 3 is a block diagram of a DNN accelerator 300 according to various embodiments. The DNN accelerator 300 can run a DNN, such as Figure 1 100 in the DNN 100. The DNN accelerator 300 includes a memory 310, a DMA (direct memory access) engine 320, and a computation block 330. In other embodiments, the DNN accelerator 300 may include alternative configurations, different, or additional components. For example, the DNN accelerator 300 may include multiple memories 310 or multiple DMA engines 320. For another example, the DNN accelerator 300 may include a single computation block 330. In addition, the functions implemented by the components of the DNN accelerator 300 may be implemented by other components or other systems included in the DNN accelerator 300.

[0064] Memory 310 stores data that the computation block 330 uses to perform deep learning operations in the DNN model. Example deep learning operations include convolution (also known as "convolution operations"), pooling operations, element-by-element operations, other types of deep learning operations, or combinations of these operations. Memory 310 can be the main memory of the DNN accelerator 300. In some embodiments, memory 310 includes one or more DRAMs (dynamic random access memories). For example, memory 310 can store input tensors, convolution kernels, or output tensors of a convolution in a convolution layer (e.g., convolution layer 30) of a DNN. The output tensor can be transferred from the local memory of the computation block 330 to the memory 310 via the DMA engine 320.

[0065] The DMA engine 320 facilitates data transfers between the memory 310 and the local memory of the compute block 330. For example, the DMA engine 320 can read data from the memory 310 and write data to the local memory of the compute block 330. For another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to the memory 310. The DMA engine 320 provides DMA functionality, allowing the compute block 330 to initiate data transfers between the memory 310 and the local memory of the compute block 330 and perform other operations while the data transfer is in progress. In some embodiments, the DMA engine 320 can read tensors from the memory 310 and modify the tensors in a manner optimized for the compute block 330 before writing the tensors to the local memory of the compute block 330.

[0066] The computation block 330 performs computations for deep learning operations. The computation block 330 may run operations in a DNN layer, or a portion of operations in a DNN layer. The computation block 330 may perform convolutions, such as standard convolutions (e.g., Figure 1 Standard convolution in 163), depthwise convolution (e.g., Figure 1 ), and point-wise convolution (e.g., Figure 1 193 in the point-by-point convolution 193) and so on. In some embodiments, the computation block 330 receives an input tensor and one or more convolution kernels and performs convolution on the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further calculated, for example, by the computation block 330 or another computation block. In some embodiments, the operation of the DNN layer can be run in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 can each perform a portion of the convolution workload. Data can be shared between the computation blocks 330.

[0067] Figure 4 is a block diagram of a computation block 400 according to various embodiments. The computation block 400 may be Figure 3 An example of the calculation block 330 in FIG. Figure 4 As shown, computation block 400 includes local memory 410, PE array 420, sparsity accelerator 430, and computation scheduler 440. In other embodiments, computation block 400 may include alternative configurations, different, or additional components. For example, computation block 400 may include multiple local memories 410, PE arrays 420, sparsity accelerators 430, or computation schedulers 440. In addition, the functions implemented by the components of computation block 400 may be implemented by other components included in computation block 400, another component in DNN accelerator 300, or other systems.

[0068] Local memory 410 is local to the computation block 400. Figure 4In an embodiment of the present invention, the local memory 410 is located within the computing block 400. In other embodiments, the local memory 410 may be located outside the computing block 400. The local memory 410 and the computing block 400 may be implemented on the same chip. The local memory 410 stores data used for convolution or data generated from convolution, such as input activation values, weights, and output activation values. In some embodiments, the local memory 410 includes one or more SRAMs (static random access memories). The local memory 410 may be byte-addressable, with each memory address identifying a single byte (eight bits) of storage space. In some embodiments, the local memory 410 may include memory banks, each of which may have a capacity of a fixed number of bytes (e.g., 32, 64, etc.).

[0069] PE array 420 performs MAC operations in convolution. PE array 420 may also perform other deep learning operations. PE array 420 may include PEs arranged in columns or in rows and columns. Each PE may perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more adders for performing accumulation. A column of PEs is referred to as a PE column. A PE column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into a MAC column. A MAC lane may also be referred to as a data transmission lane or a data loading lane. A PE column may have multiple MAC lanes. The loading bandwidth of a MAC column is the sum of the loading bandwidths of all MAC lanes associated with the MAC column. Using a specific number of MAC lanes, data can be simultaneously input into the same number of independent PEs. In some embodiments, a MAC column has four MAC lanes for inputting activation values or weights into the MAC column. Each MAC lane may have a bandwidth of 16 bytes, and the four MAC lanes may have a total loading bandwidth of 64 bytes.

[0070] In some embodiments, the PE array 420 may be capable of performing standard convolution, depthwise convolution, pointwise convolution, other types of convolution, or a combination of these. In depthwise convolution, the PE may perform a MAC operation that includes a sequence of multiplications of an input operand (e.g., input operand 217) and a weight operand (e.g., weight operand 227). Each multiplication in the sequence is a multiplication of a different activation value in the input operand and a different weight in the weight operand. The activation value and weight in the same cycle may correspond to the same channel. The multiplication sequence generates a product operand that includes a product sequence. The MAC operation may also include accumulation, wherein multiple product operands are accumulated to generate an output operand of the PE. The PE array 420 may output multiple output operands simultaneously, each output operand generated by a different PE. In standard convolution, the MAC operation may include accumulation across channels. For example, instead of generating output operands, the PE may accumulate products across different channels to generate a single output point.

[0071] In some embodiments, a PE can perform multiple rounds of MAC operations for convolution. Data (activations, weights, or both) can be reused within a single round (e.g., between different multipliers in a PE) or between different rounds of MAC operations. More details about the PE array will be provided below in conjunction with Figure 5 and Figure 6 Provide a description.

[0072] The sparsity accelerator 430 accelerates computations in the PE array 420 based on the sparsity of the input data being computed. Figure 4 A single sparsity accelerator 430 is shown, but the compute block 400 may include multiple sparsity accelerators 430. In some embodiments, each PE in the PE array 420 implements a sparsity accelerator 430 to accelerate computations in that individual PE. In other embodiments, a subset of the PE array 420 (e.g., one or more PE columns in the PE array 420) may implement a sparsity accelerator 430 to accelerate computations in the subset of PEs.

[0073] In some embodiments (e.g., embodiments in which computation block 400 performs a convolutional layer), the computation in the PE may be a MAC operation on an input operand and a weight operand. The input operand may be a portion of an input tensor for the convolution. The input operand includes a sequence of input elements (i.e., activation values). The activation values may come from different input channels. For example, each activation value comes from an input channel that is different from all other activation values in the input operand. The input operand is associated with an input bitmap, which may be stored in local memory 410. The input bitmap may indicate the locations of non-zero activation values in the input operand. The input bitmap may include a sequence of bits, with each bit corresponding to a corresponding activation value in the input operand. The positions of the bits in the input bitmap may match the positions of the corresponding activation values in the input operand. The bits in the input bitmap may be either zero or one. A zero-valued bit indicates that the corresponding activation value is zero, and a one-valued bit indicates that the corresponding activation value is non-zero. In some embodiments, the input bitmap may be generated during the execution of another DNN layer (e.g., a layer preceding the convolutional layer in the DNN).

[0074] The weight operand may be part of the kernel of a convolution. The weight operand includes a sequence of weights. The values of these weights are determined by training the DNN. The weights in the weight operand may come from different input channels. For example, each weight comes from an input channel different from all other weights in the weight operand. The weight operand is associated with a weight bitmap, which may be stored in the local memory 410. The weight bitmap may indicate the location of non-zero valued weights in the weight operand. The weight bitmap may include a sequence of bits, each bit corresponding to a corresponding weight in the weight operand. The positions of the bits in the weight bitmap may match the positions of the corresponding weights in the weight operand. The bits in the weight bitmap may be zero or one. A zero-valued bit indicates that the value of the corresponding weight is zero, and a one-valued bit indicates that the value of the corresponding weight is non-zero.

[0075] The sparsity accelerator 430 can receive an input bitmap and a weight bitmap and generate a combined sparsity bitmap for the MAC operation to be performed by the PE. In some embodiments, the sparsity accelerator 430 generates a combined sparsity bitmap 735 by performing one or more AND operations on the input bitmap and the weight bitmap. Each bit in the combined sparsity bitmap is the result of performing an AND operation on a bit in the input bitmap and a bit in the weight bitmap, that is, the product of a bit in the input bitmap and a bit in the weight bitmap. The position of the bit in the combined sparsity bitmap matches the position of the bit in the input bitmap and the position of the bit in the weight bitmap. The bits in the combined bitmap correspond to a pair of activation values and weights (activation value-weight pairs). A zero-valued bit in the combined sparsity bitmap indicates that at least one of the activation value and weight in the pair is zero. A one-valued bit in the combined sparsity bitmap indicates that both the activation value and weight in the pair are non-zero. The combined sparsity bitmap can be stored in the local memory 410.

[0076] The sparsity accelerator 430 can provide activation values and weights to the PE based on the combined sparsity bitmap. For example, the sparsity accelerator 430 can identify activation values and weights corresponding to bits in the combined sparsity bitmap and forward these activation values and weights to the PE. The sparsity accelerator 430 can skip other activation values and other weights because they do not contribute to the result of the MAC operation. In some embodiments, the local memory 310 can store non-zero activation values and weights instead of zero activation values or weights. The non-zero activation values and weights can be loaded into one or more register files of the PE, from which the sparsity accelerator 430 can retrieve the activation values and weights corresponding to bits in the combined sparsity bitmap. In some embodiments, the total number of bits in the combined sparsity bitmap is equal to the total number of activation value-weight pairs to be calculated by the PE, while the PE does not calculate other activation value-weight pairs. By skipping activation value-weight pairs corresponding to zero bits in the combined sparsity bitmap, the PE's calculations are faster than if the PE calculated all activation value-weight pairs in the input operands and weight operands.

[0077] The computation scheduler 440 schedules computations for some or all of the PEs in the PE array 420 based on the sparsity of the data to be computed by the PEs. These PEs may be part of a column in the PE array 420, or these PEs may constitute one or more columns or even the entire PE array. In some embodiments, the computation scheduler 440 may be associated with one or more other PE arrays and may schedule computations in multiple PE arrays. Figure 4 As shown, the compute scheduler 440 includes a workload module 450, a decrement counter 460, and a PE initiator 470. In other embodiments, the compute scheduler 440 may include alternative configurations, different or additional components. In addition, the functions implemented by the components of the compute scheduler 440 may be performed by other components included in the compute scheduler 440, another component in the compute block 400 or the DNN accelerator 300, or other systems.

[0078] The workload module 450 predicts the workload of the PE based on the combined sparsity bitmap of the PE (e.g., the combined sparsity bitmap generated by the sparsity accelerator 430). For each PE, the workload module 450 can determine a workload score that indicates the amount of computation to be performed by the PE. In some embodiments, the workload score can be equal to the number of ones in the combined sparsity bitmap of the PE. The workload score can also indicate the amount of time required for the PE to perform the computation, such as the time from the start of the computation to the end of the computation. In some embodiments, the workload module 450 can also rank the workload scores of the PEs. The workload module 450 can identify the highest workload score among some or all of the workload scores. The workload module 450 can also identify the lowest workload score among some or all of the workload scores.

[0079] The down counter 460 decrements from a higher value to a lower value over a sequence of clock cycles. In some embodiments, the down counter 460 may decrement from the highest workload score to the lowest workload score. In other embodiments, the down counter 460 may decrement from the highest workload score to zero. The down counter 460 may count a single value in a single clock cycle. The next value in the next clock cycle may be equal to that value minus one. In one example, the highest workload score is N (N may be an integer), and the down counter 460 counts N in the first clock cycle, N-1 in the second clock cycle, N-2 in the third clock cycle, and so on. This may continue until the down counter 460 reaches the lowest workload score or zero. For any PE whose workload score is lower than the highest workload score, the down counter 460 may reach the workload score of that PE one clock cycle after the first clock cycle.

[0080] PE initiator 470 instructs a PE to begin computation based on the value counted by down-counter 460. In some embodiments, PE initiator 470 instructs the PE(s) with the highest workload score to begin computation before all other PEs. PE initiator 470 may determine whether the value counted by down-counter 460 in a clock cycle matches any workload score determined by workload module 450. In response to determining that the value counted by down-counter 460 matches the workload score, PE initiator 470 may instruct the PE to begin computation in the next clock cycle. In response to determining that the value counted by down-counter 460 does not match any workload score, PE initiator 470 takes no further action. After the value of down-counter 460 is changed in the next clock cycle, PE initiator 470 may determine whether the new / lower value matches any workload score. The start time of computation in a PE depends on the workload of the PE, i.e., the number of ones in the combined sparsity bitmap of the PE. In some embodiments, computation in the PEs may end simultaneously. Since the PE with the highest workload starts computing first, the total time to complete all computations in the PEs can be equal to the time required to complete computations in the PE with the highest workload, which avoids the risk of compromising the performance and efficiency of the computing block 400 or the DNN accelerator 300.

[0081] Figure 5 PE array 500 according to various embodiments is shown. PE array 500 may be Figure 4 5. PE array 500 includes multiple PEs 510 (individually referred to as "PE 510"). PE 510 performs MAC operations. PE 510 may also be referred to as a neuron in a DNN. Each PE 510 has two input signals 550 and 560 and an output signal 570. Input signal 550 is at least a portion of an IFM of a layer. Input signal 560 is at least a portion of a filter of a layer. In some embodiments, input signal 550 of PE 510 may include one or more input operands, and input signal 560 may include one or more weight operands.

[0082] Each PE 510 performs a MAC operation on input signals 550 and 560 and outputs an output signal 570 as a result of the MAC operation. Some or all of the input signals 550 and 560 and the output signal 570 may be in integer format (e.g., INT8) or floating-point format (e.g., FP16 or BF16). For simplicity and illustration, the input signals and output signals of all PEs 510 use the same reference numerals, but PEs 510 may receive different input signals and output different output signals. Furthermore, a PE 510 may differ from another PE 510, for example, including more, fewer, or different components.

[0083] like Figure 5 As shown, PEs 510 are interconnected, as shown in FIG. Figure 5 The output signal 570 of PE 510 can be sent as an input signal to many other PEs 510 (and possibly back to itself) through the interconnection between PEs 510. In some embodiments, the output signal 570 of PE 510 can be combined with the output signals of one or more other PEs 510 through the accumulation operation of PE 510 to generate an internal partial sum of the PE array. More details about PE 510 will be discussed below in conjunction with Figure 5 Provide a description.

[0084] exist Figure 5 In an embodiment, the PEs 510 are arranged into columns 505 (individually referred to as "columns 505"). The inputs and weights of the layers can be distributed to the PEs 510 based on the columns 505. Each column 505 has a column buffer 520. The column buffer 520 stores data provided to the PEs 510 in the column 505 for a short period of time. The column buffer 520 can also store data output by the last PE 510 in the column 505. The output of the last PE 510 can be the sum of the MAC operations of all the PEs 510 in the column 505, which is the column-level internal partial sum of the PE array 500. In other embodiments, the inputs and weights can be distributed to the PEs 510 based on the rows in the PE array 500. The PE array 500 can include a row buffer instead of the column buffer 520. The row buffer can store the input signals of the PEs in the corresponding row and can also store the row-level internal partial sum of the PE array 500.

[0085] like Figure 5 As shown, each column buffer 520 is associated with a loader 530 and a drain 540. Data provided to a column 505 is transferred to the column buffer 520 by the loader 530, for example, via an upper memory hierarchy, e.g. Figure 4The data generated by the column 505 is extracted from the column buffer 520 by the drainer 540. In some embodiments, the data extracted from the column buffer 520 is sent to the upper memory hierarchy through the drain operation, e.g. Figure 4 In some embodiments, the drain operation does not begin until all PEs 510 in column 505 have completed their MAC operations. In some embodiments, the loader 530 or the drainer 540 may be controlled by the control module 340. Although Figure 5 Not shown, but one or more columns 505 may be associated with external adder components.

[0086] Figure 6 is a block diagram of a PE 600 according to various embodiments. The PE 600 may be Figure 5 6. PE 600 includes an input register file 610 (individually referred to as "input register file 610"), a weight register file 620 (individually referred to as "weight register file 620"), a multiplier 630 (individually referred to as "multiplier 630"), an internal adder component 640, and an output register file 650. In other embodiments, PE 600 may include fewer, more, or different components. For example, PE 600 may include multiple output register files 650. For another example, PE 600 may include a single input register file 610, weight register file 620, or multiplier 630. For another example, PE 600 may include an adder instead of internal adder component 640.

[0087] The input register file 610 temporarily stores input operands for MAC operations performed by the PE 600. In some embodiments, the input register file 610 can store a single input operand at a time. In other embodiments, the input register file 610 can store multiple input operands or a portion of an input operand at a time. The input operands include multiple input elements (i.e., input elements) in the input tensor. The input elements in the input operands can be stored sequentially in the input register file 610 so that the input elements can be processed sequentially. In some embodiments, each input element in the input operands can come from a different input channel of the input tensor. The input operands can include input elements for each input channel of the input tensor, and the number of input elements in the input operands can be equal to the number of input channels. The input elements in the input operands can have the same XY coordinates, which can be used as the XY coordinates of the input operands. For example, all input elements of the input operands can be X0Y0, X0Y1, X1Y1, etc.

[0088] The weight register file 620 temporarily stores weight operands for MAC operations performed by the PE 600. Weight operands include weights in filters of the DNN layer. In some embodiments, the weight register file 620 can store a single weight operand at a time. In other embodiments, the weight register file 620 can store multiple weight operands or a portion of a weight operand at a time. The weight operand may include multiple weights. The weights in the weight operands may be stored sequentially in the weight register file 620 so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and input operands, each weight in the weight operand may correspond to an input element in the input operand. The number of weights in the weight operand may be equal to the number of input elements in the input operand.

[0089] In some embodiments, the weight register file 620 can be the same as or similar to the input register file 610, for example, having the same size. PE 600 may include multiple register files, some of which are designated as input register files 610 for storing input operands, some as weight register files 620 for storing weight operands, and some as output register files 650 for storing output operands. In other embodiments, register files in PE 600 can be designated for other purposes, such as storing scale operands used in element-by-element addition operations. The designation of register files can be controlled by control module 340.

[0090] Multiplier 630 performs multiplication operation on input operand and weight operand. Multiplier 630 can perform multiplication operation sequence on single input operand and single weight operand, and generate product operand including product sequence. Each multiplication operation in the sequence includes multiplying the input element in the input operand with the weight in the weight operand. In some embodiments, the position (or index) of the input element in the input operand matches the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the input operand and the first weight in the weight operand, the second multiplication operation is the multiplication of the second input element in the input operand and the second weight in the weight operand, the third multiplication operation is the multiplication of the third input element in the input operand and the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation can correspond to the same depth channel, and their product can also correspond to the same depth channel.

[0091] Multiple multipliers 630 can perform multiplication operations simultaneously. These multiplication operations can be referred to as a round of multiplication operations. In a round of multiplication operations performed by these multipliers 630, each multiplier 630 can use different input operands and different weight operands. Different input operands or weight operands can be stored in different register files of the PE 600. For example, a first multiplier 630 uses a first input operand (e.g., stored in the first input register file 610) and a first weight operand (e.g., stored in the first weight register file 620), a second multiplier 630 uses a second input operand (e.g., stored in the second input register file 610) and a second weight operand (e.g., stored in the second weight register file 620), a third multiplier 630 uses a third input operand (e.g., stored in the third input register file 610) and a third weight operand (e.g., stored in the third weight register file 620), and so on. For each multiplier 630, a round of multiplication operations can include multiple cycles. A cycle consists of a multiplication operation on the input elements and the weights.

[0092] The multiplier 630 can perform multiple rounds of multiplication operations. The multiplier 630 can use the same weight operand but different input operands in different rounds. For example, in a first round, the multiplier 630 performs a sequence of multiplication operations on a first input operand stored in a first input register file, and in a second round, performs a sequence of multiplication operations on a second input operand stored in a second input register file. In the second round, another multiplier 630 can perform another sequence of multiplication operations using the first input operand and a different weight operand. In this way, the first input operand is reused in the second round. The first input operand can be further reused in additional rounds, for example, by additional multipliers 630.

[0093] The internal adder component 640 includes one or more adders, i.e., internal adders, inside the PE 600. The internal adder component 640 can perform an accumulation operation on two or more product operands from the multiplier 630 and generate an output operand of the PE 600. In some embodiments, the internal adders are arranged in a tier sequence. The tier includes one or more internal adders. For the first tier of the internal adder component 640, the internal adder can receive product operands from two or more multipliers 630 and generate a sum operand through a sequence of accumulation operations. Each accumulation operation generates the sum of two or more products, each product coming from a different multiplier 630. The sum operand includes a sequence of sums, each of which is the result of an accumulation operation and corresponds to a depth channel. For the other tiers (one or more) of the internal adder component 640, the internal adder in the tier receives a sum operand from the previous tier in the sequence. Each of these values can be generated by a different internal adder in the previous tier. The ratio of the number of internal adders in a level to the number of internal adders in a subsequent level can be 2: 1. In some embodiments, the last level of the internal adder component 640 can include a single internal adder that generates the output operand of the PE 600 .

[0094] The output register file 650 stores the output operands of the PE 600. In some embodiments, the output register file 650 can store one output operand at a time. In other embodiments, the output register file 650 can store multiple output operands or a portion of an output operand at a time. The output operands include multiple output elements in the IFM. The output elements in the output operands can be stored sequentially in the output register file 650 so that the output elements can be processed sequentially. In some embodiments, each output element in the output operands corresponds to a different depth channel, and each output element is an element of a different output channel of the depthwise convolution. The number of output elements in the output operands can be equal to the number of depth channels of the depthwise convolution.

[0095] Example sparsity acceleration in PE

[0096] Figure 7 FIGURE 7 illustrates the sparsity acceleration of PE 700 in MAC operations according to various embodiments. PE 700 may be Figure 5 An example of PE 510 in Figure 7In the embodiment of the present invention, PE 700 includes an input register file 710, a weight register file 720, a multiplier 730, an accumulator 740, and an output register file 750. In other embodiments, PE 700 may include fewer, more, or different components. PE 700 is associated with a logic operator 760 and a sparsity logic unit 770. The logic operator 760 and the sparsity logic unit 770 may be Figure 4 Components of one embodiment of the medium sparsity accelerator 430 .

[0097] The input register file 710 stores at least a portion of the input operands. The input operands include a sequence of input elements (i.e., activation values). The input operands may be part of an input tensor (e.g., an input tensor of a convolutional layer). The input operands are associated with an input bitmap 715. The input bitmap 715 may be stored in the input register file 710, in a local memory of a computational block including the PE 700, or in both. The input bitmap 715 may indicate the location of non-zero activation values in the input operands. The input bitmap 715 includes a sequence of bits, each bit corresponding to a corresponding activation value in the input operand. In some embodiments, the positions of the bits in the input bitmap 715 match the positions of the corresponding activation values in the input operands. For illustrative purposes, the input bitmap 715 includes eight bits, and the input operand includes eight activation values. In other embodiments, the input bitmap 715 may include fewer or more bits. As Figure 7 As shown, four of the eight bits in input bitmap 715 are zero-valued bits, and the remaining four are one-valued bits. A zero-valued bit indicates that the value of the corresponding activation value is zero, and a one-valued bit indicates that the value of the corresponding activation value is non-zero. Therefore, the input operand includes four zero-valued activation values and four non-zero-valued activation values.

[0098] The weight register file 720 stores at least a portion of the weight operands. The weight operands include a sequence of weights. The weight operands may be part of a filter (e.g., a filter of a convolutional layer). The weight operands are associated with a weight bitmap 725. The weight bitmap 725 may be stored in the weight register file 720, in a local memory of a computational block including the PE 700, or in both. The weight bitmap 725 may indicate the location of non-zero valued weights in the weight operands. The weight bitmap 725 includes a sequence of bits, each bit corresponding to a corresponding weight in the weight operand. In some embodiments, the positions of the bits in the weight bitmap 725 match the positions of the corresponding weights in the weight operands. For illustrative purposes, the weight bitmap 725 includes eight bits and the weight operand includes eight weights. In other embodiments, the weight bitmap 725 may include fewer or more bits. As Figure 7As shown, four of the eight bits in the weight bitmap 725 are zero-valued bits, and the remaining four are one-valued bits. The zero-valued bits indicate that the value of the corresponding weight is zero, and the one-valued bits indicate that the value of the corresponding weight is non-zero. Therefore, the weight operand includes four zero-valued weights and four non-zero-valued weights. The weight bitmap 725 can indicate the positions of the non-zero-valued weights in the weight operand.

[0099] Logic operator 760 generates a combined sparsity bitmap 735 based on input bitmap 715 and weight bitmap 725. Logic operator 760 may receive input bitmap 715 from input register file 710 or from the local memory of a computation block including PE 700. Logic operator 760 may receive weight bitmap 725 from weight register file 720 or from the local memory of a computation block. In some embodiments, logic operator 760 is an AND operator. Logic operator 760 may generate combined sparsity bitmap 735 by performing one or more AND operations on input bitmap 715 and weight bitmap 725. Each bit in combined sparsity bitmap 735 is the result of an AND operation on a bit in input bitmap 715 and a bit in weight bitmap 725. The positions of the bits in combined sparsity bitmap 735 match the positions of the bits in input bitmap 715 and the positions of the bits in weight bitmap 725. For example, the first bit in the combined sparsity bitmap 735 is the result of an AND operation on the first bit in the input bitmap 715 and the first bit in the weight bitmap 725, the second bit in the combined sparsity bitmap 735 is the result of an AND operation on the second bit in the input bitmap 715 and the second bit in the weight bitmap 725, the third bit in the combined sparsity bitmap 735 is the result of an AND operation on the third bit in the input bitmap 715 and the third bit in the weight bitmap 725, and so on.

[0100] When the value of the corresponding bit in the input bitmap 715 and the corresponding bit in the weight bitmap 725 are both one, the value of the bit in the combined sparsity bitmap 735 is one. When at least one of the values of the corresponding bit in the input bitmap 715 and the corresponding bit in the weight bitmap 725 is zero, the value of the bit in the combined sparsity bitmap 735 is zero. Figure 7 As shown, combined sparsity bitmap 735 includes six zeros and two ones.

[0101] The total number of ones in the combined sparsity bitmap 735 is equal to the number of activation value-weight pairs that will be calculated by PE 700 and will produce non-zero partial sums. The calculation of other activation value-weight pairs can be skipped without any impact on output accuracy because these pairs will produce zero partial sums due to the activation value or weight being zero. Therefore, the workload of PE 700 in that calculation round can be determined based on the total number of ones in the combined sparsity bitmap 735. The amount of time for calculation can also be estimated based on the total number of ones in the combined sparsity bitmap 735. The more ones in the combined sparsity bitmap 735, the higher the workload of PE 700 and the longer the calculation time of PE 700.

[0102] The sparsity logic unit 770 retrieves activation values and weights from the input register file 710 and the weight register file 720, respectively, based on the combined sparsity bitmap 735. To speed up computations in the PE 700, the sparsity logic unit 770 retrieves two activation value-weight pairs corresponding to ones in the combined sparsity bitmap 735 and does not retrieve six activation value-weight pairs corresponding to zeros in the combined sparsity bitmap 735. In some embodiments, the input register file 710 or the weight register file 720 stores dense data points (e.g., non-zero activation values or non-zero weights). Sparse data points (e.g., zero activation values or zero weights) are not stored in the input register file 710 or the weight register file 720. Dense data points can be compressed and stored adjacent to each other in the input register file 710 or the weight register file 720. Sparsity logic unit 770 may identify activation values and weights based on the positions of ones in combined sparsity bitmap 735 , which may indicate the positions of non-zero activation values in input operands and the positions of non-zero weights in weight operands.

[0103] The multiplier 730 receives non-zero activation value-weight pairs from the sparsity logic unit 770 and performs a multiplication operation on the activation value-weight pairs. For example, the multiplier 730 performs a multiplication operation on the activation value and weight in the individual pairs and outputs a partial sum, which is the product of the activation value and the weight. Since there are two activation value-weight pairs, the multiplier 730 can, for example, perform two multiplication operations sequentially based on the position of one in the combined sparsity bitmap 735. Without sparsity acceleration, the multiplier 730 needs to perform eight multiplication operations. By reducing the number of multiplication operations from eight to two, the MAC operations in the PE 700 are accelerated. Since the DNN accelerator typically performs a large number of MAC operations when executing the DNN, sparsity acceleration can significantly improve the efficiency and performance of the DNN accelerator.

[0104] The accumulator 740 receives the two partial sums from the multiplier 730 and accumulates the two partial sums. The result of the accumulation is a PE-level internal partial sum. The PE-level internal partial sum can be stored in the output register file 750. In some embodiments, the accumulator 740 receives one or more PE-level internal partial sums from one or more other PEs. The accumulator 740 can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of PE 700 and store the accumulated result (i.e., the multiple PE internal partial sums) in the output register file 750. The one or more other PEs can be in the same column in the PE array as PE 700. The multiple PE internal partial sums can be column-level internal partial sums. In some embodiments, the PE-level internal partial sum or the multiple PE internal partial sum of PE 700 can be sent to one or more other PEs for further accumulation.

[0105] although Figure 7 A single multiplier 730 is shown, but PE 700 may include multiple multipliers that can perform multiple multiplication operations simultaneously. These multipliers may be coupled to an internal adder component, such as internal adder component 640. Combined sparsity bitmap 735 may also be provided to a compute scheduler (e.g., compute scheduler 440) to schedule the start time of MAC operations in PE 700, thereby reducing voltage drops in the PE group that includes PE 700.

[0106] Example Compute Scheduling

[0107] Figure 8 FIG2 shows a computation schedule for a group of PEs according to various embodiments, where computations for these PEs begin simultaneously. For illustration purposes, there are five PEs in the group. In other embodiments, the group may include other numbers of PEs. The PEs in the group may be arranged in a single column in the PE array, in multiple columns in the PE array, etc. Figure 8 Combined sparsity bitmaps 810, 820, 830, 840, and 850 for five PEs are shown.

[0108] The PE is associated with one or more sparsity accelerators (e.g., sparsity accelerator 430), which can accelerate computations in the PE based on combined sparsity bitmaps 810, 820, 830, 840, and 850. The number of ones in each of combined sparsity bitmaps 810, 820, 830, 840, and 850 indicates the amount of computation to be performed by the corresponding PE. Therefore, the workload of the PE with combined sparsity bitmap 850 is the highest, followed by the PE with combined sparsity bitmap 820, then the PE with combined sparsity bitmap 840, and then the PE with combined sparsity bitmap 810. The workload of the PE with combined sparsity bitmap 830 is the lowest.

[0109] Figure 8 Also shown is a clock cycle sequence 860. The clock cycle sequence 860 may be generated by a clock generator associated with the group of PEs, such as a clock generator of a DNN accelerator that includes the group of PEs. The clock cycle sequence 860 may be used to synchronize the operation of the PEs. Figure 8 In the embodiment, computations of the PEs are not scheduled based on the combined sparsity bitmaps 810, 820, 830, 840, and 850. Rather, the start of computations is synchronized and all begin at the second clock cycle of the clock cycle sequence 860. The synchronous start of computations may result in large current transients and, therefore, large voltage drops, which may degrade the DNN accelerator or even cause functional failure.

[0110] Since the workloads of PEs are different, the computations in PEs require different numbers of clock cycles, so the computations in PEs end at different clock cycles, e.g. Figure 8 As shown, the computation of the PE with the combined sparsity bitmap 850 ends last. Therefore, the total time to complete the computation in this group of PEs is eight clock cycles.

[0111] Figure 9 FIG2 illustrates a computational schedule for a group of PEs that can reduce voltage drops according to various embodiments. For illustration, there are five PEs in the group. In other embodiments, the group may include other numbers of PEs. The PEs in the group may be arranged in a single column in a PE array, in multiple columns in a PE array, etc. Figure 9 Combined sparsity bitmaps 910, 920, 930, 940, and 950 for five PEs are shown.

[0112] The PE is associated with one or more sparsity accelerators (e.g., sparsity accelerator 430), which can accelerate computations in the PE based on combined sparsity bitmaps 910, 920, 930, 940, and 950. The number of ones in each of the combined sparsity bitmaps 910, 920, 930, 940, and 950 indicates the amount of computation to be performed by the corresponding PE. Therefore, the workload of the PE with combined sparsity bitmap 950 is the highest, followed by the PE with combined sparsity bitmap 920, then the PE with combined sparsity bitmap 940, and then the PE with combined sparsity bitmap 910. The workload of the PE with combined sparsity bitmap 930 is the lowest.

[0113] Figure 9 Also shown is a clock cycle sequence 960. The clock cycle sequence 960 may be generated by a clock generator associated with the group of PEs, such as a clock generator of a DNN accelerator that includes the group of PEs. Figure 9In the embodiment of the present invention, computations of PEs are scheduled based on combined sparsity bitmaps 910, 920, 930, 940, and 950. The computation scheduling is determined by a computation scheduler (e.g., computation scheduler 440). The computations in the PE with combined sparsity bitmap 950 and the PE with combined sparsity bitmap 920 start first in the second clock cycle. The computations of the other PEs start in subsequent clock cycles. Figure 8 Compared with the computational scheduling in Figure 9 The computation scheduling in can reduce current transients and voltage drops because fewer PEs start computation at the same time.

[0114] like Figure 9 As shown in , the computation in PE ends at different clock cycles. Figure 8 Compared with the computational scheduling in Figure 9 The computation scheduling in does not cause any computation in this group of PEs to be delayed in completion, because the computation in the PE with the combined sparsity bitmap 950 still ends last, and the total time to complete the computation in this group of PEs is eight clock cycles.

[0115] Figure 10 Another computational schedule for a group of PEs that can reduce voltage drops according to various embodiments is shown. For illustration, there are five PEs in the group. In other embodiments, the group can include other numbers of PEs. The PEs in the group can be arranged in a single column in the PE array, in multiple columns in the PE array, etc. Figure 10 Combined sparsity bitmaps 1010, 1020, 1030, 1040, and 1050 for five PEs are shown.

[0116] The PE is associated with one or more sparsity accelerators (e.g., sparsity accelerator 430), which can accelerate computations in the PE based on combined sparsity bitmaps 1010, 1020, 1030, 1040, and 1050. The number of ones in each of combined sparsity bitmaps 1010, 1020, 1030, 1040, and 1050 indicates the amount of computation to be performed by the corresponding PE. Therefore, the workload of the PE with combined sparsity bitmap 1050 is the highest, followed by the PE with combined sparsity bitmap 1020, then the PE with combined sparsity bitmap 1040, and then the PE with combined sparsity bitmap 1010. The workload of the PE with combined sparsity bitmap 1030 is the lowest.

[0117] Figure 10 Also shown is a clock cycle sequence 1060. The clock cycle sequence 1060 may be generated by a clock generator associated with the group of PEs, such as a clock generator of a DNN accelerator that includes the group of PEs. Figure 10In the embodiment of the present invention, the computation of the PEs is scheduled based on the combined sparsity bitmaps 1010, 1020, 1030, 1040, and 1050. The PEs start their computations at different times. The higher the workload of a PE, the earlier the computation in the PE starts.

[0118] In some embodiments, the computation schedule is determined by a computation scheduler that schedules PE computations using a decrement counter. The decrement counter may be decremented from eight (i.e., the number of ones in the combined sparsity bitmap 1050) to one (i.e., the number of ones in the combined sparsity bitmap 1030) or zero. For example, the decrement counter is eight in the first clock cycle of the sequence of clock cycles 1060, seven in the second clock cycle, six in the third clock cycle, and so on. The decrement continues to the eighth clock cycle (when the decrement counter is one) or the ninth clock cycle (when the decrement counter is zero). The computation of the PE will begin in the clock cycle immediately following the clock cycle in which the value of the decrement counter matches the number of ones in the combined sparsity bitmap of the PE. As Figure 10 As shown, computation in the PE with combined sparsity bitmap 1050 begins first at the second clock cycle because the value of the down counter matches the number of ones in combined sparsity bitmap 1050 in the first clock cycle. Similarly, computation in the PE with combined sparsity bitmap 1020 begins at the fifth clock cycle because the value of the down counter matches the number of ones in combined sparsity bitmap 1020 in the fourth clock cycle. Computation in the PE with combined sparsity bitmap 1040 begins at the seventh clock cycle. Computation in the PE with combined sparsity bitmap 1010 begins at the eighth clock cycle. Computation in the PE with combined sparsity bitmap 1030 begins at the ninth clock cycle.

[0119] and Figure 8 and Figure 9 Compared with the computational scheduling in Figure 10 The calculation scheduling in can further reduce the current transient and voltage drop, because no PE starts their calculation at the same time. Figure 10 As shown, the computations in the PEs end at the same time, so there is no delay in completing any computations in the group of PEs.

[0120] Example of how to schedule computation in a DNN

[0121] Figure 11 is a flow chart illustrating a method 1100 for scheduling computations in a DNN, according to various embodiments. The method 1100 may be performed by Figure 4 Although the method 1100 is based on the calculation scheduler 440 of Figure 11The flowchart shown is described, but many other methods for scheduling computations in DNNs can also be used. For example, one can change Figure 11 As another example, some of the steps may be changed, eliminated, or combined.

[0122] The computation scheduler 440 determines (1110) a workload for each corresponding PE in a group of PEs based on input operands and weight operands. The corresponding PE is configured to perform a computation (e.g., a MAC operation) on the input operands and weight operands. The input operands include a plurality of activation values for a convolution. The weight operands include a plurality of weights for the convolution. In some embodiments, the group of PEs is at least a portion of a PE array. The PE array is configured to perform at least a portion of the convolution. The PE array includes rows and columns. The group of PEs is arranged in one of the columns.

[0123] In some embodiments, the compute scheduler 440 determines the workload of the corresponding PE based on the input sparsity bitmap and the weight sparsity bitmap. The input sparsity bitmap includes a bit sequence, each bit indicating whether the value of the corresponding activation value in the input operand is zero. The weight sparsity bitmap includes another bit sequence, each bit indicating whether the value of the corresponding weight in the weight operand is zero. A combined sparsity bitmap can be generated based on the input sparsity bitmap and the weight sparsity bitmap. The combined sparsity bitmap includes multiple bits, each bit is the result of multiplying a bit in the input sparsity bitmap with a bit in the weight sparsity bitmap. The compute scheduler 440 can determine the workload based on the number of ones in the combined sparsity bitmap.

[0124] The compute scheduler 440 determines 1120 that a workload of a first PE in the set of PEs is greater than a workload of a second PE in the set of PEs. For example, the compute scheduler 440 determines that the number of ones in the combined sparsity bitmap associated with the first PE is greater than the number of ones in the combined sparsity bitmap associated with the second PE.

[0125] The compute scheduler 440 instructs (1130) the first PE to start a first computation at a first time. In some embodiments, the compute scheduler 440 determines that the workload of the first PE is greater than one or more workloads of one or more other PEs in the group of PEs. The compute scheduler 440 instructs the first PE to start the first computation at a first clock cycle in a sequence of clock cycles. The one or more computations of the one or more other PEs start one or more clock cycles after the first clock cycle in the sequence of clock cycles.

[0126] The compute scheduler 440 instructs (1140) the second PE to begin a second computation at a second time, the second time being later than the first time. In some embodiments, the compute scheduler 440 associates a sequence of numerical values with a sequence of clock cycles. Each corresponding clock cycle is associated with a greater numerical value than another clock cycle in the clock sequence that follows the clock cycle. A first clock cycle (i.e., a clock cycle in which the first computation begins) is associated with a first numerical value representing a workload of the first PE. The compute scheduler 440 determines a second numerical value representing a workload of the second PE. The compute scheduler 440 identifies a second clock cycle from the sequence of clock cycles that is associated with the second numerical value. The compute scheduler 440 instructs the second PE to begin the second computation at the second clock cycle.

[0127] In some embodiments, the computation scheduler 440 determines the first time and the second time based on the workload of the first PE and the workload of the second PE. The first computation and the second computation end simultaneously. In some embodiments, the computation scheduler 440 determines the second time based on the workload of the first PE and the workload of the second PE. The second computation ends no later than the first computation ends.

[0128] Example computing device

[0129] Figure 12 is a block diagram of an example computing device 1200 according to various embodiments. In some embodiments, the computing device 1200 may be used as Figure 3 At least a portion of the DNN accelerator 300 in. Multiple components in Figure 12 1200, but any one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system on a chip (SoC) die. Additionally, in various embodiments, computing device 1200 may not include Figure 12 1200 may include one or more of the components shown, but the computing device 1200 may include interface circuitry for coupling to the one or more components. For example, the computing device 1200 may not include the display device 1206, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which the display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include the audio input device 1218 or the audio output device 1208, but may include audio input or output device interface circuitry (e.g., a connector and supporting circuitry) to which the audio input device 1218 or the audio output device 1208 may be coupled.

[0130] The computing device 1200 may include a processing device 1202 (e.g., one or more processing devices). The "processing device" 1202 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1200 may include a memory 1204, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 1204 may include a memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-transitory computer-readable media storing instructions that are executable to perform operations for scheduling computations in the DNN, for example, in conjunction with the above. Figure 11 The method 1100 described above, or a combination thereof Figure 4 Some of the operations performed by the computation scheduler 440 are described. Instructions stored in one or more non-transitory computer-readable media may be executed by the processing device 1202.

[0131] In some embodiments, computing device 1200 may include a communication chip 1212 (e.g., one or more communication chips). For example, communication chip 1212 may be configured to manage wireless communications for transmitting data to and from computing device 1200. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, and the like that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not imply that the associated devices do not contain any wires, although in some embodiments they may not.

[0132] The communication chip 1212 may implement any of a number of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), the Long Term Evolution (LTE) project and any amendments, updates, and / or revisions (e.g., the LTE-Advanced project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband wireless access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks, which is an abbreviation for Worldwide Interoperability for Microwave Access and is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 1212 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 1212 can also operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1212 may operate according to code-division multiple access (CDMA), time division multiple access (TDMA), digital enhanced cordless telecommunication (DECT), evolution-data optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, etc. In other embodiments, the communication chip 1212 may operate according to other wireless protocols.Computing device 1200 may include antenna 1222 to facilitate wireless communications and / or receive other wireless communications (eg, AM or FM radio transmissions).

[0133] In some embodiments, the communication chip 1212 can manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1212 can include multiple communication chips. For example, the first communication chip 1212 can be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and the second communication chip 1212 can be dedicated to long-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1212 can be dedicated to wireless communications, and the second communication chip 1212 can be dedicated to wired communications.

[0134] The computing device 1200 may include a battery / power circuit 1214. The battery / power circuit 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0135] Computing device 1200 may include a display device 1206 (or corresponding interface circuitry as described above). For example, display device 1206 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat-panel display.

[0136] Computing device 1200 may include an audio output device 1208 (or corresponding interface circuitry as described above). For example, audio output device 1208 may include any device that generates an audible indicator, such as a speaker, headphones, or earbuds.

[0137] The computing device 1200 may include an audio input device 1218 (or corresponding interface circuitry as described above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., an instrument with a MIDI (Musical Instrument Digital Interface) output).

[0138] Computing device 1200 may include a GPS device 1216 (or corresponding interface circuitry as described above). As is known in the art, GPS device 1216 may communicate with a satellite-based system and may receive the location of computing device 1200.

[0139] Computing device 1200 may include other output devices 1210 (or corresponding interface circuitry as described above). Examples of other output devices 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0140] Computing device 1200 may include other input devices 1220 (or corresponding interface circuitry as described above). Examples of other input devices 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0141] The computing device 1200 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smartphone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA (personal digital assistant), an ultraportable personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 can be any other electronic device that processes data.

[0142] Select Example

[0143] The following paragraphs provide various examples of the embodiments disclosed herein.

[0144] Example 1 provides a method for scheduling computations in a DNN, comprising: determining a workload of each corresponding PE in a group of PEs based on input operands and weight operands, the corresponding PEs being configured to perform computations on the input operands and the weight operands, the input operands comprising multiple activation values of a convolution, the weight operands comprising multiple weights of the convolution; determining that a workload of a first PE in the group of PEs is greater than a workload of a second PE in the group of PEs; instructing the first PE to start a first computation at a first time; and instructing the second PE to start a second computation at a second time, the second time being later than the first time.

[0145] Example 2 provides the method of Example 1, wherein determining the workload of the corresponding PE based on the input operand and the weight operand includes: determining the workload of the corresponding PE based on an input sparsity bitmap and a weight sparsity bitmap, wherein the input sparsity bitmap includes a bit sequence, each bit of which indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, each bit of which indicates whether the value of the corresponding weight in the weight operand is zero.

[0146] Example 3 provides the method of Example 2, wherein determining the workload based on the input sparsity bitmap and the weight sparsity bitmap includes: generating a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap including a plurality of bits, each bit being the result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap; and determining the workload based on the number of ones in the combined sparsity bitmap.

[0147] Example 4 provides a method of any of the preceding examples, wherein determining that a workload of a first PE in the group of PEs is greater than a workload of a second PE in the group of PEs includes determining that a number of ones in a combined sparsity bitmap associated with the first PE is greater than a number of ones in a combined sparsity bitmap associated with the second PE.

[0148] Example 5 provides the method of any of the foregoing examples, further comprising: determining the first time and the second time based on the workload of the first PE and the workload of the second PE, wherein the time when the second calculation ends is no later than the time when the first calculation ends.

[0149] Example 6 provides the method of any of the foregoing examples, further comprising: determining the second time based on the workload of the first PE and the workload of the second PE, wherein the time when the second calculation ends is no later than the time when the first calculation ends.

[0150] Example 7 provides a method of any of the preceding examples, wherein instructing the first PE to start a first calculation at a first time includes: determining that a workload of the first processing element is greater than at least one workload of another processing element in the group of processing elements; and instructing the first processing element to start the first calculation at a first clock cycle in a sequence of clock cycles, wherein another processing element having a workload less than the first processing element starts at one or more clock cycles after the first clock cycle in the sequence of clock cycles.

[0151] Example 8 provides a method of any of the preceding examples, wherein instructing the second PE to start a second calculation at a second time includes: associating a sequence of numerical values with the sequence of clock cycles, each corresponding clock cycle being associated with a larger numerical value than another clock cycle that follows the corresponding clock cycle in the sequence of clock cycles, the first clock cycle being associated with a first numerical value representing the workload of the first PE; determining a second numerical value representing the workload of the second PE; identifying a second clock cycle associated with the second numerical value from the sequence of clock cycles; and instructing the second PE to start the second calculation at the second clock cycle.

[0152] Example 9 provides the method of any preceding example, wherein the set of PEs is at least part of a PE array configured to perform at least part of the convolution.

[0153] Example 10 provides the method of Example 9, wherein the group of processing elements is at least a portion of a processing element array, wherein the processing element array is configured to perform at least a portion of the convolution, and wherein the processing element array includes rows and columns, and the group of processing elements are arranged in one of the columns.

[0154] Example 11 provides a computation block for performing computations in a DNN, the computation block comprising: a group of PEs, each PE configured to perform computations on input operands and weight operands, wherein the input operands include multiple activation values of a convolution and the weight operands include multiple weights of the convolution; and a computation scheduler configured to: determine a workload of each PE based on the input operands and the weight operands; determine that a workload of a first PE in the group of PEs is greater than a workload of a second PE in the group of PEs; instruct the first PE to start a first computation at a first time; and instruct the second PE to start a second computation at a second time, the second time being later than the first time.

[0155] Example 12 provides the computation block of Example 11, wherein the computation scheduler is configured to determine the workload of the corresponding PE based on the input operand and the weight operand by performing the following operations: determining the workload of the corresponding PE based on an input sparsity bitmap and a weight sparsity bitmap, wherein the input sparsity bitmap includes a bit sequence, wherein each bit indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, wherein each bit indicates whether the value of the corresponding weight in the weight operand is zero.

[0156] Example 13 provides the compute block of Example 12, further comprising a sparsity accelerator configured to generate a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap comprising a plurality of bits, each of which is a result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap, wherein the compute scheduler is configured to determine a workload based on the number of ones in the combined sparsity bitmap.

[0157] Example 14 provides a compute block of any one of Examples 11-13, wherein the compute scheduler is configured to determine that a workload of a first PE in the group of PEs is greater than a workload of a second PE in the group of PEs by performing the following operations: determining that a number of ones in a combined sparsity bitmap associated with the first PE is greater than a number of ones in a combined sparsity bitmap associated with the second PE.

[0158] Example 15 provides a computing block of any one of Examples 11-14, wherein the computing scheduler is further configured to: determine the first time and the second time based on the workload of the first PE and the workload of the second PE, wherein the first computing and the second computing end at the same time.

[0159] Example 16 provides a computing block of any one of Examples 11-15, wherein the computing scheduler is further configured to: determine the second time based on the workload of the first PE and the workload of the second PE, wherein the time when the second computing ends is no later than the time when the first computing ends.

[0160] Example 17 provides a computation block of any one of Examples 11-16, wherein the computation scheduler is further configured to instruct the first PE to start the first computation at the first time by: determining that the workload of the first PE is greater than one or more workloads of one or more other PEs in the group of PEs; and instructing the first PE to start the first computation at the first clock cycle in a sequence of clock cycles, wherein one or more computations of the one or more other PEs start one or more clock cycles after the first clock cycle in the sequence of clock cycles.

[0161] Example 18 provides a computing block of any one of Examples 11-17, wherein the computing scheduler is further configured to instruct the second PE to start the second computing at the second time by performing the following operations: associating a numerical sequence with the clock cycle sequence, each corresponding clock cycle being associated with a larger numerical value than another clock cycle in the clock cycle sequence that follows the corresponding clock cycle, the first clock cycle being associated with a first numerical value representing the workload of the first PE; determining a second numerical value representing the workload of the second PE; identifying a second clock cycle associated with the second numerical value from the clock cycle sequence; and instructing the second PE to start the second computing at the second clock cycle.

[0162] Example 19 provides the computation block of any one of Examples 11-18, wherein the set of PEs is at least a portion of a PE array configured to perform at least a portion of the convolution.

[0163] Example 20 provides the computation block of Example 19, wherein the PE array comprises rows and columns, and the group of PEs is arranged in one of the columns.

[0164] Example 21 provides one or more non-transitory computer-readable media storing instructions that are executable to perform operations for scheduling computations in a DNN, the operations comprising: determining a workload for each corresponding PE in a group of PEs based on input operands and weight operands, the corresponding PEs being configured to perform computations on the input operands and the weight operands, the input operands comprising one or more activation values of a convolution, the weight operands comprising one or more weights of the convolution; determining that a workload of a first PE in the group of PEs is greater than a workload of a second PE in the group of PEs; instructing the first PE to start a first computation at a first time; and instructing the second PE to start a second computation at a second time, the second time being later than the first time.

[0165] Example 22 provides one or more non-transitory computer-readable media of Example 21, wherein determining the workload of the corresponding PE based on the input operand and the weight operand includes: determining the workload of the corresponding PE based on an input sparsity bitmap and a weight sparsity bitmap, wherein the input sparsity bitmap includes a bit sequence, each bit of which indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, each bit of which indicates whether the value of the corresponding weight in the weight operand is zero.

[0166] Example 23 provides one or more non-transitory computer-readable media of Example 22, wherein determining the workload based on the input sparsity bitmap and the weight sparsity bitmap includes: generating a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap including a plurality of bits, each bit being the result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap; and determining the workload based on the number of ones in the combined sparsity bitmap.

[0167] Example 24 provides one or more non-transitory computer-readable media of any of Examples 21-23, wherein the second time is determined based on the workload of the first PE and the workload of the second PE, and wherein the time when the second calculation ends is no later than the time when the first calculation ends.

[0168] Example 25 provides one or more non-transitory computer-readable media of any of Examples 21-24, wherein instructing the first PE to start a first calculation at a first time includes: determining that a workload of the first PE is greater than one or more workloads of one or more other PEs in the group of PEs; and instructing the first PE to start the first calculation at a first clock cycle in a sequence of clock cycles, wherein one or more calculations of one or more other PEs start one or more clock cycles after the first clock cycle in the sequence of clock cycles.

[0169] The above description of the illustrated embodiments of the present disclosure, including that described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Although specific implementations and examples of the present disclosure are described herein for illustrative purposes, various equivalent modifications may be made within the scope of the present disclosure, as will be appreciated by those skilled in the relevant art. These modifications may be made to the present disclosure in light of the above detailed description.

Claims

1. A method for scheduling computations in a deep neural network (DNN), comprising: determining a workload for each respective processing element in a set of processing elements based on input operands and weight operands, the respective processing elements being configured to perform a computation on the input operands and the weight operands, the input operands comprising one or more activation values for a convolution and the weight operands comprising one or more weights for the convolution; determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements; instructing the first processing element to begin a first computation at a first time; as well as The second processing element is instructed to begin a second computation at a second time, the second time being later than the first time.

2. The method according to claim 1, wherein Determining the workload of the corresponding processing element based on the input operand and the weight operand includes: determining a workload of the corresponding processing element based on the input sparsity bitmap and the weight sparsity bitmap, The input sparsity bitmap includes a bit sequence, each bit of which indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, each bit of which indicates whether the value of the corresponding weight in the weight operand is zero.

3. The method according to claim 2, wherein: Determining the workload based on the input sparsity bitmap and the weight sparsity bitmap includes: generating a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap comprising a plurality of bits, wherein each bit is a result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap; and The workload is determined based on the number of ones in the combined sparsity bitmap.

4. A method according to any preceding claim, wherein: Determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements includes: A determination is made that a number of ones in a combined sparsity bitmap associated with the first processing element is greater than a number of ones in a combined sparsity bitmap associated with the second processing element.

5. The method according to any preceding claim, further comprising: determining the first time and the second time based on a workload of the first processing element and a workload of the second processing element, The second calculation ends no later than the first calculation ends.

6. A method according to any preceding claim, wherein: Instructing the first processing element to start a first calculation at a first time includes: determining that the workload of the first processing element is greater than at least one workload of another processing element in the set of processing elements; and instructing the first processing element to begin the first computation on a first clock cycle in a sequence of clock cycles, The another processing element having a smaller workload than the first processing element starts one or more clock cycles after the first clock cycle in the sequence of clock cycles.

7. A method according to any preceding claim, wherein: Instructing the second processing element to start a second calculation at a second time includes: associating a sequence of numerical values with the sequence of clock cycles, each respective clock cycle being associated with a greater numerical value than another clock cycle in the sequence of clock cycles that follows the respective clock cycle, the first clock cycle being associated with a first numerical value representative of a workload of the first processing element; determining a second value representing a workload of the second processing element; identifying a second clock cycle associated with the second value from the sequence of clock cycles; and The second processing element is instructed to begin the second computation in the second clock cycle.

8. A method according to any preceding claim, wherein: The set of processing elements is at least part of a processing element array, wherein the processing element array is configured to perform at least part of the convolution, and wherein the processing element array comprises rows and columns, and the set of processing elements is arranged in one of the columns.

9. One or more non-transitory computer-readable media storing instructions executable to perform operations for computing in a network, the operations comprising: determining a workload for each respective processing element in a set of processing elements based on input operands and weight operands, the respective processing elements being configured to perform a computation on the input operands and the weight operands, the input operands comprising one or more activation values for a convolution and the weight operands comprising one or more weights for the convolution; determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements; instructing the first processing element to begin a first computation at a first time; as well as The second processing element is instructed to begin a second computation at a second time, the second time being later than the first time.

10. The one or more non-transitory computer-readable media of claim 9, wherein: Determining the workload of the corresponding processing element based on the input operand and the weight operand includes: determining a workload of the corresponding processing element based on the input sparsity bitmap and the weight sparsity bitmap, The input sparsity bitmap includes a bit sequence, each bit of which indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, each bit of which indicates whether the value of the corresponding weight in the weight operand is zero.

11. The one or more non-transitory computer-readable media of claim 10, wherein: Determining the workload based on the input sparsity bitmap and the weight sparsity bitmap includes: generating a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap comprising a plurality of bits, wherein each bit is a result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap; and The workload is determined based on the number of ones in the combined sparsity bitmap.

12. One or more non-transitory computer-readable media according to any one of claims 9-11, wherein: Determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements includes: A determination is made that a number of ones in a combined sparsity bitmap associated with the first processing element is greater than a number of ones in a combined sparsity bitmap associated with the second processing element.

13. The one or more non-transitory computer-readable media of any one of claims 9-12, further comprising: determining the first time and the second time based on a workload of the first processing element and a workload of the second processing element, The second calculation ends no later than the first calculation ends.

14. One or more non-transitory computer-readable media according to any one of claims 9-13, wherein: Instructing the first processing element to start a first calculation at a first time includes: determining that the workload of the first processing element is greater than at least one workload of another processing element in the set of processing elements; and instructing the first processing element to begin the first computation on a first clock cycle in a sequence of clock cycles, The another processing element having a smaller workload than the first processing element starts one or more clock cycles after the first clock cycle in the sequence of clock cycles.

15. One or more non-transitory computer-readable media according to any one of claims 9-14, wherein: Instructing the second processing element to start a second calculation at a second time includes: associating a sequence of numerical values with the sequence of clock cycles, each respective clock cycle being associated with a greater numerical value than another clock cycle in the sequence of clock cycles that follows the respective clock cycle, the first clock cycle being associated with a first numerical value representative of a workload of the first processing element; determining a second value representing a workload of the second processing element; identifying a second clock cycle associated with the second value from the sequence of clock cycles; and The second processing element is instructed to begin the second computation in the second clock cycle.

16. One or more non-transitory computer-readable media according to any one of claims 9-15, wherein: The set of processing elements is at least part of a processing element array, wherein the processing element array is configured to perform at least part of the convolution, and wherein the processing element array includes rows and columns, the set of processing elements being arranged in one of the columns.

17. An apparatus comprising: a computer processor for executing computer program instructions; as well as a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: determining a workload for each respective processing element in a set of processing elements based on input operands and weight operands, the respective processing elements being configured to perform a computation on the input operands and the weight operands, the input operands comprising one or more activation values for a convolution and the weight operands comprising one or more weights for the convolution; determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements; instructing the first processing element to begin a first computation at a first time; and The second processing element is instructed to begin a second computation at a second time, the second time being later than the first time.

18. The device according to claim 17, wherein Determining the workload of the corresponding processing element based on the input operand and the weight operand includes: determining a workload of the corresponding processing element based on the input sparsity bitmap and the weight sparsity bitmap, The input sparsity bitmap includes a bit sequence, each bit of which indicates whether the value of the corresponding activation value in the input operand is zero, and the weight sparsity bitmap includes another bit sequence, each bit of which indicates whether the value of the corresponding weight in the weight operand is zero.

19. The device according to claim 18, wherein Determining the workload based on the input sparsity bitmap and the weight sparsity bitmap includes: generating a combined sparsity bitmap based on the input sparsity bitmap and the weight sparsity bitmap, the combined sparsity bitmap comprising a plurality of bits, wherein each bit is a result of multiplying a bit in the input sparsity bitmap by a bit in the weight sparsity bitmap; and The workload is determined based on the number of ones in the combined sparsity bitmap.

20. The device according to any one of claims 17 to 19, wherein Determining that a workload of a first processing element in the set of processing elements is greater than a workload of a second processing element in the set of processing elements includes: A determination is made that a number of ones in a combined sparsity bitmap associated with the first processing element is greater than a number of ones in a combined sparsity bitmap associated with the second processing element.

21. The apparatus according to any one of claims 17 to 20, further comprising: determining the first time and the second time based on a workload of the first processing element and a workload of the second processing element, The second calculation ends no later than the first calculation ends.

22. The device according to any one of claims 17 to 21, wherein Instructing the first processing element to start a first calculation at a first time includes: determining that the workload of the first processing element is greater than at least one workload of another processing element in the set of processing elements; and instructing the first processing element to begin the first computation on a first clock cycle in a sequence of clock cycles, The another processing element having a smaller workload than the first processing element starts one or more clock cycles after the first clock cycle in the sequence of clock cycles.

23. The device according to any one of claims 17 to 22, wherein: Instructing the second processing element to start a second calculation at a second time includes: associating a sequence of numerical values with the sequence of clock cycles, each respective clock cycle being associated with a greater numerical value than another clock cycle in the sequence of clock cycles that follows the respective clock cycle, the first clock cycle being associated with a first numerical value representative of a workload of the first processing element; determining a second value representing a workload of the second processing element; identifying a second clock cycle associated with the second value from the sequence of clock cycles; and The second processing element is instructed to begin the second computation in the second clock cycle.

24. The device according to any one of claims 17 to 23, wherein: The set of processing elements is at least a portion of an array of processing elements.

25. The apparatus according to claim 24, wherein The processing element array is configured to perform at least a portion of the convolution, and wherein the processing element array comprises rows and columns, the set of processing elements being arranged in one of the columns.

Images