Implementing group quantization-based neural network operation on neural processing unit

Abstract

A compiler may compile a neural network and facilitate group quantization-based neural network operations on a neural processing unit (NPU). The compiler may quantize a tensor of a layer by partitioning the tensor into groups and quantizing the groups using group-specific scales. A quantized group may be stored as microscaling block. A data processing unit (DPU) in the NPU may perform group operations on the quantized groups. Results of the group operations may be accumulated by the DPU or by a digital signal processor (DSP) in the NPU to produce an output tensor of the DNN layer. The DPU may have a 32-bit load path. The DSP may determine register settings for workloads of the DPU. The compiler may interleave quantized groups to facilitate accumulations on the DPU. The compiler may dequantize the quantized group before the layer execution, and the NPU may execute the layer using the dequantized groups.

Description

IMPLEMENTING GROUP QUANTIZATION-BASED NEURAL NETWORK OPERATION ON NEURAL PROCESSING UNITCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No.63 / 701,903, filed September 30, 2024, and entitled "GROUP QUANTIZATION BASED NEURAL NETWORK OPERATION," which is incorporated by reference in its entirety for all purposes.Technical Field

[0002] This disclosure relates generally to neural networks (also referred to as "deep neural networks" or "DNN"), and more specifically, implementing group quantization-based neural network operations on neural processing units (NPUs).Background

[0003] DNNs are used extensively for a variety of artificial intelligence (Al) applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write.Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments can be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] Figures (FIGS.) 1A-1C illustrate a group quantization-based neural network operation, in accordance with various embodiments.

[0006] FIG. 2 illustrates group quantization, in accordance with various embodiments.

[0007] FIG. 3 is a block diagram of an Al system, in accordance with various embodiments.

[0008] FIG. 4 is a block diagram of a DNN module, in accordance with various embodiments.

[0009] FIG. 5 illustrates an example microscaling block, in accordance with various embodiments.

[0010] FIG. 6 illustrates a workflow in a data processing unit (DPU) with a 32-bit load path, in accordance with various embodiments.

[0011] FIG. 7 illustrates a 32-bit load path in a DPU, in accordance with various embodiments.

[0012] FIG. 8 illustrates a data loading logic, in accordance with various embodiments.

[0013] FIG. 9 illustrates a workflow in a DPU and a digital signal processor (DSP), in accordance with various embodiments.

[0014] FIG. 10 illustrates an NPU with a DPU and a DSP, in accordance with various embodiments.

[0015] FIG. 11 illustrates DPU-DSP pipelining, in accordance with various embodiments.

[0016] FIG. 12 illustrates DPU-DSP pipelining for a group quantization-based workflow, in accordance with various embodiments.

[0017] FIG. 13 illustrates interleaving input channel groups of an input tensor, in accordance with various embodiments.

[0018] FIG. 14 illustrates interleaving input channel groups of another input tensor, in accordance with various embodiments.

[0019] FIG. 15 illustrates a sparse cell, in accordance with various embodiments.

[0020] FIG. 16 illustrates a post-processing engine, in accordance with various embodiments.

[0021] FIG. 17 illustrates distribution of an input vector across ICs to different PEs groups within a DPU, in accordance with various embodiments.

[0022] FIG. 18 illustrates an example data flow for executing a DNN layer based on sparsity masks, in accordance with various embodiments.

[0023] FIG. 19 illustrates a workflow in a compiler and a DPU, in accordance with various embodiments.

[0024] FIG. 20 illustrates a DPU with group-based features, in accordance with various embodiments.

[0025] FIG. 21 illustrates an example flow of control and data for structured sparsity acceleration, in accordance with various embodiments.

[0026] FIG. 22 illustrates an example sparse cell, in accordance with various embodiments.

[0027] FIG. 23 illustrates an example PE, in accordance with various embodiments.

[0028] FIG. 24 illustrates weight loading for a group operation, in accordance with various embodiments.

[0029] FIG. 25 illustrates a MatMul layer, in accordance with various embodiments.

[0030] FIG. 26 illustrates GPTQ MatMul, in accordance with various embodiments.

[0031] FIG. 27 illustrates a DPU-DSP pipeline for executing a MatMul layer, in accordance with various embodiments.

[0032] FIG. 28 illustrates a DPU pipeline for executing a MatMul layer, in accordance with various embodiments.

[0033] FIG. 29 illustrates weight table generation for a MatMul layer, in accordance with various embodiments.

[0034] FIG. 30 illustrates weight table generation on a DSP, in accordance with various embodiments.

[0035] FIG. 31 illustrates an example transformer model, in accordance with various embodiments.

[0036] FIG. 32 illustrates an example CNN, in accordance with various embodiments.

[0037] FIG. 33 is a flowchart of a method of executing a DNN, in accordance with various embodiments.

[0038] FIG. 34 is a block diagram of an example computing device, in accordance with various embodiments.Detailed Description

[0039] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in the domains of language processing, computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more operations, such as convolution, matrix multiplication (MatMul), interpolation, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. These operations are referred to as deep learning operations or neural network operations.

[0040] Neural network operations may be tensor operations. Input or output data of neural network operations may be arranged in data structures called tensors. Taking aconvolutional layer or MatMul layer for example, the input tensors include an activation tensor and a weight tensor. Data elements in the activation tensor may be referred to as activations or input activations. Data elements in the weight tensor may be referred to as weights. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor). A neural network operation may be performed on the activation tensor and weight tensor to compute an output tensor. The output tensor may be further used as an activation tensor of a subsequent neural network operation in the DNN.

[0041] A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, fourdimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. In some embodiments, a 3D tensor may have an X-dimension, a Y-dimension, and Z dimension. The X-dimension of a tensor may be the horizontal dimension, the length of which may be the width of the tensor; the Y dimension may be the vertical dimension, the length of which may be the height of the tensor; and the Z dimension may be the channel dimension, the length of which may be the number of channels. The coordinates of the elements along a dimension may be integers in an inclusive range from 0 to (L — 1), where L is the length of the tensor in the dimension. For instance, the x coordinate of the first element in a row may be 0, the x coordinate of the second element in a row may be 1, and so on. Similarly, the y coordinate of the first element in a column may be 0, the y coordinate of the second element in a column may be 1, and so on. A 4D tensor may have a fourth dimension, which may indicate the number of batches in the operation.

[0042] Group quantization-based compression schemes are used for transformer-based computer vision and natural language processing (NLP) models. An example is generative pre-trained transformer quantization (GPTQ). GPTQ is a one-shot weight quantization method based on approximate second-order information, that is both highly accurate and highly efficient. GPTQ can quantize LLM models with more than 175 billion parameters toreduce the bit-width down to 3 or 4 bits per weight from FP16 or INT8 precision, with negligible accuracy degradation relative to the full precision baseline. Compared to normal convolution, GPTQ-based group convolutions can result in significantly better accuracy at lower bit precision. GPTQ paves the way for bringing many DNNs, such as NLP / GenAI / LLM models, to mobile or edge devices. It can be important to support GPTQ quantized models on a client device with limited memory capacity, bandwidth, and power.

[0043] Many neural processing units (NPUs) are not equipped to execute group convolutions efficiently. Many currently available solutions employ a secondary accumulator within the NPU for performing the accumulation of the group-based partial sums. After each group is accumulated, a scaling factor would be applied multiplying the accumulated values and putting them in a second accumulator. Then the first accumulator could be used for the next group. These solutions perform per vector scaling instead of per group. However, with the emergence and rise in popularity of group convolutions from an accuracy point of view or from a transformer point of view, it can be important to add group convolution-based additions to the existing NPU architecture.

[0044] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by implementing group quantization-based neural network operations on NPUs. This disclosure provides techniques that allow NPUs to execute DNN layers by performing group operations with desirable efficiency and performance. A neural network operation of a DNN layer may be converted to a plurality of group operations and an elementwise operation. For example, a convolutional layer may include a plurality of group convolutions, the output of which are partial sums, and an elementwise addition that accumulates the partial sums. As another example, a MatMul layer may include a plurality of group MatMuls and one or more accumulation operations.

[0045] In various embodiments of the present disclosure, an Al system may include a compiler and an NPU. The compiler may compile DNNs to make the DNNs executable by the DPU. The compiler may also generate data for implementing group quantization-based neural network operations on the NPU. For instance, the compiler may perform group quantization on a weight tensor of a DNN layer. The compiler may partition the weight tensor into groups. The partition may be along the input channel dimension of the weight tensor, and the groups may be channel groups. The compiler may quantize the groups into different data precisions, e.g., by using group-specific quantization parameters. Thecompiler may also perform group quantization on an activation tensor of the DNN layer. The compiler may store quantized groups as microscaling blocks. A microscaling block is a data structure in a microscaling format. The NPU may perform group operations (e.g., group convolutions or group MatMuls) on the quantized groups. The NPU may also accumulate outputs of the group operations to produce an output tensor of the DNN layer. The NPU may include a local memory, a DPU, and a DSP. The workloads for executing the DNN layer may be distributed to the DPU or DSP.

[0046] In some embodiments, the DPU performs the group operations and accumulation. The DPU may include a processing engine and a post-processing engine. The processing engine may include an array of PEs. The PEs may be MAC-based PEs ("MPEs"). The processing engine may perform the MAC operations in the group operations and produce partial sums. The partial sums may be dequantized by the post-processing engine using group-specific scales and zero points to produce partial sums of a higher precision, such as 32-bit partial sums. After the dequantization, the partial sums may be drained into the local memory. The partial sums can be loaded back from the local memory into the processing engine through a load path (e.g., a 32-bit load path) for the processing engine to perform the accumulation. The final points may be drained and stored in the local memory. The 32- bit load path can be used for other purposes, such as fused elementwise addition in residual networks.

[0047] In some embodiments, the DPU performs the group operations, and the DSP performs the accumulation. After the processing engine performs the MAC operations in the group operations and the post-processing engine dequantizes the partial sums, the higher-precision partial sums may be stored in the local memory. The DSP may read the partial sums from the local memory and perform the accumulation. The DSP may start the accumulation before the processing engine completes the group operations. The DSP may start accumulation when it can read sufficient data to perform an accumulation workload. For instance, the DSP may perform the first accumulation workload after the DPU finishes the first two group operations but before the third group operation is complete or even start. The DSP may carry out more accumulation workloads as the DPU computes more partial sums. Through such DPU-DSP pipelining, the NPU efficiency and performance can be improved.

[0048] In some embodiments, the groups may be interleaved to facilitate the DPU to perform the group operations and the accumulation. The group interleaving may be performed by the compiler. The compiler may partition each channel group of the weight tensor into multiple subgroups and interleave the subgroups from different channel groups. For instance, the compiler may form a new weight tensor by putting the subgroups into a sequence of sets. In an example having two channel groups and each channel group having two subgroups, the compiler may form two sets. The first set may have the first subgroup from the first channel group followed by the first subgroup from the second channel group. The second set may have the second subgroup from the first channel group followed by the second subgroup from the second channel group. Contexts within the new weight tensor may be distributed to PEs in the processing engine to perform MAC operations. A context may have weights having the same spatial position (e.g., the same (x,y) coordinate) but in different channels. The post-processing engine may perform dequantization as well as accumulation to produce the output tensor of the DNN layer.

[0049] In some embodiments, the compiler may dequantize weights to a data precision before the execution of the DNN layer. For instance, the compiler may dequantize weights of all the channel groups, which have different precisions, to the activation precision. The NPU may receive activations and weights of fixed precision. The NPU may execute the DNN layer in the regular manner. For larger number of groups, this technique can really lead to significant performance / power benefits.

[0050] These techniques can implement group quantitation-based neural network operations on NPUs without any additional hardware overhead or with minimal hardware overhead. Compared with currently available solution (e.g., the secondary accumulatorbased solutions), the solutions in this disclosure can reduce the area and power overheads of the matrix multiplication-based circuitry. The negligible impact on the overall area and power leads to a significant improvement in the overall TOPS / W or T0PS / mm2 of the NPU, which are critical metrics for client or edge Al platforms.

[0051] NPUs may have group-based features to implement group operations. For instance, there may be support for group-based zero point for activations and weights for each group, precision-based palletization table for each group, N:M structured sparsity-based LUT, scale and bias table, and so on. These LUTs may be configurable. The compiler may program the LUTs on a per-group basis before DNN layer execution.

[0052] Implementation of group quantization-based neural network operations can enable additional performance enhancements such as early prestart of the next convolution and skipping the reloading of weights shared among different groups. Enabling group quantization-based models can lead to significantly reduction of the overall memory footprint and thereby improve the energy efficiency of these LLM-specific models. For instance, there can be significant performance improvements during the pre-filling and decode phases of state-of-the-art transformer models.

[0053] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it can be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0054] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0055] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0056] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0057] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms"comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0058] In the following detailed description, various aspects of the illustrative implementations are described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0059] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0060] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0061] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0062] FIGS. 1A-1C illustrate a group quantization-based neural network operation, in accordance with various embodiments. For the purpose of illustration, the neural network operation shown in FIGS. 1A-1C is a convolution. In other embodiments, the neural network operation may be a different type of neural network operation, such as matrix multiplication (MatMul) operation. FIG. 1A shows input tensors of the group convolution, which include anactivation tensor 110 and weight tensors 120 (individually referred to as "weight tensor 120"). The result of the group convolution is an output tensor 130 shown in FIG. 1C.

[0063] The activation tensor 110 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 110 may be an input to the DNN, such as an image. In the embodiments of FIG. 1A, the activation tensor 110 includes activations (also referred to as "input activations") arranged in a 3D matrix. An activation is a data point in the activation tensor 110. The activation tensor 110 has a spatial size HinX WinX Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), Winis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 2D matrix of each input channel), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 110 has a spatial size of 7x7x16, i.e., the activation tensor 110 includes 16 input channels and each input channel has a 7x7 2D matrix. Each activation in the activation tensor 110 may be represented by a (x, y, z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 110 may be different.

[0064] Each weight tensor 120 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A weight tensor 120 has a spatial size Hf x Wf x Cf, where Hf is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), Wf is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, equals Cin. For purpose of simplicity and illustration, each weight tensor 120 in FIG. 1A has a spatial size of 3x3x16, i.e., the weight tensor 120 includes 16 convolutional kernels with a spatial size of 3x3. In other embodiments, the height, width, or depth of the weight tensor 120 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the activation tensor 110.

[0065] The convolution is converted to a plurality of group convolutions and an elementwise addition. Each group convolution may be performed on a portion of the activation tensor 110 and a portion of each weight tensor 120. As shown in FIG. IB, the activation tensor 110 and each weight tensor 120 are divided along the input channel dimension. The activation tensor 110 is partitioned into two channel groups: a channel group 115A and a channel group 115B. The channel group 115A and channel group 115B are each a subtensor of the activation tensor 110. Each of the channel groups has a spatial size of 7x7x8. Each weight tensor is partitioned into two channel groups: a channel group 125A and a channel group 125B. The channel group 125A and channel group 125B are each a subtensor of the weight tensor 120. Each of the channel groups has a spatial size of 3x3x8. The group convolution may include a convolution on the channel group 115A and the channel group 125A from the four weight tensors 120 and another convolution on the channel group 115B and the channel group 125B from the four weight tensors 120.

[0066] The first convolution produces a tensor 135A shown in FIG. 1C. During the first convolution, the channel group 125A from each weight tensor 120 slides across the channel group 115A and generates a 2D matrix for an output channel in the tensor 135A. In the embodiments of FIGS. 1A-1C, the 2D matrix has a spatial size of 5x5. As there are four weight tensors 120, the first convolution produces a 3D matrix (i.e., the tensor 135A) whose spatial size is 5x5x4. Similarly, the second convolution produces a tensor 135B shown in FIG. 1C. The spatial size of the tensor 135B is also 5x5x4. An accumulation operation is then performed on the tensor 135A and the tensor 135B to produce the output tensor 130 of the group convolution. The accumulation operation may be an elementwise addition, in which each activation in the tensor 135A is added with a corresponding activation in the tensor 135B. The position (e.g., the (x, y, z) coordinate) of the activation in the tensor 135A may be the same as the position of the activation in the tensor 135B. The result of the addition is an activation in the output tensor 130.

[0067] The spatial size of the output tensor 130 is the same as the spatial size of the tensor 135A or tensor 135B. In FIG. 1C, the spatial size of the output tensor 130 is represented as HoutxoutxCOut> where Houtis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel), Woutis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each outputchannel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of weight tensors 120 in the group convolution. The output tensor 130 may be used as an input tensor for the next neural network operation in the DNN.

[0068] In some embodiments, activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number. In some embodiments, the convolution may be executed as a group quantization-based convolution to save data storage and computational resources.

[0069] FIG. 2 illustrates group quantization, in accordance with various embodiments. FIG. 2 shows a tensor 200, which may be an example of the activation tensor 110 or weight tensor 120 in FIGS. 1A-1C. The spatial shape of the tensor 200 is H x W x C. The tensor 200 is divided into three subtensors (subtensor 210, subtensor 220, and subtensor 230) along the channel dimension. The spatial shape of the subtensor 210 is H X W X Cgl. The spatial shape of the subtensor 220 is H X W X Cg2. The spatial shape of the subtensor 210 is H x W x Cg3. In some embodiments, Cgl= Cg2= Cg3. The subtensor 210, subtensor 220, and subtensor 230 are quantized using different quantization parameters, such as different scales or different zero points. The data elements in the subtensor 210, subtensor 220, and subtensor 230 may have the same data type or data format before the quantization. The data type may be a floating-point data type, such as FP16, BF16, FP32, or another floatingpoint data type. After the quantization, the data precisions of the subtensor 210, subtensor 220, and subtensor 230 would be different. The subtensor 210, subtensor 220, and subtensor 230 may have different data types / formats. In an example, the data elements in the subtensor 210 may be INT5 values, the data elements in the subtensor 220 may be INT8 values, while the data elements in the subtensor 230 may be INT9 values.

[0070] Group-wise quantization can compress and accelerate the inference of DNNs by reducing the precision of the model weights. Unlike uniform quantization which applies the same quantization scheme to all weights, group-wise quantization divides the weights into groups and quantizes each group independently. Group-wise quantization can exploit the structure and redundancy in weights to achieve higher compression ratios than uniform quantization. Taking LLMs for example, LLMs typically have billions of parameters, many of which are highly correlated or have similar magnitudes. By grouping weights with similar characteristics together, group-wise quantization can use fewer bits to represent each group while minimizing the loss in model accuracy.

[0071] There are several variants of group-wise quantization, including channel-wise quantization (weights are grouped by the output channel dimension, allowing different quantization parameters for each output channel), block-wise quantization (weights are grouped into small blocks, enabling more fine-grained quantization), attention-aware quantization (weights in the attention layers are quantized separately from the weights in the feed-forward layers, as attention layers are more sensitive to quantization error), and so on. The group-wise quantization in FIG. 2 is channel-wise quantization. Example of channelwise quantization includes vector scaled quantization (VSQ) and generative pre-trained transformer quantization (GPTQ). The grouping may be done in both activations and weights and along the input channel dimension. VSQgroups may or may not have their own precision type but they have their unique scale and bias. For GPTQ, the group-based quantization may be enacted on the weight side. GPTQ-based weight groups may also have different precisions and scale and bias values. For different precision weight groups, the smaller precision weights may be scaled to the activation side precision, e.g., when the activation side precision is the largest precision type. It would be important to feed the channels for activations and weights in a group-wise fashion along with their corresponding activations and weights.

[0072] The activation or weight precision requirements, grouping dimension (e.g., Cgl, Cg2, and Cg3.), and grouping granularity may be determined based on the hardware configuration so that the hardware can execute the group quantization-based neural network operations, such as convolutions, MatMul operations, etc. The scale and bias of each group should be intrinsically supported by the underlying NPU architecture. In embodiments where groups are created along the channel dimensions (e.g., theembodiments of FIG. 2), the group quantization-based neural network operation can be mapped onto the NPU by splitting the channel groups into multiple workloads of the same HW and C dimension with each group having its own scale and bias values. This may require the ability to drain (or / and load) partial sums and sum them up externally using a compute component, such as DPU, DSP, etc.

[0073] Group-wise quantization can achieve significant compression ratios, often reaching 4-8 bits per weight, while maintaining the performance (in terms of accuracy) of the original DNN on various benchmarks. This makes group-wise quantization an essential technique for deploying DNNs (e.g., LLMs) on resource-constrained devices for accelerating inference on GPUs and NPUs.

[0074] FIG. 3 is a block diagram of an Al system 300, in accordance with various embodiments. The whole Al system 300 or a part of the Al system 300 may be implemented in one or more computing devices, such as the computing device 1200 in FIG. 12. The Al system 300 can generate and execute DNNs. As shown in FIG. 3, the Al system 300 includes a DNN module 301 and an NPU 302. In other embodiments, alternative configurations, different or additional components may be included in the Al system 300. For instance, the Al system 300 may include multiple DNN modules or multiple NPUs. Further, functionality attributed to a component of the Al system 300 may be accomplished by a different component included in the Al system 300 or a different system. In some embodiments, the Al system 300 may include different types of processing units. In an example, the DNN module 301 may be implemented by one or more central processing units (CPUs). The NPU 302 may also be referred to as an Al processor or accelerator. The DNN module 301 and NPU 302 may be implemented in the same chip or separate chips. Even though not shown in FIG. 3, the Al system 300 may include one or more graphics processing units.

[0075] The DNN module 301 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 301 may generate and train DNNs. For instance, the DNN module 301 can define the layered architecture of a DNN. The DNN module 301 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 301 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN. The DNN module 301 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 301 may pruneweights in one or more layers of a DNN by changing nonzero weight to zeros. The DNN module 301 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero weights to the total number of weights. In an example where the DNN module 301 prunes weight during DNN training, the DNN module 301 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 301 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 301 may allow the pruned weights to change values so that a pruned, zero weight may have a nonzero value after further training. The DNN module 301 may prune weights of the layer again after one or more additional epochs.

[0076] The DNN module 301 may deploy trained, compressed, or validated DNNs for use in neural network applications. In some embodiments, the DNN module 301 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 301 may facilitate deployment of the DNNs using the NPU 302. For instance, the DNN module 301 may receive data from a device or system coupled with the Al system 300 and input the received data (or data generated by the DNN module 301, e.g., based on the received data) into a DNN. The DNN module 301 may generate instructions (e.g., computer program instructions) that can be executed by the NPU 302 for DNN execution. The DNN module 301 may receive an output of the DNN from the NPU 302. The DNN module 301 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 301) to the device or system. In some embodiments, the DNN module 301 may control execution processes of trained, compressed, or validated DNNs. The DNN module 301 may function as a compiler for DNNs executed by the NPU 302. The DNN module 301 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0077] The DNN module 301 may facilitate executions of group quantization-based neural network operations. In some embodiments, the DNN module 301 performs group quantization on input tensors of DNN layers. The DNN module 301 may determine different quantization parameters (e.g., scales or zero points) for different groups within an input tensor. In some embodiments, the DNN module 301 may also interleave groups of an input tensor by partitioning each group into subgroups and rearranging the subgroups of theinput tensor. In some embodiments, the DNN module 301 may also dequantize weights that have been quantized before the DNN layer is executed. The DNN module 301 may generate configuration parameters for configuring one or more components of the NPU 302 for DNN executions. For instance, the DNN module 301 may generate configuration signals for multiplexers (MUXs). The DNN module 301 may also configure LUTs in the NPU 302. Certain aspects of the DNN module 301 are provided below in conjunction with FIG. 4.

[0078] The NPU 302 executes DNNs, including DNNs provided by the DNN module 301. For instance, the NPU 302 can execute a DNN by carrying out neural network operations in the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or performing the neural network operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. The NPU 302 may execute group quantization-based neural network operations. For example, the NPU 302 may perform group convolutions and an elementwise addition to execute a convolutional layer. As another example, the NPU 302 may perform group MatMuls and addition operations to execute a MatMul layer. Computations for executing a group quantization-based neural network operation may be assigned to various components of the NPU 302.

[0079] As shown in FIG. 3, the NPU 302 includes a memory 310, a direct memory access (DMA) engine 320, and compute blocks 330 (individually referred to as "compute block 330"). In other embodiments, alternative configurations, different or additional components may be included in the NPU 302. For example, the NPU 302 may include more than one memory 310 or DMA engine 320. As another example, the NPU 302 may include a single compute block 330. Further, functionality attributed to a component of the NPU 302 may be accomplished by a different component included in the NPU 302 or by a different system. A component of the NPU 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0080] The memory 310 stores data associated with neural network operations performed by the NPU 302. In some embodiments, the memory 310 may store data to be used by the compute blocks 330 for executing neural network operations. The memory 310 may store inputs to DNNs and outputs of DNNs. The memory 310 may also store activations (such as input activations and output activations of neural network operations) and weights (such as weights determined by training DNNs) in DNNs. In some embodiments, the memory 310may store activations and weights with floating-point precisions, such as FP4, SF4, NF4, and so on. The memory 310 may also quantized activations or weights. The memory 310 may store quantized channel groups and quantization parameters for the channel groups. In some embodiments, the memory 310 may store configuration parameters generated by the DNN module 301. The memory 310 includes one or more dynamic random-access memories (DRAMs).

[0081] The DMA engine 320 facilitates data transfer between the memory 310 and the compute blocks 330. For example, the DMA engine 320 can read data from the memory 310 and write data into a local memory of a compute block 330. As another example, the DMA engine 320 can read data from a local memory of a compute block 330 and write data into the memory 310. For instance, the DMA engine 320 may read input activations and weights of convolution from the memory 310 and load the input activations and weights to one or more compute blocks 330. The DMA engine 320 may also write output activations of convolutions computed by one or more compute blocks 330 to the memory 310. The DMA engine 320 provides a DMA feature that allows the compute block 330 to initiate data transfer between the memory 310 and the local memories of the compute blocks 330 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 320 may read tensors from the memory 310, modify the tensors in a way that is optimized for the compute block 330 before it writes the tensors into the local memories of the compute blocks 330.

[0082] The compute blocks 330 perform neural network operations in DNNs. For instance, a compute block 330 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A compute block 330 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple compute blocks 330 in parallel. For instance, multiple compute blocks 330 may each perform a portion of a workload for a neural network operation. Data may be shared between the compute blocks 330. A compute block 330 may also be referred to as a compute tile. The compute blocks 330 may be capable of running various types of neural network operations, such as convolution, matrix multiplication, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the compute blocks 330 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, thecompute block 330 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the compute block 330 or another compute block 330.

[0083] In the embodiments of FIG. 3, each compute block 330 includes a local memory 340, a DSP 350, and a DPU 355. The DPU 355 includes an IDU 360, a processing engine 370, a post-processing engine 380, and an ODU 390. Some or all the components of the compute block 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute block 330. Further, functionality attributed to a component of the compute block 330 may be accomplished by a different component included in the compute block 330, a different compute block 330, another component of the NPU 302, or a different system. A component of the compute block 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0084] The local memory 340 is local to the corresponding compute block 330. The local memory 340 is accessible to both the DSP 350 and DPU 355. In the embodiments of FIG. 3, the local memory 340 is inside the compute block 330. In other embodiments, the local memory 340 may be outside the compute block 330. Data in the local memory 340 may be transferred to or from the memory 310, e.g., through the DMA engine 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory of another compute block 330. The local memory 340 may store data received, used, or generated by the IDU 360, the processing engine 370, the post-processing engine 380, or the ODU 390. Examples of the data may include input activations, weights, output activations, configuration parameters, and so on.

[0085] In some embodiments, the local memory 340 includes one or more static randomaccess memories (SRAMs). The local memory 340 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 340 may include memory banks. The number of data banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and datalarger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0086] The DSP 350 performs computations in DNN layers, including computations in group quantization-based neural network operations. In some embodiments, the DSP 350 can perform generic computations such as addition, subtraction, multiplication, division, logical, bitwise operations, and other nonlinear computations (in terms of table look-up or polynomial approximation). The DSP 350 may be a very long instruction word (VLIW) processor. In some embodiments, the DSP 350 may have an architecture optimized for the operational needs of digital signal processing.

[0087] In some embodiments, the DSP 350 may perform some computations in a neural network operation, while other computations in the neural network operation may be performed by the DPU 355. The DSP 350 may support non-traditional operations or non- MatMul or non-convolution-based operations within DNNs. In some embodiments, the DSP 350 may perform additions of partial sums computed by the DPU 355. The DSP 350 may also generate weight tables for group quantization-based neural network operations. A weight table is a data structure that specifies the register setting of workloads, such as workloads performed by the DPU 355. The storage size of a weight table may equal the number of output channels times 16 bytes. In some embodiments, the DSP 350 may fill a weight table for each group operation. The total storage size of weight tables for a DNN layer may equal the number of groups times the number of output channels times 16 bytes. The DSP 350 may read data from the local memory 340 or write data into the local memory 340.Compared with currently available approaches that have the DMA engine 320 generate weight tables, the approach of having the DSP 350 generate weight tables can save memory traffic from the memory 310 to the local memory 340. This advantage can be significant for transformer inference.

[0088] In some embodiments, the DSP 350 may operate in accordance with a clock signal. For instance, the timing when the DSP 350 can execute instructions may be synchronized with the clock signal. In some embodiments, the DSP 350 may be pipelined along with theDMA engine 320 or the DPU 355, thereby enabling parallel computations to improve overall performance. The DSP 350 may be implemented on a microprocessor chip, which may be separate from a chip implementing the DPU 355. In some embodiments, the DSP 350 may be a Streaming Hybrid Architecture Vector Engine (SHAVE) processor. Even though FIG. 3 shows a single DSP, the compute block 330 may include multiple DSPs. The DSPs may be arranged in an array.

[0089] The IDU 360 loads data from the local memory 340 to the processing engine 370 or to the post-processing engine 380. The IDU 360 may read tensors from the local memory 340. The tensors may include activation tensors, weights tensor, and so on. The IDU 360 may perform group-wise loading of activations or weights. In some embodiments, the IDU 360 may read data from the local memory 340 and write the data into storage units in the processing engine 370. For instance, the IDU 360 may load activations into activation register files in the processing engine 370 and load weights into weight register files in the processing engine 370. The IDU 360 may have an activation reader for loading activations and a weight reader for loading weights. In some embodiments, the IDU 360 may read configuration parameters from the local memory 340 and load the configuration parameters into configuration registers or other configurable components (e.g., LUTs) of the processing engine 370 or post-processing engine 380. The IDU may also load group-specific scales and zero points into the post-processing engine 380. The scales and zero points may be stored in one or more LUTs in the IDU 360.

[0090] The processing engine 370 performs operations in DNNs. For instance, the processing engine 370 may perform group operations in a DNN layer. The group operations may be group convolutions or group MatMuls. In some embodiments, the processing engine 370 may also perform other types of operations in the DNN layer, such as additions. The processing engine 370 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 370. Each processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 370 may constitute a bigger array that includes more rows and columns. An example PE may be or may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the compute block 330 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operandand a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN or compressing the neural network operation after training. The weights in the weight operand may be in different input channels. In some embodiments, the activation operand or weight operand is a vector along the IC dimension.

[0091] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. An MAC unit may also include one or more shifters to facilitate mixed-precision computations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the IDU 360, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0092] The post-processing engine 380 processes outputs of the processing engine 370. The post-processing engine 380 may include one or more PPEs. In some embodiments, the PPEs in the post-processing engine 380 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 380 may apply group-specific scales and bias on data computed by the processing engine 370. The post-processing engine 380 may include one or more LUTs programmed with group-specific dequantization parameters. For instance, an entry of a LUT may store the scale and zero point for a channel group of a DNN layer. The post-processing engine 380 may use the zero point as a bias. The dequantization parameter may be the same as or may be determined from the quantization parameters used for quantizing the channel group. In some embodiments, each PPE in the postprocessing engine 380 may have its own LUT. In some embodiments, the dequantized values may be drained by the ODU 390 into the local memory 340. In other embodiments,the post-processing engine 380 may also accumulate the dequantized values to produce output data elements of the DNN layer. The output data elements may be drained by the ODU 390 into the local memory 340.

[0093] In some embodiments, the post-processing engine 380 computes activation functions. The post-processing engine 380 may receive outputs of the processing engine 370 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 380 may perform other types of post processing on outputs of the processing engine 370. For instance, the post-processing engine 380 may apply a bias on an output of the processing engine 370. In some embodiments, the post-processing engine 380 may be bypassed for certain neural network operations.

[0094] The ODU 390 drains data from the processing engine 370 or from the postprocessing engine 380, e.g., from register files in the processing engine 370 or from the post-processing engine 380. The drain module may write the data to the local memory 340. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the ODU 390 may drain data on a cell level. For each processing cell, the ODU 390 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the ODU 390 may use a sequence of cycles to drain data from a processing cell. The ODU 390 may drain the output of some of the PEs in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the IDU 360.

[0095] In some embodiments, the ODU 390 includes sparsity encoding logic that can convert outputs of the processing engine 370 from a dense format to a sparse format. For instance, the ODU 390 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros from data computed by the processing engine 370. The sparsity encoder may also generate sparsity maps that represent sparsity in the dense data.

[0096] In some embodiments, the data drained from the processing engine 370 are results of group operations in a DNN layer. The results may be partial sums, such as data points in the tensor 135A or tensor 135B in FIG. 1C. In other embodiments, the data drained from the processing engine 370 may be output data elements of a DNN layer, such as data points in the output tensor 130 in FIG. 1C. The sparsity encoder may generate a compressed versionof the output tensor. The sparsity encoder may identify every zero activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity maps for the output tensor. A sparsity map may indicate sparsity in at least part of the output tensor. The sparsity map may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0097] The ODU 390 may write the compressed activation tensor and the one or more sparsity maps into the local memory 340. The sparse activation tensor and the one or more sparsity maps may be further loaded to the memory 310, e.g., through the DMA engine 320. Additionally or alternatively, the sparse activation tensor and the one or more sparsity maps may be loaded by the IDU 360 to the processing engine 370 for further computation, e.g., for performing a deep learning operation in the next layer.

[0098] FIG. 4 is a block diagram of a DNN module 400, in accordance with various embodiments. The DNN module 400 may be an example of the DNN module 301 in FIG. 3. As shown in FIG. 4, the DNN module 400 includes an interface module 410, a training module 420, a compiler 430, and a datastore 440. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 400. Further, functionality attributed to a component of the DNN module 400 may be accomplished by a different component included in the DNN module 400 or a different module or system.

[0099] The interface module 410 facilitates communications of the DNN module 400 with other modules or systems. For example, the interface module 410 establishes communications between the DNN module 400 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. The interface module 410 may also facilitate transmission of data generated by the training module 420 or compiler 430 to the NPU (e.g., the NPU 302 in FIG. 3) for executing DNNs. As another example, the interface module 410 may distribute trained DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0100] The training module 420 trains DNNs by using a training dataset. The training module 420 forms the training dataset. In an example where the training module 420 trains an DNN to recognize objects in images, the training dataset includes training images andtraining labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the training module 420 to validate performance of a trained DNN. The data portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0101] The training module 420 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0102] The training module 420 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers pooling layers, fully-connected layers, MatMul layers, linear layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a featuremap that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0103] In the process of defining the architecture of the DNN, the training module 420 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0104] After the training module 420 defines the architecture of the DNN, the training module 420 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 420 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 420 uses a cost function to minimize the error.

[0105] The training module 420 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 420 finishes the predetermined number of epochs, the training module 420 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0106] The training module 420 may also verify accuracy of DNNs after training. In some embodiments, the training module 420 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the training module 420 maydetermine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The training module 420 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0107] The training module 420 may compare the accuracy score with a threshold score. In an example where the training module 420 determines that the accuracy score of the DNN is less than the threshold score, the training module 420 instructs the training module 420 to re-train the DNN. In one embodiment, the training module 420 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0108] The compiler 430 compiles information of DNNs to executable instructions that can be executed, e.g., by the NPU 302, to carry out neural network operations in DNNs. In some embodiments, the compiler 405 may generate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler 430 may use the graph to generate executable DNNs. For instance, the compiler may generate computer program instructions that are executable by the NPU 302. The computer program instructions may cause the NPU 302 to perform neural network operations in the DNNs.

[0109] The compiler 430 may facilitate mapping group quantization-based neural network operations onto the NPU. The compiler 430 may process input data of a DNN layer to make the input data ready for the NPU to perform group operations and accumulations in the DNN layer. In some embodiments, the compiler 430 may partition an input tensor (e.g.,activation tensor or weight tensor) of a DNN layer along the input channel dimension. Each portion of the input tensor is a channel group. The compiler 430 may quantize the channel groups separately by using group-specific quantization parameters, such as group-specific scales or zero points. After the quantization, the channel groups may have different precisions and data formats. In an example, the compiler 430 may change a FP32 weight tensor into channel groups in INT5, INT8, and INT9. The compiler 430 may store the quantized channel groups in a memory, e.g., the datastore 440 or the memory 310. The compiler 430 may store quantized channel groups as microscaling blocks in a microscaling format. Certain aspects about microscaling format are described below in conjunction with FIG. 5. In some embodiments, the compiler 430 may upscale the channel groups before the DNN layer execution starts. For instance, the compiler 430 may change the different data precisions of the channel groups within a weight tensor to the same data precision. The new data precision is a precision supported by the NPU or the precision of the activations. In the above example, the compiler 430 may change the precision of the weights from INT5, INT8, and INT9 to FP16. The compiler 430 may perform the dequantization on activations too.

[0110] In some embodiments, the compiler 430 may interleave groups within an input tensor of a DNN layer. For instance, the compiler 430 may divide each channel group into subgroups. The subgroups within the same channel group may have the same data precision and data type, i.e., the data precision / data type of the channel group. The compiler 430 may rearrange the subgroups within the input tensor so that the positions of some subgroups within the input tensor are different. In an example, the compiler 430 may place the first subgroup of each channel group into a set, within which the subgroups are arranged in the order following the order of their channel groups, meaning that the first subgroup of the first channel group is before the first subgroup of the second channel group, the first subgroup of the second channel group is before the first subgroup of the third channel group (if any), and so on. This set may be followed by another set that includes the second subgroup of each channel group. The compiler 430 may continue to interleave the channel groups in this manner till all the channels are included. The interleaving may result in a new tensor, which has the same spatial shape and the same channel as the input tensor but the order of the channels is different from the original input tensor. The interleaving can enable a DPU (e.g., the DPU 355 in FIG. 3) to perform both the group operations and additions in the DNN layer. Certain aspects about interleaving and performing group quantization-basedneural network operations with interleaved groups are described below in conjunction with FIGS. 13-18.

[0111] The compiler 430 may also generate executable instructions that can configure components of the NPU to perform group operations and accumulations in the DNN layer. For instance, the compiler 430 may generate configuration parameters to be used to program configurable components of the NPU. In some embodiments, the compiler 430 may generate weight tables for the group operations. The compiler 430 may generate a separate weight table for each group operation. Weight table may be a data structure to specify the register setting of workloads. The weight able for a group operation (e.g., a group convolution or group MatMul) may specific register settings for the workload of performing the group operation. The compiler 430 may generate the weight tables offline, e.g., before the execution of the DNN layer starts. In other embodiments, the compiler 430 may instruct the NPU (e.g., the DSP 350 in the NPU) to dynamically generate weight tables during the execution of the DNN layer.

[0112] In some embodiments, the compiler 430 may generate configuration parameters for LUTs in the NPU. For example, the compiler 430 may generate configuration parameters indicating the group-specific quantization parameters or dequantization parameters for the group operations of the DNN layer. These configuration parameters may be stored as entries in the corresponding LUT(s), and the NPU may use the LUT(s) to perform quantization or dequantization. As another example, the compiler 430 may generate configuration parameters for programming one or more N:M structured sparsity-based LUTs. A structured sparsity pattern may be represented by a structured sparsity ratio N:M, which represents that for every M consecutive data elements, N data elements are nonzero. N and M may be integer numbers. Examples of structured sparsity ratios may include 1:4, 2:4, 3:4, 4:8, and so on. A N:M structured sparsity-based LUT may be configured for a structured sparsity pattern and map sparsity maps of the structured sparsity pattern to read port indices ("read indices"). A read index may represent a read port. A sparsity logic in the NPU may determine one or more read indices using the LUT and determine memory addresses from the read indices and read the activations or weights stored at the memory address.

[0113] As yet another example, the compiler 430 may generate configuration signals to configure MUXs in the NPU. For instance, the compiler 430 may generate a configurationsignal that indicates a type of operation to be performed by a component of the NPU. The configuration signal may indicate that the operation is a group convolution, group MatMul, elementwise addition, and so on. The configuration signal may be provided to one or more MUXs in the component, and the one or more MUXs may use the configuration signal to select a signal from multiple input signals so that the component of the NPU may use the right data to perform the type of operation.

[0114] The datastore 440 stores data received, generated, used, or otherwise associated with the DNN module 400. For example, the datastore 440 stores the datasets used by the training module 420. The datastore 440 may also store data generated by the training module 420, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), and so on. The datastore 440 may store quantized activations, quantized weights, dequantized activations, dequantized weights, group-specific parameters, configuration parameters, or other data generated by the compiler 430. The datastore 440 may include one or more memories. In the embodiment of FIG. 4, the datastore 440 is a component of the DNN module 400. In other embodiments, the datastore 440 may be external to the DNN module 400 and communicate with the DNN module 400 through a network. In some embodiments, data stored in the datastore 440 may be transferred to an NPU memory, such as the memory 310.

[0115] FIG. 5 illustrates an example microscaling block 500, in accordance with various embodiments. The microscaling block 500 has a shared scale 510 and scalar elements 520 (individually referred to as "scalar element 520"). The shared scale 510 is shared among the scalar elements 520. The microscaling block 500 encodes a vector of k numbers (i.e., the scalar elements 520), each with value XPi, where X denotes the value of the shared scale 510, and i denotes the index of the number and is a number falling in the range from 1 to k. The k real numbers represented by the microscaling block 500 may be denoted as {v ^=1. In some embodiments, vt= XPt. In some embodiments, when X = NAN, vt= NAN for all i. In some embodiments, when |XP > vmaxFloat32(vmaxFloat32refers to the largest representable magnitude in the data format Float32), vtmay be implementation defined.

[0116] The microscaling block 500 may have a microscaling format, such as MXFP8, MXFP6, MXFP4, MXINT8, and so on. The microscaling block 500 may have one or more parameters defining the microscaling format. These parameters may be the meta data of the microscaling block 500. Examples of the parameters include a block size, scale data format,number of scale bits, element data format, element bit-width, and so on. The block size may indicate the number of scalar elements in the microscaling block 500. In the example shown in FIG. 5, the block size of the microscaling block 500 is k. The scale data format may be the data format of the shared scale 510. The number of scale bits may be the number of bits encoding the value of the shared scale 510. In an example, the scale data format may be E8M0 (an 8-bit exponent), and the number of scale bits may be 8. The element data format may indicate the data format of the scalar elements 520. Examples of the element data format include FP4, FP6, FP8, INT8, and so on. The element bit-width may indicate the number of bits needed for encoding a scalar element 520 and may correspond to the element data format. For instance, when the element data format is FP8 or INT8, the element bit-width is 8.

[0117] The microscaling block 500 is a block in a microscaling data format. Microscaling format is a hardware-specific variant of the block / group quantization technique where there are scale factors per microblock. Each microblock may be a group with its own scale factor. In some embodiments, a microscaling format may have two levels of scale factors that can be applied after every macroblock and microblock granularity. For instance, a tensor may be partitioned into macroblocks, and a macroblock may be partitioned into microblocks. The macro- and microblocks may have a block granularity that is the same as that of group size. In some emboidments, the microscaling block 500 is a quantized microblock, such as quantized channel group generated by quantizing a channel group within an input tensor of a neural network operation.

[0118] Block-wise quantization can assign scaling factors to individual blocks to mitigate outlier effects. Scaling factors may be determined through software-supported or hardware- supported methods. Software approaches may offer high precision but may require larger block sizes, whereas hardware techniques may enable finer-grained blocks using efficient scaling factors like power-of-two shared exponents. Microscaling formats can leverage multi-level scaling with dedicated hardware and support even low-precision formats like FP4 and FP6.

[0119] In some embodiments, microscaling blocks like the microscaling block 500 may be generated by a compiler, such as the compiler 430 in FIG. 4. A microscaling block may represent a quantized channel group of a DNN layer. In an example of generating a microscaling block from a channel group within a weight tensor, the compiler may identifythe maximum exponent of weights in the channel group. The compiler may map the maximum exponent to the largest binade in the scale data format. The compiler may then divide the weights in the channel group with the shared scale and determine whether any of the values exceed a representable range of the scale data format. When a value exceeds the representable range, the compiler may change the value to the maximum representable value of the scale data format. The maximum representable value may be stored as a scalar element of the microscaling block. When a value falls within the representable range, the compiler may store the value itself as a scalar element of the microscaling block. In some embodiments, the compiler may select a dimension of the weight tensor as a dimension for sharing the shared scale. For instance, the compiler may select the input channel dimension for sharing the shared scale.

[0120] FIG. 6 illustrates a workflow 600 in a DPU with a 32-bit load path, in accordance with various embodiments. The workflow 600 may be a group quantization-based workflow. The workflow 600 indicates a path to map group quantization layers onto the DPU. The DPU performing the workflow 600 may be an example of the DPU 355 in FIG. 3.

[0121] As shown in FIG. 6, the workflow includes four steps: Step 610, Step 620, Step 630, and Step 640. In Step 610, a group is loaded, for instance, from a memory in the DPU to a processing engine in the DPU. The group may be a group of quantized weights or activations. The group may be a quantized channel group with a group-specific scale and bias. Step 610 may include loading the group along with the scale and bias. The group may be loaded into the processing engine in the DPU, and the scale and bias may be loaded into the post-processing engine in the DPU. In some embodiments, the group size (e.g., the number of input channels in the group) may be determined by a configuration of the processing engine to maximize utilization of the Pes in the processing engine. In Step 620, a group operation is performed on the loaded group. The group operation may be part of a neural network operation, such as a convolution, MatMul operation, etc. The neural network operation may be converted to group operations and an elementwise addition. The group operation in Step 620 may produce partial sums. A partial sum may be the result of one or more MAC operations performed by the processing engine. The partial sums may have the same data format as the loaded group.

[0122] In Step 630, the scale and bias are applied on the partial sums to produce 32-bit partial sums. Step 630 may be dequantization performed by the post-processing engine inthe DPU. The 32-bit partial sums may have a floating-point data format, such as FP32. The 32-bit partial sums are drained in Step 630. For instance, the 32-bit partial sums are written into the memory in the DPU. In Step 640, the 32-bit partial sums are loaded back through the 32-bit load path. The 32-bit load path may be a datapath for loading 32-bit values. The 32-bit partial sums are loaded back from the memory into the processing engine. In some embodiments, the 32-bit partial sums may be loaded into one or more output register files. The 32-bit partial sums may be used as seed before starting the next group operation with the next group. In Step 650, 32-bit accumulations are performed. For instance, the processing engine may accumulate a 32-bit partial sum computed in Step 630 with a 32-bit partial sum computed from another group. The result of the 32-bit accumulation may be an output activation of the neural network operation. The 32-bit accumulation may be part of the elementwise addition.

[0123] Another round of the workflow 600 may be performed by the DPU to process the next group of the neural network operation. In some embodiments, Step 610 or Step 620 for the next group may start before Step 650 for the group is complete. For instance, the next group may be loaded, or the group operation may be performed on the next group, after Step 640 for the group is complete. The next group may be loaded along with new scale and bias. In some embodiments, the original scale and bias are modified to be applicable for subsequent rounds of the workflow 600. They may be cumulatively scaled with the previous scale and bias factors so that the final convolution result is the same as applying the individual partial sums by their corresponding scale and bias. This may be done for performing an in-place accumulation.

[0124] The workflow 600 may require the NPU to drain and load 32b partial sums. Many NPUs are integrated with the ability to drain 32-bit values as they frequently support FP32 datatype for desirable accuracy implementation. This logic can be reused to drain 32-bit partial sums. The NPU can also load 32-bit values into the existing accumulators within the processing engine. These values may be subsequently used as seeds for the next group convolution. This 32-bit load may be triggered using an existing layer computation such as a single operand-based elementwise layer operation. In some embodiments, the groups are contiguous in the input channel dimension. The total layer may be split into multiple smaller workloads due to the size of the partial sums (32-bit each) and the limited memory capacity.

[0125] FIG. 7 illustrates a 32-bit load path in a DPU, in accordance with various embodiments. The DPU in the embodiments of FIG. 7 may be an example of the DPU in the embodiments of FIG. 6. FIG. 7 shows multipliers 710A-710H (collectively referred to as "multipliers 710" or "multiplier 710"), MUXs 720A-720D (collectively referred to as "MUXs 720" or "MUX 720"), a concatenator 730, another multiplier 740, an adder 750, a MUX 760, and an accumulator 770. For the purpose of illustration and simplicity, FIG. 7 does not show all the components of the DPU 700.

[0126] The multipliers 710 may each be in a separate PE. In some embodiments, the multipliers 710A-710D may be in a PE column, and the multipliers 710E-710H may be in another PE column. For MAC operations, a multiplier 710 may receive two values, such as an activation and a weight, at a time. These two values may be quantized values loaded from a memory. The multiplier 710 computes a product of the activation and weight. The concatenator 730 may accumulate the product with one or more other products computed by one or more other multiplier 710 to produce a partial sum. The partial sum may be dequantized by the multiplier 740 and adder 750. For instance, the multiplier 740 may multiply the partial sum with a scale factor, the result of which may be accumulated with a zero point by the adder 750. In some embodiments, the multiplier 740 and adder 750 may be in a PPE of the DPU.

[0127] The dequantized partial sum may be a 32-bit partial sum. The 32-bit partial sum may be drained into the memory, then loaded back. The 32-bit partial sum may be loaded back through the activation load path. The 32-bit partial sum may be provided to the accumulator 770 through the MUX 760. The accumulator 770 may accumulate the 32-bit partial sum with one or more other 32-bit partial sums to produce an output data element of a neural network operation. The MUXs 720 and the MUX 760 may operate under configuration signals to provide data to the concatenator 730 and the accumulator 770, respectively. Certain aspects of the MUXs 720 and the MUX 760 are described below in conjunction with FIG. 8.

[0128] FIG. 8 illustrates a data loading logic, in accordance with various embodiments. The data loading logic is in an IDU of a DPU, e.g., the IDU 360 in FIG. 3. For the purpose of illustration, FIG. 8 shows loading data into four columns and four rows of a PE array. The loaded data include activations, represented as "act_data" in FIG. 8. "wt_sp" in FIG. 8 represents weight sparsity. The data loading logic may be an 8-bit / 16-bit elementwise loadunit logic. In some embodiments, the PE array has 16 rows and 16 columns. In some embodiments, instead of splitting the 16 bytes across 16 PEs with a byte / PE (or 2bytes / PE) in each PE of a row, the 16 bytes are split across 4 PEs in each column, (x, y) points, which may be data points having the same (x, y) coordinate, may be loaded within the memory at a 16-byte granularity and may be read in each clock cycle. Since a 32-bit path has 4 bytes and there are 4 sets of 4 bytes in each 16 bytes, 4 PEs are active in each row in each clock cycle. Instead of a moving pattern of single or double Is distributed across both PE and columns, there is a set of 4xls (OxF, OxFO, OxFOO, and OxFOOO) spread across different columns. This new sparsity pattern may be inserted through a mode-based MUX where an "x32 mode enable" selects the new sparsity pattern. When these changes are enacted on the IDU, the elementwise logic within the PE can be used to implement the 32-bit load.

[0129] FIG. 8 shows an 8 / 16-bit elementwise load path 801, which has a MUX 810. The MUX 810 may be an example of the MUXs 720 in FIG. 7. The MUX 810 may receive signals with data of different data types and select a signal with a data type matching the data type indicated by the "Datatype_sel" configuration signal. The 8 / 16-bit elementwise load path 801 may be extended for loading 32-bit values by adding a concatenator (e.g., the concatenator 730 in FIG. 7) and an additional MUX 820, which forms a 32-bit load path 802. The MUX 820 in the 32-bit load path 802 may be an example of the MUX 760 in FIG. 7. The MUX 820 may select 32-bit values based on the "cfg_32b_load" configuration signal and forward the 32-bit values to a 32-bit accumulator (e.g., the accumulator 770 in FIG. 7). The 32-bit load path 802 may be used for efficient implementation of group operations on the NPU as well as efficiently mapping other DNN operators such as fused elementwise addition operations that can be useful in residual networks.

[0130] FIG. 9 illustrates a workflow 900 in a DPU and a DSP, in accordance with various embodiments. The workflow 900 may be a group quantization-based workflow. The workflow 900 includes a DPU workflow 901 and a DSP workflow 902. The workflow 900 may be performed by a NPU including a DPU performing the DPU workflow 901 and a DSP performing the DSP workflow 902. An example of the NPU may be the NPU 302 in FIG. 3.

[0131] The DPU workflow 901 includes four steps: Step 910, Step 920, Step 930, and Step 940. In Step 910, a group is loaded, for instance, from a memory in the DPU to a processing engine in the DPU. The group may be a group of quantized weights or activations. The group may be a quantized channel group with a group-specific scale and bias. Step 910 mayinclude loading the group along with the scale and bias. The group may be loaded into the processing engine in the DPU, and the scale and bias may be loaded into the post-processing engine in the DPU. In some embodiments, the group size (e.g., the number of input channels in the group) may be determined by a configuration of the processing engine to maximize utilization of the Pes in the processing engine. In Step 920, a group operation is performed on the loaded group. The group operation may be part of a neural network operation, such as a convolution, MatMul operation, etc. The neural network operation may be converted to group operations and an elementwise addition. The group operation in Step 920 may produce partial sums. A partial sum may be the result of one or more MAC operations performed by the processing engine. The partial sums may have the same data format as the loaded group. In Step 930, the scale and bias are applied on the partial sums to produce 32-bit partial sums. Step 930 may be dequantization performed by the post-processing engine in the DPU. The 32-bit partial sums may have a floating-point data format, such as FP32. The 32-bit partial sums are drained in Step 930. For instance, the 32-bit partial sums are written into the memory in the DPU. In Step 940, the 32-bit partial sums are stored in the memory. Another round of the DPU workflow 901 may be performed by the DPU to process the next group of the neural network operation.

[0132] In the DSP workflow 902, the 32-bit partial sums are loaded into the DSP from the memory and the DSP computes final points from the 32-bit partial sums. The final points may be output data elements of the neural network operation. The 32-bit partial sums may be data elements of multiple tensors. The DSP may perform elementwise addition on the tensors to produce an output tensor of the neural network operation. The summation of the 32-bit partial sums may occur serially or using a parallel adder tree within the DSP. The final points may be saved back to the memory.

[0133] The workflow 900 indicates a different way from the workflow 600 in FIG. 6 for mapping group quantization-based DNN layers on the NPU. Compared to the workflow 600 in FIG. 6, the workflow 900 does not require additional hardware changes (e.g., the 32-bit load path) to support the group operations as 32-bit partial sums are not loaded back to the DPU. The workflow 900 may have a small negative performance impact.

[0134] FIG. 10 illustrates an NPU 1000 with a DPU 1010 and a DSP 1030, in accordance with various embodiments. The NPU 1000 may perform group quantization-based workflows, such as the workflow 900 in FIG. 9. The NPU 1000 may be an example of the NPU 302 in FIG.3. As shown in FIG. 10, the NPU 1000 also includes a memory 1020. Both the DPU 1010 and DSP 1030 can access the memory 1020, including writing data into the memory 1020 and reading data from the memory 1020. The NPU 1000 may include additional components not shown in FIG. 10.

[0135] The DPU 1010 includes an IDU 1011 with an activation reader 1012 and a weight reader 1013, a processing engine 1014, a post-processing engine 1015, and an ODU 1016. The IDU 1011 may load data from the memory 1020. For instance, the activation reader 1012 may load activations from the memory 1020. The weight reader 1013 may load weights from the memory 1020. The data may be input data of a DNN layer, such as a convolutional layer or MatMul layer. In some embodiments (e.g., embodiments where the DNN layer is executed based on group quantization), the IDU 1011 may load quantized values of a channel group. The IDU 1011 may also load the scale and zero point of the channel group. The IDU 1011 may load the quantized values into the processing engine 1014 and load the scale and zero point into the post-processing engine 1015.

[0136] The processing engine 1014 performs group operations in the DNN layer on the quantized values. The group operations may be group convolutions, group MatMul operations, and so on. The group operations may be performed separately. For instance, the group operations may be performed on different data (e.g., different channel groups) and produce different results. In some embodiments, each group operation may produce a tensor. The tensors produced by different group operations may have the same spatial shape. The processing engine 1014 may include a PE array. For the purpose of illustration, the PE array includes 16 columns and 16 rows. In other examples, the PE array may include a different number of columns or rows. The IDU 1011 may also load data (e.g., scales, zero points, etc.) into the post-processing engine 1015 for dequantizing values computed by the processing engine 1014. The data (e.g., 32-bit partial sums) computed by the postprocessing engine 1015 may be drained by the ODU 1016 and written into the memory 1020.

[0137] The DSP 1030 may read the 32-bit partial sums from the memory 1020 and perform an elementwise addition on the 32-bit partial sums. The output of the DSP 1030 may be an output tensor of a neural network operation.

[0138] In some embodiments, a part of the elementwise addition by the DSP 1030 may be performed in parallel with part of the group operations by the processing engine 1014. Insome embodiments, the elementwise operation starts before all the partial sums for the DNN layer are computed. The pipelining may be done potentially in fine granular batches in an interleaved manner. The parallel performance of the operations may require splitting the computation into multiple workloads that may incur additional overhead. The batching of the partial sum generation and summation may require additional compiler support, which may mostly be a one-time effort.

[0139] FIG. 11 illustrates DPU-DSP pipelining, in accordance with various embodiments. FIG. 11 shows a clock cycle sequence 1110. The clock cycle sequence 1110 may be generated by a clock generator of a NPU including the DPU and DSP, such as the NPU 302 in FIG. 3. The clock cycle sequence 1110 may control timing of operations by the DPU and DSP. FIG. 11 also shows a pipeline 1120 of tasks performed by the DPU and a pipeline 1130 of tasks performed by the DSP. tnin FIG. 11 refers to a task #n. The DPU and the DSP may simultaneously access and operate from a memory in the NPU. An operation by the DPU and an operation by the DSP may happen at the same time. This enables pipelining of operations between the DPU and the DSP, which can lead to higher performance. In the example shown in FIG. 11, ti by the DSP is performed on the output of ti by the DPU, and t2 by the DSP is performed on the output of t2by the DPU. The DSP may perform ti when the DPU performs t2. and t2by the DPU may be two group operations in a DNN layer, and ti and t2by the DSP may be workloads within an elementwise addition in the DNN layer.

[0140] FIG. 12 illustrates DPU-DSP pipelining for a group quantization-based workflow, in accordance with various embodiments. An example of the group quantization-based workflow may be the workflow 900 in FIG. 9. FIG. 12 shows a clock cycle sequence 1210. The clock cycle sequence 1210 may be generated by a clock generator of a NPU including the DPU and DSP, such as the NPU 302 in FIG. 3. The clock cycle sequence 1210 may control timing of operations by the DPU and DSP. FIG. 12 also shows a pipeline 1220 of the DPU and a pipeline 1230 of the DSP. The DPU and the DSP may simultaneously access and operate from a memory in the NPU. The DPU and DSP may perform operation in parallel, resulting in higher performance. In the example shown in FIG. 12, the DPU and DSP are performing a task ti. The DPU may perform the first stage of the task ti. For instance, the DPU performs a group operation. Data computed by the DPU may be provided to the DSP for performing the second stage of the task ti. The DSP starts the second stage of the task ti before the DPU completes the entire first stage of the task ti. As shown in FIG. 12, the DSP starts the secondstage of the task ti one clock cycle after the DPU starts the first stage of the task ti. The DSP may use data that has been computed by the DPU to perform part of the second stage of the task ti. As the DPU computes new data, the DSP may perform new workloads in the second stage of the task ti. This way, the efficiency of the NPU can be improved or even maximized.

[0141] FIG. 13 illustrates interleaving channel groups of a tensor 1300, in accordance with various embodiments. The tensor 1300 may be an input tensor of a DNN layer. In an example, the tensor 1300 is a weight tensor. The tensor 1300 includes two channel groups: channel group 1310 and channel group 1320. The channel group 1310 is partitioned into three subgroups: subgroup 1311, subgroup 1312, and subgroup 1313. The channel group 1320 is partitioned into three subgroups: subgroup 1321, subgroup 1322, and subgroup 1323. The six subgroups are interleaved. As shown in FIG. 13, the six subgroups are arranged in a sequence in which the subgroup 1311 is followed by the subgroup 1321, followed by the subgroup 1312, then the subgroup 1322, further the subgroup 1313, and last the subgroup 1323.

[0142] FIG. 14 illustrates interleaving channel groups of another tensor 1400, in accordance with various embodiments. The tensor 1400 may be an input tensor of a DNN layer. In an example, the tensor 1400 is a weight tensor. The tensor 1400 includes four channel groups: channel group 1410, channel group 1420, channel group 1430, and channel group 1440. The four channel groups are highlighted by different patterns in FIG. 14. Each of the channel groups are partitioned into three subgroups. The 12 subgroups are interleaved and rearranged into four sets. Within each set, a subgroup from the channel group 1410 is followed by a subgroup from the channel group 1420, then a subgroup from the channel group 1430, and last a subgroup from the channel group 1440. The first subgroup of each channel group is put in the first set, the second subgroup of each channel group is put in the second set, the third subgroup of each channel group is put in the third set, and the fourth subgroup of each channel group is put in the fourth set. The channel group interleaving in FIGS. 13 and 14 may be performed by a compiler that compiles the DNN.

[0143] FIG. 15 illustrates a sparse cell 1500, in accordance with various embodiments. The sparse cell includes 16 PEs arranged in PE columns 1510A-1510D (collectively referred to as "PE columns 1510" or "PE column 1510") and PE rows 1515A-1515D (collectively referred to as "PE rows 1515" or "PE row 1515"). A PE column 1510 or PE row 1515 may also bereferred to as a PE group. The sparse cell 1500 also includes weight register files 1520A- 1520D (collectively referred to as "weight register files 1520" or "weight register file 1520") and activation register files 1530A-1530D (collectively referred to as "activation register files 1530" or "activation register file 1530"). In other embodiments, the sparse cell 1500 may have fewer, more, or different components.

[0144] The activation register files 1530 may store activations of a DNN layer executed by the sparse cell 1500. The activations may be quantized values. In some embodiments, the activation register files 1530 store activations from different channel groups. In an example, the activation register file 1530A may store activations in the channel group 1410 in FIG. 14, the activation register file 1530B may store activations in the channel group 1420 in FIG. 14, the activation register file 1530C may store activations in the channel group 1430 in FIG. 14, and the activation register file 1530D may store activations in the channel group 1440 inFIG. 14. The weight register files 1520 may store weights of the DNN layers. The weights may be quantized values. The weight register files 1520 may store weights from different channel groups so that the right weights can be loaded into the right PEs for being multiplied with the right activations. Weights stored in the weight register files 1520 and activations stored in the files 1530 are loaded into the PE columns 1510 for performing MAC operations.

[0145] In the example shown in FIG. 15, each weight register file 1520 corresponds to a PE column 1510 and stores weights to be processed by one or more PEs in the PE column 1510, and each activation register file 1530 corresponds to a PE row 1515 and stores weights to be processed by one or more PEs in the PE row 1515. In some embodiments, each PE row 1515 may receive a unique HW context and all the input channels for that specific context. The context may be a vector including activations or weights having the same (x, y) coordinates in the activation tensor or weight tensor. In this mode, the activations from a fixed HW context may be multicast to all the PE rows 1515 and the group-wise channels may be selected by each PE based on a mask (e.g., sparsity masks shown in FIG. 18). The sparse cell 1500 may compute partial sums for the DNN layer. The partial sums may be further processed in a post-processing engine that includes a plurality of PPEs.

[0146] FIG. 16 illustrates a post-processing engine 1600, in accordance with various embodiments. The post-processing engine 1600 includes PPEs 1605, individually referred to as "PPE 1605." A PPE 1605 includes a multiplier 1610, an adder 1620, and a MUX 1630. ThePPEs 1605 may receive values computed by the PEs in the sparse cell 1500. The PPEs 1605 may dequantize the values. As described above, the number of PPEs 1605 in the postprocessing engine 1600 may equal the number of PEs in a PE row of the sparse cell 1500. A PPE 1605 may apply a scale 1601 and bias 1602 on partial sums computed from a channel group to dequantize the partial sums. In some embodiments, the PPE 1605 may receive four partial sums. The multiplier 1610 may multiply the scale 1601 with each partial sum. The product of the multiplication may be accumulated with the bias 1602 in the adder 1620. The result of the accumulation may be a dequantized partial sum, such as a 32-bit partial sum. The dequantized partial sums may be accumulated in the adder 1620 to compute an output data element of the DNN layer. Even though not shown in FIG. 16, the post-processing engine 1600 may have an adder tree that can take the accumulated partial sums from the PPEs 1605 and add them up to generate the final 32-bit values before applying the quantization-based operations. As shown in FIG. 16, the post-processing engine 1600 receives four scales and four biases for four channel groups, respectively. The dequantization may be performed for the channel groups separately.

[0147] The MUX 1630 may facilitate the adder 1620 performing the two types of accumulation. The MUX 1630 receives two signals: one signal is the bias 1602 and the other signal is a sum computed by the adder 1620. The MUX 1630 selects one of the two signals and transmits the selected signal to the adder 1620. When the adder 1620 receives the bias 1602 from the MUX 1630, the adder 1620 may accumulate the bias 1602 with a product computed by the multiplier 1610. When the adder 1620 receives the sum from the MUX 1630, the adder 1620 may accumulate the sum with another sum.

[0148] The post-processing engine 1600 may receive partial sums from the sparse cell 1500. In some embodiments, each PE row 1515 drains to the same set of PPEs in the postprocessing engine 1600 sequentially. The PPEs may add the values from each PE row 1515 sequentially. In some embodiments, the four PE rows 1515 may drain to the PPEs in four cycles, respectively. Since the PPEs can accumulate over multiple PE rows, they can apply a scale and bias specific to each row. Also, the number of PPEs may equal the number of PEs in a PE row 1515. Every PPE can potentially apply a scale and bias for each group and accumulate within the internal accumulator. The weight groups may split into multiple PPEs across different PE rows 1515 in a PE column 1510.

[0149] FIG. 17 illustrates distribution of an input vector across ICs to different PEs groups within a DPU, in accordance with various embodiments. The DPU may be an example of the DPU 355 in FIG. 3. The DPU in FIG. 17 includes sparse cells 1700 (individually referred to as "sparse cell 1700"). For the purpose of illustration and simplicity, FIG. 17 shows a part of the DPU, including control readers 1710 (individually referred to as "control reader 1710"), sparsity FIFOs 1720 (individually referred to as "sparsity FIFO 1720"), and data readers 1730 (individually referred to as "data reader 1730") in addition to the sparse cells 1700. FIFO stands for first-in -first out.

[0150] As shown in FIG. 17, each sparse cell 1700 includes four data storage units 1750 (individually referred to as "data storage unit 1750"), and 32 PEs 1760 (individually referred "PE 1760"). The 32 PEs 1760 are arranged in four rows. Each row includes eight PEs 1760 represented by 0-7 in FIG. 17. Each sparse cell 1700 is coupled with a control reader 1710, a sparsity FIFO 1720, and a data reader 1730. The control readers 1710, sparsity FIFOs 1720 and data readers 1730 may be components of an IDU, e.g., the IDU 360. In other embodiments, the data processing unit or a sparse cell 1700 may include fewer, more, or different components. For instance, a control reader 1710 may be coupled with more than one sparsity FIFO 1720 or data reader 1730.

[0151] The control readers 1710 facilitate transmission of data elements and sparsity masks to the sparse cells 1700. The control readers 1710 may operate on input data of DNN layers, such as activations and weights. The input data may be quantized values generated from group quantization. A control reader 1710 may operate on input data to be processed by a particular sparse cell 1700. In an example where the control reader 1710 operates on an input vector and feeds the input vector into the data reader 1730 for the sparse cell 1700, all the four data storage units 1750 in the sparse cell 1700 receive the entire input vector. The control reader 1710 may also feed four sparsity masks into the data reader 1730, and the data reader 1730 may send the four sparsity masks to the four data storage units 1750, respectively. As a result, each data storage unit 1750 in the sparse cell 1700 stores the entire input vector and one sparsity mask. The sparse cell 1700 may use the sparsity mask(s) to perform computations in the DNN layer.

[0152] In some embodiments, the input vector may be along the IC dimension so that the ICs can be spread over four MAC rows. For instance, IC0-3 may go to the first row, IC4-7 may go to the second row, IC8-11 may go to the third row, and IC12-15 may go to the fourth row.The data reader 1730 may effectively broadcast 4 data elements having the same XY position to each data storage unit 1750. As 4 ICs are sent to each data storage unit 1750, four PEs 1760 in each row of the sparse cell 1700 can be used, as shown by the dotted pattern in FIG. 17. In this way, all four PEs can be utilized. In the example shown in FIG. 17, as there are 8 MAC units available per data storage unit, the utilization rate is 50. In embodiments where the sparse cell 1700 has 4 MAC units available for a single data storage unit, this distribution pattern can yield 100% utilization.

[0153] FIG. 18 illustrates an example data flow for executing a DNN layer based on sparsity masks, in accordance with various embodiments. A data reader 1801 is fed with an input vector 1810. The data reader 1801 may be an example of the data readers 1730 in FIG. 17. For the purpose of illustration and simplicity, the input vector 1810 includes 16 data elements, each of which is represented by a cube in FIG. 18. The 16 data elements may be in 16 ICs, respectively. The input vector 1810 may be a vector in an input tensor of a DNN layer.

[0154] The data reader 1801 is also fed with four sparsity masks 1820A-1820D. Each of the sparsity masks 1820A-1820D has 16 elements, such as 16 bits, that indicate a sparsity pattern of the input vector 1810. For instance, the sparsity mask 1820A indicates that the first segment of four data elements of the input vector 1810 are nonzero while the other 12 data elements of the input vector 1810 are zero; the sparsity mask 1820B indicates that the second segment of four data elements of the input vector 1810 are nonzero while the other 12 data elements of the input vector 1810 are zero; the sparsity mask 1820C indicates that the third segment of four data elements of the input vector 1810 are nonzero while the other 12 data elements of the input vector 1810 are zero; the sparsity mask 1820D indicates that the fourth segment of four data elements of the input vector 1810 are nonzero while the other 12 data elements of the input vector 1810 are zero. The sparsity patterns encoded by the sparsity masks 1820A-1820D may not represent the real values of the data elements in the input vector 1810. Rather, the sparsity patterns are used for distributing the four segments of the input vector 1810 to different PE groups.

[0155] The data reader 1801 loads the input vector 1810 and the sparsity masks 1820A- 1820D into data storage units 1802A-1802D. The data storage units 1802A-1802D is coupled with four PE groups 1803A-1803D, respectively. Each of the PE groups 1803A-1803D includes PEs, such as PE 1760 in FIG. 17. For the purpose of illustration, each of the PEgroups 1803A-1803D has four PEs in FIG. 18. In other embodiments, there can be fewer or more PEs in a PE group. The data storage units 1802A-1802D may be examples of the data storage units 1750 in FIG. 17. The PEs 1803 may be examples of the PEs 1760 in FIG. 17.

[0156] In some embodiments, the data storage unit 1802A is loaded with the data reader 1801 and the sparsity mask 1820A; the data storage unit 1802B is loaded with the data reader 1801 and the sparsity mask 1820B; the data storage unit 1802C is loaded with the data reader 1801 and the sparsity mask 1820C; the data storage unit 1802D is loaded with the data reader 1801 and the sparsity mask 1820D. In other embodiments, the data storage units 1802A-1802D may not be loaded with the sparsity masks 1820A-1820D. Rather, the sparsity masks 1820A-1820D may be loaded into one or more sparsity storages, such as the sparsity tensor storage unit 2265 in FIG. 22.

[0157] A control logic (not shown in FIG. 18), such as the control logic 2267 in FIG. 22, may control transfer of different segments of the input vector 1810 to the PE groups 1803A- 1803D based on the sparsity masks 1820A-1820D. For instance, the control logic may use the sparsity mask 1820A to transfer the first segment 1830A of the input vector 1810 to the PE group 1803A. Similarly, the control logic may use the sparsity mask 1820B to transfer the first segment 1830B of the input vector 1810 to the PE group 1803B, use the sparsity mask 1820C to transfer the first segment 1830D of the input vector 1810 to the PE group 1803C, and use the sparsity mask 1820D to transfer the first segment 1830D of the input vector 1810 to the PE group 1803D.

[0158] The PE groups 1803A-1803D perform MAC operations using the segments 1830A- 1830D respectively. Each of the PE groups 1803A-1803D may generate an output, which may be a data element. The outputs of the PE groups 1803A-1803D are provided to an adder 1804. The adder 1804 accumulates the PE groups 1803A-1803D and generates an output data element of the DNN layer. As the execution of the DNN layer completes, an output tensor 1840 is generated.

[0159] The data distribution and data flow described above in conjunction with FIGS. 15-18 can facilitate group quantization-based DNN layer executions in a compiler aware manner. For instance, the channel groups may be interleaved (e.g., the group interleaving shown in FIGS. 13 and FIG. 14) before the group operations are performed. This method can be advantageous for DNNs with a large number of input channels compared to HW. In the stencil shown in FIG. 18, the PEs in different rows can work on different IC groups for thesame HW context. The ICs corresponding to different groups are arranged in a manner that they are loaded in different PEs in a column. For instance, 4 contiguous ICs may be loaded into a PE. Four sets of 4 IC weights may be loaded into each PE in a column and an accumulation occurs across them in the PPE. Across every 16 ICs, there are 4 sets of ICs from 4 different groups as shown in FIG. 15. This ordering may be applied to both the activations and the weights. As described above, the sparsity acceleration logic in the DPU can enable distribution of the right activations / weights to the right PEs.

[0160] FIG. 19 illustrates a workflow 1900 in a compiler and a DPU, in accordance with various embodiments. The workflow 1900 may be a group quantization-based workflow. The workflow 1900 includes four steps: Step 1910, Step 1920, Step 1930, and Step 1940. In Step 910, the compiler processes a quantized weight tensor 1901 and dequantizes the weights to activation precision. The quantized weight tensor 1901 includes three groups along the input channel dimension. The three groups are highlighted with different patterns in FIG. 19. The three groups have been quantized, e.g., by the compiler, with different quantization parameters, and therefore, have different data precisions. Through Step 1910, the compiler can convert all the data weight precisions to the activation precision. In other embodiments, the compiler may convert the weight precisions to a data precision that is different from the activation precision.

[0161] In Step 1920, the DPU loads the dequantized weights and activations from a memory into a processing engine. In Step 1930, the DPU performs a neural network operation (e.g., convolution, MatMul operating, etc.) on the dequantized weights and activations. In Step 1940, the DPU drains the final points (e.g., output data elements of the neural network operation). The final points may be drained to the memory.

[0162] The workflow 1900 can avoid the requirement of numerous workload descriptors due to group quantization. One caveat of breaking the workload into multiple subworkloads, directly proportional to the number of groups, is that it can incur significant overhead in workload descriptors, as maintenance of numerous workload descriptors is needed due to the group size. This can be a critical issue when the group size is very small and the number of groups is very large. Since one descriptor is needed per workload and one workload corresponds to one group, several descriptors corresponding to a large number of groups need to be stored. The DPU also needs to drain out all partial sums after each group-based workload to apply the scale factor in the PPE. The DSP may need to addthese partial sums parallelly. In cases of lacking enough DSP (parallel processing), the partial sum addition can become a performance bottleneck. An efficient way to circumvent both these issues is to dequantize the activations and weights before processing the neural network operation. This can be done by using the group-based scales and dequantizing the smaller weights and activations to a higher precision. For example, the group-based INT2 / INT4 / FP4 / FP8 weights can be dequantized to FP16 when activations are already in FP16. Although the compute acceleration and memory savings are lost by performing the neural network operation at higher precision, this method can bring the benefits of having a single descriptor as well as eliminating the need to accumulate the partial sums outside. For larger number of groups, this technique can really lead to significant performance / power benefits.

[0163] In addition to the hardware support required for implementing the various techniques for group quantized-based executions of DNN layers described above, the DPU may also have group-specific descriptor fields and hardware logic. These group-based logic may be based on the grouping dimensions and features needed for group quantized-based executions of DNN layers. The DPU may have group-based programmability features. For instance, the DPU can support zero points for activations and weights for each group within the IDU and processing engine. Also, the DPU may have precision-based palletization table for each group within the IDU to upscale the activations and weights from a smaller precision to a higher precision supported by the NPU. The DPU may also have N:M structured sparsity-based look-up table (LUT) within the processing engine. The DPU may further have a scale and bias table within the post-processing engine. In some embodiments, the zero point feature in the IDU may support OC and HW dimensions. The precision palletization in the IDU may support HW, IC, and OC dimensions. The structured sparsity feature in the IDU and processing engine may support HW, IC, and OC dimensions. The feature for scale and bias, which is supported by the IDU, processing engine, and postprocessing engine, may also support HW, IC, and OC dimensions.

[0164] FIG. 20 illustrates a DPU 2000 with group-based features, in accordance with various embodiments. The DPU 2000 includes an IDU 2011 with an activation reader 2012 and a weight reader 2013, a processing engine 2014, and a post-processing engine 2015. The IDU 2011 may load data into the processing engine 2014 and post-processing engine 2015. For instance, the activation reader 2012 may load activations into the processing engine 2014.The weight reader 2013 may load weights into the processing engine 2014. The weight reader 2013 may load weights into the processing engine 2014. In some embodiments (e.g., embodiments where the DNN layer is executed based on group quantization), the IDU 2011 may load quantized values of a channel group. The IDU 2011 may also load the scale and zero point of the channel group. The IDU 2011 may load the scale and zero point into the post-processing engine 2015.

[0165] The IDU 2011 also includes a LUT 2021, LUT 2022, and LUT 2023. The LUT 2021 may be inside or coupled with the activation reader 2012. The 2022 / and LUT 2023 may be inside or coupled with the weight reader 2013. In some embodiments, the LUT 2021 may store zero points for activations. The LUT 2022 may store zero points for weights. The LUT 2023 may store data for changing weight precisions. The LUT 2023 may be a precision-based palletization table. In some embodiments, the LUT 2023 may store data to be used for upscaling activations and weights for each group from a smaller precision to a higher precision supported by the NPU.

[0166] The processing engine 2014 performs computations in the DNN layer. The processing engine 2014 includes sparse cells 2016, individually referred to as "sparse cell 2016." A sparse cell 2016 includes a PE array and a sparsity acceleration logic. For the purpose of illustration, the PE array in each sparse cell 2016 includes 16 columns and 16 rows. In other examples, the PE array may include a different number of columns or rows. The sparsity acceleration logic includes a LUT 2024. Each LUT 2024 is represented by a dot in FIG. 20. The LUT 2024 may be a N:M structured sparsity-based LUT. The LUT 2024 may be used to distribute data to PEs in the sparse cell 2016. Certain aspects of N:M structured sparsitybased LUT are described below in conjunction with FIG. 21. Certain aspects of data distribution by the sparsity acceleration logic are described in conjunction with FIG. 18.

[0167] The post-processing engine 2015 includes PPEs 2018 (individually referred to as "PPEs 2018") arranged in an array. Each PPE 2018 has a LUT 2025. Each LUT 2025 is represented by a dot in FIG. 20. The LUT 2025 may store scale and bias for the PPE 2018 to perform dequantization. In some embodiments, the bias is used as a zero point for the dequantization. For the purpose of illustration, one or more components of the DPU 2000 are not shown in FIG. 20. For instance, the DPU 2000 may have an ODU.

[0168] FIG. 21 illustrates an example flow of control and data for structured sparsity acceleration, in accordance with various embodiments. The flow shown in FIG. 21 is for thestructured sparsity pattern 1:4. The flow may be in a DPU, e.g., the DPU 355 in FIG. 3. The flow starts with four compressed maps 2101, individually referred to as "compressed map 2101." Each of the compressed maps 2101 is converted to a sparsity map 2102, and four sparsity maps 2102 for the structured sparsity pattern 1:4 are generated. The decompression of the compressed maps 2101 may be done by a load module in the DPU, e.g., the IDU 360 in FIG. 3.

[0169] The sparsity maps 2102 are input into a LUT 2110. The LUT 2110 may be an example of the LUT 2024 in FIG. 20. The LUT 2110 outputs a 1-bit read index 2103 for each of the sparsity maps 2102. Four read indices 2103 are output from the LUT 2110. The read indices 2103 are provided to an address generator 2120, which generates memory addresses from the read indices 2103. The LUT 2110 and the address generator 2120 may be in a sparsity module in the DPU, e.g., the sparsity modules 2266 in FIG. 22. Activations and weights are read from the data storage unit 2130 based on the addresses generated by the address generator 2120. The data storage unit 2130 may include an activation register file and a weight register file.

[0170] The activations and weights are transmitted to four multipliers 2140, each of which receives an activation and a corresponding weight. The products computed by the multipliers 2140 are accumulated by an adder tree 2150, and the sum is stored in a data storage unit 2160. The data storage unit 2160 may be an output register file. The data storage unit 2130, multipliers 2140, adder tree 2150, and data storage unit 2160 may be in a MAC unit. The sum may be transmitted to another MAC unit for further computation. Additionally or alternatively, the data storage unit 2160 may receive a sum from another MAC unit. The two sums may be accumulated by the adder tree 2150 to compute a new sum.

[0171] FIG. 22 illustrates an example sparse cell 2200, in accordance with various embodiments. The sparse cell 2200 may be at least part of a processing engine, e.g., the processing engine 1170 in FIG. 11. The sparse cell 2200 includes 16 MAC units 2210 (individually referred to as "MAC unit 2210"), which constitutes a MAC array having four rows and four columns. A MAC unit 2210 may be at least part of a PE. The MAC array has a spatial shape of 4x4, meaning the height of the MAC array is four and the width of the MAC array is also 4. The sparse cell 2200 also includes 16 weight register files 2220 (individually referred to as "weight register file 2220"), 16 activation register files 2230 (individuallyreferred to as "activation register file 2230"), four row buffers 2240 (individually referred to as "row buffer 2240"), and sparsity modules 2266 (individually referred to as "sparsity module 2266"). In other embodiments, the sparse cell 2200 may include fewer, more, or different components. For example, the sparse cell 2200 may include a different number of MAC units 2210, weight register files 2220, activation register files 2230, row buffers 2240, or sparsity modules 2266. As another example, the sparse cell 2200 may include column buffers in lieu of or in addition to the row buffers 2240. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0172] The MAC units 2210 are configured to perform MAC operations. Each MAC unit 2210 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 2210 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 22, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 2210. The number of adders in the first tier may be half of the number of the MAC units 2210, and each adder may accumulate the outputs of two MAC units 2210. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 2200.

[0173] The weight register files 2220 store weights to be processed in MAC operations. In the embodiments of FIG. 22, four weight register files 2220 are grouped into a storage set that stores data to be used by a column of MAC units 2210. There are four storage sets corresponding to the four columns of MAC units 2210. In some embodiments, a weight register file 2220 may correspond to a MAC unit 2210 and store data to be processed by the MAC unit. In some embodiments, the four weight register files 2220 for a single column of MAC units 2210 constitute a data storage unit of the column.

[0174] The activation register files 2230 stores activations to be processed in MAC operations. In the embodiments of FIG. 22, four activation register files 2230 are groupedinto a storage set that stores data to be used by a row of MAC units 2210. There are four storage sets corresponding to the four rows of MAC units 2210. In some embodiments, an activation register file 2230 may correspond to a MAC unit 2210 and store data to be processed by the MAC unit. In some embodiments, the four activation register files 2230 for a single row of MAC units 2210 constitute a data storage unit of the row. The row buffers 2240 store outputs of the MAC units 2210. Each row buffer 2240 may drain outputs of a single row of MAC units 2210.

[0175] The sparsity module 2266 facilitates dynamic sparsity-based acceleration or workload distribution in the sparse cell 2200. In the embodiments of FIG. 22, each sparsity module 2266 includes a sparsity tensor storage unit 2265 and a control logic 2267. The sparsity tensor storage unit 2265 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 2265 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activation-weight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero. The sparsity tensor storage unit 2265 may include a LUT that stores data for structured sparsity, such as the LUT 2110 in FIG. 21.

[0176] The control logic 2267 may control transmission of activations and weights stored from the weight register files 2220 and the activation register files 2230 to the MAC units 2210 based on sparsity tensors. For instance, the control logic 2267 may select a subset of the weights stored in the weight register files 2220 and select a subset of activations stored in the activation register files 2230 based on a sparsity tensor. The control logic 2267 may transmit the selected weights and activations to the MAC units 2210 for performing MAC operations. The other weights stored in the weight register files 2220 or the other activations stored in the activation register files 2230 are skipped from computation. In the embodiments of FIG. 22, each sparsity module 2266 controls sparsity acceleration or workload distribution in a respective MAC unit 2210. As the sparsity acceleration or workload distribution is either based on both weight sparsity and activation sparsity, 16 sparsity modules 2266 are used for acceleration computations in the 16 MAC units 2210.

[0177] As shown in FIG. 22, the sparse cell 2200 is associated with MUXs (MUXs) 2203, 2204, 2205, and 2206. In other embodiments, the sparse cell 2200 may be associated with adifferent number of MUXs or other devices. The MUX 2203 facilitates loading weights, e.g., from the local memory 1140, into the weight register files 2220. The MUX 2204 facilitates loading activations, e.g., from the local memory 1140, into the activation register files 2230. The MUX 2205 facilitates loading sparsity tensors into the sparsity tensor storage unit 2265. The MUX 2206 may be a drain MUX that can facilitate draining outputs of the MAC units 2210, e.g., to the local memory 1140.

[0178] FIG. 23 illustrates an example PE 2300, in accordance with various embodiments. The PE 2300 may be a unit component of a sparse cell. In the embodiments of FIG. 23, the PE 2300 includes an MAC unit 2305, an activation register file 2310, a weight register file 2320, an output register file 2350, and a sparsity accelerator 2360. The MAC unit 2305 includes a multiplier 2330 and an adder 2340. In other embodiments, the PE 2300 may include fewer, more, or different components.

[0179] The activation register file 2310 stores an activation operand, which may be a context. The activation register file 2310 may be an example of the activation register files 1530 in FIG. 15. The weight register file 2320 stores a weight operand. The weight register file 2320 may be an example of the weight register files 1520 in FIG. 15. The activation operand and weight operand may be loaded from a memory (e.g., the memory 1140) into the activation register file 2310 and the weight register file 2320, respectively. The sparsity accelerator 2360 receives a sparsity bitmap 2315 that corresponds to the sparse tensor in the weight register file 2320. The sparsity bitmap 2315 may be a combined sparsity bitmap when the MAC unit 2305 operates in a combined sparsity mode. The sparsity bitmap 2315 may be an activation sparsity bitmap when the MAC unit 2305 operates in an activation sparsity mode. The sparsity bitmap 2315 may be a weight sparsity bitmap when the MAC unit 2305 operates in a weight sparsity mode. The sparsity bitmap 2315 may have the same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0180] Using the sparsity bitmap 2315, the sparsity accelerator 2360 selects four activations from the activation register file 2310 and selects four weights from the weight register file 2320. The sparsity accelerator 2360 transmits the selected activations and weights to the multiplier 2330. These selected data elements correspond to the nonzero valued elements of the sparsity bitmap 2315. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 2330 may compute a product basedon each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 2340. Even though FIG. 23 shows a single multiplier 2330, the MAC unit 2305 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0181] The adder 2340 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. For instance, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 2305 may operate in a dense mode in which the sparsity bitmap 2315 is not used and the sparsity accelerator 2360 is inactive. The MAC unit 2305 may process all the activations in the activation operand and all the weights in the weight operand.

[0182] In some embodiments, the PE 2300 receives one or more PE-level internal partial sums from one or more other PEs. The adder 2340 or an accumulator (not shown in FIG. 23) can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of the PE 2300 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 2350. The one or more other PEs may be in the same column as the PE 2300 in a sparse cell. The multi-unit internal partial sum may be a columnlevel internal partial sum. In some embodiments, the PE-level internal partial sum of the PE 2300 or the multi-unit internal partial sum may be sent to one or more other PEs for further accumulation.

[0183] FIG. 24 illustrates weight loading for a group operation, in accordance with various embodiments. The group operation may be a group convolution. The weight loading may be performed by an IDU, such as the IDU 360 in FIG. 3. The IDU may store the start addresses of activation or weights in a data storage unit 2440. The data storage unit may be a RAM. The start addresses may be referred to as SE pointers ("SE_PTR"), and the RAM may be referredto as a SE_RAM. The IDU may consult the SE_RAM entries to locate the start point of the next context. FIG. 24 shows how group convolution is accounted for when computing the SE pointers. For a regular convolution, the SE pointer may start at BASE_SE_PTR for the current context and increases by Z dimension of the tensor for the next context because each context consumes the entire Z dimension. BASE_SE_PTR stands for base SE pointer, which may indicate a base address of an activation or weight. The Z dimension may be the input channel dimension. In a group convolution, the context is split into groups and for each subcontext the SE pointer increments by the group size.

[0184] FIG. 24 shows a process of determining and storing SE_PTR. A MUX 2410 receives a signal 2401 and a signal 2402. The signal 2401 indicates the group size. The signal 2402 indicates the tensor size. The MUX 2410 outputs a signal 2403 that indicates the offset in the Z dimension, i.e., Z_offset. The signal 2403 and a signal 2404 are input into an adder 2420. The signal 2404 indicates BASE_SE_PTR. The adder 2420 may produce a sum by adding the signal 2403 and signal 2404 and output a signal 2405. The signal 2404 and signal 2405 are input into another MUX 2430. The MUX 2430 outputs a signal 2406 which indicates SE_PTR. The signal 2406 is stored in the data storage unit 2440, i.e., SE_RAM. The MUX 2410 and MUX 2430 operate under a configuration signal 2407. The configuration signal 2407 indicates a type of the group operation. In this example, the configuration signal 2407 indicates that the group operation is a group convolution. The MUX 2410 and MUX 2430 may select signals based on the configuration signal 2407.

[0185] Group convolution can enable IDU-based performance optimization schemes. For example, since consecutive group convolution inputs operate on independent (mutually exclusive) input activation and weight sets, the DPU does not need to wait (or stall) until the entire output is drained for the previous workload. This can lead to a significant performance uplift compared to a conventional convolution case where consecutive convolutions are dependent. The early prestart of loading the data for the next workload can begin even when the DPU is draining the value of the previous workload. The consecutive workloads corresponding to each group can be programmed in a workload queue beforehand so that the IDU can access this information and sequentially fetch data without any dependency. As another example, the IDU can identify cases where different group convolutions can share the same weight set with the aid of the compiler andeliminate re-fetching of those weights resulting in significant performance and power benefits.

[0186] FIG. 25 illustrates a MatMul layer 2500, in accordance with various embodiments. The MatMul layer 2500 may be a layer in a transformer model, such as the transformer model 3100. The MatMul layer 2500 may be part of an MHA layer. In the example shown in FIG. 25, the MatMul layer 2500 is quantized with GPTQ4-bit schema in which the group is 32 and the group size is 128. The MatMul layer 2500 has a workflow 2501 and workflow 2502. In the workflow 2501, the weight tensor is split into groups along the channel dimension, which is the dimension to be accumulated. In this case, a 4096x4096 tensor is split into 32 groups, each group has 128x4096 UINT4 data. UINT4 is a data type, i.e., 4 bits unsigned integer sub-byte-element. Each group may have its own scale. The whole scale tensor (multiply) shape is <32xlx4096>, as shown in FIG. 25. The uint4 weights are converted to FP16 weights. The tensor is reshaped. In the workflow 2502, an activation tensor having a shape <lx4096> and data type FP16 is input into the MatMul operator. The MatMul operation is performed on the activation tensor and reshaped weight tensor, both of which are in the FP16 data type. The MatMul operation produces a tensor having a shape < 1x4096>.

[0187] FIG. 26 illustrates a GPTQ MatMul, in accordance with various embodiments. In the example shown in FIG. 26, the group size is 16, and there are 8 groups. In some embodiments, the GPTQ quantization is equivalent to doing the MatMul on each group first, then accumulating together. The GPTQ MatMul may be split into 8 group MatMuls, which are shown as MatMukl through MatMul:8 in FIG. 26. Each group MatMul may take <1x16, FP16> as input activation (shown as "Act" in FIG. 26), <16xl28:INT4> as weight (shown as "W" in FIG. 26), and <lxl28:FP8> as scale (shown as "Scale" in FIG. 26). The data type of scale tensor can be FP16, FP32 or FP8 depending on the model accuracy requirement.

[0188] As each group has its own scale, the group MatMuls are performed separately. The outputs of the 8 group MatMuls are accumulate together in four accumulations, which are shown as Add:l through Add:4 in FIG. 26. Even though GPTQ can bring much better accuracy than channel-wise quantization schema, the additional accumulation may bring additional computation and have negative impact on inference speed, especially in the prefill model with a seq-length of more than IK.

[0189] Smaller workloads may lead to lower hardware utilization. For a standard MatMul <M, K> * <K, N>, the Al core hardware utilization can be very high when M, K, N is large, i.e. M=1024, N=4096, K=4096. While in the GPTQ context, the MatMul includes a number of smaller MatMuls, i.e., group=32, group_size=128, 32 * MatMul <1024, 128> * <128, 4096>. The dimension 128 can have the chance to impact hardware utilization when the tiling strategy is not well handled. Besides, the number of workloads is increased by 32 times, which can lead to the thread / workload dispatcher fail to process all those small tasks / workloads quickly enough to fully occupy hardware. There may also be more weight table loading. The weight table may be a data structure to specify the register setting of workloads. The size of NPU's weight table is OC (output channel) * 16 bytes. For the example above, the storage size of the weight table is: 32(group) * 4096 (output channel) * 16 = 2MB. 2MB can be a significant overhead and it is 25% percentage of the INT4 weight size (8MB). Its impact on the inference token rate may be significant since the second token inference may be memory bound.

[0190] FIG. 27 illustrates a DPU-DSP pipeline for executing a MatMul layer, in accordance with various embodiments. In the DPU-DSP pipeline, the DPU performs groups MatMuls, shown as MatMultO through MatMul#7 in FIG. 27, and the DSP performs accumulations of group MatMul outputs, shown as Add#0 through Add#6 in FIG. 27.

[0191] Some of the accumulations by the DSP may be performed while the DPU is performing group MatMuls. As shown in FIG. 27, Add#0 is performed in parallel with MatMul#2. Add#0 may be the accumulation of the output of MatMultO and the output of MatMultl. It can start when MatMultl is complete. The DSP does not need to wait till MatMul#7 is complete to start Add#0. Similarly, Add#l (which may be the accumulation of the output of MatMul#2 and the output of Add#0) is performed in parallel with MatMul#3, Add#2 (which may be the accumulation of the output of MatMul#3 and the output of Add#l) is performed in parallel with MatMul#4, Add#3 (which may be the accumulation of the output of MatMul#4 and the output of Add#2) is performed in parallel with MatMul#5, Add#4 (which may be the accumulation of the output of MatMul#5 and the output of Add#3) is performed in parallel with MatMul#6, and Add#5 (which may be the accumulation of the output of MatMul#6 and the output of Add#4) is performed in parallel with MatMul#7. Add#6 is performed after MatMul#7 is complete. Add#6 may be the accumulation of the output of MatMul#7 and the output of Add#5.

[0192] In LLM inference, the prefill model can be memory bound due to the long sequence length. The latency may be a sum of the time spent on MatMul operator and the accumulation operator. In a NPU in this disclosure, MatMul may run on a PE array, while the accumulation operator (which is an elementwise add operator) may run either on a PE array or a DSP. To avoid the overall latency, the accumulation operator may be scheduled on a DSP and pipelined with MatMul on a DPU, as shown in FIG. 27. This approach introduces an overhead of the last accumulation (e.g., Add#7 in FIG. 27) compared to non-GPTQ MatMul. The number is negligible when the group is large, such as 32, 64, or even higher.

[0193] FIG. 28 illustrates a DPU pipeline for executing a MatMul layer, in accordance with various embodiments. The MatMul layer may be the same as the MatMul layer in FIG. 27. However, in the embodiments of FIG. 28, no DSP is used. The group MatMuls and accumulations are all executed by the DPU. As shown in FIG. 28, the DPU performs MatMultO and MatMultl first. Then the DPU performs Add#0 to accumulate the output of MatMultO and the output of MatMultl. Next, the DPU performs MatMul#2, which is followed by Add#l, which may be the accumulation of the output of MatMul#2 and the output of Add#0. Then the DPU performs MatMul#3, followed by Add#2. This continues till the DPU finishes MatMul#7 and Add#6. Add#6 may be the accumulation of the output of MatMul#7 and the output of Add#5.

[0194] FIG. 29 illustrates weight table generation for a MatMul layer, in accordance with various embodiments. For the purpose of illustration, the MatMul layer has four group MatMuls (MatMultO through MatMul#3) and four accumulations (Add#0 through Add#3). Weights, which may have determined by training the DNN, may be stored in a memory of the NPU, such as the memory 310 in FIG. 3. Weight tables are also generated offline and stored in the memory. The weight table may be generated by a compiler before the execution of the DNN. The DMA transmits the weights and weight table from the memory into a local memory, e.g., the local memory 340 in FIG. 3. The DPU accesses the local memory to get the weights and performs the group MatMuls. The DSP also accesses the local memory. The DSP reads the weight table from the local memory and performs the accumulations.

[0195] The transfer of the weight table from the memory to the local memory can cause a significant overhead. For instance, in the LLM decoding phase, the model is memory bound due to the existence of a KV cache. The extra weight table overhead can have significantimpact on inference speed and token rate. Take LLAMAv2-7B as an example, the GPTQ can bring 25% extra memory traffic while the non-GPTQ model requires less than 1% overhead. Since the KV cache model is memory bound, 25% more memory traffic can result in an additional 25% latency.

[0196] FIG. 30 illustrates weight table generation on a DSP, in accordance with various embodiments. The DSP may be an example of the DSP 350 in FIG. 3. Different from the weight table generation in FIG. 29, the weight table generation in the embodiments of FIG. 30 is performed on the DSP in lieu of the compiler. The weight able is generated dynamically during runtime. Weights may still be stored in the memory of the NPU, and the DMA engine may transfer the weights from the memory to the local memory accessible by the DSP and DPU. The weight table is not generated or stored in the memory offline. Rather, DSP generates the weight table during the execution of the MatMul layer. As shown in FIG. 30, the DSP fills the weight table before every group MatMul. The weight table may specify register setting for performing the group MatMul. The DSP still performs the accumulations of outputs of the group MatMuls performed by the DPU. In the example shown in FIG. 30, the DSP fills the weight able for MatMul#2 after MatMultl and Add#l are complete. FIG. 30 shows a novel method to generate the weight table on DSP to save the memory traffic from DDR to the local memory. This approach invokes a DSP kernel to fill the weight table for each MatMul group. Towards that, there is a cost to run the DSP kernel, it has a chance to be overlapped with DMA.

[0197] FIG. 31 illustrates an example transformer model 3100, in accordance with various embodiments. The transformer model 3100 may transform input sequences into output sequences. In some embodiments, the transformer model 3100 is a DNN that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 3100 may be an LLM. The transformer model 3100 includes an encoder block 3110, a decoder block 3120, and a head block 3130. In other embodiment, different or additional components may be included in the transformer model 3100. Further, functionality attributed to a component of the transformer model 3100 may be accomplished by a different component included in the transformer model 3100 or a different model or module.

[0198] The encoder block 3110 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 31, the encoder block 3110 receives an input 3101 and generates an encoder output 3102. The input 3101 may be an input prompt. In some embodiments, the input 3101 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the input 3101 may include a prompt received from a user of the transformer model 3100. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder output 3102 may include one or more vectors that are contextualized representations of the input 3101. Each vector in the encoder output 3102 may represent a token in the input 3101 with contextual understanding.

[0199] The encoder block 3110 includes an embedding layer 3113, a positional encoding layer 3115, and a plurality of layers 3140 (individually referred to as "layer 3140"). In other embodiments, the encoder block 3110 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 3110 may be different from the arrangement shown in FIG. 31. For the purpose of illustration, the encoder block 3110 has N layers in FIG. 31, where N is an integer. Each layer 3140 may include one or more neural network operations. The layers 3140 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 3101. Different layers 3140 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 3140 have identical components. The components in a layer 3140 may be layers and may also be referred to as sub-layers of the layer 3140. As shown in FIG. 31, a layer 3140 includes four sub-layers: an MHA layer 3141, an add & norm layer 3142, a feed forward layer 3143, and another add & norm layer 3144.

[0200] The decoder block 3120 iteratively generates outputs 3103 using encoded representations generated by the encoder block 3110. The decoder block 3120 includes an embedding layer 3123, a positional encoding layer 3125, and a plurality of layers 3150 (individually referred to as "layer 3150"). For the purpose of illustration, the decoder block 3120 has N layers in FIG. 31, where N is an integer. In the embodiments of FIG. 2, the number of layers 3150 in the decoder block 3120 is the same as the number of layers 3140 in the encoder block 3110. In other embodiments, the number of layers 3150 in the decoderblock 3120 may be different from the number of layers 3140 in the encoder block 3110. Each layer 3150 may include one or more neural network operations. Different layers 3150 may have different internal parameters. In some embodiments, the layers 3150 may have identical components. The components in a layer 3150 may be layers and may also be referred to as sub-layers of the layer 3150. As shown in FIG. 31, a layer 3150 includes six sub-layers: an MHA layer 3151, an add & norm layer 3152, another MHA layer 3153, another add & norm layer 3154, a feed forward layer 3155, and another add & norm layer 3156.

[0201] In some embodiments, a sequence of inference stages is performed in the decoder block 3120 using encoder outputs, e.g., the encoder output 3102. A matrix may be predicted through each inference stage. The outputs 3103 may include a plurality of matrices. Each matrix may be further processed in the head block 3130 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference stage, the decoder block 3120 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 3110. The first matrix may be used by the head block 3130 to predict a first token. The predicted token may be used as a new input token, in addition to the start token(s), in the second inference stage. Similarly, a second token may be predicted through the second inference stage and may be used in the third inference stage. This iteration may continue till all the inference stages are complete.

[0202] The head block 3130 receives the output of the decoder block 3120 and processes it in a linear layer 3133 and a SoftMax layer 3135. A linear operation may be performed on the output of the decoder block 3120 in the linear layer 3133. The linear operation may include a multiplication of the output of the decoder block 3120 with a weight matrix. The output of the linear layer 3133 may be a vector. In some embodiments, the head block 3130 may function as a classifier. The number of data elements in the vector computed in the linear layer 3133 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 3133 may have M data elements representing the prediction for the M classes, respectively.

[0203] The output of the linear layer 3133 may be input into the SoftMax layer 3135. A SoftMax function may be applied on the output of the linear layer 3133 to compute probability scores. A probability score may have a value in the range from 0 to 31. In someembodiments, a probability value is computed for each data element in the vector computed in the linear layer 3133. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 3100 predicts as the next in the sequence. The final output of the transformer model 3100 may be the sequence of predicted tokens. In some embodiments, the head block 3130 may be a language modeling head.

[0204] An embedding layer (e.g., the embedding layer 3113 or the embedding layer 3123) converts an input of the embedding layer (e.g., the input 3101 or the outputs 3103) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 3113 may generate a plurality of embeddings, each of which may be converted from a different input token in the input 3101. The embeddings may capture the semantic meaning of the tokens in the input 3101. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 3101 is a prompt including a sequence of words, the embedding layer 3113 may generate an embedding from each word in the input 3101. The embedding layer 3123 in the decoder block 3120 may generate a plurality of embeddings from tokens received by the decoder block 3120 in a similar manner as the embedding layer 3113.

[0205] A positional encoding layer (e.g., the positional encoding layer 3115 or the positional encoding layer 3125) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 3104 or positional encoding vector 3105) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represent the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer.

[0206] An MHA layer (e.g., the MHA layer 3141, the MHA layer 3151, or the MHA layer 3153) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 3141 or the MHA layer 3151 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 3141, the queries, keys, and values may all come from the positional encoding layer 3115. For the MHA layer 3151, the queries, keys, and values may all come from the positional encoding layer 3125. The self-attention mechanism may enable the transformer model 3100 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0207] In some embodiments, the queries, keys, and values input into the MHA layer 3141 may be computed from vector embeddings generated by the positional encoding layer 3115. The queries, keys, and values input into the MHA layer 3151 may be computed from vector embeddings generated by the positional encoding layer 3125. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G IRWx / lmay be computed by multiply an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG IRdx / l, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K G IRWx / lmay be computed by multiple an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkE IRdx / l. Each row in the key matrix may be a key. A value matrix V E IRWx / lmay be computed by multiple an embedding matrix X E IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE IRdx / l. Each row in the value matrix may be a value.

[0208] In some embodiments, the MHA layer 3151 may implement masked multi-head selfattention. The MHA layer 3151 may prevent positions from attending to subsequentpositions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0209] In some embodiments, the MHA layer 3153 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 3153 may use outputs from the previous layer (i.e., the add & norm layer 3152) as queries and use outputs from the encoder block 3110 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 3120 to identify and emphasize the most relevant parts of the encoder's input.

[0210] In some embodiments, an MHA layer includes linear layers, a MatMul layer, a scale layer, a Softmax layer, another MatMul layer, a concatenation layer, and another linear layer. These layers may be arranged in a sequence. The MHA layer may receive three input matrices: a query matrix, a key matrix, and a value matrix, which are inputs of three linear layers, respectively. The linear layers may include matrix multiplication (MatMul) operations. For instance, a first linear layer may perform a multiplication of the query matrix with a weight matrix to compute a first parameter matrix. The first parameter matrix may be denoted as QW , where Q is the query matrix and W G ^-modei^qjs t^e wejght matrix. A second linear layer may perform a multiplication of the key matrix with a weight matrix to compute a second parameter matrix. The second parameter matrix may be denoted as KW , where K is the key matrix and W E ^-modei^k jsthe weight matrix. A third linear layer may perform a multiplication of the value matrix with a weight matrix to compute a third parameter matrix. The third parameter matrix may be denoted as VW-', where V is the value matrix and WtvEdmodeixdk jsthe weight matrix, i may indicate the index of the head. dqis the dimension of a query vector. dkis the dimension of a key vector. dvis the dimension of a value vector. In some embodiments, dq= dk= dv= dmodei / h. In some embodiments, the linear layers may be in a linear block of the MHA layer. In some embodiments, the MHA layer may include multiple linear blocks. For instance, the MHA layer includes h linear blocks. The linear blocks may have the same layers as each other.Each linear block may compute three parameter matrices from the query matrix, key matrix, and value matrix, respectively.

[0211] The MatMul layer, scale layer, mask layer, Softmax layer, and MatMul layer may be in an attention block of the MHA layer. The attention block may implement a scaled dotproduct attention mechanism. In some embodiments, the MHA layer includes a plurality of attention blocks that includes the attention block. For the purpose of illustration, the MHA layer includes h attention blocks. The attention blocks may have the same layers as each other. A linear block and an attention block may constitute a head of the MHA layer. When the MHA layer has h linear blocks and h attention blocks, the MHA layer has h heads. A head may be denoted as headi = Attention QW , KW , VW-').

[0212] A matrix multiplication operation may be performed on parameter matrices in the MatMul layer, which computes a score matrix. In some embodiments, the score matrix may establish the degree of emphasis each token should place on other tokens. The score matrix may include a plurality of scores. Each token may be assigned a score in relation to other tokens within the same time step. A higher score may indicate a higher focus or emphasis. The score matrix may be scaled in the scale layer. In some embodiments, the score matrix is scaled down in the scale layer by dividing the scores in the score matrix by the square root of the dimension of the query vector and the key vector, which may be denoted asThe output of the scale layer may be a scaled matrix, which includes adjusted scores. The mask layer may be optional in some embodiments. The mask layer may add an attention mask (which may be an input to the attention block) to the output of the scale layer to mask out some elements in the output of the scale layer. The positions of the masked-out elements may be defined by the attention mask. A Softmax function may be applied on the scaled matrix in the Softmax layer to compute an attention weight matrix. The attention weight matrix includes attention weights. The attention weights may be probability values ranging from 0 to 1. The Softmax function may emphasize high scores while diminishing low scores, which can enhance the model's ability to determine which tokens should get more attention.

[0213] In the MatMul layer, a matrix multiplication operation is performed on the attention weight matrix computed in the Softmax layer and the parameter matrix computed from value matrix in the corresponding linear layer. The result of the matrix multiplication operation is a single-head output matrix, which is an output of the attention block.

[0214] When the MHA layer has h attention blocks, there may be h single-head output matrices. The single-head output matrices are concatenated in the concatenation layer to form a concatenated matrix. A linear operation (also referred to as "linear transformation") is performed on the concatenated matrix using a weight matrix in the linear layer. In some embodiments, the MHA may be denoted as MultiHead Q, K, 7) = Concat (head- , head2, ... , headhyw° , where Concat denotes concatenation, and W° G ^hdvxdmodeijs t e wejght matrix in the corresponding linear layer.

[0215] An add & norm layer in the transformer model 3100, such as the add & norm layer 3142, 3144, 3152, 3154, and 3156, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 3142 is the MHA layer 3141. As another example, the preceding layer of the add & norm layer 3154 is the MHA layer 3153.

[0216] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sub lay er (x)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sub lay er (x denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as jj.xy= - X Zz=i^xyz > where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert ixyto a 3D tensor .xyz, e.g., by replicating every data element over z output points.

[0217] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— [ixyz. The layer normalization operation may further include a variance computation denoted asand a division1 computation denoted as Mxy= , ^=. Mxymay be a 2D tensor. The layernormalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A'xyz= X, = = Dxyzx Mxyz. The layer normalization operation may further compute x(ff2Xy+e) may be the output of the layer• ■ • normalization operation.

[0218] A feed forward layer (e.g., the feed forward layer 3143 and the feed forward layer 3155) may be a position-wise fully-connected feed forward network. In an example, the feed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0219] FIG. 32 illustrates an example CNN 3200, in accordance with various embodiments. The CNN 3200 may be at least part of a DNN that can be used for image classification or other types of Al tasks. For the purpose of illustration, the CNN 3200 includes a sequence of layers comprising a plurality of convolutional layers 3210 (individually referred to as "convolutional layer 3210"), a plurality of pooling layers 3220 (individually referred to as "pooling layer 3220"), and a plurality of fully-connected layers 3230 (individually referred to as "fully-connected layer 3230"). In other embodiments, the CNN 3200 may include fewer, more, or different layers. In an execution of the CNN 3200, the layers of the CNN 3200 execute tensor computation that includes many tensor operations, such as convolutions, interpolations, pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0220] The convolutional layers 3210 summarize the presence of features in inputs to the CNN 3200. The convolutional layers 3210 function as feature extractors. The first layer of the CNN 3200 is a convolutional layer 3210. In an example, a convolutional layer 3210 performs a convolution on an input tensor 3240 (also referred to as IFM 3240) and a filter 3250. As shown in FIG. 32, the IFM 3240 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 3240 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 3250 is represented by a 3x3x3 3D matrix. The filter 3250 includes 3 kernels, each of which may correspond to a different input channel of the IFM 3240. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 32, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 3250 in extracting features from the IFM 3240.

[0221] The convolution includes multiply-accumulate (MAC) operations with the input elements in the IFM 3240 and the weights in the filter 3250. The convolution may be a standard convolution 3263 or a depthwise convolution 3283. In the standard convolution 3263, the whole filter 3250 slides across the IFM 3240. All the input channels are combined to produce an output tensor 3260 (also referred to as output feature map (OFM) 3260). The OFM 3260 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 32. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 3260.

[0222] The multiplication applied between a kernel-sized patch of the IFM 3240 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernel-sized patch of the IFM 3240 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the "scalar product." Using a kernel smaller than the IFM 3240 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 3240 multiple times at different points on the IFM 3240. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 3240, left to right, top to bottom. The result from multiplying the kernel with the IFM 3240 one time is a single value. As the kernel is applied multiple times to the IFM 3240, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 3260) from the standard convolution 3263 is referred to as an OFM.

[0223] In the depthwise convolution 3283, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 32, the depthwise convolution 3283 produces a depthwise output tensor 3280. The depthwise output tensor 3280 is represented by a 5x5x3 3D matrix. The depthwise output tensor 3280 includes 3 output channels, each of which is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 3240 and a kernel of the filter 3250. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 3293 is then performed on the depthwise output tensor 3280 and a 1x1x3 tensor 3290 to produce the OFM 3260. The tensor 3290 is a ID tensor.

[0224] The OFM 3260 is then passed to the next layer in the sequence. In some embodiments, the OFM 3260 is passed through an activation function. An example activation function is ReLU. ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 3210 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 3260 is passed to the subsequent convolutional layer 3210 (i.e., the convolutional layer 3210 following the convolutional layer 3210 generating the OFM 3260 in the sequence). The subsequent convolutional layers 3210 perform a convolution on the OFM 3260 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 3210, and so on.

[0225] In some embodiments, a convolutional layer 3210 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 3210). The convolutional layers 3210 may perform various types of convolutions, such as 2D convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The CNN 3200 includes 326 convolutional layers 3210. In other embodiments, the CNN 3200 may include a different number of convolutional layers.

[0226] The pooling layers 3220 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 3220 is placed between two convolutional layers 3210: a preceding convolutional layer 3210 (the convolutional layer 3210 preceding the pooling layer 3220 in the sequence of layers) and a subsequent convolutional layer 3210 (the convolutional layer 3210 subsequent to the pooling layer 3220 in the sequence of layers). In some embodiments, a pooling layer 3220 is added after a convolutional layer 3210, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 3260.

[0227] A pooling layer 3220 receives feature maps generated by the preceding convolutional layer 3210 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layers 3220 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 3220 applied to a feature map of 32x6 results in an output pooled feature map of 3x3. The output of the pooling layer 3220 is input into the subsequent convolutional layer 3210 for further feature extraction. In some embodiments,the pooling layer 3220 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0228] The fully-connected layers 3230 are the last layers of the DNN. The fully-connected layers 3230 may be convolutional or not. The fully-connected layers 3230 receive an input operand. The input operand defines the output of the convolutional layers 3210 and pooling layers 3220 and includes the values of the last feature map generated by the last pooling layer 3220 in the sequence. The fully-connected layers 3230 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1, and the sum of all may be one. These probabilities are calculated by the last fully-connected layer 3230 by using a logistic function (binary classification) or a SoftMax function (multiclass classification) as an activation function. In some embodiments, the fully-connected layers 3230 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.

[0229] FIG. 33 is a flowchart of a method 3300 of executing a DNN, in accordance with various embodiments. The method 3300 may be performed by the NPU 302 in FIG. 3. Although the method 3300 is described with reference to the flowchart illustrated in FIG. 33, many other methods for executing DNNs may alternatively be used. For example, the order of execution of the steps in FIG. 33 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0230] The NPU 302 stores 3310 quantized weight groups of a neural network layer. The quantized weight groups are generated by quantizing different portions of a weight tensor of the neural network layer to different data precisions using group-specific quantization parameters. In some embodiments, the quantized weight groups are generated by a compiler, such as the compiler 430. In some embodiments, the NPU 302 stores the quantized weight groups in the memory 310 or local memory 340.

[0231] In some embodiments, the NPU 302 stores a microscaling block for a quantized weight group. In some embodiments, the NPU 302 stores the microscaling block by storing quantized weights in the quantized weight group as scalar elements and storing a shared scale used for generating the quantized weight group from a portion of the weight tensor. Insome embodiments, the NPU 302 stores the microscaling block by further storing one or more parameters indicating a data format of the shared scale, a data format of the scalar elements, or a block size. The block size is a total number of the quantized weights in the quantized weight group. In some embodiments, the NPU 302 selects an input channel dimension of the weight tensor as a dimension for sharing the shared scale.

[0232] In some embodiments, the NPU 302 generates the microscaling block from a portion of the weight tensor. The NPU 302 identifies a maximum exponent of weights in the portion of the weight tensor. The NPU 302 maps the maximum exponent to a largest binade in the data format of the scalar elements. The NPU 302 computes values by dividing the weights with the shared scale. The NPU 302 determines whether any of the values exceeds a representable range of the data format of the scalar elements. In response to determining that a value exceeds the representable range, the NPU 302 changes the value to a maximum representable value of the data format of the scalar elements. The maximum representable value is stored as a scalar element of the microscaling block. In response to determining that another value falls within the representable range, the NPU 302 stores the other value as another scalar element of the microscaling block.

[0233] The NPU 302 computes 3320 a plurality of weight groups by converting the different data predictions of the quantized weight groups to the same data precision. In some embodiments, the NPU 302 computes the plurality of weight groups using different dequantization parameters.

[0234] The NPU 302 performs 3330 a plurality of neural network operation on the plurality of weight groups and a plurality of activations groups. A neural network operation is performed on a weight group and an activation group. In some embodiments, the plurality of activation groups has the same data precision as the plurality of weight groups. In some embodiments, the plurality of neural network operation is a group convolution. In some embodiments, the NPU 302 computes the plurality of activations groups by dequantizing activations of the neural network layer.

[0235] The NPU 302 computes 3340 an output tensor of the neural network layer by accumulating results of the plurality of neural network operations. In some embodiments, the results of the plurality of neural network operations are tensors having the same spatial shape as the output tensor. The NPU 302 computes the output tensor by performing an elementwise addition on the tensors.

[0236] FIG. 34 is a block diagram of an example computing device 3400, in accordance with various embodiments. In some embodiments, the computing device 3400 can be used as at least part of the Al system 300. A number of components are illustrated in FIG. 34 as included in the computing device 3400, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 3400 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 3400 may not include one or more of the components illustrated in FIG. 34, but the computing device 3400 may include interface circuitry for coupling to the one or more components. For example, the computing device 3400 may not include a display device 3406, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 3406 may be coupled. In another set of examples, the computing device 3400 may not include an audio input device 3418 or an audio output device 3408 but may include audio input or output device interface circuitry to which an audio input device 3418 or audio output device 3408 may be coupled.

[0237] The computing device 3400 may include a processing device 3402 (e.g., one or more processing devices). The processing device 3402 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 3400 may include a memory 3404, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 3404 may include memory that shares a die with the processing device 3402. In some embodiments, the memory 3404 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for executing DNNs (e.g., the method 3300 described in conjunction with FIG. 33) or some operations performed by one or more components of the Al system 300. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 3402.

[0238] In some embodiments, the computing device 3400 may include a communication chip 3412 (e.g., one or more communication chips). For example, the communication chip3412 may be configured for managing wireless communications for the transfer of data to and from the computing device 3400. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0239] The communication chip 3412 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 3412 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 3412 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 3412 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 3412 may operate in accordance with other wireless protocols in other embodiments. The computing device 3400 may include an antenna 3422 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0240] In some embodiments, the communication chip 3412 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 3412 may include multiplecommunication chips. For instance, a first communication chip 3412 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 3412 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 3412 may be dedicated to wireless communications, and a second communication chip 3412 may be dedicated to wired communications.

[0241] The computing device 3400 may include battery / power circuitry 3414. The battery / power circuitry 3414 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 3400 to an energy source separate from the computing device 3400 (e.g., AC line power).

[0242] The computing device 3400 may include a display device 3406 (or corresponding interface circuitry, as discussed above). The display device 3406 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0243] The computing device 3400 may include an audio output device 3408 (or corresponding interface circuitry, as discussed above). The audio output device 3408 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0244] The computing device 3400 may include an audio input device 3418 (or corresponding interface circuitry, as discussed above). The audio input device 3418 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0245] The computing device 3400 may include a GPS device 3416 (or corresponding interface circuitry, as discussed above). The GPS device 3416 may be in communication with a satellite-based system and may receive a location of the computing device 3400, as known in the art.

[0246] The computing device 3400 may include another output device 3410 (or corresponding interface circuitry, as discussed above). Examples of the other output device3410 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0247] The computing device 3400 may include another input device 3420 (or corresponding interface circuitry, as discussed above). Examples of the other input device 3420 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0248] The computing device 3400 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 3400 may be any other electronic device that processes data.

[0249] The following paragraphs provide various examples of the embodiments disclosed herein.

[0250] Example 1 provides an apparatus, including a processing engine including one or more processing elements, a processing element including a multiplier and an adder, the processing engine to: receive a plurality of quantized channel groups, the plurality of quantized channel groups generated by partitioning an input tensor of a neural network layer into a plurality of channel groups and quantizing the plurality of channel groups to different data precisions, and compute a plurality of output subtensors from the plurality of quantized channel groups; and a post-processing engine to compute a plurality of dequantized subtensors by dequantizing the plurality of output subtensors, in which an output tensor of the neural network layer is computed by accumulating the plurality of dequantized subtensors.

[0251] Example 2 provides the apparatus of example 1, in which the output tensor is computed by the processing engine.

[0252] Example 3 provides the apparatus of example 2, further including a memory to store the plurality of quantized channel groups and the plurality of dequantized subtensors; a first datapath; a second datapath; and an input delivery unit to: load the plurality of quantized channel groups from the memory to the processing engine through the first datapath, and load the plurality of dequantized subtensors from the memory to the processing engine through the second datapath.

[0253] Example 4 provides the apparatus of example 3, in which the second datapath is a 32-bit datapath.

[0254] Example 5 provides the apparatus of example 3 or 4, further including an output delivery unit to write the plurality of dequantized subtensors from the post-processing engine into the memory.

[0255] Example 6 provides the apparatus of any one of examples 2-5, in which the processing element further includes a MUX and a concatenator, the MUX to select a data element from two data elements in two dequantized subtensors and to forward the selected data element to the concatenator for computing the output tensor.

[0256] Example 7 provides the apparatus of example 1, in which the output tensor is computed by the post-processing engine.

[0257] Example 8 provides the apparatus of example 7, in which the post-processing engine includes a plurality of post-processing elements, a post-processing element including a multiplier, a MUX, and an adder.

[0258] Example 9 provides the apparatus of example 7 or 8, in which the processing engine including a plurality of processing element groups, in which each of the plurality of quantized channel groups is partitioned into a plurality of channel subgroups, a processing element group to receive a channel subgroup from each of the plurality of quantized channel groups.

[0259] Example 10 provides the apparatus of example 9, in which different processing elements in the processing element group are to receive channel subgroups from different ones of the plurality of quantized channel groups.

[0260] Example 11 provides an apparatus, including a data processing unit, including a processing engine to compute a plurality of output subtensors from a plurality of quantized channel groups, respectively, in which the plurality of quantized channel groups are generated by partitioning an input tensor of a neural network layer into a plurality ofchannel groups and quantizing the plurality of channel groups, and a post-processing engine to compute a plurality of dequantized subtensors by dequantizing the plurality of output subtensors; and a digital signal processor to compute an output tensor of the neural network layer by accumulating the plurality of dequantized subtensors.

[0261] Example 12 provides the apparatus of example 11, further including a memory to store the plurality of quantized channel groups and the plurality of dequantized subtensors, in which the plurality of quantized channel groups are transmitted from the memory to the data processing unit, and the plurality of dequantized subtensors are transmitted from the data processing unit to the memory and from the memory to the digital signal processor.

[0262] Example 13 provides the apparatus of example 12, in which the data processing unit further includes an input delivery unit to load the plurality of quantized channel groups from the memory to the processing engine; and an output delivery unit to drain the plurality of dequantized subtensors from the post-processing engine into the memory.

[0263] Example 14 provides the apparatus of example 13, in which the memory is to further store a plurality of dequantization parameters, a dequantization parameter specific to a particular channel group within the input tensor, in which the input delivery unit is further to load the plurality of dequantization parameters into the post-processing engine.

[0264] Example 15 provides the apparatus of example 14, in which the memory is to further store a plurality of weight groups corresponding to the plurality of channel groups, in which the input delivery unit is further to load a weight group corresponding to a channel group into the processing engine and to load a dequantization parameter specific to the channel group into the post-processing engine in a same loading cycle.

[0265] Example 16 provides the apparatus of any one of examples 11-15, in which the processing engine is to compute the plurality of output subtensors by performing a plurality of workloads and the digital signal processor to compute the output tensor by performing a plurality of workloads, in which a first workload of the digital signal processor is performed after a first workload of the processing engine.

[0266] Example 17 provides the apparatus of example 16, in which another workload of the digital signal processor and another workload of the processing engine are at least partially performed in a same clock cycle.

[0267] Example 18 provides a method of executing a neural network, the method including storing quantized weight groups of a neural network layer, the quantized weight groupsgenerated by quantizing different portions of a weight tensor of the neural network layer to different data precisions using group-specific quantization parameters; computing a plurality of weight groups by converting the different data predictions of the quantized weight groups to a same data precision; performing a plurality of neural network operation on the plurality of weight groups and a plurality of activations groups, a neural network operation performed on a weight group and an activation group; and computing an output tensor of the neural network layer by accumulating results of the plurality of neural network operations.

[0268] Example 19 provides the method of example 18, in which the plurality of activation groups have the same data precision.

[0269] Example 20 provides the method of example 18, further including computing the plurality of activations groups by dequantizing activations of the neural network layer.

[0270] Example 21 provides the method of any one of examples 18-20, in which storing the quantized weight groups including storing a microscaling block for a quantized weight group, storing the microscaling block including storing quantized weights in the quantized weight group as scalar elements, and storing a shared scale used for generating the quantized weight group from a portion of the weight tensor.

[0271] Example 22 provides the method of example 21, in which storing the microscaling block further includes storing one or more parameters indicating a data format of the shared scale, a data format of the scalar elements, or a block size, in which the block size is a total number of the quantized weights in the quantized weight group.

[0272] Example 23 provides the method of example 22, further including generating the microscaling block from a portion of the weight tensor by: identifying a maximum exponent of weights in the portion of the weight tensor, mapping the maximum exponent to a largest binade in the data format of the scalar elements, computing values by dividing the weights with the shared scale, determining whether any of the values exceeds a representable range of the data format of the scalar elements, and in response to determining that a value exceeds the representable range, changing the value to a maximum representable value of the data format of the scalar elements, in which the maximum representable value is stored as a scalar element of the microscaling block.

[0273] Example 24 provides the method of example 23, in which the microscaling block is generated further by: in response to determining that another value falls within therepresentable range, storing the another value as another scalar element of the microscaling block.

[0274] Example 25 provides the method of any one of examples 21-24, further including selecting an input channel dimension of the weight tensor as a dimension for sharing the shared scale.

[0275] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art can recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. An apparatus, comprising: a processing engine comprising one or more processing elements, the processing engine to: receive a plurality of quantized channel groups, the plurality of quantized channel groups generated by partitioning an input tensor of a neural network layer into a plurality of channel groups and quantizing the plurality of channel groups to different data precisions, and compute a plurality of output subtensors from the plurality of quantized channel groups; and a post-processing engine to: compute a plurality of dequantized subtensors by dequantizing the plurality of output subtensors, wherein an output tensor of the neural network layer is computed by accumulating the plurality of dequantized subtensors.

2. The apparatus of claim 1, wherein the output tensor is computed by the processing engine.

3. The apparatus of claim 2, further comprising: a memory to store the plurality of quantized channel groups and the plurality of dequantized subtensors; a first datapath; a second datapath; and an input delivery unit to: load the plurality of quantized channel groups from the memory to the processing engine through the first datapath, and load the plurality of dequantized subtensors from the memory to the processing engine through the second datapath.

4. The apparatus of claim 3, wherein the second datapath is a 32-bit datapath.

5. The apparatus of claim 3 or 4, further comprising: an output delivery unit to write the plurality of dequantized subtensors from the post-processing engine into the memory.

6. The apparatus of any one of claims 2-5, wherein the processing element further comprises a MUX and a concatenator, the MUX to select a data element from two data elements in two dequantized subtensors and to forward the selected data element to the concatenator for computing the output tensor.

7. The apparatus of claim 1, wherein the output tensor is computed by the post-processing engine.

8. The apparatus of claim 7, wherein the post-processing engine comprises a plurality of post-processing elements, a post-processing element comprising a multiplier, a MUX, and an adder.

9. The apparatus of claim 7 or 8, wherein the processing engine comprising a plurality of processing element groups, wherein each of the plurality of quantized channel groups is partitioned into a plurality of channel subgroups, a processing element group to receive a channel subgroup from each of the plurality of quantized channel groups.

10. The apparatus of claim 9, wherein different processing elements in the processing element group are to receive channel subgroups from different ones of the plurality of quantized channel groups.

11. An apparatus, comprising: a data processing unit, comprising: a processing engine to compute a plurality of output subtensors from a plurality of quantized channel groups, respectively, wherein the plurality of quantized channel groups are generated by partitioning an input tensor of aneural network layer into a plurality of channel groups and quantizing the plurality of channel groups, and a post-processing engine to compute a plurality of dequantized subtensors by dequantizing the plurality of output subtensors; and a digital signal processor to compute an output tensor of the neural network layer by accumulating the plurality of dequantized subtensors.

12. The apparatus of claim 11, further comprising: a memory to store the plurality of quantized channel groups and the plurality of dequantized subtensors, wherein the plurality of quantized channel groups are transmitted from the memory to the data processing unit, and the plurality of dequantized subtensors are transmitted from the data processing unit to the memory and from the memory to the digital signal processor.

13. The apparatus of claim 12, wherein the data processing unit further comprises: an input delivery unit to load the plurality of quantized channel groups from the memory to the processing engine; and an output delivery unit to drain the plurality of dequantized subtensors from the post-processing engine into the memory.

14. The apparatus of claim 13, wherein the memory is to further store a plurality of dequantization parameters, a dequantization parameter specific to a particular channel group within the input tensor, wherein the input delivery unit is further to load the plurality of dequantization parameters into the post-processing engine.

15. The apparatus of claim 14, wherein the memory is to further store a plurality of weight groups corresponding to the plurality of channel groups, wherein the input delivery unit is further to load a weight group corresponding to a channel group into the processing engine and to load a dequantization parameter specific to the channel group into the postprocessing engine in a same loading cycle.

16. The apparatus of any one of claims 11-15, wherein the processing engine is to compute the plurality of output subtensors by performing a plurality of workloads and the digital signal processor to compute the output tensor by performing a plurality of workloads, wherein a first workload of the digital signal processor is performed after a first workload of the processing engine.

17. The apparatus of claim 16, wherein another workload of the digital signal processor and another workload of the processing engine are at least partially performed in a same clock cycle.

18. A method of executing a neural network, the method comprising: storing quantized weight groups of a neural network layer, the quantized weight groups generated by quantizing different portions of a weight tensor of the neural network layer to different data precisions using group-specific quantization parameters; computing a plurality of weight groups by converting the different data predictions of the quantized weight groups to a same data precision; performing a plurality of neural network operation on the plurality of weight groups and a plurality of activations groups, a neural network operation performed on a weight group and an activation group; and computing an output tensor of the neural network layer by accumulating results of the plurality of neural network operations.

19. The method of claim 18, wherein the plurality of activation groups have the same data precision.

20. The method of claim 18, further comprising: computing the plurality of activations groups by dequantizing activations of the neural network layer.

21. The method of any one of claims 18-20, wherein storing the quantized weight groups comprising:storing a microscaling block for a quantized weight group, storing the microscaling block comprising: storing quantized weights in the quantized weight group as scalar elements, and storing a shared scale used for generating the quantized weight group from a portion of the weight tensor.

22. The method of claim 21, wherein storing the microscaling block further comprises: storing one or more parameters indicating a data format of the shared scale, a data format of the scalar elements, or a block size, wherein the block size is a total number of the quantized weights in the quantized weight group.

23. The method of claim 22, further comprising generating the microscaling block from a portion of the weight tensor by: identifying a maximum exponent of weights in the portion of the weight tensor, mapping the maximum exponent to a largest binade in the data format of the scalar elements, computing values by dividing the weights with the shared scale, determining whether any of the values exceeds a representable range of the data format of the scalar elements, and in response to determining that a value exceeds the representable range, changing the value to a maximum representable value of the data format of the scalar elements, wherein the maximum representable value is stored as a scalar element of the microscaling block.

24. The method of claim 23, wherein the microscaling block is generated further by: in response to determining that another value falls within the representable range, storing the another value as another scalar element of the microscaling block.

25. The method of any one of claims 21-24, further comprising:selecting an input channel dimension of the weight tensor as a dimension for sharing the shared scale.

Images