Configurable multi-precision converter in neural network accelerator

Abstract

A deep neural network (DNN) accelerator may include one or more multi-precision converters for precision conversion during DNN execution. A multi-precision converter may include an input extractor for extracting an exponent and mantissa from an input value, an input generator for generating an intermediate exponent and an intermediate mantissa from the extracted exponent and the extracted mantissa, a group of exponent converts for converting the intermediate exponent to exponents of different input or output precisions, an exponent selector for selecting one of the exponent converters for a particular input or output precision, a group of mantissa converters for converting the intermediate mantissa to mantissas of different input or output precisions, a mantissa selector for selecting one of the mantissa converter for a particular input or output precision. An output value is generated from the exponent generated in the selected exponent converter and the mantissa generated in the selected mantissa converter.

Description

CONFIGURABLE MULTI-PRECISION CONVERTER IN NEURAL NETWORK ACCELERATORTechnical Field

[0001] This disclosure relates generally to neural network (also referred to as "deep neural network" or "DNN"), and more specifically, configurable multi-precision converter in DNN accelerators.Background

[0002] DNNs, due to their ability to achieve high accuracy, are used extensively for a variety of artificial intelligence (Al) applications, including image processing, scene understanding, language modeling, language translation, speech processing, gaming and many others.However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0003] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0004] FIG. 1 illustrates an example transformer model, in accordance with various embodiments.

[0005] FIG. 2 illustrates an example convolutional neural network (CNN), in accordance with various embodiments.

[0006] FIG. 3 is a block diagram of a DNN system, in accordance with various embodiments.

[0007] FIG. 4 is a block diagram of a DNN module, in accordance with various embodiments.

[0008] FIG. 5 illustrates an example compute engine, in accordance with various embodiments.

[0009] FIG. 6 illustrates an example multi-precision converter, in accordance with various embodiments.

[0010] FIG. 7 illustrates an example input extractor, in accordance with various embodiments.

[0011] FIG. 8 illustrates an example input generator, in accordance with various embodiments.

[0012] FIGS. 9A-9D illustrate exponent converters, in accordance with various embodiments.

[0013] FIGS. 10A-10D illustrate mantissa converters, in accordance with various embodiments.

[0014] FIG. 11 illustrates an example sparse cell, in accordance with various embodiments.

[0015] FIG. 12 illustrates an example sparse cell array, in accordance with various embodiments.

[0016] FIG. 13 illustrates an example processing element (PE), in accordance with various embodiments.

[0017] FIG. 14 is a flowchart of a method of converting precisions of data in DNNs, in accordance with various embodiments.

[0018] FIG. 15 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0019] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0020] Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as "input feature map (IFM)" or "input activation tensor") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor). A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

[0021] Tensors in DNNs can be saved in X-major (e.g., XYZ or XZY format), Y-major formats (e.g., YXZ or YZX format), or Z-major formats (e.g., ZXY or ZYX format). The format of a tensor may define the order in which the data points in the tensor are stored, written, or read. The first character may represent the dimension in which data points are contiguous in memory. The second character may represent the dimension in which data points can be accessed after the contiguous data points are accessed in memory. The third character may represent the dimension in which data points are accessed after the data points in the dimension represented by the second character are exhausted. Taking the ZXY format for example, the access order first starts in the Z dimension, then moves to the X dimension, and finally moves to the Y dimension. Data points in the tensor are contiguous in memory in the Z dimension, meaning data points having the same (x, y) coordinates are contiguous in memory. Using tensor permutation, the tensor may be read from memory in a different format.

[0022] The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability. DNN models may be executed, e.g., for training or inference, by DNN accelerators.

[0023] DNN accelerators are typically designed to speed up the process of carrying out DNNs. The process of carrying out DNNs is a process of executing DNNs that includes executing the layers or neural network operations in the DNNs. Some DNN accelerators arealso referred to as neural processing units (NPUs), neural engine, Al processors, or Al processing units. An example DNN accelerator includes a data processing unit (DPU) providing computation capacity, a memory providing data storage capacity, and a direct memory access (DMA) module allowing pipelined movement of parameters and data between the memory and a memory local to the DPU ("local memory"). The DPU is a compute unit, which may also be referred to as "compute block" or "compute tile." Some DNN accelerators may have multiple DPUs, i.e., multiple tiles. A DPU may include an array of PEs, a data signal processor (DSP), a local memory, other components, or some combination thereof.

[0024] Data in DNNs may have various precisions. Different data precisions may correspond to different data types or different data formats. Example data formats include fixed-point formats (e.g., INT4, INT8, INT16, INT32, etc.), floating point formats (e.g., FP16, BP16, FP32, etc.), and so on. INT stands for integer. FP stands for floating point. BP stands for brain floating point. The number in a data format may indicate the number of bits needed for representing a value having the data format. For instance, INT8 indicates an integer format or fixed-point format in which a value is represented by 8 bits, while FP16 indicates a floating point format in which a single value is represented by 16 bits. The data precision may be higher when the number in the data format is greater. A floating point format may have a fixed number of bits for sign, exponent, and mantissa, respectively. For instance, FP32 has 1 sign bit, 8 exponent bits, and 23 mantissa bits; FP16 has 1 sign bit, 5 exponent bits, and 10 mantissa bits; BF16 has 1 sign bit, 8 exponent bits, and 7 mantissa bits. There can be other floating point formats. As higher data precision may correspond to more bits for a single data element, DNN execution may be faster with lower data precision but lower data precision usually leads to lower accuracy of DNNs.

[0025] The complexity of DNNs has been increasing exponentially to achieve desirable results and accuracies, which in turn increases the compute complexity and resource requirements. One of the widely used techniques to minimize the impact of the ever- increasing compute requirement is to use lower precision arithmetic. Using lower data precision can decrease the required amount of memory and shorten or accelerate the DNN inference / training time. Though reduced precision may affect the accuracy of targeted tasks or applications and quantum of impact on accuracy could vary based on the operator stages with reduced precision.

[0026] Many Al processors, central processing units (CPUT), and graphics processing units (GPUs) can employ multiple precision computing hardware to reap the benefits of the above-mentioned scheme while maintaining targeted accuracy for DNN applications. It usually requires precision converters, which can convert from one precision to another one, at different stages in the hardware pipeline. Conventional Al processors typically use dedicated hardware blocks to achieve data precision conversion. These hardware blocks are specifically designed to handle a particular type of floating point format to another floating point format (one to one conversion). Which means for every other precision conversion requirement, an additional dedicated hardware block is needed. This requirement keeps on increasing with the evolution and exploration of new DNN models and poses a significant restriction for leveraging the potential benefits of the new DNN models. Further for cases where the input data precisions are not mappable directly on to the converter inputs, software would need to intervene and do the first level type conversion such that the data maps on to converter input. Taking an arithmetic unit designed for converting FP32 to FP16 for example, when the input data is in a different format from FP32, software is used to convert the native input precision to FP32 and then the arithmetic unit is used to convert FP32 to FP16. On the output side, a separate FP16 to FP32 converter is used to convert the output of low precision arithmetic back to the native precision.

[0027] Because currently available data converters are generally designed to satisfy a single input to single output conversion (i.e., one to one precision conversion), they are not desirably efficient with respect to area or power to meet the requirement for multiprecision data conversion (e.g., one-to-many, many-to-one, or many-to-many precision conversion). Also, as described above, software is needed to intervene and perform precision conversion to or from hardware supported data precision in absence of native hardware support for multi-precision data conversion. Therefore, these data converters can degrade the performance and power / energy efficiency of the processor.

[0028] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing DNN accelerators with configurable multi-precision converters. An example multi-precision converter in the present disclosure is capable of many-to-many precision conversion and can convert input data in various precisions to output data in various precisions. Such a multi-precision converter may be a hardware block that can be driven and configured by an instruction set architecture.

[0029] In various embodiments of the present disclosure, a DNN accelerator may include one or more multi-precision converters. A multi-precision converter may be capable of one- to-one, one-to-many, many-to-one, and many-to-many precision conversions. A multiprecision converter can convert one or more input precisions (i.e., precisions of data input into the multi-precision converter, also referred to as "input data precision") to one or more output precisions (i.e., precisions of data output from the multi-precision converter, also referred to as "output data precision"). Precision conversion may be performed during DNN execution for training or inference. Some multi-precision converters may output higher precisions compared with their inputs, and such multi-precision converters are referred to as upward multi-precision converters. Other multi-precision converters may output lower precisions compared with their inputs, and such multi-precision converters are referred to as downward multi-precision converters. In an example, a post-processing engine in the DNN accelerator may include an upward multi-precision converter for increasing the precisions of data generated in a processing engine of the DNN accelerator. After the precisions conversion, the post-processing engine may perform one or more operations on the data. The post-processing engine may also include a downward multi-precision converter for reducing the precisions of data generated in the post-processing engine before the data is sent to the processing engine for further computation.

[0030] An example multi-precision converter may include a plurality of logic units, such as an input extractor, an input generator, an exponent selector, a group of exponent converters, an exponent collector, a mantissa selector, a group of mantissa converters, a mantissa collector, a shifter, and a storage unit. The input extractor may extract exponent bits and mantissa bits from input data, such as a value computed in a neural network operation of a DNN. The input generator may generate an intermediate exponent from the extracted exponent bits and generate an intermedia mantissa from the extracted mantissa bits based on a data transfer interface in the multi-precision converter. The exponent converts can convert the intermediate exponent to exponents of different precisions. The exponent selector may select an exponent converter from the group of exponent converters for the output precision. The selected exponent converter may be used while the other exponent converters may be in idle states. The exponent collector may generate an output exponent using the converted exponent generated in the selected exponent converter, e.g., through overflow handling, bias handling, packing, etc. The mantissa converters can convertthe intermediate mantissa to mantissas of different precisions. The mantissa selector may select a mantissa converter from the group of mantissa converters for the output precision. The selected mantissa converter may be used while the other mantissa converters may be in idle states. The mantissa collector may generate an output mantissa using the converted mantissa generated in the selected mantissa converter, e.g., through denormalization, bit padding, and so on. Some of the logic units (e.g., the input extractor, input generator, exponent selector, mantissa selector, exponent collector, and mantissa collector) in the multi-precision converter may be used for multiple input or output precisions, while the other logic units (e.g., an individual exponent converter or an individual mantissa converter) may be specific to a particular input precision or output precision.

[0031] The present disclosure introduces area-efficient configurable micro-architecture for multi-precision data conversion that includes separate paths (e.g., control paths or data paths) for mantissa and exponent conversion with each path supporting multiple bitprecisions enabling configurability for any data (input data or output data) format from the set of cross-product of supported mantissa and exponent bit-precisions. Furthermore, common and shared hardware resources are designed for both exponent and mantissa data paths to further reduce area requirements. The novel, unconventional circuitry including the input extractor and input generator can handle any input data precision and transform it to the desired format by meeting the hardware interface requirements which can be consumed by the hardware efficiently. The hardware can operate with programmable instructions for both up conversion and down conversion. Also, the area-efficient configurable micro-architecture can provide flexibility to share the resources which required different precision conversions. For example, GPUs with half precision and single precision compute pipelines can share common converter. The design is fully pipelined to allow free flow of the data through the multi-precision converter such that it can be utilized to perform on the fly data conversion while consumed by the compute engines. Exponent converters and mantissa converters may be added or removed from the architecture based on requirements of particular applications, making it an easy portable option across different types of devices, ranging from servers to personal computers, handheld devices, phones, headsets, and so on. The components in the multi-precision converter can be specifically designed to keep the area and power budget as minimum as possible. Also, the overalllevels of logic in the multi-precision converter can be kept under check to ensure that the design operates at a higher frequency target.

[0032] Compared with currently available precision converters used for DNN execution, the multi-precision converters in the present disclosure can provide better performance and better efficiency in terms of area and power. For instance, a multi-precision converter in the present disclosure can handle multiple input or output precisions without the requirement of using software for precision conversion. Also, as some of the logic units in the multiprecision converter can be used for multiple input or output precisions, less area and power would be consumed for multiple input or output precisions, compared with currently available precision conversion techniques that require a whole set of logic units for every precision.

[0033] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0034] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0035] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0036] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (Band C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0037] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0038] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0039] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0040] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0041] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.Example DNNs

[0042] FIG. 1 illustrates an example transformer model 100, in accordance with various embodiments. The transformer model 100 may transform input sequences into output sequences. In some embodiments, the transformer model 100 is a neural network that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 100 may be an LLM. The transformer model 100 includes an encoder block 110, a decoder block 120, and a head block 130. In other embodiment, different or additional components may be included in the transformer model 100. Further, functionality attributed to a component of the transformer model 100 may be accomplished by a different component included in the transformer model 100 or a different model or module.

[0043] The encoder block 110 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 1, the encoder block 110 receives inputs 101 and generates encoder outputs 102. In some embodiments, the inputs 101 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the inputs 101 may include a prompt received from a user of the transformer model 100. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder outputs 102 may include one or more vectors that are contextualized representations of the input 101. Each vector in the encoder outputs 102 may represent a token in the input 101 with contextual understanding.

[0044] The encoder block 110 includes an embedding layer 113, a positional encoding layer 115, and a plurality of layers 140 (individually referred to as "layer 140"). In other embodiments, the encoder block 110 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 110 may be different from the arrangement shown in FIG. 1. For the purpose of illustration, the encoder block 110 has N layers in FIG. 1, where N is an integer. Each layer 140 may include one or more neural network operations. The layers 140 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 101. Different layers 140 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 140 have identical components. The components in a layer 140 may be layers and may also be referred to assub-layers of the layer 140. As shown in FIG. 1, a layer 140 includes four sub-layers: an MHA layer 141, an add & norm layer 142, a feed forward layer 143, and another add & norm layer 144.

[0045] The decoder block 120 iteratively generates outputs 103 using encoded representations generated by the encoder block 110. The decoder block 120 includes an embedding layer 123, a positional encoding layer 125, and a plurality of layers 150 (individually referred to as "layer 150"). For the purpose of illustration, the decoder block 120 has N layers in FIG. 1, where N is an integer. In the embodiments of FIG. 1, the number of layers 150 in the decoder block 120 is the same as the number of layers 140 in the encoder block 110. In other embodiments, the number of layers 150 in the decoder block 120 may be different from the number of layers 140 in the encoder block 110. Each layer 150 may include one or more neural network operations. Different layers 150 may have different internal parameters. In some embodiments, the layers 150 may have identical components. The components in a layer 150 may be layers and may also be referred to as sub-layers of the layer 150. As shown in FIG. 1, a layer 150 includes six sub-layers: an MHA layer 151, an add & norm layer 152, an encoder-decoder attention layer 153, another add & norm layer 154, a feed forward layer 155, and another add & norm layer 156.

[0046] In some embodiments, a sequence of inference phases is performed in the decoder block 120 using encoder outputs, e.g., the encoder outputs 102. A matrix may be predicted through each inference phase. The outputs 103 may include a plurality of matrices. Each matrix may be further processed in the head block 130 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference phase, the decoder block 120 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 110. The first matrix may be used by the head block 130 to predict a first token. The predicted token may be used as a new input token, in addition to the start token(s), in the second inference phase. Similarly, a second token may be predicted through the second inference phase and may be used in the third inference phase. This iteration may continue till all the inference phases are complete.

[0047] The head block 130 receives the output of the decoder block 120 and processes it in a linear layer 133 and a SoftMax layer 135. A linear operation may be performed on the output of the decoder block 120 in the linear layer 133. The linear operation may include a multiplication of the output of the decoder block 120 with a weight matrix. The output ofthe linear layer 133 may be a vector. In some embodiments, the head block 130 may function as a classifier. The number of data elements in the vector computed in the linear layer 133 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 133 may have M data elements representing the prediction for the M classes, respectively.

[0048] The output of the linear layer 133 may be input into the SoftMax layer 135. A SoftMax function may be applied on the output of the linear layer 133 to compute probability scores. A probability score may have a value in the range from 0 to 1. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 133. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 100 predicts as the next in the sequence. The final output of the transformer model 100 may be the sequence of predicted tokens. In some embodiments, the head block 130 may be a language modeling head.

[0049] An embedding layer (e.g., the embedding layer 113 or the embedding layer 123) converts an input of the embedding layer (e.g., the inputs 101 or the outputs 103) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 113 may generate a plurality of embeddings, each of which may be converted from a different input token in the inputs 101. The embeddings may capture the semantic meaning of the tokens in the input 101. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 101 is a prompt including a sequence of words, the embedding layer 113 may generate an embedding from each word in the input 101. The embedding layer 123 in the decoder block 120 may generate a plurality of embeddings from tokens received by the decoder block 120 in a similar manner as the embedding layer 113.

[0050] A positional encoding layer (e.g., the positional encoding layer 115 or the positional encoding layer 125) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 104 or positional encoding vector 105) on vector embeddings from the corresponding embeddinglayer to generate new vector embeddings that represents the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer.

[0051] An MHA layer (e.g., the MHA layer 141, the MHA layer 151, or the MHA layer 153) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 141 or the MHA layer 151 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 141, the queries, keys, and values may all come from the positional encoding layer 115. For the MHA layer 151, the queries, keys, and values may all come from the positional encoding layer 125. The self-attention mechanism may enable the transformer model 100 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0052] In some embodiments, the queries, keys, and values input into the MHA layer 141 may be computed from vector embeddings generated by the positional encoding layer 115. The queries, keys, and values input into the MHA layer 151 may be computed from vector embeddings generated by the positional encoding layer 125. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G IRWx / lmay be computed by multiply an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG IRdx / l, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K G IRWx / lmay be computed by multiple an embedding matrixX G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkG IRdx / l. Each row in the key matrix may be a key. A value matrix V G IRWx / lmay be computed by multiple an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE IRdx / l. Each row in the value matrix may be a value.

[0053] In some embodiments, the MHA layer 151 may implement masked multi-head selfattention. The MHA layer 151 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0054] In some embodiments, the MHA layer 153 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 153 may use outputs from the previous layer (i.e., the add & norm layer 152) as queries and use outputs from the encoder block 110 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 120 to identify and emphasize the most relevant parts of the encoder's input.

[0055] An add & norm layer in the transformer model 100, such as the add & norm layer 142, 144, 152, 154, and 156, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 142 is the MHA layer 141. As another example, the preceding layer of the add & norm layer 154 is the encoder-decoder attention layer 153.

[0056] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sub lay er (x)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sub lay er (x denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as jj.xy= - X Zz=i^xyz > where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data elementin the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and fixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert fixyto a 3D tensor [ixyz, e.g., by replicating every data element over z output points.

[0057] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— [ixyz. The layer normalization operation may further include a variance computation denoted asD2xyzand a division computation denoted as Mxy= Mxymay be a 2D tensor. The layernormalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A'xyzX, = = Dxyzx Mxyz. The layer normalization operation may further compute x(ff2Xy+e)A"xyzand LNxyz= A"xyzx yz. LNxyzmay be the output of the layernormalization operation.

[0058] A feed forward layer (e.g., the feed forward layer 143 and the feed forward layer 155) may be a position-wise fully-connected feed forward network. In an example, the feed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0059] FIG. 2 illustrates an example CNN 200, in accordance with various embodiments. For the purpose of illustration, the CNN 200 includes a sequence of layers comprising a plurality of convolutional layers 210 (individually referred to as "convolutional layer 210"), a plurality of pooling layers 220 (individually referred to as "pooling layer 220"), and a plurality of fully- connected layers 230 (individually referred to as "fully-connected layer 230"). In other embodiments, the CNN 200 may include fewer, more, or different layers. In an execution of the CNN 200, the layers of the CNN 200 execute tensor computation that includes many tensor operations, such as matrix multiplications, convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, elementwise operations (e.g., elementwiseaddition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0060] The convolutional layers 210 summarize the presence of features in inputs to the CNN 200. The convolutional layers 210 function as feature extractors. The first layer of the CNN 200 is a convolutional layer 210. In an example, a convolutional layer 210 performs a convolution on an input tensor 240 (also referred to as IFM 240) and a filter 250. As shown in FIG. 2, the IFM 240 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 240 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 250 is represented by a 3x3x3 3D matrix. The filter 250 includes 3 kernels, each of which may correspond to a different input channel of the IFM 240. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 2, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 250 in extracting features from the IFM 240.

[0061] The convolution includes MAC operations with the input elements in the IFM 240 and the weights in the filter 250. The convolution may be a standard convolution 263 or a depthwise convolution 283. In the standard convolution 263, the whole filter 250 slides across the IFM 240. All the input channels are combined to produce an output tensor 260 (also referred to as OFM 260). The OFM 260 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 2. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 260.

[0062] The multiplication applied between a kernel-sized patch of the IFM 240 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernelsized patch of the IFM 240 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the "scalar product." Using a kernel smaller than the IFM 240 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 240 multiple times at differentpoints on the IFM 240. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 240, left to right, top to bottom. The result from multiplying the kernel with the IFM 240 one time is a single value. As the kernel is applied multiple times to the IFM 240, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 260) from the standard convolution 263 is referred to as an OFM.

[0063] In the depthwise convolution 283, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 2, the depthwise convolution 283 produces a depthwise output tensor 280. The depthwise output tensor 280 is represented by a 5x5x3 3D matrix. The depthwise output tensor 280 includes 3 output channels, each of which is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 240 and a kernel of the filter 250. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 293 is then performed on the depthwise output tensor 280 and a 2x1x3 tensor 290 to produce the OFM 260.

[0064] The OFM 260 is then passed to the next layer in the sequence. In some embodiments, the OFM 260 is passed through an activation function. An example activation function is ReLU. ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 210 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 260 is passed to thesubsequent convolutional layer 210 (i.e., the convolutional layer 210 following the convolutional layer 210 generating the OFM 260 in the sequence). The subsequent convolutional layers 210 perform a convolution on the OFM 260 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 210, and so on.

[0065] In some embodiments, a convolutional layer 210 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 210). The convolutional layers 210 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The CNN 200 includes 26 convolutional layers 210. In other embodiments, the CNN 200 may include a different number of convolutional layers.

[0066] The pooling layers 220 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 220 is placed between two convolution layers 210: a preceding convolutional layer 210 (the convolution layer 210 preceding the pooling layer 220 in the sequence of layers) and a subsequent convolutional layer 210 (the convolution layer 210 subsequent to the pooling layer 220 in the sequence of layers). In some embodiments, a pooling layer 220 is added after a convolutional layer 210, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 260.

[0067] A pooling layer 220 receives feature maps generated by the preceding convolution layer 210 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids over- learning. The pooling layers 220 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. Invarious embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 220 applied to a feature map of 6x6 results in an output pooled feature map of 3x3. The output of the pooling layer 220 is inputted into the subsequent convolution layer 210 for further feature extraction. In some embodiments, the pooling layer 220 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0068] The fully-connected layers 230 are the last layers of the DNN. The fully-connected layers 230 may be convolutional or not. The fully-connected layers 230 receive an input operand. The input operand defines the output of the convolutional layers 210 and pooling layers 220 and includes the values of the last feature map generated by the last pooling layer 220 in the sequence. The fully-connected layers 230 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 2, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 230 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function. In some embodiments, the fully-connected layers 230 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.Example DNN System

[0069] FIG. 3 is a block diagram of a DNN system 300, in accordance with various embodiments. The DNN system 300 can generate (e.g., train, compress, etc.) and execute DNNs, including transformer-based models (e.g., the transformer model 100 in FIG. 1), convolution-based models (e.g., the CNN 200 in FIG. 2), other types of DNNs, or some combination thereof. As shown in FIG. 3, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 300. For instance, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Further, functionality attributed to a component of the DNN system 300 may be accomplished by adifferent component included in the DNN system 300 or a different system. In some embodiments, the DNN module 301 and DNN accelerator 302 may include different types of processing units. In an example, the DNN module 301 may be implemented, at least partially, by one or more central processing units (CPUs). The DNN module 301 and DNN accelerator 302 may be implemented in the same chip or separate chips. In some embodiments, the whole DNN system 300 or a part of the DNN system 300 may be implemented in one or more computing devices.

[0070] The DNN module 301 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 301 can define the layered architecture of a DNN. The DNN module 301 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 301 may also determine one or more hyperparameters that define how the DNN is trained. In some embodiments, the DNN module 301 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 301 may prune weights in one or more layers of a DNN by changing nonzero valued weight to zeros. The DNN module 301 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero-valued weights to the total number of weights. In an example where the DNN module 301 prunes weight during DNN training, the DNN module 301 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 301 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 301 may allow the pruned weights to change values so that a pruned, zero-valued weight may have a nonzero value after further training. The DNN module 301 may prune weights of the layer again after one or more additional epochs.

[0071] The DNN module 301 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 301 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform Al tasks (e.g., language processing, image generation, image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 301 may facilitate deployment of the DNNs using the DNN accelerator 302. For instance, the DNN module 301 may receive data from a device or system coupled with the DNN system 300 and input the received data (or data generated by the DNN module 301, e.g., based on the received data) into a DNN. The DNN module 301 may receive an output of the DNN fromthe DNN accelerator 302. The DNN module 301 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 301) to the device or system. In some embodiments, the DNN module 301 may control execution processes of trained, compressed, or validated DNNs. The DNN module 301 may function as a compiler for DNNs executed by the DNN accelerator 302. The DNN module 301 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0072] The DNN module 301 may control DNN execution or the operation of the DNN accelerator 302 during the DNN execution. In some embodiments, the DNN module 301 may perform compilation of DNNs. Additionally or alternatively, the DNN module 301 may generate configuration parameters (e.g., configuration descriptors) based on which components of the DNN accelerator 302 operate to execute DNNs. For instance, the DNN module 301 may generate configuration parameters based on which the DNN accelerator 302 may perform data precision conversion during DNN execution. Certain aspects of the DNN module 301 are provided below in conjunction with FIG. 4.

[0073] The DNN accelerator 302 executes DNNs, such as DNNs provided by the DNN module 301. For instance, the DNN accelerator 302 may receive a compiled DNN from the DNN module 301 and perform DNN inference. The DNN accelerator 302 may execute a DNN by running neural network operations in the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or a process of performing the neural network operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. As shown in FIG. 3, the DNN accelerator 302 includes a memory 310, a DMA module 320, and DPUs 330 (individually referred to as "DPU 330"). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 302. For example, the DNN accelerator 302 may include more than one memory 310 or DMA module 320. As another example, the DNN accelerator 302 may include a single DPU 330. Further, functionality attributed to a component of the DNN accelerator 302 may be accomplished by a different component included in the DNN accelerator 302 or by a different system. A component of the DNN accelerator 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0074] The memory 310 stores data associated with neural network operations performed by the DNN accelerator 302. In some embodiments, the memory 310 may store data to beused by the DPUs 330 for DNN execution. The memory 310 may store internal parameters of DNNs, such as weights, which are determined by training DNNs. The memory 310 may further store inputs to DNN layers or outputs of DNN layers, such as data input into the first layer or first module of the DNN, data generated by the DPUs 330 from performing neural network operations in the DNN, and so on. Example neural network operations include convolutions (also referred to as "convolutional operations"), layer normalization operations, SoftMax operations, matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of neural network operations, or some combination thereof. The memory 310 may also store configuration descriptors or parameters generated by the DNN module 301. The memory 310 may be a main memory of the DNN accelerator 302. In some embodiments, the memory 310 includes one or more dynamic random-access memories (DRAMs).

[0075] The DMA module 320 facilitates data transfer between the memory 310 and local memories of the DPUs 330. For example, the DMA module 320 can read data from the memory 310 and write data into a local memory of a DPU 330. As another example, the DMA module 320 can read data from a local memory of a DPU 330and write data into the memory 310. The DMA module 320 provides a DMA feature that allows the DPU 330 to initiate data transfer between the memory 310 and the local memories of the DPUs 330 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA module 320 may read tensors from the memory 310, modify the tensors in a way that is optimized for the DPU 330 before it writes the tensors into the local memories of the DPUs 330.

[0076] The DPUs 330 execute workloads in DNN execution. The DPUs 330 may perform neural network operations in DNNs for DNN training or DNN inference. For instance, a DPU 330 may execute a DNN layer by running one or more neural network operations in the DNN layer. A DPU 330 may execute a layer, or a data portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple DPUs 330 in parallel. For instance, multiple DPUs 330 may each perform a data portion of a workload for a neural network operation. Data may be shared between the DPUs 330. The DPUs 330 may be capable of running various types of neural network operations, such as convolution, layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the DPUs 330include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the DPU 330 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the DPU 330 or another DPU 330.

[0077] In the embodiments of FIG. 3, each DPU 330 includes a local memory 340, a processing engine 350, a post-processing engine 360, and a data transfer module 370. The local memory 340 can provide data storage capacity. The processing engine 350 and postprocessing engine 360 can provide computing capacity. The data transfer module 370 can transfer data between the local memory 340 and processing engine 350 or the postprocessing engine 360. Some or all the components of the DPU 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the DPU 330. Further, functionality attributed to a component of the DPU 330 may be accomplished by a different component included in the DPU 330, a different DPU 330, another component of the DNN accelerator 302, or a different system. A component of the DPU 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0078] Each local memory 340 may be local to the corresponding DPU 330. In the embodiments of FIG. 3, the local memory 340 is inside the DPU 330. In other embodiments, the local memory 340 may be outside the DPU 330. In some embodiments, the local memory 340 includes one or more SRAMs. In some embodiments, the local memory 340 may be an on-chip memory, while the memory 310 may be an off-chip memory. Data in the local memory 340 may be transferred to or from the memory 310, e.g., through the DMA module 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory of another DPU 330.

[0079] The local memory 340 may store data generated or used for controlling DNN execution. For instance, the local memory 340 may store data generated by the DNN module 301 for controlling one or more components of the DNN accelerator 301. The local memory 340 may also store data processed or computed in DNNs. For instance, the local memory 340 may store data received, processed, or computed by the processing engine 350 and the post-processing engine 360. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on. Data stored in the local memory340 may have various precisions. In some embodiments, the local memory 340 may store the same data with different formats and different precisions.

[0080] In some embodiments, the local memory 340 may include memory cuts. A memory cut includes one or more memory banks. In some embodiments, the number of memory banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a memory bank may include 8, 16, 64, or a different number of storage units. The local memory 340 may be byte- addressable, and each memory address identifies a single byte (eight bits) of storage. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0081] In some embodiments, the local memory 340 may store tensors to be processed by the processing engine 350 or the post-processing engine 360. The tensors may be input tensors of neural network operations. The local memory 340 may also store tensors generated by the processing engine 350 or the post-processing engine 360. The tensors may be output tensors of neural network operations. The layout of data points of a tensor in the local memory 340 may depend on the format in which the tensor is stored. In some embodiments, the local memory 340 may store tensors in various formats, including Z- major format, X-major format, and Y-major format. For a tensor with Z-major format, the local memory 340 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 340. For a tensor with the ZXY format or ZYX format, the local memory 340 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 340. For a tensor with X-major format, the local memory 340 may store data points having the same (y, z) coordinate contiguously. For atensor with Y-major format, the local memory 340 may store data points having the same (x, z) coordinate contiguously.

[0082] In some embodiments, the local memory 340 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0083] The processing engine 350 performs computations in neural network operations. The processing engine 350 may include compute elements that can perform computations in neural network operations. Examples of compute elements may include multipliers, adders, subtractors, shifters, normalizers, other types of compute elements, or some combination thereof. The processing engine 350 may also include other components (e.g., logic components, data storage components, etc.) that may monitor, facilitate, or control operations of compute elements in the processing engine 350.

[0084] In an example, the processing engine 350 may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the DPU 330 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0085] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0086] In some embodiments, the processing engine 350 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The processing engine 350 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0087] In some embodiments, the processing engine 350 may perform MAC operations in quantized neural network operations, such as MAC operations in a quantized convolution. In some embodiments, an MAC unit in the processing engine 350 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier thatcan multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating point format. The MAC unit may include no quantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized neural network operations.

[0088] The processing engine 350 may accelerate neural network operations based on sparsity in data. In some embodiments, the processing engine 350 may operate in a dense mode in which sparsity acceleration is not performed. The processing engine 350 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 350. Each processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 350 may constitute a bigger array that includes more rows and columns.

[0089] In some embodiments, the processing engine 350 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each processing cell in the processing engine 350 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the processing engine 350 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0090] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor mayindicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0091] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the processing engine 350 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the processing engine 350 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the processing engine 350 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0092] The post-processing engine 360 processes outputs of the processing engine 350. The post-processing engine 360 may include one or more post-processing elements. In some embodiments, the post-processing elements in the post-processing engine 360 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 360 computes activation functions. The post-processing engine 360 may receive outputs of the processing engine 350 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 360 may perform other types of post processing on outputs of the processing engine 350. For instance, the postprocessing engine 360 may apply a bias or scale on an output of the processing engine 350. In some embodiments, the post-processing engine 360 may be bypassed for certain neural network operations.

[0093] In some embodiments, the post-processing engine 360 may include one or more multi-precision converters for changing the precision of data by converting the data format. Data precision conversion may be needed for execution of neural network operations. Forexample, the post-processing engine 360 may receive data whose precision does not match the data precision requirement for a neural network operation to be performed by the postprocessing engine 360. As another example, the post-processing engine 360 may compute data whose precision does not match the data precision requirement for the next neural network operation. A multi-precision converter in the post-processing engine 360 may convert the original precision of the data to the precision required by the neural network operation. In some embodiments, a multi-precision converter in the post-processing engine 360 may be able to process various data precisions. The operation of the multi-precision converter can be configurable to meet various application needs. Certain aspects of the post-processing engine 360 are described below in conjunction with FIG. 5.

[0094] The data transfer module 370 facilitates data transfer within the DPU 330. For instance, the data transfer module 370 may load data from the local memory 340 to the processing engine 350 or the post-processing engine 360. The data transfer module 370 may also drain data computed in the processing engine 350 or the post-processing engine 360 and write the data to the local memory 340. In some embodiments, the data transfer module 370 may include one or more load modules for loading data from the local memory 340 and one or more drain modules for draining data from the processing engine 350 or the post-processing engine 360.

[0095] The data transfer module 370 may perform data read and write operations. In some embodiments, the data transfer module 370 may generate or transmit data transfer request from the processing engine 350 or the post-processing engine 360 to the local memory 340. A data transfer request may be a read request or write request. The data transfer request may include information indicating an address where the data is to be read from or written to. In some embodiments, the address corresponds to a memory cut in the local memory 340. The address may be referred to as an input address from the perspective of the local memory 340 and may be transformed into a swizzled address by the local memory 340.

[0096] The data transfer module 370 may include a buffer. The buffer may temporarily store data received by the data transfer module 370. For instance, data computed by the processing engine 350 or the post-processing engine 360 may be stored in the buffer before being processed by the data transfer module 370. In some embodiments, the buffer includes one or more circular buffers. In some embodiments, the data transfer module 370 may include multiple buffers, each of which may be used for storing data computed in aportion of the processing engine 350 or the post-processing engine 360. The portion of the processing engine 350 or the post-processing engine 360 may be a column or a row.

[0097] In some embodiments, the data transfer module 370 may include a sparsity encoder. The sparsity encoder may compress output data of DNN layers based on sparsity in the output data. In some embodiments, the sparsity encoder may receive an output tensor of a layer. The sparsity encoder may remove zero-valued data points from a tensor to generate a compressed tensor. In some embodiments, the sparsity encoder may also generate one or more sparsity tensors for the output tensor. The sparsity tensor may include sparsity elements, each of which corresponds to a different data point in the tensor and indicates whether the corresponding data point is removed or not. In some embodiments, the sparsity tensor may be a sparsity bitmap, and a sparsity element in the sparsity bitmap may be a bit. A zero bit may indicate that the corresponding data point is removed and not in the compressed tensor, while a one bit may indicate that the corresponding data point is not removed and is in the compressed tensor.

[0098] FIG. 4 is a block diagram of a DNN module 400, in accordance with various embodiments. The DNN module 400 may be an embodiment of the DNN module 301 in FIG. 3. As shown in FIG. 4, the DNN module 400 includes an interface module 410, a training module 420, a compressing module 430, a validating module 440, a compiler 450, and a datastore 460. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 400. Further, functionality attributed to a component of the DNN module 400 may be accomplished by a different component included in the DNN module 400 or a different module or system.

[0099] The interface module 410 facilitates communications of the DNN module 400 with other modules or systems. For example, the interface module 410 establishes communications between the DNN module 400 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 410 may distribute trained DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0100] The training module 420 trains DNNs by using a training dataset. The training module 420 forms the training dataset. In an example where the training module 420 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in thetraining images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the validating module 440 to validate performance of a trained DNN. The data portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0101] The training module 420 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0102] The training module 420 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height,the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0103] In the process of defining the architecture of the DNN, the training module 420 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0104] After the training module 420 defines the architecture of the DNN, the training module 420 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 420 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 420 uses a cost function to minimize the error.

[0105] The training module 420 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 420 finishes the predetermined number of epochs, the training module 420 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0106] The compressing module 430 compresses DNNs. For instance, the compressing module 430 may add pruning operations to DNN layers to reduce computational complexity or memory usage. A pruning operation may prune weight tensors of a DNN layer by changing one or more nonzero valued weights of the layer to zeros. The modification may be done before, during, or after training. Weights may be pruned during training, during inference, or a combination of both. The compressing module 430 may determine a sparsity ratio for a DNN layer. The sparsity ratio may be a ratio of the number of zero-valued weightto the total number of weights in the layer. The compressing module 430 may perform the pruning operation till the sparsity ratio of the DNN layer meets a target sparsity ration, such as 10%, 20%, 30%, 40%, 50%, and so on.

[0107] In some embodiments, the compressing module 430 may select one or more layers in a DNN and modify each selected layer with a pruning operation. For instance, the compressing module 430 may select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or of a type of layer, the compressing module 430 may determine a weight threshold that would not cause a loss of the accuracy of the DNN to exceed an accuracy loss constraint. A pruning operation may modify weights having absolute values above the weight threshold to zeros and leave the other weights unchanged. The weight pruning can reduce memory storage as zero-valued weights may not be stored. Also, the number of operations in the layer can be reduced as computations on zero-valued weights can be skipped without impacting the output of the layer. In some embodiments, the compressing module 430 may also measure energy saving, final DNN accuracy, or layer-wise sparsity caused by pruning operations.

[0108] After compressing a DNN, the compressing module 430 may fine tune the DNN, e.g., through a retraining process. The compressing module 430 may fine tunes DNNs after weights are pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For instance, after weights in a DNN are pruned, the compressing module 430 may further train the DNN by inputting a training dataset into the DNN. The values of the unpruned weights in the DNN may be modified based on outputs of the DNN and ground-truth labels of the training samples in the training dataset. In some embodiments, the values of the pruned weights (i.e., zero) are not changed during the fine-tuning process. For instance, the compressing module 430 may place a mask over a pruned weight block and the mask can prevent values in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values of all weights, including the pruned weights, may be changed during the fine-tuning process. After one or more cycles of retraining and weight changing by the compressing module 430, the compressing module 430 may perform a new pruning process, e.g., by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is done. In some embodiments, the number of epochs in the fine-tuning process may be different from the number of epochs in thetraining process in which the pre-pruning values of the weights are determined. For instance, the fine-tuning process may have less epochs than the training process. In an example, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, and so on.

[0109] The validating module 440 verifies accuracy of trained or compressed DNNs. In some embodiments, the validating module 440 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the validating module 440 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The validating module 440 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0110] The validating module 440 may compare the accuracy score with a threshold score. In an example where the validating module 440 determines that the accuracy score of the DNN is less than the threshold score, the validating module 440 instructs the training module 420 to re-train the DNN. In one embodiment, the training module 420 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0111] The compiler 450 compiles information of DNNs to executable instructions that can be executed, e.g., by the DNN accelerator 302, to carry out neural network operations in DNNs. In some embodiments, the compiler 405 may generate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neuralnetwork operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler 450 may use the graph to generate executable DNNs. For instance, the compiler may generate computer program instructions (e.g., compilation descriptors) for executing DNNs. The instructions may be stored in registers associated with components of the DNN accelerator 302.

[0112] In some embodiments, the compiler 450 may generate instructions to facilitate data precision conversion in DNNs. For instance, the compiler 450 may generate configuration parameters that can be used by multi-precision converters in the DNN accelerator 302 for changing date precision during DNN execution. The compiler 450 may generate configuration parameters that encode information with respect to the input precision (or the input data format) and output precision (or the output data format), such as configuration parameters indicating whether the input data format is integer format or floating point format, configuration parameters indicating whether the output data format is integer format or floating point format, configuration parameters indicating the number of valid sign, exponent, or mantissa bits in the input data format, configuration parameters indicating the number of valid sign, exponent, or mantissa bits in the output data format, and so on. The compiler 450 may generate these configuration parameters based on information of the DNN, e.g., information indicating the input data format and output data format.

[0113] The compiler 450 may also generate configuration parameters that specify operation modes of components in the multi-precision converter, such as configuration parameter specifies rounding mode for mantissa conversion, configuration parameters that indicates a mode for denormal numbers, and so on. A denormal number may be a non-zero number whose magnitude is smaller than the smallest normalized floating point number. The compiler 450 may also generate configuration parameters that facilitates data read or data write, such as a configuration parameter that indicates the number of data elements to be processed (e.g., the number of data elements in a tile), configuration parameter that indicates the memory address where an input data element may be fetched, configuration parameter that indicates memory address where an output data element may be stored,configuration parameter that indicates memory address where another configuration parameters may be stored, and so on.

[0114] The datastore 460 stores data received, generated, used, or otherwise associated with the DNN module 400. For example, the datastore 460 stores the datasets used by the training module 420 and validating module 440. The datastore 460 may also store data generated by the training module 420 and validating module 440, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), data for sparsity acceleration (e.g., sparsity bitmap, etc.), and so on. The datastore 460 may store control data (e.g., control parameters, etc.), compilation descriptors, or other data generated by the compressing module 430. The datastore 460 may include one or more memories. In the embodiment of FIG. 4, the datastore 460 is a component of the DNN module 400. In other embodiments, the datastore 460 may be external to the DNN module 400 and communicate with the DNN module 400 through a network.

[0115] FIG. 5 illustrates an example compute engine 500, in accordance with various embodiments. The compute engine 500 may be used in a DNN accelerator for executing neural network operations DNNs. The compute engine 500 may be an example of the processing engine 350 or the post-processing engine 360 in FIG. 3. As shown in FIG. 5, the compute engine 500 includes a multi-precision converter 510A and another multi-precision converter 510B (collectively referred to as "multi-precision converters 510" or "multiprecision converter 510") and a plurality of compute blocks 520 (individually referred to as "compute block 520"). In other embodiments, alternative configurations, different or additional components may be included in the compute engine 500. For instance, the compute engine 500 may include a different number of multi-precision converters. Further, functionality attributed to a component of the compute engine 500 may be accomplished by a different component included in the compute engine 500, a different component included in the DNN accelerator, or a different device.

[0116] The multi-precision converters 510 receive input data and change the precisions of input data to different precisions by changing the format of the input data. In some embodiments, the multi-precision converters 510 may change integer formats to integer formats, integer formats to floating point formats, or floating point formats to floating point formats. Each of the multi-precision converters 510 can convert various input precisions to various output precisions. The flexibility of the multi-precision converters 510 may make thecompute engine 500 (or the DNN accelerator) suitable for executing DNNs with various data precision requirements. The many-to-many precision conversion may be accomplished by the logic units in the multi-precision converters 510 without software intervening, which can improve the efficiency of the compute engine 500 (or the DNN accelerator). In some embodiments, some components of a multi-precision converter 510 may be capable of handling different input precisions (or different output precisions), while one or more other components may be designed for a particular precision.

[0117] In some embodiments, the multi-precision converters 510 may be of different types. For instance, the multi-precision converter 510A may be an upward precision converter while the multi-precision converter 510B may be a downward precision converter, or vice versa. In some embodiments, both of the multi-precision converters 510 may perform precision conversion for a neural network operation. In other embodiments, one or even both of the multi-precision converters 510 may be in an idle state and not need to perform any precision conversion for a neural network operation.

[0118] In some embodiments, a multi-precision converter 510 may have multiple operation modes. For example, the multi-precision converter 510 may have an operation mode for upward conversion and another operation mode for downward conversion. As another example, the multi-precision converter 510 may have an operation mode for converting floating point to integer, an operation mode for converting integer to floating point, and an operation mode for converting floating point to floating point. The multi-precision converter 510 may receive a configuration parameter that specifies an operation mode and perform precision conversion in the operation mode. A multi-precision converter 510 may support denormal numbers in one or more modes. For example, a multi-precision converter 510 may treat denormal numbers as zero. As another example, a multi-precision converter 510 may treat denormal numbers as ordinary input and perform all the required checks on the input / output with respect to the specified threshold margin based on the selected conversion type. Either mode may be enabled by a configuration parameter. Certain aspects of the multi-precision converters 510 are described below in conjunction with FIG. 6.

[0119] The compute blocks 520 performs computations in neural network operations to execute the neural network operations. In some embodiments (e.g., embodiments where the compute engine 500 is an example the processing engine 350 in FIG. 3), the compute blocks 520 may each include one or more MAC units. In other embodiments (e.g.,embodiments where the compute engine 500 is an example the post-processing engine 350 in FIG. 3), the compute blocks 520 may each include logic units (e.g., arithmetic logic units), such as activation function unit for applying activation function on data), bias unit for applying a bias value on data, scale unit for scaling data, other types of logic units, or some combination thereof.

[0120] Even though not shown in FIG. 5, the compute engine 500 may include a data path on which the multi-precision converter 510A, compute blocks 520, and plurality of compute blocks 520 are arranged in a sequence. For instance, input data may be first received by the multi-precision converter 510A, then flows to the compute blocks 520 after the precision conversion by the multi-precision converter 510A, and further, data computed by the plurality of compute blocks 520 is transmitted to the multi-precision converter 510B for another round of precision conversion. In some embodiments, the compute engine 500 may collaborate with another compute engine to perform neural network operations. The multiprecision converter 510A may perform precisions conversions to ensure that the data received from the other compute engine meets the requirement of the compute blocks 520 or the requirement of the neural network operation to be performed by the compute blocks 520. The multi-precision converter 510B may perform precision conversions to ensure that the data computed in the compute blocks 520 and output from the compute engine 500 meets the requirement of the other compute engine or the requirements of a neural network operation to be performed by the other compute engine.

[0121] In an example DNN execution process, the compute engine 500 may receive data of a particular precision from the other compute engine and may need to provide data of the same precision to the other compute engine after the compute blocks 520 complete one or more neural network operations in the DNN. The compute blocks 520 may perform the one or more neural network operations at a different precision. For such a scenario, the multiprecision converter 510A may convert the precision of the other compute engine to the precision of the compute blocks 520, and the multi-precision converter 510B may convert the precisions of the compute blocks 520 back to the precision of the other compute engine.Example Multi-precision Converter

[0122] FIG. 6 illustrates an example multi-precision converter 600, in accordance with various embodiments. The multi-precision converter 600 may be an example of the compute engine 500 in FIG. 5. The multi-precision converter 600 includes an input extractor610, an input generator 620, an exponent selector 630, exponent converts 635 (individually referred to as "exponent converter 635"), a mantissa selector 640, mantissa converts 645 (individually referred to as "mantissa converter 645"), an exponent collector 650, a mantissa collector 660, a register 670, and a shifter 680. Each of the components in the multiprecision converter 600 may include one or more logic units and may be at least part of a circuit, such as an integrated circuit. In other embodiments, alternative configurations, different or additional components may be included in the multi-precision converter 600. Further, functionality attributed to a component of the multi-precision converter 600may be accomplished by a different component included in the multi-precision converter 600 or different device.

[0123] The multi-precision converter 600 is coupled to a memory 601 and a memory control module 602. The memory 601 stores data to be converted by the multi-precision converter600. Such data may be referred to as input data. The memory 601 also stores data generated by the multi-precision converter 600. Such data may be referred to as output data. Input or output data stored in the memory 601 may have various formats and precisions. In some embodiments, the memory 601 may also store configuration parameters to be used by the memory 601 or the multi-precision converter 600. For example, the memory 601 may store configuration parameters based on which the memory control module 602 can read and write data. As another example, the memory 601 may store a configuration parameter that determines the operation mode of the multi-precision converter 600. A configuration parameter may include one or more bits. In an example, a configuration parameter specifying an operation mode of the multi-precision converter 600 may be 0 or 1 with 0 indicating a mode of converting floating point to integer while 1 indicating a mode of converting floating point to floating point. In some embodiments, the memory 601 may be a separate device from the multi-precision converter 600. For instance, the memory 601 may be an example of the local memory 340 in FIG. 3. In other embodiments, the memory 601 may be part of the multi-precision converter 600.

[0124] The memory control module 602 may read data to be converted (e.g., data with the original precision) from the memory 601 and facilitate transmission of the data to the input extractor 610. The memory control module 602 may also receive converted data (e.g., data with the converted precision) from the register 670 and write the data into the memory601. In some embodiments, the memory control module 602 may provide one or moreconfiguration parameters to one or more other components of the multi-precision converter 600. For instance, the memory 601 may store configuration parameters to be used by the input extractor 610, input generator 620, exponent selector 630, mantissa selector 640, exponent converters 635, mantissa converters 645, exponent collector 650, or mantissa collector 660.

[0125] The memory control module 602 may operate in accordance with one or more configuration parameters, such as a configuration parameter indicating a memory address from where the input data for conversion is fetched from the memory 601, a configuration parameter indicating a memory address where the converted output data is written in the memory 601, a configuration parameter indicating a memory address where one or more configuration parameters for one or more other components of the multi-precision converter 600 are stored, and so on.

[0126] In some embodiments, the memory control module 602 may be a separate device from the multi-precision converter 600. For instance, the memory control module 602 may be an example of the data transfer module 370 in FIG. 3. In other embodiments, the memory control module 602 may be part of the multi-precision converter 600.

[0127] The input extractor 610 may extract exponent bits and mantissa bits from input data elements. The input data elements may be input into the multi-precision converter 600 from the memory control module 602. An input data element may be a value computed during the execution of a neural network operation. Exponent bits are bits that encode an exponent of the data element. Mantissa bits are bits that encode a mantissa of the data element. In some embodiments, the input extractor 610 may also extract sign bits from data elements. A sign bit encodes the sign (either positive or negative) of the data element. The input extractor 610 can extract bits for various input data formats so that the restriction on the input data precision, which conventional precision converters have, can be eliminated.

[0128] In some embodiments, the input extractor 610 may operate in accordance with one or more configuration parameters. Examples of the configuration parameters may include a configuration parameter indicating the number of valid exponent bits for a data format, configuration parameter indicating the number of valid mantissa bits for a data format, configuration parameter indicating the number of valid sign bits for a data format, other types of configuration parameters, or some combination thereof. The configuration parameters may be generated offline, e.g., before the execution of the DNN starts. In someembodiments, the configuration parameters may be generated by a DNN module, such as the DNN module 301. The input extractor 610 may receive different sets of configuration parameters for different input data precisions. With the input extractor 700, the dependency of a rigid input floating point format for converter, which exists in currently available precision converters, can be removed. Certain aspects of the input extractor 610 are described below in conjunction with FIG. 7.

[0129] The input generator 620 receives exponent bits and mantissa bits from the input extractor 610 and prepares the data for generating new exponents and new mantissas. In some embodiments, the input generator 620 generates intermediate exponents from exponent bits extracted by the input extractor 610. The input generator 620 may also generate intermediate mantissas from mantissa bits extracted by the input extractor 610. In some embodiments, the input generator 620 generates intermediate exponents and intermediate mantissas based on a configuration of a hardware interface in the multiprecision converter 600. The configuration of the hardware interface may indicate a number of bits being transmitted through the hardware interface at a time. The hardware interface may be an interface to subsequent components of the multi-precision converter 600, e.g., the exponent selector 630, exponent converters 635, mantissa selector 640, or mantissa converters 645.

[0130] In some embodiments, every intermediate exponent or intermediate mantissa generated by the input generator 620 may have a fixed number of bits, which may be the number of bits the hardware interface is configured to transmit at a time, so that the intermediate exponent or intermediate mantissa can be transmitted through the hardware interface to the converter circuitry. As described above, the input extractor input extractor 610 allows input data format to be any type and extracts valid sign, exponent and mantissa bits. The extract exponent or mantissa may not meet the format of the hardware interface. Such a problem may be solved by the input generator 620. The input generator 620 may remove invalid bits from the extract exponent and mantissa bits or pad zeros to make sure unwanted data are not propagated to the converter circuitry.

[0131] In some embodiments, the input generator 620 may generate an intermediate exponent (or an intermediate mantissa) through a padding process. The padding process may include addition of one or more new bits to the extracted exponent (or mantissa) bits. The new bits may be zero bits, and the padding process is a zero padding process. In someembodiments, the input generator 620 may separate the sign bit from the exponent bits or the mantissa bits or separate the exponents bits from the mantissa bits before it fills remaining bit positions with zeros. Certain aspects of the input generator 620 are described below in conjunction with FIG. 8.

[0132] The input generator 620, exponent converters 635, and exponent collector 650 may constitute exponent converter circuitry. The exponent selector 630 selects which exponent converters 635 to be used to convert exponents with intermediate exponents generated by the input generator 620. The exponent converters 635 may be used for different output precisions. Each exponent converter 635 may be used for a particular output precision. In some embodiment, an exponent converter 635 may be configured to convert a first number of bits to a second number of bits. The first number may be the number of bits in intermediate exponents generated by the input generator 620. The second number may be the number of exponent bits that a value with the output precision has. The second number may be different for different exponent converters 635. For instance, the exponent converters 635 may include a 1-bit exponent converter for converting an intermediate exponent to a 1-bit exponent, a 2-bit exponent converter for converting an intermediate exponent to a 2-bit exponent, a 3-bit exponent converter for converting an intermediate exponent to a 3-bit exponent, and so on. Certain aspects of the exponent converters 635 are described below in conjunction with FIGS. 9A-9D.

[0133] In some embodiments, the exponent selector 630 may select one of the exponent converters 635 based on the output precision, which is pre-known. The exponent selector 630 may receive a configuration parameter that indicates the number of exponent bits for the output precision and selects an exponent converter 635 based on the configuration parameter. The configuration parameter may be generated offline, e.g., before the DNN execution starts. The exponent selector 630 may receive the configuration parameter from the DNN module 301. In some embodiments, the exponent selector 630 may activate the selected exponent converter 635 or de-activate the unselected exponent converters 635 so that the converted exponent would have the correct number of bits.

[0134] The exponent collector 650 generates output exponents from converted exponents generated by the exponent converters 635. In some embodiments, the exponent collector 650 may perform one or more operations, such as threshold checking, bias handling, underflow handling, overflow handling, packing, other types of operations, or somecombination thereof. The exponent collector 650 may compare the converted exponent with a threshold exponent of the output precision to detect underflow or overflow. For overflowing handle, the exponent collector 650 may receive carry from mantissa rounding and add it to the converted exponent.

[0135] The mantissa selector 640, mantissa converters 645, and mantissa collector 660 may constitute mantissa converter circuitry. The mantissa selector 640 selects which mantissa converters 645 to be used to convert mantissas with intermediate mantissas generated by the input generator 620. The mantissa converters 645 may be used for different output precisions. Each mantissa converter 645 may be used for a particular output precision. In some embodiment, a mantissa converter 645 may be configured to convert a first number of bits to a second number of bits. The first number may be the number of bits in intermediate mantissas generated by the input generator 620. The second number may be the number of mantissa bits that a value with the output precision has. The second number may be different for different mantissa converters 645. For instance, the mantissa converters 645 may include a 1-bit mantissa converter for converting an intermediate mantissa to a 1-bit mantissa, a 2-bit mantissa converter for converting an intermediate mantissa to a 2-bit mantissa, a 3-bit mantissa converter for converting an intermediate mantissa to a 3-bit mantissa, and so on. Certain aspects of the mantissa converters 645 are described below in conjunction with FIGS. 10A-10D.

[0136] In some embodiments, the mantissa selector 640 may select one of the mantissa converters 645 based on the output precision, which is pre-known. The mantissa selector 640 may receive a configuration parameter that indicates the number of mantissa bits for the output precision and selects a mantissa converter 645 based on the configuration parameter. The configuration parameter may be generated offline, e.g., before the DNN execution starts. The mantissa selector 640 may receive the configuration parameter from the DNN module 301. In some embodiments, the mantissa selector 640 may activate the selected mantissa converter 645 or de-activate the unselected mantissa converters 645 so that the converted mantissa would have the correct number of bits.

[0137] The mantissa collector 660 generates output exponents from converted mantissas generated by the mantissa converters 645. In some embodiments, the mantissa collector 660 may perform one or more operations, such as denormalization, shifting, bit padding, packing, rounding, normalization, other types of operations, or some combination thereof.

[0138] The register 670 receives output exponents from the exponent collector 650 and receives output mantissas from the mantissa collector 660. The output exponent and output mantissa for a data element may be collated in the register 670. In some embodiments, the register 670 has a fixed number of bits, such as 16, 32, 64, and so on. When the output data element has less bits than the fixed number of bits, the register 670 may store multiple output data elements at a time. The multiple output data elements may be collated by the shifter 680. For instance, the shifter 680 may shift the bits of a first data element so that the bits of the first data element may be followed by the bits of a second data element in the register 670.

[0139] FIG. 7 illustrates an example input extractor 700, in accordance with various embodiments. The input extractor 700 can extract exponent, mantissa, or sign bits from data elements for precision conversion. The input extractor 700 may be an example of the input extractor 610 in FIG. 6. As shown in FIG. 7, the input extractor 700 includes a multiplexer (MUX) 710, a register 720, a shifter 730, and a mantissa extractor 740. In other embodiments, the input extractor 700 may include fewer, more, or different components.

[0140] In some embodiments, the input extractor 700 receives a data element 705 and extracts a sign 701, an exponent 702, and a mantissa 703 from the data element 705. The data element 705 is an input data element that has an input data format and an input precision. The input extractor 700 also receives configuration parameters. The configuration parameters may include a configuration parameter indicating the number of valid exponent bits in the input data format and a configuration parameter indicating the number of valid mantissa bits in the input data format. The configuration parameters may be determined based on the pre-known or predetermined data format of the input data element. In some embodiments, the data element 705 may have a floating point format. The input extractor 700 may receive data in batches of a predetermined number of bits, such as 16 bits, 32 bits, 64 bits, and so on. The input extractor 700 may receive one or more data elements in a single cycle.

[0141] The MUX 710 receives the data element 705 and is configured to extract the sign 701 and the exponent 702 from the data element. The MUX 710 may be configured by at least one of the configuration parameters. In some embodiments, the most significant bit is the sign 701, and the number of valid exponent bits out of the next 8 bits is the exponent 702.

[0142] The register 720 stores input data elements. In an example where the input extractor 700 receives data in batches of 32 bits, the register 720 may be a 32-bit register. In some embodiments, the size of the register 720 is the same as the input data size, meaning the total number of bits that can be stored in the register 720 per cycle is equal to the total number of bits in the input data format. The register 720 may store a single data element at a time. In other embodiments, the size of the register 720 is smaller than the input data size, meaning the total number of bits that can be stored in the register 720 per cycle is more than the total number of bits in the input data format. The register 720 may store multiple data elements at a time.

[0143] The shifter 730 is coupled to the register 720. The shifter 730 is configured to shift bits in the register 720. In some embodiments (e.g., embodiments where the input data size is less than the size of the register 720), the shifter 730 may shift bits in the register 720 by the total number of bits in the input data format so that it can identify the next data element from the register 720. The shifter 730 may provide the next data element to the MUX 710. In some embodiments, the shifter 730 may compute the input data size, i.e., total number of bits in the input data format, based on the configuration parameters. For instance, the shifter 730 may accumulate the number of valid exponent bits, the number of valid mantissa bits, and 1 to compute the input data size.

[0144] The mantissa extractor 740 extracts the mantissa 703 from the data element 705. As shown in FIG. 7, the mantissa extractor 740 includes a MUX 750 and a subtractor 760. The MUX 750 may be configured by an output of the subtractor 760. The output of the subtractor 760 may be used as a control signal of the MUX 750. The subtractor 760 may compute the output based on the configuration parameter indicating the number of valid mantissa bits in the input data format. In some embodiments, the input extract 700 may read 32 bits in a single read operation and identify which bits from the 32 bits correspond to the sign, exponent, and mantissa, respectively. In an example, the most significant bit is the sign bit, and the next 8 bits are exponent bits. The total number of valid exponent bits may be 8 or less. The subtractor 760 may consume the number of valid exponent bits as input and subtract the number of valid exponent bits from the maximum number of exponent bits to produce a select value for the MUX 750. The maximum number of bits is 8. The MUX 750 may use the select value from the subtractor 760 to select mantissa bits. For instance, the MUX 750 may use the select value to select 23 mantissa bits from one of the eight possiblecombinations. In some embodiments, the mantissa extractor 740 receives all the bits of the data element 705. The select pin of the MUX 750 may provide the exact number of valid mantissa bits to be selected from remaining bits of the data element 705. In the example shown in FIG. 7, the MUX 750 selects 23 bits [22:0],

[0145] FIG. 8 illustrates an example input generator 800, in accordance with various embodiments. The input generator 800 may generate intermediate exponents and intermediate mantissas that meet the format requirement of a hardware interface, e.g., a converter interface. The format requirement of the converter interface may be a data size requirement, e.g., a fixed number of bits being transmitted through the converter interface at a time. The input generator 800 can facilitate latching data onto the converter interface. The input generator 800 may be an example of the input generator 620 in FIG. 6. The input generator 800 may use the output of the input extractor 610 to generate intermediate exponents and intermediate mantissas. As shown in FIG. 8, the input generator 800 includes a MUX 810, a subtractor 820, another MUX 830, and another subtractor 840. In other embodiments, the input generator 800 may include fewer, more, or different components.

[0146] The MUX 810 and subtractor 820 may generate intermediate exponents that meet the format requirement of the converter interface. For the purpose of illustration and simplicity, the converter interface requires that each intermediate exponent has 8 bits in the embodiments of FIG. 8. The subtractor 820 receives a signal indicating the format requirement of the converter interface, which is represented by {exp[7:0]} in FIG. 8. The subtractor 820 also receives the configuration parameters indicating the number of valid exponent bits in the input data format, which is represented by #valid_exp_bits in FIG. 8. The subtractor 820 may compute a control signal (e.g., a select signal) from the format requirement of the converter interface and the configuration parameter. The control signal may be a value that is equal to the result of subtracting the number of valid exponent bits (i.e., #valid_exp_bits in FIG. 8) from the maximum number of exponent bits (i.e., 8 in the embodiments of FIG. 8). The control signal is to be used by the MUX 810 to generate an intermediate exponent (represented by {exp_out[7:0]} in FIG. 8) from the extracted exponent (e.g., the exponent 702), which may not meet the format requirement of the converter interface. The intermediate exponent has 8 bits and therefore, meet the format requirement of the converter interface.

[0147] In some embodiments (e.g., embodiments where the extracted exponent has less bits than the size of the converter interface), the MUX 810 may perform a zero padding process and add zero bits to the extracted exponent. The MUX 810 may select one of its input signals for zero padding based on the control signal from the subtractor 820. In an example, the number of zeros in the selected input signal may equal the value in the control signal from the subtractor 820, i.e., the difference between the maximum number of exponent bits from the number of valid exponent bits.

[0148] The MUX 830 and subtractor 840 may generate intermediate mantissas that meet the format requirement of the converter interface. For the purpose of illustration and simplicity, the converter interface requires that each intermediate mantissa has 23 bits in the embodiments of FIG. 8. The subtractor 820 receives a signal indicating the format requirement of the converter interface, which is represented by {mant[23:0]} in FIG. 8. The subtractor 840 also receives the configuration parameters indicating the number of valid mantissa bits in the input data format, which is represented by #valid_mant_bits in FIG. 8. The subtractor 840 may compute a control signal (e.g., a select signal) from the format requirement of the converter interface and the configuration parameter. The control signal may be a value that is equal to the result of subtracting the number of valid mantissa bits (i.e., #valid_mant_bits in FIG. 8) from the maximum number of mantissa bits (i.e., 23 in the embodiments of FIG. 8). The control signal is to be used by the MUX 830 to generate an intermediate mantissa (represented by {mant_out[22:0]} in FIG. 8) from the extracted mantissa (e.g., the mantissa 703), which may not meet the format requirement of the converter interface.

[0149] In some embodiments (e.g., embodiments where the extracted mantissa has less bits than the size of the converter interface), the MUX 830 may perform a zero padding process and add zero bits to the extracted mantissa. The MUX 830 may select one of its input signals for zero padding based on the control signal from the subtractor 840. In an example, the number of zeros in the selected input signal may equal the value in the control signal from the subtractor 840, i.e., the difference between the maximum number of mantissa bits from the number of valid mantissa bits.

[0150] FIGS. 9A-9D illustrate exponent converters 900A-900D, in accordance with various embodiments. The exponent converters 900A-900D may be examples of the exponent converters 635 in FIG. 6 and configured to convert intermediate exponents to exponents inother data formats. FIG. 9A shows the exponent converter 900A that includes an exponent extractor 910, a comparator 920, another comparator 925, and a MUX 930. The exponent converter 900A is coupled to an adder 940 and a MUX 950. In other embodiments, the exponent converter 900A may include fewer, more, or different components. The exponent converter 900A receives an exponent, which may be an intermediate exponent generated by the input generator 620. The exponent converter 900A may convert exponents in various data formats to exponents of a target data format.

[0151] In some embodiments, the exponent extractor 910 receives an intermediate exponent 901, which may be generated by the input generator 620. The exponent extractor 910 generates a signed exponent 902 from the intermediate exponent 901, e.g., by extracting the exponent sign bit and exponent value bits from the intermediate exponent901. The exponent sign bit defines the sign (e.g., negative or positive) of the signed exponent 902, and the exponent value bits define the absolute value of the signed exponent902. The exponent sign bit may be the most significant bit in the signed exponent 902, and the exponent value bits may be the least significant bits in the signed exponent 902. The exponent extractor 910 may extract the exponent sign bit and exponent value bits from the intermediate exponent 901 based on the output data format, e.g., the number of exponent value bits in the output data format. The total number of bits in the exponent extractor 910 may equal the number of exponent value bits in the output data format. In some embodiments, the number of exponent value bits in the output data format is encoded in a configuration parameter used by the exponent extractor 910 for generating the signed exponent 902.

[0152] The comparator 920 may compare the signed exponent 902 with a first threshold value. The threshold value may be the minimum exponent of the input data format. The comparator 925 may compare the signed exponent 902 with a second threshold value, which may be the maximum exponent of the input data format. The comparator 920 and comparator 925 may each provide a control signal to the MUX 930. The MUX 930 may have multiple input signals, including the signed exponent 902 and two fixed exponents 903 and 904. The MUX 930 may select one of the input signals to output based on the control signals from the comparator 920 and comparator 925.

[0153] The comparators 920 and 925 may be used to check underflow and overflow, respectively. Underflow may happen when the signed exponent 902 is too small to berepresented by the input data format, i.e., it is smaller than the minimum exponent that can be represented by the input data format. Overflow may happen when the signed exponent 902 is too large to be represented by the input data format, i.e., it exceeds the maximum exponent that can be represented by the input data format. In an example, when the signed exponent 902 is smaller than the first threshold value, the MUX 930 may select the exponent 903; when the signed exponent 902 is greater than the second threshold value, the MUX 930 may select the exponent 904; when the signed exponent 902 is in a range from the first threshold value to the second threshold value (i.e., not smaller than the first threshold value and not greater than the second threshold value), the MUX 930 may select the signed exponent 903. The output of the MUX 930 is a selected exponent 905 that follows the target data format.

[0154] The adder 940 and MUX 950 may be in an exponent collector, e.g., the exponent collector 650. The adder 940 may receive a carry value 906 from mantissa converter circuitry, which may result from mantissa overflow. The adder 940 may also receive the selected exponent 905 from the MUX 930. The adder 940 may accumulate the carry value 906 and selected exponent 905 and output a sum 907. The sum 907 may be an exponent that is greater than the selected exponent 905. The carry value 906 can also be used by the MUX 950 as a control signal for selecting either the sum 907 computed by the adder 940 or the fixed exponent 904. The MUX 950 outputs a converted exponent 908, which follows the output data format. The total number of bits in the converted exponent 908 may equal the total number of bits in the signed exponent 902.

[0155] In some embodiments, the exponent converter 900A may be specific for one output data format and cannot be used for converting data to other data formats, while the adder 940 and MUX 950 can be flexible and used for different data formats. FIG. 9B shows the exponent converter 900B that is configured to specifically generate 8-bit exponents. The exponent converter 900B may be used for generating data in data formats having 8-bit exponents, such as FP32 and BF16. FIG. 9C shows the exponent converter 900C that is configured to specifically generate 5-bit exponents. The exponent converter 900C may be used for generating data in data formats having 5-bit exponents, such as FP16. FIG. 9D shows the exponent converter 900D that is configured to specifically generate 3-bit exponents. The exponent converter 900D may be used for generating data in data formats having 3-bit exponents. The exponent converters in FIGS. 9B-9D may be examples of theexponent converters 635 in FIG. 6. There may be other exponent converters configured for data formats having other numbers of exponent bits. The exponent converters 900B-900D have similar components as the exponent converter 900A.

[0156] In FIG. 9B, the exponent converter 900B receives an intermediate exponent that has 8 bits, which are all valid bits. The 8 bits include 7 exponent value bits and 1 exponent sign bit. The exponent bits are the least significant bits in the intermediate exponent, and the sign bit is the most significant bit. The signed exponent may be the same as the intermediate exponent and is compared by the comparator 920 with -126, which is the minimum value of 8-bit exponents. The signed exponent is also compared with +127, which is the maximum value of 8-bit exponents, by the comparator 925. The results of the comparisons are used as control signals of the MUX 930. The MUX 930 selects an exponent value from the signed exponent, 00, and FF and outputs the selected exponent value.

[0157] In FIG. 9C, the exponent converter 900C receives an intermediate exponent that has 8 bits. The exponent extractor 910 extracts 5 valid bits (i.e., a 5-bit signed exponent), including 4 exponent value bits and 1 exponent sign bit, from the intermediate exponent. The signed exponent is compared by the comparator 920 with -14, which is the minimum value of 5-bit exponents. The signed exponent is also compared with +15, which is the maximum value of 5-bit exponents, by the comparator 925. The results of the comparisons are used as control signals of the MUX 930. The MUX 930 selects an exponent value from the signed exponent, 00, and IF and outputs the selected exponent value.

[0158] In FIG. 9D, the exponent converter 900D receives an intermediate exponent that has 8 bits. The exponent extractor 910 extracts 3 bits (i.e., a 3-bit signed exponent), including 2 exponent value bits and 1 exponent sign bit, from the intermediate exponent. The signed exponent is compared by the comparator 920 with -2, which is the minimum value of 3-bit exponents. The signed exponent is also compared with +3, which is the maximum value of 3-bit exponents, by the comparator 925. The results of the comparisons are used as control signals of the MUX 930. The MUX 930 selects an exponent value from the signed exponent, 0, and 7. The MUX 930 outputs the selected exponent value.

[0159] In the embodiments of FIGS. 9B-9D, the size requirement of the converter interface is 8 bits. All the inputs to the converter blocks may be 8 bits with the required LSB bits being taken into consideration. The valid exponents bits can be packed to the LSB side by the input generator. Each exponent converter may be designed specifically with hardcodedMUX inputs which can be implemented by tie-high and tie-low signal in actual synthesis. The design can be ultra-light weight in terms of area and power.

[0160] FIGS. 10A-10D illustrate mantissa converters 1000A-1000D, in accordance with various embodiments. FIG. 10A shows the mantissa converter 1000A that includes rounding blocks 1010A-1010D, a MUX 1020, and an adder 1030. In other embodiments, the mantissa converter 1000A may include fewer, more, or different components. The mantissa converter 1000A may receive an intermediate mantissa 1001 that has 23 bits. The mantissa converter 1000A may also receive the sign bit 1002 of the input data element.

[0161] The rounding blocks 1010A-1010D may round the intermediate mantissa 1001 in different rounding modes. In some embodiments, the rounding block 1010A may round the intermediate mantissa 1001 toward 0. For instance, the rounding block 1010A may discard all the bits from mil (i.e., the 11thbit) to m23 (i.e., the 23rdbit) in the intermediate mantissa 1001. The rounding block 1010B may round the intermediate mantissa 1001 toward +°° or - <x>. For instance, the rounding block 1010B may add 1 or not based on the sign bit. The rounding block 1010C may round the intermediate mantissa 1001 to nearest, e.g., with ties away from zero. For instance, the rounding block 1010C may add 1 when mil is 1. The rounding block 1010D may round the intermediate mantissa 1001 to nearest with ties toward even. For instance, the rounding block 1010D may add 1 when mil is 1 and at least one other bit of mlO through m23 is 1. The accuracy of the mantissa converter 1000A may depend on the truncation of additional least significant bits and rounding mode. The four different rounding modes can support it. The rounding blocks 1010A-1010D may provide different rounding outputs, which may be used as inputs to the MUX 1020.

[0162] The MUX 1020 may select one of the rounding outputs based on a control signal 1003, which may indicate a rounding mode. The MUX 1020 may select the rounding output that was generated by the rounding block operating in the same rounding mode. The MUX 1020 provides the selected rounding output 1004 to the adder 1030. The adder 1030 accumulates the intermediate mantissa 1001 with the selected rounding output 1004. The adder 1030 outputs a converted mantissa 1005, which may be the sum of the intermediate mantissa 1001 and the selected rounding output 1004. The adder 1030 may also output a carry value 1006, e.g., when there is mantissa overflow caused by the accumulation. The carry value 1006 may be used for exponent conversion, as described above.

[0163] The mantissa converters 1000B-1000D may have similar components as the mantissa converter 1000A. The mantissa converter 1000B in FIG. 9B is configured to generate 23-bit mantissas for generating data in data formats with mantissas having 23 bits, such as FP32. In some embodiments, the mantissa converter 1000B may receive 24 bits, which includes one sign bit and 23 mantissa bits. The 23 mantissa bits may be an intermediate mantissa. The 23 mantissa bits are rounded in the four different rounding modes, and one of the rounding outputs is selected by the MUX 1050. The adder 1030 accumulates the selected rounding output with the 23 mantissa bits and computes a converted mantissa and a carry value.

[0164] The mantissa converter 1000C in FIG. 9C is configured to generate 7-bit mantissas for generating data in data formats with mantissas having 7 bits, such as BF16. In some embodiments, the mantissa converter 1000C may receive 24 bits, which includes one sign bit and 23 mantissa bits. The 23 mantissa bits may be an intermediate mantissa. The 23 mantissa bits are rounded in the four different rounding modes, and one of the rounding outputs is selected by the MUX 1050. The adder 1030 accumulates the selected rounding output with 7 mantissa bits, i.e., a 7-bit mantissa. The 7 mantissa bits are extracted from the 23 mantissa bits based on the output data format. The adder 1030 computes a converted mantissa and a carry value.

[0165] The mantissa converter 1000D in FIG. 9D is configured to generate 4-bit mantissas for generating data in data formats with mantissas having 4 bits. In some embodiments, the mantissa converter 1000C may receive 24 bits, which includes one sign bit and 23 mantissa bits. The 23 mantissa bits may be an intermediate mantissa. The 23 mantissa bits are rounded in the four different rounding modes, and one of the rounding outputs is selected by the MUX 1050. The adder 1030 accumulates the selected rounding output with 4 mantissa bits, i.e., a 4-bit mantissa. The 4 mantissa bits are extracted from the 23 mantissa bits based on the output data format. The adder 1030 computes a converted mantissa and a carry value.

[0166] The mantissa converters 1000A-1000D may be examples of the mantissa converters 635 in FIG. 6 and configured to convert intermediate exponents to mantissas in other data formats. There may be more or different mantissa converters. One or more of the mantissa converters may be selected based on the application demand, and the selection of the mantissa converters can be achieved by using one or more configuration parameters. Mantissa converter with the same or similar design can be repurposed to handle integercases by making the exponent bits constant and reporting the carry value from the mantissa conversion circuitry as an overflow.

[0167] FIG. 11 illustrates an example sparse cell 1100, in accordance with various embodiments. The sparse cell 1100 may be a processing cell in a processing engine, e.g., the processing engine 350 in FIG. 3. The sparse cell 1100 includes 16 MAC units 1110 (individually referred to as "MAC unit 1110"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 4x4, meaning the height of the MAC array is 4 and the width of the MAC array is also 4. The sparse cell 1100 also includes 16 weight register files 1120 (individually referred to as "weight register file 1120"), 16 activation register files 1130 (individually referred to as "activation register file 1130"), four row buffers 1140 (individually referred to as "row buffer 1140"), and sparsity modules 1160 (individually referred to as "sparsity module 1160"). In other embodiments, the sparse cell 1100 may include fewer, more, or different components. For example, the sparse cell 1100 may include a different number of MAC units 1110, weight register files 1120, activation register files 1130, row buffers 1140, or sparsity modules 1160. As another example, the sparse cell 1100 may include column buffers in lieu of or in addition to the row buffers 1140. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0168] The MAC units 1110 are configured to perform MAC operations. Each MAC unit 1110 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 1110 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 11, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 1110. The number of adders in the first tier may be half of the number of the MAC units 1110, and each adder may accumulate the outputs of two MAC units 1110. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 1100.

[0169] The weight register files 1120 store weights to be processed in MAC operations. In the embodiments of FIG. 11, four weight register files 1120 are grouped into a storage set that stores data to be used by a column of MAC units 1110. There are four storage sets corresponding to the four columns of MAC units 1110. In some embodiments, a weight register file 1120 may correspond to a MAC unit 1110 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 1120 constitute a weight storage unit.

[0170] The activation register files 1130 stores activations to be processed in MAC operations. In the embodiments of FIG. 11, four activation register files 1130 are grouped into a storage set that stores data to be used by a row of MAC units 1110. There are four storage sets corresponding to the four rows of MAC units 1110. In some embodiments, an activation register file 1130 may correspond to a MAC unit 1110 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 1130 constitute an activation storage unit. The row buffers 1140 store outputs of the MAC units 1110. Each row buffer 1140 may drain outputs of a single row of MAC units 1110.

[0171] The sparsity module 1160 facilitates dynamic sparsity-based acceleration in the sparse cell 1100. In the embodiments of FIG. 11, each sparsity module 1160 includes a sparsity tensor storage unit 1165 and a control logic 1167. The sparsity tensor storage unit 1165 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 1165 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activationweight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0172] The control logic 1167 may control transmission of activations and weights stored from the weight register files 1120 and the activation register files 1130 to the MAC units 1110 based on sparsity tensors. For instance, the control logic 1167 may select a subset of the weights stored in the weight register files 1120 and select a subset of activations stored in the activation register files 1130 based on a combined sparsity tensor. The selected weights and activations constitute nonzero activation-weight pairs. The control logic 1167 may transmit the selected weights and activations to the MAC units 1110 for performingMAC operations. The other weights stored in the weight register files 1120 and the other activations stored in the activation register files 1130 are skipped from computation. In the embodiments of FIG. 11, each sparsity module 1160 controls sparsity acceleration in a respective MAC unit 1110. As the sparsity acceleration is either based on both weight sparsity and activation sparsity, 16 sparsity modules 1160 are used for acceleration computations in the 16 MAC units 1110.

[0173] As shown in FIG. 11, the sparse cell 1100 is associated with multiplexers (MUXs) 1103, 1104, 1105, and 1106. In other embodiments, the sparse cell 1100 may be associated with a different number of MUXs or other devices. The MUX 1103 facilitates loading weights, e.g., from the local memory 340, into the weight register files 1120. The MUX 1104 facilitates loading activations, e.g., from the local memory 340, into the activation register files 1130. The MUX 1105 facilitates loading sparsity tensors into the sparsity tensor storage unit 1165. The MUX 1106 may be a drain MUX that can facilitate draining outputs of the MAC units 1110, e.g., to the local memory 340.

[0174] In some embodiments, the sparse cell 1100 may also execute matrix multiplications converted from Fourier transform operations. For an example Fourier transform operation, the MAC units 1110 may perform MAC operations in the two sequences of matrix multiplications converted from the Fourier transform operation. The weight register files 1120 may be used to store data points in transformation tensor of the Fourier transform operation. The activation register file 1130 may be used to store data points in the input tensor of the Fourier transform operation. The row buffers 1140 may store data points in the output tensor of the Fourier transform operation.

[0175] FIG. 12 illustrates a sparse cell array 1200, in accordance with various embodiments. The sparse cell array 1200 may be an example of the processing engine 350 in FIG. 3. In FIG. 12, the sparse cell array 1200 includes sparse cells 1210 (individually referred to as "sparse cell 1210") arranged in four columns and four rows, an activation memory 1220, and a weight memory 1230. In other embodiments, the sparse cell array 1200 may include fewer, more, or different components. For instance, the sparse cell array 1200 may include a different number of columns, rows, or sparse cells 1210.

[0176] Each sparse cell 1210 may perform sparsity accelerated MAC operations. The sparse cells 1210 may facilitate dynamic sparsity mode. For instance, the sparsity modes of a sparse cell 1210 may be dynamically changed between a combined sparsity mode, an activationsparsity mode, a weight sparsity mode, and a dense mode. An embodiment of a sparse cell 1210 may be the sparse cell 1100 in FIG. 11. The activation memory 1220 stores activations, such as activations in input tensors of neural network operations. Activations may be loaded from the activation memory 1220 to sparse cells 1210. The weight memory 1230 stores weights, such as weights in filters of neural network operations. Weights may be loaded from the weight memory 1230 to sparse cells 1210. The activation memory 1220 or weight memory 1230 may be a buffer. In other embodiments, the sparse cell array 1200 may include a dense data memory and a sparse data memory in lieu of the activation memory 1220 and weight memory 1230. The dense data memory may store dense tensors. The sparse data memory may store sparse tensors.

[0177] The sparse cell array 1200 may also execute matrix multiplications in Fourier transform operations. The activation memory 1220 may be used to store input tensors of the Fourier transform operations. The weight memory 1230 may be used to store transformation matrices of the Fourier transform operations.

[0178] FIG. 13 illustrates an example PE 1300, in accordance with various embodiments. The PE 1300 may be a unit component of a processing cell, e.g., a processing cell in the processing engine 350. In the embodiments of FIG. 13, the PE 1300 includes an MAC unit 1305, an activation register file 1310, a weight register file 1320, an output register file 1350, and a sparsity accelerator 1360. The MAC unit 1305 includes a multiplier 1330 and an adder 1340. In other embodiments, the PE 1300 may include fewer, more, or different components.

[0179] The activation register file 1310 stores an activation operand, which may be a context. The activation register file 1310 may be an example of the activation register files 1130 in FIG. 11. The weight register file 1320 stores a weight operand. The weight register file 1320 may be an example of the weight register files 1120 in FIG. 11. The activation operand and weight operand may be loaded from a memory (e.g., the memory 340) into the activation register file 1310 and the weight register file 1320, respectively. The sparsity accelerator 1360 receives a sparsity bitmap 1315 that corresponds to the sparse tensor in the weight register file 1320. The sparsity bitmap 1315 may be a combined sparsity bitmap when the MAC unit 1305 operates in a combined sparsity mode. The sparsity bitmap 1315 may be an activation sparsity bitmap when the MAC unit 1305 operates in an activation sparsity mode. The sparsity bitmap 1315 may be a weight sparsity bitmap when the MACunit 1305 operates in a weight sparsity mode. The sparsity bitmap 1315 may have the same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0180] Using the sparsity bitmap 1315, the sparsity accelerator 1360 selects four activations from the activation register file 1310 and selects four weights from the weight register file 1320. The sparsity accelerator 1360 transmits the selected activations and weights to the multiplier 1330. These selected data elements correspond to the nonzero valued elements of the sparsity bitmap 1315. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 1330 may compute a product based on each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 1340. Even though FIG. 13 shows a single multiplier 1330, the MAC unit 1305 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0181] The adder 1340 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. For instance, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 1305 may operate in a dense mode in which the sparsity bitmap 1315 is not used and the sparsity accelerator 1360 is inactive. The MAC unit 1305 may process all the activations in the activation operand and all the weights in the weight operand.

[0182] The unit-level internal partial sum may be stored in the output register file 1350. In some embodiments, the unit-level internal partial sum may be used multiple times. For instance, the activation operand may represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all the N data blocks to compute N unit-level internal partial sums, the unit-level internal partial sum iscomputed once and used N times in the convolutional layers as N unit-level internal partial sums.

[0183] In some embodiments, the PE 1300 receives one or more PE-level internal partial sums from one or more other PEs. The adder 1340 or an accumulator (not shown in FIG. 13) can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of the PE 1300 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 1350. The one or more other PEs may be in the same column as the PE 1300 in a sparse cell. The multi-unit internal partial sum may be a columnlevel internal partial sum. In some embodiments, the PE-level internal partial sum of the PE 1300 or the multi-unit internal partial sum may be sent to one or more other PEs for further accumulation.Example Method of Precision Conversion in DNN

[0184] FIG. 14 is a flowchart of a method of converting precisions of data in DNNs, in accordance with various embodiments. The method 1400 may be performed by the poseprocessing engine 360 in FIG. 3. Although the method 1400 is described with reference to the flowchart illustrated in FIG. 14, many other methods for converting precisions of data in DNNs may alternatively be used. For example, the order of execution of the steps in FIG. 14 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0185] The post-processing engine 360 generates 1410 an intermediate exponent from one or more bits of a first data element based on a hardware interface configured to transfer a predetermined number of bits at a time. The first data element has an input precision and is computed in an operation of a neural network. The intermediate exponent comprises the predetermined number of bits. In some embodiments, the post-processing engine 360 extracts the one or more bits from the first data element before the intermediate exponent is generated. In some embodiments, the post-processing engine 360 receives a configuration parameter, e.g., from the DNN module 301. The configuration parameter indicates a total number of exponent bits for the input precision. The one or more exponent bits are extracted from the first data element based on the configuration parameter. In some embodiments, the post-processing engine 360 generates the intermediate exponent by adding one or more zero bits before or after the one or more bits.

[0186] The post-processing engine 360 transmits 1420 the intermediate exponent to an exponent converter through the hardware interface. The exponent converter is selected from a group of exponent converters based on an output precision that is different form the input precision. Different exponent converters in the group correspond to different data precisions.

[0187] The post-processing engine 360 generates 1430, by using the exponent converter, a second data element having the output precision. An exponent of the second data element is generated from the intermediate exponent. The second data element is to be processed in another operation of the neural network. In some embodiments, the post-processing engine 360 generates the second data element by extracting least significant bits from the intermediate exponent based on a configuration parameter indicating a number of the least significant bits. In some embodiments, the post-processing engine 360 generates the second data element by comparing one or more least significant bits of the intermediate exponent with a constant to detect underflow or overflow. In some embodiments, the post-processing engine 360 generates the exponent of the second data element by accumulating an output of the exponent converter with a value corresponding to a carry from mantissa rounding.

[0188] In some embodiments, the post-processing engine 360 extracts one or more other bits from the first data element. The post-processing engine 360 generates an intermediate mantissa from the one or more mantissa bits. The intermediate mantissa comprises the predetermined number of bits. The post-processing engine 360 generates the mantissa of the second data element from the intermediate mantissa. In some embodiments, the postprocessing engine 360 transmits the intermediate mantissa to a mantissa converter through the hardware interface. The mantissa converter is selected from a group of mantissa converters based on the output precision. Different mantissa converters in the group correspond to the different data precisions. The mantissa of the second data element is generated from the intermediate mantissa by using the mantissa converter.

[0189] In some embodiments, the post-processing engine 360 rounds the intermediate mantissa in different rounding modes to generate different rounding outputs. The postprocessing engine 360 selects a rounding output from the rounding outputs. The postprocessing engine 360 accumulates the rounding output with the one or more other bits.Example Computing Device

[0190] FIG. 15 is a block diagram of an example computing device 1500, in accordance with various embodiments. In some embodiments, the computing device 1500 can be used as at least part of the DNN system 300. A number of components are illustrated in FIG. 15 as included in the computing device 1500, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1500 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1500 may not include one or more of the components illustrated in FIG. 15, but the computing device 1500 may include interface circuitry for coupling to the one or more components. For example, the computing device 1500 may not include a display device 1506, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1506 may be coupled. In another set of examples, the computing device 1500 may not include an audio input device 1518 or an audio output device 1508 but may include audio input or output device interface circuitry to which an audio input device 1518 or audio output device 1508 may be coupled.

[0191] The computing device 1500 may include a processing device 1502 (e.g., one or more processing devices). The processing device 1502 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1500 may include a memory 1504, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1504 may include memory that shares a die with the processing device 1502. In some embodiments, the memory 1504 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for converting precisions of data in DNNs (e.g., the method 1400 described in conjunction with FIG. 14) or some operations performed by one or more components of the DNN system 300 (e.g., the post-processing engine 360, etc.). The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1502.

[0192] In some embodiments, the computing device 1500 may include a communication chip 1512 (e.g., one or more communication chips). For example, the communication chip1512 may be configured for managing wireless communications for the transfer of data to and from the computing device 1500. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0193] The communication chip 1512 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1512 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 1512 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1512 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1512 may operate in accordance with other wireless protocols in other embodiments. The computing device 1500 may include an antenna 1522 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0194] In some embodiments, the communication chip 1512 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1512 may include multiplecommunication chips. For instance, a first communication chip 1512 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1512 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1512 may be dedicated to wireless communications, and a second communication chip 1512 may be dedicated to wired communications.

[0195] The computing device 1500 may include battery / power circuitry 1514. The battery / power circuitry 1514 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1500 to an energy source separate from the computing device 1500 (e.g., AC line power).

[0196] The computing device 1500 may include a display device 1506 (or corresponding interface circuitry, as discussed above). The display device 1506 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0197] The computing device 1500 may include an audio output device 1508 (or corresponding interface circuitry, as discussed above). The audio output device 1508 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0198] The computing device 1500 may include an audio input device 1518 (or corresponding interface circuitry, as discussed above). The audio input device 1518 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0199] The computing device 1500 may include a GPS device 1516 (or corresponding interface circuitry, as discussed above). The GPS device 1516 may be in communication with a satellite-based system and may receive a location of the computing device 1500, as known in the art.

[0200] The computing device 1500 may include another output device 1510 (or corresponding interface circuitry, as discussed above). Examples of the other output device1510 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0201] The computing device 1500 may include another input device 1520 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1520 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0202] The computing device 1500 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1500 may be any other electronic device that processes data.Select Examples

[0203] The following paragraphs provide various examples of the embodiments disclosed herein.

[0204] Example 1 provides an apparatus, including an input generator configured to generate an intermediate exponent from one or more bits in a data element having an input precision, the intermediate exponent including a predetermined number of bits; a group of exponent converters, different exponent converters in the group corresponding to different data precisions; an exponent selector configured to select an exponent converter from the group of exponent converters based on an output precision that is different from the input precision; and a register configured to store a new data element having the output precision, an exponent of the new data element generated from the intermediate exponent by using the exponent converter, the new data element to be processed in another operation of the neural network.

[0205] Example 2 provides the apparatus of example 1, further including an interface that is configured to transfer the predetermined number of bits at a time, in which the exponentselector or exponent converter is configured to receive the intermediate exponent through the interface.

[0206] Example 3 provides the apparatus of example 1 or 2, in which the input generator is further configured to generate an intermediate mantissa from one or more other bits in the data element, the intermediate mantissa including the predetermined number of bits.

[0207] Example 4 provides the apparatus of example 3, further including a group of mantissa converters, different mantissa converters in the group corresponding to the different data precisions; and a mantissa selector configured to receive the intermediate mantissa through the interface and to select a mantissa converter from the group of mantissa converters, in which a mantissa of the new data element is generated by the mantissa converter from the intermediate mantissa.

[0208] Example 5 provides the apparatus of any one of examples 1-4, further including an extractor configured to extract the one or more bits from the data elements, the one or more bits representing an exponent of the data element.

[0209] Example 6 provides the apparatus of example 5, in which the extractor is further configured to receive a configuration parameter indicating a total number of exponent bits for the input precision, and the extractor is configured to extract the one or more bits based on the configuration parameter.

[0210] Example 7 provides the apparatus of any one of examples 1-6, in which the input generator is configured to generate the intermediate exponent by adding one or more zero bits before or after the one or more bits.

[0211] Example 8 provides the apparatus of any one of examples 1-7, in which the exponent converter includes one or more comparators, a comparator configured to compare one or more least significant bits of the intermediate exponent with a constant to detect underflow or overflow.

[0212] Example 9 provides the apparatus of any one of examples 1-8, further including an exponent collector configured to receive outputs of the different exponent converters and to generator exponents for the different data precisions, in which the exponent of the new data element is generated by further using the exponent collector.

[0213] Example 10 provides the apparatus of example 9, in which the exponent collector includes an adder.

[0214] Example 11 provides a method, including generating an intermediate exponent from one or more bits of a data element based on a hardware interface configured to transfer a predetermined number of bits at a time, the data element having an input precision and computed in an operation of a neural network, the intermediate exponent including the predetermined number of bits; transmitting the intermediate exponent to an exponent converter through the hardware interface, the exponent converter selected from a group of exponent converters based on an output precision that is different form the input precision, different exponent converters in the group corresponding to different data precisions; and generating, by using the exponent converter, a new data element having the output precision, an exponent of the new data element generated from the intermediate exponent, the new data element to be processed in another operation of the neural network.

[0215] Example 12 provides the method of example 11, further including extracting the one or more bits from the data element before the intermediate exponent is generated.

[0216] Example 13 provides the method of example 12, further including receiving a configuration parameter, the configuration parameter indicating a total number of exponent bits for the input precision, in which the one or more exponent bits are extracted from the data element based on the configuration parameter.

[0217] Example 14 provides the method of any one of examples 11-13, further including extracting one or more other bits from the data element; generating an intermediate mantissa from the one or more mantissa bits, the intermediate mantissa including the predetermined number of bits; and generating the mantissa of the new data element from the intermediate mantissa.

[0218] Example 15 provides the method of example 14, further including transmitting the intermediate mantissa to a mantissa converter through the hardware interface, the mantissa converter selected from a group of mantissa converters based on the output precision, different mantissa converters in the group corresponding to the different data precisions, in which the mantissa of the new data element is generated from the intermediate mantissa by using the mantissa converter.

[0219] Example 16 provides the method of example 15, in which generating the mantissa of the new data element includes rounding the intermediate mantissa in different rounding modes to generate different rounding outputs; selecting a rounding output from the rounding outputs; and accumulating the rounding output with the one or more other bits.

[0220] Example 17 provides the method of any one of examples 11-16, in which generating the intermediate exponent includes adding one or more zero bits before or after the one or more bits.

[0221] Example 18 provides the method of any one of examples 11-17, in which generating the new data element includes extracting least significant bits from the intermediate exponent based on a configuration parameter indicating a number of the least significant bits.

[0222] Example 19 provides the method of any one of examples 11-18, in which generating the new data element includes comparing one or more least significant bits of the intermediate exponent with a constant to detect underflow or overflow.

[0223] Example 20 provides the method of any one of examples 11-19, in which generating the new data element includes generating the exponent of the new data element by accumulating an output of the exponent converter with a value corresponding to a carry from mantissa rounding.

[0224] Example 21 provides an apparatus, including a multi-precision converter including an input generator configured to generate an intermediate exponent from one or more bits in a first data element, the intermediate exponent including a predetermined number of bits, the first data element having an input precision, a group of exponent converters, different exponent converters in the group corresponding to different data precisions, an exponent selector configured to select an exponent converter from the group of exponent converters based on an output precision that is different from the input precision, and a register configured to store a second data element having the output precision, an exponent of the second data element generated from the intermediate exponent by using the exponent converter; and a compute block configured to perform an operation in a neural network using the second data element.

[0225] Example 22 provides the apparatus of example 21, in which the output precision is higher than the input precision.

[0226] Example 23 provides the apparatus of example 21 or 22, further including an additional multi-precision converter, the compute block configured to compute a third data element from the second data element, the additional multi-precision converter configured to receive the second data element and to generate a fourth data element with another output precision.

[0227] Example 24 provides the apparatus of example 23, in which the another output precision is lower than the output precision.

[0228] Example 25 provides the apparatus of any one of examples 21-24, further including an additional compute block configured to compute the data element by performing another operation in the neural network.

[0229] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

1. Claims1. An apparatus, comprising: an input generator configured to generate an intermediate exponent from one or more bits in a first data element having an input precision, the intermediate exponent comprising a predetermined number of bits; a group of exponent converters, different exponent converters in the group corresponding to different data precisions; an exponent selector configured to select an exponent converter from the group of exponent converters based on an output precision that is different from the input precision; and a register configured to store a second data element having the output precision, an exponent of the second data element generated from the intermediate exponent by using the exponent converter.

2. The apparatus of claim 1, further comprising: an interface that is configured to transfer the predetermined number of bits at a time, wherein the exponent selector or exponent converter is configured to receive the intermediate exponent through the interface.

3. The apparatus of claim 1 or 2, wherein the input generator is further configured to generate an intermediate mantissa from one or more other bits in the first data element, the intermediate mantissa comprising the predetermined number of bits.

4. The apparatus of claim 3, further comprising: a group of mantissa converters, different mantissa converters in the group corresponding to the different data precisions; and a mantissa selector configured to receive the intermediate mantissa through the interface and to select a mantissa converter from the group of mantissa converters, wherein a mantissa of the second data element is generated by the mantissa converter from the intermediate mantissa.

5. The apparatus of any one of claims 1-4, further comprising: an extractor configured to extract the one or more bits from the first data elements, the one or more bits representing an exponent of the first data element.

6. The apparatus of claim 5, wherein the extractor is further configured to receive a configuration parameter indicating a total number of exponent bits for the input precision, and the extractor is configured to extract the one or more bits based on the configuration parameter.

7. The apparatus of any one of claims 1-6, wherein the input generator is configured to generate the intermediate exponent by adding one or more zero bits before or after the one or more bits.

8. The apparatus of any one of claims 1-7, wherein the exponent converter comprises one or more comparators, a comparator configured to compare one or more least significant bits of the intermediate exponent with a constant to detect underflow or overflow.

9. The apparatus of any one of claims 1-8, further comprising: an exponent collector configured to receive outputs of the different exponent converters and to generator exponents for the different data precisions, wherein the exponent of the second data element is generated by further using the exponent collector.

10. The apparatus of claim 9, wherein the exponent collector comprises an adder.

11. A method, comprising: generating an intermediate exponent from one or more bits of a first data element based on a hardware interface configured to transfer a predetermined number of bits at a time, the first data element having an input precision and computed in an operation of a neural network, the intermediate exponent comprising the predetermined number of bits;transmitting the intermediate exponent to an exponent converter through the hardware interface, the exponent converter selected from a group of exponent converters based on an output precision that is different form the input precision, different exponent converters in the group corresponding to different data precisions; and generating, by using the exponent converter, a second data element having the output precision, an exponent of the second data element generated from the intermediate exponent, the second data element to be processed in another operation of the neural network.

12. The method of claim 11, further comprising: extracting the one or more bits from the first data element before the intermediate exponent is generated.

13. The method of claim 12, further comprising: generating a configuration parameter, the configuration parameter indicating a total number of exponent bits for the input precision, wherein the one or more exponent bits are extracted from the first data element based on the configuration parameter.

14. The method of any one of claims 11-13, further comprising: extracting one or more other bits from the first data element; generating an intermediate mantissa from the one or more mantissa bits, the intermediate mantissa comprising the predetermined number of bits; and generating the mantissa of the second data element from the intermediate mantissa.

15. The method of claim 14, further comprising: transmitting the intermediate mantissa to a mantissa converter through the hardware interface, the mantissa converter selected from a group of mantissa converters based on the output precision, different mantissa converters in the group corresponding to the different data precisions,wherein the mantissa of the second data element is generated from the intermediate mantissa by using the mantissa converter.

16. The method of claim 15, wherein generating the mantissa of the second data element comprises: rounding the intermediate mantissa in different rounding modes to generate different rounding outputs; selecting a rounding output from the rounding outputs; and accumulating the rounding output with the one or more other bits.

17. The method of any one of claims 11-16, wherein generating the intermediate exponent comprises: adding one or more zero bits before or after the one or more bits.

18. The method of any one of claims 11-17, wherein generating the second data element comprises: extracting least significant bits from the intermediate exponent based on a configuration parameter indicating a number of the least significant bits.

19. The method of any one of claims 11-18, wherein generating the second data element comprises: comparing one or more least significant bits of the intermediate exponent with a constant to detect underflow or overflow.

20. The method of any one of claims 11-19, wherein generating the second data element comprises: generating the exponent of the second data element by accumulating an output of the exponent converter with a value corresponding to a carry from mantissa rounding.

21. An apparatus, comprising: a multi-precision converter comprising:an input generator configured to generate an intermediate exponent from one or more bits in a first data element, the intermediate exponent comprising a predetermined number of bits, the first data element having an input precision, a group of exponent converters, different exponent converters in the group corresponding to different data precisions, an exponent selector configured to select an exponent converter from the group of exponent converters based on an output precision that is different from the input precision, and a register configured to store a second data element having the output precision, an exponent of the second data element generated from the intermediate exponent by using the exponent converter; and a compute block configured to perform an operation in a neural network using the second data element.

22. The apparatus of claim 21, wherein the output precision is higher than the input precision.

23. The apparatus of claim 21 or 22, further comprising: an additional multi-precision converter, the compute block configured to compute a third data element from the second data element, the additional multi-precision converter configured to receive the second data element and to generate a fourth data element with another output precision.

24. The apparatus of claim 23, wherein the another output precision is lower than the output precision.

25. The apparatus of any one of claims 21-24, further comprising: an additional compute block configured to compute the data element by performing another operation in the neural network.

Images