Floating-point sum-of-accumulate unit for easy variable data precision

Abstract

A fused dot product (MAC) circuit may support variable precision of floating-point data elements to perform calculations (e.g., MAC operations) in deep learning operations. The operating mode of the circuit may be selected based on the precision of the input elements. The operating mode may be FP16 mode or FP8 mode. In FP8 mode, the product exponent may be calculated based on the exponent of the floating-point input elements. The maximum exponent may be selected from one or more product exponents. The global maximum exponent may be selected from multiple maximum exponents. The product mantissa is calculated based on the difference between the global maximum exponent and the corresponding maximum exponent and may be aligned with another product mantissa. The adder tree may accumulate the aligned product mantissas and calculate a partial sum mantissa. The partial sum mantissa may be normalized using the global maximum exponent.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to U.S. Patent Application No. 18 / 362,529, filed on 31 July 2023, entitled “FLOATING-POINT MULTIPLY-ACCUMULATE UNIT FACILITATING VARIABLE DATA PRECISION,” which is incorporated herein by reference in its entirety.

[0002] [Technical field] The present invention generally relates to multiply-accumulate (MAC) operations, and more specifically to a floating-point MAC (FPMAC) unit that facilitates variable data precision. [Background technology]

[0003] Deep neural networks (DNNs) are widely used in various artificial intelligence applications, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the expense of considerable computational cost. DNNs have extremely high computational requirements, as each inference can require hundreds of millions of MAC operations and the reading and writing of large amounts of data. Therefore, techniques are needed to improve the efficiency of DNNs. [Brief explanation of the drawing]

[0004] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, similar reference numerals indicate similar structural elements. Embodiments shown in the figures of the accompanying drawings are illustrative, not limiting. [Figure 1] This document presents exemplary DNNs in various embodiments. [Figure 2] This section illustrates exemplary convolutions in various embodiments. [Figure 3] This is a block diagram of a DNN accelerator in various embodiments. [Figure 4] Exemplary processing elements (PEs) having FPMAC units according to various embodiments are shown. [Figure 5A] This document describes FPMAC units capable of skipping mantissa multiplication in various embodiments. [Figure 5B] This document describes FPMAC units capable of skipping mantissa multiplication in various embodiments. [Figure 6] This document illustrates FPMAC units that support variable floating-point precision in various embodiments. [Figure 7] This document demonstrates FP16 mantissa calculation in FPMAC units under various embodiments. [Figure 8A] This document demonstrates FP8 mantissa calculation in FPMAC units under various embodiments. [Figure 8B] This document demonstrates FP8 mantissa calculation in FPMAC units under various embodiments. [Figure 9A] This shows data paths in FPMAC units that support variable floating-point precision according to various embodiments. [Figure 9B] This shows data paths in FPMAC units that support variable floating-point precision according to various embodiments. [Figure 10] This shows a maximum exponential module with OR trees in various embodiments. [Figure 11] Exemplary PE arrays in various embodiments are shown. [Figure 12] This is a block diagram of PE in various embodiments. [Figure 13] This flowchart shows methods for performing FPMAC calculations using various embodiments. [Figure 14] This is a block diagram illustrating an exemplary computing device in various embodiments. [Modes for carrying out the invention]

[0005] [Summary] In the past decade, there has been a rapid increase in artificial intelligence (AI)-based data processing, especially that based on deep neural networks (DNNs). DNNs are widely used in the areas of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve accuracies beyond human levels. Combined with the rapid increase in the computing power of execution platforms, significant improvements in DNN model size and accuracy have led to the adoption of DNN applications even within resource-constrained mobile devices and edge devices with limited energy availability.

[0006] A DNN layer may include one or more deep learning operations such as convolution, pooling, element-wise operations, linear operations, non-linear operations, etc. The deep learning operations in a DNN layer may be performed on one or more internal parameters of the DNN layer and the input data received by the DNN layer. The internal parameters (e.g., weights) of a DNN layer may be determined during the training phase.

[0007] The internal parameters or input data of a DNN layer may be elements of a tensor. A tensor is a data structure having a plurality of elements across one or more dimensions. Exemplary tensors include a vector which is a one-dimensional tensor and a matrix which is a two-dimensional tensor. Three-dimensional tensors and even higher-dimensional tensors can also exist. A DNN layer may have an input tensor (also referred to as an “input feature map (IFM)”) including one or more input activations (also referred to as “input elements” or “activations”), a weight tensor including one or more weights, and an output tensor (also referred to as an “output feature map (OFM)”) including one or more output activations (also referred to as “output elements” or “activations”). The weight tensor of a convolution may be a kernel, a filter, or a group of filters.

[0008] Increasing the size of a DNN leads to an increase in the resources required for DNN training and inference. Larger-width floating-point formats often cannot achieve high energy efficiency. Lower-precision integer formats can achieve high energy efficiency but often require extensive model tuning or optimizer hyperparameters. Narrow-bitwidth integer formats show some advantages for inference, but FP8 can achieve desirable accuracy across the range of DNNs for both training and inference without requiring extensive tuning or optimizer hyperparameters. Many existing DNNs use the FP16 (half-precision floating-point) format, HF16 (IEEE half-precision floating-point format), and BF16 (Brain Floating Point) format. However, the FP8 (8-bit floating-point) format can accelerate deep learning training and inference for better performance and energy efficiency. Migrating to the FP8 format can be beneficial. However, currently available DNN accelerators typically support the FP16 format and various integer formats but cannot support the FB8 format.

[0009] Embodiments of the present disclosure provide a DNN accelerator having an FPMAC unit that can support variable FP formats including FP16 and FP8 formats such as HF16, BF16, HF8, BF8, other types of FP18, and FP8 formats, or some combination thereof. The FPMAC unit may include a fused dot product MAC circuit having one or more merged data paths that support both the FP16 format and the FP8 format. In an exemplary FPMAC unit, for maximum reuse of hardware components for FP16 mantissa multiplication, the FP16 mantissa multiplication may be reconfigured into a 2-way FP8 dot product. The reconfigurability can reduce the energy overhead required to support the FP8 format.

[0010] In various embodiments, the FPMAC unit may support variable precision of floating-point data elements to perform calculations (e.g., MAC operations) in deep learning operations. The control module may select the operating mode of the FPMAC unit from a plurality of operating modes based on the precision of at least one of the floating-point data elements to be processed. One or more multiplication and alignment modules may operate in the selected operating mode.

[0011] In an example where the operating mode is for a floating-point data format with lower precision (e.g., FP8 format), the multiplication and alignment modules within the FPMAC unit may calculate one or more product exponents based on the exponents of the floating-point data elements and select the largest exponent from the one or more product exponents. This largest exponent is local to the multiplication and alignment module and can therefore also be called the local largest exponent. One or more local largest exponents calculated by one or more multiplication and alignment modules within the FPMAC unit may be transmitted to a maximum exponent module within the FPMAC unit. The maximum exponent module may select the largest exponent from one or more maximum exponents. The largest exponent selected by the maximum exponent module can be applied to multiple or all multiplication and alignment modules and is therefore also called the global largest exponent. The multiplication and alignment modules may also calculate a product mantissa in which one or more bits are shifted based on the difference between the global largest exponent and the local largest exponent. The shift can align the product mantissa with one or more other product mantissas. The adder tree within the FPMAC unit may accumulate the aligned product mantissas and calculate a partial sum mantissa. The partial sum mantissa may be normalized using the global maximum exponent. The result of the normalization may also be the output of the FPMAC unit.

[0012] The FPMAC unit can skip the calculation of the mantissa if the mantissa multiplication does not affect the output of the adder tree. For example, if the mantissa were calculated and aligned and had a bit width (e.g., number of bits) that exceeds the bit width limit of the adder tree, the mantissa would not be calculated. Skipping mantissa multiplication can be facilitated by using an OR tree in the maximum exponent module to reduce the time delay required to determine whether the bit width of the mantissa exceeds the bit width limit. In another example, if another mantissa is infinite or NaN (not a number), the mantissa would not be calculated. Skipping mantissa multiplication can reduce the energy consumed by the FPMAC unit to perform that calculation.

[0013] By using an FPMAC unit configurable for both FP16 and FP8 formats, this disclosure enables significant area reduction compared to separate FP16 dot product and FP8 dot product implementations. The combined, reconfigurable design can minimize energy overhead while supporting multiple input format encodings.

[0014] For illustrative purposes, specific numbers, materials, and configurations are described to provide a complete understanding of exemplary implementations. However, it will be apparent to those skilled in the art that this disclosure may be implemented without specific details, or / or using only some of the embodiments described. In other examples, well-known features are omitted or simplified so as not to obscure the exemplary embodiments.

[0015] Furthermore, references are made to the attached drawings, which form part of this disclosure, and the drawings illustrate possible embodiments. It should be understood that other embodiments may be used and structural or logical modifications may be made without departing from the scope of this disclosure. Accordingly, the following detailed description should not be construed as restrictive.

[0016] Various operations may be described sequentially as multiple separate operations or calculations in the manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these operations are necessarily order-dependent. In particular, these operations do not have to be performed in the order presented. The described operations may be performed in a different order than in the embodiments described. In further embodiments, various further operations may be performed, or the described operations may be omitted.

[0017] For the purposes of this disclosure, the phrase "A or B" or "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B or C" or "A, B and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in relation to a range of scale, "between" includes both ends of that range.

[0018] This description uses the phrases “in an embodiment” or “in an embodiment,” which may each refer to one or more of the same or different embodiments. Terms such as “equipped,” “included,” and “having” as used in reference to embodiments of this disclosure are synonymous. This disclosure may use descriptions based on perspective views, such as “top,” “bottom,” “upper,” “bottom,” and “side,” to describe various features of the drawings, but these terms are for illustrative purposes only and do not suggest a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” and “third” to describe common objects simply indicates that different instances of similar objects are being referred to, and is not intended to suggest that the objects thus described must be in a given order, temporally, spatially, in rank, or in any other manner.

[0019] In the following detailed description, various aspects of exemplary implementations are described using terminology commonly used by those skilled in the art to communicate their research to others skilled in the art.

[0020] The terms “substantially,” “close,” “approximately,” “near,” and “about” generally indicate that the values ​​are within + / - 20% of a target value based on specific values ​​described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally indicate that the values ​​are within + / - 5 to 20% of a target value based on specific values ​​described herein or known in the art.

[0021] Furthermore, the terms “equipped,” “possessing,” “included,” “contained,” “having,” “possessing,” or any other variation thereof, are intended to cover non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that includes a list of elements is not necessarily limited to these elements alone, and may include other elements that are not explicitly listed or that are specific to such a method, process, device, or DNN accelerator. Also, the term “or” indicates an inclusive “or” and not an exclusive “or.”

[0022] Each of the systems, methods, and devices disclosed herein has several novel embodiments, and not one of these embodiments alone may possess all the desired attributes disclosed herein. Details of one or more implementations of the objects described herein are given in the following description and accompanying drawings.

[0023] [Example DNN] Figure 1 shows exemplary DNN100 in various embodiments. For illustrative purposes, DNN100 in Figure 1 is a CNN. In other embodiments, DNN100 may be other types of DNNs. DNN100 is trained to receive an image and output the classification of objects in the image. In the embodiment of Figure 1, DNN100 receives an input image 105 containing objects 115, 125, and 135. DNN100 includes a series of layers, including a plurality of convolutional layers 110 (individually referred to as “convolutional layer 110”), a plurality of pooling layers 120 (individually referred to as “pooling layer 120”), and a plurality of fully connected layers 130 (individually referred to as “fully connected layer 130”). In other embodiments, DNN100 may include fewer layers, more layers, or different layers. In DNN100 inference, the layers of DNN100 perform numerous tensor operations, such as convolution (e.g., multiply-accumulate (MAC) operations), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication), other types of tensor operations, or any combination thereof.

[0024] The convolutional layer 110 summarizes the presence of features in the input image 105. The convolutional layer 110 functions as a feature extractor. The first layer of the DNN 100 is the convolutional layer 110. In one example, the convolutional layer 110 performs convolution on the input tensor 140 (also called IFM140) and the filter 150. As shown in Figure 1, the IFM140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM140 contains three input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix contains seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3x3x3 three-dimensional matrix. The filter 150 contains three kernels, each of which may correspond to a different input channel of the IFM140. A kernel is a 2D matrix of weights, with the weights arranged in the columns and rows. A kernel can be smaller than an IFM. In the embodiment shown in Figure 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel contains three weights in each row and three weights in each column. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140.

[0025] The convolution involves a MAC operation using the input elements in IFM140 and the weights in filter 150. The convolution may be a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides across IFM140. All input channels are combined to generate an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements (also called output points) in each row and five output elements in each column. For illustrative purposes, the standard convolution includes one filter in the embodiment shown in Figure 1. In embodiments with multiple filters, the standard convolution may generate multiple output channels in OFM 160.

[0026] The multiplication applied between a kernel-sized patch and a kernel in IFM140 may be a dot product. The dot product is an element-wise multiplication between a kernel-sized patch and the corresponding kernel in IFM140, which is then summed, always resulting in a single value. Because this results in a single value, the operation is often called a "scalar product." It is intentional to use a kernel smaller than IFM140, as this allows the same kernel (set of weights) to be multiplied by IFM140 multiple times at different points on IFM140. Specifically, the kernel is applied systematically from left to right and top to bottom to each overlapping portion or kernel-sized patch of IFM140. The result of multiplying a kernel by IFM141 once is a single value. When a kernel is applied to IFM140 multiple times, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from the standard convolution 163 (i.e., OFM160) is called OFM.

[0027] In depth convolution 183, the input channels are not combined. Rather, MAC operations are performed on each individual input channel and each individual kernel to generate output channels. As shown in Figure 1, depth convolution 183 generates depth output tensor 180. Depth output tensor 180 is represented by a 5x5x3 3D ​​matrix. Depth output tensor 180 contains three output channels, each of which is represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements in each row and five output elements in each column. Each output channel is the result of a MAC operation between the input channels of IFM 140 and the kernels of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal stripes) is the result of a MAC operation between the second input channel (patterned with horizontal stripes) and the second kernel (patterned with horizontal stripes), and the third output channel (patterned with diagonal stripes) is the result of a MAC operation between the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depth convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are collectively referred to as depth channels. After the depth convolution, a pointwise convolution 193 is performed on the depth output tensor 180 and the 1x1x3 tensor 190 to generate OFM 160.

[0028] Next, OFM160 is passed to the next layer in the sequence. In some embodiments, OFM160 is passed through an activation function. An exemplary activation function is ReLU. ReLU is a computation that directly returns the value provided as input, or returns zero if the input is less than or equal to zero. The convolutional layer 110 may take several images as input and compute a convolution with each of these kernels. This process can be repeated multiple times. For example, OFM160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 following the convolutional layer 110 that generates OFM160 in the sequence). The subsequent convolutional layer 110 performs a convolution on OFM160 using the new kernel to generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelized again by further subsequent convolutional layers 110, and so on.

[0029] In some embodiments, the convolutional layer 110 has four hyperparameters, namely, the number of kernels, a kernel of size F (for example, the kernel has dimensions of F × F × D pixels), S steps in which the window corresponding to the kernel is dragged on the image (for example, a step of 1 means moving the window by 1 pixel at a time), and zero padding P (for example, adding a black contour of P pixel thickness to the input image of the convolutional layer 110). The convolutional layer 110 may perform various types of convolution, such as 2D convolution, dilated or atrous convolution, spatially separable convolution, depth separable convolution, transposed convolution, etc. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0030] The pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features in patches of the feature map. The pooling layer 120 is placed between two convolutional layers 110, namely, a preceding convolutional layer 110 (a convolutional layer 110 that precedes the pooling layer 120 in the layer sequence) and a succeeding convolutional layer 110 (a convolutional layer 110 that follows the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after an activation function (e.g., ReLU) is applied to OFM 160.

[0031] The pooling layer 120 receives the feature map generated by the preceding convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation preserves these important properties while reducing the size of the feature map. Therefore, the pooling operation improves the efficiency of the DNN and avoids overfitting. The pooling layer 120 may perform the pooling operation through mean pooling (calculating the average value of each patch on the feature map), max pooling (calculating the maximum value of each patch on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of 2 pixels, and as a result, the pooling operation reduces the size of the feature map by half, for example, the number of pixels or values ​​in the feature map is reduced to one-quarter of the size. In one example, a pooling layer 120 applied to a 6x6 feature map produces a pooled feature map with a 3x3 output. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates separately for each feature map to create a new set of the same number of pooled feature maps.

[0032] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be a convolutional layer. The fully connected layer 130 accepts an input operand. The input operand defines the outputs of the convolutional layer 110 and the pooling layer 120 and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operand to generate a vector. The vector may contain as many elements as there are classes, where element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1, and the sum of all is 1. These probabilities are calculated in the last fully connected layer 130 by using the logistic function (binary classification) or the softmax function (multi-class classification) as the activation function.

[0033] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. In the embodiment of Figure 1, there are three objects 115, 125, and 135 in the input image, so N is equal to 3. Each element of the operand represents the probability that the input image 105 belongs to a certain class. To calculate the probabilities, the fully connected layer 130 multiplies each input element by its weight, takes the sum, and then applies an activation function (e.g., logistic for N=2, softmax for N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In one example, the vector contains three probabilities: a first probability that object 115 is a tree, a second probability that object 125 is a car, and a third probability that object 135 is a person. In other embodiments, where the input image 105 contains different objects or different numbers of objects, the individual values ​​may be different.

[0034] [Example of convolution] Figure 2 shows exemplary convolutions in various embodiments. The convolution may be performed in a convolutional layer of a DNN, for example, in convolutional layer 110 in Figure 1. The convolutional layer may also be a front-end layer. The convolution can be performed on an input tensor 210 and a filter 220 (each referred to as "filter 220"). The result of the convolution is the output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator that includes one or more computation blocks. An example of a DNN accelerator may be the DNN accelerator 300 in Figure 3. An example of a computation block may be the computation block 325 in Figure 3.

[0035] In the embodiment shown in Figure 2, the input tensor 210 includes activations (also called "input activations," "elements," or "input elements") arranged in a 3D matrix. Activations in the input tensor 210 are data points in the input tensor 210. The input tensor 210 has a spatial size H in ×W in ×C in It has H in is the height of the 3D matrix (i.e., the length along the Y-axis, which represents the number of activations in the column of the 2D matrix for each input channel), and W in is the width of the 3D matrix (i.e., the length along the X-axis, which represents the number of activations in each row of the 3D matrix for each input channel), and C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For the purposes of simplification and illustration, the input tensor 210 has a spatial size of 7 × 7 × 3, i.e., the input tensor 210 contains 3 input channels, each input channel having a 7 × 7 2D matrix. Each input element of the input tensor 210 may be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 210 may vary.

[0036] Each filter 220 contains weights arranged in a 3D matrix. The weight values ​​may be determined through training the DNN. The filter 220 has a spatial size H f ×Wf ×C f has, and H f is the height of the filter (i.e., the length along the Y-axis, which indicates the number of weights in each column of the kernel), and W f is the width of the filter (i.e., the length along the X-axis, which indicates the number of weights in each row of the kernel), and C f is the depth of the filter (i.e., the length along the Z-axis, which indicates the number of channels). In some embodiments, C f is equal to C in . For simplicity and illustrative purposes, each filter 220 in FIG. 2 has a spatial size of 3×3×3, i.e., the filter 220 includes three convolutional kernels with a spatial size of 3×3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0037] Activations or weights can take one or more bytes in memory. The number of bytes for activations or weights may depend on the data format. For example, when an activation or weight has an integer format (e.g., INT8), the activation takes 1 byte. When an activation or weight has a floating-point format (e.g., FP16 or BF16), the activation or weight takes 2 bytes. Other data formats may be used for activations or weights.

[0038] In convolution, each filter 220 slides across the input tensor 210 to generate a 2D matrix for the output channels in the output tensor 230. In the embodiment of FIG. 2, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activations (also referred to as "output activations", "elements", or "output elements") arranged in a 3D matrix. The activations in the output tensor 230 are data points in the output tensor 230. The output tensor 230 has a spatial size of H out ×W out ×C out and H outis the height of the 3D matrix (i.e., the length along the Y-axis, which represents the number of output activations in the column of the 2D matrix for each output channel), and W out is the width of the 3D matrix (i.e., the length along the X-axis, which represents the number of output activations in the row of the 2D matrix for each output channel), and C out This is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of output channels). out H may be equal to the number of filters 220 in the convolution. out and W out This may depend on the input tensor 210 and the heights and weights of each filter 220.

[0039] As part of the convolution, MAC operations can be performed on the input tensor 210 and the 3x3x3 subtensor 215 in each filter 220 (highlighted by a dot pattern in Figure 2). The result of the MAC operations on the subtensor 215 and one filter 220 is the output activation. In some embodiments (e.g., embodiments where the convolution is integral convolution), the output activation may contain 8 bits, e.g., 1 byte. In other embodiments (e.g., embodiments where the convolution is floating-point convolution), the output activation may contain more than 1 byte. For example, the output element may contain 2 bytes.

[0040] After the MAC operations on subtensor 215 and all filters 220 are completed, vector 235 is generated. Vector 235 is highlighted with diagonal lines in Figure 2. Vector 235 contains a sequence of output activations arranged along the Z-axis. Output activations in vector 235 have the same (x,y) coordinates, but they correspond to different output channels and have different Z coordinates. The dimension of vector 235 along the Z-axis may be equal to the total number of output channels in output tensor 230.

[0041] After vector 235 is generated, further MAC operations are performed to generate further vectors until output tensor 230 is generated. For example, filter 220 may move along the X or Y axis on input tensor 210, and MAC operations can be performed on filter 220 and another subtensor in input tensor 210 (the subtensor having the same size as filter 220). The amount of movement of filter 220 on input tensor 210 during different computational rounds of the convolution is called the stride size of the convolution. The stride size may be 1 (i.e., the amount of movement of filter 220 is one activation), 2 (i.e., the amount of movement of filter 220 is two activations), etc. The height and width of output tensor 230 may be determined based on the stride size.

[0042] In some embodiments, MAC operations on a 3×3×3 subtensor (e.g., subtensor 215) and filter 220 may be performed by multiple PEs. One or more PEs may receive activation operands (e.g., activation operand 217 shown in Figure 2) and weight operands (e.g., weight operand 227 shown in Figure 2). Activation operand 217 includes a sequence of activations having the same (Y,Z) coordinates but different X coordinates. Weight operand 227 includes a sequence of weights having the same (Y,Z) coordinates but different X coordinates. The length of activation operand 217 is the same as the length of weight operand 227. The activations in activation operand 217 and the weights in weight operand 227 may be supplied to the PE sequentially. The PE may receive activation-weight pairs containing the activations and their corresponding weights at once and multiply the activations and weights. The positions of the activations in activation operand 217 may coincide with the positions of the weights in weight operand 227.

[0043] The activation or weight may be a floating-point number. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits including one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing the exponent of the floating-point number, and bits representing the mantissa of the floating-point number. The mantissa is part of the floating-point number that represents the significant figures of the floating-point number. The mantissa is multiplied by the base raised to the power of the exponent to give the actual value of the floating-point number.

[0044] Floating-point numbers may have various precisions and data formats, such as FP32 (single-precision floating-point) format, FP16 format, and FP8 format. A floating-point number in FP16 format may be represented by 16 bits, including a sign bit, several bits representing the exponent (e.g., 5 bits or 8 bits), and several bits representing the mantissa (e.g., 10 bits or 7 bits). The FP8 format has lower precision than the FP16 format. A floating-point number in FP8 format may be represented by 8 bits, including a sign bit, several bits representing the exponent (e.g., 5 bits or 4 bits), and several bits representing the mantissa (e.g., 2 bits or 3 bits).

[0045] The multiplication of floating-point activation and floating-point weights may include the calculation of a product exponent based on the exponents of two floating-point numbers and the calculation of a product mantissa based on the mantissas of two floating-point numbers. The product exponent may be the sum of the two exponents. The product mantissa may be the product of the two mantissas.

[0046] To sum the products of multiple activation-weight pairs, the product mantissa may be aligned by shifting bits in the product mantissa based on the product exponent. For example, the largest exponent may be selected from the product exponents, and the difference between the largest exponent and the product exponents of the activation-weight pairs may determine how much to shift one or more bits in the product mantissa of the activation-weight pairs. The shifted product mantissa may be accumulated to calculate a partial sum mantissa, which may be further normalized based on the largest exponent.

[0047] In some embodiments, the number of mantissa bits in a floating-point number may need to be adjusted to satisfy the number of mantissa bits in the target FP format (for example, some bits in the mantissa may need to be truncated). Furthermore, the exponent bits may be determined based on the number of exponent bits in the FP format. Normalization of an FP number may include, for example, changing the exponential form of the FP number to satisfy the FP format. In some embodiments, normalization of a floating-point number may include removing one or more leading zeros in the floating-point number. The leading zeros may be zero bits that precede non-zero bits in the floating-point number. The normalized floating-point number may not have leading zeros. The decimal point may also be moved, and the exponent may be adjusted according to the removal of leading zeros. The result of normalization may also be the result of MAC operations on multiple activation-weight pairs. Floating-point activation and MAC operations on floating-point weights may be performed by a PE having an FPMAC unit such as FPMAC unit 410 in Figure 4, FPMAC unit 500 in Figure 5A, or FPMAC unit 600 in Figure 6.

[0048] [Example DNN accelerator] Figure 3 is a block diagram of the DNN accelerator 300 in various embodiments. The DNN accelerator 300 can perform deep learning operations in a DNN. The DNN accelerator 300 may be used for DNN training and inference. In the embodiment of Figure 3, the DNN accelerator 300 includes memory 310, a direct memory access (DNA) engine 320, and a compute block 330 (each referred to as “Compute Block 330”). In other embodiments, alternative configurations, different components, or further components may be included in the DNN accelerator 300. For example, the DNN accelerator 300 may include more than one memory 310 or more than one DMA engine 320. As another example, the DNN accelerator 300 may include a single compute block 330. Furthermore, functions attributable to the components of the DNN accelerator 300 may be achieved by different components included in the DNN accelerator 300 or by different systems. The components of the DNN accelerator 300 may be implemented as hardware, software, firmware, or any combination thereof.

[0049] Memory 310 stores data related to deep learning operations performed by the DNN accelerator 300. In some embodiments, memory 310 may store data used by the compute block 330 to perform deep learning operations. For example, memory 310 may store weights, such as the weights of the convolutional layers determined by training the DNN. Memory 310 may also store data generated by the compute block 330 from performing deep learning operations in the DNN. Exemplary deep learning operations include convolution (also called “convolution operations”), pooling operations, element-wise operations, activation functions, other types of deep learning operations, or any combination thereof. Memory 310 may also be the main memory of the DNN accelerator 300. In some embodiments, memory 310 includes one or more dynamic random-access memories (DRAM).

[0050] The DMA engine 320 facilitates data transfer between memory 310 and the local memory of the compute block 330. For example, the DMA engine 320 can read data from memory 310 and write data to the local memory of the compute block 330. As another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to memory 310. The DMA engine 320 provides DMA functionality that allows the compute block 330 to initiate data transfer between memory 310 and its local memory and to perform other operations while the data transfer is taking place. In some embodiments, the DMA engine 320 may read a tensor from memory 310 and modify the tensor in a manner optimized for the compute block 330 before writing the tensor to the local memory of the compute block 330.

[0051] The compute block 330 can perform deep learning operations in the DNN. For example, the compute block 330 may perform deep learning operations or parts of deep learning operations in the DNN layer at once. The compute block 330 may be capable of performing various types of deep learning operations, such as convolution, pooling, element-wise operations, linear operations, and nonlinear operations. In one example, the compute block 330 may perform a convolution, such as a standard convolution or a depth convolution. In some embodiments, the compute block 330 receives an input tensor and one or more convolution kernels and performs a convolution using the input tensor and convolution kernels. The result of the convolution may be an output tensor, which can be further computed, for example, by the compute block 330 or another compute block 330. In some embodiments, the operation of the DNN layer may be performed in parallel by multiple compute blocks 330. For example, multiple compute blocks 330 may each perform a part of the workload for convolution. Data may be shared among the compute blocks 330. The calculation block 330 may also be called a calculation tile. In some embodiments, each calculation block 330 may also be a processing unit.

[0052] In the embodiment shown in Figure 3, each compute block 330 includes a local memory 340, a PE array 350, a control module 360, a sparsity accelerator 370, and a post-processing unit 380. Some or all components of the compute block 330 can be implemented on the same chip. In other embodiments, alternative configurations, different components, or further components may be included in the compute block 330. Furthermore, functions resulting from the components of the compute block 330 may be achieved by different components included in the compute block 330, different compute blocks 330, different components of the DNN accelerator 300, or different systems. For example, the control module 360 ​​may not be part of the compute block 330, or it may not be part of the DNN accelerator 300. As another example, the control module 360 ​​may be part of the PE array 350, part of the PE row within the PE array 350, or part of the PE within the PE array 350. The components of the compute block 330 may be implemented in hardware, software, firmware, or any combination thereof.

[0053] The local memory 340 is local to the corresponding compute block 330. In the embodiment shown in Figure 3, the local memory 340 is located inside the compute block 330. In other embodiments, the local memory 340 may be located outside the compute block 330. The local memory 340 may store data received, used, or generated by the PE array 350 and the post-processing unit 380. Examples of data may include input activations, weights, output activations, sparsity bitmaps, etc. Data in the local memory 340 may be transferred to and from memory 210, for example, through the DMA engine 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory of another compute block 330.

[0054] In some embodiments, the local memory 340 includes one or more static random access memories (SRAM). The local memory 340 may be byte-addressable, with each memory address identifying a single byte (8 bits) of storage. In some embodiments, the local memory 340 may include memory banks. The number of data banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 3048, or other numbers. A memory bank may include multiple storage units. In one example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit within a memory bank may have a memory address. In one example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with contiguous memory addresses, i.e., adjacent storage units. For example, one memory unit may store an integer in INT8 format or a floating-point number in FP8 format, while two memory units may be required to store a number in FP16 or BF16 format having 16 bits. In some embodiments, the 16 bits can be transferred from local memory 340 in a single read cycle. In other embodiments, the 16 bits can be transferred from local memory 340 in multiple read cycles, such as two cycles.

[0055] The PE array 350 may include PEs arranged in columns, or in columns and rows. Each PE can perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more adders (also called "adders") for performing accumulation. A column of PEs is called a PE column. A PE column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into a MAC column. A MAC lane may also be called a data transmission lane or data loading lane. A PE column may have multiple MAC lanes. The loading bandwidth of a MAC column is the sum of the loading bandwidths of all MAC lanes associated with the MAC column. Using a certain number of MAC lanes, data can be fed simultaneously to the same number of independent PEs. In some embodiments, where a MAC column has four MAC lanes for feeding activation or weights into the MAC column, and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes may have a total loading bandwidth of 64 bytes.

[0056] In some embodiments, the PE array 350 may be capable of depth convolution, standard convolution, or both. In depth convolution, the PE may perform a MAC operation which includes a sequence of multiplications for the input operands and weight operands. Each multiplication in the sequence (also called a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activations and weights in the same cycle may correspond to the same channel. The sequence of multiplications produces a product operand which includes a sequence of products. The MAC operation may also include an accumulation which is performed by accumulating multiple product operands to produce an output operand of the PE. The PE array 350 may output multiple output operands at once, each output operand being produced by a different PE. In standard convolution, the MAC operation may include an accumulation which is performed across channels. For example, instead of producing an output operand, the PE may accumulate products across different channels to produce a single output point.

[0057] The PEs within the PE array 350 may include one or more configurable FPMAC units capable of processing floating-point data elements at various accuracies, including FP16 and FP8 formats. Examples of FPMAC units include FPMAC unit 410 in Figure 4, FPMAC unit 500 in Figure 5A, and FPMAC unit 600 in Figure 6. An FPMAC unit may include a fused dot product MAC circuit having one or more merged data paths for floating-point data with different accuracies. In some embodiments, an FPMAC unit may receive a configuration signal from the control module 360 ​​and operate according to the configuration signal. For example, an FPMAC unit may have multiple operating modes for processing data at different accuracies. The configuration signal may indicate which operating mode the FPMAC unit is operating in.

[0058] The control module 360 ​​controls the operating mode of the FPMAC unit in the PE array 350. In some embodiments, the control module 360 ​​may control the operating mode of the FPMAC unit based on the precision of the data processed by the FPMAC unit. For example, the control module 360 ​​may determine whether the data precision is lower or higher. The control module 360 ​​may determine the data precision based on the data format. For example, data having an FP16 format may be determined to have higher precision, and data having an FP8 format may be determined to have lower precision. In embodiments, the control module 360 ​​may generate a configuration signal in response to the determination that the data precision is lower and transmit the configuration signal to the FPMAC unit. The configuration signal may configure the operating mode of the FPMAC unit to a mode that can be used to process data elements with lower precision. In response to the determination that the data precision is higher, the control module 360 ​​may generate a different configuration signal and transmit the different configuration signal to the FPMAC unit. The different configuration signal may configure the operating mode of the FPMAC unit to a different mode that can be used to process data elements with higher precision.

[0059] In some embodiments, the control module 360 ​​may transmit the same configuration signal to multiple FPMAC units, for example, FPMAC units used to perform MAC operations in floating-point activation or convolution with floating-point weights. The control module 360 ​​may provide different configuration signals to the same FPMAC units at different times, for example, in different computational rounds in which the FPMAC units process data with different precisions.

[0060] The control module 360 ​​may also control the clock cycles in the operation of the FPMAC unit, for example, through one or more clocks. The FPMAC unit may use multiple cycles to calculate the product of a floating-point activation and a floating-point weight. In embodiments, the MAC operation on the activation operand and the weight operand may include three or more cycles. The FPMAC unit may calculate the product exponent of the activation-weight pair and find the maximum exponent in the first cycle. In the second cycle, the FPMAC unit may calculate the product mantissa and align the product mantissas. In the third cycle, the FPMAC unit may accumulate the aligned product mantissas and normalize the sum.

[0061] The control module 360 ​​may facilitate skipping the mantissa multiplication in the second cycle in some embodiments, such as embodiments where the result of mantissa multiplication does not affect the result of the MAC operation, or embodiments where the result of mantissa multiplication is known. For example, before the start of the second cycle, the control module 360 ​​may decide whether to skip the second and third cycles based on the product exponent and the maximum exponent. The control module 360 ​​may also decide whether the product mantissa has a bit width that exceeds the bit width limit if it has been shifted (for example, shifted based on the difference between the maximum exponent and the corresponding product exponent).

[0062] The bit width may also be the number of bits in the shifted mantissa. The bit width limit may also be the bit width limit of the adder tree in the FPMAC unit that computes the mantissa together with other mantissas. The adder tree may be implemented with a fixed width, resulting in truncation of the mantissa when the alignment causes a large shift that exceeds its width. This data-dependent situation occurs in many floating-point addition operations that require alignment, but can become more likely as the number of terms in the fused dot product increases. The energy consumed when computing the mantissa is wasted when not used in the adder tree. Therefore, mantissa multiplication skipping can reduce the amount of power required for the MAC operation without affecting the output of the MAC operation. The control module 360 ​​may compare the alignment shift amount with a threshold related to the adder tree width. After determining that the bit width of the mantissa exceeds the bit width limit, the control module 360 ​​transmits a gate signal to the FPMAC unit. After receiving the gate signal, the FPMAC unit may skip the calculations in the second and third cycles.

[0063] In another embodiment, the control module 360 ​​may generate and provide a gate signal to skip mantissa multiplication when the result of the adder tree can be known without mantissa multiplication, for example, if one of the multipliers generates infinity or NaN. These conditions must be determined during the first cycle so that the gate signal becomes available before the start of the second cycle.

[0064] The sparsity accelerator 370 accelerates computations in the PE array 350 based on sparsity in activations or weights. In some embodiments (for example, an embodiment in which computation block 330 performs a convolutional layer), computations in the PE may also be MAC operations on input operands and weight operands. The input operands may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operands may include one or more weights in the filter of the convolution. The values ​​of the weights are determined through training the DNN. The weights in the weight operands may be in different input channels.

[0065] In some embodiments, an input operand is associated with an activation bitmap that may be stored in local memory 340. The activation bitmap can indicate the location of non-zero activations in the input operand. The activation bitmap may contain multiple bits, each bit corresponding to a specific activation in the input operand. The location of a bit in the activation bitmap may coincide with the location of a corresponding activation in the input operand. The bits in the activation bitmap may be 0 or 1. A 0-value bit indicates that the value of the corresponding activation is zero, and a 1-value bit indicates that the value of the corresponding activation is non-zero. In some embodiments, the activation bitmap may be generated during the execution of another DNN layer, for example, a layer placed before a convolutional layer in the DNN.

[0066] In some embodiments, a weight operand is associated with a weight bitmap that may be stored in local memory 340. The weight bitmap can indicate the positions of non-zero weights within the weight operand. The weight bitmap may contain multiple bits, each of which corresponds to a specific weight within the weight operand. The positions of the bits in the weight bitmap may coincide with the positions of the corresponding weights within the weight operand. The bits in the weight bitmap may be 0 or 1. A 0-value bit indicates that the corresponding weight has a value of zero, and a 1-value bit indicates that the corresponding weight has a value of non-zero.

[0067] In some embodiments, the sparsity accelerator 370 may receive an activation bitmap and a weight bitmap and generate a combined sparsity bitmap for MAC operations to be performed by the PE. In some embodiments, the sparsity accelerator 370 generates a combined sparsity bitmap 735 by performing one or more AND operations on the activation bitmap and the weight bitmap. Each bit in the combined sparsity bitmap is the result of an AND operation on the bit in the activation bitmap and the bit in the weight bitmap, i.e., the product of the bit in the activation bitmap and the bit in the weight bitmap. The position of a bit in the combined sparsity bitmap corresponds to the position of a bit in the activation bitmap and the position of a bit in the weight bitmap. The bits in the combined bitmap correspond to activation and weight pairs (activation-weight pairs). A zero bit in the combined sparsity bitmap indicates that at least one of the activation and weight in the pair is zero. A single bit in the combined sparsity bitmap indicates that both the activation and weight in the pair are non-zero. The combined sparsity bitmap may be stored in local memory 340.

[0068] The sparsity accelerator 370 may provide activations and weights to the PE based on a combined sparsity bitmap. For example, the sparsity accelerator 370 may identify one or more non-zero activation-weight pairs from local memory 340 based on a combined sparsity bitmap. The local memory 340 may store the input operands and weight operands in a compressed format such that non-zero activations and non-zero weights are stored, but zero activations and zero weights are not. Non-zero activations of the input operands may constitute a compressed input operand. Non-zero weights of the weight operands may constitute a compressed weight operand. For non-zero activation-weight pairs, the sparsity accelerator 370 may determine the location of the activations in the compressed input operand and the location of the weights in the compressed weight operand based on the activation bitmap, weight bitmap, and combined bitmap. The activations and weights can be read from local memory 340 based on the locations determined by the sparsity accelerator 370.

[0069] In some embodiments, the sparsity accelerator 370 includes sparsity acceleration logic that can compute a position bitmap based on an activation bitmap and a weight bitmap. The sparsity accelerator 370 may determine the position indices of activations and weights based on the position bitmap. For example, the position index of an activation in a compressed input operand may be equal to the number of ones in the activation position bitmap generated by the sparsity accelerator 370, and the position index of a weight in a compressed weight operand may be equal to the number of ones in the weight position bitmap generated by the sparsity accelerator 370. The position indices of activations or weights indicate the position of the activation or weight in the compressed input operand or the compressed weight operand. The sparsity accelerator 370 may read the activations and weights from one or more memories based on these position indices.

[0070] The sparsity accelerator 370 can transfer identified non-zero activation-weight pairs to the PE. The sparsity accelerator 370 may skip other activations and weights because they do not contribute to the MAC operation result. In some embodiments, local memory 340 may store non-zero activations and weights, but not zero activations or weights. Non-zero activations and weights may be loaded into one or more register files of the PE, from which the sparsity accelerator 370 may retrieve the activations and weights corresponding to those in the combined sparsity bitmap. In some embodiments, the total number of 1s in the combined sparsity bitmap is equal to the total number of activation-weight pairs computed by the PE, while the PE computes no other activation-weight pairs. By skipping activation-weight pairs corresponding to zero bits in the combined sparsity bitmap, the PE's computation becomes faster compared to the PE computed all activation-weight pairs in the input operands and weight operands.

[0071] The sparsity accelerator 370 may be implemented in hardware, software, firmware, or any combination thereof. In some embodiments, at least a portion of the sparsity accelerator 370 may be located inside a PE. Although Figure 4 shows a single sparsity accelerator 370, the compute block 330 may include multiple sparsity accelerators. In some embodiments, each PE in the PE array 350 is implemented with a sparsity accelerator 370 to accelerate computations and reduce power consumption in the individual PE. In other embodiments, a subset of the PE array 350 (e.g., one or more PE columns in the PE array 350) may be implemented with a sparsity accelerator 370 to accelerate computations in the subset of PEs.

[0072] The post-processing unit 380 processes the output of the PE array 350. In some embodiments, the post-processing unit 380 computes an activation function. The post-processing unit 380 may receive the output of the PE array 350 as input to the activation function. The post-processing unit 380 may transmit the output of the activation function to local memory 340. The output of the activation function may be retrieved later by the PE array 350 from local memory 340 for further computation. For example, the post-processing unit 380 may receive the output tensor of a DNN layer from the PE array 350 and compute one or more activation functions for the output tensor. The results of the computation by the post-processing unit 380 may be stored in local memory 340 and later used as the input tensor for the next DNN layer. In addition to or instead of the activation function, the post-processing unit 380 may perform other types of post-processing on the output of the PE array 350. For example, the post-processing unit 380 may apply a bias to the output of the PE array 350.

[0073] In some embodiments, local memory 340 may be associated with a load path, and a drain path may be used for data transfer within the compute block 330. For example, data may be transferred from local memory 340 to the PE array 350 via the load path. Data may also be transferred from the PE array 350 to local memory 340 via the drain path. A post-processing unit 380 may be located on the drain path to process the output of the PE array before the data is written to local memory 340.

[0074] [Example FPMAC unit in PE] Figure 4 shows an exemplary PE400 having an FPMAC unit 410 in various embodiments. The PE400 also includes an input storage unit 420, a weight storage unit 430, an accumulator 480, and an output storage unit 490. The FPMAC unit 410 includes multipliers 450A-D (collectively referred to as “multiple multipliers 450” or “multiplier 450”) and an adder tree 440. The adder tree 440 includes adders 460A and 460B and an adder 465. In other embodiments, alternative configurations, different components, or further components may be included in the PE400. For example, the PE400 may include more than one FPMAC unit. The FPMAC unit 410 may include a different number of multipliers. The adder tree 440 may include a different number of adders. Furthermore, functions resulting from the components of the PE400 may be achieved by different components contained within the PE400, different components contained within the PE array in which the PE400 is located, or by different systems. The locations of the components of the PE400 in Figure 4 are for illustrative purposes only. While the locations of the components may reflect the direction of data flow in the PE400, the locations of the components in Figure 4 do not necessarily represent the physical locations of the components within the PE400.

[0075] PE400 may perform sequential cycles of MAC operations. In a MAC operation cycle, PE400 may process multiple input operands and multiple weight operands, for example, if there are multiple multipliers 450 in the FPMAC unit 410. Activations are provided to and stored in the input storage unit 420. In some embodiments, the input storage unit 420 may store activations for up to four input operands in a MAC operation cycle. Weights are provided to and stored in the weight storage unit 430. The weight storage unit 430 may store weights for up to four weight operands in a MAC operation cycle. Multipliers 450 may fetch activations and weights from the input storage unit 420 and the weight storage unit 430 and calculate their product. In an exemplary round, each multiplier 450 receives an activation and its corresponding weight and outputs the product of the activation and the weight. In other cycles, the activations and weights may be reused by different multipliers 450. The activations and weights from the input memory unit 420 and the weight memory unit 430 may be reused more than once.

[0076] The activations or weights may be data elements having a floating-point format such as FP16 or FP8. In some embodiments, the multiplier 450 may calculate an 8-way FP16 dot product or a 16-way FP8 dot product. The multiplier 450 may have two or more operating modes. For example, the multiplier 450 may have an FP16 operating mode for calculating an 8-way FP16 dot product and an FP8 operating mode for calculating a 16-way FP8 dot product. Further details regarding the FP16 and FP8 operating modes are provided below in reference to Figures 6, 7, 8A, and 8B.

[0077] The adder tree 440 receives the dot product calculated by the multiplier 450 and accumulates the dot products. In some embodiments, the dot product received by the adder tree 440 may also be an aligned product mantissa. Adder 460A receives the product calculated by multipliers 450A and 450B and calculates the first sum. Adder 460B receives the product calculated by multipliers 450C and 450D and calculates the second sum. Adder 465 receives the first and second sums from pipeline registers 470A and 470B, accumulates the sums, and generates the output of the FPMAC unit 410. Although not shown in Figure 4, the FPMAC unit 410 may include a normalization module that can normalize the output of the adder tree 440.

[0078] The output of the FPMAC unit 410 is further provided to the accumulator 480. The accumulator 480 adds the output of the FPMAC unit 410 to a value stored in the output storage unit 490. This value may be the output of another PE400 that has been transmitted to the PE400 and stored in the output storage unit 490. The output of the accumulator 480 can be stored in the output storage unit 490.

[0079] Figures 5A and 5B show various embodiments of an FPMAC unit 500 capable of skipping mantissa multiplication. The FPMAC unit 500 may also be an embodiment of the FPMAC unit 410 in Figure 4. As shown in Figure 5A, the FPMAC unit 500 includes a product and align module 510 (each referred to as the "product and align module 510"), a maximum exponent module 520, an adder tree 530, and a normalization module 540. The product and align module 510 and the maximum exponent module 520 may also be embodiments of the multiplier 450 in the FPMAC unit 410 in Figure 4. The adder tree 530 and the normalization module 540 may also be embodiments of the adder tree 440 in the FPMAC unit 410 in Figure 4.

[0080] In other embodiments, alternative configurations, different components, or further components may be included in the FPMAC unit 500. Furthermore, functions attributable to the components of the FPMAC unit 500 may be achieved by different components included in the FPMAC unit 500, different components included in the PE where the FPMAC unit 500 is located, or different devices. The positions of the components of the FPMAC unit 500 in Figure 5 are for illustrative purposes only. The positions of the components may reflect the direction of data flow in the FPMAC unit 500, but the positions of the components within the FPMAC unit 500 do not necessarily represent the physical locations of the components within the FPMAC unit 500.

[0081] The FPMAC unit 500 may receive an activation operand containing a sequence of floating-point activations and a weight operand containing a sequence of floating-point weights. The activations and weights may be distributed to the product and alignment module 510. For example, the product and alignment module 510 may receive the activations and weights. The product and alignment module 510 may calculate the product exponent and product mantissa based on the floating-point activation-weight pairs.

[0082] As shown in Figure 5A, the product and alignment module 510 includes an adder 512, a subtractor 514, a multiplier 516, and a shifter 518. The adder 512 may calculate a product exponent (represented by "ep" in Figure 5A) by accumulating the exponent (represented by "ea" in Figure 5A) of a first floating-point number (e.g., activation) with the exponent (represented by "eb" in Figure 5A) of a second floating-point number (e.g., weight). The product exponent may be transmitted to the maximum exponent module 520.

[0083] The Max Exponent Module 520 may receive different product exponents from different product and alignment modules 510, as shown in the product exponents listed in Table 501 in Figure 5B. The Max Exponent Module 520 outputs the Max Exponent (represented by "maxexp" in Figures 5A and 5B), which can be the largest product exponent received by the Max Exponent Module 520. In the embodiment of Figure 5B, the Max Exponent is the third product exponent in Table 501. The Max Exponent Module 520 provides the Max Exponent to the subtractor 514 in each product and alignment module 510 that provided the product exponent to the Max Exponent Module 520.

[0084] The subtractor 514 in the product and alignment module 510 may subtract the product exponent calculated by the adder 512 from the maximum exponent, or vice versa, to calculate the difference between the product exponent and the maximum exponent. This difference is transmitted to the shifter 518. This difference may also be called the shift coefficient. Table 502 in Figure 5B shows the difference between the product exponent in Table 501 and the maximum exponent in Table 501.

[0085] The multiplier 516 multiplies the mantissa of the first floating-point number (represented by "ma" in Figure 5A) and the mantissa of the second floating-point number (represented by "mb" in Figure 5A) to calculate the product mantissa (represented by "mp" in Figure 5A). The product mantissa may be transmitted to the shifter 518.

[0086] The shifter 518 shifts one or more bits in the product mantissa based on the difference calculated by the subtractor 514. The shift of the product mantissa in the product and align module 510 aligns the product mantissa calculated by these product and align module 510, allowing for the output of aligned product mantissas, as shown in Table 503 in Figure 5B. This alignment facilitates the accumulation of product mantissas by the adder tree 530.

[0087] In some embodiments, the operation of the adder 512, the maximum exponent module 520, and the subtractor 514 may be performed in the first clock cycle, while the operation of the multiplier 516 and the shifter 518 may be performed in the second clock cycle following the first clock cycle. The operation of the multiplier 516 and the shifter 518 may be skipped in some embodiments. In one example, the operation of the multiplier 516 and the shifter 518 is performed when the amount shifted by the shifter 518 is a threshold, for example, the mantissa of the adder tree 530 has a fixed bit width (in Figures 5A and 5B, "W") f The operation may be skipped when it exceeds a threshold shift amount which can cause it to exceed (represented by ). As shown in Figure 5B, the bit width of the last aligned product mantissa in table 503 exceeds the fixed bit width of the adder tree 530. Therefore, the operation of the multiplier 516 and shifter 518 to calculate the last aligned product mantissa in table 503 can be skipped.

[0088] In another example, the operation of the multiplier 516 and shifter 518 may be skipped when the output of the adder tree 530 can be known without a product mantissa to be calculated, for example, when another product and align module 510 outputs a product mantissa of infinity or NaN. In some embodiments, the product and align module 510 may receive a gate signal from, for example, the control module 360. After receiving the gate signal, the product and align module 510 skips the operation of the multiplier 516 and shifter 518 in the second cycle.

[0089] The adder tree 530 may receive one or more aligned product mantissas from the product and align module 510. The adder tree 530 may include multiple adders (not shown in Figure 5A or Figure 5B) arranged in a hierarchy. The number of adders in the first hierarchy may be half the number of product and align module 510 in the FPMAC unit 500. Each adder in the first hierarchy may be associated with two product and align module 510 and accumulate the aligned product mantissas calculated by the two product and align module 510. The number of adders in the second hierarchy may be half the number of adders in the first hierarchy. This may continue up to the last hierarchy which may have a single adder. Each adder may receive two numbers and output the sum of the two numbers.

[0090] The output of adder tree 530 ("partial sum mantissa") is W f The partial sum mantissa may have a bit width equal to +Log2N, where N is the number of aligned product mantissas accumulated by the adder tree 530. The partial sum mantissa is transmitted to the normalization module 540. The maximum exponent is also transmitted to the normalization module 540. The normalization module 540 may normalize the partial sum mantissa based on the maximum exponent, for example, by shifting one or more bits in the partial sum mantissa based on the maximum exponent. The result of the normalization may also be the result of a MAC operation.

[0091] Figure 6 shows an FPMAC unit 600 supporting variable floating-point precision in various embodiments. The FPMAC unit 600 may also be an embodiment of the FPMAC unit 410 in Figure 4. As shown in Figure 6, the FPMAC unit 600 includes a multiplier and align module 610 (each referred to as the "multiplier and align module 610"), a maximum exponent module 620, an adder tree 630, and a normalization module 640. The multiplier and align module 610 and the maximum exponent module 620 may also be embodiments of the multiplier 450 in the FPMAC unit 410 in Figure 4. The adder tree 630 and the normalization module 640 may also be embodiments of the adder tree 440 in the FPMAC unit 410 in Figure 4.

[0092] In other embodiments, alternative configurations, different components, or further components may be included in the FPMAC unit 600. Furthermore, functions attributable to the components of the FPMAC unit 600 may be achieved by different components included in the FPMAC unit 600, different components included in the PE where the FPMAC unit 600 is located, or different devices. The positions of the components of the FPMAC unit 600 in Figure 6 are for illustrative purposes only. The positions of the components may reflect the direction of data flow in the FPMAC unit 600, but the positions of the components within the FPMAC unit 600 do not necessarily represent the physical locations of the components within the FPMAC unit 600.

[0093] The FPMAC unit 600 may receive an activation operand containing a sequence of floating-point activations and a weight operand containing a sequence of floating-point weights. The activations and weights may be distributed to the product and align module 610. For example, the product and align module 610 may receive the activations and weights. The product and align module 610 may calculate the product exponent and product mantissa based on the floating-point activation-weight pairs.

[0094] The FPMAC unit 600 (in particular the product and alignment module 610) may be configurable for multiple operating modes, such as FP16 mode, FP8 mode, etc. The operating modes of the FPMAC unit 600 may be controlled by the control module 360. In some embodiments, the FPMAC unit 600 may support multiple formats for each precision. For example, the FPMAC unit 600 may support E5M10 and E8M7 for FP16, or E5M2 and E4M3 for FP8, using, for example, the wider of the possible exponential width and mantissa width. Each FP16 or FP8 input element may be in a different format. The FPMAC unit 600 may reuse a higher precision multiplier to compute a lower precision dot product by performing local exponential difference and alignment in a lower precision mode. In one example, the multiplier may compute a × b in FP16, while it may compute a × b + c × d in the FP8 mode being computed. In some embodiments, the FP8 input elements may be packed within the same bits as the FP16 input element so that two FP8 input elements can fit within the same bit width as a single FP16 input element.

[0095] In FP16 mode, the product and align module 610 receives the activation and weights in FP16 format. The adder 612 within the product and align module 610 may calculate the product exponent (represented by "ep" in Figure 6) by accumulating the exponent (represented by "ea" in Figure 6) of a first floating-point number (e.g., activation) with the exponent (represented by "eb" in Figure 6) of a second floating-point number (e.g., weight). The product exponent may be transmitted to the maximum exponent module 620.

[0096] The maximum exponent module 620 may receive different product exponents from different product and alignment modules 610. The maximum exponent module 620 outputs the maximum exponent which can be the largest product exponent received by the maximum exponent module 620. The maximum exponent module 620 provides the maximum exponent to the subtractor 614 in each product and alignment module 610 that provided the product exponent to the maximum exponent module 620. The subtractor 614 in the product and alignment module 610 may subtract the product exponent calculated by the adder 612 from the maximum exponent, or vice versa, to calculate the difference between the product exponent and the maximum exponent. This difference is transmitted to the shifter 618.

[0097] The multiplier 616 in the multiplication and alignment module 610 multiplies the mantissa of a first floating-point number (represented by "ma" in Figure 6) and the mantissa of a second floating-point number (represented by "mb" in Figure 6) to calculate the product mantissa (represented by "mp" in Figure 6). The product mantissa may be transmitted to the shifter 618. The shifter 618 in the multiplication and alignment module 610 shifts one or more bits in the product mantissa based on the difference calculated by the subtractor 614. The shift of the product mantissa in the multiplication and alignment module 610 aligns the product mantissas calculated by these multiplication and alignment modules 610, allowing the aligned product mantissa to be output. This alignment facilitates the accumulation of product mantissas by the adder tree 630. Further details regarding the FP16 mode are described below in relation to Figure 7.

[0098] In FP8 mode, the product and align module 610 may receive two activations and two weights in FP8 format for a calculation round. An adder 611 within the product and align module 610 may calculate the first product exponent by accumulating the activation exponent (represented by "ea0" in Figure 6) with the weight exponent (represented by "eb0" in Figure 6). Another adder 611 may calculate the second product exponent by accumulating the other activation exponent (represented by "ea1" in Figure 6) with the other weight exponent (represented by "eb1" in Figure 6). The two product exponents are transmitted to the maximum value detector 615, which selects the higher product exponent as the local maximum exponent (represented by "ep" in Figure 6). The local maximum exponent may be transmitted to the maximum exponent module 620. The difference (or absolute difference) between the product exponents may also be transmitted to the shifter 613 within the product and align module 610. In some embodiments, the maximum value detector 615 may include a subtractor for calculating the difference (or absolute difference).

[0099] The max exponent module 620 may receive multiple local max exponents from different product and align modules 610. The max exponent module 620 outputs a global max exponent (represented by "max ep" in Figure 6) which may be the largest max exponent received by the max exponent module 620. The max exponent module 620 provides the global max exponent to the subtractor 614 in each product and align module 610 that provided the max product exponent to the max exponent module 620. The subtractor 614 in the product and align module 610 may calculate the difference by subtracting the maximum product exponent calculated by the adder 612 from the global max exponent, or vice versa. This difference is transmitted to the shifter 618.

[0100] The shifter 613 may align the mantissas (represented as "mb0" and "mb1" in Figure 6) of two floating-point numbers (e.g., two weights or two activations) based on the maximum exponent determined by the maximum value detector 615. The aligned mantissas and the mantissas of the other two floating-point numbers (e.g., two activations or two weights) are transmitted to the multiplier 616 via the multiplexer 617. The multiplier 616 accumulates the mantissas (represented as "ma" in Figure 6) to calculate the product mantissa (represented as "mp" in Figure 6). The product mantissa may be a two-way dot product. The product mantissa may be transmitted to the shifter 618. Further details regarding the FP8 mode are described below in relation to Figures 8A and 8B.

[0101] In both FP16 and FP8 modes, the shifter 618 in the multiplication and alignment module 610 shifts one or more bits in the product mantissa based on the difference calculated by the subtractor 614. The shift of the product mantissa in the multiplication and alignment module 610 aligns the product mantissa calculated by these multiplication and alignment modules 610, allowing the aligned product mantissa to be output. This alignment facilitates the accumulation of product mantissas by the adder tree 630.

[0102] The adder tree 630 may receive one or more aligned product mantissas from the product and align modules 610. The adder tree 630 may include a plurality of adders (not shown in Figure 6) arranged in a hierarchy. The number of adders in the first hierarchy may be half the number of product and align modules 610 in the FPMAC unit 500. Each adder in the first hierarchy may be associated with two product and align modules 610 and accumulate the aligned product mantissas calculated by the two product and align modules 610. The number of adders in the second hierarchy may be half the number of adders in the first hierarchy. This may continue up to the last hierarchy which may have a single adder. Each adder may receive two numbers and output the sum of the two numbers. In some embodiments, the adder tree 630 may maintain intermediate sums in a carry-save format. In one example, each adder in the adder tree 630 may receive four numbers (for example, two carry-storage numbers) and output two numbers (for example, one carry-storage number).

[0103] The output of adder tree 630 ("partial sum mantissa") is W f The partial sum mantissa may have a bit width equal to +Log2N, where N is the number of aligned product mantissas accumulated by the adder tree 630. The partial sum mantissa is transmitted to the normalization module 640. The maximum exponent is also transmitted to the normalization module 640. The normalization module 640 may normalize the partial sum mantissa based on the maximum exponent, for example, by shifting one or more bits in the partial sum mantissa based on the maximum exponent. The result of the normalization may also be the result of a MAC operation.

[0104] The FPMAC unit 600 may allow mantissa multiplication skipping as described above. In FP16 mode, the operations in the multiplier 616 and shifter 618 may occur in a cycle after the cycle in which the operations in the adder 612, maximum exponent module 620, and subtractor 614 are performed. The operations in the multiplier 616 and shifter 618 may be skipped, for example, based on gate signals from the control module 360. In FP8 mode, the operations in the shifter 613, multiplier 616, and shifter 618 may occur in a cycle after the cycle in which the operations in the adder 611, maximum value detector 615, maximum exponent module 620, and subtractor 614 are performed. The operations in the shifter 613, multiplier 616, and shifter 618 may be skipped, for example, based on gate signals from the control module 360.

[0105] Figure 7 shows FP16 mantissa calculation in an FPMAC unit according to various embodiments. The embodiments of the FPMAC unit may be FPMAC unit 500 in Figure 5A or FPMAC unit 600 in Figure 6. In the embodiment of Figure 7, the mantissa multiplier (e.g., multiplier 616) may be reconfigured to support a single FP16 mantissa multiplication at a time. Dots in Figure 7 represent bits.

[0106] In the embodiment shown in Figure 7, the FPMAC unit operates in FP16 mode using one of the inputs (amant16) Booth encoded with + / -, 2x, and 1x signals. The negation of these unsigned inputs is performed within the multiplier by negating the + / - Booth signal when the product code (XOR of the codes of the two inputs) is 1. As shown in Figure 7, six XOR gates 710 (individually referred to as "XOR gate 710") are used to determine the product code. The use of these XOR gates can result in lower area consumption compared to negating either the mantissa input or the multiplier output, which both require far more XOR gates (e.g., 22 XOR gates) and add an extra 1. In some embodiments, the mantissa may be unsigned while the Booth-encoded partial product is signed. By inverting the sign Booth selection, neither the input mantissa (e.g., amant16 or bamant16) nor the output product mantissa needs to be negated separately. The extra 1 (since the negation of two's complement involves inverting the bits and adding 1) may already be included in the multiplier when the partial product is negative, providing further power savings.

[0107] Figures 8A and 8B show FP8 mantissa calculations in an FPMAC unit according to various embodiments. An embodiment of the FPMAC unit may be the FPMAC unit 600 in FIG. 6. In the embodiments of FIGS. 8A and 8B, a mantissa multiplier (e.g., multiplier 616) may be reconfigured to perform two FP8 mantissa multiplications with the sum of two products at once. In FIG. 8A, one of the sorted FP8 mantissas of the inputs is Booth-encoded with two 4b mantissas (i.e., mantissas each having 4 bits) aligned to the top (larger exponent) and bottom (smaller exponent) 4b of the 11b mantissa input. In this case, the negation is split into two, and the sign corresponding to the smaller exponent negates the + / - Booth signal of the lower Booth row, while the sign corresponding to the larger exponent negates the + / - Booth signal of the upper Booth row. In some embodiments, ma1 and mb1 may be the mantissas corresponding to the larger product exponent between each pair of FP8 product exponents. When ea1 + eb1 > ea0 + eb0, ma1, ma0, mb1, and mb0 may remain as they are. When ea1 + eb1 < ea0 + eb0, ma1 and ma0 may be swapped, and mb1 and mb0 may be swapped. The relative alignment of the two products may be achieved by shifting the other input mantissa in parallel with the Booth encoder. This can balance the Booth encoder delay of one of the multiplier inputs with the alignment delay for the other input.

[0108] As shown in FIG. 8A, the upper input (mb1) is used for the upper row, and the lower input (mb0) is used for the lower row. Several options can be used for alignment. The lower input may be aligned based on the least significant bit (LSB) of the shift8 exponent difference, while the upper input may be shifted by 7 when the exponent difference is greater than 7, for example, in an embodiment where the FP8 mantissa is 4b and aligns within an 11b multiplier. In other embodiments (e.g., embodiments where other floating-point formats are used), the shift amount may be different. Next, a single-stage shifter in the multiplier output re-aligns the dot product over the multiplier output for global alignment and preparation of the adder tree.

[0109] Figure 8B shows various options for aligning FP8 mantissas. Alignment may be performed by the shifter 613 in Figure 6. When shift8=0, the mantissas are vertically aligned, and as a result, the multiplier product contains the correct sum of the two FP8 mantissa products. When shift8=14, which may be the maximum separation achievable for two FP8 mantissa products in an 11b×11b multiplier, the mantissas are diagonally aligned. For shift8 values ​​between 0 and 14, there are multiple options for alignment, as shown in Figure 8B for shift8=7. When the upper mantissa is not aligned above the multiplier, either a final product alignment or exponent adjustment may be necessary to maintain consistency. For large differences between exponents that require truncation, one option is to shift out or truncate the lower mantissa, resulting in a loss of symmetry between the inputs. An alternative method is to saturate the shift of the lower mantissa so that bits are not truncated before multiplication. The multiplier output then requires an extra shift to insert an extra sign extension bit in the middle of the product.

[0110] Figures 9A and 9B show data paths in FPMAC units supporting variable floating-point precision according to various embodiments. An embodiment of the FPMAC unit may be the FPMAC unit 600 in Figure 6. In Figure 9A, two inputs (ain and bin) are received. The two inputs may be floating-point activation and floating-point weights. The two inputs are gated with an 8b mode selection signal to prevent switching on the unselected portion of the data path. Special number logic detects exponent and mantissa fields that are all 1 or all 0 and sets the appropriate inf / nan / zero / subnormal signals. 16b and 8b exponent extraction logic uses a single-stage shifter to align the exponents depending on the encoding (e.g., HF or BF encoding) and sets the exponent to 1 when subnormal using the OR of the LSB and the subnormal signal. The product exponents (exp16 and exp8) may be calculated by adding the input exponents. Using the XOR of two FP8 exponents together with maxexp8 results in a smaller exponent, which is then subtracted from the maximum exponent to find the FP8 mantissa shift amount.

[0111] The mantissa may be extracted from the input as shown in Figure 9B, and shifted by 3 for 16b mode coding and by 1 for 8b mode coding to correctly align the mantissa depending on the coding. A leading 1 may be added when the value is not zero or subnormal. Following local 8b maximum exponent logic, the 8b mantissa is sorted according to the larger and smaller exponents within each pair. In some embodiments, the shift8 signal is subtracted and compared to 7, which can be the largest possible shift for the FP8 mantissa in the 11b mantissa multiplier.

[0112] Figure 10 shows a maximum exponential module 1000 having an OR tree according to various embodiments. The maximum exponential module 1000 may also be an embodiment of the maximum exponential module 520 in Figure 5 or the maximum exponential module 620 in Figure 6. In some embodiments (for example, embodiments in which mantissa multiplication skipping may be used to save power), it may be timing-critical to determine whether the mantissa shift amount is greater than the fixed bit width of the adder tree in order to stop mantissa multiplication in a timely manner in the next clock cycle. This signal may be one of the conditions for determining whether the mantissa multiplier should be gated to reduce power.

[0113] In the embodiment shown in Figure 10, a two-stage peculative OR tree is used to reduce or minimize the time delay caused by the determination of whether the mantissa shift amount is greater than the fixed bit width of the adder tree. The time delay in the two-stage peculative OR tree can be smaller than the time delay in tree-based comparison and selection implementations, especially for wide dot products with many terms.

[0114] In some embodiments, the maximum exponent module 1000 may start from the most significant bit (MSB). The maximum exponent module 1000 may OR the bits across all product exponents to determine whether the maximum MSB bit is 1 or 0. The maximum exponent module 1000 may further combine the result of the OR operation with the individual product exponent MSB bits to determine whether each product exponent is still in conflict to determine the maximum exponent in FP16 operation mode or the global maximum exponent in FP8 operation mode. The resulting "smaller" signal may indicate that a particular product exponent is smaller than the maximum value, for example, because the input "smaller" signal is 1, or because the maximum exponent bit is 1 while the product exponent bit is 0.

[0115] The maximum exponent module 1000 may also use inference to pre-calculate maxexp, which is the maximum exponent in FP16 operating mode or the global maximum exponent in FP8 operating mode, without knowing the upper bit maxexp value. In the embodiment of Figure 10, there are two stages of inference before continuing with the normal MSB-LSB signal dependency. If a particular product exponent is known to be zero, the input "smaller" signal to the MSB OR tree can be set to prevent its product from affecting the maximum exponent.

[0116] For FP8 operation mode, a two-stage OR tree may be used to find the local maximum product exponent (maxexp8) between each pair of products. This local maximum follows the same MSB-LSB arrival profile as the global maxexp. Since the FP8 product exponent occupies only the lower 6b, the upper 3b of the global maximum detection logic may be removed from the critical path. Compared to conventional compare and select implementations, the two-stage inference OR tree can achieve lower critical path delays. The upper 3b and lower 6b may depend on the floating-point format of the input data. Different floating-point formats may have different boundaries.

[0117] [Example PE array] Figure 11 shows PE array 1100 in various embodiments. PE array 1100 may also be an embodiment of PE array 350 in Figure 3. PE array 1100 includes a plurality of PE1110s (each referred to as "PE1110"). PE1110s perform MAC operations. PE1110s may also be called neurons in a DNN. An embodiment of PE1100 may also be PE400 in Figure 4.

[0118] In the embodiment shown in Figure 11, each PE1110 has two input signals 1150 and 1160 and an output signal 1170. Input signal 1150 is at least a portion of the IFM to the layer. Input signal 1160 is at least a portion of the filter to the layer. In some embodiments, the input signal 1150 of PE1110 includes one or more activation operands, and the input signal 1160 includes one or more weight operands.

[0119] Each PE1110 performs a MAC operation on input signals 1150 and 1160 and outputs an output signal 1170, which is the result of the MAC operation. Some or all of the input signals 1150 and 1160 and the output signal 1170 may be in integer format such as INT8, or floating-point format such as FP16 or BF16. For the sake of simplification and illustration, all PE1110 input and output signals have the same reference sign, however, PE1110s may receive different input signals from each other and output different output signals from each other. Also, a PE1110 may be different from another PE1110, for example, by containing more components, fewer components, or different components.

[0120] As shown in Figure 11, the PE1110s are connected to each other as indicated by the dashed arrows in Figure 11. The output signal 1170 of PE1110 may be transmitted as an input signal to many other PE1110s via the interconnection between the PE1110s (and possibly returned to itself). In some embodiments, the output signal 1170 of PE1110 may incorporate the output signals of one or more other PE1110s through the accumulation operation of PE1110 to generate an internal partial sum of the PE array. Further details regarding the PE1110s are described below in reference to Figure 11B.

[0121] In the embodiment shown in Figure 11, the PE1110s are located in column 1105 (each referred to as “column 1105”). The inputs and weights of the layer may be distributed to the PE1110s based on column 1105. Each column 1105 has a column buffer 1120. The column buffer 1120 temporarily stores the data provided to the PE1110s in column 1105. The column buffer 1120 may also store the data output by the last PE1110 in column 1105. The output of the last PE1110 may also be the sum of the MAC operations of all PE1110s in column 1105, which is the column-level internal partial sum of the PE array 1100. In other embodiments, the inputs and weights may be distributed to the PE1110s based on rows in the PE array 1100. The PE array 1100 may include row buffers instead of column buffers 1120. The row buffer may store the input signals of the corresponding row's PE, and may also store the row-level internal partial sums of the PE array 1100.

[0122] As shown in Figure 11, each column buffer 1120 is associated with a load 1130 and a drain 1140. Data provided to column 1105 is transmitted to column buffer 1120 through the load 1130, for example, through a higher memory hierarchy, such as local memory 340 in Figure 3. Data generated by column 1105 is extracted from column buffer 1120 through drain 1140. In some embodiments, the data extracted from column buffer 1120 is sent to a higher memory hierarchy, such as local memory 340 in Figure 3, through a drain operation. In some embodiments, the drain operation is not initiated until all PEs 1110 in column 1105 have completed these MAC operations. Although not shown in Figure 11, one or more columns 1105 may be associated with an external adder assembly.

[0123] Figure 12 is a block diagram of PE1200 in various embodiments. PE1200 may also be the embodiment of PE1110 in Figure 11. Another embodiment of PE1200 may be PE400 in Figure 4. PE1200 includes an input register file 1210 (each referred to as "input register file 1210"), a weight register file 1220 (each referred to as "weight register file 1220"), a multiplier 1230 (each referred to as "multiplier 1230"), an internal adder assembly 1240, and an output register file 1250. In other embodiments, PE1200 may include fewer, more, or different components. For example, PE1200 may include multiple output register files 1250. As another example, PE1200 may include a single input register file 1210, a weight register file 1220, or a multiplier 1230. As yet another example, PE1200 may include an adder instead of the internal adder assembly 1240.

[0124] The input register file 1210 temporarily stores the activation operands for MAC operations performed by the PE 1200. In some embodiments, the input register file 1210 may store a single activation operand at a time. In other embodiments, the input register file 1210 may store multiple activation operands or a portion of activation operands at a time. An activation operand contains multiple input elements (i.e., input elements) in an input tensor. The input elements of an activation operand may be stored sequentially in the input register file 1210 so that the input elements can be processed sequentially. In some embodiments, each input element in an activation operand may come from a different input channel of the input tensor. An activation operand may contain input elements from each of the input channels of the input tensor, and the number of input elements in an activation operand may be equal to the number of input channels. The input elements in an activation operand may have the same (X,Y) coordinates, which may be used as the (X,Y) coordinates of the activation operand. For example, all input elements of an activation operand may be X0Y0, X0Y1, X1Y1, etc. An embodiment of the input register file 1210 may also be the input storage unit 420 in Figure 4.

[0125] The weight register file 1220 temporarily stores weight operands for MAC operations performed by PE1200. The weight operands include the weights in the DNN layer's filter. In some embodiments, the weight register file 1220 may store a single weight operand at a time. In other embodiments, the input register file 1210 may store multiple weight operands or a portion of weight operands at a time. A weight operand may contain multiple weights. Since the weights of a weight operand may be stored sequentially in the weight register file 1220, the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and activation operands, each weight in the weight operand may correspond to an input element of the activation operand. The number of weights in a weight operand may be equal to the number of input elements in the activation operand.

[0126] In some embodiments, the weight register file 1220 may be the same as or similar to the input register file 1210, and may have the same size, for example. The PE 1200 may include multiple register files, some of which are designated as the input register file 1210 for storing activation operands, some as the weight register file 1220 for storing weight operands, and some as the output register file 1250 for storing output operands. In other embodiments, the register files within the PE 1200 may be designated for other purposes, for example, for storing scale operands used in element-wise addition operations. An embodiment of the weight register file 1220 may be the weight storage unit 430 in Figure 4.

[0127] The multiplier 1230 performs multiplication operations on the activation operand and the weight operand. The multiplier 1230 may perform a sequence of multiplication operations on a single activation operand and a single weight operand to produce a product operand containing a sequence of products. Each multiplication operation in the sequence involves multiplying the input element in the activation operand by the weight in the weight operand. In some embodiments, the position (or index) of the input element in the activation operand coincides with the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the activation operand by the first weight in the weight operand, the second multiplication operation is the multiplication of the second input element in the activation operand by the second weight in the weight operand, the third multiplication operation is the multiplication of the third input element in the activation operand by the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation may correspond to the same depth channel, and their product may also correspond to the same depth channel.

[0128] Multiple multipliers 1230 may perform multiplication operations simultaneously. These multiplication operations may be referred to as a round of multiplication operations. In a round of multiplication operations by the multipliers 1230, each multiplier 1230 may use different activation operands and different weight operands. Different activation operands or weight operands may be stored in different register files of PE1200. For example, the first multiplier 1230 uses a first activation operand (e.g., stored in the first input register file 1210) and a first weight operand (e.g., stored in the first weight register file 1220), the second multiplier 1230 uses a second activation operand (e.g., stored in the second input register file 1210) and a second weight operand (e.g., stored in the second weight register file 1220), the third multiplier 1230 uses a third activation operand (e.g., stored in the third input register file 1210) and a third weight operand (e.g., stored in the third weight register file 1220), and so on. For each multiplier 1230, the round of multiplication operations may include multiple cycles. Each cycle includes multiplication operations on the input elements and weights.

[0129] The multiplier 1230 may perform multiplication operations over multiple rounds. The multiplier 1230 may use the same weight operand but different activation operands in different rounds. For example, the multiplier 1230 performs a sequence of multiplication operations on a first activation operand stored in a first input register file in the first round, and on a second activation operand stored in a second input register file in the second round. In the second round, a different multiplier 1230 may perform another sequence of multiplication operations using the first activation operand and different weight operands. In this way, the first activation operand is reused in the second round. The first activation operand may be reused further in subsequent rounds, for example, by a further multiplier 1230. An embodiment of the multiplier 1230 1210 may be the multiplier 450 in Figure 4.

[0130] The internal adder assembly 1240 includes one or more adders, i.e., internal adders, within the PE 1200. The internal adder assembly 1240 may perform an accumulation operation on two or more product operands from the multiplier 1230 to produce an output operand of the PE 1200. In some embodiments, the internal adders are arranged in a series of hierarchies. Each hierarchy includes one or more internal adders. For the first hierarchy of the internal adder assembly 1240, the internal adder may receive product operands from two or more multipliers 1230 and produce a sum operand through a sequence of accumulation operations. Each accumulation operation produces a sum of two or more products, each product from a different multiplier 1230. The sum operand includes a sequence of sums, each sum being the result of an accumulation operation and corresponding to a depth channel. For other hierarchies of the internal adder assembly 1240, an internal adder in one hierarchy receives a sum operand from a preceding hierarchy in the sequence. Each of these numbers may be generated by different internal adders in the preceding hierarchy. The ratio of the number of internal adders in one hierarchy to the number of internal adders in a subsequent hierarchy may be 2:1. In some embodiments, the final hierarchy of the internal adder assembly 1240 may include a single internal adder that generates the output operand of PE1200. Embodiments of the internal adder assembly 1240 may include an adder tree 440 or accumulator 480 in Figure 4.

[0131] The output register file 1250 stores the output operands of the PE 1200. In some embodiments, the output register file 1250 may store an output operand at a time. In other embodiments, the output register file 1250 may store multiple output operands or a portion of output operands at a time. An output operand includes multiple output elements in the IFM. The output elements of an output operand may be stored sequentially in the output register file 1250 so that the output elements can be processed sequentially. In some embodiments, each output element in an output operand corresponds to a different depth channel and is an element of a different output channel of the depth convolution's output channels. The number of output elements in an output operand may be equal to the number of depth channels in the depth convolution. Embodiments of the output register file 1250 may include the output storage unit 490 shown in Figure 4.

[0132] [Example of how to perform FPMAC calculations] Figure 13 is a flowchart of method 1300 for performing an FMAC operation according to various embodiments. Method 1300 may be performed by the calculation block 330 in Figure 3. Although method 1300 is described with reference to the flowchart shown in Figure 13, many other methods for performing an FPMAC operation may be used as alternatives. For example, the execution order of the steps in Figure 13 may be changed. As another example, some of the steps may be modified, deleted, or combined.

[0133] The computation block 330 selects an operating mode from a plurality of operating modes of the circuit based on the precision of at least one of the floating-point data elements 1310. The circuit may be a circuit of an FPMAC unit, such as the FPMAC unit 600 in Figure 6. In some embodiments, the operating mode corresponds to a first precision, such as FP8 precision. The plurality of operating modes further include another operating mode corresponding to a second precision, such as FP16 precision. The second precision is higher than the first precision. In some embodiments, the floating-point data elements may include activations and weights for deep learning operations, such as convolution.

[0134] The calculation block 330 calculates the product exponent based on the exponents of the floating-point data elements. The calculation block 330 may also calculate the product exponent by accumulating the exponents of two or more floating-point data elements.

[0135] The calculation block 330 selects one or more maximum exponents. The maximum exponents are derived from one or more product exponents. In some embodiments, the calculation block 330 may select the larger product exponent from two product exponents as the maximum exponent.

[0136] The calculation block 330 selects a global maximum exponent from one or more maximum exponents.1340 In some embodiments, the calculation block 330 performs one or more OR operations on the bits in one or more maximum exponents.The calculation block 330 selects a global maximum exponent based on the results of one or more OR operations.In some embodiments, the calculation block 330 selects a global maximum exponent based on at least one MSB of one or more maximum exponents.

[0137] The calculation block 330 calculates the result of a sum-of-products operation based on the product exponent, one or more maximum exponents, and a global maximum exponent.1350 In some embodiments, the calculation block 330 calculates the product mantissa based on the mantissa of a floating-point data element, the maximum exponent, and a global maximum exponent.In embodiments, the calculation block 330 aligns one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent.

[0138] In some embodiments, one or more product exponents are calculated in a first cycle. The product mantissa is calculated in a second cycle. In the second cycle, the calculation block 330 shifts one or more bits in the product mantissa based on the global maximum exponent. Before the second cycle, the calculation block 330 determines whether shifting one or more bits would cause the product mantissa to exceed the bit width limit. In some embodiments, depending on the determination that shifting one or more bits would cause the product mantissa to exceed the bit width limit, the calculation block 330 may skip calculating the product mantissa.

[0139] [Example computing device] Figure 14 is a block diagram of exemplary computing device 1400 according to various embodiments. In some embodiments, computing device 1400 can be used as at least part of the DNN accelerator 300 in Figure 3. Although numerous components are shown as being included in computing device 1400 in Figure 14, one or more of these components may be omitted or duplicated depending on the application. In some embodiments, some or all of the components included in computing device 1400 may be mounted on one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-chip (SoC) die. Furthermore, in various embodiments, computing device 1400 may not include one or more of the components shown in Figure 14, but computing device 1400 may include interface circuits for coupling with one or more components. For example, computing device 1400 may not include display device 1406, but may include display device interface circuits (e.g., connectors and driver circuits) to which display device 1406 can be coupled. In another example set, the computing device 1400 may not include the audio input device 1418 or the audio output device 1408, but may include an audio input or output device interface circuit (e.g., a connector and support circuit) to which the audio input device 1418 or the audio output device 1408 can be coupled.

[0140] The computing device 1400 may include a processing device 1402 (e.g., one or more processing devices). The processing device 1402 processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1400 may also include a memory 1404, which itself may include one or more memory devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard drive. In some embodiments, the memory 1404 may include memory that shares a die with the processing device 1402. In some embodiments, the memory 1404 includes one or more non-temporary computer-readable media that store executable instructions for performing operations, e.g., the method 1300 described above in relation to Figure 13, or some operations performed by the computing block 330 described above in relation to Figure 3. Instructions stored in one or more non-temporary computer-readable media may be executed by the processing device 1402.

[0141] In some embodiments, the computing device 1400 may include a communication chip 1412 (e.g., one or more communication chips). For example, the communication chip 1412 may be configured to manage wireless communication for transferring data to and from the computing device 1400. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can communicate data using modulated electromagnetic radiation through a non-solid medium. This term does not mean that the devices in question do not include any wires, although in some embodiments they may be wire-free.

[0142] The communication chip 1412 may implement any of a number of wireless standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.10 family), Institute of Electrical and Electronics Engineers (IEEE) standards including the IEEE 802.16 standard (e.g., IEEE 802.16-2005 revised edition), and Long-Term Evolution (LTE) projects with any modification, update, and / or revision (e.g., the Advanced LTE project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2")). IEEE 802.16-compliant broadband wireless access (BWA) networks are commonly referred to as WiMAX networks, an acronym for Worldwide Interoperability for Microwave Access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standard. The communication chip 1412 may operate in accordance with Global System for Mobile Communications (GSM), General-Purpose Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. The communication chip 1412 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1412 may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimization (EV-DO) and their derivatives, as well as any other wireless protocol designated as 3G, 4G, 5G, and later. In other embodiments, the communication chip 1412 may operate in accordance with other wireless protocols. The computing device 1400 may include an antenna 1422 for facilitating wireless communication and / or receiving other wireless communications (such as AM or FM radio transmissions).

[0143] In some embodiments, the communication chip 1412 may manage wired communication such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1412 may include multiple communication chips. For example, the first communication chip 1412 may be dedicated to short-range wireless communication such as Wi-Fi or Bluetooth, and the second communication chip 1412 may be dedicated to long-range wireless communication such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1412 may be dedicated to wireless communication, and the second communication chip 1412 may be dedicated to wired communication.

[0144] The computing device 1400 may include a battery / power supply circuit 1414. The battery / power supply circuit 1414 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 1400 to an energy source separate from the computing device 1400 (e.g., AC line power).

[0145] The computing device 1400 may include a display device 1406 (or a corresponding interface circuit as described above). The display device 1406 may include any of the following visual indicators, such as a head-up display, computer monitor, projector, touchscreen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0146] The computing device 1400 may include an audio output device 1408 (or a corresponding interface circuit as described above). The audio output device 1408 may include any device that generates an audible indicator, such as a speaker, headset, or earphones.

[0147] The computing device 1400 may include an audio input device 1418 (or a corresponding interface circuit as described above). The audio input device 1418 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (for example, an instrument with a MIDI (musical instrument digital interface) output).

[0148] The computing device 1400 may include a GPS device 1416 (or a corresponding interface circuit as described above). The GPS device 1416 may communicate with a satellite-based system and receive the position of the computing device 1400, as is known in the art.

[0149] The computing device 1400 may include another output device 1410 (or a corresponding interface circuit as described above). Examples of other output devices 1410 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or further storage devices.

[0150] The computing device 1400 may include another input device 1420 (or a corresponding interface circuit as described above). Examples of other input devices 1420 may include an accelerometer, gyroscope, compass, image capture device, keyboard, cursor control device such as a mouse, stylus, touchpad, barcode reader, quick response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0151] The computing device 1400 may have any desired shape factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, PDA (Personal Digital Assistant), ultramobile personal computer, etc.), a desktop computer system, a server or other network-connected computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1400 may also be any other electronic device that processes data.

[0152] [Example Selection] The following paragraphs provide various examples of embodiments disclosed herein.

[0153] Example 1 provides a device for multiply-accumulate operations on floating-point data elements, the device comprising: a control module configured to select an operating mode from a plurality of operating modes of the device based on the precision of at least one of the floating-point data elements; one or more product and align modules configured to operate in an operating mode by calculating one or more product exponents based on the exponents of the floating-point data elements and selecting the largest exponent from one or more product exponents; and a maximum exponent module configured to select a global maximum exponent from one or more maximum exponents calculated by one or more product and align modules.

[0154] Example 2 provides the apparatus of Example 1, wherein the selected operating mode corresponds to a first precision, and the multiple operating modes further include another operating mode corresponding to a second precision, the second precision being higher than the first precision, and the bit width of a floating-point data element having the second precision is equal to the total bit width of two or more floating-point data elements having the first precision.

[0155] Example 3 provides the apparatus of Example 1 or 2, wherein the product and alignment module is further configured to operate in operating mode by calculating the product mantissa based on the mantissa, maximum exponent, and global maximum exponent of the floating-point data elements.

[0156] Example 4 provides the apparatus of Example 3, wherein calculating the mantissa involves aligning one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element, based on the largest exponent.

[0157] Example 5 provides the apparatus of Example 4, wherein aligning one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent includes determining the difference between the maximum exponent and the exponent of one or more floating-point data elements, and aligning one or more bits in the mantissa of one floating-point data element with one or more bits in the mantissa of another floating-point data element based on the difference.

[0158] Example 6 provides one of the devices from Examples 3-5, where one or more product exponents are calculated before the product mantissa is calculated.

[0159] Example 7 provides one of the devices from Examples 1 to 6, wherein the control module is further configured to generate a gate signal based on a global maximum exponent and a product exponent calculated by another product and alignment module, and to transmit the gate signal to another product and alignment module, the gate signal preventing any product mantissa from being calculated after the product and alignment module.

[0160] Example 8 provides one of the devices from Examples 1 to 7, wherein the maximum exponent module includes one or more groups of OR operators configured to perform one or more OR operations on bits in one or more maximum exponents, and the maximum exponent module is configured to select a global maximum exponent based on the results of one or more OR operations.

[0161] Example 9 provides a device of any one of Examples 1 to 8, further comprising one or more adders configured to accumulate one or more product mantissas from one or more product and align modules, the product mantissas being calculated by the product and align modules based on the mantissas of floating-point data elements, and the one or more product mantissas being aligned based on the global maximum exponent.

[0162] Example 10 provides the apparatus of Example 9 and further includes a normalization module configured to compute the result of a sum-of-accumulate operation on a floating-point data element by normalizing the output of one or more adders based on the global maximum exponent.

[0163] Example 11 provides a method for a sum-of-products operation on a floating-point data element, the method comprising: selecting an operating mode from a plurality of operating modes of a circuit based on the precision of at least one of the floating-point data elements; calculating a product exponent based on the exponents of the floating-point data elements by the circuit of the operating mode; selecting one or more maximum exponents by the circuit of the operating mode, the maximum exponent being selected from one or more of the product exponents; selecting a global maximum exponent from one or more maximum exponents by the circuit of the operating mode; and calculating the result of a sum-of-products operation by the circuit of the operating mode based on the product exponents, one or more maximum exponents, and the global maximum exponent.

[0164] Example 12 provides the method of Example 11, where the operating modes correspond to a first precision, and the multiple operating modes further include other operating modes corresponding to a second precision, where the second precision is higher than the first precision.

[0165] Example 13 provides the method of Example 11 or 12, wherein the step of calculating the result of a sum-of-products operation includes the step of calculating the product mantissa based on the mantissa, maximum exponent, and global maximum exponent of the floating-point data element.

[0166] Example 14 provides the method of Example 13, wherein the step of calculating the product mantissa includes aligning one or more bits in the mantissa of one floating-point data element with one or more bits in the mantissa of another floating-point data element, based on the largest exponent.

[0167] Example 15 provides the method of Example 13 or 14, wherein one or more product exponents are calculated in a first cycle, and a product mantissa is calculated in a second cycle, the method further comprising the steps of shifting one or more bits in the product mantissa based on the global maximum exponent in the second cycle, and determining before the second cycle whether shifting one or more bits would cause the product mantissa to exceed a bit width limit.

[0168] Example 16 provides one of the methods from Examples 11 to 15, wherein the step of selecting the global maximum exponent includes the step of performing one or more OR operations on bits in one or more maximum exponents, and the step of selecting the global maximum exponent based on the results of one or more OR operations.

[0169] Example 17 provides one of the methods from Examples 11 to 16, wherein the step of selecting the global largest index includes the step of selecting the global largest index based on the MSB of at least one of the one or more largest indices.

[0170] Example 18 provides one or more non-temporary computer-readable media for storing executable instructions to perform an operation, the operation comprising: selecting an operating mode from several operating modes of a circuit based on the precision of a floating-point data element; calculating one or more product exponents based on the exponents of a floating-point data element; selecting the largest exponent from one or more product exponents; and selecting a global maximum exponent from one or more maximum exponents calculated by one or more product and alignment modules.

[0171] Example 19 provides one or more non-temporary computer-readable media of Example 18, wherein the operating modes correspond to a first precision, and the multiple operating modes further include other operating modes corresponding to a second precision, where the second precision is higher than the first precision.

[0172] Example 20 provides one or more non-temporary computer-readable media of Example 18 or 19, the operation further comprising calculating a product mantissa based on the mantissa, maximum exponent, and global maximum exponent of a floating-point data element, the calculation of the product mantissa comprising aligning one or more bits in the mantissa of one floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent.

[0173] The above description of exemplary implementations of the Disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the Disclosure to the exact form disclosed. Specific implementations and examples of the Disclosure are described herein for illustrative purposes only, but various equivalent modifications are possible within the scope of the Disclosure, as will be apparent to those skilled in the art. These modifications may be made to the Disclosure in light of the above detailed description.

Claims

1. A device for multiply-accumulate operations on floating-point data elements, A control module configured to select an operating mode from a plurality of operating modes of the device based on the precision of at least one of the floating-point data elements, One or more product and alignment modules configured to operate in the operating mode by calculating one or more product exponents based on the exponents of the floating-point data elements and selecting the largest exponent from the one or more product exponents, A maximum exponent module configured to select a global maximum exponent from one or more maximum exponents calculated by the one or more product and alignment modules, and A device that includes this.

2. The apparatus according to claim 1, wherein the selected operating mode corresponds to a first precision, and the plurality of operating modes further include another operating mode corresponding to a second precision, the second precision being higher than the first precision, and the bit width of a floating-point data element having the second precision is equal to the total bit width of two or more floating-point data elements having the first precision.

3. The apparatus according to claim 1 or 2, wherein the product and alignment module is further configured to operate in the operating mode by calculating a product mantissa based on the mantissa of the floating-point data elements, the maximum exponent, and the global maximum exponent.

4. Calculating the aforementioned multiplicative mantissa is, The apparatus according to claim 3, comprising aligning one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent.

5. Aligning one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent is: Determining the difference between the maximum exponent and the exponent of the floating-point data element or the exponent of another floating-point data element, Based on the difference, align one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of the other floating-point data element. The apparatus according to claim 4, including the apparatus described in claim 4.

6. The apparatus according to claim 3, wherein the one or more product exponents are calculated before the product mantissa is calculated.

7. The control module is Based on the global maximum index and the product index calculated by another product and alignment module, a gate signal is generated. The gate signal is further configured to transmit the aforementioned gate signal to the other multiplication and alignment module, The apparatus according to claim 1, wherein the gate signal prevents the calculation of either product mantissa after the product and alignment modules.

8. The apparatus according to claim 1, wherein the maximum exponent module includes one or more groups of OR operators configured to perform one or more OR operations on bits in one or more maximum exponents, and the maximum exponent module is configured to select the global maximum exponent based on the results of the one or more OR operations.

9. The system further includes one or more adders configured to accumulate one or more product mantissas from the one or more product and sorting modules, The apparatus according to claim 1, wherein the product mantissa is calculated by the product and alignment module based on the mantissas of the floating-point data elements, and the one or more product mantissas are aligned based on the global maximum exponent.

10. The apparatus according to claim 9, further comprising a normalization module configured to calculate the result of the sum-of-accumulate operation on the floating-point data elements by normalizing the output of one or more adders based on the global maximum exponent.

11. A method for multiply-accumulate operations on floating-point data elements, The steps include selecting an operating mode from a plurality of operating modes of the circuit based on the precision of at least one of the floating-point data elements, The circuit in the aforementioned operating mode performs the steps of calculating the product exponent based on the exponents of the floating-point data elements, The circuit in the aforementioned operating mode selects one or more maximum exponents, wherein the maximum exponents are selected from one or more of the product exponents. The circuit in the aforementioned operating mode selects the global maximum index from the one or more maximum indices, The circuit in the operating mode calculates the result of the sum-of-products operation based on the product exponent, the one or more maximum exponents, and the global maximum exponent. A method that includes this.

12. The method according to claim 11, wherein the operating mode corresponds to a first accuracy, and the plurality of operating modes further include another operating mode corresponding to a second accuracy, the second accuracy being higher than the first accuracy.

13. The step of calculating the result of the sum-of-products operation is: The method according to claim 11 or 12, comprising the step of calculating a product mantissa based on the mantissa of the floating-point data element, the maximum exponent, and the global maximum exponent.

14. The step of calculating the multiplicative mantissa is: The method according to claim 13, further comprising the step of aligning one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent.

15. The step of aligning one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent is: The steps include determining the difference between the maximum exponent and the exponent of the floating-point data element or the exponent of another floating-point data element, The steps of aligning one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of the other floating-point data element based on the difference: The method according to claim 14, including the method described in claim 14.

16. The one or more product exponents are calculated in the first cycle, the product mantissa is calculated in the second cycle, and the method is In the second cycle, the steps include shifting one or more bits in the mantissa based on the global maximum exponent, The steps include determining whether shifting one or more bits before the second cycle would cause the mantissa to exceed the bit width limit, and The method according to claim 13, further comprising:

17. The step of selecting the global largest index is, The steps of performing one or more OR operations on the bits in the one or more largest exponents, The step of selecting the global maximum exponent based on the result of one or more OR operations. The method according to claim 11, including the method described in claim 11.

18. The step of selecting the global largest index is, The method according to claim 11, further comprising the step of selecting the global maximum exponent based on the most significant bit of at least one of the one or more maximum exponents.

19. A computer program that causes a computer to perform an action, The aforementioned operation is, For multiply-accumulate operations, the operating mode is selected from multiple operating modes of the circuit based on the precision of the floating-point data elements, Calculating one or more product exponents based on the exponents of the floating-point data elements, Selecting the largest index from the aforementioned one or more product indices, Selecting the global maximum exponent from one or more maximum exponents calculated by one or more product and alignment modules. A computer program that includes [this].

20. The computer program according to claim 19, wherein the operating mode corresponds to a first precision, and the plurality of operating modes further include another operating mode corresponding to a second precision, the second precision being higher than the first precision.

21. The aforementioned operation is, The computer program according to claim 19, further comprising calculating a product mantissa based on the mantissa of the floating-point data element, the maximum exponent, and the global maximum exponent.

22. Calculating the aforementioned multiplicative mantissa is, The computer program according to claim 21, comprising aligning one or more bits in the mantissa of a floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent.

23. The step of aligning one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of another floating-point data element based on the maximum exponent is: The steps include determining the difference between the maximum exponent and the exponent of the floating-point data element or the exponent of another floating-point data element, The steps of aligning one or more bits in the mantissa of the floating-point data element with one or more bits in the mantissa of the other floating-point data element based on the difference: The computer program according to claim 22, including the computer program described in claim 22.

24. The one or more product exponents are calculated in the first cycle, the product mantissa is calculated in the second cycle, and the operation is In the second cycle, one or more bits in the multiplicand are shifted based on the global maximum exponent, Before the second cycle, determine whether shifting one or more bits would cause the mantissa to exceed the bit width limit. The computer program according to claim 21, further comprising:

25. Selecting the aforementioned global largest index means Performing one or more OR operations on the bits in the one or more largest exponents, Selecting the global maximum exponent based on the result of one or more OR operations. The computer program according to claim 19, including the computer program described in claim 19.

26. One or more non-temporary computer-readable storage media storing a computer program according to any one of claims 19 to 25.

Images