Optimized multiplication and accumulation operator for calculations involved in AI
The HMAC operator addresses resource inefficiencies in deep learning by performing efficient tensor conversion between quantization domains, reducing operations from three or four to one, thereby enhancing computational efficiency and resource utilization.
Patent Information
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-07-25
- Publication Date
- 2026-04-03
AI Technical Summary
Existing deep learning techniques using 8-bit integer quantization suffer from resource inefficiency and high performance and power consumption due to the reliance on 32-bit floating-point arithmetic for tensor element representation changes between quantization domains.
A hybrid multiplication and accumulation (HMAC) operator that performs efficient tensor conversion between quantization domains by multiplying a floating-point number by an integer and adding an integer operand, reducing operations to one from three or four, using a structure that includes a multiplier, shifter, adder, and saturation circuit to handle exponent and saturation conditions.
This approach significantly reduces the number of operations required, achieving a computational efficiency gain of the order of 3n operations, while maintaining accurate results by handling exponent and saturation conditions, thus optimizing hardware resource usage.
Smart Images
Figure 00000010_0000
Abstract
Description
Title of the invention: Optimized multiplication and accumulation operator for calculations involved in AI technical field
[0001] The invention relates to deep learning techniques in the field of artificial intelligence, and more particularly to a hardware multiplication and accumulation operator adapted to perform some of the calculations involved. Background
[0002] In deep learning techniques, for some of the layers of deep neural networks, arithmetic involving small number of bits (for example 8-bit integers, called INT8) is used instead of 32-bit floating-point arithmetic (numbers in FP32 format according to the IEEE 754 standard), which is more expensive in terms of hardware resources.
[0003] Classical deep learning techniques rely on a simple integer quantization transformation between FP32 floating-point numbers and INT8 numbers. A real number X represented in FP32 format is mapped to an INT8 number x within a given quantization domain defined by a 32-bit integer zero point z (INT32) and an FP32 floating-point scaling factor S. The quantized number X, denoted ~X, is then expressed as:
[0004] [Math.l] (xz)S
[0005] The number x in INT8 format is the one that is stored and used for arithmetic operations. The quantization domain corresponds to a low-resolution range of variation of the number ~X around a central value z located in a high-dynamic range (that of the factor S, an FP32 number). Since the values of x vary in the range [-128, 127], the range of variation of ~X becomes:
[0006] [Math.2] e[(-128-z)S, (127-z)S]
[0007] This range of variation, which in practice is supposed to cover a Gaussian distribution of the values of X around the zero point z, does not cover the entire range of variation of the number X. Thus, for values of X outside the range, ~X is saturated at the boundaries of the range.
[0008] There may be a quantization domain per activation tensor (a multidimensional matrix of activation values), and a tensor channel (one of its dimensions) may sometimes have a different quantization domain. Once FP32 numbers are quantized, they can be exploited using resource-inefficient operations on small integers x.
[0009] Despite the advantages of this quantization, FP32 arithmetic remains used in calculations that modify the representation of tensor elements between quantization domains. FP32 arithmetic is more expensive to implement in terms of performance, power consumption, and silicon area. Summary
[0010] Generally, a hybrid multiplier-accumulation hardware operator is provided, configured to multiply a floating-point formatted multiplicand by an integer formatted multiplicand and add an integer formatted result of the multiplication to an integer formatted operand.
[0011] The operator may include a first multiplicand input for the floating-point multiplicand having a mantissa field and an exponent field; a second multiplicand input for the integer multiplicand; an accumulator input for the integer operand; an integer multiplier configured to multiply the mantissa field of the floating-point multiplicand by the integer multiplicand and produce a result in a fixed-point format corresponding to a zero exponent of the floating-point multiplicand; a connected shifter to receive the result of the multiplier and shift it left or right depending on the value of the exponent field of the floating-point multiplicand; and an integer adder receiving the integer operand and a window on the result of the shifter, the window capturing a number of bits to the left of the fixed decimal point position at least equal to the number of bits in the integer operand.
[0012] The left and right shift amplitude of the shifter can be limited to the number of bits of the integer operand and the operator can include a saturation circuit connected to the adder and the shifter and configured to produce the result of the adder when no bit to the left of the window is significant and the exponent field codes a value less than the number of bits of the integer operand, and to produce a saturated result when at least one bit to the left of the window is significant or the exponent field codes a value greater than or equal to the number of bits of the integer operand.
[0013] The saturation circuit can be configured to round the adder result based on significant values of the bits located to the right of the window. Summary description of the drawings
[0014] Embodiments will be described below, by way of non-limiting example, in relation to the accompanying figures, among which:
[0015] Fig. 1 represents a block diagram of an embodiment of a hybrid multiplication and addition operator. Detailed description
[0016] A new type of hybrid hardware fused multiplication and addition (HFMA) operator is proposed below, providing efficient tensor conversion between different quantization domains. The structure of this HFMA operator arises from a specific analysis and decomposition of the operations involved in the conversion between quantization domains. The resulting structure multiplies a signed integer (e.g., INT32) by a floating-point number (e.g., FP32) and adds a signed integer operand (INT32) to the product to provide an integer result in the same format as the operand. Because the result is in the same format as the addition operand, the structure can be used to compute the accumulation of products, and can therefore also be referred to as the "hybrid multiplication and accumulation" or HMAC ("Hybrid Multiply-ACcumulate") operator.
[0017] The most common calculation in deep learning networks is the accumulated dot product of an activation vector X t by a weight vector IV, with the addition of a bias B j according to the following relationship (since we now work exclusively on quantified values, we omit the prefix to clarify the writing):
[0018] [Math.3] YOU
[0019] Each term is quantified in its own quantization domain, respectively (zy, Sy), (zx, Sx), (0, Sw) and (0, SXSW). The zero point of the weights IV, and of the biases B j is 0. The quantization domain (0, SXSW) of B 7 has the same scale factor as the quantization domain resulting from the product X t IV ;.
[0020] The different terms are expressed as follows:
[0021] [Math.4] xi = (xr^Sx [Math.4] [Math.4] [Math.4]
[0022] Let us recall that the S values are scaling factors in FP32, the z values are the zero points in INT32, and the xt and yj values are INT8 numbers which are used for storage and recurring operations.
[0023] By making substitutions in these relations, we obtain:
[0024] [Math.5] - zA^E^-+sa^
[0025] The integer representing the quantified result is then expressed as:
[0026] [Math.6] y. = int ( zy+— ( + Cj J ) with [Math.6] Cj^bj-z^Wi
[0027] Each term cj is an INT32 integer constant computable offline using simple integer arithmetic, such as a classical multiplication-accumulation (MAC) operator operating on INT32 integers. The sum V”w is a constant integer for the set of j that can be calculated once and stored to calculate all the terms c j.
[0028] The basic recurrent calculation on index j, classically involving operations in FP32, becomes an integer product operation with addition that multiplies the integer V” rw . z. in INT32 by a real number SA in FP32, and adds an integer zy in H# >Wi sy INT32, the result being again an INT32 number. This is exactly what the HFMA operator calculates. The yj values are then saturated to fit once more into an INT8 integer and are stored and reused to represent the quantized values of the Y j-
[0029] With conventional means, the operation would consist of converting the integer multiplicand and the integer addition operand into FP32 numbers, performing a fused multiplication and addition (FMA) of the resulting FP32 numbers, and converting the FP32 result into an INT32 integer, i.e. three or four operations instead of one, one of which involves a complex FMA operator of FP32 numbers.
[0030] This gain in the number of operations occurs, in the example above, once every n + 1 iterations on i. In some cases, n can be of the same order as the number of iterations on j, so that we obtain a gain of the order of 3n operations.
[0031] The HFMA operator can also be used for one-off operations such as the sum of quantized vectors:
[0032] [Math. 8] Y^Xj + Wt [Math. 8] xi = (xrzx)Sx [Math. 8] wi = (wi - z^S” [Math. 8] Yj = (yfZy)Sy [Math. 8] y. = int\ (Wi-^^+Zy [Math. 8] ls / Ç \ \ y. ^int[ (xrzx)fy+int[ (wrzw) / y+zy) J
[0033] This corresponds to a nested application of the HFMA or HMAC operator (with two additions of INT32 integers).
[0034] This calculation can be further optimized when the quantization domain of one of the inputs is the same as that of the output, which often occurs in practice. For example, if zx = Sy and zx = zy, then:
[0035] [Math.9]
[0036] This corresponds to a single application of the HMAC operator (with an addition of INT32 integers).
[0037] The HMAC operator can also be used in combination with an operator designated by FSCALE to quantify FP32 values into INT8 values without involving FP32 arithmetic. An FSCALE operation scales an FP32 value by an integer power of two, in the form of a simple integer addition over the 8-bit exponent field of the FP32 numeric representation.
[0038] For example, in the case of X; = (x; - zx)Sx with d. - we have:
[0039] [Math. 10] (X. \ / \ 7Ç + zx J = in^FSCALE(e)m + z J
[0040] Here, the FSCALE operation produces a result in FP32, while e is INT8, and m and z are INT32. Thus, the calculation of xt is again an HMAC calculation which produces an INT32 value which will be transformed into an INT8 value.
[0041] Figure 1 represents a block diagram of an embodiment of a hybrid multiplication and addition (or accumulation) operator. The operator is configured to perform the operation OpO x Opl + Op2, where the multiplicand OpO is a floating-point number, for example FP32, the multiplicand Opl is an integer, for example INT32, and the operand Op2 is an integer of the same format (INT32) as the operand Opl.
[0042] The floating-point multiplicand OpO includes an 8-bit exponent field EXP and a 24-bit mantissa field MANT. The mantissa field MANT of The operator OpO, treated as an integer, and the integer multiplicand Opl are fed into an integer multiplier 10. The multiplier 10 thus produces a 57-bit product P. The product P is in fact a fixed-point number whose position is conventionally chosen for an exponent EXP equal to 0. In this case, the product corresponds to the multiplication of a 24-bit number l,xx...xx (after the decimal point) by a 32-bit integer XX...XX (before the decimal point), so that the fixed point is positioned between bits 23 and 24 of the 57-bit product P.
[0043] A left-shift circuit 12 receives the product P at positions 31 and 87 of a 119-bit virtual number having 31 bits set to 0 to the right of the product P and 31 bits set to 0 to the left of the product P. The shift command of circuit 12 is the exponent EXP of the multiplicand OpO. When this exponent is positive, circuit 12 performs a corresponding left shift. When the exponent is negative, circuit 12 performs a corresponding right shift.
[0044] The "useful" output of shifter 12 is taken from a window covering positions 55 to 87 of the shifted number, namely the 33 bits before the fixed decimal point, forming a 32-bit integer with a carry bit. The bits not taken on the right side of the window will be used for rounding, while the bits not taken on the left side will be used for saturation.
[0045] Note that the magnitude of the left or right shift considered is the size of the integer addition operand, here 32 bits (31 plus the position without the shift for the zero exponent), corresponding to a signed exponent of 6 bits and not 8 bits. This restricted magnitude corresponds to the edge positions where significant bits of the shifted product can still be found within the window and constitute a usable integer. To the left, this corresponds to the product of an integer equal to 1 and a floating-point number with an exponent of +31. To the right, this corresponds to the product of an integer whose most significant bit is 1 and a floating-point number with an exponent of -31. Beyond an exponent of ±31, the result is systematically saturated or rounded to 0.
[0046] The contents of the 33-bit window are provided to an adder 14 which also receives the 32-bit integer operand Op2. The adder 14 produces a 34-bit result, the 34th bit being a carry bit.
[0047] The result of the adder 14 is provided to a saturation and rounding circuit 16 which also receives the 55 bits on the right and the 31 bits on the left of the window of the shift circuit 12. The circuit 16 produces a final result RI in the form of a signed integer INT32, according to the following conditions.
[0048] The result RI is saturated to the largest positive integer when at least one of the following conditions is satisfied:
[0049] the exponent is greater than +31 and the integer multiplicand Opl is non-zero and the sign of the multiplication is positive,
[0050] at least one of the 31 bits on the left side of the window is set to 1 and the multiplication sign is positive, or
[0051] one of the two most significant bits of the result (bits 32 and 33) of the addition is set to 1.
[0052] The result RI is saturated to the smallest negative integer when at least one of the The following conditions are met:
[0053] the exponent is less than -31 and the integer multiplicand Opl is non-zero and the sign of the multiplication is negative,
[0054] at least one of the 31 bits on the left side of the window is set to 1 and the multiplication sign is negative, or
[0055] one of the two most significant bits of the result of the addition is 0.
[0056] In the description of the circuit in [Fig. 1], it has been assumed for clarity that the numbers are positive. In practice, numbers are signed so that, for negative numbers, the significant value of the bits is 0 instead of 1. The circuit in [Fig. 1] is extended in a known way to also handle negative numbers and invert the required bits according to a sign bit. For example, for a negative product P, the left and right bits at the input of shifter 12 are all 1 instead of 0.
[0057] Furthermore, since the operator involves a floating-point number FP32 as the multiplicand, this number can be undefined (NaN) or infinite in certain situations. If it is infinite and the integer multiplicand Opl is non-zero, the result RI is saturated to the largest positive integer or the smallest negative integer depending on the sign of the result. If it is undefined or infinite and multiplied by an integer zero, the result RI can be conventionally saturated, for example, to the smallest negative integer.
[0058] Finally, the RI result is rounded according to the bits having a significant value to the right of the window, for example according to the IEEE 754 standard governing floating-point numbers.
[0059] The operator can also raise the IEEE 754 flags for overflow, underflow, inaccuracy, and invalidity. Underflow is raised when the unrounded result is not zero and strictly between -1 and 1 (so-called subnormal numbers).
Claims
Demands
1. Hybrid multiply-accumulate hardware operator configured to multiply a floating-point (FP32) formatted multiplicand (OpO) by an integer (INT32) formatted multiplicand (Opl) and add a result of the multiplication to an integer (INT32) formatted operand (Op2), in which the result of the multiplication is an integer of the same format as the operand, up to a carry bit.
2. An operator according to claim 1, comprising: a first multiplicand input (OpO) for the floating-point multiplicand having a mantissa field (MANT) and an exponent field (EXP); a second multiplicand input (Opl) for the integer multiplicand; an accumulator input (Op2) for the integer operand; an integer multiplier (10) configured to multiply the mantissa field of the floating-point multiplicand by the integer multiplicand and produce a result in a fixed-point format corresponding to a zero exponent of the floating-point multiplicand; a shifter (12) connected to receive the result of the multiplier and shift it left or right depending on the value of the exponent field of the floating-point multiplicand;and an integer adder (14) receiving the integer operand (Op2) and a window on the result of the shifter, the window capturing a number of bits to the left of the fixed decimal point position equal to the number of bits of the integer operand up to a carry bit.
3. An operator according to claim 2, wherein the left and right shift amplitude of the shifter is limited to the number of bits in the integer operand, and the operator comprises a saturation circuit (16) connected to the adder and the shifter and configured to: produce the adder result when no bit to the left of the window is significant and the exponent field encodes a value less than the number of bits in the integer operand, and produce a saturated result when at least one bit to the left of the window is significant or the exponent field 9 encodes a value greater than or equal to the number of bits of the integer operand.
4. Operator according to claim 3, wherein the saturation circuit (16) is configured to round the result of the adder according to significant values of the bits located to the right of the window.