Optimized multiply accumulate operators for AI computations
By designing hybrid hardware multiplier accumulation and additional transportation (HFMA), the disadvantages of FP32 arithmetic in deep learning have been solved, and high -efficiency tensor conversion and calculations are achieved, and the performance and efficiency of deep learning networks are improved.
Patent Information
- Application Number
- CN202410982336.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-25
- Filing Date
- 2024-07-22
- Publication Date
- 2025-01-28
AI Technical Summary
In deep learning technology, FP32 arithmetic is used to modify the calculation of the representation of tensor elements between quantized domains, resulting in higher implementation costs of performance, power consumption, and silicon area.
Design a hybrid hardware multiplication and accumulation of additional transport computing (HFMA). This operator can multiply the number of floating points into integer and add to the number of integer operations. Condition.
Through the HFMA operator, the computing in the deep learning network can be performed without FP32 arithmetic, reducing the number of operations, improving performance and efficiency, and reducing the cost of power consumption and silicon area.
Smart Images
Figure CN119376691A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to deep learning techniques in the field of artificial intelligence, and more specifically to hardware multiply-accumulate operators suitable for performing some of the computations involved. Background Art
[0002] In deep learning techniques, some of the layers of a deep neural network use arithmetic involving small integers (e.g., 8-bit integers, referred to as INT8) rather than more hardware-intensive 32-bit floating-point arithmetic (FP32 number format according to the IEEE 754 standard).
[0003] Conventional deep learning techniques are based on simple integer quantization transformations between FP32 floating point numbers and INT8 numbers. A real number X represented in FP32 format is mapped to an INT8 number x within a given quantization domain defined by a 32-bit integer zero point z (INT32) and an FP32 floating point scaling factor S. Then, the quantized number X, denoted as ~X, is represented by the following formula:
[0004] ~X=(xz)S
[0005] The number x in INT8 format is the number that is stored and used for arithmetic operations. The quantization domain corresponds to the low-resolution range of the number ~X around the central value z that is located in the high dynamic range (the high dynamic range of the factor S (FP32 number)). When the value of x changes in the range [-128, 127], the range of variation of ~X becomes:
[0006] ~X∈[(-128-z)S,(127-z)S]
[0007] This range of variation (which in practice is considered to cover a Gaussian distribution of X values around a zero point z) does not cover the entire range of variation of the number X. Therefore, for values of X outside this range, ~X saturates at the boundaries of the range.
[0008] There can be one quantization domain per activation tensor (a multidimensional matrix of activation values), and channels of a tensor (one of its dimensions) can sometimes have different quantization domains. Once FP32 numbers have been quantized, they can be exploited using resource-efficient operations on small integers x.
[0009] Despite the advantages of this quantization, FP32 arithmetic is still used in calculations that modify the representation of tensor elements between quantization domains. FP32 arithmetic is more expensive to implement in terms of performance, power consumption, and silicon area. Summary of the invention
[0010] A hybrid hardware multiply-accumulate operator is generally provided and is configured to multiply a floating point multiplicand by an integer multiplicand and to add the integer multiplication result to an integer operand.
[0011] The operator may include a first multiplicand input of the floating-point multiplicand, the floating-point multiplicand having a mantissa field and an exponent field; a second multiplicand input of the integer multiplicand; an accumulator input of the integer operand; an integer multiplier configured to multiply the mantissa field of the floating-point multiplicand by the integer multiplicand and produce a result in a fixed-point format corresponding to a zero exponent of the floating-point multiplicand; a shifter connected to receive the result of the multiplier and shift the result left or right depending on the value of the exponent field of the floating-point multiplicand; and an integer adder receiving the integer operand and a window on the result of the shifter, the window capturing a number of bits to the left of a fixed decimal point position equal to the number of bits of the integer operand within a carry position.
[0012] The left shift amplitude and the right shift amplitude of the shifter may be limited to the number of bits in the integer operand, and the operator may include a saturation circuit, which is connected to the adder and the shifter, and the saturation circuit is configured to: produce the result of the adder when no bit on the left side of the window is valid and the exponent field encodes a value less than the number of bits in the integer operand; and produce a saturated result when at least one bit on the left side of the window is a valid value or the exponent field encodes a value greater than or equal to the number of bits in the integer operand.
[0013] The saturation circuit may be configured to round the adder result according to the effective value of the bit to the right of the window. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The following non-limiting description is provided in relation to the accompanying drawings, in which:
[0015] Figure 1 A block diagram of a mixed multiply-add operator is shown. DETAILED DESCRIPTION
[0016] In the following, a new type of hardware hybrid fused multiply-add (HFMA) operator for efficient tensor conversion between different quantization domains is provided. The structure of the HFMA operator is derived from a specific analysis and decomposition of the operations involved in converting between quantization domains. The resulting structure multiplies a signed integer (e.g., INT32) by a floating point number (e.g., FP32), and adds a signed integer operand (INT32) to the product, thereby providing an integer result in the same format as the operands. Since the result is in the same format as the addition operand, the structure can be used to compute the accumulation of the product, so the structure can also be called a "hybrid multiply-accumulate" (HMAC) operator.
[0017] The most common computation in deep learning networks is to transform the activation vector X according to the following relationshipi With the weight vector W i The accumulated scalar product of plus the bias B j (Because the quantized values are mentioned exclusively below, the prefix "~" is omitted to clarify the writing):
[0018]
[0019] Each of the terms is quantized in its own quantization domain, which is (z y ,S y )、(z x ,S x )、(0,S w ) and (0,S x S w ). Weight W i and bias B j The zero point of B is 0. j The quantization domain (0,S x S w ) has the same value as the product X i W i The resulting quantization domain has the same scaling factor.
[0020] The different terms are represented as follows:
[0021] X i =(x i -z x )S x
[0022] W i =w i S w
[0023] B j =S x S w b j
[0024] Y j =(y j -z y )S y
[0025] Remember that the value S is the FP32 scaling factor, the value z is the INT32 zero point, and x i and j It is an INT8 number used for storage and loop operations.
[0026] Substituting in these relations yields:
[0027]
[0028] Then, the integer representing the quantized result is expressed as:
[0029]
[0030] in
[0031]
[0032] Each item c j is an INT32 integer constant that can be computed offline using simple integer arithmetic, such as the normal multiply-accumulate (MAC) operator that operates on INT32 integers. For a range of j values, the sum is an integer constant that can be computed once and stored to compute all terms c j .
[0033] The basic loop computation on index j, which conventionally involves operations in FP32 arithmetic, becomes an integer product and add operation that converts the INT32 integer Multiplying real numbers in FP32 format And add the INT32 integer z y , the result is also an INT32 number. This is exactly what the HFMA operator calculates. Then, y j The values are saturated to fit back into INT8 integers and are stored and reused to represent the quantized y j value.
[0034] Using conventional means, this operation would include converting the integer multiplicand and integer addition operand to FP32 numbers, performing a fused multiply-add (FMA) on the resulting FP32 numbers, and converting the FP32 result to an INT32 integer, i.e., three or four operations, rather than one, one of which involves a complex FMA operator for the FP32 numbers.
[0035] In the above example, this gain in the number of operations occurs once for every n+1 iterations on pair i. In some cases, n may be of the same order as the number of iterations on pair j, resulting in a gain of the order of 3n operations.
[0036] The HFMA operator can also be used for on-time operations such as summation of quantized vectors:
[0037] Y i =X i +W i
[0038] X i =(x i -z x )Sx
[0039] W i =(w i -z w )S w
[0040] Y j =(y j -z y )S y
[0041]
[0042] This corresponds to a nested application of the HFMA operator or the HMAC operator (with two INT32 integer additions).
[0043] The computation can be further optimized when the quantization domain of one of the inputs is the same as the quantization domain of the output, which often happens in practice. For example, if S x =S y And z x =z y ,but:
[0044]
[0045] This corresponds to a single application of the HMAC operator (with one INT32 integer addition).
[0046] The HMAC operator can also be used in conjunction with an operator that specifies FSCALE to quantize FP32 values to INT8 values without involving FP32 arithmetic. The FSCALE operation scales the FP32 value by an integer power of two in the form of simple integer addition of the 8-bit exponent field of the FP32 numerical representation.
[0047] For example, in X i =(x i -z x )S x and In this case, we get:
[0048]
[0049] Here, the FSCALE operation produces an FP32 result, while e is INT8, and m and z are INT32. Therefore, x i The calculation is again an HMAC calculation, producing an INT32 value which will be transformed into an INT8 value.
[0050] Figure 1A block diagram of a mixed multiply-add (or accumulate) operator is shown. The operator is configured to perform an operation Op0 Op1 + Op2, where the multiplicand Op0 is a floating point number (e.g., FP32), the multiplicand Op1 is an integer (e.g., INT32), and the operand Op2 is an integer of the same format (INT32) as the operand Op1.
[0051] The floating-point multiplicand Op0 includes an 8-bit exponent field EXP and a 24-bit mantissa field MANT. The mantissa field MANT of the operator Op0 (considered as an integer) and the integer multiplicand Op1 are provided to the integer multiplier 10. The multiplier 10 generates a 57-bit product P. The product P is actually a fixed-point number, the position of which is conventionally selected so that the exponent EXP is equal to 0. In this case, the product corresponds to the number 1.xx...xx (with 24 bits after the decimal point) multiplied by the integer XX... XX is 32 bits (before the decimal point), so that the fixed decimal point is located between the 23rd and 24th bits of the 57-bit product P.
[0052] The left shift circuit 12 receives the product P at position 31 and position 87 of the 119-bit virtual number, which has 31 bits from 0 to the right of the product P and 31 bits from 0 to the left of the product P. The shift of the circuit 12 is controlled by the exponent EXP of the multiplicand Op0. When the exponent is positive, the circuit 12 performs a corresponding left shift. When the exponent is negative, the circuit 12 performs a corresponding right shift.
[0053] The "useful" output of shifter 12 is taken from a window covering position 55 to position 87 of the shifted number, i.e., 33 bits before the fixed decimal point, forming a 32-bit integer with a carry bit. Unused bits to the right of the window are used for rounding, while unused bits to the left are used for saturation.
[0054] Note that the left or right shift magnitude considered is the size of the integer addition operand, here 32 bits (31 plus the unshifted positions when the exponent is zero), corresponding to a 6-bit instead of an 8-bit signed exponent. This constrained magnitude corresponds to the extreme positions where the significant bits of the shifted product can still be in the window and form a usable integer. On the left, this corresponds to the product of an integer equal to 1 and a floating point with an exponent of +31. On the right, this corresponds to the product of an integer with a most significant bit of 1 and a floating point with an exponent of -31. When the exponent exceeds ±31, the result is systematically saturated or rounded to 0.
[0055] The contents of the 33-bit window are provided to adder 14, which also receives a 32-bit integer operand Op2. Adder 14 produces a 34-bit result, the 34th bit being the carry bit.
[0056] The result of adder 14 is fed to a saturation and rounding circuit 16 which also receives 55 bits to the right and 31 bits to the left of the window of shifter 12. Circuit 16 produces a final result R1 in the form of a signed integer INT32 under the following conditions.
[0057] The result R1 is saturated to the maximum positive integer when at least one of the following conditions is met:
[0058] the exponent is greater than +31, the integer multiplicand Op1 is nonzero, and the sign of the multiplication is positive,
[0059] At least one of the 31 bits to the left of the window is 1, and the sign of the multiplication is positive, or
[0060] One of the two most significant bits (bit 32 and bit 33) in the result of the addition is 1.
[0061] The result R1 is saturated to the smallest negative integer when at least one of the following conditions is met:
[0062] the exponent is less than -31, the integer multiplicand Op1 is nonzero, and the sign of the multiplication is negative,
[0063] At least one of the 31 bits to the left of the window is 1, and the sign of the multiplication is negative, or
[0064] One of the two most significant bits of the addition result is 0.
[0065] exist Figure 1 In the description of the circuit shown, it is assumed for clarity that the numbers are positive. In practice, numbers are signed so that for negative numbers the effective value of a bit is 0 rather than 1. Figure 1 The circuit shown is extended in a known manner to also handle negative numbers, inverting the required bits depending on the sign bit. For example, for a negative product P, the bits to the left and right at the input of the shifter 12 are both 1s instead of 0s.
[0066] Furthermore, since the operator involves a floating point number FP32 as a multiplicand, the number may be indefinite (NaN) or infinite in some cases. If the number is infinite and the integer multiplicand Op1 is non-zero, the result R1 is saturated to the largest positive integer or the smallest negative integer depending on the sign of the result. If the number is indefinite or infinite and is multiplied by a zero integer, the result R1 may be saturated by convention, for example, to the smallest negative integer.
[0067] Finally, the result R1 is rounded by the number of bits of the significant value to the right of the window (eg, according to the IEEE 754 standard governing floating point numbers).
[0068] The operator may also raise the IEEE 754 overflow, underflow, inexact, and invalid flags. The underflow flag is raised when the unrounded result is non-zero and strictly between -1 and 1 (a so-called subnormal number).
Claims
1. A hybrid hardware multiply-accumulate operator, the hybrid hardware multiply-accumulate operator configured to multiply a multiplicand (Op0) formatted as a floating point number (FP32) by a multiplicand (Op1) formatted as an integer (INT32), and add the result of the multiplication to an operand (Op2) formatted as an integer (INT32), wherein the result of the multiplication is an integer of the same format as the operands, except for a carry bit.
2. The operator according to claim 1, comprising: a first multiplicand input (Op0) of a floating point multiplicand having a mantissa field (MANT) and an exponent field (EXP); Second multiplicand input (Op1) of integer multiplicand; Accumulator input for integer operands (Op2); an integer multiplier (10) configured to multiply the mantissa field of the floating-point multiplicand by the integer multiplicand and produce a result in a fixed-point format corresponding to a zero exponent of the floating-point multiplicand; a shifter (12) connected to receive the result of the multiplier and to shift the result left or right depending on the value of the exponent field of the floating point multiplicand; and An integer adder (14) receives the integer operand (Op2) and a window on the result of the shifter, the window capturing the number of bits to the left of a fixed decimal point position equal to the number of bits of the integer operand within a carry position.
3. The operator according to claim 2, wherein the left shift amplitude and the right shift amplitude of the shifter are limited by the number of bits of the integer operand, and the operator comprises a saturation circuit (16) connected to the adder and the shifter, and the saturation circuit is configured to: producing the result of the adder when no bits to the left of the window are significant and the exponent field encodes a value less than the number of bits of the integer operand, and A saturated result is produced when at least one bit to the left of the window is a valid value or the exponent field encodes a value greater than or equal to the number of bits in the integer operand.
4. The operator according to claim 3, wherein the saturation circuit (16) is configured to round the adder result according to the effective value of the bits to the right of the window.