Method and apparatus for approximating computation of a softmax function

Abstract

The application discloses a method and device for approximating calculation of a softmax function. The method simplifies the exponential operation of e into twice constant multiplication and once exponential operation of e with the output range of (1, 2), which is mainly to limit the range of e exponent to utilize the characteristics of small range and high precision of the cordic algorithm; and simplifies N times division operation into once reciprocal operation with fixed input range, N times multiplication and shift operation. The device of the application comprises a constant multiplication unit, an exponential calculation unit, a floating point addition unit, a mantissa calculation unit, a subtraction array unit and a multiplication unit, adopts a carry-save adder instead of a traditional adder, and further shortens the critical path. The scheme of the application greatly improves the calculation speed while maintaining high precision, and reduces the consumption of calculation resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of very large scale integrated circuit design, and in particular to a high-precision, low-latency softmax function approximation calculation method and device. BACKGROUND

[0002] In recent years, deep learning has become a key tool for solving complex problems in image recognition, natural language processing, autonomous driving, etc. When designing hardware-based deep neural networks, the implementation of activation functions becomes a major challenge.

[0003] As a basic component in deep neural networks, activation functions are difficult to achieve high-speed calculation while maintaining ideal fitting accuracy in hardware due to their non-linear characteristics. Softmax function, as a common activation function, can map the output of multiple neurons to the interval (0, 1), and the sum of all outputs is 1, which can be considered as a probability. It is commonly used in situations where multiple outputs are weighted or individual weights are calculated. The exponential and division calculations involved in the softmax function require a large amount of hardware resources. The optimization of the exponential part mostly uses piecewise linear approximation or lookup table methods, but these methods need to be re-segmented and approximated or tabled when the calculation accuracy requirement changes, lacking a certain universality. Therefore, there is an urgent need for a high-precision, low-latency universal softmax function approximation calculation method and device. SUMMARY

[0004] The present application aims to provide a softmax function approximation calculation method and its implementation hardware architecture, achieving high precision while maintaining low latency.

[0005] To achieve the above-mentioned application purposes, the technical solutions adopted by the present application are as follows:

[0006] A softmax function approximation calculation method includes the following steps:

[0007] Step 1, calculate each element x i of the input array x and store them respectively;

[0008] Step 2, accumulate the results of each element calculation ;

[0009] Step 3, calculate the reciprocal of the accumulated , N is the number of elements of array x;

[0010] Step 4, multiply the reciprocal calculated in step 3 with the stored in step 1.

[0011] Further, in step 1, the e index calculation uses the following formula:

[0012]

[0013] Wherein Y is the element x i The integer value of The cordic hyperbolic model algorithm is used to obtain Y The multiplication of 2

[0014] The present application also provides a device for implementing the above-mentioned method for approximating the calculation of the softmax function, comprising:

[0015] A constant multiplication unit for performing two constant multiplication operations: the first multiplication is to multiply the input floating point number by a constant And separate the obtained integer part and decimal part; the second multiplication is to multiply the decimal part by ln2 and output to the exponent calculation unit;

[0016] An exponent calculation unit for calculating the exponent operation using the cordic algorithm, which includes multiple selectors, shifters and adders, and adopts a pipeline multi-stage iterative structure, with register temporary storage between stages;

[0017] A floating point addition unit adopting a multi-stage addition structure, each stage including multiple two-input floating point adders, when the input number is N, if Is not an integer, the stage number is Downward rounding plus one;

[0018] A mantissa calculation unit, which inputs the mantissa output by the floating point addition unit, for calculating the reciprocal of the denominator of the softmax function;

[0019] A subtraction array unit, which inputs the floating point number exponent part and the floating point number exponent sum output by the floating point addition unit, for calculating the difference between each exponent value and the exponent sum, and converting the subtraction operation into an addition operation by adding a sign bit and calculating the complement code to obtain the value;

[0020] A multiplication unit, which inputs the decimal part output by the exponent calculation unit and multiplies it with the output of the mantissa calculation unit, and shifts the obtained result according to the output difference value of the subtraction array unit, and finally outputs the calculation result of the softmax function.

[0021] Further, the mantissa calculation unit uses the cordic algorithm to calculate the reciprocal of the denominator.

[0022] Further, the constant multiplication unit multiplies the input floating point number by a constant ​Approximately 1.4427; the decimal part is multiplied by ln2, approximately 0.6931.

[0023] Further, the multiplication operation of the constant multiplication unit is completed by a shift and addition operation, and the addition operation is completed by a carry-save adder.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] The present application optimizes the calculation of the softmax function, converts the exponential operation into a shift and subtraction operation by using the cordic hyperbolic model method, and converts the multiple division operation of the softmax function into a reciprocal operation and a multiple multiplication operation, wherein the multiplication operation can be realized by shift accumulation, thereby reducing the consumption of hardware resources. The present application also proposes a softmax hardware optimization and implementation device based on the approximate method, which uses a carry-save adder to shorten the operation time, so that the calculation delay is lower while maintaining high precision of the calculation result, and is more suitable for application scenarios with high precision and real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the overall hardware architecture diagram of the present application.

[0027] Figure 2 is a schematic diagram of a carry-save adder of the constant multiplication unit.

[0028] Figure 3 is a single-level iteration schematic diagram in the exponential calculation unit.

[0029] Figure 4 is a 3-level floating-point adder schematic diagram in the floating-point addition unit.

[0030] Figure 5 is a hardware structure diagram of a two-input floating-point adder. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0032] The expression of the softmax function is as follows:

[0033]

[0034] wherein x i represents the i-th element of the input array x, p i represents the softmax value of the element, and the array x has a total of N elements.

[0035] The softmax function hardware scheme of the present embodiment is characterized in that the following formula is used:

[0036]

[0037] The above formula is explained as follows: first, based on the mathematical transformation of the traditional softmax function to reduce the range of e exponential operation, the e exponential operation is converted into 2 整数 小数 , the range of e 小数 is limited to [1, 2], and the specific conversion formula is as follows:

[0038]

[0039] Where x i represents the i-th element of the input array x, Y i is the integer part of the result obtained by multiplying the element x i , X i is equal to , that is, the e 小数 mentioned above.

[0040] Then, the transformed e 小数 above is converted into a shift-add operation using the cordic hyperbolic model method. The cordic hyperbolic model method is essentially based on a hyperbola as a reference line, and the initial vector (1, 0) is used as the initial state of the rotation vector. The end point of the vector after rotation is always on the hyperbola, and the rotation angle θ i of the vector is i, which represents the number of rotations. By controlling the angle θ i , tanhθ i is equal to 2 -i , and after a certain number of rotations, the sum of the rotation angles is θ. The horizontal and vertical coordinates of the vector obtained after the above iteration are where K is the scaling factor, and the scaling factor is a constant 1.2051 after 8 rotations. The specific iteration formula is as follows:

[0041]

[0042] In the formula, x i , y i are the coordinates of the rotation vector after i rotations, and z i is the difference between the rotation angle i and the target angle. x i+1 , y i+1 , z i+1 are the results after one iteration of x i , y i , z i , i represents the number of rotations, and the initial i is equal to 0. At this time, x0, y0 are 1, 0, and z0 is the input, that is, the e 小数 ​​The smaller value in the equation. When the iteration count is 8, adding x8 and y8 and multiplying by 1.2051 gives e. 小数 .

[0043] Then add the denominators of the transformed softmax function. Then, the reciprocal of the denominator is calculated. The reciprocal operation is transformed into the following iterative formula using the Cordic algorithm:

[0044]

[0045] The principle of the cordic algorithm here is similar to that described earlier, but the x-coordinate of the endpoint of the rotating vector remains constant, and the vector rotates by an angle θ each time. i 'i' represents the number of rotations, controlled by the angle θ. i Make tanθ i Equal to 2 -i The x and y in the formula i The x and y coordinates of the vector after i rotations, z i It is the difference between the angles rotated i times and the target angle. The initial value x is equal to the input, which is used as the denominator for calculating the reciprocal. y0 and z0 are 1 and 0 respectively. After a certain number of iterations, the final z... i That is the final reciprocal value.

[0046] Finally, the division between the numerator and denominator of the transformed softmax function is converted into the multiplication of the numerator and the reciprocal of the denominator, ultimately yielding the softmax value of the i-th element of the input array x.

[0047] The hardware design scheme for the softmax function in this embodiment takes N random floating-point numbers as input, with a default precision of 32 bits and an output range of (0,1). It mainly includes: a constant multiplication unit, an exponent calculation unit, a floating-point addition unit, a mantissa calculation unit, a subtraction array unit, and a multiplication unit. The constant multiplication unit performs two multiplication operations; the exponent calculation unit performs exponentiation using the Cordic hyperbolic model algorithm; the floating-point addition unit performs multi-level floating-point addition operations; the mantissa calculation unit calculates the reciprocal of the mantissa; the subtraction array unit calculates the difference between each exponent and the sum of exponents; and the multiplication unit multiplies the output of the exponent calculation unit by the reciprocal, then shifts the result of the subtraction array to obtain the final probability value.

[0048] The function and implementation process of each unit are described in detail below:

[0049] The constant multiplication unit mainly performs two multiplications. The multiplication unit consists of a shifter and an accumulator. According to formula (2), the first multiplication multiplies the input floating-point number by... The second multiplication multiplies the decimal part by ln2. In terms of precision, the input is a 32-bit floating point number, which is composed of 1-bit sign bit, 8-bit exponent bit and 23-bit mantissa bit. The first multiplication constant is replaced by 1.4427, and the result is still taken as 32-bit after multiplication. The second multiplication constant ln2 is replaced by 0.6931, and since it is the multiplication of two decimal parts less than 1, the 11 bits of mantissa are taken to participate in the calculation, which can ensure very high precision. The accumulator uses a carry-save adder, and takes three numbers as an example. As shown in Figure 2 , the mantissa parts a, b and c are added after three times of shifting, and the bit number of the three mantissa is 23-bit. The same bits of the three numbers are added respectively, and the "carry" and "sum" are calculated by the full adder, which are represented by sm and cm respectively. The serial number after the sign represents the bit number. The "carry" sm of the previous stage is passed to the next stage to be added with the "sum" sm of the next stage, and the final value is obtained by the full adder, which further reduces the calculation delay.

[0050] The exponential calculation unit is mainly used to calculate the exponential part. According to formula (2), X i satisfies the Cordic algorithm to calculate the exponential range in the range of (1, 2). The entire exponential calculation unit is composed of three selectors, two shifters, three adders and a lookup table. The lookup table mainly records the value of tanh -1 (2 -i ), which is stored by using 32-bit floating point number. The actual lookup table value precision is changed by taking part of the mantissa value and adding the exponent. Generally, the mantissa takes eight bits to achieve very high precision. In addition, when calculating the final exponential value, it also needs to multiply the correction factor K. The value of K is related to the number of rotations of the Cordic algorithm. When the number of iterations reaches eight times, the value of K tends to 1.2051, and the value of tanh -1 (2 -i ) is listed as follows:

[0051]

[0052] The floating point addition unit is composed of multiple two-input floating point adders (see Figure 4 ), and the structure of a single two-input floating point adder is as shown in Figure 5 . The number of stages of the floating point addition unit is determined by the number of inputs. Generally speaking, when the number of input data is N, the number of stages of the adder is If If not, rounding up one and delaying the adder of the previous stage. For the input floating point number, the difference of the exponent part exp1 and exp2 is calculated and then passed to the shift unit, and the result is used to control the selector. The two mantissas mant1 and mant2 are passed to the two selectors, and the output of the selector is determined by the comparison unit. When exp1 is greater than exp2, mant2 is output to the shift unit; when exp1 is less than exp2, mant1 is output to the shift unit. The shifted mantissa is added to the other mantissa, and the carry value is passed to the three units, the shift unit, the selector MUX and the full adder. When the carry value is 1, the output exponent is incremented by 1, and the added mantissa is right shifted by one bit. At this time, the selector MUX selects the output mantissa of the shift end. When the carry value is 0, the selector MUX directly selects the mantissa and outputs.

[0053] The mantissa counting unit, in this embodiment, uses the cordic algorithm to calculate the reciprocal. The specific algorithm is shown in formula (5), i represents the number of iterations, y i and z i represent the value after i iterations. The input mantissa x is first shifted through the shift register, and then added or subtracted with the input y i , and the output result is passed to the selector. The input mantissa x i is right shifted by i bits, and the input z i is the initial value of 0, and the result of the addition or subtraction operation after shifting i bits of 1 is passed to the two-way selector and selected by y i signal. When y i is positive, the output is the subtraction result; when y i is negative, the output is the addition result. After a certain number of iterations, z i outputs the final reciprocal value.

[0054] The subtraction array unit of this embodiment, the input is the exponent part of the floating point number and the exponent of the floating point number. The subtraction array unit first adds the sign bit to the input unsigned number, which is 0 by default, and then performs the complement operation. The input is added to the complement to convert the subtraction operation to addition operation for evaluation. If the sign bit of the obtained number is 0, it is directly output; if the sign bit is 1, the result is output after the complement operation.

[0055] The multiplication unit of the embodiment inputs the exponential difference value of the subtraction array unit, the reciprocal of the mantissa calculation unit and the decimal part of the exponential calculation unit, and outputs the probability value of softmax. The probability output of the final floating point number is realized by multiplying the reciprocal and the decimal and combining the exponential difference value shift. In terms of precision, the input reciprocal part precision is determined by the input precision of the floating point addition unit. Taking eight bits as an example, the input decimal part is also eight bits, so the multiplication result is 16 bits. Under the premise of no data loss, high-precision floating point number output can be realized through the shift operation.

[0056] The device is implemented on the development board ZynqUltraScale+ZCU102 Evaluation Board, and 1000 random data sets within the interval [0, 20] are input, each data set length is between 4 and 20, and the results are error analyzed based on the calculation results of the softmax function provided by matlab. The final average error is kept below 0.005, indicating that the device method is suitable for application scenarios with high requirements for the precision of the softmax function.

Claims

1. An apparatus for approximated computation of a softmax function, the apparatus comprising: The device is used to perform the following steps: Step 1, calculate each element x of the input array x. i corresponding And store them separately; Step 2, calculate the results of each element. The results are accumulated; Step 3, calculate the accumulated result. The reciprocal of , where N is the number of elements in array x; Step 4, store the result obtained in Step 1. Multiply by the reciprocal obtained in step 3; The device specifically comprises: a constant multiplication unit for performing two constant multiplication operations: a first multiplication is multiplying an input floating point number by a constant and separating an integer part and a decimal part of the result; a second multiplication is multiplying the decimal part by ln2 and outputting the result to an exponent calculation unit; An index calculation unit for calculating an exponential operation by using a cordic algorithm; the unit comprises multiple selectors, shifters and adders, and adopts a pipeline multi-stage iteration structure, and the stages are temporarily stored by registers; The floating point addition unit adopts a multi-stage addition structure, each stage including a plurality of two-input floating point adders, when the input number is N, if is not an integer, the stage number is rounded down and added by one; A mantissa calculation unit, which takes the mantissa output by the floating point addition unit as input, and is used for calculating the reciprocal of the denominator of a softmax function; A subtraction array unit, which takes the floating point index part and the floating point index sum output by the floating point addition unit as input, and is used for calculating the difference between each index value and the index sum, and converting the subtraction operation into an addition operation by adding a sign bit and calculating a complement code to obtain the result; A multiplication unit, which takes the decimal part output by the index calculation unit as input, and is multiplied by the output of the mantissa calculation unit respectively, and the result is shifted according to the difference value output by the subtraction array unit, and finally the calculation result of the softmax function is output.

2. The apparatus of claim 1, wherein, In step 1, the e index calculation adopts the following formula: wherein Y is an element x i multiplying the integer value of, using the cordic hyperbolic model algorithm to obtain, and 2 Y the multiplication of is converted into a shift operation.

3. The apparatus of claim 1, wherein, The mantissa calculation unit adopts a cordic algorithm to calculate the reciprocal of the denominator.

4. The apparatus of claim 1, wherein, The constant multiplication unit multiplies the input floating point number by a constant Approximately 1.4427; the fractional part is multiplied by ln2, approximately 0.6931.

5. The apparatus of claim 4, wherein, The multiplication operation of the constant multiplication unit is completed by shifting and addition operation, and the addition operation is completed by using a carry-save adder.

Images