Fast computing device and method of nonlinear activation function based on coefficient lookup table
By using a coefficient lookup table-based method, the activation function is divided into intervals and approximated by a quadratic function, which solves the problem of slow computation speed of nonlinear activation functions and achieves high-efficiency computation speed and accuracy improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2024-09-19
- Publication Date
- 2026-07-21
Smart Images
Figure CN119249050B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of neural network technology, and in particular to a fast calculation device and method for nonlinear activation functions based on coefficient lookup tables. Background Technology
[0002] With the rise of deep learning, the number of layers in neural networks is constantly increasing, and the parameter scale is becoming increasingly large. Training models also requires a large amount of computing resources. As the basic operation of each neuron in a neural network, the computational efficiency of activation functions directly affects the training speed of the model. In particular, for complex nonlinear activation functions such as Sigmoid, Tanh, and Mish, the computational overhead is especially huge, which seriously restricts the real-time performance of neural networks.
[0003] Among related technologies, the CORDIC algorithm and Taylor expansion method can be used. The CORDIC algorithm approximates mathematical functions using a series of rotations and iterations, involving only simple addition, subtraction, and shift operations, making it very suitable for hardware implementation. However, this algorithm is mainly used to calculate trigonometric and hyperbolic functions, lacking versatility; and achieving high accuracy requires multiple iterations, resulting in slow speed. The Taylor expansion method uses the Taylor formula to expand the activation function into a power series for approximation, but achieving high accuracy generally requires expansion to the fourth order or higher, incurring huge multiplication overhead and failing to address the slow computation speed of nonlinear activation functions. Summary of the Invention
[0004] This application provides a fast calculation device and method for nonlinear activation functions based on coefficient lookup tables to solve the problem of slow calculation speed for nonlinear activation functions.
[0005] The first aspect of this application provides a fast calculation device for nonlinear activation functions based on a coefficient lookup table, comprising: The input selection unit is used to select the input floating-point number based on the activation function type; The coefficient lookup table unit is used to divide the entire region of the activation function into several intervals, and to divide the non-infinite intervals in the intervals into several segments; to approximate each segment using a quadratic function, in order to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein, the quadratic function coefficients include linear coefficients, quadratic coefficients, and constant terms; A preprocessing unit is used to extract features from the floating-point number to obtain the sign bit, exponent, and mantissa; The arithmetic unit is used to calculate an index value and a second mantissa sub-segment based on the sign bit, exponent, and mantissa; to obtain the corresponding quadratic function coefficients from the coefficient lookup table based on the index value; and to obtain the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations. The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine it with the sign bit and exponent of the floating-point number to output the value in floating-point format.
[0006] The aforementioned device, through a coefficient lookup table, can transform the calculation of nonlinear activation functions into simple quadratic function multiplication and addition operations, thereby effectively improving the calculation speed of nonlinear activation functions and solving the problem of slow calculation speed of nonlinear activation functions.
[0007] Optionally, the endpoints of the interval include 0, powers of 2 or their opposites, positive infinity, and negative infinity; When the interval is located in the non-negative number field, the left endpoint of the interval is closed and the right endpoint of the interval is open; When the interval is located in a non-positive number field, the left endpoint of the interval is open and the right endpoint of the interval is closed.
[0008] Optionally, the coefficient lookup table unit is also used to determine the number of segments in the non-infinite interval and the number of decimal places in the coefficients of the quadratic function; Perform the best uniform approximation on the next segment or the first segment, and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the coefficients of the linear term; The quadratic coefficient is calculated based on the coefficient of the linear term, the number of decimal places of the coefficient of the quadratic function, and the number of digits in the first mantissa sub-segment. Perform the best uniform approximation again, and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the constant term; The coefficients of the first term, the coefficients of the second term, and the constant term are stored in hexadecimal form. Based on the coefficients of the first term, the coefficients of the second term, and the constant term, a coefficient lookup table for the non-infinite interval is generated.
[0009] Optionally, the floating-point number is an IEEE 754 single-precision floating-point number.
[0010] Optionally, the interval includes an infinite interval and a non-infinite interval; the non-infinite interval includes an intermediate interval and an interval containing zero.
[0011] Optionally, the intervals of the non-negative number field in the coefficient lookup table are arranged in ascending order; the intervals of the non-positive number field in the coefficient lookup table are arranged in descending order, and the array index of the arrangement of each interval in the coefficient lookup table is used as the index value.
[0012] Optionally, the arithmetic unit is further configured to perform a first classification based on the sign bit to determine whether the floating-point number is located in the interval of the non-negative number field or the interval of the non-positive number field. The floating-point number is determined based on the true value of the exponent to determine whether it is in an infinite interval, an intermediate interval, or an interval containing zero; When the floating-point number is in the range containing zero, the mantissa is right-shifted to adjust the order, and the index value and the second mantissa are calculated using the first mantissa sub-segment and the number of segments; When the floating-point number is in an infinite range, the constant term of the last segment of the middle interval with the largest endpoint is used as the constant term of the floating-point number, and the index value is calculated. When the floating-point number is in the middle range, the index value and the second mantissa sub-segment are calculated using the first mantissa sub-segment.
[0013] Optionally, the arithmetic unit is further configured to calculate the coefficients of the first and second terms in the corresponding quadratic function using signed integer multiplication to obtain the values of the first and second terms. Based on the second last digit sub-field, calculate the maximum number of decimal places, the number of decimal places for the first term, and the number of decimal places for the second term; Based on the maximum number of decimal places, the number of decimal places in the first term, and the number of decimal places in the second term, the values of the constant term, the first term, and the second term are shifted and added together to obtain the value of the quadratic polynomial.
[0014] Optionally, the post-processing output unit is used to determine whether the floating-point number is infinity or NOT based on the exponent and mantissa; If the floating-point number is infinity or NOT, assign a value to the output according to the form of the activation function; Determine whether the floating-point number falls within the linear interval of the activation function based on the sign bit and exponent; If the floating-point number falls within the linear interval of the activation function, the output value is equal to the floating-point number; If the floating-point number is not infinite or a non-number and does not fall within the linear interval of the activation function, the value of the quadratic polynomial is normalized to obtain the mantissa and the exponent of the output value. Output a floating-point number in format based on the mantissa and the exponent of the output value.
[0015] A second aspect of this application provides a fast calculation method for nonlinear activation functions based on coefficient lookup tables, applied to the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the first aspect. The method includes: The entire region of the activation function is divided into several intervals, and the non-infinite intervals within those intervals are further divided into several segments. For each segment, a quadratic function is used for approximation to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein, the quadratic function coefficients include the coefficients of the first term, the coefficients of the quadratic term, and the constant term; Feature extraction is performed on the input floating-point number to obtain the sign bit, exponent, and mantissa; The index value and the second mantissa subfield are calculated based on the sign bit, exponent, and mantissa. The corresponding quadratic function coefficients are obtained from the coefficient lookup table based on the index value. Based on the second tail number segment, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations, the value of the quadratic polynomial is obtained; The value of the quadratic polynomial in integer form is normalized and combined with the sign bit and exponent of the floating-point number to output the value in floating-point format.
[0016] The above method combines the lookup table method and the piecewise approximation method, which can significantly improve the calculation accuracy with a small reduction in calculation speed, and the calculation speed is better than the Taylor expansion method. It is suitable for fast calculation of nonlinear activation functions with high accuracy requirements.
[0017] As can be seen from the above technical solutions, this application provides a fast calculation device and method for nonlinear activation functions based on coefficient lookup tables. The device includes: an input selection unit, used to select the input floating-point number according to the activation function type; a coefficient lookup table unit, used to divide the entire activation function region into several intervals, and divide the non-infinite intervals in the intervals into several segments; approximating each segment using a quadratic function to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein, the quadratic function coefficients include linear coefficients, quadratic coefficients, and a constant term; a preprocessing unit, used to extract features from the input floating-point number to obtain the sign bit, exponent, and mantissa; an arithmetic unit, used to calculate the index value and the second mantissa sub-segment according to the sign bit, exponent, and mantissa; obtain the corresponding quadratic function coefficient in the coefficient lookup table according to the index value; and obtain the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficient, and fixed-point multiplication and addition operations; and a post-processing output unit, used to normalize the value of the quadratic polynomial in integer form and combine it with the sign bit and exponent of the floating-point number to output the value in floating-point format. This addresses the issue of slow computation speed of nonlinear activation functions. Attached Figure Description
[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a framework diagram of the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 2The image of the Sigmoid function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 3 The image of the Tanh function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 4 The image of the Softplus function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 5 The image of the Mish function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 6 This is an image of the Swish function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 7 This is a graph of the GELU function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application; Figure 8 This refers to the partial coefficient lookup table of the Sigmoid function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application. Figure 9 This refers to the coefficient lookup table for the Mish function in the fast calculation device for nonlinear activation functions based on coefficient lookup tables described in the embodiments of this application. Figure 10 This is a flowchart illustrating the fast calculation method for nonlinear activation functions based on coefficient lookup tables described in this application. Detailed Implementation
[0020] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application.
[0021] To address the issue of slow computation speed of nonlinear activation functions, see [link to relevant documentation]. Figure 1 This application provides a fast calculation device for nonlinear activation functions based on coefficient lookup tables, comprising: The input selection unit is used to select the input floating-point number based on the activation function type.
[0022] It should be understood that floating-point numbers can be IEEE 754 single-precision floating-point numbers; activation function types include: See Sigmoid function. Figure 2 The output value can be mapped to the range [0, 1], and the expression is: ; Tanh function, see Figure 3 The output value can be mapped to the range [-1, 1], and the expression is: ; See the Softplus function. Figure 4 It can be viewed as a smoothing of the ReLU function, expressed as: ; The Mish function, see Figure 5 This can be viewed as a smoothing of the ReLU function, expressed as: ; The Swish function, see Figure 6 The shape of the function can be adjusted by parameters, and the expression is: ; in, =1; GELU function, see Figure 7 The Gaussian error linear unit activation function is approximately expressed as: .
[0023] The above six activation functions are all non-linear activation functions. Since non-linear activation functions are all complex, they are used as examples in the device of this application.
[0024] In other embodiments, the device is also applicable to half-precision or double-precision floating-point numbers.
[0025] The coefficient lookup table unit is used to divide the entire activation function region into several intervals, and further divide the non-infinite intervals into several segments. A quadratic function is used to approximate each segment to establish a coefficient lookup table containing the optimal quadratic function coefficients. The quadratic function coefficients include linear coefficients, quadratic coefficients, and constant terms.
[0026] It should be understood that the interval division and segmentation are the same for all six nonlinear activation functions.
[0027] In some embodiments, the endpoints of the interval include 0, powers of 2 or their opposites, positive infinity, and negative infinity; When the interval is located in the non-negative number field, the left endpoint of the interval is closed and the right endpoint of the interval is open. When the interval is located in the non-positive number field, the left endpoint of the interval is open and the right endpoint of the interval is closed.
[0028] In some embodiments, the interval includes an infinite interval and a non-infinite interval; the non-infinite interval includes the intermediate interval and the interval containing zero.
[0029] (1) Intermediate interval: The activation function is divided into intervals according to the sign bit and the exponent. The sign bit and exponent of all floating-point numbers in the interval are the same, and the endpoints of the interval are powers of 2 or their opposites. For example, a single-precision floating-point number with a sign bit of 0 and an exponent of (01111111)2 can be divided into an interval, which is represented as [1, 2) after conversion to decimal; a floating-point number with a sign bit of 1 and an exponent of (01111111)2 can also be divided into an interval, which is represented as (-2, -1) after conversion to decimal. Similarly, intervals such as [0.125, 0.25), [0.5, 1), [2, 4), and (-8, -4] can also be divided.
[0030] (2) Zero-containing intervals: These are intervals near and containing 0, represented as (-|min1|, 0] and [0, min2), where |min1| is the node with the smallest absolute value in the negative number field, and min2 is the node with the smallest absolute value in the positive number field. Both |min1| and min2 are powers of 2. There is more than one floating-point exponent for these two intervals. Generally, |min1| = min2 is set to facilitate subsequent index value lookup. Zero-containing intervals can be represented as (-min, 0] and [0, min), where min > 0.
[0031] (3) Infinite intervals: Intervals extending to positive or negative infinity, namely (-∞, -|max1|] and [max2, +∞), where |max1| is the interval node with the largest absolute value in the negative number field, and max2 is the interval node with the largest absolute value in the positive number field, and both |max1| and max2 are powers of 2. There is more than one type of floating-point exponent for these two intervals. Generally, |max1| = max2 is set to facilitate subsequent index value lookup. Infinite intervals can be represented as (-|max|, 0] and [0, max), where max > 0.
[0032] When determining an infinite interval, the constant or linear part of the corresponding activation function should be preserved as much as possible, so that subsequent function fitting is unnecessary.
[0033] It should be understood that the intervals can be specifically divided into (-∞, -16], (-16, -8], (-8, -4], (-4, -2], (-2, -1], (-1, -0.5], (-0.5, -0.25], (-0.25, -0.125], (-0.125, 0], [0, 0.125), [0.125, 0.25), [0.25, 0.5), [0.5, 1), [1, 2), [2, 4), [4, 8), [8, 16), [16, +∞). Among them, the infinite intervals are (-∞, -16] and [16, +∞), the intervals containing zero are (-0.125, 0] and [0, 0.125), and the rest are intermediate intervals.
[0034] After dividing the intervals, further segmentation is needed. Subsequently, a quadratic function is used on each segment for optimal uniform approximation. Since infinite intervals do not require fitting, segmentation is only performed on the middle interval and the interval containing zero. The number of segments on an interval, size, should be a power of 2. Each segment has the same length, and generally, the number of segments is the same across all intervals (except infinite intervals). For example, the middle interval [1, 2) is divided into 8 segments: the first segment is [1, 1.125), the second segment is [1.125, 1.25), and so on. The interval containing zero, [0, 0.5) is divided into 8 segments: the first segment is [0, 0.0625), the second segment is [0.0625, 0.125), and so on.
[0035] When performing piecewise fitting of activation functions, the process is performed on only one interval at a time. The independent variable x is fixed in the interval [0, 1). The actual fitting interval is changed by shifting and stretching the function expression. For example, if the original function f(x) is fitted in the interval [0, 1), then f(x+1) is fitted in the interval [1, 2), f(-0.5×x) is fitted in the interval (-0.5, 0], and f(2×(x+1)) is fitted in the interval [2, 4). When fitting the intermediate interval, the general formula is f(a×(x+1)), where a is the coefficient; when fitting the interval containing zero, the general formula is f(±min×x).
[0036] Since the independent variable x is fixed in the interval [0, 1), the first fitting is performed on the first segment [0, 1 / size). In the second fitting, the function is shifted to the left by a distance of 1 / size; the independent variable x is still fitted on [0, 1 / size), but it is equivalent to fitting the second segment [1 / size, 2 / size). In the third fitting, the function is shifted to the left again by 1 / size, and it is equivalent to fitting the third segment [2 / size, 3 / size), and so on. It can be concluded that for any segment of any interval (except an infinite interval) of the activation function, the independent variable x is always on [0, 1 / size) when approximating the function. Note that fitting over non-positive domains starts from the right endpoint of the interval.
[0037] Therefore, the coefficient lookup table for an activation function has 384 entries. A portion of the coefficient lookup table can be found in [link to relevant documentation]. Figure 8 and Figure 9 .
[0038] The preprocessing unit is used to extract features from floating-point numbers to obtain the sign bit, exponent, and mantissa.
[0039] It should be understood that the exponent of a floating-point number is in offset binary form, and 127 needs to be subtracted to obtain its true value; when extracting the mantissa, the hidden 1 in the highest bit needs to be padded, and after padding, there are 24 bits, which is called the complete mantissa dx.
[0040] The sign bit, exponent, and mantissa are used for subsequent calculations of the index value in the coefficient lookup table and for the final post-processing steps.
[0041] In some embodiments, the intervals of the non-negative fields in the coefficient lookup table are arranged in ascending order; the intervals of the non-positive fields in the coefficient lookup table are arranged in descending order, and the array index of the arrangement of each interval in the coefficient lookup table is used as the index value.
[0042] In some embodiments, the coefficient lookup table unit is also used to determine the number of segments in a non-infinite interval and the number of decimal places for the quadratic function coefficients.
[0043] Let `size` be the number of segments in an interval, `t` be the number of decimal places for the constant term, `p` be the number of decimal places for the linear coefficient, and `q` be the number of decimal places for the quadratic coefficient. The purpose of `t`, `p`, and `q` is to truncate the calculated original coefficients according to these values during subsequent function approximation, to adapt to the bit depth requirements of different computational elements, or to control the fitting accuracy. A larger number of decimal places results in higher accuracy, but also increases computational cost. For example, we can set `t = 21` for the constant coefficient, `p = 12` for the linear coefficient, and `q = 10` for the quadratic coefficient.
[0044] Perform the best uniform approximation on the next or first segment, and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the coefficients of the linear term.
[0045] It should be understood that optimal uniform approximation is a function approximation method that minimizes the absolute value of the maximum error between the original function and the fitted polynomial. A strictly optimal uniform approximation polynomial is difficult to obtain; therefore, the Remez algorithm is often used iteratively. This optimal uniform approximation uses a quadratic polynomial. To reduce error, the coefficients of the quadratic function can be extracted step by step. After the first optimal uniform approximation, the coefficients of the original linear term are extracted first. And based on the p-value, the decimal places are truncated to obtain the coefficient C1 of the linear term.
[0046] The coefficient of the quadratic term is obtained by calculating the number of decimal places of the coefficient of the linear term, the coefficient of the quadratic function, and the number of digits in the first mantissa.
[0047] It should be understood that, in order to compensate for the shortcomings to a certain extent... The error arising during truncation can be compensated for by using the following formula when calculating the coefficient C2 of the quadratic term: ; in, These are the coefficients of the original quadratic term obtained from the first best uniform approximation.
[0048] Will C2 is obtained by truncating the decimal places according to the q value. The error caused by truncation is ( -C1)·x, while the corrected error is ( -C1)·x 2 • size: This formula can reduce the error, but it cannot eliminate the error.
[0049] We then perform the best uniform approximation again and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the constant term.
[0050] It should be understood that the error generated after truncating the coefficients of the first and second terms is already quite large. Therefore, another best uniform approximation is performed. Let the activation function be f(x), then the function to be approximated is f(x) - C2·x. 2 -C1·x, using a constant term for approximation, yields the original constant term C0. By truncating the decimal places according to the value of t, the constant term C0 is obtained.
[0051] Store the coefficients of the first term, the coefficients of the second term, and the constant term in hexadecimal form.
[0052] It should be understood that the constant term C0, the linear coefficient C1, and the quadratic coefficient C2 of this segment are stored in hexadecimal form in sequence. Since the position of the decimal point is fixed for each coefficient, the decimal point can be omitted and the data can be stored directly in integer form.
[0053] Generate a coefficient lookup table for non-infinite intervals based on the coefficients of the linear term, the quadratic term, and the constant term.
[0054] It should be understood that once all segments of an interval have been calculated, a coefficient lookup table for that interval is generated.
[0055] The concepts of the first and second last digit segments differ for the middle interval and the interval containing zero.
[0056] (1) The first and second mantissa segments of the intermediate interval: If the fitting is of the intermediate interval, then the 1 in the general formula represents the 1 implied in the highest bit of the IEEE 754 single-precision floating-point number, and the independent variable x represents the mantissa part. And the highest log2 (size) bit of the mantissa of x in each segment is always 0, and only the bits after it participate in the best uniform approximation operation of the function. The highest log2 (size) bit of the mantissa is called the first mantissa segment, and the k bits immediately following the first mantissa segment are called the second mantissa segment. The number of bits of the second mantissa segment can be varied to adapt to different specifications of multiplication and addition operation units.
[0057] The purpose of shifting the function to the left by 1 / size each time is to subtract the first mantissa segment to ensure that only the subsequent mantissas participate in the calculation. However, in reality, the first mantissa segment of the fitted segment is incremented by 1 each time. For example, when size=8, the first mantissa segment has 3 bits. When fitting the first segment, the first mantissa segment is (000)2. When fitting the second segment, the first mantissa segment is (001)2, and so on. When fitting the 8th segment, the first mantissa segment is (111)2. Since the first mantissa segment is only a part of the mantissa and is not related to the exponent, the value of the first mantissa segment is not related to the current interval, but only to the segment number. Therefore, the first mantissa segment can be used to calculate the index value.
[0058] (2) First and second mantissa segments in the zero-containing interval: If the fitting is of the zero-containing interval, then the general expression does not contain +1, and the independent variable x does not directly correspond to the mantissa part of the floating-point number. However, if it is represented in the format of a general binary fixed-point decimal, then the log2(size) bits after the decimal point of x correspond to the first mantissa segment, and the decimal bits after the first mantissa segment actually participate in the operation. Therefore, the k bits after the first mantissa segment can still be set as the second mantissa segment. To obtain the first and second mantissa segments from the floating-point format, it is necessary to first convert it to the fixed-point decimal format, that is, to perform a right shift and exponent alignment operation so that the true value of the exponent is log2(min), and then extract the first and second mantissa segments.
[0059] Since the number of segments is size 8, in some embodiments, the first mantissa sub-segment has 3 bits, and the second mantissa sub-segment has k=19 bits, that is, the last bit of the 23-bit mantissa of the single-precision floating-point number does not participate in the calculation.
[0060] The arithmetic unit is used to calculate the index value and the second mantissa sub-segment based on the sign bit, exponent, and mantissa; to obtain the corresponding quadratic function coefficients from the coefficient lookup table based on the index value; and to obtain the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations.
[0061] In some embodiments, the arithmetic unit is further configured to perform a first classification based on the sign bit to determine whether the floating-point number is located in the interval of the non-negative number field or the interval of the non-positive number field.
[0062] It should be understood that the arithmetic unit contains multipliers and adders. For the non-negative number field, the intervals can be arranged in ascending order, i.e., the interval [0, min) is first, and the interval [max / 2, max) is last. For the non-positive number field, the intervals can be arranged in descending order, i.e., the interval (-min, 0] is first, and the interval (-max, -max / 2] is last. Moreover, within each interval, the intervals are also arranged in descending order. For example, the first term in the interval (-min, 0] is the coefficient of (-min / size, 0). The index value is the array number of the coefficients. Since three consecutive coefficients are extracted each time, it is only necessary to determine the array number of the constant term C0. The index value of the linear term coefficient C1 is C0 number + 1, and the index value of the quadratic term coefficient C2 is C0 number + 2. When calculating the index value, the sign bit is first determined as 0 or 1 for the first classification.
[0063] Determine whether the floating-point number is in an infinite interval, a middle interval, or a zero-containing interval based on the exponent's true value: When the floating-point number is in a zero-containing interval, right-shift the mantissa to align the exponent, and use the first mantissa sub-segment and the number of segments to calculate the index value and the second mantissa sub-segment; When the floating-point number is in an infinite interval, use the constant term of the last segment of the middle interval with the largest endpoint as the constant term of the floating-point number, and calculate the index value; When the floating-point number is in a middle interval, use the first mantissa sub-segment to calculate the index value and the second mantissa sub-segment.
[0064] It should be understood that when calculating the index value and the second mantissa subfield based on the exponent, it is acceptable to set the sign bit to 0. The subsequent specific process is as follows: If the true value of the exponent is less than log2(min), then the corresponding interval contains zeros. The 24-bit complete mantissa dx needs to be right-shifted to align the exponent so that its true value equals log2(min). The 23rd bit of the aligned sequence (assuming the least significant bit is the 1st bit) and the following log2(size)-1 bits correspond to the first mantissa sub-segment, where size is the number of interval segments. The k bits following the first mantissa sub-segment correspond to the second mantissa sub-segment.
[0065] The index value can be calculated using the first tail segment. Let's assume the index of the C0 array containing the zero interval [0, min) is m. Then the index value of the currently input C0 is... ; in, This indicates that the first mantissa subarray of the binary number is converted to decimal.
[0066] After obtaining the C0 index value, the C1 and C2 index values can be further obtained.
[0067] If the exponent's true value is greater than or equal to log2(max), it corresponds to an infinite interval. No function approximation is needed; simply let C0 be the last segment of the interval [max / 2, max) with the largest endpoint. Further, C1 and C2 can be obtained. Note that if this infinite interval corresponds to the constant part of a non-linear activation function, such as the Sigmoid function which can be considered a constant 1 in the interval [8, +∞), then the quadratic polynomial value calculated using C0, C1, and C2 is correct because C1 and C2 are both 0, and the result is independent of the input. However, if this infinite interval corresponds to the linear part of a non-linear activation function, such as the Mish function which can be considered a linear function y=x in the interval [8, +∞), then the binomial value calculated using C0, C1, and C2 is incorrect. Therefore, special assignment is needed in the post-processing steps, without referring to the value of the quadratic polynomial.
[0068] If the true value of the exponent is greater than or equal to log2(min) and less than log2(max), then it corresponds to the middle interval and no shifting is required. The 23rd bit of the complete mantissa dx (assuming the least significant bit of the complete mantissa dx is the 1st bit) and the following log2(size)-1 bits correspond to the first mantissa sub-segment, where size is the number of interval segments; the k bits after the first mantissa sub-segment correspond to the second mantissa sub-segment.
[0069] The index value can be calculated using the first mantissa segment. Let's assume the true value of the input exponent is exp, and the index of the C0 array containing the first segment of data in the zero interval [0, min) is m. Then the current input C0 index value is: ; in, This indicates that the first mantissa subarray of the binary number is converted to decimal.
[0070] After obtaining the C0 index value, the C1 and C2 index values can be further obtained.
[0071] If the sign bit is 1, m needs to be changed to the C0 array index of the first segment of data in the zero interval (-min, 0]. If the exponent true value is greater than or equal to log2(max), C0 is directly set to the C0 of the last segment in the middle interval (-max, max / 2] with the smallest endpoint. The rest of the process remains unchanged.
[0072] In a specific embodiment, the C0 array index of the first segment of data containing the zero interval (-0.125, 0] is 0, the C0 array index of the first segment of data containing the zero interval [0, 0.125) is 192, the C0 array index of the last segment in the middle interval (-16, -8] with the smallest endpoint value is 189, and the C0 array index of the last segment in the middle interval [8, 16] with the largest endpoint value is 381. Substituting these values into the aforementioned formula yields the formula for calculating the index value.
[0073] In some embodiments, the arithmetic unit is further configured to calculate the coefficients of the first and second terms in the coefficients of the corresponding quadratic function using signed integer multiplication, so as to obtain the values of the first and second terms.
[0074] It should be understood that the constant term equal to C0 does not need to be calculated.
[0075] Based on the second decimal segment, calculate the maximum number of decimal places, the number of decimal places for the first term, and the number of decimal places for the second term.
[0076] It should be understood that the maximum number of decimal places needs to be recorded before adding the terms to determine how to shift the terms during addition and ensure the correctness of the result. The decimal places of the constant term (bits0 = t) are: The decimal places of the linear term (bits1) are: ; Where k is the number of bits in the second mantissa sub-array. The number of bits in the first tail number sub-field.
[0077] The number of decimal places (bits2) for the quadratic term is: The maximum number of decimal places, max_bits, is the maximum value of bits0, bits1, and bits2.
[0078] Based on the maximum number of decimal places, the number of decimal places in the linear term, and the number of decimal places in the quadratic term, the constant term, the linear term, and the quadratic term are shifted and added together to obtain the value of the quadratic polynomial.
[0079] It should be understood that the left shift of the constant term is max_bits-bits0, the left shift of the linear term is max_bits-bits1, and the left shift of the quadratic term is max_bits-bits2. The value of the quadratic polynomial, sum, is obtained by adding the signed integers after shifting each term to the left.
[0080] Since the sign of the output value of mainstream activation functions is very easy to determine, for example, for activation functions such as Tanh and Mish, the output is greater than 0 when the input is greater than 0 and less than 0 when the input is less than 0. For example, the Sigmoid function is always greater than 0. Therefore, the absolute value of sum can be taken and the sign bit can be directly assigned in the post-processing step.
[0081] The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine it with the sign bit and exponent of the floating-point number so that the output value is a floating-point number.
[0082] In some embodiments, the post-processing output unit is used to determine whether the floating-point number is infinity or NOT based on the exponent and mantissa.
[0083] If the floating-point number is infinity or NOT, assign a value to the output according to the activation function form.
[0084] Specifically, if the exponent of the input floating-point number is all 1s and the mantissa is all 0s, then it represents infinity; if the exponent of the input floating-point number is all 1s and the mantissa is not all 0s, then it represents NOT. In this case, the output can be directly assigned according to the specific activation function form. For example, for the Sigmoid function, when the input is positive infinity, the output is 1, that is, the sign bit is 0, the exponent is (01111111)2, and the mantissa is all 0s; when the input is negative infinity, the output is 0, and the sign bit, exponent, and mantissa are all 0.
[0085] Determine whether a floating-point number falls within the linear interval of the activation function based on the sign bit and exponent.
[0086] If the floating-point number falls within the linear interval of the activation function, the output value is equal to the floating-point number.
[0087] Specifically, for activation functions such as Mish, Swish, and GELU, these functions tend to be linear functions y=x on the infinite interval [max, +∞). If the sign bit of the input floating-point number is 0 and the true value of the exponent is greater than or equal to log2(max), then the output floating-point number is directly set to be equal to the input floating-point number.
[0088] If the floating-point number is not infinite or a NOT number and does not fall within the linear interval of the activation function, the value of the quadratic polynomial is normalized to obtain the mantissa and the exponent of the output value.
[0089] Specifically, if neither of the above two cases applies, then `sum` needs to be normalized to calculate the mantissa and output the exponent of the floating-point number. Let's assume `sum` is a 64-bit integer, and that its absolute value has already been calculated during the quadratic polynomial calculation, so `sum` is always greater than or equal to 0. Starting from the most significant bit of `sum` (let's call the least significant bit the 1st bit), iterate through the bits one by one. If the current bit is 1, output its index `i`, and output that bit and the following 23 bits of data. If there are fewer than 23 bits after the current bit, pad with zeros. If `sum` has no digits that are 1, output `i` as -1, indicating a result of 0. If `i` is -1, the output value is equal to the output value of the activation function when the input is 0. This is generally a simple rational number; for example, the Sigmoid function outputs 0.5 when the input is 0, and functions like Tanh and Mish output 0 when the input is 0. This can be directly converted to floating-point form. If zero is not 1, then remove the highest 1 from the found 24-bit data, and the resulting 23-bit data is the mantissa of the output floating-point number. The exponent res_exp in the biased form of the output floating-point number can be calculated using the following formula: ; Among them, max_bits is the maximum number of bits.
[0090] Output a floating-point number in format based on the mantissa and the exponent of the output value.
[0091] Specifically, the sign bit of the floating-point number output by the Sigmoid function is always 0, while the sign bit of the floating-point number output by activation functions such as Tanh, Mish, and Swish is equal to the sign bit of the input floating-point number.
[0092] This application also provides a method for fast calculation of nonlinear activation functions based on coefficient lookup tables in some embodiments, which is applied to the fast calculation device for nonlinear activation functions based on coefficient lookup tables in the above embodiments. See [link to relevant documentation]. Figure 10 The methods include: S100: Divide the entire activation function region into several intervals, and divide the non-infinite intervals into several segments.
[0093] S200: Approximate each segment using a quadratic function to establish a coefficient lookup table containing the optimal quadratic function coefficients.
[0094] The coefficients of a quadratic function include the coefficients of the linear term, the coefficients of the quadratic term, and the constant term. S300: Extract features from the input floating-point number to obtain the sign bit, exponent, and mantissa.
[0095] S400: Calculate the index value and the second mantissa subfield based on the sign bit, exponent, and mantissa.
[0096] S500: Retrieve the corresponding quadratic function coefficients from the coefficient lookup table based on the index value.
[0097] S600: Based on the second mantissa subarray, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations, the value of the quadratic polynomial is obtained.
[0098] S700: Normalizes the value of a quadratic polynomial in integer form and combines it with the sign bit and exponent of the floating-point number to output the value in floating-point format.
[0099] It should be understood that in neural networks, activation functions map the input of neurons to their output. Without activation functions, a neural network is merely a linear model, unable to learn nonlinear relationships in the data. Activation functions, by introducing nonlinearity, endow neural networks with stronger learning and expressive capabilities, enabling them to handle more complex problems. Therefore, the choice of activation function is crucial to the performance of a neural network model; a suitable activation function can effectively improve the model's robustness and output accuracy. Moreover, most nonlinear activation functions exhibit nonlinearity only within a finite interval near 0, and when the absolute value of the input is large, the output is a constant or linearly related to the input. Therefore, fast computation can be achieved by simply fitting the nonlinear part. However, the lookup table method subdivides the activation function into many segments, with each segment containing a fixed activation function value, and stores all values in a lookup table. For a specific input, the lookup table index is calculated according to certain rules, and the corresponding value is retrieved to obtain the output. This method requires almost no computation, making it the fastest. Furthermore, its segmentation method is independent of the activation function form, offering good versatility. However, achieving high accuracy requires significant storage resources. Piecewise approximation involves intentionally dividing the activation function into a few segments, approximating each segment using a linear function or a polynomial of higher degree. This method incurs no storage overhead, but due to the limited number of segments, a separate segmentation method needs to be defined for each activation function. Furthermore, because each segment is relatively long, approximation using a linear function generally results in low accuracy, while using a high-degree polynomial increases computational cost with the polynomial degree, leading to some real-time performance limitations.
[0100] The above method integrates the lookup table method and the piecewise approximation method. It not only inherits the advantages of the conventional lookup table method's versatility but also consumes less storage resources under the same accuracy requirements. Furthermore, this method has a smaller computational load, and its computation speed is only slightly slower than the conventional lookup table method, outperforming the piecewise high-order polynomial approximation method. It achieves a good trade-off between storage resources and computation speed, ensuring high accuracy, and its computation speed is superior to the Taylor expansion method, making it suitable for applications requiring fast computation of nonlinear activation functions with high accuracy requirements.
[0101] Based on the above method, the maximum absolute error in the interval containing zero and the intermediate interval, as well as the maximum absolute error in the entire number field, were statistically analyzed, as shown in the table below: Table 1. Error Statistics of Sigmoid Function Table 2. Error statistics of Tanh function Table 3 Softplus function error statistics Table 4. Error Statistics of the Mish Function Table 5. Error Statistics of Swish Function Table 6. Error Statistics of GELU Function As shown in the tables, this method achieves high accuracy in fast activation function calculation, with a maximum error on the order of 10⁻⁵, significantly outperforming the lookup table method and the piecewise approximation method. Furthermore, since it only requires calculating a quadratic polynomial, its computational speed is superior to the Taylor expansion method.
[0102] As can be seen from the above technical solutions, this application provides a fast calculation device and method for nonlinear activation functions based on coefficient lookup tables. The device includes: an input selection unit for selecting the input floating-point number according to the activation function type; a coefficient lookup table unit for dividing the entire activation function region into several intervals and dividing the non-infinite intervals into several segments; approximating each segment using a quadratic function to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein the quadratic function coefficients include linear coefficients, quadratic coefficients, and a constant term; a preprocessing unit for extracting features from the input floating-point number to obtain the sign bit, exponent, and mantissa; an arithmetic unit for calculating the index value and the second mantissa sub-segment based on the sign bit, exponent, and mantissa; obtaining the corresponding quadratic function coefficient in the coefficient lookup table based on the index value; and obtaining the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficient, and fixed-point multiplication and addition operations; and a post-processing output unit for normalizing the value of the quadratic polynomial in integer form and combining it with the sign bit and exponent of the floating-point number to output the value in floating-point format. This addresses the issue of slow computation speed of nonlinear activation functions.
[0103] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.
Claims
1. A fast calculation device for nonlinear activation functions based on coefficient lookup tables, characterized in that, include: The input selection unit is used to select the input floating-point number based on the activation function type; The coefficient lookup table unit is used to divide the entire region of the activation function into several intervals, and to divide the non-infinite intervals in the intervals into several segments; to approximate each segment using a quadratic function, in order to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein, the quadratic function coefficients include linear coefficients, quadratic coefficients, and constant terms; A preprocessing unit is used to extract features from the floating-point number to obtain the sign bit, exponent, and mantissa; The arithmetic unit is used to calculate an index value and a second mantissa sub-segment based on the sign bit, exponent, and mantissa; to obtain the corresponding quadratic function coefficients from the coefficient lookup table based on the index value; and to obtain the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations. The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine it with the sign bit and exponent of the floating-point number to output the value in floating-point format. The endpoints of the interval include 0, powers of 2 or their opposites, positive infinity, and negative infinity; When the interval is located in the non-negative number field, the left endpoint of the interval is closed and the right endpoint of the interval is open; When the interval is located in a non-positive number field, the left endpoint of the interval is open and the right endpoint of the interval is closed. The arithmetic unit is also used to perform a first classification based on the sign bit to determine whether the floating-point number is located in the interval of the non-negative number field or the interval of the non-positive number field. The floating-point number is determined based on the true value of the exponent to determine whether it is in an infinite interval, an intermediate interval, or an interval containing zero; When the floating-point number is in the range containing zero, the mantissa is right-shifted to adjust the order, and the index value and the second mantissa are calculated using the first mantissa sub-segment and the number of segments; When the floating-point number is in an infinite range, the constant term of the last segment of the middle interval with the largest endpoint is used as the constant term of the floating-point number, and the index value is calculated. When the floating-point number is in the middle range, the index value and the second mantissa are calculated using the first mantissa sub-segment; The arithmetic unit is also used to calculate the coefficients of the first term and the coefficients of the second term in the corresponding quadratic function using signed integer multiplication, so as to obtain the values of the first term and the second term. Based on the second last digit sub-field, calculate the maximum number of decimal places, the number of decimal places for the first term, and the number of decimal places for the second term; Based on the maximum number of decimal places, the number of decimal places in the first term, and the number of decimal places in the second term, the values of the constant term, the first term, and the second term are shifted and added together to obtain the value of the quadratic polynomial.
2. The fast calculation device for nonlinear activation functions based on coefficient lookup tables according to claim 1, characterized in that, The coefficient lookup table unit is also used to determine the number of segments in the non-infinite interval and the number of decimal places in the coefficients of the quadratic function; Perform the best uniform approximation on the next segment or the first segment, and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the coefficients of the linear term; The quadratic coefficient is calculated based on the coefficient of the linear term, the number of decimal places of the coefficient of the quadratic function, and the number of digits in the first mantissa sub-segment. Perform the best uniform approximation again, and truncate the coefficients of the quadratic function according to the number of decimal places to obtain the constant term; The coefficients of the first term, the coefficients of the second term, and the constant term are stored in hexadecimal form. Based on the coefficients of the first term, the coefficients of the second term, and the constant term, a coefficient lookup table for the non-infinite interval is generated.
3. The fast calculation device for nonlinear activation functions based on coefficient lookup tables according to claim 1, characterized in that, The floating-point number is an IEEE 754 single-precision floating-point number.
4. The fast calculation device for nonlinear activation functions based on coefficient lookup tables according to claim 1, characterized in that, The interval includes infinite intervals and non-infinite intervals; the non-infinite intervals include intermediate intervals and intervals containing zero.
5. The fast calculation device for nonlinear activation functions based on coefficient lookup tables according to claim 4, characterized in that, The intervals of the non-negative number field in the coefficient lookup table are arranged in ascending order; the intervals of the non-positive number field in the coefficient lookup table are arranged in descending order, and the array index of the arrangement of each interval in the coefficient lookup table is used as the index value.
6. The fast calculation device for nonlinear activation functions based on coefficient lookup tables according to claim 1, characterized in that, The post-processing output unit is used to determine whether the floating-point number is infinity or not a number based on the exponent and mantissa. If the floating-point number is infinity or NOT, assign a value to the output according to the form of the activation function; Determine whether the floating-point number falls within the linear interval of the activation function based on the sign bit and exponent; If the floating-point number falls within the linear interval of the activation function, the output value is equal to the floating-point number; If the floating-point number is not infinite or a non-number and does not fall within the linear interval of the activation function, the value of the quadratic polynomial is normalized to obtain the mantissa and the exponent of the output value. Output a floating-point number in format based on the mantissa and the exponent of the output value.
7. A fast calculation method for nonlinear activation functions based on coefficient lookup tables, characterized in that, The method applied to the fast calculation device for nonlinear activation functions based on coefficient lookup tables according to any one of claims 1-6, the method comprising: The entire region of the activation function is divided into several intervals, and the non-infinite intervals within those intervals are further divided into several segments. For each segment, a quadratic function is used for approximation to establish a coefficient lookup table containing the optimal quadratic function coefficients; wherein, the quadratic function coefficients include the coefficients of the first term, the coefficients of the quadratic term, and the constant term; Feature extraction is performed on the input floating-point number to obtain the sign bit, exponent, and mantissa; The index value and the second mantissa subfield are calculated based on the sign bit, exponent, and mantissa. The corresponding quadratic function coefficients are obtained from the coefficient lookup table based on the index value. Based on the second tail number segment, the corresponding quadratic function coefficients, and fixed-point multiplication and addition operations, the value of the quadratic polynomial is obtained; The value of the quadratic polynomial in integer form is normalized and combined with the sign bit and exponent of the floating-point number to output the value in floating-point format.