Non-linear activation function rapid calculation device and method based on coefficient lookup table

Abstract

The invention provides a nonlinear activation function rapid calculation device and method based on a coefficient lookup table, and the device comprises an input selection unit which is used for selecting an input floating-point number; the coefficient lookup table unit is used for dividing the whole region of the activation function into a plurality of intervals and dividing a non-infinite interval into a plurality of sections; carrying out approximation on each section by using a quadratic function so as to establish a coefficient lookup table; the preprocessing unit is used for performing feature extraction on the floating-point number to obtain a sign bit, an order code and a mantissa; the operation unit is used for calculating an index value and a second mantissa subsegment; obtaining a corresponding quadratic function coefficient in a coefficient lookup table according to the index value; obtaining a value of a quadratic polynomial based on the second mantissa sub-segment, a corresponding quadratic function coefficient and fixed-point number multiplication and addition operation; and the post-processing output unit is used for normalizing the value of the quadratic polynomial and outputting a value in a floating-point number format in combination with the sign bit and the order code of the floating-point number, so that the problem that the calculation speed of the activation function is relatively low is solved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a device and method for fast calculation of a nonlinear activation function based on a coefficient lookup table. Background Art

[0002] With the rise of deep learning, the number of neural network layers continues to increase, the parameter scale is getting larger and larger, and a large amount of computing resources are also required when training models. As the basic operation of each neuron in the neural network, the computational efficiency of the activation function directly affects the speed of model training. Especially for complex nonlinear activation functions such as Sigmoid, Tanh, Mish, etc., the computational overhead is particularly huge, which seriously restricts the real-time performance of the neural network.

[0003] In the related technology, CORDIC algorithm and Taylor expansion method can be used. CORDIC algorithm is an algorithm that uses a series of rotation and iterative operations to approximate mathematical functions. It only has simple addition, subtraction and shift operations, which is very suitable for hardware implementation. However, this algorithm is mainly used to calculate trigonometric and hyperbolic functions, and its versatility is poor. It also requires multiple iterations to achieve high precision, which is slow. Taylor expansion method uses Taylor formula to expand the activation function into a power series for approximation, but this method generally needs to be expanded to the fourth order or even higher to achieve high precision, and the multiplication operation overhead is huge, which cannot solve the problem of slow calculation speed of nonlinear activation functions. Summary of the invention

[0004] The present application provides a device and method for fast calculation of a nonlinear activation function based on a coefficient lookup table to solve the problem of slow calculation speed of the nonlinear activation function.

[0005] The first aspect of the present application provides a fast calculation device for a nonlinear activation function based on a coefficient lookup table, comprising:

[0006] An input selection unit, used to select an input floating point number according to the activation function type;

[0007] A coefficient lookup table unit is used to divide the entire area of ​​the activation function into a number of intervals, and divide the non-infinite interval in the interval into a number of segments; use a quadratic function to approximate each segment to establish a coefficient lookup table containing the best quadratic function coefficients; wherein the quadratic function coefficients include a linear term coefficient, a quadratic term coefficient, and a constant term;

[0008] A preprocessing unit, used for performing feature extraction on the floating point number to obtain a sign bit, an exponent and a mantissa;

[0009] an operation unit, configured to calculate an index value and a second mantissa subsegment according to the sign bit, the exponent and the mantissa; obtain a corresponding quadratic function coefficient in the coefficient lookup table according to the index value; and obtain a value of a quadratic polynomial based on the second mantissa subsegment, the corresponding quadratic function coefficient and a fixed-point multiplication and addition operation;

[0010] The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine the sign bit and the exponent of the floating point number to output the value in floating point format.

[0011] The above device can convert the calculation of the nonlinear activation function into a simple quadratic function multiplication and addition operation through a coefficient lookup table, thereby effectively improving the calculation speed of the nonlinear activation function and solving the problem of slow calculation speed of the nonlinear activation function.

[0012] Optionally, the endpoints of the interval include 0, a power of 2 or a negative number of a power of 2, positive infinity and negative infinity;

[0013] When the interval is in the non-negative number domain, the left endpoint of the interval is closed and the right endpoint of the interval is open;

[0014] When the interval is located in a non-positive number domain, the left endpoint of the interval is open and the right endpoint of the interval is closed.

[0015] Optionally, the coefficient lookup table unit is further used to determine the number of segments of the non-infinite interval and the number of decimal places of the quadratic function coefficient;

[0016] Performing the best consistent approximation on the next segment or the first segment and truncating the coefficient of the quadratic function according to the number of decimal places of the coefficient to obtain the coefficient of the first term;

[0017] Calculate according to the coefficient of the linear term, the number of decimal places of the coefficient of the quadratic function and the number of places of the first mantissa subsegment to obtain the coefficient of the quadratic term;

[0018] Performing the best consistent approximation again and truncating the coefficient according to the number of decimal places of the quadratic function coefficient to obtain a constant term;

[0019] Saving the linear term coefficient, the quadratic term coefficient and the constant term in hexadecimal form;

[0020] A coefficient lookup table of the non-infinite interval is generated according to the linear term coefficient, the quadratic term coefficient and the constant term.

[0021] Optionally, the floating point number is an IEEE754 single-precision floating point number.

[0022] Optionally, the interval includes an infinite interval and a non-infinite interval; the non-infinite interval includes an intermediate interval and an interval containing zero.

[0023] Optionally, the intervals of the non-negative number domain in the coefficient lookup table are arranged in order from small to large; the intervals of the non-positive number domain in the coefficient lookup table are arranged from large to small, and the array sequence number of the arrangement of each interval in the coefficient lookup table is used as the index value.

[0024] Optionally, the operation unit is further used to perform a first classification according to the symbol class to confirm whether the floating point number is located in an interval of the non-negative number domain or an interval of the non-positive number domain;

[0025] Determine, based on the true number of the exponent, whether the floating point number is in an infinite interval, an intermediate interval, or an interval containing zero;

[0026] When the floating point number is in a zero-containing interval, the mantissa is right-shifted to the right order, and the index value and the second mantissa subsegment are calculated using the first mantissa subsegment and the segment number;

[0027] When the floating-point number is in an infinite interval, the constant term of the last segment of the middle interval with the largest endpoint is used as the constant term of the floating-point number to calculate the index value;

[0028] When the floating point number is in the middle interval, the index value and the second mantissa subsegment are calculated using the first mantissa subsegment.

[0029] Optionally, the operation unit is further used to calculate the linear term coefficient and the quadratic term coefficient in the corresponding quadratic function coefficient by using signed integer multiplication to obtain the linear term value and the quadratic term value;

[0030] According to the second mantissa subsegment, the maximum number of decimal places, the number of decimal places of the first term and the number of decimal places of the second term are calculated;

[0031] According to the maximum number of decimal places, the number of decimal places of the linear term and the number of decimal places of the quadratic term, the constant term, the linear term value and the quadratic term value are shifted and added to obtain the value of the quadratic polynomial.

[0032] Optionally, the post-processing output unit is used to determine whether the floating point number is infinite or non-number according to the exponent and the mantissa;

[0033] If the floating point number is infinite or non-number, assign a value to the output according to the activation function form;

[0034] Determine whether the floating point number falls within the linear interval of the activation function according to the sign bit and the exponent;

[0035] If the floating point number falls within the linear interval of the activation function, the output value is equal to the floating point number;

[0036] If the floating point number is not infinite or non-number and does not fall within the linear interval of the activation function, normalize the value of the quadratic polynomial to obtain the mantissa and the order of the output value;

[0037] Outputs a floating-point value based on the mantissa and the exponent of the output value.

[0038] The second aspect of the present application provides a method for fast calculation of a nonlinear activation function based on a coefficient lookup table, which is applied to the device for fast calculation of a nonlinear activation function based on a coefficient lookup table described in the first aspect, and the method comprises:

[0039] Dividing the entire activation function region into a number of intervals, and dividing a non-infinite interval in the interval into a number of segments;

[0040] Using a quadratic function to approximate each segment to establish a coefficient lookup table containing optimal quadratic function coefficients; wherein the quadratic function coefficients include a linear term coefficient, a quadratic term coefficient, and a constant term;

[0041] Perform feature extraction on the input floating point number to obtain the sign bit, exponent and mantissa;

[0042] Calculate the index value and the second mantissa subsegment according to the sign bit, the exponent and the mantissa;

[0043] Obtaining a corresponding quadratic function coefficient in the coefficient lookup table according to the index value;

[0044] deriving a value of a quadratic polynomial based on the second mantissa subsegment, the corresponding quadratic function coefficient, and a fixed-point multiplication and addition operation;

[0045] The value of the quadratic polynomial in integer form is normalized and combined with the sign bit and the exponent of the floating point number to output a value in floating point format.

[0046] The above method combines the lookup table method and the piecewise approximation method, which can significantly improve the calculation accuracy with a small reduction in calculation speed. The calculation speed is better than the Taylor expansion method and is suitable for fast calculation of nonlinear activation functions with high requirements on accuracy.

[0047] It can be seen from the above technical scheme that the present application provides a device and method for fast calculation of a nonlinear activation function based on a coefficient lookup table, the device comprising: an input selection unit, used to select an input floating-point number according to the type of activation function; a coefficient lookup table unit, used to divide the entire area of ​​the activation function into several intervals, and divide the non-infinite intervals in the intervals into several segments; using a quadratic function for approximation on each segment to establish a coefficient lookup table containing the best quadratic function coefficients; wherein the quadratic function coefficients include linear term coefficients, quadratic term coefficients and constant terms; a preprocessing unit, used to extract features from the input floating-point number to obtain a sign bit, an exponent and a mantissa; an operation unit, used to calculate an index value and a second mantissa sub-segment according to the sign bit, the exponent and the mantissa; obtaining the corresponding quadratic function coefficient in the coefficient lookup table according to the index value; obtaining the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficient and the fixed-point number multiplication and addition operation; a post-processing output unit, used to normalize the value of the quadratic polynomial in integer form, and combine the sign bit and the exponent of the floating-point number to output the value in floating-point format. Solve the problem of slow calculation speed of non-linear activation functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A framework diagram of a nonlinear activation function fast calculation device based on a coefficient lookup table according to an embodiment of the present application;

[0050] Figure 2 It is the Sigmoid function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0051] Figure 3 It is the Tanh function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0052] Figure 4 It is the Softplus function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0053] Figure 5 It is the Mish function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0054] Figure 6It is the Swish function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0055] Figure 7 It is the GELU function image in the nonlinear activation function fast calculation device based on coefficient lookup table described in the embodiment of the present application;

[0056] Figure 8 It is a partial coefficient lookup table of the Sigmoid function in the nonlinear activation function fast calculation device based on the coefficient lookup table described in the embodiment of the present application,

[0057] Fig. 9 It is a coefficient lookup table of the Mish function part in the nonlinear activation function fast calculation device based on the coefficient lookup table described in the embodiment of the present application;

[0058] Fig.10 A schematic diagram of a flow chart of a method for fast calculation of a nonlinear activation function based on a coefficient lookup table according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following embodiments do not represent all implementation methods consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application.

[0060] To solve the problem of slow calculation speed of nonlinear activation functions, see Figure 1 Some embodiments of the present application provide a fast calculation device for a nonlinear activation function based on a coefficient lookup table, including:

[0061] The input selection unit is used to select the input floating point number according to the activation function type.

[0062] It should be understood that the floating point number may be an IEEE754 single precision floating point number; the activation function types include:

[0063] Sigmoid function, see Figure 2 , the output value can be mapped to [0, 1], the expression is:

[0064]

[0065] Tanh function, see Figure 3 , the output value can be mapped to [-1, 1], the expression is:

[0066]

[0067] Softplus function, see Figure 4 , which can be regarded as the smoothing of the ReLU function, is expressed as:

[0068] f(x)=ln(1+e x );

[0069] Mish function, see Figure 5 , which can be regarded as the smoothing of the ReLU function, and the expression is:

[0070] f(x) = x·tanh(ln(1+e x ));

[0071] Swish function, see Figure 6 , the function shape can be adjusted by parameters, the expression is:

[0072] f(x) = x·Sigmoid(βx);

[0073] Among them, β is 1;

[0074] GELU function, see Figure 7 , Gaussian error linear unit activation function, the approximate expression is:

[0075]

[0076] The above six activation functions are all nonlinear activation functions. Since nonlinear activation functions are complex, they are suitable for use as examples in the device of the present application.

[0077] In some other embodiments, the apparatus is also applicable to half-precision or double-precision floating point numbers.

[0078] The coefficient lookup table unit is used to divide the entire area of ​​the activation function into a number of intervals, and divide the non-infinite intervals in the intervals into a number of segments; a quadratic function is used for approximation on each segment to establish a coefficient lookup table containing the best quadratic function coefficients; wherein the quadratic function coefficients include linear term coefficients, quadratic term coefficients and constant terms.

[0079] It should be understood that the interval division and segmentation of the six nonlinear activation functions are the same.

[0080] In some embodiments, the endpoints of the interval include 0, a power of 2 or the opposite of a power of 2, positive infinity, and negative infinity;

[0081] When the interval is in the non-negative domain, the left endpoint of the interval is closed and the right endpoint of the interval is open;

[0082] When the interval is located in a non-positive number domain, the left endpoint of the interval is open and the right endpoint of the interval is closed.

[0083] In some embodiments, the interval includes an infinite interval and a non-infinite interval; the non-infinite interval includes an intermediate interval and a zero-containing interval.

[0084] (1) Intermediate interval: The activation function is divided into intervals according to the sign bit and the exponent. The sign bit and exponent of all floating-point numbers in the interval are the same, and the interval endpoints are all powers of 2 or their opposites. For example, a single-precision floating-point number with a sign bit of 0 and an exponent of (01111111)2 can be divided into an interval, which is converted to decimal and represents [1, 2); a floating-point number with a sign bit of 1 and an exponent of (01111111)2 can also be divided into an interval, which is converted to decimal and represents (-2, -1). Similarly, it can be divided into intervals such as [0.125, 0.25), [0.5, 1), [2, 4), (-8, -4], etc.

[0085] (2) Zero-containing interval: that is, the interval near 0 and including 0, represented by (-|min1|, 0] and [0, min2), where |min1| is the interval node with the smallest absolute value in the negative domain, min2 is the interval node with the smallest absolute value in the positive domain, and |min1| and min2 are both powers of 2. There is more than one floating point exponent corresponding to these two intervals. Generally, |min1|=min2 is set to facilitate the subsequent search for index values. The zero-containing interval can be represented by (-min, 0] and [0, min), min>0.

[0086] (3) Infinite interval: An interval extending to positive infinity or negative infinity, i.e. (-∞, -|max1|] and [max2, +∞), where |max1| is the interval node with the largest absolute value in the negative domain, max2 is the interval node with the largest absolute value in the positive domain, and both |max1| and max2 are powers of 2. There is more than one floating point exponent corresponding to these two intervals. Generally, |max1|=max2 is set to facilitate subsequent search for index values. Infinite intervals can be represented as (-|max|, 0] and [0, max), with max>0.

[0087] When determining the infinite interval, the constant part or linear part of the corresponding activation function should be guaranteed as much as possible, and there is no need for subsequent function fitting.

[0088] It should be understood that the interval can be specifically divided into (-∞, -16], (-16, -8], (-8, -4], (-4, -2], (-2, -1], (-1, -0.5], (-0.5, -0.25], (-0.25, -0.125], (-0.125, 0], [0, 0.125), [0.125, 0.25), [0.25, 0.5), [0.5, 1), [1, 2), [2, 4), [4, 8), [8, 16), [16, +∞). Among them, the infinite interval is (-∞, -16] and [16, +∞), the zero-containing interval is (-0.125, 0] and [0, 0.125), and the rest are intermediate intervals.

[0089] After the interval is divided, it is necessary to further divide it into segments, and then use quadratic functions on each segment for the best consistent approximation. Since infinite intervals do not need to be fitted, they are only segmented on the middle interval and the interval containing zeros. The number of segments on an interval should be a power of 2, and the length of each segment is the same, and generally all intervals (except infinite intervals) have the same number of segments. For example, the middle interval [1, 2) is divided into 8 segments, the first segment is [1, 1.125), the second segment is [1.125, 1.25), and so on; the zero-containing interval [0, 0.5) is divided into 8 segments, the first segment is [0, 0.0625), the second segment is [0.0625, 0.125), and so on.

[0090] When performing piecewise fitting of the activation function, it is performed only on one interval at a time. The independent variable x is fixed in the interval [0, 1). The actual fitting interval is changed by shifting and stretching the function expression. For example, if the original function f(x) is fitted on the interval [0, 1), then f(x+1) is fitted on the interval [1, 2), f(-0.5×x) is fitted on the interval (-0.5, 0), and f(2×(x+1)) is fitted on the interval [2, 4). When fitting the intermediate interval, the general expression is f(a×(x+1)), where a is the coefficient; when fitting the interval containing zero, the general expression is f(±min×x).

[0091] Since the independent variable x is fixed on the interval [0, 1), the first fitting is performed on the first segment [0, 1 / size). In the second fitting, the function is shifted to the left by a distance of 1 / size, and the independent variable x is still fitted on [0, 1 / size), but the equivalent fitting is the second segment [1 / size, 2 / size). In the third fitting, the function is shifted to the left by 1 / size again, and the equivalent fitting is the third segment [2 / size, 3 / size), and so on. It can be concluded that for any segment of any interval of the activation function (except the infinite interval), the independent variable x is always on [0, 1 / size) when the function is approximated. Note that the fitting on the non-positive domain starts from the right endpoint of the interval.

[0092] Therefore, the coefficient lookup table of an activation function has a total of 384 items. The coefficient lookup table can be found in Figure 8 and Fig. 9 .

[0093] The preprocessing unit is used to extract features of floating point numbers to obtain the sign bit, exponent and mantissa.

[0094] It should be understood that the exponent of a floating-point number is in the form of a shift code, and 127 needs to be subtracted to obtain its true value; when extracting the mantissa, the hidden 1 in the highest bit needs to be completed, and after completion there are 24 bits, which is called the complete mantissa dx.

[0095] The sign bit, exponent, and mantissa are used in subsequent calculations of the index value of the coefficient lookup table and in the final post-processing steps.

[0096] In some embodiments, the intervals of the non-negative number domain in the coefficient lookup table are arranged in order from small to large; the intervals of the non-positive number domain in the coefficient lookup table are arranged from large to small, and the array sequence number of each interval in the coefficient lookup table is used as the index value.

[0097] In some embodiments, the coefficient lookup table unit is further used to determine the number of segments of the non-infinite interval and the number of decimal places of the quadratic function coefficient.

[0098] The number of segments of an interval can be set as size, the number of decimal places of the constant term as t, the number of decimal places of the coefficient of the first-order term as p, and the number of decimal places of the coefficient of the second-order term as q. The role of t, p, and q is to truncate the calculated original coefficients according to this value in the subsequent function approximation process to adapt to the number of bits required by different computing elements, or to control the accuracy of the fitting. The larger the number of bits retained, the higher the accuracy, but the corresponding calculation overhead will also increase. For example, the number of decimal places of the coefficient of the constant term can be t = 21, the number of decimal places of the coefficient of the first-order term can be p = 12, and the number of decimal places of the coefficient of the second-order term can be q = 10.

[0099] The best consistent approximation is performed on the next segment or the first segment and truncated according to the number of decimal places of the coefficient of the quadratic function to obtain the coefficient of the linear term.

[0100] It should be understood that the best consistent approximation is a function approximation method that minimizes the absolute value of the maximum error between the original function and the fitting polynomial. The strict best consistent approximation polynomial is difficult to obtain, so the Remez algorithm is often used to approximate it by iteration. This best consistent approximation uses a quadratic polynomial. In order to reduce the error, the quadratic function coefficients can be extracted step by step. After the first best consistent approximation, the original linear term coefficient C1' is first extracted, and its decimal places are truncated according to the p value to obtain the linear term coefficient C1.

[0101] The coefficient of the quadratic term is obtained by calculating according to the coefficient of the linear term, the number of decimal places of the coefficient of the quadratic function and the number of digits of the first mantissa subsegment.

[0102] It should be understood that in order to compensate for the error caused by intercepting C1' to a certain extent, the following formula can be used when calculating the quadratic term coefficient C2:

[0103] C2”=C2'+(C1'-C1)·size;

[0104] Where C2' is the coefficient of the original quadratic term obtained by the first best consistent approximation.

[0105] C2" is truncated according to the q value to obtain C2. The error caused by truncation of C1' is (C1'-C1)·x, and the error caused by correction is (C1'-C1)·x 2 ·size, this formula can reduce the error but cannot eliminate it.

[0106] The best consistent approximation is performed again and truncated according to the number of decimal places of the quadratic function coefficient to obtain the constant term.

[0107] It should be understood that the error generated after the first and second term coefficients are truncated is already large, so another best consistent approximation is performed. Let the activation function be f(x), then the function to be approximated is f(x)-C2·x 2 -C1·x, use the constant term to approximate, get the original constant term C0', and then truncate it according to the t value to get the constant term C0.

[0108] Save the linear term coefficient, quadratic term coefficient, and constant term in hexadecimal format.

[0109] It should be understood that the constant term C0, the linear term coefficient C1, and the quadratic term coefficient C2 are saved in hexadecimal form in sequence. Since the position of the decimal point is fixed for each coefficient, the decimal point can be omitted and saved directly in integer form.

[0110] Generate a coefficient lookup table of a non-infinite interval based on the linear term coefficient, the quadratic term coefficient, and the constant term.

[0111] It should be understood that when all segments of an interval are calculated, a coefficient lookup table for the interval is generated.

[0112] For the intermediate interval and the zero-containing interval, the concepts of the first mantissa subsegment and the second mantissa subsegment are different.

[0113] (1) The first mantissa subsegment and the second mantissa subsegment of the middle interval: If the middle interval is fitted, the 1 in the general expression represents the 1 implied in the highest bit of the IEEE754 single-precision floating-point number, and the independent variable x represents the mantissa part. In each segment, the highest log2(size) bit of the mantissa of x is always 0, and only the bits after it participate in the best consistent approximation operation of the function. The highest log2(size) bit of the mantissa is called the first mantissa subsegment, and the k bits immediately after the first mantissa subsegment are called the second mantissa subsegment. The number of bits of the second mantissa subsegment can be changed to accommodate multiplication and addition units of different specifications.

[0114] The purpose of shifting the function to the left by 1 / size each time is to subtract the first mantissa subsegment to ensure that only the subsequent mantissas are involved in the calculation. However, in fact, the first mantissa subsegment of the fitted segment is successively increased by 1. For example, when size = 8, the first mantissa subsegment has 3 bits. When fitting the first segment, the first mantissa subsegment is (000)2, and when fitting the second segment, the first mantissa subsegment is (001)2. And so on, when fitting the 8th segment, the first mantissa subsegment is (111)2. Since the first mantissa subsegment is only a partial mantissa and has nothing to do with the exponent, the value of the first mantissa subsegment is independent of the current interval and is only related to the sequence number of the segment. Therefore, the first mantissa subsegment can be used to calculate the index value.

[0115] (2) The first mantissa subsegment and the second mantissa subsegment of the zero-containing interval: If the fitting is a zero-containing interval, there is no +1 in the general expression, and the independent variable x does not directly correspond to the mantissa part of the floating-point number. However, if it is expressed in the format of a general binary fixed-point decimal, the log2(size) bits after the decimal point of x correspond to the first mantissa subsegment, and the decimal bits after the first mantissa subsegment actually participate in the calculation, and the k bits after the first mantissa subsegment can still be set as the second mantissa subsegment. To obtain the first mantissa subsegment and the second mantissa subsegment from the floating-point format, it is necessary to first convert it into a fixed-point decimal format, that is, to perform a right shift operation on the order so that the true value of the order code is log2(min), and then extract the first mantissa subsegment and the second mantissa subsegment.

[0116] Since the segment number size is 8, in some embodiments, the first mantissa subsegment has 3 bits, and the second mantissa subsegment has k=19 bits, that is, the last bit of the 23-bit mantissa of the single-precision floating point number does not participate in the calculation.

[0117] The operation unit is used to calculate the index value and the second mantissa sub-segment according to the sign bit, the exponent and the mantissa; obtain the corresponding quadratic function coefficient in the coefficient lookup table according to the index value; and obtain the value of the quadratic polynomial based on the second mantissa sub-segment, the corresponding quadratic function coefficient and the fixed-point number multiplication and addition operation.

[0118] In some embodiments, the operation unit is further used to perform a first classification according to the sign class to confirm whether the floating point number is located in an interval of the non-negative number domain or an interval of the non-positive number domain.

[0119] It should be understood that the operation unit includes a multiplier and an adder. For the non-negative number domain, the intervals can be arranged in order from small to large, that is, the interval [0, min) is at the front and the interval [max / 2, max) is at the end; for the non-positive number domain, the intervals can be arranged in order from large to small, that is, the interval (-min, 0] is at the front and the interval (-max, -max / 2] is at the end, and each interval is also arranged from large to small, for example, the first item in the interval (-min, 0] is the coefficient of (-min / size, 0]. The index value is the array number of the coefficient. Since three consecutive coefficients are extracted each time, it is only necessary to determine the array number of the constant term C0. The index value of the linear term coefficient C1 is C0 number + 1, and the index value of the quadratic term coefficient C2 is C0 number + 2. When calculating the index value, first determine whether the sign bit is 0 or 1, and perform the first classification.

[0120] According to the true number of the exponent, confirm whether the floating-point number is in an infinite interval, an intermediate interval, or an interval containing zero: when the floating-point number is in an interval containing zero, shift the mantissa right to the exponent, and use the first mantissa subsegment and the segment number to calculate the index value and the second mantissa subsegment; when the floating-point number is in an infinite interval, use the constant term of the last segment of the intermediate interval with the largest endpoint as the constant term of the floating-point number to calculate the index value; when the floating-point number is in the intermediate interval, use the first mantissa subsegment to calculate the index value and the second mantissa subsegment.

[0121] It should be understood that when calculating the index value and the second mantissa sub-segment according to the exponent, the sign bit may be set to 0. The subsequent specific process is as follows:

[0122] If the true value of the exponent is less than log2(min), then the corresponding zero-containing interval requires the 24-bit complete mantissa dx to be right-shifted to make the true value of the exponent equal to log2(min). The 23rd bit of the post-order sequence (assuming the lowest bit of the post-order sequence is the first bit) and the following log2(size)-1 bits correspond to the first mantissa subsegment, where size is the number of interval segments; the k bits after the first mantissa subsegment correspond to the second mantissa subsegment.

[0123] The index value can be calculated using the first mantissa subsegment. Let's assume that the C0 array number of the first segment of data containing the zero interval [0, min) is m, then the C0 index value of the current input is

[0124] index(C0)=(x1) 10 3+m;

[0125] Among them, (x1) 10 Indicates converting the first mantissa subfield of a binary number to decimal.

[0126] After obtaining the C0 index value, the C1 and C2 index values ​​can be further obtained.

[0127] If the true value of the order code is greater than or equal to log2(max), it corresponds to an infinite interval, and there is no need to perform function approximation. C0 is directly set to the C0 of the last segment on the intermediate interval [max / 2, max) with the largest endpoint, and C1 and C2 can be further obtained. Note that if the infinite interval corresponds to the constant part of the nonlinear activation function, for example, the Sigmoid function can be regarded as a constant 1 on the interval [8, +∞), then the quadratic polynomial value calculated using C0, C1, and C2 is correct, because C1 and C2 are both 0, and the result is independent of the input; but if the infinite interval corresponds to the linear part of the nonlinear activation function, for example, the Mish function can be regarded as a linear function y=x on the interval [8, +∞), then the binomial value calculated using C0, C1, and C2 is wrong, so it is necessary to perform special assignment in the post-processing step without referring to the value of the quadratic polynomial.

[0128] If the true value of the exponent is greater than or equal to log2(min) and less than log2(max), it corresponds to the middle interval and no shifting is required. The 23rd bit of the complete mantissa dx (assuming the lowest bit of the complete mantissa dx is the 1st bit) and the log2(size)-1 bits after it correspond to the first mantissa subsegment, where size is the number of interval segments; the k bits after the first mantissa subsegment correspond to the second mantissa subsegment.

[0129] The index value can be calculated using the first mantissa subsegment. Let's assume that the true value of the input exponent is exp, and the C0 array number of the first segment of data containing the zero interval [0, min) is m. Then the current input C0 index value is:

[0130] index(C0)=[exp-log2(min)]·size·3+(x1) 10 3+m;

[0131] Among them, (x1) 10 Indicates converting the first mantissa subfield of a binary number to decimal.

[0132] After obtaining the C0 index value, the C1 and C2 index values ​​can be further obtained.

[0133] If the sign bit is 1, m needs to be changed to the C0 array number of the first segment of data containing the zero interval (-min, 0], and when the true value of the exponent is greater than or equal to log2(max), C0 is directly set to the C0 of the last segment on the middle interval with the smallest endpoint (-max, max / 2], and the rest of the process remains unchanged.

[0134] In a specific embodiment, the C0 array number of the first segment of data containing the zero interval (-0.125, 0] is 0, the C0 array number of the first segment of data containing the zero interval [0, 0.125] is 192, the C0 array number of the last segment of the middle interval (-16, -8] with the smallest endpoint value is 189, and the C0 array number of the last segment of the middle interval [8, 16) with the largest endpoint value is 381. Substituting the above formula into the above formula, the calculation formula of the index value can be obtained.

[0135] In some embodiments, the operation unit is further used to calculate the linear term coefficient and the quadratic term coefficient in the corresponding quadratic function coefficient using signed integer multiplication to obtain the linear term value and the quadratic term value.

[0136] It should be understood that the constant term equal to C0 does not have to be calculated.

[0137] According to the second mantissa subsegment, the maximum number of decimal places, the number of decimal places of the first term and the number of decimal places of the second term are calculated.

[0138] It should be understood that the maximum number of decimal places needs to be recorded before adding each term, so as to determine how to shift each term when adding to ensure the correct result. The number of decimal places of the constant term, bits0 = t. The number of decimal places of the linear term, bits1, is:

[0139] bits1 = p + k + log2 (size);

[0140] Wherein, k is the number of bits in the second mantissa subsegment, and log2(size) is the number of bits in the first mantissa subsegment.

[0141] The number of decimal places in the quadratic term, bits2, is:

[0142] bits2=p+2·[k+log2(size)]

[0143] The maximum number of decimal places, max_bits, is the maximum value of bits0, bits1, and bits2.

[0144] According to the maximum number of decimal places, the number of decimal places of the linear term and the number of decimal places of the quadratic term, the constant term, the linear term value and the quadratic term value are shifted and added to obtain the value of the quadratic polynomial.

[0145] It should be understood that the constant term is left-shifted by max_bits-bits0, the linear term is left-shifted by max_bits-bits1, the quadratic term is left-shifted by max_bits-bits2, and the value sum of the quadratic polynomial is obtained by left-shifting each term and adding the signed integers.

[0146] Since the sign of the output value of the mainstream activation function is very easy to determine, for example, for activation functions such as Tanh and Mish, the output is greater than 0 when the input is greater than 0, and the output is less than 0 when the input is less than 0. For example, the Sigmoid function is always greater than 0, so the absolute value of the sum can be taken and the sign bit can be directly assigned in the post-processing step.

[0147] The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine the sign bit and the exponent of the floating point number so that the output value is a value in the floating point format.

[0148] In some embodiments, the post-processing output unit is used to determine whether the floating point number is infinite or non-number according to the exponent and the mantissa.

[0149] If the floating point number is infinite or non-number, the output is assigned a value according to the activation function.

[0150] Specifically, if the exponent of the input floating-point number is all 1 and the mantissa is all 0, it is infinity; if the exponent of the input floating-point number is all 1 and the mantissa is not all 0, it is non-number. At this time, the output can be directly assigned according to the specific activation function form. For example, for the Sigmoid function, when the input is positive infinity, the output is 1, that is, the sign bit is 0, the exponent is (01111111)2, and the mantissa is all 0; when the input is negative infinity, the output is 0, and the sign bit, exponent, and mantissa are all 0.

[0151] The sign bit and the exponent are used to determine whether the floating point number falls within the linear interval of the activation function.

[0152] If the floating-point number falls within the linear interval of the activation function, the output value is equal to the floating-point number.

[0153] Specifically, for activation functions such as Mish, Swish, and GELU, these functions tend to the linear function y=x on the infinite interval [max, +∞). If the sign bit of the input floating-point number is 0 and the true value of the exponent is greater than or equal to log2(max), the output floating-point number is directly set to be equal to the input floating-point number.

[0154] If the floating point number is not infinite or non-number and does not fall within the linear interval of the activation function, the value of the quadratic polynomial is normalized to obtain the mantissa and the exponent of the output value.

[0155] Specifically, if it is not the above two cases, it is necessary to normalize sum, find the mantissa and the exponent of the output floating point number. Let sum be a 64-bit integer, and the absolute value of sum has been found in the step of calculating the quadratic polynomial, and sum is always greater than or equal to 0. Starting from the highest bit of sum (let the lowest bit be the first bit), traverse to the lower bits one by one. If the current bit is 1, output the serial number i of the bit, and output the 23 bits of data of the bit and after it; if there are less than 23 bits after the bit, it is necessary to fill with zeros; if no digit of sum is 1, the output i is -1, indicating that the result is 0. If i is -1, the output value is equal to the output value when the activation function input is 0, which is generally a simple rational number. For example, the Sigmoid function outputs 0.5 when the input is 0, and the Tanh, Mish and other functions output 0 when the input is 0. It can be directly converted into floating point form. If zero is not 1, remove the 1 in the highest bit of the 24-bit data found, and the resulting 23-bit data is the mantissa of the output floating-point number. The exponent res_exp in the shifted form of the output floating-point number can be calculated according to the following formula:

[0156] res_exp=127+i-max_bits;

[0157] Among them, max_bits is the maximum number of decimal places.

[0158] Outputs a floating-point value based on the mantissa and the exponent of the output value.

[0159] Specifically, the sign bit of the floating-point number output by the Sigmoid function is always 0, and the sign bit of the floating-point number output by activation functions such as Tanh, Mish, and Swish is equal to the sign bit of the input floating-point number.

[0160] Some embodiments of the present application also provide a method for fast calculation of a nonlinear activation function based on a coefficient lookup table, which is applied to a device for fast calculation of a nonlinear activation function based on a coefficient lookup table in the above embodiment, see Fig.10 , methods include:

[0161] S100: Divide the entire area of ​​the activation function into a number of intervals, and divide the non-infinite interval in the interval into a number of segments.

[0162] S200: Use a quadratic function to perform approximation on each segment to establish a coefficient lookup table containing optimal quadratic function coefficients.

[0163] Among them, the coefficients of the quadratic function include the coefficients of the linear term, the coefficients of the quadratic term and the constant term;

[0164] S300: extracting features of the input floating point number to obtain a sign bit, an exponent and a mantissa.

[0165] S400: Calculate the index value and the second mantissa subsegment according to the sign bit, the exponent and the mantissa.

[0166] S500: Obtain corresponding quadratic function coefficients in a coefficient lookup table according to the index value.

[0167] S600: Obtaining a value of a quadratic polynomial based on the second mantissa subsegment, the corresponding quadratic function coefficients, and fixed-point number multiplication and addition operations.

[0168] S700: normalize the value of the quadratic polynomial in integer form, and combine it with the sign bit and the exponent of the floating point number to output a value in floating point format.

[0169] It should be understood that in a neural network, an activation function can map the input of a neuron to the output. If the activation function is not used, the neural network is just a linear model and cannot learn the nonlinear relationship in the data. The activation function, by introducing nonlinearity, gives the neural network stronger learning and expression capabilities, enabling it to handle more complex problems. Therefore, the choice of activation function is crucial to the performance of the neural network model, and a suitable activation function can effectively improve the robustness and output accuracy of the model. Moreover, most nonlinear activation functions only show nonlinearity in a limited interval near 0. When the absolute value of the input is large, the output is a constant or linearly related to the input. Therefore, only the nonlinear part needs to be fitted to achieve fast calculation. However, the lookup table method subdivides the activation function into many segments, the activation function on each segment is a fixed value, and all values ​​are stored as a lookup table. For a specific input, it is only necessary to find the lookup table index according to certain rules, and the corresponding value can be taken out to get the output. This method requires almost no calculation, so the calculation speed is the fastest. At the same time, its segmentation method is independent of the activation function form, and it has good versatility, but if you want to achieve higher accuracy, you need to consume a lot of storage resources. The piecewise approximation divides the activation function into a few segments, and uses a linear function or a higher-order polynomial to approximate each segment. This method has no storage overhead, but due to the small number of segments, each activation function needs to be segmented separately; and because each segment is long, if a linear function is used for approximation, the accuracy is generally low, and if a high-order polynomial is used for approximation, the computing power consumption will increase with the increase of the polynomial degree, and the real-time performance is lacking.

[0170] The above method combines the lookup table method and the piecewise approximation method. It not only inherits the advantages of the conventional lookup table method in terms of good versatility, but also consumes less storage resources under the same accuracy requirements. At the same time, the method has a small amount of calculation, and the calculation speed is not much lower than that of the conventional lookup table method, which is better than the piecewise high-order polynomial approximation method. It achieves a good compromise between storage resources and calculation speed, ensures high accuracy, and has a calculation speed that is better than the Taylor expansion method. It is suitable for fast calculation of nonlinear activation functions with high accuracy requirements.

[0171] According to the above method, the maximum absolute error on the zero-containing interval and the middle interval as well as the maximum absolute error of the entire domain are counted, as shown in the following table:

[0172] Table 1 Sigmoid function error statistics

[0173]

[0174] Table 2 Tanh function error statistics

[0175]

[0176] Table 3 Error statistics of Softplus function

[0177]

[0178]

[0179] Table 4 Error statistics of Mish function

[0180]

[0181] Table 5 Swish function error statistics

[0182]

[0183] Table 6. GELU function error statistics

[0184]

[0185] As can be seen from the tables, this method has a high accuracy in fast calculation of activation functions, with a maximum error of 10-5, which is significantly better than the lookup table method and the piecewise approximation method. And because only quadratic polynomials are calculated, the calculation speed is better than the Taylor expansion method.

[0186] It can be seen from the above technical scheme that the embodiment of the present application provides a device and method for fast calculation of a nonlinear activation function based on a coefficient lookup table, and the device includes: an input selection unit, which is used to select an input floating-point number according to the type of activation function; a coefficient lookup table unit, which is used to divide the entire area of ​​the activation function into several intervals, and divide the non-infinite intervals in the interval into several segments; a quadratic function is used for approximation on each segment to establish a coefficient lookup table containing the best quadratic function coefficients; wherein the quadratic function coefficients include linear term coefficients, quadratic term coefficients and constant terms; a preprocessing unit, which is used to extract features of the input floating-point number to obtain a sign bit, an exponent and a mantissa; an operation unit, which is used to calculate an index value and a second mantissa sub-segment according to the sign bit, the exponent and the mantissa; according to the index value, the corresponding quadratic function coefficient is obtained in the coefficient lookup table; based on the second mantissa sub-segment, the corresponding quadratic function coefficient and the fixed-point number multiplication and addition operation, the value of the quadratic polynomial is obtained; a post-processing output unit, which is used to normalize the value of the quadratic polynomial in integer form, and combine the sign bit and exponent of the floating-point number to output the value in floating-point format. Solve the problem of slow calculation speed of non-linear activation functions.

[0187] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A fast calculation device for nonlinear activation function based on coefficient lookup table, characterized in that: include: An input selection unit, used to select an input floating point number according to the activation function type; A coefficient lookup table unit is used to divide the entire area of ​​the activation function into a number of intervals, and divide the non-infinite interval in the interval into a number of segments; use a quadratic function to approximate each segment to establish a coefficient lookup table containing the best quadratic function coefficients; wherein the quadratic function coefficients include a linear term coefficient, a quadratic term coefficient, and a constant term; A preprocessing unit, used for performing feature extraction on the floating point number to obtain a sign bit, an exponent and a mantissa; an operation unit, configured to calculate an index value and a second mantissa subsegment according to the sign bit, the exponent and the mantissa; obtain a corresponding quadratic function coefficient in the coefficient lookup table according to the index value; and obtain a value of a quadratic polynomial based on the second mantissa subsegment, the corresponding quadratic function coefficient and a fixed-point multiplication and addition operation; The post-processing output unit is used to normalize the value of the quadratic polynomial in integer form and combine the sign bit and the exponent of the floating point number to output the value in floating point format.

2. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The endpoints of the interval include 0, a power of 2 or the opposite of a power of 2, positive infinity, and negative infinity; When the interval is in the non-negative number domain, the left endpoint of the interval is closed and the right endpoint of the interval is open; When the interval is located in a non-positive number domain, the left endpoint of the interval is open and the right endpoint of the interval is closed.

3. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The coefficient lookup table unit is also used to determine the number of segments of the non-infinite interval and the number of decimal places of the quadratic function coefficient; Performing the best consistent approximation on the next segment or the first segment and truncating the coefficient of the quadratic function according to the number of decimal places of the coefficient to obtain the coefficient of the first term; Calculate according to the coefficient of the linear term, the number of decimal places of the coefficient of the quadratic function and the number of places of the first mantissa subsegment to obtain the coefficient of the quadratic term; Performing the best consistent approximation again and truncating the coefficient according to the number of decimal places of the quadratic function coefficient to obtain a constant term; Saving the linear term coefficient, the quadratic term coefficient and the constant term in hexadecimal form; A coefficient lookup table of the non-infinite interval is generated according to the linear term coefficient, the quadratic term coefficient and the constant term.

4. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The floating point number is an IEEE754 single-precision floating point number.

5. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The interval includes an infinite interval and a non-infinite interval; the non-infinite interval includes an intermediate interval and an interval containing zero.

6. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 5, characterized in that: The intervals of the non-negative number domain in the coefficient lookup table are arranged in order from small to large; the intervals of the non-positive number domain in the coefficient lookup table are arranged from large to small, and the array sequence number of the arrangement of each interval in the coefficient lookup table is used as the index value.

7. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 6, characterized in that: The operation unit is further used to perform a first classification according to the symbol class to confirm whether the floating point number is located in the interval of the non-negative number domain or the interval of the non-positive number domain; Determine, based on the true number of the exponent, whether the floating point number is in an infinite interval, an intermediate interval, or an interval containing zero; When the floating point number is in a zero-containing interval, the mantissa is right-shifted to the right order, and the index value and the second mantissa subsegment are calculated using the first mantissa subsegment and the segment number; When the floating-point number is in an infinite interval, the constant term of the last segment of the middle interval with the largest endpoint is used as the constant term of the floating-point number to calculate the index value; When the floating point number is in the middle interval, the index value and the second mantissa subsegment are calculated using the first mantissa subsegment.

8. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The operation unit is also used to calculate the linear term coefficient and the quadratic term coefficient in the corresponding quadratic function coefficient by using signed integer multiplication to obtain the linear term value and the quadratic term value; According to the second mantissa subsegment, the maximum number of decimal places, the number of decimal places of the first term and the number of decimal places of the second term are calculated; According to the maximum number of decimal places, the number of decimal places of the linear term and the number of decimal places of the quadratic term, the constant term, the linear term value and the quadratic term value are shifted and added to obtain the value of the quadratic polynomial.

9. The nonlinear activation function fast calculation device based on coefficient lookup table according to claim 1, characterized in that: The post-processing output unit is used to determine whether the floating point number is infinite or non-number according to the exponent and the mantissa; If the floating point number is infinite or non-number, assign a value to the output according to the activation function form; Determine whether the floating point number falls within the linear interval of the activation function according to the sign bit and the exponent; If the floating point number falls within the linear interval of the activation function, the output value is equal to the floating point number; If the floating point number is not infinite or non-number and does not fall within the linear interval of the activation function, normalize the value of the quadratic polynomial to obtain the mantissa and the order of the output value; Outputs a floating-point value based on the mantissa and the exponent of the output value.

10. A fast calculation method for a nonlinear activation function based on a coefficient lookup table, characterized in that: A fast calculation device for a nonlinear activation function based on a coefficient lookup table applied to any one of claims 1 to 9, the method comprising: Dividing the entire activation function region into a number of intervals, and dividing a non-infinite interval in the interval into a number of segments; Using a quadratic function to approximate each segment to establish a coefficient lookup table containing optimal quadratic function coefficients; wherein the quadratic function coefficients include a linear term coefficient, a quadratic term coefficient, and a constant term; Perform feature extraction on the input floating point number to obtain the sign bit, exponent and mantissa; Calculate the index value and the second mantissa subsegment according to the sign bit, the exponent and the mantissa; Obtaining a corresponding quadratic function coefficient in the coefficient lookup table according to the index value; deriving a value of a quadratic polynomial based on the second mantissa subsegment, the corresponding quadratic function coefficient, and a fixed-point multiplication and addition operation; The value of the quadratic polynomial in integer form is normalized and combined with the sign bit and the exponent of the floating point number to output a value in floating point format.

Images