An activation function processing method, device and integrated circuit based on hardware chip
By using the combination method of Taylor expansion and register group on the hardware chip, the calculation process of the activation function is simplified, the problem of high computational complexity and cost of activation function in the prior art is solved, and the low complexity and low cost are achieved Accurate calculation.
Patent Information
- Application Number
- CN202011625104.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-30
AI Technical Summary
When calculating activation functions in hardware, it is difficult to obtain accurate results under low complexity and low cost conditions. High-order polynomial calculations are complex and costly, and register storage schemes require thousands of registers.
Using the activation function processing method based on the hardware chip, the Taylor expansion initialization register is used to store the first M term polynomial coefficient value and the remaining term compensation value, and the preset domain and register group address are used to simplify the calculation and reduce the number of registers.
Obtaining accurate calculation results of activation functions at low cost and low complexity reduces the number of registers, simplifies the calculation process, and improves the calculation accuracy.
Smart Images

Figure CN112651497B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an activation function processing method, device and integrated circuit based on a hardware chip. Background Art
[0002] The accuracy of the activation function calculation results plays an important role in the hardware acceleration of the neural network model. Currently, there are two solutions for calculating the activation function in hardware:
[0003] The first method is to approximate the function value of the activation function through a high-order polynomial. However, if you want to improve the accuracy of the calculation result, you need to increase the order of the polynomial. When the order of the polynomial is high, the calculation process is complicated, which increases the area, power consumption, and delay of the hardware circuit.
[0004] The second method is to store the function value of the activation function at each independent variable in a register, and directly obtain it from the register when the function value of the activation function needs to be calculated. Although this solution simplifies the calculation process, if you want to achieve an error level of one ten-thousandth, at least thousands of registers are required, which is costly.
[0005] Therefore, both of the above two solutions cannot obtain accurate calculation results of the activation function under low complexity and low cost conditions. Summary of the invention
[0006] In order to solve the above technical problems, the present application provides an activation function processing method, device and integrated circuit based on a hardware chip, which can obtain accurate calculation results of the activation function under low complexity and low cost conditions.
[0007] The embodiments of the present application disclose the following technical solutions:
[0008] In a first aspect, the present application provides an activation function processing method based on a hardware chip, the method comprising:
[0009] Initializing and configuring the registers, including: according to the Taylor expansion of the activation function at the target value; obtaining the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; storing the coefficient values and the compensation values in the register group corresponding to the target value;
[0010] Obtaining a first independent variable within a preset definition domain of the activation function;
[0011] Determine a first difference between the first independent variable and the initial value, and calculate the difference between the first difference and 2 -K Determine the address of the register group and the corresponding target value by a multiple of
[0012] According to the address, reading the coefficient value and the compensation value in the register group;
[0013] Determine the sum of the first M polynomials by using the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder by using the first independent variable, the target value and the compensation value;
[0014] The function value of the activation function at the first independent variable is determined according to the sum of the first M polynomials and the approximate value of the remainder.
[0015] Optionally, a compensation coefficient is determined according to the first independent variable and the target value, and an approximate value of the remainder is determined by the compensation value and the compensation coefficient;
[0016] Wherein, the compensation coefficient is not positively correlated with the M.
[0017] Optionally, the compensation coefficient is as follows:
[0018]
[0019] Where N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
[0020] Optionally, an approximate value of the remainder is obtained by the following formula:
[0021]
[0022] Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in the one register.
[0023] Optionally, a second difference between the first independent variable and the target value is determined, and the sum of the first M polynomials is determined based on the second difference and the coefficient value.
[0024] Optionally, the sum of the first M polynomials is obtained by the following formula:
[0025]
[0026] Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value among the M coefficient values.
[0027] Optionally, the function value of the activation function at the first independent variable is determined according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula:
[0028] f(x)=f lut (x)+Δ(x)
[0029] Wherein, f(x) is the function value of the activation function at the first independent variable.
[0030] In a second aspect, the present application provides an activation function processing device based on a hardware chip, comprising: a configuration module, a determination module, a reading module and a calculation module;
[0031] The configuration module is used to initialize the configuration of the registers according to the Taylor expansion of the activation function at the target value; obtain the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; and store the coefficient values and the compensation values in the register group corresponding to the target value;
[0032] The determination module is used to obtain a first independent variable within a preset definition domain of the activation function; determine a first difference between the first independent variable and an initial value, and calculate a first difference between the first independent variable and an initial value according to the first difference and 2 -K Determine the address of the register group and the corresponding target value by a multiple of
[0033] The reading module is used to read the coefficient value and the compensation value in the register group according to the address;
[0034] The calculation module is used to determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value; and determine the function value of the activation function at the first independent variable based on the sum of the first M polynomials and the approximate value of the remainder.
[0035] Optionally, the calculation module is specifically used to determine a compensation coefficient according to the first independent variable and the target value, and determine an approximate value of the remainder by using the compensation value and the compensation coefficient;
[0036] Wherein, the compensation coefficient is not positively correlated with the M.
[0037] Optionally, the compensation coefficient is as follows:
[0038]
[0039] Where N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
[0040] Optionally, the calculation module is specifically used to obtain an approximate value of the remainder by using the following formula:
[0041]
[0042] Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in the one register.
[0043] Optionally, the calculation module is specifically used to determine a second difference between the first independent variable and the target value, and determine the sum of the first M polynomials according to the second difference and the coefficient value.
[0044] Optionally, the calculation module is specifically configured to obtain the sum of the first M polynomials by using the following formula:
[0045]
[0046] Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value among the M coefficient values.
[0047] Optionally, the calculation module is specifically used to determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula:
[0048] f(x)=f lut (x)+Δ(x)
[0049] Wherein, f(x) is the function value of the activation function at the first independent variable.
[0050] In a third aspect, the present application provides an integrated circuit, comprising any one of the devices introduced in the second aspect above.
[0051] It can be seen from the above technical solution that the present application has the following advantages:
[0052] The present application provides an activation function processing method, device and integrated circuit based on a hardware chip, the method comprising: obtaining a first independent variable within a preset definition domain of the activation function; determining a first difference between the first independent variable and an initial value, and determining a first difference between the first independent variable and the initial value according to the difference between the first independent variable and the initial value; -KThe address of the register group and the corresponding target value are determined by a multiple of; wherein the Taylor expansion of the activation function at the target value includes the first M polynomials and the remainder; the register group includes M+1 registers; M registers respectively store the coefficient values of each term of the first M polynomials, and 1 register stores the compensation value used to calculate the approximate value of the remainder; since the first independent variable comes from a preset domain interval, and the preset domain interval does not exceed the interval from negative infinity to positive infinity, the number of required registers is reduced, thereby reducing the cost; the target values in adjacent register groups differ by 2 -K ; K is an integer, M is a positive integer; Since the target values in adjacent register groups differ by 2 -K Therefore, when the computer performs binary operations, it can directly obtain the first independent variable as 2 -K multiples of, without the need for addition, subtraction and if judgment, thereby simplifying the calculation process. According to the address, read the coefficient value in the register group and read the compensation value; determine the sum of the first M polynomials by the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder by the first independent variable, the target value and the compensation value; determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder. The technical solution provided by the present application uses the function value at the pre-stored target value, the multi-order derivative value of the function and the remainder compensation value, and adopts a nonlinear method to obtain the function value of the input first independent variable. For the target value, the number of target values that need to be stored is reduced. Therefore, from the overall use of the number of registers, compared with the traditional method, the number of registers is saved a lot. In other words, for the calculation accuracy of the function value, this method is better than the case where the traditional method uses several times the number of registers. Therefore, the technical solution provided by the present application can obtain accurate calculation results of the activation function under low cost and low complexity conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 A flowchart of an activation function processing method based on a hardware chip provided in this application;
[0055] Figure 2 A schematic diagram of a register provided for this application;
[0056] Figure 3A computer calculation logic diagram provided for this application;
[0057] Figure 4 Another computer calculation logic diagram provided for this application;
[0058] Figure 5A Another computer calculation logic diagram provided for this application;
[0059] Figure 5B A processing logic diagram of an integrated circuit provided by the present application;
[0060] Figure 6 A schematic diagram of an activation function processing device based on a hardware chip provided in this application. DETAILED DESCRIPTION
[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0062] When using Field Programmable Gate Array (FPGA) or System on Chip (SoC) hardware to develop certain functions of a product, special function operations other than addition, subtraction, and multiplication may be encountered during its operation, such as e x , trigonometric functions, hyperbolic functions, etc. With the development of artificial intelligence (AI) technology in recent years, hyperbolic functions (such as Tanh functions) are frequently used by neural network models as an activation function, thereby enhancing the expression ability of neural network models through nonlinear mapping. In addition, activation functions also include Sigmoid functions and ReLU functions.
[0063] The accuracy of the activation function calculation results has a certain impact on the final results of the hardware accelerator of the large-scale neural network model. The higher the accuracy of the activation function calculation results, the more accurate the hardware accelerator calculation results. However, there are two solutions for calculating the activation function in hardware:
[0064] The first method is to approximate the function value of the activation function through a high-order polynomial. However, if you want to improve the accuracy of the calculation result, you need to increase the order of the polynomial. When the order of the polynomial is high, the calculation process is complicated, which increases the area, power consumption, and delay of the hardware circuit.
[0065] The second method is to store the function value of the activation function at each independent variable in a register, and directly obtain it from the register when the function value of the activation function needs to be calculated. Although this solution simplifies the calculation process, if you want to achieve an error level of one in ten thousand, at least thousands of registers are required, which is costly. Although some products use linear interpolation methods to appropriately reduce the use of registers; however, due to the limited expressiveness of the linear function itself and the errors introduced by it, the effect of register saving or accuracy improvement is not significant.
[0066] Under low-complexity and low-cost conditions, it is difficult to directly obtain or calculate the exact function value corresponding to the independent variable of the activation function in hardware.
[0067] In order to solve the problem of calculating the above activation function in hardware, the present application provides an activation function processing method based on a hardware chip, including: initializing and configuring registers, including: according to the Taylor expansion of the activation function at the target value; obtaining the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; storing the coefficient values and the compensation values in the register group corresponding to the target value; obtaining a first independent variable within a preset definition domain of the activation function; determining a first difference between the first independent variable and the initial value, and calculating the compensation value according to the difference between the first difference and the 2 -K The address of the register group and the corresponding target value are determined by a multiple of ; since the first independent variable comes from a preset domain interval, and the preset domain interval is smaller than the interval from negative infinity to positive infinity, the number of required registers is reduced, thereby reducing the cost; the target values in adjacent register groups differ by 2 -K ; K is an integer, M is a positive integer; Since the target values in adjacent register groups differ by 2 -K Therefore, when the computer performs binary operations, it can directly obtain the first independent variable as 2 -K multiples of, without the need for addition, subtraction and if judgment, thereby simplifying the calculation process. According to the address, read the coefficient value in the register group and read the compensation value; determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value; determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder.
[0068] The technical solution provided by the present application can control the use of the number of registers while taking into account the simplicity of the operation logic, and thus can use a small number of registers to achieve an error level of one ten-thousandth. Specifically, the technical solution provided by the present application uses the function value at the pre-stored target value, the multi-order derivative value of the function and the remainder compensation value, and adopts a nonlinear method to obtain the function value of the input first independent variable. For the target value, the number of target values that need to be stored is reduced. Therefore, in terms of the overall use of the number of registers, compared with the traditional method, the number of registers is saved a lot. In other words, for the calculation accuracy of the function value, this method is better than the traditional method using several times the number of registers. Therefore, the technical solution provided by the present application can obtain accurate calculation results of the activation function under low cost and low complexity conditions.
[0069] Embodiment 1:
[0070] Embodiment 1 of the present application provides an activation function processing method based on a hardware chip, which is described in detail below with reference to the accompanying drawings.
[0071] See also Figure 1 , which is a flowchart of an activation function processing method based on a hardware chip provided by this application.
[0072] The method comprises the following steps:
[0073] Step 101: Initialize and configure the registers.
[0074] Specifically, according to the Taylor expansion of the activation function at the target value; the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms are obtained; and the coefficient values and the compensation values are stored in the register group corresponding to the target value. Therefore, the embodiment of the present application stores the coefficient values and the compensation values in the register group corresponding to the target value in advance, and after the target value is determined, the corresponding coefficient values and compensation values can be quickly read in the register group.
[0075] It should be noted that, taking the preset domain as [0.0625, 5.0625] as an example, the minimum target value can be equal to the boundary value, that is, the target value is equal to the initial value, for example, the target value and the initial value are both 0.0625. The distance between adjacent target values is 2 -K , the number of target values can be obtained based on the preset domain length and the distance between adjacent target values, that is, the number of target values is obtained based on the ratio of the preset domain length to the distance between adjacent target values. For example: q = (5.0625-0.0625) / 2 -K Then, q is rounded to obtain Q, where q is the ratio of the preset domain length to the distance between adjacent target values, Q is the number of target values, and Q refers to the smallest integer not less than q. When K=3, Q=40 in this embodiment.
[0076] Step 102: Obtain a first independent variable within a preset definition domain of the activation function.
[0077] The preset domain is an interval within an open interval from negative infinity to positive infinity. For example, the preset domain is [-4, 5]. It can be seen that this interval is only an interval and can only take values between -4 and 5, but cannot take values such as -5 or 6. The value range of this interval can be determined according to the specific activation function. For example, when the activation function is less than or equal to -4, and the calculation result of the activation function at the first independent variable is approximately equal to the first independent variable, -4 can be used as the minimum value boundary of the preset domain. Similarly, the maximum value boundary of the preset domain can also be determined to obtain the preset domain of the activation function.
[0078] The first independent variable is an independent variable in the preset definition domain, and the calculation result of the activation function is the value of the activation function at the first independent variable. For example, the first independent variable can be any value between -4 and 5, can be 0, or can be 3. When the first independent variable is 0, the calculation result of the activation function is the value of the activation function at 0.
[0079] This embodiment does not limit the type of activation function. Commonly, activation functions that need to obtain function values by looking up a table in a hardware operator include Tanh function, Sigmoid function, etc.
[0080] To facilitate understanding by those skilled in the art, the following is introduced by taking the Tanh function as an example. For those skilled in the art, the Sigmoid function may also adopt a similar processing method. For the sake of brevity, only the Tanh function is introduced as an example in this embodiment.
[0081] When the activation function is the Tanh function, it can be seen from the characteristics of the Tanh function that the Tanh function satisfies the following characteristics:
[0082] Tanh(-x)=-Tant(x) (1)
[0083]
[0084]
[0085] That is, the Tanh function satisfies formula (1-3). From formula (1), we can know that we only need to know the value of the Tanh function on x∈[0,+∞). The value of the Tanh function on x∈(-∞,0] can be obtained by transformation through formula (1). First, due to the limitation of the number of bits of binary representation in computer hardware, there is a difference between the decimal value expected to be expressed and the binary value actually corresponding to the computer after conversion to decimal. For example, the minimum number that can be represented by a normalized 16-bit floating point number is 2 -14 In other words, a normalized 16-bit floating point number cannot represent a value greater than 2 -14 A smaller positive real number; if the distance between two normalized 16-bit floating-point numbers is less than or approximately 2 -14 , this gap can be ignored in the calculation. According to formula (2), when x≤x′, it is considered that tanh(x)≈x; in this embodiment, x is selected ′ =0.0625, so that tanh(x)-x≈2 -14 , that is, if x∈[0,0.0625), directly return x as the function value of tanh(x). According to formula (3), when x≥x″, it is considered that tanh(x)≈1, that is, directly return 1 as the function value of tanh(x); in this embodiment, x″=5.0625 is selected, so that tanh(x)-x≈2 -14 .
[0086] Then, the preset domain of the Tanh function is obtained, that is, [0.0625, 5.0625], and the first independent variable takes values within the preset domain. Then, for other values, the function value of the Tanh function can be obtained by the formulas (1)-(3). Therefore, it is only necessary to store the parameter values corresponding to the target values contained in the preset domain of the Tanh function in the register, thereby reducing the number of required registers and reducing costs. For example, if you want to get the value of tanh(0.01), according to the formula (2), directly return 0.01, which is the approximate value of tanh(0.01); for another example, if you want to get the value of tanh(-4), according to the formula (1), find the function value corresponding to the first independent variable being 4 in the preset domain, and after inverting the sign bit, it is the value corresponding to tanh(-4).
[0087] Step 103: Determine a first difference between the first independent variable and the initial value, and -K The multiple determines the address of the register group and the corresponding target value.
[0088] The first independent variable has been introduced in detail in step 102, and will not be described again here. The initial value is a boundary value within the preset definition domain. For ease of understanding, the boundary value introduced in this embodiment is a smaller boundary value, i.e., 0.0625. Of course, in some other embodiments, the initial value may also be 5.0625.
[0089] The target value is selected from the preset definition domain, and the target value is closest to the first independent variable and less than the value of the first independent variable. The target value is used to form a corresponding relationship with the coefficient value stored in the register. When the target value is confirmed by the first independent variable and the initial value, the register can be accessed to obtain the value in the register. The specific implementation process will be introduced in detail later.
[0090] The Taylor expansion of the activation function at the target value includes the first M polynomials and the remainder. In order to facilitate the calculation of the Tanh function, the function is expanded at the target value to obtain the Taylor expansion, which is as follows:
[0091]
[0092] Among them, x0 is the target value, Tanh ′ (x0) is the value of the first-order derivative of the Tanh function at the target value x0, x is the first independent variable, R n (x) is the nth order Taylor remainder. In this embodiment, Tanh(x0) is defined as ′ (x0),Tanh″(x0) / 2……Tanh (n) (x0) / n! are the coefficients of the polynomial.
[0093] For ease of calculation, the Taylor expansion is divided into two parts for calculation, the first part is the first M polynomials; the second part is the remainder. For a function that needs to be approximated, if there are high-order derivatives at the target value point, the more terms expanded using the Taylor series, the more accurate it is. However, the more expanded terms there are, the more complicated the calculation will be, and the delay and power consumption will increase. Therefore, the embodiment of the present application uses as few expanded terms as possible to approximate the target function, that is, the first part. The second part is then used to approximate the error caused by too few expanded terms in the first part. Through the two parts, the primary approximation problem of the target function is converted into two sub-problems, which not only simplifies the calculation, but also ensures the accuracy of the results.
[0094] Wherein, M is a positive integer. The specific value of M is not limited in this embodiment. M can be 1, 2, 3, or a larger value. For ease of understanding, the following takes M as 3 as an example for introduction. That is, in this embodiment, the first part includes the first three polynomials, which are
[0095] The remainder is the remaining polynomial in the Taylor expansion except the first M terms. In this embodiment, it is the remaining polynomial except the first 3 terms, that is,
[0096] The first difference is obtained by subtracting the first independent variable from the initial value. In this embodiment, the first difference is (x-0.0625). -K In conventional techniques, when calculating the multiple, addition and subtraction and IF judgment statements, or more complex division, are required to obtain the multiple. This embodiment is different from conventional techniques, and the distance between adjacent target values is defined as 2 -K , you can quickly get the multiple relationship.
[0097] For ease of understanding, K is taken as a negative number for example. Of course, in this application, K should be a positive integer, and the principle is the same. For example, when K=-2, when it is necessary to calculate how many times the decimal number 10 is the decimal number 4, the binary number corresponding to 10 is 1010B, and the binary number corresponding to 4 is 100B. There are two "0"s at the end of 100B. Therefore, on the basis of 1010B, the last two digits are directly removed to get 10B, and the decimal number corresponding to 10B is 2, so the computer can quickly know that 10 is 2 times 4. Another example: when K=-3, it is necessary to calculate how many times 10 is 8. The binary number corresponding to 8 is 1000B, and there are three "0"s at the end. Therefore, on the basis of 1010B, the last three digits are directly removed to get 1B, and the decimal number corresponding to 1B is 1, so the computer can quickly know that 10 is 1 times 8. When K is a positive integer, the principle is similar and will not be repeated here.
[0098] The address of the register group is used to access the data stored in each register in the register group. After the multiple is determined, it is possible to directly jump to the register address corresponding to the multiple according to the multiple.
[0099] The register group includes M+1 registers; M registers store the coefficient values of each term of the first M polynomials respectively, and 1 register stores the compensation value used to calculate the approximate value of the remainder; the target values in adjacent register groups differ by 2 -K .
[0100] In this embodiment, M=3, and the register group includes 4 registers. In order to control the number of registers required on the hardware device and the accuracy of the calculation results of the control activation function, K is selected as 3 in this embodiment. Of course, this application is not limited to K=3, and K can also be 2 or 4. In this embodiment, when K is 3, the cost and calculation results are comprehensively considered, and the effect is better.
[0101] When K = 3, 2 -K=0.125, the target values in adjacent register groups differ by 0.125, and each group of registers includes 4 registers. When the preset definition domain is 0.0625-5.0625, it can be known that the length of the preset definition domain is 5, and 5 is 40 times of 0.125. Therefore, in this embodiment, only 40*4=160 registers are needed, which greatly reduces the number of required registers.
[0102] See also Figure 2 , which is a schematic diagram of a register provided in this application.
[0103] The figure only shows the first three groups of registers in detail, and the subsequent registers are similar. It can be seen from the figure that each group of registers includes 4 registers. Taking the first group of registers as an example, the M registers in the register group are register A, register B and register C, and the other register is register D. Each register stores data. The data in register A is the value of Tanh (0.0625), the data in register B is Tanh` (0.0625), the data in register C is the value of Tanh`` (0.0625) / 2, and the data in register D is the value of △max (0.0625).
[0104] In the first set of registers, the target value is 0.0625, and in the second set of registers, the target value is 0.0625+2 -3 , in the third group of registers, the target value is 0.0625+2*2 -3 The data in each group of registers are all known values in advance, so the value in the register corresponding to any optional target value can be obtained. For example, Tanh``(0.0625+2 -3 ) / 2 is known, △max(0.0625+2*2 -3 ) are known, etc.
[0105] Among them, for each group of registers, the values stored in the first three registers (such as register A, register B and register C) are the coefficients corresponding to the first three polynomials in the Taylor expansion, and the value stored in the last register (such as register D) is the compensation value of the approximate value of the remainder.
[0106] Specifically, the address of the register group can be determined according to the multiple. When the multiple is 1, the register group is confirmed to be the first group. When the multiple is 2, the register group is determined to be the second group. And so on. The position of the register group is directly determined by the display lookup table (Look Up Table, LUT), and there is no need to traverse all the register groups, thereby reducing the time of the difference value and improving the efficiency of the lookup table. It should be noted that the multiple and the register group number start from 0. It should be noted that when the obtained multiple is a non-integer, the smallest integer not greater than the multiple is taken as the multiple.
[0107] To facilitate understanding by those skilled in the art, the address of a register determined by a computer is described in detail below in conjunction with the accompanying drawings.
[0108] See also Figure 3 , which is a computer calculation logic diagram provided by this application.
[0109] In the figure, sign is the symbol part, exponent is the exponent part, and fraction is the decimal part.
[0110] 301: Determine a first difference between a first independent variable and an initial value, as shown in part 301 in the figure, wherein the first independent variable is represented by a binary 16-bit half floating point number.
[0111] 302: Perform 17-bit fixed-point alignment on the first binary difference, as shown in part 302 in the figure, the first 3 bits are the integer part, and the last 14 bits are the decimal part.
[0112] 303: After intercepting the first 6 bits after fixed-point alignment, add two "0"s at the end to form an 8-bit binary number, as shown in part 303 in the figure. Use this 8-bit binary number as the first address of the register group of the LUT to obtain the value in the register. Among them, the 6 bits are the first difference and 2 -K multiples of.
[0113] 304: Confirm that the first 6 bits of the binary integer intercepted after the fixed-point alignment in 303 are consistent with 2 -3 The product of the first difference is subtracted from the product to obtain the second difference, and then the sum of the M-term polynomial is calculated.
[0114] In addition, in this embodiment, a first difference between the first independent variable and the initial value is determined, and the first difference is calculated based on the difference between the first independent variable and 2 -K Specifically, first obtain the first difference and 2 -K multiples of , by combining this multiple with 2 -K The value obtained by multiplying and adding the initial value is the corresponding target value, that is, x0 = initial value + multiple · 2 -K It should be noted that when the obtained multiple is a non-integer, the smallest integer not greater than the multiple is taken as the multiple.
[0115] Step 104: Read the coefficient value and the compensation value in the register group according to the address.
[0116] After confirming the address of the register group, the address of the register group can be directly accessed to obtain the data stored in the four registers in the register group, including the coefficient value and the compensation value.
[0117] For example, when the second group of registers is known according to the address of the register group, the values stored in the four registers in the second group of registers are read successively, that is, Tanh (0.0625+2 -3 )、Tanh`(0.0625+2 -3 )、Tanh``(0.0625+2 -3 ) / 2 and △max(0.0625+2 -3 ). Among them, Tanh(0.0625+2 -3 )、Tanh`(0.0625+2 -3 ) and Tanh``(0.0625+2 -3 ) / 2 is the coefficient value, △max(0.0625+2 -3 ) is the compensation value. Step 105: Determine the sum of the first M polynomial terms through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value.
[0118] To facilitate understanding by those skilled in the art, the following first introduces how to determine the sum of the first M polynomials using the first independent variable, the target value and the coefficient value.
[0119] From the above Taylor expansion, we can see that the polynomial of Taylor expansion also includes (x-x0) 0 、(x-x0)、(x-x0) 2 .
[0120] Therefore, it is necessary to determine the second difference between the first independent variable and the target value, that is, (x-x0), and determine the sum of the first M polynomials based on the second difference and the coefficient value.
[0121] Specifically, the sum of the first M polynomials is obtained by the following formula:
[0122]
[0123] Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value of the M coefficient values. In this embodiment, the activation function is taken as the Tanh function, and M=3 is selected. Therefore, the above formula can be converted to:
[0124]
[0125] Then the sum of the first M polynomials in the Taylor expansion can be determined.
[0126] Furthermore, (x-x0) is the second difference, that is, the second difference is equal to (x-x0). From step 103, it can be seen that x0=initial value+multiple·2 -K , that is, the second difference = x-(initial value + multiple · 2 -K ) = x-initial value-multiple · 2 -K = first difference - multiple · 2 -K Since the first difference has been obtained in advance, in order to simplify the calculation, the second difference can be directly obtained by calculating the first difference and the multiple and 2 -K The difference of the products is obtained.
[0127] The following describes how to determine the approximate value of the remainder using the first independent variable, the target value, and the compensation value.
[0128] determining a compensation coefficient according to the first independent variable and the target value, and determining an approximate value of the remainder by using the compensation value and the compensation coefficient;
[0129] Wherein, the compensation coefficient is not positively correlated with the M.
[0130] Specifically, the compensation coefficient is as follows:
[0131]
[0132] Where N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
[0133] This embodiment does not limit the specific value of N. For ease of understanding, N=5 is used as an example for introduction below. When N=5 and M=3, the compensation coefficient is finite, that is, 32 values. It should be noted that when the compensation coefficient is 32, the 32nd value is 1. In another case, the supplementary coefficient can also be 31. The compensation coefficient can also be stored in a register, then it is necessary to add a register to each group of registers. In order to save the compensation coefficient, each group of registers includes 5 registers; or a new register table can be created, and the compensation value can be stored separately in another register table for query use. In some other embodiments, it may not be saved in the register. After determining the fixed-point alignment and obtaining the 17-bit number, the compensation coefficient is calculated in real time. This embodiment does not limit this. The above introduction is only an optional solution.
[0134] After the compensation coefficient is obtained, the approximate value of the remainder can be determined according to the compensation coefficient.
[0135] The approximate value of the remainder is obtained by the following formula:
[0136]
[0137] Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in the one register.
[0138] In order to calculate the remainder of the Taylor expansion, the following is defined in this embodiment:
[0139]
[0140] Among them, Δ max (x) is the compensation value of the remainder approximation, which is stored in the register, x i is the target value corresponding to the first independent variable x, Δx max Indicates the maximum distance between the independent variable x and the corresponding target value, which can be expressed as 2 -K -2 -K-N In this embodiment, when K=3 and N=5, Δx max =2 -3 -2 -8 .
[0141] See also Figure 4 , this figure is another computer calculation logic diagram provided by this application.
[0142] As shown in part 302 of the figure, the last 11 bits of the 17 bits after the fixed-point alignment are selected, and the first N bits of the last 11 bits are defined as θ. For example, when N=5, the first 5 bits of the last 11 bits of the 17 bits after the fixed-point alignment are taken as θ, as shown in part 401 of the figure. Once the 17 bits are determined, the values of the first M terms of the polynomial and the approximate value of the remainder can be uniquely determined.
[0143] The figure shows that the compensation values are stored separately in a register table, so only 32 additional registers are needed.
[0144] Step 106: Determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder.
[0145] After obtaining the sum of the first M polynomial terms and the approximate value of the remainder, the function value of the activation function at the first independent variable can be obtained by adding these two values.
[0146] Specifically, the function value of the activation function at the first independent variable is determined according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula:
[0147] f(x)=f lut (x)+Δ(x)
[0148] Wherein, f(x) is the function value of the activation function at the first independent variable.
[0149] When the activation function is the Tanh function, the above formula is transformed into:
[0150] tanh(x)=tanh lut (x)+Δ(x)
[0151] Wherein, tanh(x) is the function value of the activation function at the first independent variable.
[0152] The solution introduced in the above embodiment takes M=3 as an example, M may also be 2, and the following description will take M=2 as an example.
[0153] When M = 2, it is necessary to replace the first M polynomials with The remaining registers are moved to the remainder, thereby reducing the use of one register in each group of registers, that is, a total of 40 registers are reduced.
[0154] Because the compensation value and compensation coefficient are used to correct the approximate value of the remainder, the use of 40 registers is reduced while ensuring that the error meets the requirements, thereby reducing the hardware area by nearly 20%. When M=2, two multiplications of the quadratic term are reduced, and one addition operation of the sum is reduced, thereby reducing the calculation delay by 1 / 3, and the benefits are obvious.
[0155] The specific calculation process is similar to that when M=3, and will not be repeated here.
[0156] The Taylor expansion polynomial also includes (x-x0) 0 、(x-x0)、(x-x0) 2 In addition, in some embodiments, (x-x0) can also be calculated by shifting.
[0157] Specifically, the last 11 bits after the 17-bit fixed-point alignment are determined in the above manner. If there are one or more "0"s at the beginning of the first position of the 11-bit number, it is shifted left by n bits, and after the shift, n "0"s are added to the last position. For example, when the 11-bit binary number is 001 1001 1001B, there are two consecutive "0"s B at the first position, so it is shifted left by 2 bits, and n = 2 at this time. After the shift, it is 1 1001 1001B, and then 2 "0"s are added to the last position to get 110 0110 0100B. Remove the first digit on the left to get the last 10 digits, that is, the fraction (the decimal part of a 16-bit half-floating point number); then calculate 14-nM as the exponent (the exponent part of a 16-bit half-floating point number), where n=2, M=3, then 14-nM=9, the binary number corresponding to 9 is 1001B, 1001B is less than 5 digits, and at this time, it is necessary to fill "0" on the left to full 5 digits, and finally get the exponent. Since the target value corresponding to the first independent variable obtained in this embodiment is not greater than the first independent variable, that is, the result of (x-x0) is non-negative; therefore, the sign part of the 16-bit half-floating point number can be ignored in the above calculation, and the default is 0B. Finally, the three parts are spliced to obtain the 16-bit value of (x-x0), that is, 0 01001 1001100100B.
[0158] In addition, when M=2, the present application embodiment also provides another computer calculation logic diagram, see Figure 5A , the figure and Figure 4 In comparison, only one register table is needed to complete the determination of the compensation coefficient. As shown in part 302 of the figure, in the 17-bit number obtained after fixed-point alignment, the last 11 bits of the 17 bits are selected, and then the first 5 bits of the last 11 bits are directly right-filled with "11", which is At the address of the register table, as shown in 701 in the figure. It should be noted that when θ is greater than 31, the content of the subsequent corresponding register table address will be empty. Compared with using two register tables, merging the two register tables into one register table can avoid the waste when the content of the register address is empty when M=2, and at the same time reduce the number of register tables. The method of determining the sum of the M-term polynomial is similar to 301-304, except that when M=2, when reading the value of the register group, only 3 values are read continuously, for example, reading Figure 5A "Tanh(0.0625)", Tanh`(0.0625) and △max(0.0625), while the fourth value "0" is read alone when determining the compensation coefficient.
[0159] The above introduces the activation function processing method based on hardware chips. The following introduces its internal processing logic from the perspective of integrated circuits.
[0160] See also Figure 5B , which is a processing logic diagram of an integrated circuit provided by this application.
[0161] The processing logic may be used in a processor of an integrated circuit, such as a CPU, etc. The processing logic includes the following steps:
[0162] Step 501: Get the first independent variable x.
[0163] For example, when calculating the function value of the Tanh function when x is 0.325, x exists as a binary number inside the processor and is represented by a 16-bit half-floating point number.
[0164] Step 502: After saving the first left digit of x, S is set to 0B. The variable used for saving is recorded as S for ease of understanding.
[0165] Among them, the first left bit is the sign part. Since tanh(x) and x have the same sign, that is, the positive and negative relationship is the same, the sign bit can be defaulted to 0B in the intermediate process of tanh(x) calculation, and finally the sign bit of x is assigned to tanh(x) to get the final result.
[0166] Step 503: Is x≤0.0625 true? If yes, go to step 504; if no, go to step 505.
[0167] Step 504: When x≤0.0625, the Tanh function takes the value x, so x is returned.
[0168] Step 505: Continue to determine whether x≥5.0625? If yes, execute step 506; if no, execute step 507.
[0169] Step 506: When x≥5.0625, the Tanh function takes the value of 1, so return 1.
[0170] Step 507: Calculate x-0.0625. The specific calculation process can refer to the method introduced in the above embodiment, which will not be repeated here.
[0171] Step 508: 17-bit fixed-point alignment processing.
[0172] Step 509: Take the first 6 bits and right-fill with 00B to determine the first address of the register group to determine the coefficient value and the compensation value.
[0173] Step 510: Take bits 7-11 to determine the compensation coefficient.
[0174] Step 511: The last 11 bits are used to determine x-x0.
[0175] The specific implementation of step 508, step 509, step 510 and step 511 is described in the above embodiment, and will not be repeated here. In addition, step 509, step 510 and step 511 are executed in no particular order.
[0176] Step 512: Calculate and obtain a 16-bit half-floating point value of the Tanh function.
[0177] Step 513: Set the left first position to S.
[0178] Step 514: Output the value of the Tanh function.
[0179] In this embodiment, the activation function processing method includes: initializing and configuring the register, including: according to the Taylor expansion of the activation function at the target value; obtaining the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; storing the coefficient values and the compensation values in the register group corresponding to the target value; obtaining the first independent variable within the preset definition domain of the activation function; determining the first difference between the first independent variable and the initial value, and calculating the compensation value according to the difference between the first difference and the 2 -K The address of the register group and the corresponding target value are determined by a multiple of ; wherein the register group includes M+1 registers; M registers respectively store the coefficient values of each term of the first M polynomials, and 1 register stores the compensation value used to calculate the approximate value of the remainder; since the first independent variable comes from a preset domain interval, and the preset domain interval is smaller than the interval from negative infinity to positive infinity, the number of required registers is reduced, thereby reducing the cost; the target values in adjacent register groups differ by 2 -K ; K and M are both positive integers; Since the target values in adjacent register groups differ by 2 -K Therefore, when the computer performs binary operations, it can directly obtain the first independent variable as 2 -K multiples without the need for addition, subtraction and if judgment, thereby simplifying the calculation process. According to the address, read the coefficient value in the register group and read the compensation value; determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value; determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder. Therefore, the technical solution provided by the present application can obtain accurate calculation results of the activation function under low cost and low complexity conditions.
[0180] Embodiment 2:
[0181] Embodiment 2 of the present application provides an activation function processing device based on a hardware chip, which is described below in conjunction with the accompanying drawings.
[0182] See also Figure 6 , which is a schematic diagram of an activation function processing device based on a hardware chip provided in this application.
[0183] The device includes: a configuration module 601 , a determination module 602 , a reading module 603 and a calculation module 604 .
[0184] The configuration module 601 is used to initialize the configuration of the register according to the Taylor expansion of the activation function at the target value; obtain the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; and store the coefficient values and compensation values in the register group corresponding to the target value.
[0185] The determination module 602 is used to obtain a first independent variable within a preset definition domain of the activation function; determine a first difference between the first independent variable and the initial value, and calculate the first difference between the first independent variable and the initial value according to the difference between the first independent variable and the initial value. -K The multiple determines the address of the register group and the corresponding target value.
[0186] The reading module 603 is used to read the coefficient value and the compensation value in the register group according to the address.
[0187] The calculation module 604 is used to determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value; and determine the function value of the activation function at the first independent variable based on the sum of the first M polynomials and the approximate value of the remainder.
[0188] Optionally, the calculation module 604 is specifically configured to determine a compensation coefficient according to the first independent variable and the target value, and determine an approximate value of the remainder by using the compensation value and the compensation coefficient.
[0189] Wherein, the compensation coefficient is not positively correlated with the M.
[0190] Optionally, the compensation coefficient is as follows:
[0191]
[0192] Where N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
[0193] Optionally, the calculation module 603 is specifically configured to obtain an approximate value of the remainder by using the following formula:
[0194]
[0195] Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in the one register.
[0196] Optionally, the calculation module 604 is specifically used to determine a second difference between the first independent variable and the target value, and determine the sum of the first M polynomials according to the second difference and the coefficient value.
[0197] Optionally, the calculation module 604 is specifically configured to obtain the sum of the first M polynomials by using the following formula:
[0198]
[0199] Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value among the M coefficient values.
[0200] Optionally, the calculation module 604 is specifically used to determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula:
[0201] f(x)=f lut (x)+Δ(x)
[0202] Wherein, f(x) is the function value of the activation function at the first independent variable.
[0203] In this embodiment, the activation function processing device includes: a configuration module, a determination module, a reading module and a calculation module; the configuration module is used to initialize the configuration of the register, according to the Taylor expansion of the activation function at the target value; obtain the coefficient values of the first M polynomial terms of the Taylor expansion and the compensation values of the approximate values of the remainder terms; store the coefficient values and the compensation values in the register group corresponding to the target value. The determination module is used to obtain the first independent variable within the preset definition domain of the activation function; determine the first difference between the first independent variable and the initial value, and calculate the compensation value according to the first difference and 2 -KThe address of the register group and the corresponding target value are determined by a multiple of ; wherein the Taylor expansion of the activation function at the target value includes the first M polynomials and the remainder; the register group includes M+1 registers; M registers respectively store the coefficient values of each term of the first M polynomials, and 1 register stores the compensation value used to calculate the approximate value of the remainder; since the first independent variable comes from a preset domain interval, and the preset domain interval is smaller than the interval from negative infinity to positive infinity, the number of required registers is reduced, thereby reducing the cost; the target values in adjacent register groups differ by 2 -K ; K is an integer, M is a positive integer; Since the target values in adjacent register groups differ by 2 -K Therefore, when the computer performs binary operations, it can directly obtain the first independent variable as 2 -K multiples without the need for addition, subtraction and if judgment, thereby simplifying the calculation process. The reading module is used to read the coefficient value in the register group and read the compensation value according to the address; the calculation module is used to determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder through the first independent variable, the target value and the compensation value; determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder. Therefore, the technical solution provided by the present application can obtain accurate calculation results of the activation function under low cost and low complexity conditions.
[0204] Embodiment three:
[0205] Embodiment 3 of the present application further provides an integrated circuit, comprising the device introduced in the above embodiments.
[0206] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and integrated circuit embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and integrated circuit embodiments described above are merely schematic, and the units and modules described as separate components may or may not be physically separated. In addition, some or all of the units and modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0207] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0208] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present application without departing from the scope of the technical solution of the present application, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of protection of the technical solution of the present application.
Claims
1. An activation function processing method based on a hardware chip, characterized in that: The method comprises: Initializing and configuring the registers, including: according to the Taylor expansion of the activation function at the target value; obtaining the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; storing the coefficient values and the compensation values in the register group corresponding to the target value; Obtaining a first independent variable within a preset definition domain of the activation function; Determine a first difference between the first independent variable and the initial value, and calculate the difference between the first difference and 2 -K Determine the address of the register group and the corresponding target value by a multiple of According to the address, reading the coefficient value and the compensation value in the register group; Determine the sum of the first M polynomials by using the first independent variable, the target value and the coefficient value; determine the approximate value of the remainder by using the first independent variable, the target value and the compensation value; Determining a function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder; The approximate value of the remainder is obtained by the following formula: Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in a register, N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
2. The method according to claim 1, characterized in that A second difference between the first independent variable and the target value is determined, and the sum of the first M polynomials is determined based on the second difference and the coefficient value.
3. The method according to claim 2, characterized in that The sum of the first M polynomials is obtained by the following formula: Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value among the M coefficient values.
4. The method according to any one of claims 1 to 3, characterized in that: The function value of the activation function at the first independent variable is determined according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula: f(x)=f lut (x)+Δ(x) Wherein, f(x) is the function value of the activation function at the first independent variable.
5. An activation function processing device based on a hardware chip, characterized in that: include: Configuration module, determination module, reading module and calculation module; The configuration module is used to initialize the configuration of the registers, according to the Taylor expansion of the activation function at the target value; obtain the coefficient values of each term of the first M polynomials of the Taylor expansion and the compensation values of the approximate values of the remainder terms; store the coefficient values and the compensation values in the register group corresponding to the target value; The determination module is used to obtain a first independent variable within a preset definition domain of the activation function; determine a first difference between the first independent variable and an initial value, and calculate a first difference between the first independent variable and an initial value according to the first difference and 2 -K Determine the address of the register group and the corresponding target value by a multiple of The reading module is used to read the coefficient value and the compensation value in the register group according to the address; The calculation module is used to determine the sum of the first M polynomials through the first independent variable, the target value and the coefficient value; Determine an approximate value of the remainder by using the first independent variable, the target value and the compensation value; determine a function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value of the remainder; The calculation module is specifically used to obtain the approximate value of the remainder through the following formula: Among them, Δ(x) is the approximate value of the remainder, Δ max (x) is the compensation value stored in a register, N is a positive integer, θ is a non-negative integer, and θ is less than or equal to (2 N -1), θ is positively correlated with the multiple.
6. The device according to claim 5, characterized in that The calculation module is specifically used to determine a second difference between the first independent variable and the target value, and determine the sum of the first M polynomials according to the second difference and the coefficient value.
7. The device according to claim 6, characterized in that The calculation module is specifically used to obtain the sum of the first M polynomials through the following formula: Among them, f lut (x) is the sum of the M-term polynomials, x0 is the target value, (x-x0) is the second difference, is the Mth value among the M coefficient values.
8. The device according to any one of claims 5 to 7, characterized in that: The calculation module is specifically used to determine the function value of the activation function at the first independent variable according to the sum of the first M polynomials and the approximate value, which is specifically obtained by the following formula: f(x)=f lut (x)+Δ(x) Wherein, f(x) is the function value of the activation function at the first independent variable.
9. An integrated circuit, characterized in that: The invention comprises the device according to any one of claims 5 to 8.
