A reconfigurable activation function hardware device adapted for deep learning hardware accelerators

Abstract

The application provides a reconfigurable activation function hardware device suitable for deep learning hardware accelerator, comprising a function type judging unit, a ReLU calculation unit, a simplified function calculation unit, a variable precision unit and an optimized function calculation unit. The application makes full use of the correlation between different nonlinear activation function calculation expressions, can realize approximate calculation of nine commonly used activation functions of neural networks, i.e. ReLU function, ReLU6 function, PReLU function, Leaky ReLU function, Sigmoid function, Tanh function, Swish function, H-Sigmoid function and H-Swish function, thereby adapting to multifunctional deep learning hardware accelerator, achieving a good balance between calculation resources and approximate accuracy, and having the characteristics of high calculation efficiency, flexibility, reconfigurability and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to a reconfigurable activation function hardware device suitable for deep learning hardware accelerator. BACKGROUND

[0002] Nonlinear Activation Function (NAF) is an important component of Deep Neural Network (DNN). Due to its complex arithmetic operation, it brings key challenges to the efficient hardware implementation of DNN. The problem solved by the present application is how to design a flexible and reconfigurable hardware acceleration architecture for various activation functions, which can adapt to multi-functional DNN accelerator and be efficiently implemented on FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) platform.

[0003] DNN accelerator has become a research hotspot in academia and industry in recent years, and nonlinear activation function is an important component of deep neural network structure. At present, emerging deep learning (DL) technology, such as neural architecture search (NAS), is proposed to find the optimal DNN structure for various practical deep learning tasks. In order to achieve high precision of DNN design, various NAFs are involved in emerging computing operations, such as Swish function in Squeeze-and-Excitation (SE) block. The optional activation layer in the predefined NAS search space also changes from single ReLU function to various nonlinear functions, including Sigmoid function and Tanh function. In addition, the emerging DNN structure often contains multiple nonlinear activation functions, which puts high requirements on general DNN accelerator. However, in hardware design, directly replacing nonlinear function with rough simplified function or ignoring these complex nonlinear functions will cause unacceptable precision loss. Therefore, for deep learning system, a general computing module that can be converted into different nonlinear activation functions is urgently needed.

[0004] Nonlinear activation function is widely used in the field of computer vision tasks. According to the relevance of mathematical representation of different functions, common NAFs can be roughly divided into two categories: one is ReLU series, and the other is Sigmoid, Swish, Tanh series, such as Figure 1As shown. In order to facilitate hardware implementation, in the lightweight convolutional neural network (Convolutional Neural Network, CNN), the Sigmoid function and the Swish function are roughly simplified into H-Sigmoid function and H-Swish function, but the network needs to be retrained or fine-tuned to ensure that the accuracy of the network is not affected.

[0005] Prior art solutions:

[0006] In order to efficiently implement the nonlinear activation function, the approximation scheme can be divided into the following types:

[0007] (1) The method based on look-up table (Look-Up-Table, LUT) uses LUT or storage unit to store the sampling value of NAF. However, using this method to approximate more NAF or obtain higher approximation accuracy will cause significant increase of storage resources.

[0008] (2) Piecewise linear (Piecewise Linear, PWL) approximation method divides the input range into several segments, and uses a linear function to fit the curve of each segment. Increasing the approximation accuracy can be achieved by increasing the number of PWL segments or using non-uniform distribution of PWL segments, but this also increases the difficulty of curve segmentation and linear fitting.

[0009] Similar scheme 1 (reference: X. Feng, Y. Li, Y. Qian, J. Gao, W. Cao, and L. Wang, “A high-precision flexible symmetry-aware architecture for element-wise activation functions,” in Proc. Int. Conf. Field Programmable Technol. (ICFPT), 2021, pp. 1-4): Using piecewise linear approximation method to efficiently support multiple nonlinear functions, nonlinear calculation is divided into multiple linear function form, but the calculation logic in the module is fixed.

[0010] (3) Use iterative methods such as coordinated rotation digital computer (Coordinated Rotation Digital Computer, CORDIC) and polynomial approximation methods such as Taylor's theorem to implement NAF, the former has long delay time due to repeated iteration operations, and the latter needs more hardware resources due to multiple cascaded multiplication and addition operations.

[0011] Similar scheme 2: Using Newton iteration method, optimize the design of a single nonlinear function.

[0012] In addition to the above-mentioned shortcomings of each scheme itself, the prior art has the following problems:

[0013] (1) Most of the work (including similar scheme 2) are optimized for a single nonlinear activation function, and the correlation between different nonlinear functions is not fully utilized, which cannot meet the high requirements of reconfigurability and flexibility of efficient DNN accelerators.

[0014] (2) In particular, for similar scheme 1, a corresponding segmentation fitting software model needs to be designed to segment different nonlinear functions and provide linear fitting support for the scheme; the correlation between different functions is not fully utilized, and it is simply converted to segmented linear calculation, the underlying design is fixed, and a fixed number of calculation cycles are required for the calculation of complex functions and simple functions, which will cause unnecessary delay for system-level design. SUMMARY

[0015] The present application aims to solve the problem that most existing nonlinear activation function NAF designs only focus on one type of activation function (with a dedicated hardware architecture), which cannot adapt to multi-functional deep neural network DNN accelerators. The present application utilizes the correlation between different nonlinear activation functions NAF, based on the optimized relationship between the initial approximation value of the activation function and its reciprocal function, designs a corresponding calculation process, and merges and reuses the calculation logic and data flow of certain nonlinear activation functions NAF to minimize hardware consumption. The present application also provides a variable precision unit that can dynamically adjust the approximation precision of the function to adapt to the needs of different scenarios.

[0016] The present application specifically provides a reconfigurable activation function hardware device for deep learning hardware accelerator, comprising a function type judgment unit, a ReLU calculation unit, a simplified function calculation unit, a variable precision unit and an optimized function calculation unit.

[0017] The function type judgment unit dynamically selects the calculation path of the activation function according to the input data x to be calculated and the type of activation function to be calculated.

[0018] The ReLU calculation unit determines the calculation interval to which the input data to be calculated belongs according to the sign bit of the data to be calculated, and performs different calculation operations according to the type of activation function, realizes the ReLU series activation function through data shift operation or direct data assignment, and the ReLU series activation function includes ReLU function, ReLU6 function, Leaky ReLU function.

[0019] The simplified function calculation unit obtains the calculation result of the H-Sigmoid activation function as an initial approximation value of the Sigmoid function calculation, and simultaneously realizes the PReLU function by multiplexing the multiplier in the simplified function calculation unit and the corresponding parameters in the PReLU function calculation.

[0020] The variable precision unit approximates the reciprocal function 1+e -x of the Sigmoid to obtain an approximate value appro_sig_rec of the reciprocal function of the Sigmoid, wherein the approximate precision of the reciprocal function of the Sigmoid can be dynamically adjusted by changing the numerical value of the parameter L based on the precision adjustable scheme. According to the precision adjustable scheme, the parameter L can directly affect the calculation parameter value, the number and size of the divided subintervals in the variable precision unit. Meanwhile, the calculation result of the whole part 1-e -2x of the Tanh function is obtained by multiplexing the calculation logic based on the e exponential, and the approximate values of the reciprocal function of the Sigmoid and the whole part of the Tanh function are provided;

[0021] The optimized function calculation unit performs the following calculation on the initial approximation value (the H-Sigmoid calculation result) and the reciprocal function approximation value (the approximate value appro_sig_rec of the reciprocal function of the Sigmoid) to obtain the optimized approximation value of the Sigmoid function:

[0022] sigmoid(x)=2×H-Sigmoid(x)-H-Sigmoid(x) 2 ×appro_sig_rec

[0023] The optimized function calculation unit includes a first multiplier, a second multiplier and a third multiplier.

[0024] The first multiplier is used to calculate the square of the H-Sigmoid calculation result, the second multiplier is used to calculate the multiplication of the square value of the H-Sigmoid calculation result and the approximate value appro_sig_rec of the reciprocal function of the Sigmoid, and the third multiplier is used to realize the multiplication of the whole part of the H-Swish function and the Tanh function and the calculation result of the Sigmoid activation function.

[0025] Wherein, the appro_sig_rec is the approximation value of the Sigmoid inverse function, the calculation result of the H-Sigmoid activation function is recorded as the intermediate result A1, based on the above formula, the multiplication of the intermediate result A1 and the intermediate result A1 is performed in the No.1 multiplier of the optimization function calculation unit, the multiplication of the square value of the intermediate result A1 and the approximation value of the Sigmoid inverse function appro_sig_rec is performed in the No.2 multiplier, and then the result obtained by the No.2 multiplier is subtracted from the 2x A1 result obtained by the shift operation (by adding the complement code), so as to finally obtain the approximation value of the optimized Sigmoid function;

[0026] The H-Swish function and the Tanh function are realized by multiplexing the No.3 multiplier inside the optimization function calculation unit; the Tanh function and the Swish function are realized by multiplying the Sigmoid function calculation result with the corresponding integral part of the non-linear function, wherein the integral part of the Tanh function is 1-e -2x , and the integral part of the Swish function is x.

[0027] The simplified function calculation unit inside includes a No.4 multiplier, and the simplified function calculation unit specifically performs the following operations:

[0028] According to the H-Sigmoid activation function expression:

[0029]

[0030] The PReLU activation function expression is:

[0031]

[0032] Wherein, a is a constant less than 1.

[0033] According to the H-Sigmoid activation function expression and the PReLU activation function expression, the H-Sigmoid activation function needs the No.4 multiplier to realize the function calculation when the input range of the to-be-calculated data x is (-3, 3), and the PReLU activation function needs the No.4 multiplier to realize the function calculation when the input range of x is (-∞, 0), the No.4 multiplier is used inside the simplified function calculation unit to complete the operation of the H-Sigmoid function and the a x operation of the PReLU de, and the parameter multiplied by x is changed according to the calculation requirement, so as to complete the calculation, wherein a is a constant less than 1.

[0034] The variable precision unit specifically performs the following operations:

[0035] The original function f(x) is converted into the exponential function with base 2:

[0036]

[0037] Let input variable P = log2e x, convert f(x) to f # (P) = f(ln2 P) = 2 -P = 2 -(Pm+Pn) , where f # (P) represents function calculation with input independent variable P, input variable P is divided into integer part Pmand decimal part Pn, in this way, for e -x approximate calculation is converted to shift operation (2 -Pm calculation processing) and approximate calculation of 2 -Pn , the input range of approximate calculation is reduced to 0 to ln2, 0 < Pn≤ 1; According to the above reasoning, it can be seen that the data range that needs to be approximated is converted from approximate calculation of e -x , x is any data, to approximate calculation of 2 -Pn , the range of Pnis (0, 1), and the conversion to the x input domain is between 0 and ln2;

[0038] Based on the Taylor expansion formula, the expansion of function f(x) at x = x0is expressed as (because x is not much different from x0, the high-order polynomials of the expansion are omitted here):

[0039]

[0040] Where x0is any number;

[0041] Introduce parameter L, L is a positive integer greater than or equal to 0, divide the input range of function f(x) (0, ln2] into 2 L segments, and the length of each segment is 2 -L × ln2, the sub-interval after cutting is expressed as [n·2 -L × ln2, (n+1)·2 -L × ln2), where n is a positive integer in [0, 2 L ];

[0042] Let x0= ln2 P0, δx = 2 -L × ln2, get f(x) = f(ln2 P0) + 2 L × (f(ln2 (P0+2 -L )) - f(ln2 P0)) × (log2e x - P0);

[0043] For f # (P), let P0= n·2 -L , then in the input range [n·2 -L , (n+1)·2 -L ], we get:

[0044]

[0045] Since α and β are both computational parameters related to L, the approximate accuracy of the exponential function can be adjusted by the parameter L.

[0046] The following calculation is achieved through shift and addition operations in shift and addition unit 1 within the variable precision unit: P = log₂e·x, based on the approximation of log₂e: 1.4735 = 1 + 0.5 - 0.0625 = 2 0 +2 -1 -2 -4 The transformed data P to be calculated is obtained, where the integer part Pm of P participates in the subsequent shift operation, and the fractional part Pn completes the approximate calculation in formula (1).

[0047] f is calculated in shift-add operation unit 2 inside the variable precision unit. # (P) = α + P >> β1 + P >> β2, where β1 and β2 are positive integers satisfying β = 2 -β1 +2 -β2 >> indicates a shift operation;

[0048] Based on the transformation relationship between the independent variables of a function: P = log₂e·x, for the function f # (P), the subintervals after cutting are represented as [n·2] -L ,(n+1)·2 -L Given the parameter L, the subinterval to which the input Pn belongs is determined based on the high-order value of the fractional part Pn, thus determining the specific value of n and obtaining the parameter values ​​of α, β1, and β2, thereby realizing the calculation of formula (1). f(x) is then calculated using f... # After (P) is represented, based on the transformation relationship between x and P, the segmented subintervals are changed from the original [n·2] within the computational domain of x. -L ×ln 2,(n+1)·2 -L ×ln 2), transformed into [n·2] within the computational domain of P. -L ,(n+1)·2 -L That is, by transforming the computational domain (x becomes P), the partitioned computational subintervals are transformed into [n·2]. -L ,(n+1)·2 -L ).

[0049] The device completes the calculation of the reconfigurable activation function by performing the following steps:

[0050] Step 1: Input the data to be calculated and the type of the activation function to be calculated into the reconfigurable activation function hardware device;

[0051] Step 2: The function type determination unit dynamically selects the computational operation to be performed based on the type of the activation function.

[0052] The system determines whether the input data is a ReLU function, a ReLU6 function, or a Leaky ReLU function. If it is, the ReLU calculation unit determines the calculation interval of the input data and obtains the calculation result of the corresponding activation function. For example, if the activation function is a ReLU function, and the input data is in the interval (-∞, 0), the output result is 0; if the input data is in the interval (0, +∞), the output result is the input value. The "x×0.125" operation in the Leaky ReLU function is implemented by right-shifting the input by three bits, saving hardware resources.

[0053] Determine if it is a PReLU function. If it is, determine the calculation interval to which the input data to be calculated belongs. When the input is less than 0, the x×α operation is implemented by reusing the No. 4 multiplier inside the simplified function calculation unit, where α is a constant less than 1, thus obtaining the calculation result of the PReLU function.

[0054] Determine if it is an H-Sigmoid function. If so, determine the calculation interval of the input data to be calculated. If the input data is in the interval (-3, 3), the calculation is performed using multiplier 4 and adder within the simplified function calculation unit. The operation yields the calculation result of the H-Sigmoid activation function, denoted as intermediate result A1;

[0055] Determine if it is an H-Swish function. If so, based on the calculation expressions of the H-Sigmoid function and the H-Swish function, the H-Swish function is expressed as H-Swish(x) = x × H-Sigmoid(x). Based on the calculation result of the H-Sigmoid activation function (intermediate result A1), the multiplier No. 1 inside the optimization function calculation unit is reused to perform the multiplication calculation of the input data and the H-Sigmoid calculation result, and the calculation result of the H-Swish activation function is obtained.

[0056] Determine if it is the Sigmoid function. If so, since the H-Sigmoid activation function is a hardware-friendly simplification of the Sigmoid activation function, it can be considered as an initial approximation of the Sigmoid function. Based on the optimization relationship between the calculation result of the H-Sigmoid function (initial approximation) and the approximate value appro_sig_rec of the inverse Sigmoid function, the optimized approximation of the Sigmoid function is obtained based on the following formula:

[0057] Sigmoid(x) = 2×H - Sigmoid(x) - H - Sigmoid(x) 2 ×appro_sig_rec

[0058] First, calculate the approximation value appro_sig_rec of the Sigmoid reciprocal function. The Sigmoid reciprocal function is expressed as 1 + e -x , because the hardware implementation of the exponential unit e is very complex, so it is converted to the hardware implementation of the exponential unit with base 2, that is Let P = log2e·x. Among them, P is divided into an integer part Pm and a fractional part Pn, then it is converted to 1 + 2 -(Pm+Pn) . In this way, for e -x the approximate calculation is transformed into a shift operation (the calculation process of 2 -P ) and the approximate calculation of 2 -Pn . Then the input range of the approximate calculation is reduced to between 0 and ln2, 0 < Pn ≤ 1; the optimized approximate value of the Sigmoid function is the calculation result of the Sigmoid activation function;

[0059] For the calculation of 2 -Pn it is based on the Taylor expansion formula. Convert 2 -Pn to the calculation form of a polynomial, and divide the calculation interval of Pn. Calculate and optimize the polynomial of 2 -Pn in each calculation interval, and convert the polynomial to the form of α + Pn >> β1 + Pn >> β2. Among them, α, β1, and β2 are all calculation parameters related to the interval division parameter L, and specific values are obtained according to their respective calculation formulas; >> represents a shift operation;

[0060] Judge whether it is a Swish function. If so, because according to the calculation expressions of the Sigmoid function and the Swish function, the Swish function is expressed as Swish(x) = x × Sigmoid(x). Therefore, based on the Sigmoid calculation result, that is, the intermediate result A2, the multiplication calculation of the input data and the Sigmoid calculation result is realized through the multiplier No. 3 inside the optimization function calculation unit, and the calculation result of the Swish activation function is obtained;

[0061] Judge whether it is a Tanh function. If so, because according to the calculation expressions of the Sigmoid function and the Tanh function, the Tanh function is expressed as Tanh(x) = (1 - e -2x ) × Sigmoid(2x). Because 1 - e -2x and the Sigmoid reciprocal function 1 + e -x are only different in addition, subtraction, and shift operations, so 1 - e -2xThe function can reuse e in the variable precision unit. -x The computational logic is implemented in shift-addition operation unit 3 within the variable precision unit. Based on the Sigmoid calculation result, i.e., the intermediate result A2, the multiplier No. 3 within the optimized function calculation unit is used to perform the multiplication of the input data and the Sigmoid calculation result to obtain the calculation result of the Tanh activation function.

[0062] Step 3: Based on the calculation operations in Step 2, obtain and output the calculation results of different types of nonlinear activation functions.

[0063] In step 2, the specific steps for calculating the reciprocal of the sigmoid function include:

[0064] The data to be calculated is input into the variable precision unit. First, based on the relationship between the exponential function e and the base-2 exponential function, the input data is multiplied by log₂e. Since the approximate value of log₂e is 1.4735, which can be represented in binary as 1.0000 + 0.1000 - 0.0001, this multiplication operation can be converted into a data shift and addition operation, implemented in the shift and addition operation unit 1 within the variable precision unit. At this point, the input data becomes temporary data B1. Using the decimal point of temporary data B1 as the boundary, it is divided into temporary data B2 for the integer part and temporary data B3 for the decimal part. Temporary data B2 corresponds to Pn, and temporary data B3 corresponds to Pm. Under the given parameter L, the calculation interval to which temporary data B3 belongs is flexibly determined by the high-order data of temporary data B3, determining the value of n. Substituting this into the calculation formulas for α, β₁, and β₂ yields the corresponding specific value. This is implemented in the shift and addition operation unit 2 within the variable precision unit, resulting in 2... -Pn The calculation result is denoted as temporary data B4; based on the calculation formula 1+2 -(Pm+Pn) The temporary data B4 is shifted and added according to the specific value of the temporary data B2. This is implemented in the shift and addition operation unit 3 to obtain the approximate value (appro_sig_rec) of the final Sigmoid reciprocal function.

[0065] The result of the H-Sigmoid activation function calculation is denoted as intermediate result A1, based on the formula Sigmoid(x) = 2 × H-Sigmoid(x) - H-Sigmoid(x). 2×appro_sig_rec, in the first multiplier of the optimization function calculation unit, intermediate result A1 is multiplied by intermediate result A1. In the second multiplier, the square of intermediate result A1 is multiplied by the approximation value appro_sig_rec of the reciprocal Sigmoid function. Then, the result obtained from the second multiplier is subtracted from the intermediate result A1 (2×A1 obtained through a shift operation). Finally, the optimized approximation value of the Sigmoid function is obtained. The optimized approximation value of the Sigmoid function is the calculation result of the Sigmoid activation function. Since this result can be reused in the calculation of other activation functions, it is denoted as intermediate result A2.

[0066] Because the Sigmoid function is centrally symmetric about (0, 0.5), the value for a negative input is obtained using the formula Sigmoid(x) = 1 - Sigmoid(-x). In conjunction with the absolute value module in the simplified function calculation unit, the negative input is converted to a positive input. Based on the calculation result for a positive input, the shift and addition operation unit 4 within the variable precision unit, combined with the shift and addition operations in the formula, yields the corresponding calculation result for a negative input.

[0067] This scheme utilizes the correlation between different nonlinear activation functions (NAFs) to provide flexible and efficient hardware support for ReLU, ReLU6, Leaky ReLU, PReLU, Sigmoid, H-Sigmoid, Swish, H-Swish, and Tanh functions. Other functions that can be represented or approximated as containing the aforementioned nonlinear activation functions can also be supported by this scheme. For example, the GELU function can be approximated by the formula f(x) = x × Sigmoid(1.702·x), and the calculation scheme of this invention can support the Sigmoid function after a shift transformation of the input.

[0068] Beneficial effects: (1) Based on the optimization relationship between the function itself and its reciprocal function, this scheme further derives the correlation between the calculation expressions of different nonlinear activation functions, designs the corresponding calculation process, merges and reuses the calculation logic and data flow of some nonlinear activation functions NAF, and designs a reconfigurable nonlinear activation function module.

[0069] (2) In order to achieve a good balance between function approximation accuracy and hardware logic overhead, this solution designs a variable precision unit to dynamically adapt to different scenario requirements. Attached Figure Description

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0071] Figure 1 This is a schematic diagram of a nonlinear activation function.

[0072] Figure 2 This is a schematic diagram of the architecture of a hardware device for reconfigurable activation functions.

[0073] Figure 3 This is a flowchart of the hardware device computation for reconfigurable activation functions.

[0074] Figure 4 This is a schematic diagram of an adjustable precision scheme. Detailed Implementation

[0075] (I) Theoretical Part of Hardware Device for Reconfigurable Activation Functions

[0076] For a nonlinear function, its expression y = f(x) (where x is the independent variable, y is the dependent variable and y ≠ 0) can be equivalent to the expression To facilitate subsequent derivations and to distinguish between different functions, we can use the function f. * The expression can be represented by (y) = 1 / y - 1 / (f(x)). Applying Newton's iterative method, we can base the expression on the initial approximation y. i We obtain equation f * A new approximation y(y) = 0 j , is represented as:

[0077] f * (y j )=f′ * (y i )×(y j -y i )+f*(y i )=0 (1)

[0078]

[0079] Where 1 / (f(x)) is the reciprocal of y. Therefore, based on the above formula, an initial approximation value y can be obtained. i Optimized approximation y j Considering that in actual hardware implementation, 1 / (f(x)) should be an approximation rather than an exact value, we choose 1 / y. x This is an approximation of 1 / (f(x)), for ease of subsequent derivation. We assume an error Δ. j Δ i and Δ x Between the true value y and its approximation (y >> Δ) j ,Δ i ,Δ x To satisfy equation y j =y+Δ j ,yi =y+Δ i ,y x =y+Δ x Therefore, formula (2) takes the form of:

[0080]

[0081] It can be seen that adjusting the approximation precision of the reciprocal of the nonlinear function can affect the deviation of the new approximation value of the nonlinear function. Therefore, taking y>0 as an example, we will discuss which conditions need to be met to optimize the initial approximation value in the following four cases.

[0082] Case 1 (Δ) i >0,Δ x >0): To improve the accuracy of the new approximation, then y-Δ i <y+Δ j <y+Δ i Based on formula (3), the inequality can be obtained. Therefore, it is necessary to satisfy |Δ x |<|Δ i This is used to optimize the initial approximation.

[0083] Case 2 (Δ) i <0,Δ x <0): The same conclusion can be obtained through derivation similar to that of case 1.

[0084] Case 3 (Δ) i >0,Δ x <0): Because y+Δ i It must be greater than y + Δ x In this case, we only need to determine y-Δ i <y+Δ j Whether or not these conditions are met, the following derivation is obtained:

[0085]

[0086]

[0087] Because Δ i >0 and y>>Δ i , Approximately 1, based on the above derivation, we can obtain |Δ x |Needs to be less than|Δ i This is used to optimize the initial approximation.

[0088] Case 4 (Δ) i <0,Δ x >0): The same conclusion can be obtained through derivation similar to that of case 3.

[0089] Therefore, as long as the error |Δ x |less than the initial approximation error|Δ i The initial approximation can then be optimized based on formula (3). In addition, assuming there is an error here... Make and y x =y+Δ x Therefore, the relationship between these two errors can be obtained as follows:

[0090]

[0091] It can be seen that the specific numerical relationship between the errors depends on y and y x The value of |y| is such that when the output value of the nonlinear activation function is between 0 and 1, then... It can be deduced that a more accurate approximation of y can be obtained by approximating the reciprocal function of the activation function. x Thus, for some nonlinear activation functions containing fractions, where the complexity of fractional calculations prevents further improvement in their approximation accuracy, while satisfying... Under certain conditions, a more accurate approximation can be obtained by approximating the reciprocal function of the activation function (or the reciprocal of the fraction in the activation function).

[0092] Because the output range of the Sigmoid function is between 0 and 1, such as Figure 1 As shown in (4), the initial approximation of the Sigmoid function can be optimized according to formulas (3) and (6). For efficient hardware implementation, this invention selects the H-Sigmoid function as the initial approximation of the Sigmoid function in the design. Therefore, the calculation formula of the Sigmoid function can be expressed as:

[0093] Sigmoid(x)=2×H-Sigmoid(x)-H-Sigmoid(x) 2 ×appro_sig_rec

[0094] Here, appro_sig_rec is an approximation of the reciprocal of the Sigmoid function.

[0095] Other functions can also be transformed into the form of multiplying the algebraic part by the fractional part:

[0096]

[0097]

[0098]

[0099] In this way, different nonlinear activation functions can be linked together.

[0100] The activation function names and abbreviations supported by this invention are as follows:

[0101] Corrected linear unit (ReLU);

[0102]

[0103] Suppress its maximum corrected linear unit (ReLU6); (No corresponding Chinese name)

[0104]

[0105] Parametric Corrected Linear Unit (PReLU);

[0106]

[0107] Leaky ReLU with leakage correction;

[0108]

[0109] The sigmoid activation function;

[0110]

[0111] Hyperbolic tangent activation function (Tanh);

[0112]

[0113] Swish activation function (Swish); (No corresponding Chinese name)

[0114]

[0115] Hardware-friendly H-Sigmoid activation function; (No corresponding Chinese name)

[0116]

[0117] Hardware-friendly Swish activation function (H-Swish); (No corresponding Chinese name)

[0118]

[0119] (II) Hardware Design of Reconfigurable Activation Function

[0120] Figure 2The block diagram in the document provides an overall architectural overview of the reconfigurable activation function hardware device, which is mainly divided into five parts: function type determination unit, ReLU calculation unit, simplified function calculation unit, variable precision unit, and optimized function calculation unit.

[0121] The function type determination unit dynamically selects the calculation path of the activation function based on the input data x to be calculated and the type of activation function to be calculated;

[0122] In the ReLU computation unit, ReLU series activation functions, including ReLU function, ReLU6 function, and Leaky ReLU function, are implemented through data shifting operations or direct data assignment, depending on the sign bit and function type.

[0123] In the simplified function calculation unit, the calculation result of the H-Sigmoid activation function is obtained, which can be used as the initial approximation value (intermediate result A1) for the Sigmoid function calculation. At the same time, the PReLU function can be implemented by reusing multiplier No. 4 with corresponding parameters.

[0124] In variable precision units, the reciprocal function of sigmoid, 1+e... -x An approximate value for the reciprocal function is obtained through approximate calculation. The input is transformed in shift-addition unit 1, and the simplified approximate calculation is implemented in shift-addition unit 2. Combined with an adjustable precision scheme, the approximate precision of the function can be flexibly adjusted, thus achieving a good balance with hardware resources. Simultaneously, the calculation result of the integer part of the Tanh function can be obtained by reusing the calculation logic based on the exponent e. That is, shift-addition unit 3 can simultaneously provide approximate values ​​for the reciprocal function of Sigmoid and the integer part of the Tanh function.

[0125] In the optimization function calculation unit, the initial approximation (intermediate result A1) and the reciprocal function approximation (appro_sig_rec) are calculated in multipliers 1 and 2 as follows:

[0126] Sigmoid(x)=2×H-Sigmoid(x)-H-Sigmoid(x) 2 ×appro_sig_rec

[0127] The optimized approximation of the Sigmoid function is obtained (intermediate result A2). Note that the shift-addition operation unit 4, in conjunction with the absolute value module in the simplified function calculation unit, is responsible for generating the Sigmoid function result with positive and negative inputs based on the symmetry of the function. The H-Swish function can be implemented by reusing multiplier 1 of the optimized function calculation unit. The Tanh function and the Swish function can be implemented by multiplying the optimized approximation of the Sigmoid function (intermediate result A2) by the corresponding polynomial part of the nonlinear function based on multiplier 3.

[0128] By operating the data streams of various nonlinear activation functions based on different computational parts in the device of this invention, and by reusing computational logic, the burden on hardware resources can be reduced and resources can be utilized to the maximum extent.

[0129] like Figure 3 As shown, the device completes the calculation of the reconfigurable activation function by performing the following steps:

[0130] Step 1: Input the data to be calculated and the type of the activation function to be calculated into the reconfigurable activation function hardware device;

[0131] Step 2: The function type determination unit dynamically selects the computational operation to be performed based on the type of the activation function.

[0132] The system determines whether the input data is a ReLU function, a ReLU6 function, or a Leaky ReLU function. If it is, the ReLU calculation unit determines the calculation interval of the input data and obtains the calculation result of the corresponding activation function. For example, if the activation function is a ReLU function, and the input data is in the interval (-∞, 0), the output result is 0; if the input data is in the interval (0, +∞), the output result is the input value. The "x×0.125" operation in the Leaky ReLU function is implemented by right-shifting the input by three bits, saving hardware resources.

[0133] Determine if it is a PReLU function. If it is, determine the calculation interval to which the input data to be calculated belongs. When the input is less than 0, the x×α operation is implemented by reusing the No. 4 multiplier inside the simplified function calculation unit, where α is a constant less than 1, thus obtaining the calculation result of the PReLU function.

[0134] Determine if it is an H-Sigmoid function. If so, determine the calculation interval of the input data to be calculated. If the input data is in the interval (-3, 3), the calculation is performed using multiplier 4 and adder within the simplified function calculation unit. The operation yields the calculation result of the H-Sigmoid activation function, denoted as intermediate result A1;

[0135] Determine if it is an H-Swish function. If so, based on the calculation expressions of the H-Sigmoid function and the H-Swish function, the H-Swish function is expressed as H-Swish(x) = x × H-Sigmoid(x). Based on the calculation result of the H-Sigmoid activation function (intermediate result A1), the multiplier No. 1 inside the optimization function calculation unit is reused to perform the multiplication calculation of the input data and the H-Sigmoid calculation result, and the calculation result of the H-Swish activation function is obtained.

[0136] Determine if it is a Sigmoid function. If so, based on the optimization relationship between the calculation result of the H-Sigmoid function and the approximate value appro_sig_rec of the reciprocal Sigmoid function, obtain the optimized approximate value of the Sigmoid function using the following formula:

[0137] Sigmoid(x)=2×H-Sigmoid(x)-H-Sigmoid(x) 2 ×appro_sig_rec

[0138] First, calculate the approximate value of the reciprocal of the Sigmoid function, appro_sig_rec. The reciprocal of the Sigmoid function is represented as 1 + e -x Because the hardware implementation of the exponent unit e is very complex, it is converted to a hardware implementation with a base-2 exponent unit, i.e. Let P = log₂e⁻¹x, where P is divided into an integer part Pm and a fractional part Pn, then it is converted to 1 + 2. -(Pm+Pn) In this way, for e -x The approximate calculation is transformed into a shift operation (2 -Pm (calculation processing) and for 2 -Pn If approximate calculations are used, the input range for the approximate calculations is reduced to between 0 and ln2. <Pn≤1;

[0139] For 2 -Pn The calculation is based on the Taylor expansion formula, which is 2 -Pn Convert to a polynomial computational form, and divide Pn into computational intervals. For each computational interval, 2 -Pn The polynomial is calculated and optimized to transform it into the form α+Pn>>β1+Pn>>β2, where α, β1, and β2 are calculation parameters related to the interval partitioning parameter L, and their specific values ​​are obtained according to their respective calculation formulas; >> indicates a shift operation.

[0140] Determine if it is the Swish function. If so, based on the calculation expressions of the Sigmoid and Swish functions, the Swish function is expressed as Swish(x) = x × Sigmoid(x). Therefore, based on the Sigmoid calculation result, i.e., the intermediate result A2, the multiplier No. 3 inside the function calculation unit is optimized to perform the multiplication calculation between the input data and the Sigmoid calculation result, thus obtaining the calculation result of the Swish activation function.

[0141] Determine if it is a Tanh function. If it is, because according to the calculation expressions of the Sigmoid function and the Tanh function, the Tanh function is expressed as Tanh(x) = (1-e^(-x)). -2x )×Sigmoid(2x). Because 1-e -2x and the reciprocal of the Sigmoid function 1+e -x The only difference is between addition, subtraction, and shift operations, so 1-e -2x The function can reuse e in the variable precision unit. -x The computational logic is implemented in shift-addition operation unit 3 within the variable precision unit. Based on the Sigmoid calculation result, i.e., the intermediate result A2, the multiplier No. 3 within the optimized function calculation unit is used to perform the multiplication of the input data and the Sigmoid calculation result to obtain the calculation result of the Tanh activation function.

[0142] Step 3: Based on the calculation operations in Step 2, obtain and output the calculation results of different types of nonlinear activation functions.

[0143] In step 2, the specific steps for calculating the reciprocal of the sigmoid function include:

[0144] The data to be calculated is input into the variable precision unit. First, based on the relationship between the exponential function e and the base-2 exponential function, the input data is multiplied by log₂e. Since the approximate value of log₂e is 1.4735, which can be represented in binary as 1.0000 + 0.1000 - 0.0001, this multiplication operation can be converted into a data shift and addition operation, implemented in the shift and addition operation unit 1 within the variable precision unit. At this point, the input data becomes temporary data B1. Using the decimal point of temporary data B1 as the boundary, it is divided into temporary data B2 for the integer part and temporary data B3 for the decimal part. Temporary data B2 corresponds to Pn, and temporary data B3 corresponds to Pm. Under the given parameter L, the calculation interval to which temporary data B3 belongs is flexibly determined by the high-order data of temporary data B3, determining the value of n. Substituting this into the calculation formulas for α, β₁, and β₂ yields the corresponding specific value. This is implemented in the shift and addition operation unit 2 within the variable precision unit, resulting in 2... -PnThe calculation result is denoted as temporary data B4; based on the calculation formula 1 + 2 -(Pm+Pn) Perform a shift operation and an addition operation on the temporary data B4 according to the specific value of the temporary data B2, which is implemented in the shift-and-add operation unit 3 to obtain an approximate value (appro_sig_rec) of the final Sigmoid reciprocal function.

[0145] The calculation result of the H-Sigmoid activation function is denoted as intermediate result A1. Based on the formula Sigmoid(x) = 2 × H-Sigmoid(x) - H-Sigmoid(x) 2 × apprp_sig_rec, perform the multiplication of intermediate result A1 with intermediate result A1 (implemented by multiplier 1), the multiplication of the square value of intermediate result A1 with the approximate value of the Sigmoid reciprocal function (implemented by multiplier 2), and the shift operation and addition operation on intermediate result A1 in the optimization function calculation unit to obtain the calculation result of the Sigmoid activation function. Since this result can be reused in the calculations of other activation functions, it is denoted as intermediate result A2.

[0146] Because the Sigmoid function is a function symmetric about the center (0, 0.5), the value when the input is negative is obtained through the formula Sigmoid(x) = 1 - Sigmoid(-x). In cooperation with the absolute value module in the simplified function calculation unit, the negative input is changed to a positive input, and based on the calculation result of the positive input in the shift-and-add operation unit 4 inside the variable precision unit, the corresponding calculation result for the negative input is obtained by combining the shift and addition operations in the formula;

[0147] (III) Precision-adjustable scheme and design (the specific content of this part is the specific content of the calculation method in step 2 for 2 -Pn ).

[0148] According to the reciprocal expression of the Sigmoid function, the key to its hardware implementation is the approximate implementation of the e exponential function. In this scheme, the variable precision unit design is based on the Taylor series expansion and can flexibly adjust the approximation precision of the function. First, according to the characteristics of the hardware design, the original function is converted into an exponential function with base 2, expressed as Let P = log2e · x, then f(x) can be converted to f # (P) = f(ln2 · P) = 2 -P = 2 -(Pm+Pn) . Among them, the input variable P can be divided into an integer part Pm and a fractional part Pn. In this way, e -x can be realized through a shift operation and for 2 -Pn (0 < Pn ≤ 1). In this way, the input range can be reduced to between 0 and ln2.

[0149] Based on the Taylor expansion formula, the function f(x) at x = x0 is expanded as follows: This is the first-order representation of the Taylor expansion, omitting the higher-order parts. To flexibly adjust the approximation precision, a parameter L is introduced, where L is a positive integer greater than or equal to 0, uniformly dividing the input range of the function f(x) (0, ln 2] into 2... L Segments, each segment is 2 units long -L ×ln2. For example... Figure 4 As shown, the sub-intervals after segmentation can be represented as [n·2] -L ×ln2,(n+1)·2 -L ×ln2), where n is [0,2]. L A positive integer. Let x0 = ln 2 × P0, δx = 2 -L ×ln2, we get f(x)=f(ln2·P0)+2 L ×(f(ln2·(P0+2 -L ))-f(ln2·P0))×(log2e·x-P0). Therefore, for f # For (P), let P0 = n·2 -L Then, within the input range [n·2] -L ,(n+1)·2 -L )get:

[0150]

[0151] In this way, the approximate accuracy of the exponential function can be adjusted by the parameter L, avoiding the waste of resources caused by the mismatch between high theoretical accuracy and low quantization bit width.

[0152] In hardware design, according to P = log₂e·x, the approximate value of log₂e is 1.4735 = 1.0000 + 0.1000 - 0.0001 (binary). This can be obtained through... Figure 2 The shift and addition operations are implemented in the shift and addition unit 1. The integer part of the calculation result participates in the subsequent shift operation, and the fractional part completes the approximate calculation in formula (10). When the value of L is very small, the value of β can be regarded as the sum of two exponential functions, thereby further reducing the complexity of the hardware. Therefore, the calculation in the shift and addition unit 2 can be simplified to f # (P) = α + P > β1 + P >> β2, where β1 and β2 are positive integers satisfying β = 2 -β1 +2 -β2 >> indicates a shift operation.

[0153] Since the input is converted to [n·2] -L ,(n+1)·2-L Given a parameter L, the sub-interval to which the input belongs can be flexibly determined based on the high-order value of the fractional part, thus effectively reducing the complexity of the design. For example, if L is set to 2, the input range can be divided into 4 parts: (0, 1 / 4), [1 / 4, 1 / 2), [1 / 2, 3 / 4), [3 / 4, 1]. Assuming the fractional part Pn1 of the input value is 0.10xxx (binary), it can be determined that the value belongs to the sub-interval [1 / 2 (binary: 0.10), 3 / 4 (binary: 0.11)]. With n = 2, substituting n and L into formula (10), we obtain the calculated values ​​of α, β1 and β2 corresponding to this interval, and 2 -Pn1 Approximate value.

[0154] Example 1:

[0155] Taking the implementation of the Sigmoid function as an example, its implementation process is based on its simplified function, the H-Sigmoid function. The input data is processed by the simplified function calculation unit to obtain the intermediate result 1, which is the result of the H-Sigmoid function; simultaneously, the input data is processed by the variable precision unit to obtain the approximate value of the reciprocal function, i.e., 1+e. -x The approximate calculation results are as follows.

[0156] (1) The calculation in the simplified function calculation unit is the calculation implemented by the H-Sigmoid function, which can be expressed as:

[0157]

[0158] (2) In the calculation within the variable precision unit, the input value variable x is first converted to the form of variable P using P = log₂e·x. Specifically, this is achieved by taking an approximation of log₂e, 1.4735 = 1.0000 + 0.1000 - 0.0001 (binary). Figure 2 The shift-addition operation unit 1 implements the operation through shift and addition. Given the parameter L, the sub-interval to which the fractional part Pn of the input P belongs is flexibly determined based on the high-order digits of the fractional part, thus determining the specific value of n and obtaining the corresponding calculated values ​​of α, β1, and β2. These are then processed in the shift-addition operation unit 2 to obtain 2. -Pn As a result. Because e -x This can be achieved through shift operations and for 2 -Pn The calculation is used to approximate this, and by combining the integer part Pm of the input P, ​​1+e can be obtained in the shift-addition unit 3. -x The approximate calculation result is appro_sig_rec.

[0159] Therefore, based on intermediate result 1 and the approximate value of the reciprocal function, approximate optimization can be performed in the optimization function calculation unit to obtain intermediate result 2, which is the calculation result of Sigmoid. The specific calculation formula is expressed as follows:

[0160] Sig,oid(x)=2×H-Sigmoid(x)-H-Sigmoid(x) 2 ×appro_sig_rec

[0161] The calculation result of the corresponding negative input can be obtained by shifting and adding the absolute value module in the simplified function calculation unit through the shifting and adding unit 4.

[0162] When selecting 12-bit fixed-point input data for the hardware design, the interval partitioning parameter L is set to 2 to achieve a better balance between approximate accuracy and hardware design complexity. Within the input range (-8, 8), the approximate mean absolute error (MAE) is calculated to be 1.7 × 10⁻⁶. -3 The maximum absolute error (MXE) is 9.7 × 10⁻⁶. -3 The mean square error (MSE) is 7.5 × 10⁻⁶. -6 .

[0163] In the design work of the Sigmoid function (reference: Z. Qin, Y. Qiu, H. Sun, Z. Lu, Z. Wang, Q. Shen, and H. Pan, “Anovel approximation methodology and its efficient vlsiimplementation for the Sigmoid function,” IEEE Trans. Circuits Syst., II, Exp. Briefs, vol. 67, no. 12, pp. 3422–3426, 2020), under the same input settings, the MAE is 1.8 × 10⁻⁶. -3 MXE is 1.1 × 10 -2 In contrast, the method of this invention achieves better performance, supports more nonlinear activation functions, and does not incur significant resource overhead.

[0164] Compared to other designs, most of these works involve custom hardware designs for a single type of activation function, lacking the flexibility to support different types of activation functions. This makes it impossible to meet the multi-network, multi-tasking requirements of multi-functional hardware accelerator designs.

[0165] Works based on piecewise linear approximations (reference: X. Feng, Y. Li, Y. Qian, J. Gao, W. Cao, and L. Wang, “A high-precision flexible symmetry-aware architecture for element-wise activation functions,” in Proc. Int. Conf. Field Programmable Technol. (ICFPT), 2021, pp. 1–4.) can support multiple functions, but due to their fixed computational architecture, even the ReLU function requires 8 cycles to produce a result, leading to unnecessary delays when applied to accelerators. The design of this invention benefits from reconfigurable design, where the computation time of different activation functions dynamically varies according to the actual computational path, avoiding unnecessary delays: for ReLU series functions, 1 cycle is required to produce a result; for H-Sigmoid functions, 3 cycles are required; for H-Swish functions, 4 cycles are required; for Sigmoid functions, 6 cycles are required; and for Swish and Tanh functions, 7 cycles are required.

[0166] The reconfigurable activation function hardware device of this invention is applied to a convolutional neural network (CNN) inference accelerator as part of a convolution computation result processing unit. Specifically, the convolution computation result is used as the data to be computed, and the activation function type is used as the module's input to obtain the output of the corresponding activation function. This allows the accelerator to flexibly support multiple network models by supporting different activation functions in the network model. Experimental results show that on the Intel Arria 10 SoC FPGA platform, the system throughput for the lightweight CNN network MobileNet V3 is 79.34 GOPS, and the system throughput for the SE-ResNet network is 152.33 GOPS.

[0167] This solution primarily relies on the computational expression of the Sigmoid function, but it can be applied to other nonlinear activation functions that involve fractions. When the approximation accuracy of other nonlinear functions cannot be further improved due to the complexity of fractional calculations, the method proposed in this invention can be used to further improve the approximation accuracy of the nonlinear function, as long as the constraints on the approximation error of the output and reciprocal function are met.

[0168] This invention provides a reconfigurable activation function hardware device adapted to deep learning hardware accelerators. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A reconfigurable activation function hardware device adapted to a deep learning hardware accelerator, characterized in that, The function type judgment unit, the ReLU calculation unit, the simplified function calculation unit, the variable precision unit and the optimized function calculation unit are included. The function type judgment unit dynamically selects a calculation path of the activation function according to the input to-be-calculated data x and the to-be-calculated activation function type. The ReLU calculation unit judges a calculation interval to which the input to-be-calculated data belongs according to a sign bit of the to-be-calculated data x, and performs different calculation operations according to the activation function type, so as to realize ReLU series activation functions through data shift operation or direct data assignment, wherein the ReLU series activation functions include ReLU function, ReLU6 function and Leaky ReLU function. The simplified function calculation unit obtains a calculation result of the H-Sigmoid activation function as an initial approximation value of the Sigmoid function calculation, and simultaneously realizes PReLU function by reusing a multiplier in the simplified function calculation unit and cooperating with corresponding parameters in PReLU function calculation. The variable precision unit approximates the inverse function of Sigmoid 1+e -x to obtain an approximate value of the inverse function of Sigmoid appro_sig_rec, and obtains the whole expression part 1-e of the Tanh function by multiplexing the calculation logic based on the e exponential implementation -2x The calculation result provides the approximate values of the inverse function of Sigmoid and the whole expression part of the Tanh function; The optimized function calculation unit performs the following calculation on the initial approximation value and the reciprocal function approximation value to obtain an optimized approximation value of the Sigmoid function: Sigmoid(x) = 2 x H-Sigmoid(x) - H-Sigmoid(x) 2 x approach sig rec The optimized function calculation unit includes a first multiplier, a second multiplier and a third multiplier. Wherein, the approximation value of the Sigmoid reciprocal function is appro_sig_rec, the calculation result of the H-Sigmoid activation function is an intermediate result A1, based on the above formula, multiplication is performed on the intermediate result A1 and the intermediate result A1 in the first multiplier of the optimized function calculation unit, multiplication is performed on the square value of the intermediate result A1 and the approximation value of the Sigmoid reciprocal function appro_sig_rec in the second multiplier, subtraction operation is performed on the result obtained by the second multiplier and the result 2xA1 obtained by shift operation of the intermediate result A1, and finally the optimized approximation value of the Sigmoid function is obtained. H-Swish function and Tanh function are realized by multiplexing optimization function calculation unit inside No. 3 multiplier; Tanh function and Swish function are realized by multiplying the Sigmoid function calculation result with the corresponding integral part in the nonlinear function, wherein the integral part of the Tanh function is 1-e -2x , and the integral part of the Swish function is x; The simplified function calculation unit internally includes a fourth multiplier, and the simplified function calculation unit specifically performs the following operations: According to the H-Sigmoid activation function expression and the PReLU activation function expression, the H-Sigmoid activation function needs a No. 4 multiplier to realize function calculation when the input range of the data to be calculated x is (-3, 3) and the PReLU activation function needs a No. 4 multiplier to realize function calculation when the input range of x is (-∞, 0), and the No. 4 multiplier is used inside the simplified function calculation unit to complete the H-Sigmoid function operation and the PReLU de's a x operation, where a is a constant less than 1. The variable precision unit specifically performs the following operations: The original function f(x) is converted into an exponential function with base 2: Let the input variable P = log2e·x, convert f(x) to f # (P) = f(ln2·P) = 2 -P = 2 -(Pm+Pn) , where f # (P) represents a function calculation with the input independent variable P, the input variable P is divided into an integer part Pm and a decimal part Pn, in this way, the approximation calculation of e -x is converted into a shift operation and an approximation calculation of 2 -Pn , and then the input range of the approximation calculation is reduced to 0 to ln2, 0 < Pn≤1; Based on the Taylor expansion formula, the expansion of the function f(x) at x=x0 is: Wherein, x0 is an arbitrary number; A parameter L is introduced, L is a positive integer greater than or equal to 0, the input range (0, ln 2] of the function f(x) is uniformly divided into 2 L segments, the length of each segment is 2 -L ×ln 2, the sub-interval after the division is represented as [n·2 -L ×ln 2,(n+1)·2 -L ×ln 2), where n is a positive integer in [0,2 L ]. Let x0=ln 2xP0, δx=2 -L xln 2, get f(x)=f(ln 2P0)+2 L x(ln 2(P0+2 -L xln 2)-f(ln 2P0))x(log2e x-P0) For f # (P), let P0= n·2 -L , then in the input range [n·2 -L ,(n+1)·2 -L ] we get: The approximation accuracy of the exponential function can be adjusted by the parameter L; The shift and addition operation in the shift and addition operation unit 1 in the variable precision unit is realized by shift and addition operation, that is, P=log2e·x, based on the approximation value of log2e to obtain the converted to-be-calculated data P, wherein the integer part Pm of P participates in subsequent shift operation, and the decimal part Pn completes the approximation calculation in formula (1); f is calculated in the shift-add operation unit 2 inside the variable precision unit # (P) = a + P » β1 + P » β2, where β1 and β2 are positive integers satisfying β = 2 -β1 + 2 -β2 » denotes a shift operation; Based on the conversion relationship between the function arguments: P = log2e·x, for the function f # (P), the cut sub-interval is expressed as [n·2 -L ,(n+1)·2 -L ), under the given parameter L, according to the high value of the decimal part Pn, the sub-interval to which the input Pn belongs is determined, the specific value of n is determined, the parameter values of α, β1 and β2 are obtained, and the calculation of formula (1) is realized; The device completes the calculation of the reconfigurable activation function by performing the following steps: Step 1, input to-be-calculated data and the type of to-be-calculated activation function into the reconfigurable activation function hardware device; Step 2, the function type judgment unit dynamically selects the calculation operation to be performed according to the type of the activation function: determining whether it is a ReLU function, or a ReLU6 function, or a Leaky ReLU function, if it is any of them, determining the calculation interval to which the input data to be calculated belongs through the ReLU calculation unit, and obtaining the calculation result of the corresponding activation function; determining whether it is a PReLU function, if it is, determining the calculation interval to which the input data to be calculated belongs, wherein when the input is less than 0, the operation of x*alpha is realized by reusing the No. 4 multiplier in the simplified function calculation unit, wherein alpha is a constant less than 1, thereby obtaining the calculation result of the PReLU function; determining whether it is an H-Sigmoid function, if yes, determining the calculation interval to which the input data to be calculated belongs, wherein the input data to be calculated is in the interval (-3, 3), and the determination is realized by the 4th multiplier and the adder in the simplified function calculation unit obtaining the calculation result of the H-Sigmoid activation function by operation, and recording it as an intermediate result A1; determining whether it is a H-Swish function, if it is, according to the calculation expression of the H-Sigmoid function and the H-Swish function, the H-Swish function is represented as H-Swish(x) = x*H-Sigmoid(x), based on the calculation result of the H-Sigmoid activation function, i.e. the intermediate result A1, the multiplication calculation of the input data and the H-Sigmoid calculation result is realized by reusing the No. 1 multiplier in the optimized function calculation unit, thereby obtaining the calculation result of the H-Swish activation function; determining whether it is a Sigmoid function, if it is, according to the optimization relationship between the calculation result of the H-Sigmoid function and the approximate value of the Sigmoid reciprocal function, i.e. appro_sig_rec, the optimized approximate value of the Sigmoid function is obtained based on the following formula: Sigmoid(x) = 2 x H-Sigmoid(x) - H-Sigmoid(x) 2 xappro_sig_rec First, an approximation value of the sigmoid reciprocal function, appro_sig_rec, is calculated, which is expressed as 1+e -x , let P=log2e·x, where P is divided into an integer part Pm and a decimal part Pn, then converted to 1+2 -(Pm+Pn) , the approximation calculation of e -x is converted to a shift operation and an approximation calculation of 2 -Pn , and the input range of the approximation calculation is reduced to 0 to ln2, 0<Pn≤1; the optimized approximation value of the sigmoid function is the calculation result of the sigmoid activation function; For 2 -Pn The calculation is based on the Taylor expansion formula, converting 2 -Pn into a polynomial form of calculation, and dividing the calculation interval of Pn, calculating and optimizing the polynomial of 2 -Pn in each calculation interval, and converting the polynomial into the form of α+Pn>>β1+Pn>>β2, wherein α, β1 and β2 are calculation parameters related to the interval division parameter L; >> represents a shift operation; determining whether it is a Swish function, if it is, according to the calculation expression of the Sigmoid function and the Swish function, the Swish function is represented as Swish(x) = x*Sigmoid(x), based on the Sigmoid calculation result, i.e. the intermediate result A2, the multiplication calculation of the input data and the Sigmoid calculation result is realized by reusing the No. 3 multiplier in the optimized function calculation unit, thereby obtaining the calculation result of the Swish activation function; determine whether it is a Tanh function, if so, according to the calculation expression of the Sigmoid function and the Tanh function, the Tanh function is expressed as Tanh(x) = (1-e -2x ) x Sigmoid(2x), 1-e -2x The function is calculated in the variable precision unit, the calculation logic of e -x is multiplexed in the shift and addition operation unit 3 inside the variable precision unit; based on the Sigmoid calculation result, that is, the intermediate result A2, the multiplication calculation of the input data and the Sigmoid calculation result is realized by optimizing the multiplier No. 3 in the function calculation unit, and the calculation result of the Tanh activation function is obtained; Step 3, based on the calculation operation of step 2, the calculation results of different types of nonlinear activation functions are obtained and output.

Images