A hardware implementation device and method for activation function

Abstract

The present invention discloses a hardware implementation device and method for an activation function, belonging to the technical field of hardware implementation of function calculation. Aiming at the problem that it is difficult to balance high performance and hardware resources when implementing activation functions by hardware in the traditional method in the prior art, the invention comprises an e-index calculation module and a CORDIC division module. The input signal is calculated in the e-index calculation module through a small exponential lookup table with a base of 2 and shift and addition operations to obtain the e-index e. ‑x or e index e 2x The adder of the present invention is based on carry-save addition to realize the form of "I 1 +I 2 ‑I 3 " calculation function; the CORDIC division module is based on the CORDIC algorithm, and calculates the division function result in the form of #imgabs0# through multiple iterations. The circuit structure of the present invention is simple, without complex control logic, convenient circuit layout and wiring, high calculation accuracy, and supports sigmoid and tanh activation function calculations in any input range during calculation, solving the problem that traditional hardware implementation methods cannot balance performance and hardware resources.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware implementation of function computing, and more specifically, to a hardware implementation device and method for an activation function. Background Art

[0002] With the rapid development of artificial intelligence and other fields, neural networks have become a hot topic of research in recent years. Artificial neural network is a mathematical model that uses a structure similar to the synaptic connection of the brain to process information. Its main task is to build a practical artificial neural network model based on the principles of biological neural networks and the needs of practical applications, design corresponding learning algorithms, simulate certain intelligent activities of the human brain, and then implement it technically to solve practical problems.

[0003] As an important part of neural network algorithms, the hardware implementation of activation functions has also become a hot topic of research. Activation functions are functions that run on neurons in artificial neural networks and are responsible for mapping the input of neurons to the output. Activation functions play a very important role in enabling artificial neural network models to learn and understand very complex and nonlinear functions. Currently, many applications of neural network algorithms require not only real-time performance but also high precision, which inevitably places high precision requirements on the hardware implementation of activation functions.

[0004] When implementing activation functions on hardware, it is necessary to consider the hardware resources occupied during implementation, operating efficiency, calculation accuracy, etc. Commonly used activation functions include sigmoid function, tanh function and ReLU function. Among all activation functions, sigmoid and tanh functions are more difficult to implement in hardware. In 1957, Volder proposed the Coordinate Rotation Digital Computer (CORDIC) algorithm in his paper to calculate trigonometric functions, multiplication and division. The CORDIC algorithm is a hardware-friendly algorithm that can convert more complex function calculations into simple shift and addition operations. Researchers often use it to implement the required target function. The current implementation methods of sigmoid and tanh are difficult to guarantee the performance of the hardware while ensuring accuracy. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the problem that it is difficult to balance high performance and hardware resources when implementing activation functions in hardware using traditional methods in the prior art, the present invention provides a hardware implementation device and method for an activation function, which can better balance hardware performance and the use of hardware resources, and ensure higher hardware performance by using fewer hardware resources.

[0007] 2. Technical solution

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] A hardware implementation device for an activation function comprises an e-index calculation module, a CORDIC division module, an adder, a selector and a top-level control module, wherein the output end of the e-index calculation module is connected to the input end of the adder, the output end of the adder is connected to the output end of the CORDIC division module, the output end of the CORDIC division module is connected to the input end of the selector, the top-level control module is connected to the e-index calculation module, the CORDIC division module and the selector, the e-index calculation module is used for calculating an exponential function, and the CORDIC division module calculates a formula in the form of The top-level control module sends the clock signal CLK and the selection signal SEL to control the overall operation of the hardware. The clock signal CLK connects the e-index module and the CORDIC division module, and the selection signal SEL connects the e-index module and the selector. The top-level control module connects the e-index calculation module and the CORDIC division module and makes corresponding selections, and finally outputs the result of the sigmoid activation function or the tanh activation function.

[0010] Furthermore, the e exponent calculation module includes a shift unit, an adder and a power exponent operation unit, the shift unit includes a shift unit A, a shift unit B, a shift unit C, a shift unit D and a shift unit E, the adder includes an adder A and an adder B; the outputs of the shift unit A, the shift unit B and the shift unit C are all connected to the input of the adder A, the output of the adder A and the output of the shift unit E are all connected to the input of the adder B, the output of the adder B is connected to the input of the shift unit D and the input of the power exponent operation unit, and the output of the power exponent operation unit is also connected to the input of the shift unit D. The e exponent calculation module calculates the e exponent e by a small exponent lookup table with a base of 2 and shift and addition operations. -x and e index e 2x .

[0011] Furthermore, the adder A is a three-input M-bits adder, including 2M full adders, and the 2M full adders are grouped in M ​​to form two full adder groups; the input signal of the nth full adder in the first full adder group is the nth bit of the input signal, the output signal of the nth full adder in the first full adder group is used as the input signal of the nth full adder corresponding to the second full adder group, and the carry signals of the nth full adder in the first full adder group and the nth full adder corresponding to the second full adder group are both used as the input signal of the n+1th full adder corresponding to the second full adder group, M is a natural number greater than 1, and n is a natural number less than or equal to M.

[0012] Furthermore, the CORDIC division module includes a selector, a variable memory, an adder, a shift calculation unit, an iteration counter and an iteration controller. During calculation, each iteration variable corresponds to a selector, the output of the selector is connected to the input of the variable register, the output of the variable register is connected to the input of the adder, and the output of the adder is connected to the input of the selector; the selector is also connected to the iteration controller, the input of the adder is also connected to the shift calculation unit, and the shift calculation unit is connected to the iteration calculator. The CORDIC division module calculates the following through multiple iterations: The result of the division function.

[0013] A method for hardware implementation of an activation function, using the activation function hardware implementation device, an input signal calculates the e-index through an e-index calculation module, then adds the calculated value of the e-index calculation module to a fixed value through an addition module, and then calculates the division function result through a CORDIC division module using multiple iterations, and finally selects and outputs the result of a sigmoid activation function or a tanh activation function through a selector.

[0014] Furthermore, when the sigmoid function is selected, the input signal is passed through the e-index calculation module to calculate the e-index e -x , and then add e through the addition module -x It is added to a fixed value, and then the CORDIC division module is used to calculate the division function result using multiple iterations. After the division module is completed, the function value corresponding to the sigmoid function is obtained.

[0015] Furthermore, when the tanh function is selected, the input signal is passed through the e-index calculation module to calculate the e-index e 2x , and then add e through the addition module 2x After adding the fixed value, the CORDIC division module is used to calculate the division function result using multiple iterations. The CORDIC division module calculation value is shifted and then the fixed value is subtracted from the calculated value after the shift operation in the subtractor to finally obtain the function value corresponding to the tanh function.

[0016] Furthermore, the e-exponent calculation module first performs a shift operation on the input signal, and then uses adder A to add the shift results of shift unit A, shift unit B and shift unit C. Adder B performs addition / subtraction operations on the calculation result of adder A and the shift result of shift unit E. The operation result is divided into an integer part and a decimal part. The decimal part is calculated by the power exponential operation unit and then shifted in shift unit D; when the integer part is greater than or equal to 0, the number of left shifts of mobile unit D is the corresponding value of the integer part, and when the integer part is less than 0, the number of left shifts of mobile unit D is the corresponding value of the absolute value of the integer part.

[0017] Furthermore, the e-index calculation module calculates e -x When the value is equal to 0, the shift unit A, the shift unit B and the shift unit C respectively perform right shift operations of one bit, four bits and seven bits on the input signal; the e exponent calculation module calculates e 2x When the value is , the shift unit A, the shift unit B and the shift unit C respectively perform no shift, right shift by seven bits and right shift by three bits on the input signal.

[0018] The e index will be e when implemented -x Convert to exponent of 2 e 2x Convert to The two exponentiation parts of 2 are further converted into shift and 3-input addition operations: x>>1+x>>4-x>>7 and x+x>>7-x>>3; the exponentiation part is calculated to obtain the sum of the integer and the positive decimal I+D, and the final calculation is 2 I Using shift operation, 2 D A small exponential lookup table with base 2 is used, and the input range of the lookup table is [0, 1). The use of the lookup table can convert the e exponent of a large input range into a power of 2 exponent that is easy to implement in a small range.

[0019] Furthermore, in the CORDIC division module, the selector selects the iterative input value of the divider and sends the iterative input value to the variable register cache. The iterative input value is calculated by the shift calculation unit and then input into the adder. The output result of the adder is used as the initial value of the next iteration. When the number of iterations reaches the preset value, the z-value variable register outputs the calculated value; the shift calculation unit performs corresponding shift operations according to the count of the iteration counter. The iteration counter is used to record the number of current iterations and provide the shift operation unit with a specific number of shift bits. The output of the iteration counter controls the operation of the shift calculation unit; the sign bit of y in the y-value variable register controls the selection bit of the adder, and the adder is selected to perform addition or subtraction operations. The CORDIC division module adopts the vector mode in the linear coordinate system of the CORDIC algorithm to realize the reciprocal division operation. The input initial values ​​y and z of the iterative equation are set to 1 and 0 respectively, and the initial value x is the value to be calculated.

[0020] The hardware implementation of the activation function of the present invention is based on carry-save addition and the CORDIC algorithm. The adder A of the e-exponent calculation unit adopts carry-save addition operation. When calculating the input data of M-bits, only a delay of (M+1)τ is required, which can solve the high delay problem of the traditional ripple carry adder. At the same time, the circuit structure is simple, there is no complex control logic, and the circuit layout and wiring are convenient. When calculating the exponent, a small exponential lookup table with a base of 2 is used. The input range of the lookup table is [0,1). The lookup table can be used to convert the e-exponent of a large range input into a power of 2 that is easy to implement in a small range. The CORDIC algorithm is executed in conjunction with the CORDIC division module, and the form of The division function result, when the number of iterations reaches a certain value, the system calculation can reach an accuracy of 10 -5 .

[0021] 3. Beneficial effects

[0022] Compared with the prior art, the advantages of the present invention are:

[0023] The activation function hardware implementation method of the present invention is based on carry-save addition and CORDIC algorithm. Compared with traditional hardware implementation methods such as lookup table method, linear piecewise approximation method, nonlinear piecewise approximation method, etc., the adder based on carry-save addition of the present invention only needs a delay of (M+1)τ when calculating M-bits input data, which can solve the high delay problem of traditional ripple carry adder, and has the advantages of simple circuit structure, no complex control logic, and convenient circuit layout and wiring. A small exponential lookup table with base 2 is used in exponential calculation, and the input range of the lookup table is [0,1). The use of the lookup table can convert the e exponent of a large range input into a power exponent of 2 that is easy to implement in a small range.

[0024] The CORDIC division module of the present invention uses the CORDIC algorithm to calculate the function result through multiple iterations, and the accuracy can reach 10 -5 . The present invention supports the calculation of sigmoid and tanh activation functions in any input range, and can better balance the use of hardware performance and hardware resources, that is, while ensuring high performance, the present invention uses fewer hardware resources. In addition, the present invention studies and analyzes the relationship between the two activation functions, and can choose to implement two types of activation functions, sigmoid function and tanh function, after further configuration during calculation, while the traditional method only implements one function. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the hardware device structure of the present invention;

[0026] Figure 2 This is a hardware implementation structure diagram of the e-index calculation module of the present invention;

[0027] Figure 3 The structure diagram of the three-input adder based on carry-save addition of the present invention;

[0028] Figure 4 FIG. 4 is a hardware implementation structure diagram of the CORDIC division module of the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] Example 1

[0031] Figure 1 The figure shows a schematic diagram of the structure of the hardware device of the "S" type activation function based on carry-save addition and CORDIC algorithm in this embodiment. The hardware device of this embodiment includes an e-index calculation module and a CORDIC division module, and the e-index calculation module and the CORDIC division module are connected. The hardware system also includes a top-level control module, which is responsible for connecting the e-index calculation module and the CORDIC division module. The e-index calculation module and the CORDIC division module are both connected to the clock signal clk, and the e-index calculation module is also connected to the selection signal sel. The input signal first passes through the e-index calculation module, and then the calculated value of the e-index calculation module is added to 1 in an adder, and then this value is passed through the CORDIC division module as the input value of the CORDIC division module; the result after the CORDIC division module calculates enters the selector, and finally outputs the result of the activation function.

[0032] The activation function in this embodiment is analyzed below:

[0033] If the sigmoid function is expressed as The tanh function is expressed as The following deduction can be made:

[0034]

[0035] According to formula (1), the tanh function can be expressed by the sigmoid function. That is to say, in the specific hardware implementation, the tanh function and the sigmoid function can share a hardware architecture to reduce the use of hardware resources.

[0036] From formula (1), we can see that the key to implementing the sigmoid function and the tanh function is the division and the e-exponent. Division can be implemented through the vector mode in the linear coordinate system of the CORDIC algorithm, so the biggest difficulty in implementing the two functions is the implementation of the e-exponent. Further analysis and deduction:

[0037] e used in the sigmoid function and tanh function respectively -x and e 2x It can be expressed as follows:

[0038]

[0039]

[0040] In formula (2) and formula (3), r1 and r2 are real numbers. A real number can be expressed as the sum of an integer and a pure decimal. Therefore, r1 and r2 can be expressed as follows, where I represents an integer and D represents a pure decimal:

[0041] r1=I1+D1 (4)

[0042] r2=I2+D2 (5)

[0043] I1 and I2 in formula (4) and formula (5) can be calculated by formula (6) and formula (7), where floor() is a rounding function:

[0044] I1=floor((-x)log2e) (6)

[0045] I2=floor((2x)log2e) (7)

[0046] Accordingly, D1 and D2 can be calculated using formula (8) and formula (9):

[0047] D1=(-x)log2e-floor((-x)log2e) (8)

[0048] D2=(2x)log2e-floor((2x)log2e) (9)

[0049] Thus we can get:

[0050]

[0051]

[0052] The shift operations in formula (10) and formula (11) are very common in hardware systems and are very easy to implement, but the power of 2 requires special processing. Considering that D1, D2∈[0,1), the range is very small, a lookup table can be used to implement the power of 2 in the specific implementation.

[0053] After the above analysis and derivation process, the large-scale e exponent (-∞, +∞) can be converted into a small-scale and easy-to-implement power of 2. The following will continue to analyze how to calculate I1, D1, I2 and D2.

[0054] (-x)log2e and (2x)log2e can be expressed as follows:

[0055]

[0056] (2x)log2e=x·log2e 2 (13)

[0057] After calculation, we can get, log2e 2 ≈2.885390081777927, therefore, formula (12) and formula (13) can be transformed into:

[0058] (-x)log2e≈x·(-1.442695040888963)

[0059] =x·(-2+0.557304959111037)

[0060] =-2x+0.557304959111037x (14)

[0061] (2x)log2e≈x·2.885390081777927

[0062] =x·(2+0.885390081777927)

[0063] =2x+0.885390081777927x (15)

[0064] From formula (14) and formula (15), we can find that the calculation of (-x)log2e and (2x)log2e can be divided into two parts. The first part is to calculate -2x or 2x, and the second part is to calculate the multiplication of decimals. For the calculation of -2x or 2x, the hardware implementation can directly use the shift operation to solve it, while the multiplication of decimals requires another solution. Considering that 0.557304959111037≈0.5546875, 0.885390081777927≈0.8828125, and 0.5546875 and 0.8828125 are (0.1000111)2 and (0.1110001)2 respectively at the binary level, we have:

[0065] 0.557304959111037x≈x·(0.1000111)2 (16)

[0066] 0.885390081777927x≈x·(0.1110001)2 (17)

[0067] It is not difficult to find that (0.1000111)2 and (0.1110001)2 can be calculated as follows:

[0068] (0.1000111)2=(0.1000000)2+(0.0001000)2-(0.0000001)2 (18)

[0069] (0.1110001)2=(1.0000000)2+(0.0000001)2-(0.0010000)2 (19)

[0070] Thus:

[0071] x·(0.1000111)2=x>>1+x>>4-x>>7 (20)

[0072] x·(0.1110001)2=x+x>>7-x>>3 (21)

[0073] After the above series of derivations, the more complex decimal multiplication is finally transformed into shift and addition operations that are easy to implement in hardware. After calculating (-x)log2e and (2x)log2e, the integer parts of the two are taken as I1 and I2, and the decimal parts are taken as D1 and D2. At this point, the calculated values ​​of I1, I2, D1 and D2 are obtained.

[0074] Figure 2The basic structure diagram of the e-exponent calculation module of this embodiment is shown. The e-exponent calculation module includes several shift units and adders. In this example, it includes five shift units SU1, SU2, SU3, SU4 and SU5. The shift units perform shift operations. Shift units SU1, SU2 and SU3 are all connected to adder Adder; Adder is a three-input adder, and adder Adder performs the addition operation of the three numbers of shift units SU1, SU2 and SU3. The calculation result of adder Adder is added or subtracted from the calculation result of shift unit SU5, and the obtained value is divided into an integer part I and a decimal part D. The decimal part D is calculated by the power exponential operation unit of 2 and then enters the shift unit SU4 to perform the shift operation. The number of shifts performed by the shift unit SU4 is confirmed according to the integer part I. When the integer part I is greater than or equal to zero, a left shift operation is performed, and when the integer part I is less than zero, a right shift operation is performed.

[0075] The following is a detailed description of the e index calculation module. -x and e 2x The calculation process is described in the following.

[0076] Among them, e -x The calculation process includes the following steps:

[0077] Step 11: Input signal x;

[0078] Step 12: The input signal x is shifted by the shift units SU1, SU2, SU3, and SU5 respectively, and the input signal x is shifted right by one bit, right by four bits, right by seven bits, and left by one bit respectively, and the shift calculation results of x>>1, x>>4, x>>7, and x<<1 are calculated;

[0079] Step 13: Input the shift calculation results x>>1, x>>4, x>>7 of x right shifted by one bit, four bits, and seven bits into the three-input adder Adder, and calculate the intermediate result S1 according to formula (18);

[0080] Step 14: add the intermediate result S1 calculated by the adder Adder and the shift calculation result x<<1 of the shift unit SU5 shifting one bit to the left, and obtain a result divided into two parts I+D, where I represents the integer part and D represents the decimal part;

[0081] Step 15: The fractional part D is passed through the power of 2 exponential operation unit to obtain 2 D ;

[0082] Step 16: Calculation result of power exponential operation unit 2 D Enter the shift unit SU4 and shift left by I bits (when the integer part I≥0) or right by (-I) bits (when the integer part I<0) to get e -x .

[0083] e 2x The calculation process is the same as e -x They are basically the same, except that the number of shifts in the shift unit calculation is different. The specific calculation process is as follows:

[0084] Step 21: Input signal x;

[0085] Step 22: The input signal x is shifted by the shift units SU1, SU2, SU3, and SU5, respectively, and the input signal x is not shifted, right shifted by seven bits, right shifted by three bits, and left shifted by one bit are respectively performed to obtain the shift calculation results of x, x>>7, x>>3, and x<<1;

[0086] Step 23: Input the shift calculation results x, x>>7, and x>>3 of x without shift, right shift by seven bits, and right shift by three bits into a three-input adder Adder, and calculate the intermediate result S2 according to formula (19);

[0087] Step 24: Subtract the intermediate result S2 calculated by the adder Adder from the shift calculation result x<<1 of the left shift operation of the shift unit SU5, and the result is divided into two parts I+D, where I represents the integer part and D represents the decimal part;

[0088] Step 25: The fractional part D is passed through the power of 2 exponential operation unit to obtain 2 D ;

[0089] Step 26: Calculation result of power exponential operation unit 2 D Enter the shift unit SU4 and shift left by I bits (when the integer part I≥0) or right by (-I) bits (when the integer part I<0) to get e 2x .

[0090] The structure of the three-input adder Adder is as follows: Figure 3 As shown, the three-input M-bits adder Adder includes 2M full adders. The 2M full adders are grouped in M ​​to form two full adder groups, which are used for the adder input layer and output layer respectively. When describing, we can also use the first layer to represent the input layer and the second layer to represent the output layer. The input layer and the output layer each include M full adders. The output signal of the nth full adder in the input layer is sent to the nth full adder in the output layer as the input signal. At the same time, the carry signal of the nth full adder in the input layer is sent to the n+1th full adder in the output layer as the input signal. The input signal of the n+1th full adder in the output layer also includes the carry signal of the nth full adder in the output layer. The two input terminals of the 0th full adder in the output layer are set to 0 and 1 respectively. M is a natural number greater than 1, and n is a natural number less than or equal to M.

[0091] like Figure 3As shown, in this structure, when interconnected, the output and carry signal of the first layer adder group are used as the input signal of the second layer adder group, and the carry of the previous adder in the second layer adder group is also used as the input of the next adder. 1 ,I 2 and I 3 Each bit of the first adder is used as the input signal of the first layer. For example, the first adder of the first layer receives I 1 ,I 2 and I 3 The 0th bit data information of the first adder in the second layer is set to 0 and 1 respectively, and finally the M-bits data output by the second layer are combined to obtain the required calculation result.

[0092] The three-input adder Adder of this embodiment does not adopt the traditional adder structure, but proposes an improved adder based on the carry-save adder. Figure 3 With the interconnection shown in the figure, the adder can realize the form of "I 1 +I 2 -I 3 "Calculation function, where I 1 ,I 2 ,I 3 All are M-bits numbers. Compared with the traditional adder, the improved carry-save adder proposed in the present invention has great advantages. When the ripple carry adder realizes the same function, it requires at least 2Mτ delay (τ is the full adder calculation delay), while the adder only needs (M+1)τ delay; the carry-lookahead adder can also reduce the delay, but the carry-lookahead adder requires additional control logic and complex wiring, while the adder of this embodiment has a simple structure and does not require additional complex logic control. In terms of hardware cost, the original carry-save adder needs to use a ripple carry adder in the third stage to calculate the final result, while the adder of this embodiment omits this part, reducing the cost.

[0093] The input signal x is calculated by the e exponential calculation module to obtain e -x and e 2x , corresponding to the sigmoid activation function or the tanh activation function respectively. -x and e 2x The values ​​obtained by adding 1 are used as the input signals of the CORDIC division module, and finally the results of the sigmoid activation function or the tanh activation function are output.

[0094] When the sigmoid function is selected, the input signal is calculated through the e-index calculation module to calculate the e-index e -x , and then add e through the addition module -xIt is added to a fixed value, and then the CORDIC division module is used to calculate the division function result using multiple iterations. After the division module is completed, the function value corresponding to the sigmoid function is obtained.

[0095] When the tanh function is selected, the input signal is passed through the e-index calculation module to calculate the e-index e 2x , and then add e through the addition module 2x After adding the fixed value, the CORDIC division module is used to calculate the division function result using multiple iterations. After the calculation is completed, The calculated value is shifted left by one bit and sent to the subtractor for calculation. Finally, the function value corresponding to the tanh function is obtained.

[0096] The structure of the CORDIC division module is as follows Figure 4 As shown, it includes three selectors, three variable registers, three adders, two shift calculation units, an iteration counter and an iteration controller, each iteration variable corresponds to a selector, the output end of each selector is connected to the input end of a variable register, the output end of the variable register is connected to the input end of the adder, the output end of the adder is connected to the input end of the selector, the selector is also connected to the iteration controller, the input ends of two adders are also connected to the shift calculation unit, the shift calculation unit is controlled by the iteration calculator, and when the number of iterations reaches a preset value, the variable register outputs the calculation result.

[0097] In the CORDIC division module, the selector is used to select the iterative input value of the divider, and the iterative initial value is sent to the calculation module only in the first iteration. In other cases, the result of each iteration is selected as the initial value of the next iteration; the variable register is used to cache the current iterative input value for use by the shift calculation unit and the adder; the shift calculation unit performs corresponding shift operations according to the count of the iteration counter; the iteration counter is used to record the number of current iterations and provide the shift operation unit with a specific number of shift bits; the adder can choose to perform addition or subtraction operations, which is determined by the selection bit. When specifically connected, the control bit of the selector is controlled by the iteration controller, and the output of the selector is used as the input of the variable register; the output of the iteration counter controls the operation of the shift calculation unit, and the sign bit of y in the current y value register controls the selection of the adder function; the output result of the adder is used as the initial value of the next iteration.

[0098] The CORDIC division module executes the vector mode of the CORDIC algorithm in the linear coordinate system and calculates the vector mode of the form The initial values ​​of the iteration variables x, y, and z are the values ​​to be calculated, 1, and 0, respectively. When the number of iterations reaches a preset value, the output value is output by the variable register of the current variable z. The number of iterations in this embodiment is 13 times, which is obtained through system simulation. The number of iterations in this embodiment is set to 13 times, and the accuracy reaches 10 -5 In practical applications, to ensure system accuracy, the number of iterations is no less than 13 times.

[0099] According to the above scheme, the hardware implementation of the "S" type activation function based on carry-save addition and CORDIC algorithm is synthesized under TSMC 40nm process, and the maximum frequency of the system clock signal clk can reach 1.5GHz. At this frequency, the system area is about 4048.64μm 2 The power consumption is only 1.75mW. The simulation results show that the sigmoid function in this embodiment can reach 10 -4 The accuracy of the tanh function can reach 10 -3 accuracy.

[0100] It can be seen from the above results that the "S" type activation function implementation method of this embodiment not only has high calculation accuracy and good performance, but also can use less hardware resources. As a typical operation in fields such as artificial intelligence, this hardware implementation method has good reference significance and broad application prospects.

[0101] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited to this, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of this patent. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A hardware implementation device for an activation function, characterized in that: The method comprises an e-index calculation module, a CORDIC division module, an adder, a selector and a top-level control module. The output end of the e-index calculation module is connected to the input end of the adder, the output end of the adder is connected to the output end of the CORDIC division module, the output end of the CORDIC division module is connected to the input end of the selector, the top-level control module is connected to the e-index calculation module, the CORDIC division module and the selector, the e-index calculation module is used for calculating the exponential function, and the CORDIC division module calculates the exponential function through multiple iterations. The CORDIC division module includes a selector, a variable memory, an adder, a shift calculation unit, an iteration counter and an iteration controller. During calculation, each iteration variable corresponds to a selector. The output end of the selector is connected to the input end of the variable register, the output end of the variable register is connected to the input end of the adder, and the output end of the adder is connected to the input end of the selector. The selector is also connected to the iteration controller, and the input end of the adder is also connected to the shift calculation unit. The shift calculation unit is connected to the iteration calculator. In the CORDIC division module, the selector selects the iterative input value of the divider and sends the iterative input value to the variable register cache. The iterative input value is calculated by the shift calculation unit and input into the adder. The output result of the adder is used as the initial value of the next iteration. When the number of iterations reaches a preset value, the z value variable register outputs the calculated value. The shift calculation unit performs corresponding shift operations according to the count of the iteration counter. The iteration counter is used to record the number of current iterations and provide the shift calculation unit with a specific number of shift bits. The output of the iteration counter controls the operation of the shift calculation unit. The sign bit of y in the y value variable register controls the selection bit of the adder, and the adder is selected to perform addition or subtraction operations. The CORDIC division module uses the vector mode under the linear coordinate system of the CORDIC algorithm to implement the reciprocal division operation. The input initial values ​​y and z of the iteration equation are set to 1 and 0 respectively, and the initial value x is the value to be calculated.

2. The hardware implementation device of an activation function according to claim 1, characterized in that: The e-exponent calculation module includes a shift unit, an adder and a power exponent operation unit. The shift unit includes a shift unit A, a shift unit B, a shift unit C, a shift unit D and a shift unit E. The adder includes an adder A and an adder B. The output ends of the shift unit A, the shift unit B and the shift unit C are all connected to the input end of the adder A, the output end of the adder A and the output end of the shift unit E are all connected to the input end of the adder B, the output end of the adder B is connected to the input end of the shift unit D and the input end of the power exponent operation unit, and the output end of the power exponent operation unit is also connected to the input end of the shift unit D.

3. The hardware implementation device of an activation function according to claim 2, characterized in that: The adder A is a three-input M-bits adder, including 2M full adders, and the 2M full adders are grouped into M to form two full adder groups; the input signal of the nth full adder in the first full adder group is the nth bit of the input signal, the output signal of the nth full adder in the first full adder group is used as the input signal of the nth full adder corresponding to the second full adder group, and the carry signals of the nth full adder in the first full adder group and the nth full adder corresponding to the second full adder group are both used as the input signal of the n+1th full adder corresponding to the second full adder group, M is a natural number greater than 1, and n is a natural number less than or equal to M.

4. A method for hardware implementation of an activation function, characterized in that: Using a hardware implementation device of an activation function as described in any one of claims 1 to 3, the input signal calculates the e-index through an e-index calculation module, and then the calculation value of the e-index calculation module is added to the fixed value through an addition module, and then the division function result is calculated using multiple iterations through a CORDIC division module, and finally the result of a sigmoid activation function or a tanh activation function is selected and output through a selector.

5. The method for hardware implementation of an activation function according to claim 4, characterized in that: When the sigmoid function is selected, the input signal is calculated through the e-index calculation module to calculate the e-index e -x , and then add e through the addition module -x It is added to a fixed value, and then the CORDIC division module is used to calculate the division function result using multiple iterations. After the division module is completed, the function value corresponding to the sigmoid function is obtained.

6. The method for hardware implementation of an activation function according to claim 5, characterized in that: When the tanh function is selected, the input signal is passed through the e-index calculation module to calculate the e-index e 2x , and then add e through the addition module 2x After adding the fixed value, the CORDIC division module is used to calculate the division function result using multiple iterations. The CORDIC division module calculation value is shifted and then the fixed value is subtracted from the calculated value after the shift operation in the subtractor to finally obtain the function value corresponding to the tanh function.

7. A method for hardware implementation of an activation function according to claim 5 or 6, characterized in that: The e exponential calculation module first performs a shift operation on the input signal, and then uses adder A to add the shift results of shift unit A, shift unit B and shift unit C. Adder B performs addition / subtraction operations on the calculation result of adder A and the shift result of shift unit E. The operation result is divided into an integer part and a decimal part. The decimal part is calculated by the power exponential operation unit and then shifted in shift unit D. When the integer part is greater than or equal to 0, the number of left shifts of mobile unit D is the corresponding value of the integer part. When the integer part is less than 0, the number of left shifts of mobile unit D is the corresponding value of the absolute value of the integer part.

8. The method for hardware implementation of an activation function according to claim 7, characterized in that: The e index calculation module calculates e -x When the value is , the shift unit A, the shift unit B and the shift unit C respectively perform right shift operations of one bit, four bits and seven bits on the input signal; The e index calculation module calculates e 2x When the value is , the shift unit A, the shift unit B and the shift unit C respectively perform no shift, right shift by seven bits and right shift by three bits on the input signal.

9. A method for hardware implementation of an activation function according to claim 5 or 6, characterized in that: In the CORDIC division module, the selector selects the iterative input value of the divider and sends the iterative input value to the variable register cache. The iterative input value is calculated by the shift calculation unit and then input into the adder. The output result of the adder is used as the initial value of the next iteration. When the number of iterations reaches the preset value, the z value variable register outputs the calculated value. The shift calculation unit performs corresponding shift operations according to the count of the iteration counter. The iteration counter is used to record the number of current iterations and provide the shift calculation unit with a specific number of shift bits. The output of the iteration counter controls the operation of the shift calculation unit. The sign bit of y in the y value variable register controls the selection bit of the adder, and selects the adder to perform addition or subtraction operations.

Images