A digital circuit module for computing a tanh function based on a range addressable lookup table

Abstract

The application discloses a digital circuit module for calculating a tanh function based on a range-addressable lookup table, comprising an input module, an address mapping module, a memory module, a selection output module and control logic. Under the control of the control logic, the circuit generates an address and looks up a function value in three clock cycles to calculate the tanh function. The application comprises a bit width selection method for input data and output data under a given maximum absolute error, and a selection method for memory storage values, and can realize circuit modules with different precisions. The application has the advantages of simple and reliable circuit structure, fast calculation speed, low power consumption, less storage space compared to a common lookup table method, easy expansion and convenience in chip implementation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural network and digital circuit design, in particular to the circuit implementation of nonlinear activation function, and specifically to a digital circuit module for calculating tanh function based on range-addressable lookup table. BACKGROUND

[0002] Nowadays, artificial neural networks have been applied in many fields, such as image and speech recognition, natural language processing, etc. In artificial neural networks, an activation function is used for each neuron to introduce nonlinearity into the neural network, thereby improving the expression ability of the neural network to the model. When calculating, the neural network first performs linear operations such as convolution on the input data, and then takes the result as the input quantity of the activation function. Nowadays, the emerging network evolution tends to use fewer linear calculation operations, and many optimizations have been made for linear calculation. As a result, the performance bottleneck of the neural network gradually shifts to the complex activation function.

[0003] In the current mainstream artificial neural network, the tanh function is a common nonlinear activation function. Since it requires complex exponential and division operations, using software to calculate will consume a lot of time, limiting the performance of the neural network, so we want to design a dedicated hardware module to calculate the nonlinear activation function.

[0004] The approximation methods for nonlinear activation functions mainly include lookup table, piecewise linear approximation, piecewise nonlinear approximation, and hybrid method. Among them, the lookup table method is simple in thought and fast in calculation, and is a commonly used method. However, the lookup table method requires a large number of storage units to store function values, and the number of storage units will increase sharply as the accuracy increases. SUMMARY

[0005] In order to alleviate the large demand for storage units of the lookup table method, and make full use of the characteristics of the tanh function itself (including the function value of tanh approaching zero point approaching the independent variable x, the function value of infinite distance tending to 1 and -1, and the tanh itself being an odd function), the most suitable data bit width and lookup table storage content are selected for a certain accuracy requirement, and the technical scheme adopted by the present application is as follows:

[0006] The present application provides a digital circuit module for calculating tanh function based on range-addressable lookup table, comprising: an input module, an address mapping module, a temporary register module, a memory module, a selection output module, and a control logic unit.

[0007] The input module accepts external input data data as the independent variable of the tanh function, and decomposes the input data data into a sign bit sign and an unsigned number unsigned as the output of the input module; the format of the input data is defined as follows: the format of an n-bit fixed-point number is used, and the data format is {S, I, F}, wherein S is a 1-bit sign bit indicating the positive or negative of the data, I is n0-bit integer bits, and F is n1-bit decimal bits, and n=1+n0+n1; the input range of the data is (-2 n0 , 2 n0 ), n is the data bit width, n0 is the integer bit width, and n1 is the decimal bit width;

[0008] The address mapping module accepts the unsigned number unsigned given by the input module and the external input overflow bit, processes to generate the address addr2 of the corresponding function value in the memory module, and generates a read enable signal ren; wherein the external input overflow bit indicates the range of the input data, and if overflow=1, it indicates that the input is not between (-2 n0 , 2 n0 );

[0009] The temporary register module accepts the unsigned number unsigned output by the input module and stores it;

[0010] The memory module has two modes of reading and writing; when the write enable signal wen given by the outside is 1, the memory module stores the external input data data as the data, and the storage address is the external input address addr1; when the read enable signal ren generated by the address mapping module is 1, the memory module outputs the data stored at the address addr2; the data format of the storage unit of the memory module is an unsigned fixed-point number, and only the decimal bits are stored, and the bit width is n2-1; wherein 1 represents a sign bit, and the sign bit does not need to be stored;

[0011] The selection output module selects one of the output data of the temporary storage register module or the memory module according to the control signal from the control logic unit, removes the integer bits, adds the sign bit and outputs the output data as the output; the selection logic of the selection output module is that when the unsigned number of the input data data is less than the pre-defined boundary value x_start and the overflow bit is 0, the unsigned number is directly removed of the integer bits, added with the sign bit and outputted as the output, wherein the decimal part is added with the zero until the whole data bit width is equal to the bit width n2 of the output output; otherwise, the function value of the input data is searched in the memory module, and then the sign bit is added and outputted as the output; the output data output is a signed fixed-point number with the bit width n2 and the data format {S, F}, wherein S is a 1-bit sign bit and F is the decimal bit of the fixed-point number without the integer bit; the pre-defined boundary value x_start has the format {I, F} and the bit width n-1 without the sign bit;

[0012] The control logic unit coordinates the execution of the modules, and the whole calculation process is completed in 3 clock periods under the control of the control logic unit and a calculation completion signal fin is outputted; when the reset signal rst inputted from outside is 1, the whole digital circuit module is reset; in the first clock period, the input data is processed by the input module and the address mapping module to obtain the ren signal and the address addr and the data unsigned which needs to be temporarily stored; in the second clock period, the output values of the temporary storage register module and the memory module are selected according to the signal from the control logic unit; in the third clock period, the selected data is outputted through the output register, and the calculation completion signal fin becomes 1, indicating that the calculation has been completed.

[0013] According to the preferred scheme of the application, the data bit width n of the input data data, the integer bit width n0 and the decimal bit width n1, the data bit width n2 of the output data output and the boundary value x_start of the selection output module are determined by the following method:

[0014] Firstly, the maximum absolute error e of the calculation of the tanh function is determined;

[0015] Then, n2 is determined, and the selected n2 satisfies 1 / 2 n2 ≤e / 10;

[0016] Then, n1 is determined, and the selected n1 satisfies 1 / 2 n1 ≤e and 1 / 2 n1 ≤2(e-1 / 2 n2 );

[0017] Then n0 is determined to satisfy 1-tanh(2 n0 )≤2(e-1 / 2 n2 ), and n0 is selected as the minimum value satisfying the condition;

[0018] Then n is determined to satisfy n=1+n0+n1;

[0019] Finally, x_start is determined as the solution of the equation x-tanh(x)=e.

[0020] According to a preferred scheme of the present application, the demarcation value x_start of the output module is selected as the value closest to the solution of the equation x-tanh(x)=e in the fixed-point number format.

[0021] According to a preferred scheme of the present application, the mapping rule of the address mapping module is determined by the following method:

[0022] The interval [x_start, 2 n0 ) is divided into N subintervals, each of which covers the whole interval without overlapping and is connected with the previous one. After x_start is determined, x_start is the starting point of the first subinterval, also denoted as x0, and the end point of the first subinterval is x1, which is also the starting point of the second subinterval, and so on. The variable k is used to denote the label of a subinterval, k=1, 2, 3,..., N; x0 and the end point xk of the kth subinterval are n-1-bit unsigned fixed-point numbers, with n1-bit decimal width and n0-bit integer width. xk is selected as the largest number in the format satisfying tanh(xk)-tanh(xk-1)≤2(e-1 / 2 n2 ). That is, starting from x0, x0 is increased by 1 from the least significant bit to obtain the next bit number x of x0 in the format, and then it is checked whether the number x satisfies tanh(xk)-tanh(xk-1)≤2(e-1 / 2 n2 ). If not, the last number x satisfying the above inequality is taken as x1, and the remaining xk is determined according to the same rule. According to this rule, the Kth subinterval is determined, so that xK≥2 n0 , and the interval [x_start, 2 n0 ) is completely divided into subintervals, and the end point xK of the last subinterval is taken as 2 n0 , and the total number of subintervals N=K.

[0023] According to a preferred scheme of the present application, the mapping rule of the address mapping module is: when overflow=0, if the input unsigned number is in the kth subinterval, the binary form of k-1 is output as the address addr2; when overflow=1, the binary form of N is output as the address addr2.

[0024] According to a preferred scheme of the present application, the storage data value of the memory module is: at address k-1, the number closest to (tanh(xk)-tanh(xk-1)) / 2 represented by the storage data format, wherein the storage data format is an unsigned fixed-point number without integer bits and with decimal bit width n2-1; at address N, the number closest to (1-tanh(2 n0 )) / 2 represented by the data format.

[0025] According to a preferred scheme of the present application, the module needs to store corresponding data in the memory module according to the above-mentioned storage data value before performing the calculation operation.

[0026] According to a preferred scheme of the present application, the address addr is selected by the control logic unit from two signals of address addr1 and address addr2, and the logic of selecting the above-mentioned two data is: when the write enable signal wen given externally is 1, the address addr1 is selected as the address addr; when the write enable signal wen is 0, the address addr2 is selected as the address addr.

[0027] The present application has the advantages that a specific circuit module design method for calculating the tanh function by using a range-addressable lookup table is proposed, a plurality of input data are mapped to the same storage unit within a given maximum absolute value error e, the use of the storage unit is reduced, an optimized partition method is proposed, a more optimal data mapping scheme can be selected, and the use of the storage unit is further reduced; in addition, a bit width selection method for input data and output data is proposed, the length of a single storage unit can be reduced as much as possible, thereby reducing the use area of the memory; and the properties of the tanh function are used, the input smaller than x_start is directly taken as the output, the negative input is converted into a positive number and then calculated, and the area of the memory is also reduced. Compared with the ordinary lookup table method, the present application reduces the use of memory resources, has a simple structure, does not have a complex unit similar to a multiplier, has a fast calculation speed, low power consumption, small occupied area, can be expanded, and can meet the application scenarios of the current neural network edge calculation. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a structural block diagram of the module of the present application.

[0029] Figure 2 is an input data format diagram of the module of the present application.

[0030] Figure 3 is a circuit timing diagram of the module of the present application.

[0031] Figure 4 is a function simulation waveform diagram of the module of the present application. DETAILED DESCRIPTION

[0032] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0033] The maximum absolute error e of tanh is calculated as 0.02 in this embodiment, and the following technical solution is adopted:

[0034] A digital circuit module for calculating tanh function based on range addressable lookup table, comprising: an input module, an address mapping module, a memory module, a temporary register module, a selection output module, a control logic unit; the structural block diagram is as shown in Figure 1 .

[0035] The input module accepts external input data as the independent variable of tanh function, and decomposes the input data into sign and unsigned, as the output of the module. The input data adopts the format of 8-bit fixed-point number, including 1-bit sign, 1-bit integer and 6-bit decimal. The input range of data is (-2, 2), and the data format is as shown in Figure 2 .

[0036] The address mapping module accepts the unsigned given by the input module, and the input overflow of external neural network, processes to generate the address addr2 of the corresponding function value in the memory module, and generates the read enable signal ren, the bit width of addr2 is 5. The input overflow of external neural network represents the range of input data, if overflow = 1, it represents that the input is not between (-2, 2).

[0037] The temporary register module accepts the unsigned output by the input module and stores it.

[0038] The memory module has two modes of reading and writing. When the write enable signal wen given by the outside is 1, the memory module stores the external input data as the data, and the storage address is the data addr1 input by the outside; when the read enable signal ren generated by the address mapping module is 1, the memory module outputs the data stored at the address addr2. The data format of the storage unit of the memory module is unsigned fixed-point number, and only has decimal place, and the bit width is 8.

[0039] The selection output module includes two parts, MUX and output register, which selects one of the output data of the temporary register or the memory module according to the control signal given by the control logic, and the selected data is output as output after adding sign bit. The temporary register stores the unsigned part of the input data, and output is a 9-bit signed fixed-point number, including 1-bit sign bit and 8-bit decimal bit, without integer bit. The selection logic of the above selection output module is that when the unsigned number of the input data data is less than the pre-given data x_start = 0.40625 (converted to fixed-point number format as 7'h1a) and overflow = 0, it means that the data is small enough, and the unsigned data is directly removed from the integer bit, and two 0s are added at the end of the decimal bit and the sign bit is added at the highest bit, and the operation is output as output; otherwise, the function value of the input data is looked up in the memory module, and the operation of supplementing the data bit and adding the sign bit is output as output.

[0040] The control logic unit coordinates the execution of each module, and the entire calculation process is completed in 3 clock cycles under its control, as shown in Figure 3 When the reset signal rst input from outside is 1, the entire digital circuit module is reset; in the first clock cycle, the input data passes through the input module and the address mapping module to obtain the ren signal and the address addr, and the data unsigned that needs to be temporarily stored, and the address addr is selected from the two signals of address addr1 and address addr2 by the control logic unit. The logic of selecting the above two data is: when the write enable signal wen given from outside is 1, select address addr1 as address addr; when the write enable signal wen is 0, select address addr2 as address addr; in the second clock cycle, the output values of the temporary register module and the memory module will be selected according to the signals given by the control logic; in the third clock cycle, the selected data is output through the output register, and at the same time, the calculation completion signal fin becomes 1, indicating that the calculation has been completed.

[0041] When the input is in the following interval range, the specific mapping rule of the above address mapping module and the storage data of the memory module under the corresponding address are as follows, and the binary format data of the address and the storage value of the memory module are also written:

[0042] The interval [0.40625, 0.4375) outputs addr2 = 5' b00000, and the memory module stores 0.3984375 (8'h66);

[0043] Interval [0.4375, 0.46875), output addr2 = 5' b00001, memory module stores 0.42578125 (8'h6d);

[0044] Interval [0.46875, 0.5), output addr2 = 5' b00010, memory module stores 0.44921875 (8'h73);

[0045] Interval [0.5, 0.546875), output addr2 = 5' b00011, memory module stores 0.48046875 (8'h7b);

[0046] Interval [0.546875, 0.59375), output addr2 = 5' b00100, memory module stores 0.515625 (8'h84);

[0047] Interval [0.59375, 0.640625), output addr2 = 5' b00101, memory module stores 0.55078125 (8'h8d);

[0048] Interval [0.640625, 0.6875), output addr2 = 5' b00110, memory module stores 0.58203125 (8'h95);

[0049] Interval [0.6875, 0.734375), output addr2 = 5' b00111, memory module stores 0.609375 (8'h9c);

[0050] Interval [0.734375, 0.78125), output addr2 = 5' b01000, memory module stores 0.640625 (8'h a4);

[0051] Interval [0.78125, 0.84375), output addr2 = 5' b01001, memory module stores 0.671875 (8'h ac);

[0052] Interval [0.84375, 0.90625), output addr2 = 5' b01010, memory module stores 0.703125 (8'h b4);

[0053] Interval [0.90625, 0.984375), output addr2 = 5' b01011, memory module stores 0.73828125 (8'h bd);

[0054] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0055] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0056] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0057] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0058] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0059] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0060] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7);

[0061] Interval [1.40625, 1.59375), output addr2 = 5' b10000, memory module stores 0.90234375 (8'he7).

[0062] The application needs to store the above-mentioned values into the corresponding storage units through the corresponding interfaces before calculation.

[0063] The above embodiment is implemented on vivado, and passes the functional simulation and actual FPGA on-board verification. The test finds that the actual function of the circuit is consistent with the expectation, and the output is consistent with the theoretical value. In this embodiment, e=0.02, a total of 20 storage units are used, and the data bit width of each unit is 8 bits. Using the ordinary lookup table method, 64 storage units are needed, and the data bit width of each unit is 8 bits. The storage usage of the present embodiment is reduced by 68.8% compared with the ordinary lookup table method. In another test result of e=0.002, a total of 296 storage units are used, and the data bit width of each unit is 11 bits. Using the ordinary lookup table method, 512 storage units are needed, and the data bit width of each unit is 11 bits. The storage usage is reduced by 42.2% compared with the ordinary lookup table method. Figure 4 For the functional simulation waveform of this embodiment, it can be seen from the figure that the calculation is completed within three clock cycles.

[0064] The above embodiment is only used to illustrate the technical solutions of the present application, but not to limit it. The protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A digital circuit module for computing a tanh function based on a range addressable lookup table, characterized by, The application relates to a circuit for calculating the value of a hyperbolic tangent function, comprising: an input module, an address mapping module, a temporary register module, a memory module, a selection output module, and a control logic unit; the input module accepts external input data data as the independent variable of a tanh function, and decomposes the input data data into a sign bit sign and an unsigned number unsigned, as the output of the input module; the address mapping module accepts the unsigned number unsigned given by the input module and an external input overflow bit, processes to generate the address addr2 of the corresponding function value in the memory module, and generates a read enable signal ren; the mapping rule of the address mapping module is determined by the following method: dividing the interval [x_start, 2 n0 ) into N sub-intervals, each sub-interval covers the whole interval from the start to the end, and does not overlap with each other; after x_start is determined, x_start is the start point of the first sub-interval, also recorded as x0, the end point of the first sub-interval is x1, which is also the start point of the second sub-interval, and so on, the variable k is used to represent the label of a sub-interval, k takes 1, 2, 3, …, N; x0 and the end point xk of the kth sub-interval are n-1 bit unsigned fixed point numbers, the decimal width is n1, and the integer width is n0, xk is selected as the largest number that can satisfy tanh(xk)-tanh(xk-1)≤2(e-1 / 2 n2 ) under this format, that is, starting from x0, x0 is increased by 1 from the least significant bit to obtain the next bit number x of x0 that can be represented under this format, and then it is checked whether the number x satisfies tanh(xk)-tanh(xk-1)≤2(e-1 / 2 n2 ), until it does not satisfy, the last number x that satisfies the above inequality is taken as x1, and the rest of xk is also determined according to this rule; according to this rule, the Kth sub-interval is determined, so that xK≥2 n0 , at this time all sub-intervals in the interval [x_start, 2 n0 ) have been divided, the end point xK of the last sub-interval takes 2 n0 , and the total number of sub-intervals N=K; the temporary register module stores the unsigned number unsigned output by the input module; the memory module has two modes of reading and writing; when an external write enable signal wen is 1, the memory module stores external input data data as data, and the storage address is an external input address addr1; when the read enable signal ren generated by the address mapping module is 1, the memory module outputs the data stored at the address addr2; The storage data values of the memory module are: at address k-1, a number closest to (tanh(xk)-tanh(xk-1)) / 2 represented in a storage data format, which is an unsigned fixed-point number without integer bits and with a decimal bit width of n2-1; at address N, a number closest to (1-tanh(2 n0 )) / 2 represented in the data format. the selection output module selects one of the output data of the temporary register module or the memory module according to the control signal given by the control logic unit, and outputs the selected data after removing the integer bit, adding the sign bit sign and zero padding at the end; the selection logic of the selection output module is that when the unsigned number unsigned of the input data data is less than a pre-set boundary value x_start and the overflow bit is 0, the unsigned number unsigned is directly output after removing the integer bit, adding the sign bit sign and zero padding at the end, otherwise, the function value of the input data data is searched in the memory module, and then the sign bit sign is added to output the data; the data bit width n, the integer bit width n0 and the decimal bit width n1 of the input data data, the data bit width n2 of the output data output, and the boundary value x_start of the selection output module are determined by the following method: firstly, the maximum absolute error e of calculating the tanh function is determined; Then determine n2, the selected n2 satisfies 1 / 2 n2 ≤ e / 10; Next, determine n1, to satisfy 1 / 2 n1 ≤ e and 1 / 2 n1 ≤ 2 (e - 1 / 2 n2 ); Then determine n0, to meet 1-tanh(2 n0 )≤2(e-1 / 2 n2 ), n0 select the minimum value to meet the conditions; then, n is determined to satisfy n=1+n0+n1; finally, x_start is determined to be the solution of the equation x-tanh(x)=e. The control logic unit coordinates the execution of each module, and the entire calculation process is completed in 3 clock cycles under its control, and outputs a calculation completion signal fin; when the externally input reset signal rst is 1, the entire digital circuit module is reset; in the first clock cycle, the input data passes through the input module and the address mapping module to obtain the ren signal and the address addr, and the data unsigned that needs to be temporarily stored; in the second clock cycle, the output values of the temporary storage register module and the memory module are selected according to the signals given by the control logic unit; in the third clock cycle, the selected data is output through the output register, and at the same time, the calculation completion signal fin becomes 1, indicating that the calculation has been completed.

2. The digital circuit module for computing tanh function based on range addressable lookup table according to claim 1, wherein, The demarcation value x_start of the selection output module is the value closest to the solution of the equation x-tanh(x)=e in the fixed-point number format.

3. The digital circuit module for computing tanh function based on range addressable lookup table according to claim 1, wherein, Before performing the calculation operation, the storage data value stores the corresponding data in the memory module.

4. The digital circuit module for computing tanh function based on range addressable lookup table according to claim 1, wherein, The address addr is selected by the control logic unit from the two signals of address addr1 and address addr2, and the logic of selecting the above two data is: when the externally given write enable signal wen is 1, the address addr1 is selected as the address addr; when the write enable signal wen is 0, the address addr2 is selected as the address addr.

5. The digital circuit module for computing tanh function based on range addressable lookup table according to claim 1, wherein, The mapping rule of the address mapping module is: when overflow=0, if the input unsigned number is in the kth partition, output k-1 in binary form as the address addr2; when overflow=1, output N in binary form as the address addr2.

Images