Method and system for implementing computation of tanh activation function based on FPGA

Abstract

The application relates to a calculation method for realizing a Tanh excitation function based on FPGA, which comprises the following steps: obtaining the quantization bit number during FPGA realization, determining the decimal bit quantization bit number, determining the accuracy of input and output values of the Tanh excitation function and the range of the output values based on the decimal bit quantization bit number, processing the values greater than 0 based on four linear fittings, one quadratic fitting and two table storage respectively, and mapping and processing the values less than 0 based on the odd symmetry of the Tanh function. The application can realize the Tanh function with high accuracy under the condition that the FPGA resource is limited.

Description

Technical Field

[0001] This invention relates to the field of computer vision and artificial intelligence application technology, specifically to a method and system for calculating the Tanh activation function based on FPGA. Background Technology

[0002] With the continuous development of computer performance and in-depth research in the field of machine learning, convolutional neural networks (CNNs) have become one of the most popular machine learning algorithms in recent years. Due to their complex network structure and training methods, CNNs are extremely computationally intensive algorithms. Traditional CNNs rely on general-purpose central processing units (CPUs) for computation, which is not only slow and inefficient but also difficult to meet real-time requirements. Currently, hardware implementation of CNNs mainly focuses on acceleration based on graphics processing units (GPUs). GPU-based CNNs can effectively combine software and hardware, especially since GPU manufacturers such as NVIDIA and AMD have optimized deep learning computations, significantly accelerating neural network training. However, the enormous power consumption of GPUs limits the application of CNNs in mobile systems. Besides GPUs, ASIC chips can also accelerate CNN training because they are low-power and offer high processing speeds. However, the extremely high price and low flexibility of ASIC chips make them unsuitable for the flexible structure of CNNs. As a reprogrammable hardware architecture, FPGA can not only achieve low-power computing, but its parallel computing characteristics are also a perfect match for convolutional neural networks. Therefore, FPGA-based convolutional neural network accelerators have become the mainstream hardware design direction.

[0003] A typical CNN consists of multiple computational layers, including convolutional layers and optional subsampling layers. Each neuron is connected to multiple neurons in the previous layer. A typical convolutional layer transforms N input feature maps into M output feature maps, with the transformation relationship determined by N×M convolutional kernels, i.e., the connection weights between neurons in two layers. The convolutional results of all input feature maps are summed and added to the bias, and finally, a nonlinear transformation by an activation function (commonly Sigmoid, Tanh, ReLU, etc.) is applied to obtain a single point value on the output feature map, i.e., a single output neuron. However, the activation function Tanh, due to the need for exponential calculations, imposes significant resource and computational overhead on FPGA implementations, and the implementation accuracy may not always meet requirements. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and system for calculating the Tanh excitation function based on FPGA, which can achieve high-precision implementation of the Tanh excitation function while minimizing resource overhead and saving storage resources.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for calculating the Tanh excitation function based on FPGA includes the following steps:

[0007] Obtain the quantization bit depth for FPGA implementation, determine the fractional quantization bit depth, and based on the fractional quantization bit depth, determine the precision of the input and output values ​​of the Tanh excitation function, as well as the range of the output values;

[0008] For input values ​​greater than 0, processing is performed based on four-segment linear fitting, one-segment quadratic fitting, and two-segment table storage respectively;

[0009] The mapping process for values ​​less than 0 is based on the odd symmetry of the Tanh function.

[0010] Furthermore, the number of decimal quantization bits is 8 bits.

[0011] Furthermore, the determination of the range of input and output values ​​of the Tanh excitation function based on the number of decimal quantization bits specifically involves:

[0012] For the excitation function - hyperbolic tangent function Both the input x and the output y are fractionally quantized to 8 bits, and the highest precision of the input x and the output y is HP = 1 / 2^8 = 0.00390625. The output range of y is [-1, +1].

[0013] Furthermore, when the input x is greater than 0, it is processed based on four-segment linear fitting, one-segment quadratic fitting, and two-segment table storage, as follows:

[0014] Given input x = (0, 0.30859375], the output y is represented by a first-order linear fit:

[0015] y = 249 * HP * x + HP;

[0016] With input x = (0.30859375, 0.57421875], the output y is represented by a first-order linear fit: y = 216*HP*x + 10*HP;

[0017] With input x as (0.57421875, 0.84765625], the output y is represented by a first-order linear fit: y = 164*HP*x + 39*HP;

[0018] With input x = (0.84765625, 1.14453125], the output y is represented by a first-order linear fit: y = 112 * HP * x + 82 * HP;

[0019] Given input x as (1.14453125, 2.17578125], the output y is represented by a quadratic fit:

[0020] y=-37*HP*x*x+160*HP*x+75*HP;

[0021] Given input x as (2.17578125, 2.48828125], the output y is stored as a fixed value in the table.

[0022] y = 0.984375;

[0023] Given the input x as (2.48828125, 3.46484375], the output y is stored as a fixed value in the table.

[0024] y = 0.99609375;

[0025] Given input x = (3.46484375, +∞], the output y = 1.

[0026] Furthermore, when the input x is less than 0, mapping processing is performed based on the odd symmetry of the Tanh function, as follows:

[0027] When x = 0, y = 0; when x > 0, y = y + ; then when x < 0, y = -y + .

[0028] Based on this mapping relationship, the output value can be obtained for all input ranges.

[0029] A computational system for implementing the Tanh activation function based on FPGA includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the computational method for implementing the Tanh activation function based on FPGA as described above.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] This invention is based on fixed-point processing implemented on actual FPGAs. While achieving high-precision implementation of the Tanh function, it has low resource overhead and saves a lot of storage resources. It can achieve high-precision Tanh function under FPGA resource constraints. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2This is a complete implementation flowchart of one embodiment of the present invention;

[0034] Figure 3 This is a fitting graph of the precision of FPGA implementation and full-precision quantization (8 bits) in one embodiment of the present invention;

[0035] Figure 4 This is an error diagram of the precision and full-precision quantization (8 bits) implemented by FPGA in one embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] Please refer to Figure 1-4 This invention addresses the problems of high resource overhead and poor accuracy in implementing the Tanh activation function in convolutional neural networks on FPGAs. It proposes a method with lower resource overhead, saving significant storage resources, and enabling high-precision Tanh function implementation even under FPGA resource constraints. The details are as follows:

[0038] Obtain the quantization bit depth for FPGA implementation, determine the fractional quantization bit depth, and based on the fractional quantization bit depth, determine the precision of the input and output values ​​of the Tanh excitation function, as well as the range of the output values;

[0039] For input values ​​greater than 0, processing is performed based on four-segment linear fitting, one-segment quadratic fitting, and two-segment table storage respectively;

[0040] The mapping process for values ​​less than 0 is based on the odd symmetry of the Tanh function.

[0041] In this embodiment, since the FPGA only supports fixed-point arithmetic, all data needs to be quantized. The FPGA implementation's fractional quantization is determined to be 8 bits, and the input and output ranges of the excitation function Tanh are determined based on this 8-bit fractional quantization. For the Tanh hyperbolic tangent function implemented using the FPGA, tanh... Both the input x and the output y are quantized to 8 bits, and the highest precision of the input x and the output y is HP = 1 / 2^8 = 0.00390625. The input range of x is (-∞, +∞), and the decimal value is represented as an 8-bit value. The output range of y is [-1, +1], and the decimal value is represented as an 8-bit value.

[0042] Based on the quantized data, processing is performed separately for x equal to 0, x > 0, and x < 0. When the input x equals 0, y equals 0. For values ​​of x greater than 0, a four-segment linear fitting, a one-segment quadratic fitting, and a two-segment table storage method are used to replace traditional exponential calculations or simple table storage methods. The specific implementation method is as follows:

[0043] With input x = (0, 0.30859375], the output y is represented by a first-order linear fit: y = 249 * HP * x + HP;

[0044] With input x = (0.30859375, 0.57421875], the output y is represented by a first-order linear fit: y = 216*HP*x + 10*HP;

[0045] With input x as (0.57421875, 0.84765625], the output y is represented by a first-order linear fit: y = 164*HP*x + 39*HP;

[0046] With input x = (0.84765625, 1.14453125], the output y is represented by a first-order linear fit: y = 112 * HP * x + 82 * HP;

[0047] Given input x as (1.14453125, 2.17578125], the output y is represented by a quadratic fit:

[0048] y=-37*HP*x*x+160*HP*x+75*HP;

[0049] Given the input x as (2.17578125, 2.48828125], the output y is stored as a fixed value: y = 0.984375;

[0050] Given the input x as (2.48828125, 3.46484375], the output y is stored as a fixed value: y = 0.99609375;

[0051] Given the input x as (3.46484375, +∞], the output y = 1;

[0052] For input values ​​x < 0, a mapping based on the odd symmetry of the Tanh function is used to obtain all output y values ​​within the range where x is less than 0. That is, when x = 0, y = 0; when x > 0, y = y + ; therefore, when x < 0, y = -y + . This mapping relationship yields output values ​​for all input ranges.

[0053] refer to Figure 3 In this embodiment, the fitting degree between the value y of the Tanh function and the full-precision (8-bit) miniaturization is realized based on the method of the present invention on FPGA (only the region x>0 is drawn, the region x<0 is symmetrical and has consistent performance). The result achieved by this method is basically consistent with the full-precision result.

[0054] refer to Figure 4In this embodiment, the error between the value y of the Tanh function implemented by the FPGA based on the method of the present invention and the full precision (8-bit) fractional quantization is shown (only the region x>0 is drawn, the region x<0 is symmetrical and has consistent performance). It can be seen that the difference between all the implementation results and the full precision does not exceed 1 decimal place error, that is, 1 / 2^8 = 0.00390625, thus achieving high precision.

[0055] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A method for calculating a Tanh activation function based on FPGA, characterized in that, The method comprises the following steps: acquiring the quantization bit number in FPGA implementation, determining the decimal bit quantization bit number, and determining the input and output value accuracy of the Tanh excitation function and the output value range based on the decimal bit quantization bit number; processing the input value greater than 0 based on four linear fitting, one quadratic fitting, and two table storage respectively; performing mapping processing on the value less than 0 based on the odd symmetry of the Tanh function; the decimal bit quantization bit number is 8 bits; the input value and output value range of the Tanh excitation function is determined based on the decimal bit quantization bit number, and specifically: The input x and the output y of the activation function - hyperbolic tangent function tanh x = Both the input x and the output y are subjected to 8-bit small quantization, and the highest precision of the input x and the output y is HP = 1 / 2^8 = 0.00390625, and the output range of y is [-1, +1]; when the input x is greater than 0, the processing is respectively based on four linear fittings, one quadratic fitting, and two table storage, and the specific processing is as follows:​ when the input x is (0, 0.30859375], the output y is expressed by linear fitting; y = 249 HP x + HP; For input x in (0.30859375, 0.57421875], output y is represented by a linear fit of degree one: y = 216 HP x + 10 HP; For input x in (0.57421875, 0.84765625], output y is represented by a linear fit of the form: y = 164 HP x + 39 HP; For input x in (0.84765625, 1.14453125], output y is represented by a linear fit of degree one: y = 112 HP x + 82 HP; when the input x is (1.14453125, 2.17578125], the output y is expressed by quadratic fitting; y = -37 HP x x + 160 HP x + 75 HP; when the input x is (2.17578125, 2.48828125], the output y is a table fixed value; y=0.984375; when the input x is (2.48828125, 3.46484375], the output y is a table fixed value; y=0.99609375; when the input x is (3.46484375, +∞), the output y = 1.

2. The method of claim 1, wherein the Tanh activation function is implemented based on an FPGA. when the input x is less than 0, the mapping processing is performed based on the odd symmetry of the Tanh function, and specifically as follows: when x = 0, y = 0; when x > 0, y = y +; then when x < 0, y = -y +; thus, the output value in all input ranges is obtained through the mapping relationship.

3. A system for implementing the method of claim 1 or 2 for computing a Tanh activation function based on FPGA, characterized in that, a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to specifically execute the steps in the Tanh excitation function calculation method based on FPGA implementation according to any one of claims 1-2.

Images