Calculation method and device of neural network activation function and storage medium

Abstract

The embodiment of the invention discloses a calculation method and device of a neural network activation function and a medium. The method comprises the following steps: dividing an activation function input interval into an activation area and a universal area according to calculation characteristics of different activation functions; calculating table items of the active area and the universal area; and loading the table items to different calculation units, carrying out type conversion and judgment on the input data, and carrying out table look-up calculation in different forms according to the size of the input data. According to the method, a more efficient result is obtained by using a kb table item aiming at the characteristic of stable change of a universal region of an activation function; according to the importance and change intensity of an active region, the precision and the running speed of an activation function are improved by adopting a mode of combining a lookup table and piecewise linear approximation. In the operation of the neural network model, the operation speed of the activation function operator can be increased by looking up the table, the operator precision is more flexibly improved by increasing the table item difference, and the storage space of the table item is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network computing, and in particular, to a calculation method, device, and storage medium for a neural network activation function. Background Art

[0002] To accelerate the calculation of activation functions, the common implementation methods on hardware currently mainly include the look-up table method, piecewise linear approximation method, Taylor series expansion, etc. For example, the Chinese patent application with the publication number CN112749803A discloses a quantization method for calculating the activation function of a neural network. According to different types of activation functions, different look-up tables are generated, the input floating-point data is quantized into fixed-point data, and the look-up table operation is directly performed according to the result to obtain the fixed-point result of its output. Another example is the Chinese patent application with the publication number CN110688088A, which discloses a general non-linear activation function calculation device and method for neural networks. This solution uses the piecewise linear approximation method to perform piecewise linear fitting on the sigmoid function, and uses the mathematical relationship between the sigmoid function and the tanh function to share the address index unit and look-up table resources, and performs operations on different non-linear functions in the neural network through mode selection and corresponding linear changes.

[0003] However, when high precision is required, the linear approximation must increase the number of segmentation intervals and the number of table entries, and the look-up table method must increase the number of table entries, etc. This will consume a large amount of storage resources and is only applicable to a limited input interval. The fusion implementation of different methods is a relatively cutting-edge research direction at present. Summary of the Invention

[0004] Aiming at the technical defects mentioned in the background art, the purpose of the embodiments of the present invention is to provide a calculation method, device, and storage medium for a neural network activation function.

[0005] To achieve the above object, in a first aspect, the embodiments of the present invention provide a calculation method for a neural network activation function, including:

[0006] According to the calculation characteristics of different activation functions, divide the input interval of the activation function into an activation area and a general area, and obtain the input quantization coefficient and the output quantization coefficient;

[0007] Calculate a first table entry and a second table entry based on the input quantization coefficient and the output quantization coefficient; wherein, the first table entry corresponds to the activation area, and the second table entry corresponds to the general area;

[0008] Load the first table entry and the second table entry into different calculation units, perform type conversion and judgment of the input data, and perform different forms of look-up table calculation according to the size of the input data.

[0009] Among them, the input quantization coefficient is the quantization coefficient obtained by converting the floating-point input into fixed-point data, which is calculated based on the floating-point boundary point and the fixed-point boundary point;

[0010] The output quantization coefficient is the quantization coefficient obtained by converting the fixed-point data into floating-point output, which is given according to experience or comprehensive test results.

[0011] As a specific implementation manner of the present application, calculating the first table entry specifically includes:

[0012] For the activation region, generate multiple fixed-point inputs;

[0013] Convert the fixed-point input into a floating-point input according to the input quantization coefficient;

[0014] Send the floating-point input into the activation function to obtain the real floating-point output;

[0015] Convert the floating-point output into a fixed-point output according to the output quantization coefficient;

[0016] Generate a mapping table of the input-to-output correspondence relationship based on the multiple fixed-point inputs and fixed-point outputs, that is, the first table entry.

[0017] As a specific implementation manner of the present application, calculating the second table entry specifically includes:

[0018] For the general region, obtain the slope k and intercept b in different directions;

[0019] If the (k, b) value in the forward general region of the activation function is (1.0, 0), and the (k, b) value in the negative general region is (0, 0);

[0020] Apply the existing calculation rules of the input-output quantization coefficient to convert the floating-point (k, b) value into a fixed-point (k, b) value as the second table entry.

[0021] As a specific implementation manner of the present application, perform different forms of look-up table calculations according to the size of the input data, specifically:

[0022] If it is a floating-point input, convert the floating-point input into a fixed-point input;

[0023] If the size of the fixed-point input is within the INT-2n range, perform the look-up table calculation in the activation region;

[0024] If the size of the fixed-point input exceeds the INT-2n range, perform the look-up table calculation in the general region.

[0025] Among them, performing the look-up table calculation in the activation region specifically includes:

[0026] Obtain the query data of the fixed-point input, and split the query data to obtain the first index and the second index;

[0027] Perform a look-up table operation according to the first index, find LUT[N] and LUT[N + 1] among 257 table entries, and calculate the width of the small interval;

[0028] Multiply the second index by the width of the small interval to obtain the distance diff between the floating-point value within the small interval and the value of the left boundary point;

[0029] Add the final left boundary value LUT[N] of the small interval and the distance diff to obtain the final fixed-point output result.

[0030] Among them, the look-up table calculation in the general area is specifically as follows:

[0031] Obtain the query data of the fixed-point input, select the positive and negative (k, b) table entries with the query data, and perform the calculation process of kx + b to obtain the final fixed-point output result.

[0032] In a second aspect, an embodiment of the present invention further provides a calculation device for a neural network activation function, including a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. Among them, the memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to call the program instructions to execute the method of the first aspect above.

[0033] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method of the first aspect above.

[0034] Implementing the embodiments of the present invention has the following advantages:

[0035] In the existing solutions, often only the accuracy of a limited input interval can be taken into account, or the accuracy can be improved by increasing the number of table entries. In the embodiments of the present invention, by dividing the input interval of the neural network activation function and implementing different look-up table methods for different regions. Specifically, for the characteristic of stable change in the general area of the activation function, using the kb table entries to obtain more efficient results; for the importance and severity of change in the activation area, a combination of look-up tables and piecewise linear approximation is adopted to improve the accuracy and operating speed of the activation function. That is, during the operation of the neural network model, the embodiments of the present invention can improve the operating speed of the activation function operator through look-up tables, improve the operator accuracy more flexibly by increasing the table entry difference, and reduce the storage space of the table entries.

[0036] In addition, in the process of mutual conversion between floating-point and fixed-point before and after the operation, the differences between floating-point numbers can be amplified by fine-tuning the quantization coefficients to increase the differences between table entries, thereby improving the accuracy and greatly increasing the flexibility and upper limit of accuracy control. Brief Description of the Drawings

[0037] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art.

[0038] Figure 1 It is a flowchart of the calculation method of the neural network activation function provided by the embodiment of the present invention;

[0039] Figure 2 It is a schematic diagram of a common neural network function;

[0040] Figure 3 It is a mapping relationship diagram of floating-point input and fixed-point input;

[0041] Figure 4 It is a tabulation logic diagram of the activation area search mapping table;

[0042] Figure 5 It is a schematic diagram of the activation area table lookup operation unit;

[0043] Figure 6 It is a schematic diagram of a general-purpose differential segment operation unit;

[0044] Figure 7 It is a structural diagram of the calculation device of the neural network activation function provided by the embodiment of the present invention. Detailed Embodiments

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0047] Explanation of Related Terms:

[0048] General Region: The input interval where the gradient of the activation function hardly changes

[0049] Activation region: The input interval where the gradient of the activation function changes drastically

[0050] Floating-point data: The original input floating-point type data of the activation function

[0051] Fixed-point data: The fixed-point (integer) type data after quantization of the original input data

[0052] Quantization coefficient: The conversion coefficient between floating-point data and fixed-point data

[0053] The inventive concept of the present invention is to propose a software and hardware solution that combines the use of a lookup table and piecewise linear approximation, and divides the input interval of the activation function into an activation region and a general region: for the general region, the gradient of the activation function hardly changes, and the fitting method of kx + b can meet the accuracy requirements and save the number of table entries; for the activation region, the gradient of the activation function changes relatively drastically, and the lookup table method is used for calculation. Through the tabulation algorithm of this solution, the accuracy level of combining the two methods of lookup table and piecewise linear approximation, as well as a smaller number of table entries and better computational performance, are achieved.

[0054] The technical solution of the present invention can be summarized into the following two parts:

[0055] (I) Software calculation part:

[0056] Interval division: In this part, according to the calculation characteristics of different activation functions, it is divided into an activation region and a general region, and the quantization coefficients of input and output are obtained, etc.

[0057] Table entry generation: In this part, n table entries for the activation region and (k, b) table entries for the general region are calculated respectively.

[0058] (II) Hardware device part:

[0059] In this part, first, the table entries of the software part need to be loaded into the hardware device, and the type conversion and judgment of the input data are performed. According to the different sizes of the input, they are respectively sent to different hardware calculation units for different forms of table lookup calculation.

[0060] Please refer to Figure 1 , which is the calculation method of the neural network activation function provided by the embodiment of the present invention, and includes the following steps:

[0061] S1, according to the calculation characteristics of different activation functions, divide the input interval of the activation function into an activation region and a general region, and obtain the input quantization coefficient and the output quantization coefficient.

[0062] For conventional activation functions such as sigmoid, tanh, etc., such as Figure 2As shown, in order to enhance the training efficiency and stability of a neural network, there is usually an input interval where the gradient changes significantly. During backpropagation, the weight parameters of neurons are quickly changed or differentiated, which is the activation region described in this embodiment; in other regions, the gradient changes gently or approaches a constant, which is generally referred to as the saturation region, that is, the general region described in this embodiment.

[0063] Taking floating-point ±x as the boundary points, assuming that within the floating-point [-x, x) interval, the trend of a conventional activation function changes significantly, that is, [-x, x) is used as the activation region, and outside this interval, it tends to be a straight line, with (-∞, -x) and [x, +∞) as the general regions. During the fixed-point calculation process, set ±X (bit length of 2n) as the boundary points for fixed-point calculation. The activation region for fixed-point calculation can be expressed as [-X, X), or [-2 (2n-1 ), 2 (2n-1) -1), and can also be expressed as the upper and lower bounds of INT-2n. The general region can be analogously represented as that of the floating-point region. The quantization coefficient for converting floating-point input to fixed-point data can be calculated through the boundary points of floating-point and fixed-point, and the quantization coefficient for converting fixed-point data to floating-point output can be given by experience or comprehensive test results.

[0064] Among them, the mapping relationship between floating-point input and fixed-point input can be referred to Figure 3 .

[0065] S2. Calculate the first table entry and the second table entry based on the input quantization coefficient and the output quantization coefficient.

[0066] Among them, the first table entry corresponds to the activation region, and the second table entry corresponds to the general region.

[0067] For the generation process of the table entry for the general region (i.e., the second table entry), first obtain the slopes k and intercepts b in different directions. Taking the Relu activation function as an example, the (k, b) values in its positive general region are (1.0, 0), and the (k, b) values in its negative general region are (0, 0). Apply the existing calculation rules for input-output quantization coefficients to convert the floating-point (k, b) values into fixed-point (k, b) values as the table entries for the general region.

[0068] For the generation process of the table entry for the activation region (i.e., the first table entry), as Figure 4 shown, a number of fixed-point inputs need to be generated, and the corresponding table entries of the fixed-point outputs are calculated to form a mapping table of the input-to-output correspondence. The specific process is as follows:

[0069] In this embodiment, use INT-2n as the upper and lower bounds of the activation region. When making the table, obtain any combination of 2 n data of n-bit as the first n bits, and the last n-bit are 0. The combination can obtain 2 nData of size INT-2n, and additionally capture the right boundary point of the activation region, a total of 2 n +1 fixed-point type data as the fixed-point input of the table entry, and convert it into its corresponding floating-point input through the quantization coefficient of the input data. After obtaining the floating-point input corresponding to the fixed-point input, send the floating-point input into the activation function to obtain the true floating-point output, and convert it into a fixed-point output through the output quantization coefficient as the table entry of the activation region.

[0070] It should be noted that in the tabulation logic, the scaling effect can be transferred to specific table entries by changing the quantization coefficients of the input and output. The calculation accuracy can be improved by increasing the difference between table entries. The specific optimization ideas include moderately increasing the K value of the stable region, moderately increasing the size of 2 n +1 table entries and the differences between them, etc.

[0071] S3, load the first table entry and the second table entry into different computing units, perform type conversion and judgment of the input data, and perform different forms of table look-up calculations according to the size of the input data.

[0072] When specifically implemented, different forms of table look-up calculations are performed according to the size of the input data, specifically:

[0073] If it is a floating-point input, convert the floating-point input into a fixed-point input;

[0074] If the size of the fixed-point input is within the range of INT-2n, perform table look-up calculation in the activation region;

[0075] If the size of the fixed-point input exceeds the range of INT-2n, perform table look-up calculation in the general region.

[0076] Furthermore, the specific process of performing table look-up calculation in the activation region is as follows:

[0077] Obtain the query data of the fixed-point input, and split the query data to obtain the first index and the second index;

[0078] Perform table look-up operation according to the first index, find LUT[N] and LUT[N + 1] among 257 table entries, and calculate the width of the small interval;

[0079] Multiply the second index by the width of the small interval to obtain the distance diff between the floating-point value in the small interval and the value of the left boundary point;

[0080] Add the final left boundary value LUT[N] of the small interval and the distance diff to obtain the final fixed-point output result.

[0081] Furthermore, the specific process of performing table look-up calculation in the general region is as follows:

[0082] Obtain the query data of the fixed-point input, select the positive and negative (k, b) table entries with the query data, and perform the calculation process of kx + b to obtain the final fixed-point output result.

[0083] After completing the table-making process of the software part, it is necessary to store the table entry information in different regions into different computing units of the hardware, and perform the conversion process of the input data. If it is a floating-point input, it is first necessary to convert it into a fixed-point input, and then send it to the interval judgment unit of the hardware. The following description will be made with n being 8 as an example.

[0084] If the size of the fixed-point input is within the range of INT-2n, that is, INT16, then perform the table-lookup operation process in the activation area, as Figure 5 shown: Obtain the low 16-bit INT16 data of the fixed-point input data, and use the first eight bits and the last eight bits of the INT16 data as indexes respectively to perform the table-lookup operation. The high eight bits H8 find LUT[N] and LUT[N + 1] among 257 table entries according to the specific INT8 size N, and calculate the interval size. This step can be regarded as the floating-point [-x, x) being divided into 256 intervals, and the corresponding small interval and its width are found according to the floating-point input range; the result obtained by multiplying the low eight bits L8 by the small interval width can be regarded as the distance diff between the floating-point value in the small interval and the value of the left boundary point; finally, adding the left boundary value LUT[N] of the small interval and diff can obtain the final fixed-point output result.

[0085] If the data exceeds the representation range of INT-2n, that is, INT16, then perform the calculation process in the general area, as Figure 6 shown. Obtain the high 16-bit INT16 data of the input fixed-point data (the reduction multiple of the input data is offset by increasing the output quantization coefficient), select the positive and negative (k, b) table entries according to the obtained INT16 data, and thus perform the calculation process of kx + b to obtain the final fixed-point output result, and select whether to convert the fixed-point output into a fixed-point output according to actual needs.

[0086] It should be noted that the software calculation scheme is based on int2n. For the convenience of description, int16 is selected for example, and int16 can be int24 or int28.

[0087] From the above description, it can be known that when implementing the embodiments of the present invention, the advantages are as follows:

[0088] In existing solutions, often only the accuracy of a limited input range can be taken into account, or the accuracy can be improved by increasing the number of table entries. In the embodiments of the present invention, the input range of the neural network activation function is segmented, and different table lookup methods are implemented for different regions. Specifically, for the characteristic of stable change in the general area of the activation function, the kb table entries are used to obtain more efficient results; for the importance and drastic change degree of the activation area, a combination of a lookup table and piecewise linear approximation is adopted to improve the accuracy and running speed of the activation function. That is, during the operation of the neural network model, the embodiments of the present invention can improve the running speed of the activation function operator by table lookup, more flexibly improve the operator accuracy by increasing the difference in table entries, and reduce the storage space of the table entries.

[0089] In addition, in the process of mutual conversion between floating-point and fixed-point before and after the operation, the differences between floating-point numbers can be enlarged by fine-tuning the quantization coefficient to increase the differences between table entries, thereby improving the accuracy and greatly increasing the flexibility and upper limit of accuracy control.

[0090] Based on the same inventive concept, as Figure 7 shown, the embodiments of the present invention further provide a calculation device for a neural network activation function, which may include: one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104. The above-mentioned processors 101, input devices 102, output devices 103, and memory 104 are interconnected through a bus 105. The memory 104 is used to store a computer program, the computer program includes program instructions, and the processor 101 is configured to call the program instructions to execute the method in the method embodiment part described above.

[0091] It should be understood that in the embodiments of the present invention, the so-called processor 101 may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0092] The input device 102 may include a keyboard, etc., and the output device 103 may include a display (such as an LCD), a speaker, etc.

[0093] The memory 104 may include a read-only memory and a random access memory, and provide instructions and data to the processor 101. A part of the memory 104 may also include a non-volatile random access memory. For example, the memory 104 may also store information about the device type.

[0094] In a specific implementation, the processor 101, input device 102, and output device 103 described in the embodiments of the present invention may execute the implementation manners described in the embodiments of the calculation method of the neural network activation function provided by the embodiments of the present invention, which will not be elaborated here.

[0095] Correspondingly, the embodiments of the present invention provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the following is implemented: the above-mentioned calculation method of the neural network activation function.

[0096] The computer-readable storage medium may be an internal storage unit of the system described in any of the foregoing embodiments, such as the hard disk or memory of the system. The computer-readable storage medium may also be an external storage device of the system, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the system. Further, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the system. The computer-readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium may also be used to temporarily store data that has been output or will be output.

[0097] As described above, the above are only specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A calculation method for a neural network activation function, characterized in that, Including: According to the calculation characteristics of different activation functions, divide the input interval of the activation function into an activation region and a general region, and obtain an input quantization coefficient and an output quantization coefficient; Calculate a first table entry and a second table entry based on the input quantization coefficient and the output quantization coefficient; wherein, the first table entry corresponds to the activation region, and the second table entry corresponds to the general region; Load the first table entry and the second table entry into different calculation units to perform type conversion and judgment of input data, and perform different forms of look-up table calculation according to the size of the input data.

2. The calculation method according to claim 1, characterized in that, The input quantization coefficient is a quantization coefficient for converting a floating-point input into a fixed-point data, which is calculated based on floating-point boundary points and fixed-point boundary points; The output quantization coefficient is a quantization coefficient for converting fixed-point data into a floating-point output, which is given according to experience or comprehensive test results.

3. The calculation method according to claim 2, wherein, Specifically, calculating the first table entry is as follows: For the activation region, generate multiple fixed-point inputs; Convert the fixed-point inputs into floating-point inputs according to the input quantization coefficient; Send the floating-point inputs into the activation function to obtain the real floating-point output; Convert the floating-point output into a fixed-point output according to the output quantization coefficient; Generate a mapping table of the input-to-output correspondence relationship according to multiple fixed-point inputs and fixed-point outputs, that is, the first table entry.

4. The calculation method according to claim 3, characterized in that, Specifically, generating multiple fixed-point inputs is as follows: Taking the upper and lower bounds of INT-2n as the activation region, obtaining any combination of 2 n data as the first n bits; Capture the right boundary point of the activation area, and use 2 n +1 fixed-point type data as the fixed-point input of the table entry.

5. The calculation method according to claim 2, characterized in that Specifically, calculating the second table entry is as follows: For the general region, obtain slopes k and intercepts b in different directions; If the (k, b) values in the positive direction of the activation function general region are (1.0, 0), and the (k, b) values in the negative direction of the general region are (0, 0); Apply the existing calculation rules of input and output quantization coefficients to convert the floating-point (k, b) values into fixed-point (k, b) values as the second table entry.

6. The calculation method according to claim 1, characterized in that, Perform different forms of look-up table calculation according to the size of the input data, specifically: If it is a floating-point input, convert the floating-point input into a fixed-point input; If the size of the fixed-point input is within the INT-2n range, perform the look-up table calculation in the activation region; If the size of the fixed-point input exceeds the INT-2n range, perform the look-up table calculation in the general region.

7. The calculation method according to claim 6, wherein, Specifically, performing the look-up table calculation in the activation region is as follows: Obtain the query data of the fixed-point input, and split the query data to obtain a first index and a second index; Perform a look-up table operation according to the first index, find LUT[N] and LUT[N + 1] among 257 table entries, and calculate the width of the small interval; Multiply the second index by the width of the small interval to obtain the distance diff between the floating-point value in the small interval and the value of the left boundary point; Add the final left boundary value LUT[N] of the small interval and the distance diff to obtain the final fixed-point output result.

8. The calculation method according to claim 6, characterized in that, Specifically, performing the look-up table calculation in the general region is as follows: Obtain the query data of the fixed-point input, select the (k, b) table entries in the positive and negative directions with the query data, and perform the calculation process of kx + b to obtain the final fixed-point output result.

9. A computing device for a neural network activation function, characterized in that, It includes a processor, an input device, an output device, and a memory. The processor, input device, output device, and memory are interconnected. Among them, the memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to call the program instructions and execute the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1-8.

Images