Activation method and device of neural network, NPU, equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 伟光有限公司(CN)
- Filing Date
- 2022-11-29
- Publication Date
- 2026-07-03
Smart Images
Figure CN115906971B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, NPU, device, and storage medium for activating a neural network. Background Technology
[0002] With the development of deep learning, neural networks have been widely used in various fields, and the scale of models has also developed in a wider and deeper direction. In order to reduce the consumption of computing resources and expand the application scenarios of neural networks, related technologies have reduced the model size and saved storage space by quantization while preserving the model structure.
[0003] To utilize NPUs (Neural-network Processing Units) for neural network computation, many related technologies employ LUTs (Look-Up Tables) to simplify the computation process. However, when using self-optimized LUTs for activation computation, these technologies often require adaptive recoding of the NPU, resulting in resource consumption. Summary of the Invention
[0004] This application provides a method, apparatus, NPU, device, and storage medium for activating a neural network, which reuses the quantization calculation module, improves the activation calculation effect, and simplifies the improvement process. The technical solution is as follows:
[0005] On one hand, embodiments of this application provide an activation method for a neural network, the method comprising:
[0006] The target first-level interval to which the input value belongs is determined from the first-level lookup table. The first-level lookup table contains the correspondence between the first-level interval and the interval parameter. The first-level interval is obtained by dividing the range of the input value. The input value is a fixed-point number, and the range of the input value is a range of fixed-point numbers.
[0007] Based on the target interval parameters of the target first-level interval, the target transformation parameters of the target second-level interval to which the input value belongs are determined from the second-level lookup table. The target second-level interval belongs to the target first-level interval, and each first-level interval is divided into at least one second-level interval. The second-level lookup table contains the correspondence between the second-level intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fitting of the activation function corresponding to the second-level interval.
[0008] By reusing the quantization algorithm module, based on the target transformation parameters and the input value, the activation value corresponding to the input value is determined. The activation value is a fixed-point number. The quantization algorithm module is used to perform data quantization and dequantization processing.
[0009] On the other hand, embodiments of this application provide an activation device for a neural network, the device comprising:
[0010] The system comprises a first lookup table unit, a second lookup table unit, and a calculation unit, wherein the calculation unit is connected to both the first and second lookup table units.
[0011] The first lookup table unit is used to store a first-level lookup table, which contains the correspondence between first-level intervals and interval parameters. The first-level intervals are obtained by dividing the range of input values, where the input values are fixed-point numbers and the range of input values is a range of fixed-point numbers.
[0012] The second lookup table unit is used to store the second lookup table, which contains the correspondence between the second-level intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fit of the activation function corresponding to the second-level interval.
[0013] The calculation unit is used to determine the target first-level interval to which the input value belongs from a first-level lookup table; based on the target interval parameter of the target first-level interval, determine the target transformation parameter of the target second-level interval to which the input value belongs from a second-level lookup table, wherein the target second-level interval belongs to the target first-level interval; obtain the target transformation parameter corresponding to the target second-level interval from the second-level lookup table; and, through a reused quantization algorithm module, determine the activation value corresponding to the input value based on the target transformation parameter and the input value, wherein the activation value is a fixed-point number. The quantization algorithm module is used to perform data quantization and dequantization processing.
[0014] On the other hand, embodiments of this application provide an NPU, which includes programmable logic circuits and / or program instructions, used to implement the neural network activation method as described above when the NPU is running.
[0015] On the other hand, embodiments of this application provide an NPU, which includes the activation device for the neural network described above.
[0016] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the neural network activation method as described above.
[0017] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one program that is loaded and executed by a processor to implement the neural network activation method as described above.
[0018] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the neural network activation method described above.
[0019] In this embodiment, the computer device determines the target primary interval based on the input value, reads the target interval parameters within the target primary interval, and determines the target secondary interval based on the target interval parameters. Then, it determines the target transformation parameters stored in the target secondary interval. This two-level lookup table approach improves the accuracy of activation calculation while ensuring that the lookup table occupies minimal storage space, thus improving resource utilization. Based on the target transformation parameters and input values read from the newly constructed secondary lookup table, the computer device performs activation calculations using a reused quantization algorithm module to obtain activation values. The target transformation parameters can be used to complete the activation calculation according to the inherent calculation logic of the quantization calculation module. This allows the reuse of the quantization calculation module to complete activation calculations based on the optimized LUT table, reducing the optimization cost associated with using a completely new LUT table for activation calculations. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This application shows a structural block diagram of a computer device provided in an exemplary embodiment;
[0022] Figure 2 A flowchart illustrating an exemplary embodiment of the present application provides a method for activating a neural network.
[0023] Figure 3 A schematic diagram illustrating the interval division provided in an exemplary embodiment of this application is shown;
[0024] Figure 4 A schematic diagram of a quantization calculation module provided in an exemplary embodiment of this application is shown;
[0025] Figure 5 A flowchart illustrating the activation calculation provided in an exemplary embodiment of this application is shown;
[0026] Figure 6 A schematic diagram of a multiplexed quantization calculation module provided in an exemplary embodiment of this application is shown;
[0027] Figure 7 A schematic diagram illustrating a table lookup process provided in an exemplary embodiment of this application is shown;
[0028] Figure 8 A flowchart illustrating a secondary lookup table lookup method provided in an exemplary embodiment of this application is shown.
[0029] Figure 9 A structural block diagram of an activation device for a neural network provided in an exemplary embodiment of this application is shown. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0031] For ease of understanding, the terms used in the embodiments of this application will be explained below.
[0032] Model quantization is a method of compressing models. To meet the accuracy requirements of various AI applications, the width, number of layers, depth, and various parameters of deep neural network structures have increased rapidly. This results in deep learning models occupying more storage space and requiring longer inference latency, hindering industrial deployment. Current models run on four types of chips: CPUs (Central Processing Units), GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits). However, compared to deep learning models, the computing power of these chips is limited. For chips on edge devices, there are many limitations in terms of storage, memory, power consumption, and latency, with inference efficiency being particularly important.
[0033] In the context of computer vision and deep learning, "model" specifically refers to a convolutional neural network. Quantization, or quantization, is the process of approximating a continuous value of a signal as a finite number of discrete values; it's a method of information compression. A model consists of weights and biases, both of which are stored using the float32 data type. Floating-point numbers occupy 32 bits in storage, while fixed-point numbers (int8) occupy only 8 bits. Model quantization is essentially a model compression method that uses fixed-point numbers to represent floating-point numbers for computation.
[0034] A Look-Up Table (LUT) is essentially a type of RAM (Random Access Memory). A LUT replaces the logical AND and OR gates required to obtain data with two tables similar to truth tables. The LUT stores all the results of the input variables and the output variables after passing through the logic gates. After data is pre-written into RAM, inputting a signal is equivalent to inputting an address to look up the table; if the corresponding value is found at that address, that value is output as the result.
[0035] The Least Squares Method (LSM) is a commonly used method for solving unconstrained optimization problems. In this embodiment, the LSM is used to calculate the optimal fitted line by linearly fitting discrete values. The LSM finds the best matching function for the data by minimizing the sum of squared errors. This embodiment uses the LSM for linear fitting to obtain a linear expression with smaller errors.
[0036] Please refer to Figure 1 This diagram illustrates a structural block diagram of a computer device provided in an exemplary embodiment of this application. The computer device 100 may include one or more components such as a processor 110 and a memory 120.
[0037] The processor 110 integrates an NPU (Neural Network Processing Unit) for performing neural network processing, executing neural network activation methods, and implementing artificial intelligence (AI) functions. The processor 110 may include one or more processing cores. The processor 110 connects to various parts within the computer device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by accessing data stored in memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content displayed on the touchscreen; and the modem handles wireless communication. It is understood that the modem may not be integrated into the processor 110 and can be implemented separately by a dedicated processor.
[0038] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described below, etc.; the data storage area may store data (such as audio data, telephone directory, etc.) created according to the use of the computer device 100.
[0039] In addition, those skilled in the art will understand that the structure of the computer device 100 shown in the above figures does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the computer device 100 may also include components such as an input unit, audio circuitry, a Wi-Fi module, a power supply, and a Bluetooth module, which will not be described in detail here.
[0040] Please refer to Figure 2 This document illustrates a flowchart of a neural network activation method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example, and the method includes the following steps:
[0041] Step 201: Determine the target first-level interval to which the input value belongs from the first-level lookup table. The first-level lookup table contains the correspondence between the first-level interval and the interval parameters. The first-level interval is obtained by dividing the range of the input value. The input value is a fixed number and the range of the input value is a range of fixed numbers.
[0042] Given an input value, the computer device first searches the first-level lookup table based on the input value. Based on the mapping relationship stored in the first-level lookup table, which is the correspondence between the first-level interval and the interval parameter, the computer device needs to determine its target first-level interval based on the input value, and then read the target interval parameter.
[0043] In the embodiments of this application, the LUT table is used to simplify the activation calculation process of the quantization model. Therefore, during the calculation process, the input value is a fixed-point number, and the corresponding input value range is a fixed-point number range.
[0044] In an illustrative example, such as Figure 3 As shown, the activation function interval corresponding to the first-level interval is obtained by uniformly dividing the activation function, and the division granularity is the first-level division granularity.
[0045] It should be noted that, in the embodiments of this application, the activation function used in the neural network can be the sigmoid function. It can also be the Tanh function. Or it could be the ReLU function. Nonlinear functions, etc., are not limited in this application.
[0046] Step 202: Based on the target interval parameters of the target first-level interval, determine the target transformation parameters of the target second-level interval to which the input value belongs from the second-level lookup table. The target second-level interval belongs to the target first-level interval, and each first-level interval is divided into at least one second-level interval. The second-level lookup table contains the correspondence between the second-level intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fitting of the activation function corresponding to the second-level interval.
[0047] Given a target primary interval, the corresponding target interval parameter is read from the primary lookup table. Based on the target interval parameter, the computer calculates and determines the target secondary interval corresponding to the input value, and then determines the target transformation parameter stored in the obtained target secondary interval. In one possible implementation, such as... Figure 3 As shown, the activation function intervals corresponding to the second-level intervals are obtained by non-uniformly dividing the activation function. Specifically, within the same first-level interval, the activation function is uniformly divided to obtain the second-level intervals, while the number of second-level intervals varies between different first-level intervals.
[0048] The secondary lookup table stores the correspondence between secondary intervals and transformation parameters. The transformation parameters are obtained by transforming the fitting parameters. When performing linear fitting on the activation function curves corresponding to the secondary intervals, the fitted line can be expressed as:
[0049]
[0050] Substituting the starting value of the second-level interval into Formula 1, we get:
[0051]
[0052] in, Let be the value of the starting point of the nth secondary interval. According to Formula 1 and Formula 2, the parameters in the fitted line can be simplified, and the fitted line can be expressed by the following formula:
[0053]
[0054] Furthermore, to reduce the size of the neural network model so that it can still be used in edge computing scenarios with limited computing resources, this embodiment uses model quantization for neural network computation. Accordingly, during the activation process through linear fitting, the fitted line represented by Equation 3 needs to be quantized. The formula based on linear quantization is shown below:
[0055]
[0056] Where R is a floating-point number, which is the actual data before quantization; Q is the fixed-point number obtained after quantization of the actual floating-point data; S is the quantization scale when quantizing the floating-point number; and Z is the quantization zero point, which represents the data offset introduced during the quantization process.
[0057] Substituting the linear quantization formula into Equation 3, we can obtain the quantized fitted straight line, as shown below:
[0058]
[0059] By performing a mathematical transformation on formula 4, we can obtain:
[0060]
[0061] Formula 5 uses the activation value This represents the expression for the fitted straight line, and the two parameters in this formula are the fitting parameters. Based on the slope parameter... intercept parameter The above formula can be simplified to obtain the expression for the fitted line:
[0062]
[0063] Optionally, the fitting parameters can be determined using the least squares method.
[0064] Furthermore, since the secondary lookup table stores the transformation parameters of the corresponding secondary intervals, when the target secondary index is determined, the target secondary index can be used as the lookup address to determine the target secondary interval, and the target transformation parameters stored in the target secondary interval can be read so that the activation value can be further calculated based on the target transformation parameters.
[0065] Step 203: By reusing the quantization algorithm module, the activation value corresponding to the input value is determined based on the target transformation parameters and the input value. The activation value is a fixed number. The quantization algorithm module is used to perform data quantization and dequantization processing.
[0066] When the target transformation parameters stored in the secondary lookup table are read, the activation calculation is completed using the above-mentioned fitted linear expression (Formula 5) to obtain the activation value. In this embodiment of the application, in order to reduce the optimization cost of the LUT table, the above-mentioned fitted linear expression is mathematically transformed so that its calculation logic conforms to the inherent operation logic of the quantization algorithm module, that is, the computer device can complete the activation calculation by reusing the quantization calculation module.
[0067] In summary, the computer device determines the target primary interval based on the input value, reads the target interval parameters within the primary interval, and determines the target secondary interval based on these parameters. Then, it reads the target transformation parameters stored in the secondary interval. This two-level lookup table approach improves activation calculation accuracy while ensuring the lookup table requires minimal storage space, thus increasing resource utilization. Based on the target transformation parameters and input values read from the newly constructed secondary lookup table, the computer device performs activation calculations using a reused quantization algorithm module to obtain the activation value. The target transformation parameters can be used to complete the activation calculation according to the inherent calculation logic of the quantization calculation module, thereby enabling the reuse of the quantization calculation module to perform activation calculations based on the optimized LUT table, reducing the optimization cost associated with using a completely new LUT table.
[0068] In this embodiment, the computer device completes the activation calculation by reusing the quantization algorithm module. In one possible implementation, the quantization algorithm module is used for data quantization and dequantization processing. This application takes linear quantization as an example for illustration. The quantization algorithm module includes the following calculation process:
[0069] In the process of data dequantization, linear dequantization calculation can be expressed by the following formula:
[0070]
[0071] Where R is the floating-point number obtained by dequantizing the fixed-point data; round() indicates the function to round the value; S is the scaling factor, which represents the scaling ratio between the fixed-point and floating-point numbers in the dequantization calculation. Optionally, the scaling factor S can be calculated using the following formula:
[0072]
[0073] in, The maximum value among the floating-point numbers obtained from dequantization calculations. This is the minimum value among the floating-point numbers obtained from dequantization. The maximum value among the fixed-point numbers before dequantization. S represents the minimum value of the fixed-point number before dequantization. In other words, S indicates the ratio between the range of floating-point numbers and the range of fixed-point numbers before and after quantization. Illustratively, in the quantization process of converting floating-point numbers to 8-bit fixed-point numbers, the scaling factor can be expressed as: Z represents the quantization zero, which characterizes the computational error introduced during the quantization process. Optionally, the quantization zero Z can be calculated using the following formula:
[0074] It should be noted that the formulas for calculating S or Z above are for illustrative purposes only, and this application does not impose any limitations on them.
[0075] The above linear inverse quantization formula contains the following calculation logic: First, perform... The subtraction calculation is performed; then the result of the subtraction calculation is multiplied by S; finally, the overall result is rounded to the nearest integer. Optionally, if the processor does not support floating-point calculation, the initial data may be amplified to achieve full quantization inference. The fixed-point number conversion operation involves multiples, and therefore, when performing the above-mentioned inverse quantization calculation, a right shift calculation is needed to restore the data.
[0076] Correspondingly, such as Figure 4As shown, the quantization calculation module includes, in addition to the input module 401 and the output module 408, the following calculation modules in sequence: subtraction calculation module 402, multiplication calculation module 403, right shift calculation module 404, and rounding calculation module 405. The quantization calculation module utilizes the subtraction calculation module 402 to perform... The operation involves ZP1 indicating the quantization zero Z in the aforementioned linear inverse quantization formula (Formula 4), which is then used by the multiplication calculation module 403. The calculation is performed, and the data is further restored using the right shift calculation module 404, and the rounding calculation module 404 is used for further calculation. Calculation. It should be noted that, due to the inherent data calculation methods of the hardware, ZP1 must be an integer in the above calculation module.
[0077] The quantization module also undertakes the task of performing quantization calculations. During the data quantization process, linear quantization calculations can be expressed by the following formula:
[0078]
[0079] In this formula, which is the same as the linear dequantization formula above, Q is the fixed-point data obtained after quantization, R is the floating-point data before quantization, S is the scaling factor, and Z is the quantization zero point.
[0080] The quantification formula contains the following calculation logic: First, perform... The division operation; and then, based on the fact that the result of the division operation may not be an integer, the following... Perform rounding; then add the rounded result to the quantization zero point Z. Correspondingly, such as... Figure 4 As shown, the quantization calculation module includes the following calculation modules in sequence: right shift calculation module 404, rounding calculation module 405, and addition calculation module 406. The quantization calculation module utilizes the right shift calculation module 404 to perform... The calculation is then performed by the rounding calculation module 405. The calculations, and the addition calculation module 406, are performed. The calculation process involves pruning the quantization results to prevent numerical overflow. Therefore, the quantization module includes a pruning module 407. It should also be noted that, due to the inherent data calculation methods of the hardware, ZP2 must be an integer in the aforementioned calculation modules.
[0081] Optionally, the above-mentioned quantization calculation modules correspond to the hardware in the NPU, wherein the subtraction calculation module 402 can be a subtractor, the multiplication calculation module 403 can be a multiplier, the right shift calculation module 404 can be implemented using a shifter, and the addition calculation module 406 can be an adder.
[0082] In order to reuse the above quantization algorithm module for LUT-based activation function calculation, the activation calculation formula needs to be mathematically transformed so that the calculation logic of the activation calculation formula is consistent with the inherent logic of the quantization calculation module.
[0083] Based on the quantization calculation module, the first calculation module is the addition calculation module, and the first step in the corresponding activation calculation should also be an addition operation, based on formula 6. You can raise this issue. As a common factor, Formula 6 can be transformed into:
[0084]
[0085] After transformation, the addition operation in Formula 7 above... The calculation can be performed using the addition module; correspondingly, This corresponds to ZP1. Furthermore, the multiplication operation in Formula 7 can be further calculated using the multiplication module within the quantization calculation module. Wherein, because... This represents the slope parameter, used for fitting calculations where the activation function curve is relatively smooth. The value of could be infinitesimally small, therefore The value of can be infinite. To avoid numerical overflow, when The value is less than When, set The value is equal to .in, It represents the difference between the maximum and minimum values that can be expressed in a fixed-point number.
[0086] It should be noted that, due to the inherent data calculation methods of the hardware, ZP1 must be an integer. Therefore, when storing this parameter, it needs to be rounded down to ensure that it is an integer. Furthermore, considering the actual data storage format, the above formula 7 can be expressed as:
[0087]
[0088] Based on Formula 8 above, To activate the parameters required in the calculation, the activation calculation is performed using a LUT table. It should be the first transformation parameter stored in the second-level lookup table, which is derived from the slope parameter in the fitting parameters. and intercept parameters The result is obtained through calculation and conversion.
[0089] It should also be noted that in Formula 8 above... Similarly, for the parameters required in the activation calculation, since the LUT table can only store integers, when storing this slope parameter in the secondary lookup table, it is necessary to store the amplified result obtained after amplifying the slope parameter, that is, the second transformation parameter Mul obtained after calculation and transformation. Accordingly, combined with the actual stored data, the above formula 8 can be expressed as:
[0090]
[0091] Wherein, the second transformation parameter is 2 of the slope parameter. N The data is restored by shifting it to the right, that is, In the right shift calculation, the result of the right shift is the result obtained by shifting to the right by N positions, where N is a positive integer.
[0092] Based on the formula transformation, when the above activation calculation is completed through the LUT table, the secondary lookup table contains the correspondence between the secondary interval, the first transformation parameter and the second transformation parameter. The first transformation parameter is the rounded result of the intercept-slope ratio, and the second transformation parameter is the amplified result of the slope parameter.
[0093] Formula 9 is an activation calculation formula derived from the fitted linear expression (Formula 6) and conforms to the inherent calculation logic of the quantization calculation module. When performing activation calculations based on the LUT table, the second-level lookup table stores the first transformation parameter and the second transformation parameter. For the method of activation calculation based on two parameters in computer equipment, please refer to [link to relevant documentation]. Figure 5 The diagram illustrates a flowchart of an activation calculation provided in an exemplary embodiment of this application. The activation calculation may include the following steps:
[0094] Step 501: The sum of the input value and the first target transformation parameter is determined as the first intermediate value.
[0095] Here, the first target transformation parameter is the first transformation parameter stored in the second-level target interval, which can be expressed as: The first intermediate value can be obtained as: That is, as Figure 6 As shown, when the first target conversion parameter is read by looking up the second-level lookup table, corresponding to the quantization calculation module 610, the computer device first uses the subtraction calculation module 611 to determine the first intermediate value calculation 621 based on the activation calculation formula (formula 9) 620.
[0096] Step 502: Determine the second intermediate value as the product of the second target transformation parameter and the first intermediate value.
[0097] The second transformation target parameter is the second transformation parameter stored in the target secondary interval. Based on the second transformation parameter, which can be represented as Mul, the second intermediate value can be obtained. That is, as Figure 6 As shown, when the second target conversion parameter is read by looking up the second lookup table, corresponding to the quantization calculation module 610, after the computer device completes the addition operation, it uses the multiplication calculation module 612 to determine the second intermediate value calculation 622 based on the activation calculation formula (formula 9) 620.
[0098] Step 503: Determine the right shift result of the second intermediate value as the activation value, wherein the second transformation parameter is 2 of the slope parameter. N The result of shifting the bits to the right by N bits is the result of shifting the bits to the right, where N is a positive integer.
[0099] Based on the target second transformation parameter being the slope parameter, 2 N When activating the calculation, a corresponding data restoration process is required, namely, a right shift calculation. Correspondingly, such as... Figure 6 As shown, the right shift calculation 623 described above can be completed using the right shift calculation module 613 in the quantization calculation module 610.
[0100] In Formula 9 above, to ensure that ZP1 is an integer, the second-level lookup table stores the first conversion parameter obtained by rounding. Considering that the rounding calculation will introduce a certain error during the operation, the second-level lookup table can contain the correspondence between the second-level interval, the first conversion parameter, the second conversion parameter and the third conversion parameter, provided that the storage space allows. The third conversion parameter is the rounded result of the product of the slope parameter and the error parameter, and the error parameter is the difference between the intercept slope ratio and the first conversion parameter.
[0101] The third transformation parameter indicates the error introduced by rounding the ratio of the intercept parameter and the slope parameter. This error can be expressed as... To ensure that the computational logic matches the inherent computational logic of the quantization module while restoring the error, an addition operation can be introduced based on Formula 9, utilizing the addition module within the quantization module to complete the error restoration. That is, based on Formula 9, it is necessary to... Perform addition operations.
[0102] It should be noted that the addition calculation module in the quantization calculation module is used to perform the calculation. When performing addition, The corresponding value is ZP2. Due to the inherent limitations of the hardware's calculation method, ZP2 must be an integer, and therefore the third conversion parameter needs to be obtained through rounding. In other words, the activation calculation method can be expressed by the following formula:
[0103]
[0104] Accordingly, the activation calculation based on Formula 10 may include the following steps:
[0105] 1. The sum of the input value and the first target transformation parameter is determined as the first intermediate value.
[0106] This step is the same as step 501, and will not be repeated here.
[0107] 2. The product of the second target transformation parameter and the first intermediate value is determined as the second intermediate value.
[0108] This step is the same as step 502, and will not be repeated here.
[0109] 3. The right shift result of the second intermediate value is determined as the third intermediate value, where the second transformation parameter is 2 times the slope parameter. N The result of shifting the bits to the right by N bits is the result of shifting the bits to the right, where N is a positive integer.
[0110] The third intermediate value can be represented as: That is, after reading the amplification factor N through the second-level lookup table, corresponding to the quantization calculation module, the computer device performs a right shift calculation based on the activation calculation formula (Formula 10) after completing the multiplication operation using the right shift calculation module.
[0111] It should be noted that in the same secondary lookup table, each secondary transformation parameter is obtained by amplification using the same amplification factor, that is, the value of N is unique. Therefore, in one possible implementation, the secondary lookup table stores only one amplification factor N.
[0112] 4. The sum of the third intermediate value and the third target transformation parameter is determined as the activation value.
[0113] The third target transformation parameter is the third transformation parameter stored in the second-level target interval, and the third transformation parameter can be expressed as: .like Figure 6 As shown, when the third target conversion parameter is read by looking up the second-level lookup table, corresponding to the quantization calculation module 610, after the computer device completes the right shift calculation, it uses the addition calculation module 614 to perform the above addition calculation 631 based on the activation calculation formula (formula 10) 630.
[0114] In summary, when storage space allows, introducing a third transformation parameter into the activation calculation can restore the errors caused by formula transformation, thereby improving the accuracy of the activation calculation.
[0115] In the activation calculation based on the lookup table, when an input value is obtained, the first-level interval in the first-level lookup table corresponding to that input value is first determined. Each entry in the first-level lookup table corresponds to one first-level interval. In one possible implementation, the first-level intervals are obtained by uniformly dividing the input value range based on the first-level partitioning granularity; that is, the number of first-level intervals can be calculated using the following formula:
[0116]
[0117] in, This represents the number of first-level intervals in the first-level lookup table; The range of input values, that is, the range of values for the quantized fixed-point numbers, can be represented as: ; This represents the first-level granularity. Since determining the first-level interval corresponding to an input value in the first-level lookup table does not require high precision, in one possible implementation, the range of input values within the first-level interval can be represented by 8-bit data, meaning there exists a maximum input value range. This indicates that a valid input value can be any value between 0 and 255. Illustratively, with an input value range of 0 to 255 and a first-level partition granularity of 8, the first-level lookup table includes 32 first-level intervals.
[0118] Furthermore, based on the input value, the minimum value within the range of input values, and the first-level partition granularity, the target first-level index corresponding to the input value is determined. In one possible implementation, the target first-level index can be calculated using the following formula:
[0119]
[0120] in, For input values, The minimum value in the input range. This represents the floor function.
[0121] Indicative, such as Figure 7 As shown, with an input value of 123, a minimum value of 0 in the input value range, and a first-level partitioning granularity of 8, the target first-level index can be obtained from Formula 11 as 15.
[0122] Furthermore, based on the one-to-one correspondence between the target first-level index and each first-level interval, once the target first-level index is calculated, the first-level interval corresponding to the target first-level index in the first-level lookup table is determined as the target first-level interval.
[0123] Since LUT tables are essentially data stored in memory, the process of reading the corresponding entry content through the target first-level index is equivalent to reading the data information stored at the memory address. In this embodiment, the target first-level index is used as an address, and the interval parameters stored at that address are the target interval parameters. In one possible implementation, the target interval parameters include the starting second-level index of the second-level interval within the target first-level interval, the number of target second-level intervals within the target first-level interval, and the starting value of the interval within the target first-level interval. The second-level intervals within each first-level interval are uniformly divided; that is, the second-level intervals are obtained by uniformly dividing the curves within each first-level interval.
[0124] In one possible implementation, the target secondary interval can be determined based on the target interval parameters. For details on how to determine the target secondary interval, please refer to [reference needed]. Figure 8 The diagram illustrates a flowchart of determining a target secondary interval provided by an exemplary embodiment of this application.
[0125] Step 801: Based on the first-level partitioning granularity and the number of target second-level intervals, determine the second-level partitioning granularity of the second-level intervals in the target first-level intervals.
[0126] In this embodiment, to save storage space while ensuring the accuracy of activation calculation, a suitable granularity needs to be selected for the secondary intervals. Correspondingly, for curve intervals with significant changes in the slope of the activation function curve, a smaller granularity is used to determine the fitting parameters. That is, intervals with larger slope changes than the primary intervals should contain more secondary intervals so that the curve segments in each secondary interval are as close to a straight line as possible. In the secondary lookup table, the numerical ranges contained in each secondary interval are non-uniform, but it should be noted that the secondary intervals within each primary interval are uniformly divided. Since the granularity of the secondary intervals varies depending on the degree of change in the curve slope within the primary interval, it is necessary to first determine the granularity of the secondary intervals when determining the target secondary intervals based on the primary intervals.
[0127] Each entry in the first-level lookup table stores at least a first-level index and the number of corresponding second-level intervals within the first-level interval. The calculation method for determining the second-level partition granularity based on the first-level partition granularity and the number of second-level intervals can be expressed by the following formula:
[0128]
[0129] in, For secondary granularity, Let P be the number of second-level intervals, representing the first-level partitioning granularity. It should be noted that the first-level intervals are obtained by uniformly dividing the activation curve. For the same first-level lookup table, the first-level partitioning granularity... It is a fixed value.
[0130] Indicative, such as Figure 7 As shown, when the first-level partitioning granularity is 8, and P is read as 3 in the first-level lookup table, the second-level partitioning granularity within the target first-level interval can be calculated using formula 12 as 8 / 3.
[0131] In the above process, the granularity of the second-level division is determined based on the number of second-level intervals stored in the target first-level interval. To improve the accuracy of the activation values, it is necessary to improve the linear fitting effect, that is, to set a smaller granularity of the second-level division in the first-level interval where the curve slope changes greatly, and to increase the number of second-level intervals for linear fitting. In one possible implementation, the number of second-level intervals in each first-level interval is positively correlated with the rate of change of the second derivative of the activation function within the first-level interval.
[0132] Accordingly, when constructing the secondary lookup table, i points on the corresponding activation function curve are determined as reference points in each primary interval, and the second derivative of each reference point is calculated. Then, the i calculated values in each primary interval are summed. The sum of the second derivatives in each primary interval characterizes the change in the slope of the curve within the primary interval; that is, the slope of the activation curve changes faster in the primary interval where the sum of the second derivatives is larger. Based on the determined number of secondary intervals and according to the sum of the second derivatives in each primary interval, the number of secondary intervals in each primary interval is determined proportionally. This ensures that more secondary intervals are divided in primary intervals with larger curve slope changes, resulting in a smaller granularity of secondary division; conversely, fewer secondary intervals are divided in primary intervals with smaller curve slope changes, saving storage space in the secondary lookup table. Thus, while controlling the number of secondary intervals, a better fitting effect can be obtained.
[0133] It should be noted that, due to limitations in hardware calculation methods, the total number of secondary intervals must be a power of 2.
[0134] In a schematic representation, when constructing the secondary lookup table, the activation curve is divided into 32 primary intervals. Ten points are determined within each primary interval, and the second derivative of each point is calculated and summed. When the total number of secondary intervals is 128, the number of secondary intervals in each primary interval is determined proportionally based on the sum of the second derivatives in each primary interval. The specific method for determining the number of secondary intervals proportionally could be multiplying the total number of secondary intervals by the proportion and rounding, or sorting based on the sum of the second derivatives and determining the number of secondary intervals according to a predefined proportion, etc. These methods are for illustrative purposes only and are not limited in this application.
[0135] Step 802: Determine the index offset based on the input value, the starting value of the target first-level interval, and the second-level partitioning granularity.
[0136] Each entry in the first-level lookup table stores at least the first-level index, the starting value of the interval, the starting second-level index, and the number of second-level intervals. The starting value indicates the initial value of the input value range within the target first-level interval, and the second-level partitioning granularity indicates the granularity applicable when further dividing the target first-level interval into second-level intervals. The calculation process for determining the index offset based on the input value, the starting value of the interval, and the second-level partitioning granularity can be expressed by the following formula:
[0137]
[0138] in, Indicates the index offset. For input values, The starting value of the target first-level interval. The granularity of the second-level division within the target first-level interval. This indicates that the floor function operation is performed.
[0139] Indicative, such as Figure 7 As shown, with the input value 123, the target first-level index 15 is determined as the first-level interval, and then the value is read. Given a value of 120 and P = 3, and based on a first-level partitioning granularity of 8, the second-level partitioning granularity is calculated. The value is 8 / 3, which can be calculated using Formula 13, representing the index offset. 1
[0140] Step 803: Determine the target secondary index based on the index offset and the starting secondary index.
[0141] Here, the index offset represents the deviation between the starting secondary index stored in the target primary interval and the target secondary index corresponding to the input value. Furthermore, the process of calculating and determining the target secondary index can be expressed by the following formula:
[0142]
[0143] Where E is the target secondary index. The starting secondary index is read from the target primary range. This is the index offset, which is calculated in the same way as in step 402.
[0144] Indicative, such as Figure 7As shown, each first-level interval of the first-level lookup table stores the first-level index, the interval start value, the starting second-level index, and the number of second-level intervals. With an input value of 123, for a first-level lookup table with a first-level partition granularity of 8, the target first-level index is 15. Then, it is read from the corresponding target first-level interval. The target interval start value is 120, the starting second-level index is 60, and the number of second-level intervals is 3. The index offset is calculated to be 1. According to Formula 14, the target second-level index is 61.
[0145] Step 804: Determine the second-level interval corresponding to the target second-level index in the second-level lookup table as the target second-level interval.
[0146] Based on the one-to-one correspondence between secondary indexes and secondary intervals, and the fact that the secondary lookup table stores the mapping relationship between secondary indexes and secondary intervals, when the target secondary interval is determined based on the target interval parameter, the secondary interval corresponding to the target secondary index in the secondary lookup table can be identified as the target secondary interval. Since the secondary lookup table is essentially data stored in RAM (Random Access Memory), and the corresponding target secondary index is essentially a lookup address, the computer device can query the secondary lookup table to determine the target secondary interval indicated by the target secondary index, given the lookup address.
[0147] Step 805: Obtain the target transformation parameters corresponding to the target secondary interval from the secondary lookup table.
[0148] The second-level lookup table stores the transformation parameters in tabular form. The target second-level interval serves as the target interval in the second-level lookup table, storing the target transformation parameters used to determine the activation value. Furthermore, the computer device can read the target transformation parameters from the target second-level interval.
[0149] In summary, in this embodiment, the computer device, based on the input value, first determines the target primary interval in the primary lookup table by searching through a primary lookup table with the same granularity. Then, based on the target interval parameters stored in the primary lookup table, the target secondary interval is determined in a secondary lookup table with a different granularity, thus assisting the search process. Even when the secondary interval is non-uniform, the target secondary interval can still be quickly determined without additional hardware such as comparators. This saves storage space for the secondary intervals and reduces device power consumption, enabling the search and reading process for non-uniform lookup tables.
[0150] Please refer to Figure 9 The diagram illustrates a structural block diagram of an activation device for a neural network provided in an exemplary embodiment of this application. The device includes:
[0151] The system comprises a first lookup table unit 901, a second lookup table unit 902, and a calculation unit 903, with the calculation unit 903 connected to both the first lookup table unit 901 and the second lookup table unit 902. It should be noted that the first lookup table unit 901, the second lookup table unit 902, and the calculation unit 903 are all hardware units within the NPU.
[0152] The first lookup table unit 901 is used to store a first-level lookup table. The first-level lookup table contains the correspondence between the first-level interval and the interval parameter. The first-level interval is obtained by dividing the range of input values. The input value is a fixed-point number, and the range of input values is a range of fixed-point numbers.
[0153] The second lookup table unit 902 is used to store a secondary lookup table, which contains the correspondence between the secondary intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fit of the activation function corresponding to the secondary interval.
[0154] The calculation unit 903 is used to determine the target first-level interval to which the input value belongs from a first-level lookup table; based on the target interval parameter of the target first-level interval, determine the target transformation parameter of the target second-level interval to which the input value belongs from a second-level lookup table, wherein the target second-level interval belongs to the target first-level interval; obtain the target transformation parameter corresponding to the target second-level interval from the second-level lookup table; and, through a reused quantization algorithm module, determine the activation value corresponding to the input value based on the target transformation parameter and the input value, wherein the activation value is a fixed-point number. The quantization algorithm module is used to perform data quantization and dequantization processing.
[0155] Optionally, the second lookup table unit 902 is further configured to:
[0156] The fitting parameters are stored, including slope parameters and intercept parameters. The secondary lookup table contains the correspondence between the secondary interval, the first transformation parameter, and the second transformation parameter. The first transformation parameter is the rounded result of the intercept-slope ratio, and the second transformation parameter is the amplified result of the slope parameter.
[0157] Optionally, the second lookup table unit 902 is further configured to:
[0158] The target transformation parameters are stored, including a first target transformation parameter and a second target transformation parameter.
[0159] Optionally, the computing unit 903 is further configured to:
[0160] The sum of the input value and the first target transformation parameter is determined as the first intermediate value;
[0161] The product of the second target transformation parameter and the first intermediate value is determined as the second intermediate value;
[0162] The activation value is determined by shifting the second intermediate value to the right, wherein the second transformation parameter is 2 of the slope parameter. N The right shift result is obtained by shifting the bit to the right by N bits, where N is a positive integer.
[0163] Optionally, the second lookup table unit 902 is further configured to:
[0164] The target transformation parameters are stored. These target transformation parameters are obtained by transforming the fitting parameters, which include a slope parameter and an intercept parameter. The secondary lookup table contains the correspondence between the secondary interval, the first transformation parameter, the second transformation parameter, and the third transformation parameter. The first transformation parameter is the rounded result of the intercept-slope ratio. The second transformation parameter is the amplified result of the slope parameter. The third transformation parameter is the rounded result of the product of the slope parameter and the error parameter. The error parameter is the difference between the intercept-slope ratio and the first transformation parameter.
[0165] Optionally, the second lookup table unit 902 is further configured to:
[0166] The target transformation parameters are stored, including a first target transformation parameter, a second target transformation parameter, and a third target transformation parameter.
[0167] Optionally, the computing unit 903 is further configured to:
[0168] The sum of the input value and the first target transformation parameter is determined as the first intermediate value;
[0169] The product of the second target transformation parameter and the first intermediate value is determined as the second intermediate value;
[0170] The third intermediate value is determined by shifting the second intermediate value to the right, wherein the second transformation parameter is 2 of the slope parameter. N The right shift result is the result obtained by shifting to the right by N bits, where N is a positive integer;
[0171] The sum of the third intermediate value and the third target transformation parameter is determined as the activation value.
[0172] Optionally, the first lookup table unit 901 is further configured to:
[0173] The correspondence between the first-level interval and the interval parameters is stored. The first-level interval is obtained by uniformly dividing the range of input values based on the first-level division granularity.
[0174] Optionally, if the target first-level interval to which the input value belongs is determined from the first-level lookup table, the calculation unit 903 is further configured to:
[0175] Based on the input value, the minimum value of the input value range, and the first-level partitioning granularity, determine the target first-level index corresponding to the input value;
[0176] The first-level interval corresponding to the target first-level index in the first-level lookup table is determined as the target first-level interval.
[0177] Optionally, the first lookup table unit 901 is further configured to:
[0178] The system stores the correspondence between primary intervals and interval parameters. The secondary intervals within each primary interval are evenly divided. The target interval parameters include the starting secondary index of the secondary interval in the target primary interval, the number of target secondary intervals in the target primary interval, and the starting value of the target primary interval.
[0179] Optionally, when the target secondary interval to which the input value belongs is determined from the secondary lookup table based on the target interval parameters of the target primary interval, the calculation unit 903 is further configured to:
[0180] Based on the first-level partitioning granularity and the number of target second-level intervals, determine the second-level partitioning granularity of the second-level intervals within the target first-level intervals;
[0181] The index offset is determined based on the input value, the starting value of the target first-level interval, and the second-level partitioning granularity.
[0182] Based on the index offset and the starting secondary index, determine the target secondary index;
[0183] The second-level interval corresponding to the target second-level index in the second-level lookup table is determined as the target second-level interval;
[0184] Obtain the target transformation parameters corresponding to the target secondary interval from the secondary lookup table.
[0185] Optionally, the first lookup table unit 901 is further configured to:
[0186] The system stores the correspondence between first-level intervals and interval parameters. The interval parameters include the starting second-level index of the second-level interval in the target first-level interval, the number of target second-level intervals in the target first-level interval, and the interval starting value of the target first-level interval. The number of second-level intervals in each first-level interval is positively correlated with the rate of change of the second derivative of the activation function within the first-level interval.
[0187] Optionally, the second lookup table unit 902 is further configured to:
[0188] The target transformation parameters are stored. The target transformation parameters are obtained by transforming the fitting parameters, which are determined by the least squares method.
[0189] In summary, in this embodiment, based on the input value, the computer device determines the target primary index through the calculation unit, reads the target interval parameters from the first lookup table unit, and uses the calculation unit to determine the target secondary index based on the target interval parameters. Then, it reads the target transformation parameters corresponding to the target secondary interval from the second lookup table unit. The calculation unit performs activation calculations based on the target transformation parameters and the input value using the quantization calculation module to obtain the activation value. While reducing the storage space occupied by excessive activation calculations by searching the non-uniform secondary lookup table, the activation calculation formula is transformed and the transformation parameters that conform to the calculation logic of the quantization calculation module are stored, thereby reusing the quantization calculation module for activation calculations and reducing the optimization cost of the LUT table.
[0190] This application also provides an NPU. The NPU includes programmable logic circuitry and / or program instructions, which, when the NPU is running, are used to implement the neural network activation method as described in the above embodiments.
[0191] This application also provides a computer-readable storage medium storing at least one program, which is loaded and executed by an NPU to implement the neural network activation method described in any of the above embodiments.
[0192] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0193] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An activation method of a neural network, characterized by, The method includes: The target first-level interval to which the input value belongs is determined from the first-level lookup table. The first-level lookup table contains the correspondence between the first-level interval and the interval parameter. The first-level interval is obtained by dividing the range of the input value. The input value is a fixed-point number, and the range of the input value is a range of fixed-point numbers. Based on the target interval parameters of the target primary interval, the target transformation parameters of the target secondary interval to which the input value belongs are determined from the secondary lookup table. The target secondary interval belongs to the target primary interval, and each primary interval is divided into at least one secondary interval. The secondary lookup table contains the correspondence between the secondary intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fit of the activation function corresponding to the secondary interval. The fitting parameters include slope parameters and intercept parameters. The secondary lookup table contains the correspondence between the secondary intervals, the first transformation parameter, and the second transformation parameter. The first transformation parameter is the rounded result of the ratio of intercept to slope, and the second transformation parameter is the amplified result of the slope parameter. The target transformation parameters include the first target transformation parameter and the second target transformation parameter. By reusing the subtraction calculation module in the quantization algorithm module, the sum of the input value and the first target conversion parameter is determined as the first intermediate value; By reusing the multiplication calculation module in the quantization algorithm module, the product of the second target transformation parameter and the first intermediate value is determined as the second intermediate value; By using the right-shift calculation module in the reuse quantization algorithm module, the right-shift result of the second intermediate value is determined as the activation value, wherein the second target transformation parameter is 2 of the slope parameter. N The right shift result is obtained by shifting to the right by N bits, where N is a positive integer. The activation value is a fixed number. The quantization algorithm module is used to perform data quantization and dequantization processing. The quantization algorithm module corresponds to the hardware in the neural network processor (NPU), the subtraction calculation module is a subtractor, the multiplication calculation module is a multiplier, and the right shift calculation module is a shifter.
2. The method of claim 1, wherein, The fitting parameters include a slope parameter and an intercept parameter. The secondary lookup table contains the correspondence between the secondary interval, the first transformation parameter, the second transformation parameter, and the third transformation parameter. The first transformation parameter is the rounded result of the intercept-slope ratio, the second transformation parameter is the amplified result of the slope parameter, and the third transformation parameter is the rounded result of the product of the slope parameter and the error parameter. The error parameter is the difference between the intercept-slope ratio and the first transformation parameter.
3. The method of claim 2, wherein, The target transformation parameters include a first target transformation parameter, a second target transformation parameter, and a third target transformation parameter; The method further includes: By reusing the subtraction calculation module in the quantization algorithm module, the sum of the input value and the first target conversion parameter is determined as the first intermediate value; By reusing the multiplication calculation module in the quantization algorithm module, the product of the second target transformation parameter and the first intermediate value is determined as the second intermediate value; By using the right-shift calculation module in the reuse quantization algorithm module, the right-shift result of the second intermediate value is determined as the third intermediate value, wherein the second target transformation parameter is 2 of the slope parameter. N The right shift result is the result obtained by shifting to the right by N bits, where N is a positive integer; The sum of the third intermediate value and the third target transformation parameter is determined as the activation value by using the addition calculation module in the reuse quantization algorithm module.
4. The method of claim 1, wherein, The first-level interval is obtained by uniformly dividing the range of input values based on the first-level division granularity; Determining the target first-level interval to which the input value belongs from the first-level lookup table includes: Based on the input value, the minimum value of the input value range, and the first-level partitioning granularity, determine the target first-level index corresponding to the input value; The first-level interval corresponding to the target first-level index in the first-level lookup table is determined as the target first-level interval.
5. The method of claim 4, wherein, The secondary intervals within each primary interval are evenly divided, and the target interval parameter includes the starting secondary index of the secondary interval in the target primary interval, the number of target secondary intervals in the target primary interval, and the interval starting value of the target primary interval; The step of determining the target transformation parameter of the target secondary interval to which the input value belongs from the secondary lookup table based on the target interval parameter of the target primary interval includes: Based on the first-level partitioning granularity and the number of target second-level intervals, determine the second-level partitioning granularity of the second-level intervals within the target first-level intervals; The index offset is determined based on the input value, the starting value of the target first-level interval, and the second-level partitioning granularity. Based on the index offset and the starting secondary index, determine the target secondary index; The second-level interval corresponding to the target second-level index in the second-level lookup table is determined as the target second-level interval; Obtain the target transformation parameters corresponding to the target secondary interval from the secondary lookup table.
6. The method of claim 5, wherein, The number of secondary intervals in each of the first-level intervals is positively correlated with the rate of change of the second derivative of the activation function within the first-level interval.
7. The method of claim 1, wherein, The fitting parameters were determined using the least squares method.
8. An activation device for a neural network, characterized by The device includes: The system comprises a first lookup table unit, a second lookup table unit, and a calculation unit, wherein the calculation unit is connected to both the first lookup table unit and the second lookup table unit. The first lookup table unit is used to store a first-level lookup table. The first-level lookup table contains the correspondence between first-level intervals and interval parameters. The first-level intervals are obtained by dividing the range of input values. The input values are fixed-point numbers and the range of input values is a range of fixed-point numbers. Each first-level interval is divided into at least one second-level interval. The second lookup table unit is used to store a secondary lookup table. The secondary lookup table contains the correspondence between the secondary intervals and the transformation parameters. The transformation parameters are obtained by transforming the fitting parameters of the linear fitting of the activation function corresponding to the secondary interval. The fitting parameters include a slope parameter and an intercept parameter. The secondary lookup table contains the correspondence between the secondary intervals, the first transformation parameter, and the second transformation parameter. The first transformation parameter is the rounded result of the intercept-slope ratio, and the second transformation parameter is the amplified result of the slope parameter. The calculation unit is configured to: determine the target first-level interval to which the input value belongs from a first-level lookup table; determine the target transformation parameter of the target second-level interval to which the input value belongs from a second-level lookup table based on the target interval parameter of the target first-level interval, wherein the target second-level interval belongs to the target first-level interval, and the target transformation parameter includes a first target transformation parameter and a second target transformation parameter; determine the sum of the input value and the first target transformation parameter as a first intermediate value through the subtraction calculation module in the multiplexing quantization algorithm module; determine the product of the second target transformation parameter and the first intermediate value as a second intermediate value through the multiplication calculation module in the multiplexing quantization algorithm module; and determine the right shift result of the second intermediate value as an activation value through the right shift calculation module in the multiplexing quantization algorithm module, wherein the second target transformation parameter is 2 of the slope parameter. N The right shift result is obtained by shifting the data to the right by N bits, where N is a positive integer. The activation value is a fixed-point number. The quantization algorithm module is used for data quantization and dequantization processing. The quantization algorithm module corresponds to the hardware in the neural network processor (NPU). The subtraction calculation module is a subtractor, the multiplication calculation module is a multiplier, and the right shift calculation module is a shifter.
9. A neural network processing unit, NPU, characterized by The NPU includes programmable logic circuits and / or program instructions, which, when the NPU is running, are used to implement the activation method of the neural network as described in any one of claims 1 to 7.
10. A neural network processing unit, NPU, characterized by The NPU includes the activation device for the neural network as described in claim 8.
11. A computer device, comprising: The computer device includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the neural network activation method as described in any one of claims 1 to 7.
12. A computer storage medium, characterized in that The computer storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the activation method of the neural network according to any one of claims 1 to 7.
13. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer storage medium; the processor of the computer device reads the computer instructions from the computer storage medium, and the processor executes the computer instructions, so that the computer device executes the activation method of the neural network according to any one of claims 1 to 7.