Reverse simulation method of neural network processor and related device
Through the reverse simulation method of the neural network processor, quantitative models and transformation rules are built, which solves the problem of low accuracy after the deployment of the neural network model, and achieves the consistency and accuracy of the output results.
Patent Information
- Application Number
- CN202411152985.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, after the neural network model is quantitatively trained and deployed to the neural network processor, the output results differ, resulting in a decrease in accuracy.
A reverse simulation method for neural network processors is proposed. By constructing the quantized model and simulated output data of hardware operators, quantized conversion rules are established, and the input and output of hardware operators are directly simulated to ensure that the model output is consistent with the processor environment, thus Improve quantitative effect.
The difference in model quantization effect is reduced, and the accuracy of the neural network model after it is deployed to the neural network processor is improved, ensuring consistency of the output results.
Smart Images
Figure CN119227758A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model quantization technology, and in particular to a reverse simulation method and related devices for a neural network processor. Background Art
[0002] Neural Processing Unit (NPU) usually refers to hardware specifically used to accelerate neural network calculations. These hardware can efficiently handle computationally intensive tasks in neural networks. In order to reduce the memory usage and computing requirements of neural network models when applied to neural network processors, it is usually necessary to quantize the neural network model and reduce the accuracy of the model parameters.
[0003] The existing method is usually to perform model quantization training on the neural network model, and then deploy the quantized neural network model to the neural network processor. However, due to the difference between the environment for model quantization training and the operating environment of the neural network processor, there are certain differences between the output results of the neural network model after model quantization training in the training environment and the output results after deployment to the neural network processor, which affects the accuracy of the neural network model after model quantization is deployed to the neural network processor. Summary of the invention
[0004] Based on the above needs, the present application proposes a reverse simulation method and related devices for a neural network processor, which can improve the model quantization effect of the neural network model, and thereby improve the accuracy of the neural network model after being deployed on the neural network processor.
[0005] To achieve the above objectives, this application proposes the following technical solutions:
[0006] According to a first aspect of an embodiment of the present application, a reverse simulation method of a neural network processor is provided, comprising:
[0007] Based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters, construct a quantization model corresponding to the hardware operator; wherein the hardware operator is a hardware operator of a neural network processor;
[0008] Determine the simulated output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model; wherein the simulated output data is simulated data of the calculation result of the hardware operator;
[0009] Based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator, construct a quantization conversion rule corresponding to the hardware operator;
[0010] Based on the quantization conversion rules corresponding to the hardware operators, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, and the quantized neural network model is deployed to the neural network processor.
[0011] Optionally, before constructing a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters, the method further includes:
[0012] The process of the neural network processor running the neural network model is analyzed to determine the hardware operator corresponding to the neural network processor.
[0013] Optionally, determining the simulation output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model includes:
[0014] Input data of the hardware operator is input into the quantization model so that the quantization model determines original calculation data corresponding to the input data, and quantizes the original calculation data based on the quantization coefficients in the quantization model to obtain simulation output data corresponding to the hardware operator.
[0015] Optionally, based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator, constructing a quantization conversion rule corresponding to the hardware operator includes:
[0016] Based on the correspondence between the input data of the hardware operator and the simulation output data, and / or the correspondence between the original calculation data corresponding to the input data of the hardware operator and the simulation output data, a quantization conversion rule corresponding to the hardware operator is constructed.
[0017] Optionally, the process of constructing the input data of the hardware operator includes:
[0018] constructing input data of the hardware operator based on a numerical range of a quantized data type corresponding to the input data of the hardware operator;
[0019] Based on the input data corresponding to the critical value of the numerical range of the quantized data type, the range of the original calculation data of the hardware operator is determined; the range of the original calculation data of the hardware operator is used to verify whether the original calculation data corresponding to the input data determined by the quantized model is correct.
[0020] Optionally, based on the quantization conversion rule corresponding to the hardware operator, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, including:
[0021] Based on the quantization conversion rule corresponding to the hardware operator, query the target simulation output data corresponding to the real input data of the hardware operator;
[0022] The target simulated output data corresponding to the real input data is used as the output data of the forward propagation when the neural network model is quantized and trained, and / or the quantized neural network model is tested and verified using the target simulated output data corresponding to the real input data.
[0023] According to a second aspect of an embodiment of the present application, a reverse simulation device for a neural network processor is provided, comprising:
[0024] A model building module, used to build a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters; wherein the hardware operator is a hardware operator of a neural network processor;
[0025] A simulation data determination module, used to determine the simulation output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model; wherein the simulation output data is the simulation data of the calculation result of the hardware operator;
[0026] A rule construction module, used to construct a quantization conversion rule corresponding to the hardware operator based on the corresponding relationship between the input data of the hardware operator and the simulated output data of the hardware operator;
[0027] A model processing module is used to perform quantization training on the neural network model and / or test and verify the quantized neural network model based on the quantization conversion rules corresponding to the hardware operator, and deploy the quantized neural network model to the neural network processor.
[0028] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;
[0029] The memory is connected to the processor and is used to store programs;
[0030] The processor is used to implement the reverse simulation method of the above-mentioned neural network processor by running the program in the memory.
[0031] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the reverse simulation method of the above-mentioned neural network processor is implemented.
[0032] According to a fifth aspect of an embodiment of the present application, there is provided a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the above-mentioned reverse simulation method of the neural network processor.
[0033] The reverse simulation method of the neural network processor proposed in the present application builds a quantization model corresponding to the hardware operator based on the operator parameters in the hardware operator and the quantization coefficients corresponding to the operator parameters; wherein the hardware operator is the hardware operator of the neural network processor; based on the input data and quantization model of the hardware operator, the simulation output data corresponding to the hardware operator is determined; wherein the simulation output data is the simulation data of the calculation result of the hardware operator; based on the correspondence between the input data of the hardware operator and the simulation output data of the hardware operator, the quantization conversion rule corresponding to the hardware operator is built; based on the quantization conversion rule corresponding to the hardware operator, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, and the quantized neural network model is deployed to the neural network processor. By adopting the technical solution of the present application, the calculation result of the hardware operator of the neural network processor can be simulated, and the simulation output result can be directly used as the output result of the neural network model, so as to ensure that the output result of the neural network model is consistent with the output result in the operating environment of the neural network processor, thereby reducing the effect difference of the model quantization of the neural network model and improving the accuracy of the neural network model after being deployed to the neural network processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of a flow chart of a reverse simulation method of a neural network processor provided in an embodiment of the present application;
[0036] Figure 2 A schematic flow chart of another reverse simulation method of a neural network processor provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of a processing flow for constructing input data of a hardware operator provided in an embodiment of the present application;
[0038] Figure 4 A schematic diagram of the structure of a reverse simulation device for a neural network processor provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The technical solution of the embodiment of the present application is applicable to the application scenario of the neural network model, and is specifically used in the application scenario of the model quantization of the neural network model. The technical solution of the embodiment of the present application can reduce the effect difference of the model quantization of the neural network model and improve the accuracy of the neural network model after being deployed on the neural network processor.
[0041] With the rapid development of neural network technology, AI algorithms have an increasing demand for computing power, so a number of customized chips for neural networks have emerged, generally referred to as neural network processors (NPUs). NPUs have the advantages of high computing power and low power consumption, and have gradually become the main carrier for the deployment of AI algorithms.
[0042] NPU generally customizes basic computing units at the hardware level to support neural network operators, such as MAC arrays (a computing array composed of multipliers and adders), convolution units, etc. At the same time, in order to achieve higher computing power within limited hardware resources, NPU can be used to deploy neural network models and reduce the memory usage and computing requirements when the neural network model is deployed on the neural network processor and runs. It is usually necessary to quantize the neural network model and reduce the accuracy of the model parameters.
[0043] The existing method is usually to perform model quantization training on the neural network model, and then deploy the quantized neural network model to the neural network processor. However, due to the difference between the environment for model quantization training and the operating environment of the neural network processor, there are certain differences between the output results of the neural network model after model quantization training in the training environment and the output results after deployment to the neural network processor, which affects the accuracy of the neural network model after model quantization is deployed to the neural network processor.
[0044] Based on this, the present application proposes a reverse simulation method for a neural network processor. This technical solution can ensure that the output results of the neural network model are consistent with the output results in the operating environment of the neural network processor, reduce the difference in the effect of model quantization of the neural network model, and thus solve the problem of low accuracy of the prior art neural network model after being deployed on the neural network processor.
[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0046] Exemplary Methods
[0047] See also Figure 1 As shown, the embodiment of the present application proposes a reverse simulation method of a neural network processor. The method includes:
[0048] S101 . Construct a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters.
[0049] In this embodiment, the operator used by the neural network processor when running the neural network model deployed thereon is used as the hardware operator corresponding to the neural network processor. The number of hardware operators corresponding to the neural network processor is at least one, for example, the hardware operators corresponding to the neural network processor include: at least one of a matrix multiplication operator (such as a Linear operator), a convolution operator, a BatchNorm operator, a single-input operator, and a dual-input operator.
[0050] The operator parameters in the hardware operator usually include weights and activation values, where the activation value may include input data and output data, for example, the input data is the output data activation value of the previous hardware operator of the current hardware operator, and the output data is the activation value output by the activation function in the current hardware operator. In the process of quantizing the model, the operator parameters are quantized, that is, the weights and activation values are quantized. Generally, the goal of model quantization is to convert the weights and activation values of the model from high-precision representations (such as 32-bit floating point numbers) to lower-precision representations (such as 8-bit integers) to reduce storage requirements and computational complexity, that is, quantization from float32 to int8 or uint8. Among them, the range of int8 is -128 to 127, and the range of uint8 is 0 to 255. Through the range of the data types of the weights and activation values of the operators in the neural network model and the range of the quantized data types, the quantization coefficients corresponding to the weights and activation values can be determined, that is, the operator parameters in each hardware operator corresponding to the neural network processor and the quantization coefficients corresponding to each operator parameter can be determined in advance.
[0051] There are corresponding calculation rules for various hardware operators. The model corresponding to the hardware operator can be constructed according to the operator parameters in the hardware operator and the corresponding calculation rules. In order to construct the quantization model corresponding to the hardware operator, in the process of model construction, it is not only necessary to set the quantization coefficients corresponding to each operator parameter in the hardware operator according to the operator parameters in the hardware operator and the corresponding calculation rules, so as to construct a quantization model that can output the output data of the hardware operator after quantization. Among them, each hardware operator is corresponding to a quantization model. Specifically, the calculation rules of the constructed quantization model need to quantize the parameters used in the calculation rules based on the quantization coefficients corresponding to the operator parameters, and update the quantized calculation rules. Moreover, after the quantization model determines the output data based on the quantized calculation rules, it is also necessary to quantize according to the quantization coefficients corresponding to the output data and then output the final quantized output data.
[0052] S102: Determine simulation output data corresponding to the hardware operator based on the input data and the quantization model of the hardware operator.
[0053] The quantization model of the hardware operator is constructed through the above steps. This embodiment also needs to construct the input data of the hardware operator. The constructed input data of the hardware operator needs to cover all numerical ranges of the input data, so that the input data of the hardware operator can include all input possibilities.
[0054] After constructing the input data and quantization model of the hardware operator, this embodiment uses the quantization model of the hardware operator to process and calculate the input data of the hardware operator according to the calculation rules after the hardware operator is quantized, and then quantizes the output data, thereby determining the simulated output data corresponding to the hardware operator. The simulated output data corresponding to the hardware operator is simulated data of the calculation result determined when the hardware operator is running in the hardware (i.e., the neural network processor).
[0055] Furthermore, based on the input data and the quantization model of the hardware operator, the simulation output data corresponding to the hardware operator is determined. The specific steps are as follows:
[0056] The input data of the hardware operator is input into the quantization model so that the quantization model determines the original calculation data corresponding to the input data, and quantizes the original calculation data based on the quantization coefficient in the quantization model to obtain the simulation output data corresponding to the hardware operator.
[0057] In this embodiment, the input data of the constructed hardware operator is input into the quantization model. The quantization model can process and calculate the input data according to the calculation rules after the hardware operator is quantized, and obtain the original calculation data corresponding to the input data. Since the input data covers all the numerical ranges of the input data, the corresponding original calculation data determined according to the input data also covers all the numerical ranges of the output data. In addition, since the quantization of the hardware operator requires quantization of both the weight and the activation value, the original calculation data corresponding to the input data is the activation value after the activation function is processed, then the original calculation data corresponding to the input data also needs to be quantized according to the quantization coefficient corresponding to the operator parameter in the quantization model, and the quantized data is used as the simulated output data finally output by the quantization model. Since the original calculation data corresponding to the input data covers all the numerical ranges of the output data, the simulated output data finally output by the quantization model covers all the numerical ranges of the output data after quantization.
[0058] S103 . Constructing a quantization conversion rule corresponding to the hardware operator based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator.
[0059] After the analog output data corresponding to the input data of the hardware operator is determined through the above steps, the corresponding relationship between each input data of the hardware operator and each analog output data of the hardware operator can be determined, and then based on the corresponding relationship between the input data of the hardware operator and the analog output data of the hardware operator, the quantization conversion rule corresponding to the hardware operator is constructed, so that all input data and all analog output data are stored in the quantization conversion rule, and the input data and the corresponding analog output data are associated. This embodiment can use the constructed quantization conversion rule corresponding to the hardware operator to query the analog output data corresponding to the input data of the hardware operator, and realize the reverse simulation of the hardware operator, that is, directly simulate the input and output of the hardware operator.
[0060] In this embodiment, the number of hardware operators corresponding to the neural network processor is at least one, and for each hardware operator, a corresponding quantization conversion rule is constructed, and finally, the quantization conversion rules corresponding to each hardware operator of the neural network processor are obtained.
[0061] Furthermore, based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator, a quantization conversion rule corresponding to the hardware operator is constructed. The specific steps are as follows:
[0062] Based on the correspondence between the input data of the hardware operator and the simulation output data, and / or the correspondence between the original calculation data corresponding to the input data of the hardware operator and the simulation output data, a quantization conversion rule corresponding to the hardware operator is constructed.
[0063] After the input data of the hardware operator is input into the quantization model corresponding to the hardware operator, the quantization model can calculate the original calculation data corresponding to the input data, and then determine the analog output data corresponding to the input data. Therefore, there is a corresponding relationship between the input data, the original calculation data and the analog output data, that is, the quantization conversion rule corresponding to the hardware operator can be constructed according to the corresponding relationship between the input data of the hardware operator and the analog output data of the hardware operator, so as to query the analog output data corresponding to the input data using the quantization conversion rule; the quantization conversion rule corresponding to the hardware operator can also be constructed according to the corresponding relationship between the original calculation data corresponding to the input data of the hardware operator and the analog output data, so as to query the analog output data corresponding to the original calculation data using the quantization conversion rule; the quantization conversion rule corresponding to the hardware operator can also be constructed according to the corresponding relationship between the input data of the hardware operator, the original calculation data corresponding to the input data of the hardware operator and the analog output data of the hardware operator, so as to query the original calculation data corresponding to the input data using the quantization conversion rule, and then query the corresponding analog output data.
[0064] S104: Based on the quantization conversion rules corresponding to the hardware operators, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, and the quantized neural network model is deployed to the neural network processor.
[0065] After constructing the quantization conversion rules corresponding to the hardware operators, the neural network model can be quantized and trained through the quantization conversion rules corresponding to the hardware operators, that is, the corresponding operators in the neural network model are quantized and trained. After the quantization training of the neural network model is completed, the quantized neural network model needs to be tested and verified to further improve the quantization accuracy of the neural network model. Among them, the test and verification of the quantized neural network model is also implemented based on the quantization conversion rules corresponding to the hardware operators. After the neural network model is quantized and tested and verified, the quantized neural network model is deployed to the neural network processor so that the neural network processor runs the quantized neural network model, thereby reducing the memory usage and computing requirements when the neural network model is deployed on the neural network processor and runs.
[0066] In this embodiment, the neural network model is quantized and trained based on the quantization conversion rules corresponding to the hardware operators, and each corresponding operator in the neural network model is trained separately based on the quantization conversion rules corresponding to each hardware operator. The quantized neural network model is tested and verified based on the quantization conversion rules corresponding to each hardware operator, and each corresponding operator in the quantized neural network model is tested and verified separately.
[0067] Specifically, based on the quantization conversion rules corresponding to the hardware operators, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified. The specific steps are as follows:
[0068] First, based on the quantization conversion rules corresponding to the hardware operators, query the target simulation output data corresponding to the real input data of the hardware operators.
[0069] According to the quantization conversion rules corresponding to each hardware operator constructed in this embodiment, the target simulation output data corresponding to the real input data input into each hardware operator can be queried. The operators when the neural network model is running in the training environment correspond to the hardware operators when the neural network model is running in the neural network processor environment. The real input data of the operator existing in the neural network model is the real input data of the hardware operator corresponding to the operator, and the target simulation output data of the operator existing in the neural network model is the target simulation output data of the hardware operator corresponding to the operator. When there are multiple operators in the neural network model, the target simulation output data corresponding to the real input data of the current operator is the real input data of the next operator.
[0070] For example, if there are three operators in the neural network model, namely, operator A, operator B, and operator C, operator A is the first operator in the neural network model, operator B is the second operator in the neural network model, and operator C is the third operator in the neural network model. The input data input into the neural network model is the real input data of operator A. The target simulation output data corresponding to the real input data of operator A is queried using the quantization conversion rule of the hardware operator corresponding to operator A, and the target simulation output data corresponding to the real input data of operator B is queried using the quantization conversion rule of the hardware operator corresponding to operator B, and the target simulation output data corresponding to the real input data of operator B is queried using the quantization conversion rule of the hardware operator corresponding to operator B, and the target simulation output data corresponding to the real input data of operator B is queried using the real input data of operator C, and the target simulation output data corresponding to the real input data of operator C is queried using the quantization conversion rule of the hardware operator corresponding to operator C.
[0071] Second, the target simulated output data corresponding to the real input data is used as the output data of the forward propagation when the neural network model is quantized and trained, and / or the quantized neural network model is tested and verified using the target simulated output data corresponding to the real input data.
[0072] In this embodiment, quantization-aware training (QAT) can be used for quantization training of the neural network model, that is, QAT quantization is performed on the neural network model. In the process of QAT quantization of the neural network model, the neural network model simulates quantization during forward propagation and uses full-precision representation during back propagation to ensure accurate calculation of the gradient, so that the neural network model can gradually adapt to the changes brought about by quantization during the training process, thereby maintaining high performance after quantization.
[0073] In the prior art, the real input data of the neural network model is usually directly input into the model, the neural network model simulates quantization during forward propagation, and outputs the output data of the forward propagation. Then, during the backward propagation, the neural network model adjusts and optimizes the model parameters of the neural network model based on the difference between the output data of the forward propagation and the output data represented by the full precision. However, in the prior art, there is a certain difference between the training environment of the neural network model during quantization training and the operating environment when the neural network model is deployed in the neural network processor and runs, which leads to a certain difference between the output data of the simulated quantization when the neural network model performs forward propagation in the training environment and the output data when the neural network model runs in the operating environment of the neural network processor. This further leads to a decrease in the quantization effect of the quantized neural network model obtained by quantization training in the training environment when it is deployed in the neural network processor and runs, thereby affecting the accuracy of the quantized neural network model after being deployed in the neural network processor.
[0074] Therefore, this embodiment does not input the real input data into the neural network model to simulate quantization during forward propagation of the neural network model, but directly queries the target simulated output data corresponding to the real input data from the constructed quantization conversion rules as the output data of the forward propagation when the neural network model is quantized. Since the target simulated output data corresponding to the real input data stored in the quantization conversion rules is the calculation result of the hardware operator when the directly simulated neural network model is deployed on the neural network processor, it can ensure the consistency between the output data of the forward transmission used in the quantization training of the neural network model and the output data of the neural network model when it is running in the operating environment of the neural network processor, thereby ensuring the quantization effect of the quantized neural network model obtained by the quantization training of the neural network model in the training environment when it is deployed in the neural network processor for operation, thereby improving the accuracy of the quantized neural network model after being deployed on the neural network processor.
[0075] In this embodiment, the target simulated output data corresponding to the real input data of each operator is used as the output data of the forward propagation when the operator in the neural network model is quantized and trained. Each operator of the neural network model is quantized and trained separately, and finally the neural network model after each operator is quantized and trained is used as the quantized neural network model.
[0076] After the quantization training of the neural network model is completed, the quantized neural network model is obtained, and the target simulated output data corresponding to the real input data can be used to test and verify the quantized neural network model, that is, the real input data is input into the quantized neural network model to obtain the real output data of the output, and the quantized neural network model is fine-tuned based on the difference between the real output data and the target simulated output data corresponding to the real input data. Specifically, when the neural network model has multiple operators, the real input data of the operator is input into the operator in the quantized neural network model to obtain the real output data of the operator output, and the operator in the quantized neural network model is fine-tuned based on the difference between the real output data and the target simulated output data corresponding to the real input data. In this way, each operator in the quantized neural network model is tested and verified, so as to obtain the final quantized neural network model.
[0077] From the above introduction, it can be seen that the reverse simulation method of the neural network processor proposed in the embodiment of the present application, based on the operator parameters in the hardware operator and the quantization coefficients corresponding to the operator parameters, constructs a quantization model corresponding to the hardware operator; wherein the hardware operator is the hardware operator of the neural network processor; based on the input data and quantization model of the hardware operator, determines the simulation output data corresponding to the hardware operator; wherein the simulation output data is the simulation data of the calculation result of the hardware operator; based on the correspondence between the input data of the hardware operator and the simulation output data of the hardware operator, constructs the quantization conversion rule corresponding to the hardware operator; based on the quantization conversion rule corresponding to the hardware operator, quantizes the neural network model and / or tests and verifies the quantized neural network model, and deploys the quantized neural network model to the neural network processor. By adopting the technical solution of this embodiment, the calculation result of the hardware operator of the neural network processor can be simulated, and the simulation output result can be directly used as the output result of the neural network model, so as to ensure that the output result of the neural network model is consistent with the output result in the operating environment of the neural network processor, thereby reducing the effect difference of model quantization of the neural network model and improving the accuracy of the neural network model after being deployed to the neural network processor.
[0078] As an optional implementation, see Figure 2 As shown, in another embodiment of the present application, a reverse simulation method of a neural network processor is disclosed, comprising the following steps:
[0079] S201. Analyze the process of the neural network processor running the neural network model, and determine the hardware operator corresponding to the neural network processor.
[0080] This embodiment requires analyzing the running behavior of the neural network processor when running the neural network model, that is, analyzing the running process, determining the operators used by the neural network processor when running the neural network model, and using the analyzed operators used by the neural network processor when running the neural network model as the hardware operators corresponding to the neural network processor.
[0081] S202: construct a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters.
[0082] S203 . Determine simulation output data corresponding to the hardware operator based on the input data and the quantization model of the hardware operator.
[0083] S204 : constructing a quantization conversion rule corresponding to the hardware operator based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator.
[0084] S205. Based on the quantization conversion rules corresponding to the hardware operators, perform quantization training on the neural network model and / or test and verify the quantized neural network model, and deploy the quantized neural network model to the neural network processor.
[0085] The execution method of steps S202 to S205 in this embodiment is the same as the execution method of steps S101 to S104 in the above embodiment, and will not be described in detail in this embodiment.
[0086] As an optional implementation, see Figure 3 As shown, in another embodiment of the present application, a process of constructing input data of a hardware operator is disclosed, comprising the following steps:
[0087] S301 . Construct input data of a hardware operator based on a numerical range of a quantized data type corresponding to the input data of the hardware operator.
[0088] Generally, the goal of model quantization is to convert the model's weights and activation values from high-precision representation (such as 32-bit floating point numbers) to lower-precision representation (such as 8-bit integers) to reduce storage requirements and computational complexity, that is, quantize from float32 to int8 or uint8. Among them, the input data of the hardware operator is the output data of the previous operator of the hardware operator, and the output data is the activation value after the activation function. Therefore, the input data of the hardware operator is also the activation value. In the process of quantizing the neural network model, the input data is also quantized, that is, from high-precision representation (such as 32-bit floating point numbers) to lower-precision representation (such as 8-bit integers). If the quantized data type corresponding to the input data of the hardware operator is int8, then the value range of the quantized data type is -128 to 127. If the quantized data type corresponding to the input data of the hardware operator is uint8, then the value range of the quantized data type is 0 to 255. The input data for constructing a hardware operator needs to be constructed based on the numerical range of the quantized data type corresponding to the input data of the hardware operator to ensure that the constructed input data of the hardware operator can cover all numerical ranges of the quantized data type corresponding to the input data of the hardware operator.
[0089] S302 : Determine the range of original calculation data of the hardware operator based on input data corresponding to a critical value of a numerical range of the quantized data type.
[0090] This embodiment first determines the input data corresponding to the critical value of the numerical range of the quantized data type. For example, if the quantized data type is int8, the numerical range of the quantized data type is -128 to 127, and the critical value of the numerical range of the quantized data type is -128 and 127; if the quantized data type is uint8, the numerical range of the quantized data type is 0 to 255, and the critical value of the numerical range of the quantized data type is 0 and 255. Then, based on the input data corresponding to the critical value of the numerical range of the quantized data type, the original calculation data calculated by the quantized calculation rule corresponding to the hardware operator is determined, that is, the input data corresponding to the critical value of the numerical range of the quantized data type is input into the quantization model corresponding to the hardware operator, and the quantization model calculates the original calculation data corresponding to the input data. Since the input data is determined based on the critical value of the numerical range of the quantized data type, the original calculation data corresponding to the input data is also the critical value of the original calculation data, so that the coverage range of the original calculation data of the hardware operator can be determined.
[0091] The coverage of the original calculation data of the hardware operator can verify whether the original calculation data corresponding to the input data determined by the quantization model is correct. That is, if the input data of the hardware operator is within the numerical range of the quantization data type corresponding to the input data of the hardware operator, then the original calculation data corresponding to the input data determined by the quantization model must also be within the coverage of the original calculation data of the hardware operator. Therefore, the coverage of the original calculation data of the hardware operator is used to verify whether the original calculation data corresponding to the input data determined by the quantization model is correct. If the original calculation data corresponding to the input data determined by the quantization model is within the coverage of the original calculation data of the hardware operator, it means that the original calculation data corresponding to the input data determined by the quantization model is correct. If the original calculation data corresponding to the input data determined by the quantization model is not within the coverage of the original calculation data of the hardware operator, it means that the original calculation data corresponding to the input data determined by the quantization model is incorrect. At this time, the constructed quantization model may be inaccurate and the quantization model needs to be adjusted.
[0092] Specifically, for the Linear operator in the matrix multiplication operator, the corresponding calculation rules are as follows:
[0093] Y=X*W T +b
[0094]
[0095] If the quantized data type of the input data is int8, the value range of the quantized data type corresponding to the input data of the hardware operator is determined to be -128 to 127.
[0096] The above X is the input data, the size is 128*128*256, the subscripts i and j start from 0, and the critical value -128 of the numerical range of the quantized data type is assigned to the input data, and the corresponding input data is:
[0097]
[0098] The above W is the weight, the size is 128*256, and the subscripts i and j start from 0. The weight is:
[0099]
[0100] When b=0, the range of the original calculation data Y of the hardware operator is 0 to 2,097,152.
[0101] Assign the critical value 127 of the numerical range of the quantized data type to the input data, and the corresponding input data is:
[0102]
[0103] At this time, the range of the original calculation data Y of the hardware operator is -2,080,768 to 0.
[0104] Combining the two ranges of the original calculation data Y of the hardware operator together, it is determined that the final coverage range of the original calculation data Y of the hardware operator is -2,080,768 to 2,097,152.
[0105] Furthermore, for the Linear operator, the rule for constructing the quantization model of the operator can be:
[0106]
[0107]
[0108]
[0109]
[0110]
[0111] Among them, scale w Indicates the quantization coefficient corresponding to the weight, scale x Represents the quantization coefficient of the input data, scale y represents the quantization coefficient of the output data, represents the weight before quantization, W ij represents the quantized weight, Represents the input data before quantization, x i represents the quantized input data, Represents the output data before quantization, y j Represents the quantized output data.
[0112] For the convolution operator, when the convolution kernel size of the convolution operator is 1, it is equivalent to a matrix multiplication operator. When the convolution kernel is greater than 1, the center point of the convolution kernel can be configured to be the same as the matrix multiplication operator, except for the center point of the convolution kernel, the rest are 0.
[0113] For the BatchNorm operator, the construction of input data and the construction of the quantization model are the same as those of the matrix multiplication operator.
[0114] For a single-input operator, the numerical range of the quantization data type corresponding to the input data is -128 to 127, there will be 256 outputs, and a quantization conversion rule of 256 length is directly constructed.
[0115] For a two-input operator, construct the input X (1) , X (2) , the size is 256*256,
[0116]
[0117]
[0118] Construct a 256*256 quantization conversion rule with a total of 64k to cover all outputs.
[0119] Exemplary Devices
[0120] Accordingly, the present application also provides a reverse simulation device for a neural network processor, see Figure 4 As shown, the device comprises:
[0121] The model building module 100 is used to build a quantization model corresponding to the hardware operator based on the operator parameters in the hardware operator and the quantization coefficients corresponding to the operator parameters; wherein the hardware operator is a hardware operator of a neural network processor;
[0122] The simulation data determination module 110 is used to determine the simulation output data corresponding to the hardware operator based on the input data and the quantization model of the hardware operator; wherein the simulation output data is the simulation data of the calculation result of the hardware operator;
[0123] A rule construction module 120, configured to construct a quantization conversion rule corresponding to the hardware operator based on a correspondence between the input data of the hardware operator and the simulated output data of the hardware operator;
[0124] The model processing module 130 is used to perform quantization training on the neural network model and / or test and verify the quantized neural network model based on the quantization conversion rules corresponding to the hardware operators, and deploy the quantized neural network model to the neural network processor.
[0125] From the above introduction, it can be seen that the reverse simulation device of the neural network processor proposed in the embodiment of the present application can simulate the calculation results of the hardware operators of the neural network processor, and directly use the simulation output results as the output results of the neural network model, thereby ensuring that the output results of the neural network model are consistent with the output results in the operating environment of the neural network processor, thereby reducing the difference in the effect of model quantization of the neural network model and improving the accuracy of the neural network model after being deployed to the neural network processor.
[0126] As an optional implementation, disclosed in another embodiment of the present application, the inverse simulation device of the neural network processor also includes: an operator analysis module.
[0127] The operator analysis module is used to analyze the process of the neural network processor running the neural network model and determine the hardware operator corresponding to the neural network processor.
[0128] As an optional implementation, another embodiment of the present application discloses that the simulation data determination module 110 is specifically used to:
[0129] The input data of the hardware operator is input into the quantization model so that the quantization model determines the original calculation data corresponding to the input data, and quantizes the original calculation data based on the quantization coefficient in the quantization model to obtain the simulation output data corresponding to the hardware operator.
[0130] As an optional implementation, another embodiment of the present application discloses that the rule construction module 120 is specifically used for:
[0131] Based on the correspondence between the input data of the hardware operator and the simulation output data, and / or the correspondence between the original calculation data corresponding to the input data of the hardware operator and the simulation output data, a quantization conversion rule corresponding to the hardware operator is constructed.
[0132] As an optional implementation, another embodiment of the present application discloses that the reverse simulation device of the neural network processor also includes: a data construction module and a range determination module.
[0133] A data construction module, used to construct input data of the hardware operator based on a numerical range of a quantized data type corresponding to the input data of the hardware operator;
[0134] The range determination module is used to determine the range of the original calculation data of the hardware operator based on the input data corresponding to the critical value of the numerical range of the quantized data type; the range of the original calculation data of the hardware operator is used to verify whether the original calculation data corresponding to the input data determined by the quantized model is correct.
[0135] As an optional implementation, another embodiment of the present application discloses that the model processing module 130 is specifically used for:
[0136] Based on the quantization conversion rules corresponding to the hardware operators, query the target simulation output data corresponding to the real input data of the hardware operators;
[0137] The target simulated output data corresponding to the real input data is used as the output data of the forward propagation when the neural network model is quantized and trained, and / or the quantized neural network model is tested and verified using the target simulated output data corresponding to the real input data.
[0138] The reverse simulation device of the neural network processor provided in this embodiment belongs to the same application concept as the reverse simulation method of the neural network processor provided in the above embodiments of this application, and can execute the reverse simulation method of the neural network processor provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the reverse simulation method of the neural network processor. For technical details not fully described in this embodiment, please refer to the specific processing content of the reverse simulation method of the neural network processor provided in the above embodiments of this application, and will not be repeated here.
[0139] Exemplary Electronic Devices
[0140] Another embodiment of the present application also provides an electronic device, see Figure 5 As shown, the device includes:
[0141] Memory 200 and processor 210;
[0142] The memory 200 is connected to the processor 210 and is used to store programs;
[0143] The processor 210 is used to implement the reverse simulation method of the neural network processor disclosed in any of the above embodiments by running the program stored in the memory 200.
[0144] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .
[0145] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.
[0146] A bus may include a pathway that transfers information between components of a computer system.
[0147] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0148] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.
[0149] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.
[0150] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.
[0151] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.
[0152] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.
[0153] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the reverse simulation method of any neural network processor provided in the above embodiments of the present application.
[0154] Exemplary computer program products and storage media
[0155] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the inverse simulation method of the neural network processor according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.
[0156] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0157] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the inverse simulation method of the neural network processor according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.
[0158] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0159] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0160] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.
[0161] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.
[0162] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0163] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0164] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.
[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0166] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0167] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0168] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A reverse simulation method for a neural network processor, characterized in that: include: Based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters, construct a quantization model corresponding to the hardware operator; wherein the hardware operator is a hardware operator of a neural network processor; Determine the simulated output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model; wherein the simulated output data is simulated data of the calculation result of the hardware operator; Based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator, construct a quantization conversion rule corresponding to the hardware operator; Based on the quantization conversion rules corresponding to the hardware operators, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, and the quantized neural network model is deployed to the neural network processor.
2. The method according to claim 1, characterized in that Before constructing a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters, the method further includes: The process of the neural network processor running the neural network model is analyzed to determine the hardware operator corresponding to the neural network processor.
3. The method according to claim 1, characterized in that Determining simulation output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model includes: Input data of the hardware operator is input into the quantization model so that the quantization model determines original calculation data corresponding to the input data, and quantizes the original calculation data based on the quantization coefficients in the quantization model to obtain simulation output data corresponding to the hardware operator.
4. The method according to claim 3, characterized in that Based on the correspondence between the input data of the hardware operator and the simulated output data of the hardware operator, a quantization conversion rule corresponding to the hardware operator is constructed, including: Based on the correspondence between the input data of the hardware operator and the simulation output data, and / or the correspondence between the original calculation data corresponding to the input data of the hardware operator and the simulation output data, a quantization conversion rule corresponding to the hardware operator is constructed.
5. The method according to claim 1, characterized in that The process of constructing the input data of the hardware operator includes: constructing input data of the hardware operator based on a numerical range of a quantized data type corresponding to the input data of the hardware operator; Based on the input data corresponding to the critical value of the numerical range of the quantized data type, the range of the original calculation data of the hardware operator is determined; the range of the original calculation data of the hardware operator is used to verify whether the original calculation data corresponding to the input data determined by the quantized model is correct.
6. The method according to claim 1, characterized in that Based on the quantization conversion rules corresponding to the hardware operators, the neural network model is quantized and trained and / or the quantized neural network model is tested and verified, including: Based on the quantization conversion rule corresponding to the hardware operator, query the target simulation output data corresponding to the real input data of the hardware operator; The target simulated output data corresponding to the real input data is used as the output data of the forward propagation when the neural network model is quantized and trained, and / or the quantized neural network model is tested and verified using the target simulated output data corresponding to the real input data.
7. A reverse simulation device for a neural network processor, characterized in that: include: A model building module, used to build a quantization model corresponding to the hardware operator based on operator parameters in the hardware operator and quantization coefficients corresponding to the operator parameters; wherein the hardware operator is a hardware operator of a neural network processor; A simulation data determination module, used to determine the simulation output data corresponding to the hardware operator based on the input data of the hardware operator and the quantization model; wherein the simulation output data is the simulation data of the calculation result of the hardware operator; A rule construction module, used to construct a quantization conversion rule corresponding to the hardware operator based on the corresponding relationship between the input data of the hardware operator and the simulated output data of the hardware operator; A model processing module is used to perform quantization training on the neural network model and / or test and verify the quantized neural network model based on the quantization conversion rules corresponding to the hardware operator, and deploy the quantized neural network model to the neural network processor.
8. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the reverse simulation method of the neural network processor as described in any one of claims 1 to 6 by running the program in the memory.
9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, it implements the reverse simulation method of the neural network processor as described in any one of claims 1 to 6.
10. A computer program product, characterized in that It comprises computer program instructions, which, when executed by a processor, enable the processor to implement the reverse simulation method of a neural network processor as claimed in any one of claims 1 to 6.