Pre-quantification model deployment method oriented to DL compiler

Abstract

The invention belongs to the field of deep learning compilers, and discloses a DL compiler-oriented pre-quantization model deployment method, which comprises the following steps of: 1, analyzing a front-end pre-quantization deep learning model by a front-end framework analyzer, and outputting a graph containing a Q operator; 2, performing Target-oriented graph segmentation operation, i.e., a target platform, and after the operation, segmenting the graph into two parts: operators supported by an accelerator are operated on the accelerator, and operators not supported by the accelerator are operated on a CPU (Central Processing Unit); and step 3, the DL compiler adopts different scheduling and code generation aiming at the two parts, and adopts two stages, namely a function-stage accelerator code generator and an operator-stage accelerator code generator, to generate an accelerator hardware instruction and a driving function. The method supports an accelerator-oriented graph cutting function, degradation from a Q operator to a common operator can be completed, and input and output formats of accelerator sub-functions can meet hardware requirements.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning compilers, and in particular to a pre-quantization model deployment method for DL ​​compilers. Background Art

[0002] The emergence of Deep Learning Compiler (DL Compiler) has redefined the challenges of deploying deep learning models on various hardware platforms. DL compilers usually convert models represented in framework-specific representations into a common intermediate representation (IR). Then, the DL compiler gradually reduces the graph from graph level to tensor level. In the graph-level IR, the compiler optimizes the computational graph of the model. In the tensor-level IR, it optimizes the loop structure of tensor operators, etc., and makes corresponding optimizations for different hardware. After continuous optimization, the DL compiler is able to handle the diversity of the front-end (framework) and back-end (hardware platform) more effectively, thereby simplifying the deployment of deep learning models. In general, deep learning models require a lot of computing and memory resources. Related research has implemented various technologies (algorithms, software, and hardware) to reduce the computing and memory burden of deep learning models, thereby simplifying their development and deployment process.

[0003] Among these technologies, quantization is a widely studied method. Quantization allows the model to consume less computing and memory resources while keeping its accuracy close to that of the unquantized model. The paper (Efficient Execution of Quantized Deep Learning Models: A Compiler Approach) proposes a QNN (Quantized Neural Network) as a graph-level IR dialect that can enhance deep learning compilers and support a variety of mainstream deep learning frameworks. This approach provides an end-to-end solution that can read pre-quantized models and support a variety of hardware backends.

[0004] However, this approach does not support deploying pre-quantized models on customized AI accelerators, such as the Versatile Tensor Accelerator (VTA). Summary of the invention

[0005] The purpose of the present invention is to provide a pre-quantization model deployment method for a DL compiler to solve the above-mentioned technical problems.

[0006] In order to solve the above technical problems, the specific technical solution of a pre-quantization model deployment method for a DL compiler of the present invention is as follows:

[0007] A pre-quantization model deployment method for a DL compiler includes the following steps:

[0008] Step 1: The front-end framework parser parses the front-end pre-quantized deep learning model and outputs a graph containing the Q operator. The Q operator is an operator that contains quantization information and is one level higher than the graph-level operator. It will be downgraded to a lower-level graph-level operator in the subsequent process.

[0009] Step 2: Perform a graph splitting operation for the target platform. After this operation, the graph will be divided into two parts: operators supported by the accelerator will run on the accelerator, and operators not supported by the accelerator will run on the CPU; Step 3: The DL compiler will use different scheduling and code generation for the two parts of step 2. For operators running on the CPU, the DL compiler will use the Host default process for processing and finally generate source code running on the CPU; while operators running on the accelerator will be optimized through accelerator-specific scheduling, and a two-level, function-level and operator-level accelerator code generator will be used to generate accelerator hardware instructions and driver functions.

[0010] Further, the step 1 includes the calculation of the quantized 2D convolution operator: a: first, the input data and weight data are moved from the system memory to the local buffer of the accelerator through the DMA Load instruction; the GEMM calculation is performed, and the accelerator accumulator Buffer is cleared before the calculation; the Bias data is moved from the system memory to the local buffer of the accelerator through the DMA Load instruction, and the vector ALU ADD operation of the GEMM result and the Bias is performed; the vector ALU RMUL, vector ALU RSHR operation and vector ALU ADD operation are performed, that is, the Requantize process is performed: multiply by multiplier, divide by 2^shift and add output zero point; perform vector ALU MIN / MAX operation to limit the result to the Int8 data range [-128,127]; finally, the result is stored in the system memory through the DMA Store instruction. The calculation process of b is basically the same as a, the only difference is that before b performs the GEMM operation, the Bias is first moved to the accelerator accumulator Buffer through the DMA Load instruction, and then the GEMM multiplication and accumulation calculation is performed on this basis.

[0011] Further, the step 2 includes the following steps:

[0012] First, through pattern matching, the operator a is grouped to obtain the grouped b, where each triangle in b represents the matching pattern, i.e., Q.conv2d+Q.Bias_add+Q.Requantize+Q.Relu; then, the Target tag is added to the matching pattern through the Annotation operation, indicating that this part of the operator will run on the Target; through the Partitioning operation, the Group with the Target tag is converted into a function module represented at the graph level, i.e., c, and an accelerator attribute is added to the function, which instructs the DL compiler to use an accelerator-specific process to process the function; then, the graph is traversed and modified, and the accelerator function name is changed to a special function name with Target; next, pre-optimization is performed to meet the requirements of GraphPacking; finally, Graph is executed. Packing converts the Q operator graph into a common graph-level operator graph and performs Layout transformation so that the input and output of the accelerator function module meet the accelerator hardware requirements, that is, d, d means that the Pack operation needs to be performed before entering the accelerator module; the Unpack operation needs to be performed before exiting the accelerator module and entering the Host module; if there is only an Elementwise operator between the two accelerator modules, no additional Pack and Unpack operations are required.

[0013] Furthermore, the step 3 adopts function-level and operator-level transformation methods to transform the accelerator sub-function and the accelerator operator, including the following steps:

[0014] First, we use an illegal accelerator graph, i.e., a graph whose input and output do not meet the accelerator hardware requirements, as input and perform function-level transformation. The transformation process is as follows:

[0015] Step 3.1: Traverse the graph and collect the location information of all sub-functions in the graph. The search results are recorded in a list. Each element of the list is a tuple. The tuple contains two parameters: the sub-function name and the location of the sub-function in the graph. The list is represented as follows:

[0016] List[Tuple(sub_func,index),…];

[0017] Step 3.2: Traverse the list obtained in step (1), collect the accelerator submodules and their location information in the graph according to the special function name of the accelerator function, and record them in the accelerator submodule list, which is represented as follows: List(Tuple(accelerator_func,index));

[0018] Step 3.3: Traverse the accelerator submodule list to determine whether the current accelerator function needs to perform Pack and Unpack operations;

[0019] Step 3.4: Traverse the graph and perform function-level transformation of the graph, that is, treat the sub-function as a normal operator. For each accelerator sub-module, if its pack flag in the pack list is True, perform the Pack operation on its input; if the unpack flag is True, perform the Unpack operation on its output. After a round of traversal, the input and output of all accelerator sub-modules are legal, that is, they meet the accelerator hardware requirements. The graph output in this step is the legal accelerator graph;

[0020] Then, taking the accelerator legal graph as input, the operator-level graph transformation is performed to downgrade the Q operator to facilitate subsequent processing, and the graph is traversed. If the current operator is an accelerator submodule, each operator in it is further traversed to perform operator-level transformation.

[0021] Furthermore, the step 3.3 determines whether the current accelerator function needs to perform Pack and Unpack operations. The determination rule is: if the current accelerator submodule is the first accelerator submodule, and there are non-element-wise operators in the previous operators, then the Pack operation needs to be performed; if the current accelerator submodule is the last accelerator submodule, and there are non-element-wise operators in the subsequent operators, then the Unpack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the previous accelerator submodule, then the Pack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the next accelerator submodule, then the Unpack operation needs to be performed. The traversal results are recorded in the pack list, and each element of the list is a three-tuple, namely, the accelerator submodule function name, whether to pack the flag, and whether to unpack the flag. The pack list is expressed as follows:

[0022] List(Tuple(accelerator_func,pack_flag,unpack_flag),…).

[0023] Furthermore, the specific processing steps of performing operator-level transformation in step 3.4 are as follows:

[0024] Step 3.4.1: Process the Q.conv2d operator: If the current operator is Q.conv2d, replace it with the Conv2d_zp operator, which uses the Input zero point as the Padding value and performs the convolution operation of the Padding Input and Weight. The Conv2d_zp operator is downgraded to the accelerator GEMM Loop, Load, and Store operations through the subsequent accelerator-specific optimization process;

[0025] Step 3.4.2: Process the Q.Bias_add operator: If the current operator is Q.Bias_add, replace it with the Bias_add_zp operator, which adds the zero-fused Bias constant to the output of Conv2d_zp. The Bias constant here is Qfused_bias = Qbias - ∑zpin*Qwgt. The Conv2d_zp operator is converted to accelerator vector ALU ADD and Load operations through subsequent accelerator-specific scheduling;

[0026] Step 3.4.3: Process the Q.Requantize operator: If the current operator is Q.Requantize, replace it with a combination of a quantized multiplication operator Round_multiply with Rounding, a shift division operator Round_rshift with Rounding, and an Add operator that adds an output zero point. The Round_multiply and Round_rshift operators are converted to accelerator vector ALU RMUL and RSHR operations through subsequent accelerator-specific scheduling. The Add operator is converted to accelerator vector ALU ADD operation through subsequent accelerator-specific scheduling.

[0027] Step 3.4.4: Process Q.Relu operator: If the current operator is Q.Relu, replace it with a combination of min and max operators, i.e. limit the result to the range [-128, 127]. The min and max operators are converted into accelerator vector ALU MIN and MAX operators through subsequent accelerator-specific scheduling.

[0028] Furthermore, the workflow of the accelerator two-level code generator in step 3 is as follows: first, the accelerator function-level code generator cooperates with the Host code generator to generate a C language version of the operator execution control flow and the accelerator driver code; wherein, the accelerator function-level code generator traverses each accelerator sub-function in the graph and generates an independent accelerator driver function for it, and the driver function completes two tasks: initializing the accelerator and starting the accelerator computing kernel, and the accelerator kernel executes the hardware instructions in the instruction array in sequence; then, the accelerator operator-level generator traverses each operator in the accelerator sub-function, and generates a Python version of the operator execution control flow according to the connection relationship and parameter information of the graph, and calls the accelerator Runtime through the Pybind11 module to generate accelerator hardware instruction data, and saves the data in a C language array; wherein, py2cpp is a Python package bound to the accelerator C or C++ Runtime API, generated by the Pybind11 module, which can realize the calling of C or C++ functions in Python.

[0029] A pre-quantization model deployment method for a DL compiler of the present invention has the following advantages: (1) The present invention supports a graph cutting function for an accelerator, which can not only complete the downgrading of the Q operator to a common operator, but also enable the input and output formats of the accelerator sub-function to meet the hardware requirements. (1) It supports two-level transformation of the graph, wherein the function-level transformation performs Pack and Unpack operations; the operator transformation performs operator downgrading. (3) The Q.Conv2d operator has a zero-point filling attribute during the downgrading process, and supports the ACC accumulator to be initialized as Bias scheduling. The Q.Requantize operator is further disassembled into a tensor-level operator combination (round_rmul and round_rshr) during the downgrading process to better support the fusion of the alu operator into the convolution operation function. (4) It supports a two-level code generator for the accelerator, wherein the function-level code generator generates an accelerator driver function, and the operator-level code generator generates an accelerator hardware instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall process structure of the pre-quantization model deployment for the DL compiler of the present invention;

[0031] Figure 2 It is a working schematic diagram of the Target-oriented graph segmentation module of the present invention;

[0032] Figure 3 This is a technical schematic diagram of the Graph Packing module of the present invention;

[0033] Figure 4 It is the calculation flow graph of the quantized 2D convolution operator of the present invention;

[0034] Figure 5 It is a working schematic diagram of the two-stage code generator of the accelerator of the present invention;

[0035] Figure 6 It is a schematic diagram of the binding relationship between the Pybind11 Python package py2cpp and the accelerator Runtime of the present invention. DETAILED DESCRIPTION

[0036] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a pre-quantization model deployment method for a DL compiler of the present invention in conjunction with the accompanying drawings.

[0037] The present invention is applicable to scenarios where the quantization method used by the pre-quantization model complies with the "Tensorflow lite int8 Quantization Specification".

[0038] The following formula 1 describes the calculation process of the 2D quantized convolution operator. A is the quantized input, QB is the quantized weight, Q C is the quantized output, scale A 、scale B 、scale C and zp A 、zp B 、zp C They are the quantization scale and quantization zero point for , n represents batch, ci represents input channel, co represents output channel, h represents input height, w represents input width, kh and kw represent convolution kernel height and width respectively.

[0039]

[0040]

[0041] According to the Tensorflow Lite int8 quantization specification, zp B is 0, the above formula evolves into the following formula 2. The calculation process can be described in several steps: the first step is to perform Q A and Q B The ordinary convolution operation, where Q A is the input after zero-point padding. This operation can be mapped to a graph-level convolution operator with zero-point padding. The second step is to subtract term 2, which can be mapped to a graph-level addition operator. The third step is to multiply by (scale A ·scale B ) / scale C , which can be mapped to a graph-level requantize operator; finally, add the output zero point zp C , which can be mapped to a graph-level addition operator. Item 2 is a constant, and the result can be calculated in advance and then fused into the bias constant, which can reduce ALU operations.

[0042]

[0043]

[0044] like Figure 1 As shown, a pre-quantization model deployment method for a DL compiler of the present invention comprises the following steps:

[0045] Step 1: The front-end framework parser parses the front-end pre-quantized deep learning model and outputs a graph containing the Q operator. The Q operator is an operator that contains quantization information and is one level higher than the graph-level operator. It will be downgraded to a lower-level graph-level operator in the subsequent process. Taking the 2D quantized convolution operator as an example, the 2D quantized convolution operator of the front-end model is converted into a combination of several Q operators and graph-level operators, namely Q.conv2d->Q.Bias_add->Q.Requantize->Q.Relu.

[0046] Figure 4 A computational flow chart of a quantized 2D convolution operator provided for an embodiment of the present invention. The computational process in (a) of the figure is: the input data and weight data are transferred from the system memory to the local buffer of the accelerator through the DMA Load instruction; the GEMM calculation is performed, and the accelerator accumulator buffer is cleared before the calculation; the Bias data is transferred from the system memory to the local buffer of the accelerator through the DMA Load instruction, and the vector ALU ADD operation of the GEMM result and the Bias is performed; the vector ALU RMUL, vector ALU RSHR operation and vector ALU ADD operation are performed, that is, the Requantize process is performed: multiply by the multiplier (Rounding Multiply), divide by 2^shift (Rounding Right Shift) and add the output zero point (Add output zero point); the vector ALU MIN / MAX operation is performed to limit the result to the Int8 data range [-128,127]; finally, the result is stored in the system memory through the DMA Store instruction. The calculation process of (b) is basically the same as that of (a). The only difference is that before executing the GEMM operation, (b) first moves the Bias to the accelerator accumulator Buffer through the DMA Load instruction, and then performs the GEMM multiplication and accumulation calculation on this basis. The advantage of doing so is that it reduces the number of hardware instructions and saves computing overhead, that is, the accumulator clearing process and the vector ALU ADD operation process are omitted.

[0047] Step 2: Perform a graph splitting operation for the target (i.e., target platform). After this operation, the graph will be divided into two parts: one part (i.e., operators supported by the accelerator) will run on the accelerator, and the other part (i.e., operators not supported by the accelerator) will run on the host (i.e., CPU) side.

[0048] Figure 2Schematic diagram of the Target-oriented graph partitioning module technology provided for an embodiment of the present invention. First, through pattern matching, the operators in (a) are grouped to obtain the grouped (b), and each triangle in (b) represents a matching pattern, namely Q.conv2d+Q.Bias_add+Q.Requantize+Q.Relu. Next, the Target tag is added to the matching pattern through the Annotation operation, indicating that this part of the operators will run on the Target (accelerator); through the Partitioning operation, the Group with the Target tag is converted into a function module represented at the graph level, namely (c), and an accelerator attribute is added to the function, which instructs the DL compiler to use an accelerator-specific process to process the function. Then, the graph is traversed and modified, and the accelerator function name is modified to a special function name with Target (accelerator name). Next, pre-optimization is performed, such as constant folding, to meet the requirements of Graph Packing. Finally, perform Graph Packing (two-level graph transformation) to convert the Q operator graph into a common graph-level operator graph, and perform Layout transformation (i.e., Pack and Unpack) to make the input and output of the accelerator function module meet the accelerator hardware requirements, i.e., (d). (d) indicates that before entering the accelerator module, a Pack operation needs to be performed; before exiting the accelerator module and entering the Host module, an Unpack operation needs to be performed; if there is only an Elementwise operator between the two accelerator modules, no additional Pack and Unpack operations are required.

[0049] Step 3: The DL compiler will use different scheduling and code generation for the two parts of step 2. For operators running on the CPU, the DL compiler will use the Host default process to process and finally generate source code running on the CPU. Operators running on the accelerator will be optimized through accelerator-specific scheduling, and a two-level (function-level and operator-level) accelerator code generator will be used to generate accelerator hardware instructions and driver functions.

[0050] Figure 3 The technical principle diagram of the Graph Packing module provided by the embodiment of the present invention. A two-level (function level and operator level) transformation method is used to transform the accelerator sub-functions and accelerator operators in the graph. First, the accelerator illegal graph (i.e., the graph whose input and output do not meet the accelerator hardware requirements) is used as input to perform function-level transformation. The transformation process is as follows:

[0051] (1) Traverse the graph and collect the location information of all sub-functions in the graph. The search results are recorded in a list. Each element of the list is a tuple. The tuple contains two parameters: the sub-function name and the location of the sub-function in the graph. The list can be expressed as follows:

[0052] List[Tuple(sub_func,index),…]

[0053] (2) Traverse the list obtained in step (1), collect the accelerator submodules and their location information in the graph according to the special function name of the accelerator function, and record them in the accelerator submodule list. The list can be expressed as follows:

[0054] List(Tuple(accelerator_func,index))

[0055] (3) Traverse the accelerator submodule list to determine whether the current accelerator function needs to perform Pack and Unpack operations. The determination rule is: if the current accelerator submodule is the first accelerator submodule and there are non-element-wise operators in the previous operators, a Pack operation needs to be performed; if the current accelerator submodule is the last accelerator submodule and there are non-element-wise operators in the subsequent operators, an Unpack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the previous accelerator submodule, a Pack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the next accelerator submodule, an Unpack operation needs to be performed. The traversal results are recorded in the pack list, and each element of the list is a three-tuple, namely (accelerator submodule function name, whether to pack flag, whether to unpack flag). The pack list can be expressed as follows:

[0056] List(Tuple(accelerator_func,pack_flag,unpack_flag),…)

[0057] (4) Traverse the graph and perform function-level transformation of the graph, that is, treat the sub-functions as ordinary operators. For each accelerator sub-module, if its pack flag in the pack list is True, perform the Pack operation on its input; if the unpack flag is True, perform the Unpack operation on its output. After one round of traversal, the input and output of all accelerator sub-modules are legal, that is, they meet the accelerator hardware requirements. The graph output in this step is the legal accelerator graph.

[0058] Then, the accelerator legal graph is used as input to perform operator-level graph transformation and downgrade the Q operator to facilitate subsequent processing. Traverse the graph. If the current operator is an accelerator submodule, further traverse each operator in it and perform operator-level transformation. The specific processing steps are as follows:

[0059] (1) Process the Q.conv2d operator. If the current operator is Q.conv2d, it is replaced by the Conv2d_zp operator, which uses the zero point of the Input as the Padding value and performs the convolution operation of the Padding Input and Weight. The Conv2d_zp operator is downgraded to the accelerator GEMM Loop, Load, and Store operations through the subsequent accelerator-specific optimization process.

[0060] (2) Process the Q.Bias_add operator. If the current operator is Q.Bias_add, replace it with the Bias_add_zp operator, which adds the Bias constant after zero-point fusion to the output result of Conv2d_zp. The Bias constant here is Q fused_bias =Q bias -∑zp in *Q wgt The Conv2d_zp operator is converted into accelerator vector ALU ADD and Load operations through subsequent accelerator-specific scheduling.

[0061] (3) Process the Q.Requantize operator. If the current operator is Q.Requantize, replace it with a combination of the quantized multiplication operator Round_multiply with Rounding, the shift division operator Round_rshift with Rounding, and the Add operator that adds output zero. The Round_multiply and Round_rshift operators are converted to accelerator vector ALU RMUL and RSHR operations through subsequent accelerator-specific scheduling, and the Add operator is converted to accelerator vector ALU ADD operation through subsequent accelerator-specific scheduling.

[0062] (4) Process the Q.Relu operator. If the current operator is Q.Relu, it is replaced by a combination of the min operator and the max operator, which limits the result to the range [-128, 127]. The min and max operators are converted into accelerator vector ALU MIN and MAX operators through subsequent accelerator-specific scheduling.

[0063] Figure 5A schematic diagram of the operation of the two-level code generator for the accelerator provided in the embodiment of the present invention. First, the accelerator function-level code generator cooperates with the Host code generator to generate the C language version of the operator execution control flow and the accelerator driver code; wherein, the accelerator function-level code generator traverses each accelerator sub-function in the graph, and generates an independent accelerator driver function for it, and the driver function completes two tasks: initializing the accelerator and starting the accelerator computing kernel, and the accelerator kernel executes the hardware instructions in the instruction array in sequence. Then, the accelerator operator-level generator traverses each operator in the accelerator sub-function, and generates a Python version of the operator execution control flow according to the connection relationship and parameter information of the graph, and calls the accelerator Runtime through the Pybind11 module to generate accelerator hardware instruction data, and saves the data in the C language array; wherein, py2cpp is a Python package bound to the accelerator C or C++ Runtime API, generated by the Pybind11 module, which can realize the call of C or C++ functions in Python, and its binding relationship is as follows Figure 6 shown.

[0064] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A pre-quantized model deployment method for DL ​​compilers, It is characterized in that The steps include: Step 1: The front-end framework parser parses the front-end pre-quantized deep learning model and outputs a graph containing the Q operator. The Q operator is an operator that contains quantization information and is one level higher than the graph-level operator. It will be downgraded to a lower-level graph-level operator in the subsequent process. Step 2: Perform a graph splitting operation for the target platform. After this operation, the graph will be divided into two parts: operators supported by the accelerator will run on the accelerator, and operators not supported by the accelerator will run on the CPU; Step 3: The DL compiler will use different scheduling and code generation for the two parts of step 2. For operators running on the CPU, the DL compiler will use the Host default process for processing and finally generate source code running on the CPU; while operators running on the accelerator will be optimized through accelerator-specific scheduling, and a two-level, function-level and operator-level accelerator code generator will be used to generate accelerator hardware instructions and driver functions.

2. The pre-quantization model deployment method for DL ​​compiler according to claim 1, It is characterized in that The step 1 includes the calculation of the quantized 2D convolution operator: a: firstly, the input data and weight data are transferred from the system memory to the local buffer of the accelerator through the DMA Load instruction; Execute GEMM calculation, clear the accelerator accumulator buffer before calculation; move the bias data from the system memory to the local buffer of the accelerator through the DMA Load instruction, and perform the vector ALU ADD operation of the GEMM result and the bias; perform vector ALU RMUL, vector ALU RSHR and vector ALU ADD operations, that is, perform the Requantize process: multiply by multiplier, divide by 2^shift and add output zero point; perform vector ALU MIN / MAX operations to limit the result to the Int8 data range [-128, 127]; finally, store the result in the system memory through the DMA Store instruction. The calculation process of b is basically the same as a, the only difference is that before executing the GEMM operation, b first moves the bias to the accumulator buffer of the accelerator through the DMA Load instruction, and then performs the GEMM multiplication and accumulation calculation on this basis.

3. The pre-quantization model deployment method for DL ​​compiler according to claim 1, It is characterized in that The step 2 includes the following steps: First, through pattern matching, operators a are grouped to obtain grouped b. Each triangle in b represents a matching pattern, namely Q.conv2d + Q.Bias_add + Q.Requantize + Q.Relu. Then, the Target tag is added to the matching pattern through the Annotation operation, indicating that this part of the operators will run on the Target. Through the Partitioning operation, the Group with the Target tag is converted into a function module represented at the graph level, namely c, and an accelerator attribute is added to the function, which instructs the DL compiler to use an accelerator-specific process to process the function. Then, the graph is traversed and modified to change the accelerator function name to a special function name with Target. Next, pre-optimization is performed to meet the requirements of Graph Packing. Finally, Graph Packing is executed. Packing converts the Q operator graph into a common graph-level operator graph and performs Layout transformation so that the input and output of the accelerator function module meet the accelerator hardware requirements, that is, d, d means that the Pack operation needs to be performed before entering the accelerator module; the Unpack operation needs to be performed before exiting the accelerator module and entering the Host module; if there is only an Elementwise operator between the two accelerator modules, no additional Pack and Unpack operations are required.

4. The pre-quantization model deployment method for DL ​​compiler according to claim 1, It is characterized in that The step 3 adopts function-level and operator-level transformation methods to transform the accelerator sub-function and the accelerator operator, including the following steps: First, we use an illegal accelerator graph, i.e., a graph whose input and output do not meet the accelerator hardware requirements, as input and perform function-level transformation. The transformation process is as follows: Step 3.1: Traverse the graph and collect the location information of all sub-functions in the graph. The search results are recorded in a list. Each element of the list is a tuple. The tuple contains two parameters: the sub-function name and the location of the sub-function in the graph. The list is represented as follows: List[Tuple(sub_func, index), …]; Step 3.2: Traverse the list obtained in step (1), collect the accelerator submodules and their location information in the graph according to the special function name of the accelerator function, and record them in the accelerator submodule list, which is represented as follows: List(Tuple(accelerator_func, index)); Step 3.3: Traverse the accelerator submodule list to determine whether the current accelerator function needs to perform Pack and Unpack operations; Step 3.4: Traverse the graph and perform function-level transformation of the graph, that is, treat the sub-function as a normal operator. For each accelerator sub-module, if its pack flag in the pack list is True, perform the Pack operation on its input; if the unpack flag is True, perform the Unpack operation on its output. After a round of traversal, the input and output of all accelerator sub-modules are legal, that is, they meet the accelerator hardware requirements. The graph output in this step is the legal accelerator graph; Then, taking the accelerator legal graph as input, the operator-level graph transformation is performed to downgrade the Q operator to facilitate subsequent processing, and the graph is traversed. If the current operator is an accelerator submodule, each operator in it is further traversed to perform operator-level transformation.

5. The pre-quantization model deployment method for DL ​​compiler according to claim 4, It is characterized in that The step 3.3 determines whether the current accelerator function needs to perform Pack and Unpack operations. The determination rule is: if the current accelerator submodule is the first accelerator submodule, and there are non-element-wise operators in the previous operators, then the Pack operation needs to be performed; if the current accelerator submodule is the last accelerator submodule, and there are non-element-wise operators in the subsequent operators, then the Unpack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the previous accelerator submodule, then the Pack operation needs to be performed; if there are non-element-wise operators between the current accelerator submodule and the next accelerator submodule, then the Unpack operation needs to be performed. The traversal results are recorded in the pack list, and each element of the list is a three-tuple, namely, the accelerator submodule function name, whether to pack the flag, and whether to unpack the flag. The pack list is expressed as follows: List(Tuple(accelerator_func, pack_flag, unpack_flag), …).

6. The method for deploying a pre-quantization model for a DL compiler according to claim 4, It is characterized in that The specific processing steps of performing operator-level transformation in step 3.4 are as follows: Step 3.4.1: Process the Q.conv2d operator: If the current operator is Q.conv2d, replace it with the Conv2d_zp operator, which uses the Input zero point as the Padding value and performs the convolution operation of the Padding Input and Weight. The Conv2d_zp operator is downgraded to the accelerator GEMM Loop, Load, and Store operations through the subsequent accelerator-specific optimization process; Step 3.4.2: Process the Q.Bias_add operator: If the current operator is Q.Bias_add, replace it with the Bias_add_zp operator, which adds the zero-fused Bias constant to the output of Conv2d_zp. The Bias constant here is Qfused_bias=Qbias-∑zpin*Qwgt. The Conv2d_zp operator is converted to accelerator vector ALU ADD and Load operations through subsequent accelerator-specific scheduling; Step 3.4.3: Process the Q.Requantize operator: If the current operator is Q.Requantize, replace it with a combination of a quantized multiplication operator Round_multiply with Rounding, a shift division operator Round_rshift with Rounding, and an Add operator that adds an output zero point. The Round_multiply and Round_rshift operators are converted to accelerator vector ALU RMUL and RSHR operations through subsequent accelerator-specific scheduling. The Add operator is converted to accelerator vector ALU ADD operation through subsequent accelerator-specific scheduling. Step 3.4.4: Process Q.Relu operator: If the current operator is Q.Relu, replace it with a combination of min and max operators, i.e. limit the result to the range [-128, 127]. The min and max operators are converted into accelerator vector ALU MIN and MAX operators through subsequent accelerator-specific scheduling.

7. The pre-quantization model deployment method for DL ​​compiler according to claim 1, It is characterized in that The workflow of the accelerator two-level code generator in step 3 is as follows: first, the accelerator function-level code generator cooperates with the Host code generator to generate the C language version of the operator execution control flow and the accelerator driver code; wherein, the accelerator function-level code generator traverses each accelerator sub-function in the graph and generates an independent accelerator driver function for it, and the driver function completes two tasks: initializing the accelerator and starting the accelerator computing kernel, and the accelerator kernel executes the hardware instructions in the instruction array in sequence; then, the accelerator operator-level generator traverses each operator in the accelerator sub-function, and generates the Python version of the operator execution control flow according to the connection relationship and parameter information of the graph, and calls the accelerator Runtime through the Pybind11 module to generate the accelerator hardware instruction data, and saves the data in the C language array; wherein, py2cpp is a Python package bound to the accelerator C or C++ Runtime API, generated by the Pybind11 module, which can realize the calling of C or C++ functions in Python.

Images