Construction method of TVM quantization model based on grammar layer conversion and related device

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a construction method of a TVM quantization model based on grammar layer conversion and a related device. The method comprises the following steps: acquiring a PyTorch model file and a quantization parameter file of a model to be subjected to parameter quantization; the PyTorch model file is imported into a PyTorch framework, and a Python reasoning script of model reasoning is obtained; analyzing the Python reasoning script line by line according to a grammar rule, and converting the analyzed Python reasoning script into a TVM relay reasoning script according to a grammar conversion rule in combination with model information in a PyTorch model file; analyzing the converted TVM relay reasoning script in the TVM framework, and storing the TVM relay reasoning script as a TVM model file; and calculating and generating a quantized TVM model parameter file according to the quantized parameter file. The method solves the problem of insufficient precision of a direct quantification model in a tvm architecture, and has the advantages of simple program and easiness in debugging and expansion compared with a traditional construction method based on computational graph semantic conversion.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method for constructing a TVM quantization model based on syntax layer conversion and a related device. Background Art

[0002] Parameter quantization is a model compression and acceleration technology widely used in the field of deep learning. It reduces the computational and storage overhead of the model by reducing the number of bits used to represent model parameters. Parameter quantization refers to converting the parameters (such as weights, activation values, etc.) in deep learning models from the original floating-point representation (usually 32 bits) to fixed-point or integer representations with lower bit widths (such as 16 bits, 8 bits, 4 bits, 2 bits, or even 1 bit).

[0003] The advantage of parameter quantization is that it reduces storage space and memory usage: by reducing the bit width of the parameters, the storage space and runtime memory usage of the model parameters can be significantly reduced. For example, quantizing 32-bit floating-point parameters to 8-bit integers can reduce the storage space by about 75%. Parameter quantization can speed up calculations: parameters with lower bit widths are easier to process on hardware, which can speed up the calculation of the model. At the same time, the bandwidth and power consumption required to read and process parameters with lower bit widths are also lower. Parameter quantization can improve the deployment capabilities of edge devices: for edge devices and embedded devices with limited computing resources, parameter quantization makes deep learning models easier to deploy and run on these devices.

[0004] Parameter quantization can reduce model size, improve computing efficiency, facilitate dedicated hardware acceleration, and even improve model generalization capabilities. It is a key step in the practical deployment of neural networks and a key module of TVM (Tensor Virtual Machine, an open source, end-to-end deep learning model compilation framework). Currently, the TVM framework has limited quantization algorithms, resulting in reduced accuracy of some TVM quantized models. The better algorithms in some third-party model quantization tools cannot be directly used by the TVM framework. Adding new quantization algorithms or traditional methods based on computational graph semantic parsing and conversion to the TVM framework has the problems of large code size, complex structure, and difficult debugging and maintenance.

[0005] Therefore, a simpler and more efficient method for constructing TVM quantitative models is urgently needed. Summary of the invention

[0006] The purpose of the present invention is to provide a method for constructing a TVM quantization model based on syntax layer conversion and related devices, which are used to solve the technical problems that the TVM quantization model constructed by the existing method has a complex structure and is difficult to debug and maintain.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for constructing a TVM quantization model based on syntax layer conversion, comprising: Get the PyTorch model file and quantization parameter file of the model to be quantized; Import the PyTorch model file into the PyTorch framework to obtain the Python inference script for model inference; According to the grammatical rules, the Python inference script is parsed, and combined with the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the grammatical conversion rules; Parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; The TVM model file is quantized according to the quantization parameter file to generate a quantized TVM model parameter file.

[0008] A further improvement of the present invention is that: the step of converting the parsed Python inference script into a TVM relay inference script according to the syntax conversion rules in combination with the model information in the PyTorch model file includes: Determine whether a quantization calculation operator is needed. If a quantization calculation operator is needed, refer to the quantization parameter file and insert the corresponding quantization calculation operation statement in the TVM relay inference script; The calculation operator is quantized based on the quantized calculation operation statement. Based on the quantized calculation operator and the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the syntax conversion rules.

[0009] A further improvement of the present invention is that: in the step of parsing the Python reasoning script according to the grammatical rules, and converting the parsed Python reasoning script into the TVM relay reasoning script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the Python reasoning script is converted into the TVM relay reasoning script by converting the Python script line by line at the grammatical layer into the TVM relay grammar.

[0010] A further improvement of the present invention is that the step of obtaining the PyTorch model file and the quantization parameter file of the model to be parameterized specifically includes: Run a third-party model quantization tool to process the model to be parameter quantized to obtain a PyTorch model file and a quantization parameter file; the third-party model quantization tool is Tensorflow Lite, TensorRT, PaddleSlim or PyTorch.

[0011] A further improvement of the present invention is that: in the step of parsing the Python inference script according to the grammar rules, and combining the model information in the PyTorch model file, converting the parsed Python inference script into the TVMrelay inference script according to the grammar conversion rules, the grammar rules include Python script parsing grammar definition and Relay script generation grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

[0012] A further improvement of the present invention is that: in the step of parsing the Python inference script according to the grammatical rules, and converting the parsed Python inference script into the TVMrelay inference script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the grammatical conversion rules are specifically: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

[0013] A further improvement of the present invention is that: in the process of converting the parsed Python inference script into a TVM relay inference script, if a calculation operator that needs to be quantized is encountered, a quantization parameter file is referred to, and in the step of inserting a corresponding quantization calculation operation statement in the TVM relay inference, for the calculation operator that needs to be quantized, quantization parameters are read from the quantization parameter file to generate a corresponding Relay quantization calculation operation statement; the quantization parameters include at least one of an input scale, an input offset, a weight scale, a weight offset, an output scale, and an output offset.

[0014] In a second aspect, the present invention provides a device for constructing a TVM quantization model based on syntax layer conversion, comprising: The model quantization tool running module is used to obtain the PyTorch model file and quantization parameter file of the model to be quantized; A Python script acquisition module is used to import the PyTorch model file in the PyTorch framework and obtain a Python inference script for model inference; A grammar conversion module is used to parse the Python inference script according to grammar rules, and convert the parsed Python inference script into a TVM relay inference script according to grammar conversion rules in combination with model information in the PyTorch model file; The TVM model generation module is used to parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; The quantization parameter calculation module is used to quantize the TVM model file according to the quantization parameter file to generate a quantized TVM model parameter file.

[0015] A further improvement of the present invention is that: in the step of converting the parsed Python inference script into the TVM relay inference script according to the syntax conversion rules in combination with the model information in the PyTorch model file, it is determined whether a quantization calculation operator is required. If a quantization calculation operator is required, the corresponding quantization calculation operation statement is inserted into the TVM relay inference script with reference to the quantization parameter file; The calculation operator is quantized based on the quantized calculation operation statement. Based on the quantized calculation operator and the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the syntax conversion rules.

[0016] A further improvement of the present invention is that: in the step of parsing the Python reasoning script according to the grammatical rules, and converting the parsed Python reasoning script into the TVM relay reasoning script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the Python reasoning script is converted into the TVM relay reasoning script by converting the Python script grammatical layer line by line into the TVM relay grammar.

[0017] A further improvement of the present invention is that the step of obtaining the PyTorch model file and the quantization parameter file of the model to be parameterized specifically includes: Run a third-party model quantization tool to process the model to be parameter quantized to obtain a PyTorch model file and a quantization parameter file; the third-party model quantization tool is Tensorflow Lite, TensorRT, PaddleSlim or PyTorch.

[0018] A further improvement of the present invention is that: in the step of parsing the Python inference script according to the grammar rules, and combining the model information in the PyTorch model file, converting the parsed Python inference script into the TVMrelay inference script according to the grammar conversion rules, the grammar rules include Python script parsing grammar definition and Relay script generation grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

[0019] A further improvement of the present invention is that: in the step of parsing the Python inference script according to the grammatical rules, and converting the parsed Python inference script into the TVMrelay inference script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the grammatical conversion rules are specifically: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

[0020] A further improvement of the present invention is that: in the process of converting the parsed Python inference script into a TVM relay inference script, if a calculation operator that needs to be quantized is encountered, a quantization parameter file is referred to, and in the step of inserting a corresponding quantization calculation operation statement in the TVM relay inference, for the calculation operator that needs to be quantized, quantization parameters are read from the quantization parameter file to generate a corresponding Relay quantization calculation operation statement; the quantization parameters include at least one of an input scale, an input offset, a weight scale, a weight offset, an output scale, and an output offset.

[0021] In a third aspect, the present invention provides an electronic device, comprising: A memory for storing executable instructions; The processor is used to implement the method for constructing a TVM quantization model based on syntax layer conversion when executing the executable instructions stored in the memory.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the method for constructing a TVM quantization model based on syntax layer conversion.

[0023] In a fifth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method for constructing a TVM quantization model based on syntax layer conversion.

[0024] Compared with the prior art, the present invention has the following unexpected beneficial effects: The present invention provides a method for constructing a TVM quantization model based on syntax layer conversion, including: obtaining a PyTorch model file and a quantization parameter file of a model to be parameterized; importing the PyTorch model file in a PyTorch framework, and obtaining a Python reasoning script for model reasoning; parsing the Python reasoning script according to syntax rules, and combining the model information in the PyTorch model file, according to syntax conversion rules, converting the Python reasoning script into a TVM relay reasoning script; parsing the converted TVM relay reasoning script in the TVM framework, and saving it as a TVM model file; according to the quantization parameter file, calculating and generating a quantized TVM model parameter file. The present invention solves the problem of insufficient precision of the direct quantization model in the TVM architecture by converting the Python script syntax layer line by line into a TVM relay syntax, and then directly parsing it into a TVM quantization model using the TVM architecture, and compared with the traditional construction method based on the semantic conversion of the computational graph, it has the advantages of simple program, easy debugging and expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 A schematic diagram of a method for constructing a TVM quantization model based on syntax layer conversion proposed in an embodiment of the present invention; Figure 2 A schematic diagram of a method for constructing a TVM quantization model based on syntax layer conversion proposed in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a device for constructing a TVM quantization model based on syntax layer conversion proposed in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an electronic device proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0027] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0029] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0031] The disclosed embodiment provides a method for constructing a TVM quantization model based on syntax layer conversion and a related device, which is used to solve the technical problems of complex structure, difficult debugging and maintenance of TVM quantization models constructed by existing methods. The exemplary application of the electronic device provided by the disclosed embodiment is described below. The electronic device provided by the disclosed embodiment can be implemented as a smart phone, a smart watch, a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), an intelligent voice interaction device, a smart home appliance, and a vehicle-mounted terminal, etc. Various types of terminals or user terminals can also be implemented as a server. As another practicable method of the disclosed embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms, etc., but is not limited to this.

[0032] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0033] See also Figure 1 and Figure 2 As shown, this embodiment proposes a method for constructing a TVM quantization model based on syntax layer conversion, including: S1. Run a third-party model quantization tool to process the model to be quantized and obtain the PyTorch model file and the quantization parameter json file. In a specific implementation, the third-party model quantization tool is Tensorflow Lite; TensorFlow Lite is an inference framework launched by Google for embedded devices. It supports float16 and int8 low precision, and supports post-training quantization and quantization-aware training.

[0034] In a specific implementation, the third-party model quantization tool is TensorRT; TensorRT is a neural network inference engine proposed by Nvidia. It supports 8-bit quantization after training. It uses a cross-entropy-based model quantization algorithm to achieve this by minimizing the difference between two distributions.

[0035] In a specific implementation, the third-party model quantization tool is PaddleSlim; PaddleSlim is a model quantization tool proposed by Baidu, included in the PaddlePaddle framework, which supports quantization-aware training, offline quantization, weight global quantization, and channel-level quantization.

[0036] In a specific implementation, the third-party model quantization tool is PyTorch; PyTorch 1.3 has supported quantization since then. It is based on QNNPACK and supports post-training quantization, dynamic quantization, and quantization-aware training.

[0037] In a specific implementation, the present invention runs a third-party model quantization tool PyTorch, processes the model to be parameter quantized, and obtains a PyTorch model file and a quantization parameter json file; S2, import the PyTorch model file obtained in S1 into the PyTorch framework, and obtain the Python inference script for model inference through torch.fx.GraphModule based on the model information in the PyTorch model file, as shown below: … Model_0_conv= getattr(selfmodel, "0").conv(x); x= None model_0_bn=getattr(selfmodel, "0").bn(model_0_conv); model_0_conv =None model_0_act=getattr(self.model, "0").act(model_0_bn); model_0_bn=None … S3. According to the grammatical rules, parse the Python inference script obtained in step S2 line by line, and convert the parsed Python inference script into a TVM relay inference script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, as shown below: … %10=nn.conv2d(%9, %model 0_conv_weight, strides=[2, 2], padding=[1,1, 1, 1j, channels=16, kernel_size=[3, 3]); %11 =nn.bias_add(%10,%model_0_conv_bias); %12 =sigmoid(%11); %13 = multiply(%11, %12); … In a specific implementation, when a quantizable calculation operator is encountered during conversion, a quantization parameter file is referenced and a quantization calculation operation statement is inserted into the TVMrelay inference script, as shown below: … %6=multiply(%5, 5.56522f) %7=round(%6); %8=clip(%7,a_min=-127f,a_max-127f); %9 = cast(%9, dtype="int8"); … In a specific implementation, the grammar rules include a Python script parsing grammar definition and a Relay script generating grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

[0038] In a specific implementation, according to the grammar rules, the Python inference script is parsed line by line, and combined with the model information in the PyTorch model file, according to the grammar conversion rules, the parsed Python inference script is converted into a TVMrelay inference script, and the grammar conversion rules are specifically: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

[0039] In a specific implementation, the Python script parsing syntax is defined as:

[0040] In a specific implementation, the relay script generation syntax is defined as:

[0041] In a specific implementation, the grammar conversion algorithm can be described as:

[0042] S4. Parse the TVM relay inference script in the TVM framework and save it as a TVM model json file; S5. Based on the quantization parameter json file, the TVM framework calculates and generates the quantized TVM model parameter json file.

[0043] See also Figure 3 As shown, the embodiment of the present invention also provides a device for constructing a TVM quantization model based on syntax layer conversion, including: The model quantization tool running module is used to obtain the PyTorch model file and quantization parameter file of the model to be quantized; A Python script acquisition module is used to import the PyTorch model file in the PyTorch framework and obtain a Python inference script for model inference; A syntax conversion module is used to parse the Python inference script line by line according to the syntax rules, and convert the parsed Python inference script into a TVMrelay inference script according to the syntax conversion rules in combination with the model information in the PyTorch model file; The TVM model generation module is used to parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; The quantization parameter calculation module is used to calculate and generate a quantized TVM model parameter file according to the quantization parameter file.

[0044] In a specific implementation, the Python inference script is parsed line by line according to the grammatical rules, and in combination with the model information in the PyTorch model file, the parsed Python inference script is converted into a TVMrelay inference script according to the grammatical conversion rules. In the process of parsing the Python inference script line by line to convert it into the TVM relay inference script, if a quantizable calculation operator is encountered, the corresponding quantized calculation operation statement is inserted into the TVM relay inference with reference to the quantization parameter file.

[0045] In a specific implementation, the Python reasoning script is parsed line by line according to grammar rules, and in combination with model information in a PyTorch model file, the parsed Python reasoning script is converted into a TVM relay reasoning script according to grammar conversion rules. The Python script grammar layer is converted line by line into the TVM relay grammar to achieve conversion of the Python reasoning script into a TVM relay reasoning script.

[0046] In a specific implementation, the step of obtaining the PyTorch model file and the quantization parameter file of the model to be parameterized includes: Run a third-party model quantization tool to process the model to be parameter quantized to obtain a PyTorch model file and a quantization parameter file; the third-party model quantization tool is Tensorflow Lite, TensorRT, PaddleSlim or PyTorch.

[0047] In a specific implementation, the step of parsing the Python inference script line by line according to the grammar rules, and converting the parsed Python inference script into a TVMrelay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, wherein the grammar rules include a Python script parsing grammar definition and a Relay script generating grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

[0048] In a specific implementation, the step of parsing the Python inference script line by line according to the grammar rules, and converting the parsed Python inference script into the TVMrelay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, the grammar conversion rules are specifically: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

[0049] In a specific implementation, in the process of converting the Python inference script into the TVM relay inference script by line-by-line parsing, if a quantizable calculation operator is encountered, refer to the quantization parameter file, in the step of inserting the corresponding quantization calculation operation statement in the TVM relay inference, for the calculation operator that needs to be quantized, read the quantization parameter from the quantization parameter file, and generate the corresponding Rela quantization calculation operation statement; the quantization parameter includes at least one of an input scale, an input offset, a weight scale, a weight offset, an output scale, and an output offset.

[0050] See also Figure 4 As shown, an embodiment of the present invention also provides an electronic device 100 for implementing a method for constructing a TVM quantization model based on syntax layer conversion; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0051] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the method for constructing a TVM quantization model based on syntax layer conversion by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0052] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0053] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for constructing a TVM quantization model based on syntax layer conversion, and the processor 102 may execute the plurality of instructions to implement: Get the PyTorch model file and quantization parameter file of the model to be quantized; Import the PyTorch model file into the PyTorch framework to obtain the Python inference script for model inference; According to the grammatical rules, the Python inference script is parsed line by line, and combined with the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the grammatical conversion rules; Parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; According to the quantization parameter file, a quantized TVM model parameter file is calculated and generated.

[0054] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0055] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a TVM quantization model based on syntax layer conversion, characterized in that: include: Get the PyTorch model file and quantization parameter file of the model to be quantized; Import the PyTorch model file into the PyTorch framework to obtain the Python inference script for model inference; According to the grammatical rules, the Python inference script is parsed, and combined with the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the grammatical conversion rules; Parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; The TVM model file is quantized according to the quantization parameter file to generate a quantized TVM model parameter file.

2. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 1, characterized in that: The step of converting the parsed Python inference script into a TVM relay inference script according to the syntax conversion rules in combination with the model information in the PyTorch model file includes: Determine whether a quantization calculation operator is needed. If a quantization calculation operator is needed, refer to the quantization parameter file and insert the corresponding quantization calculation operation statement in the TVMrelay inference script; The calculation operator is quantized based on the quantized calculation operation statement. Based on the quantized calculation operator and the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the syntax conversion rules.

3. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 1, characterized in that: In the step of parsing the Python reasoning script according to the grammatical rules, and converting the parsed Python reasoning script into the TVM relay reasoning script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the Python reasoning script is converted into the TVM relay reasoning script by converting the Python script line by line at the grammatical layer into the TVM relay grammar.

4. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 1, characterized in that: The step of obtaining the PyTorch model file and the quantization parameter file of the model to be parameterized specifically includes: Run a third-party model quantization tool to process the model to be parameter quantized to obtain a PyTorch model file and a quantization parameter file; the third-party model quantization tool is Tensorflow Lite, TensorRT, PaddleSlim or PyTorch.

5. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 1, characterized in that: In the step of parsing the Python inference script according to the grammar rules, and converting the parsed Python inference script into the TVM relay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, the grammar rules include Python script parsing grammar definition and Relay script generation grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

6. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 5, characterized in that: In the step of parsing the Python inference script according to the grammar rules, and converting the parsed Python inference script into the TVM relay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, the grammar conversion rules are specifically as follows: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

7. The method for constructing a TVM quantization model based on syntax layer conversion according to claim 2, characterized in that: In the process of converting the parsed Python inference script into a TVM relay inference script, if a calculation operator that needs to be quantized is encountered, refer to the quantization parameter file, and in the step of inserting the corresponding quantization calculation operation statement in the TVM relay inference, for the calculation operator that needs to be quantized, read the quantization parameter from the quantization parameter file and generate the corresponding Relay quantization calculation operation statement; the quantization parameter includes at least one of an input scale, an input offset, a weight scale, a weight offset, an output scale, and an output offset.

8. A device for constructing a TVM quantization model based on syntax layer conversion, characterized in that: include: The model quantization tool running module is used to obtain the PyTorch model file and quantization parameter file of the model to be quantized; A Python script acquisition module is used to import the PyTorch model file in the PyTorch framework and obtain a Python inference script for model inference; A grammar conversion module is used to parse the Python inference script according to grammar rules, and convert the parsed Python inference script into a TVM relay inference script according to grammar conversion rules in combination with model information in the PyTorch model file; The TVM model generation module is used to parse the converted TVM relay inference script in the TVM framework and save it as a TVM model file; The quantization parameter calculation module is used to quantize the TVM model file according to the quantization parameter file to generate a quantized TVM model parameter file.

9. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 8, characterized in that: In the step of converting the parsed Python inference script into a TVM relay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, it is determined whether a quantization calculation operator is required. If a quantization calculation operator is required, the corresponding quantization calculation operation statement is inserted into the TVM relay inference script with reference to the quantization parameter file; The calculation operator is quantized based on the quantized calculation operation statement. Based on the quantized calculation operator and the model information in the PyTorch model file, the parsed Python inference script is converted into a TVM relay inference script according to the syntax conversion rules.

10. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 8, characterized in that: In the step of parsing the Python reasoning script according to the grammatical rules, and converting the parsed Python reasoning script into the TVM relay reasoning script according to the grammatical conversion rules in combination with the model information in the PyTorch model file, the Python reasoning script is converted into the TVM relay reasoning script by converting the Python script grammatical layer line by line into the TVM relay grammar.

11. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 8, characterized in that: The step of obtaining the PyTorch model file and the quantization parameter file of the model to be parameterized specifically includes: Run a third-party model quantization tool to process the model to be parameter quantized to obtain a PyTorch model file and a quantization parameter file; the third-party model quantization tool is Tensorflow Lite, TensorRT, PaddleSlim or PyTorch.

12. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 8, characterized in that: In the step of parsing the Python inference script according to the grammar rules, and converting the parsed Python inference script into the TVM relay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, the grammar rules include Python script parsing grammar definition and Relay script generation grammar definition, wherein: Python script parsing syntax definition includes variable definition, operation definition and parameter definition; The syntax definition generated by the Relay script includes variable definition, operation definition and parameter definition, and the operation definition includes a variety of quantization-related operations.

13. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 12, characterized in that: In the step of parsing the Python inference script according to the grammar rules, and converting the parsed Python inference script into the TVM relay inference script according to the grammar conversion rules in combination with the model information in the PyTorch model file, the grammar conversion rules are specifically as follows: Convert the variables, operations, and parameters in the Python inference script according to the grammatical rules of the Relay script to form the corresponding TVM relay inference script.

14. The device for constructing a TVM quantization model based on syntax layer conversion according to claim 9, characterized in that: In the process of converting the parsed Python inference script into a TVM relay inference script, if a calculation operator that needs to be quantized is encountered, refer to the quantization parameter file, and in the step of inserting the corresponding quantization calculation operation statement in the TVM relay inference, for the calculation operator that needs to be quantized, read the quantization parameter from the quantization parameter file and generate the corresponding Relay quantization calculation operation statement; the quantization parameter includes at least one of an input scale, an input offset, a weight scale, a weight offset, an output scale, and an output offset.

15. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

16. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement the method described in any one of claims 1 to 7.

17. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images