AI compiler model compiling multistage performance evaluation method and system

Abstract

The invention relates to an AI compiler model compiling multistage performance evaluation method and system, and the method comprises the steps: generating a plurality of candidate solutions for each operator through an AI compiler, and enabling each candidate solution to have different intermediate expressions; time evaluation is carried out on the intermediate expressions corresponding to all the candidate solutions, M optimal candidate solutions are selected, and M is a positive integer; the intermediate expressions corresponding to the M candidate solutions are compiled to generate corresponding assembly instructions, time evaluation is carried out, N optimal candidate solutions are selected, and N is a positive integer and smaller than M; the N candidates are decompiled to generate binary instructions, the binary instructions are imported into a chip for actual measurement, and the optimal running time and the corresponding running instruction are obtained according to the actual measurement result. The partial optimal solution selected step by step is transmitted to the next stage for more accurate evaluation, compared with an existing time evaluation method, the accuracy is higher, the compiling time can be greatly shortened, and the selection problem of multi-back-end implementation can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of AI compiler technology, and in particular to a method and system for evaluating the multi-level performance of AI compiler model compilation. Background Art

[0002] The AI ​​compiler is a compiler for a specific field, which mainly solves problems such as training, reasoning, and deployment of neural networks. In order to solve the problem of the endless emergence of new operators and the explosion of operator combinations in deep learning, AI compilers often propose corresponding code generation (codegen) technology. Taking the typical deep learning compiler TVM as an example, the deep learning network is first converted into its internal intermediate representation (IR). This IR can summarize the operators in the deep learning network. To implement a new operator, it only needs to convert the operator into the relevant internal IR, and then convert the IR into hardware executable instructions or codes through the instruction generation module.

[0003] However, how to ensure the efficiency of the operators automatically generated by the deep learning compiler has become a new problem that must be faced. In order to solve this problem, a very large solution space is usually constructed, and then an optimal solution is found in the solution space through a certain evaluation model. After obtaining the optimal solution, the fusion or splitting information of the operator can be determined, and then the final instruction is generated accordingly. In this step, the accuracy of time evaluation is very important. Usually, the accuracy is higher than 90% to be acceptable in engineering, because time evaluation, as the most important part of the graph fusion strategy, will directly affect the fusion behavior between operators. For example, the fusion benefits of counting nodes A and B are usually calculated by adding the running time of A to the running time of B and then subtracting the running time of AB fused together. Subsequent fusion behaviors are all judged based on the relevant fusion benefits. Generally, AI chips are designed as a multi-level storage structure. The use of multi-level pipelines in the calculation process can effectively cover up the delay of data transmission. Due to the limited on-chip space, operators need to be split into smaller blocks for batch execution. Therefore, the composition of the solution space includes the splitting of operators of different dimensions (N, H, W, C) at different storage levels (shared memory, local memory), as well as the pipeline settings at different storage levels. Usually, the number of candidate solutions in the original solution space is in the tens of thousands. A simple way is to build a mathematical model to evaluate the running time of the fused candidate solution based on the intermediate expression IR in the compilation process. This method has high running efficiency, but the time evaluation error is large. If each feasible solution is actually executed on the chip, the compilation time will be unbearable.

[0004] There are currently several main types of time assessment methods:

[0005] 1) Building a time evaluation model (cost model) based on the IR level. In order to solve the problem of inaccurate evaluation, more hardware implementation details will be considered during the evaluation, but the effect is limited. IR is usually a general expression of the upper layer, and mixing too many hardware details is not conducive to software design;

[0006] 2) Further compile the IR into low-level instructions such as assembly instructions, and then do time evaluation. The evaluation method is also based on a similar mathematical model. Although this solution is more efficient than direct actual measurement, the compilation time is still unbearable, so this method is rarely used;

[0007] 3) Use reinforcement learning to build a time evaluation model (cost model), that is, perform time evaluation at the IR level, perform actual measurement execution through random sampling, and then correct the IR level time evaluation model, and make the evaluation more accurate through continuous iteration. If the sampling is too dense, the compilation time will be unbearable. If the sampling is too sparse, it will make the model difficult to converge and the evaluation error will be large. Usually, the compilation time of this solution is also very long;

[0008] 4) Build a neural network to learn the above evaluation process, and use the trained model to perform time evaluation. In this way, the optimal solution corresponding to IR can be given at one time, which is the most efficient among all algorithms. However, the evaluation network is complex to train. Due to the complexity of the network fragments, the diversity of operator fusion strategies and the emergence of new operators, its robustness is very poor.

[0009] In addition, the above methods also face the problem of multiple backend selection. For example, if an operator developer develops an efficient operator in advance and then generates the operator through automatic code generation by the compiler, how to evaluate the pros and cons of these two methods during the compilation process? None of the above methods provides a solution, so the usual practice is to give priority to the operator implementation method already available in the operator library. Automatic code generation is performed only when there is no corresponding implementation in the operator library.

[0010] In summary, the time evaluation model based on the IR level has high operating efficiency but large errors. The time evaluation scheme based on the instruction level or actual measurement will lead to too long compilation time. The compilation time of the scheme based on reinforcement learning is still relatively long. The scheme based on training neural networks for time evaluation will lead to maintenance costs and poor robustness. Moreover, none of the above schemes can solve the problem of selecting multiple backend implementations. Summary of the invention

[0011] The purpose of the present invention is to provide a multi-level performance evaluation method and system for AI compiler model compilation, so as to at least solve one of the technical problems existing in the existing time evaluation method.

[0012] To achieve the above object, the present invention provides a multi-level performance evaluation method for AI compiler model compilation, comprising:

[0013] Use the AI ​​compiler to generate several candidate solutions for each operator, each with a different intermediate expression;

[0014] Perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer;

[0015] Compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions, where N is a positive integer and is less than the M;

[0016] The N candidates are decompiled to generate binary instructions and imported into the chip for actual measurement, and the optimal running time and corresponding running instructions are obtained according to the actual measurement results.

[0017] Optionally, the candidate solution includes splitting the operator into different storage levels in different dimensions and pipeline settings of the different storage levels.

[0018] Optionally, after generating several candidate solutions for each operator, candidate solutions that do not meet the performance requirements are filtered out based on hardware information.

[0019] Optionally, before decompilation of the N candidates to generate binary instructions, the AI ​​compiler model compilation multi-level performance evaluation method further includes:

[0020] According to the description information of the original operator, it is matched with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated. If the match fails, it is directly skipped.

[0021] Optionally, for a network segment including multiple operators, a time evaluation is performed for each operator in the network segment, and the optimal running time and corresponding running instructions of each operator are obtained according to the fusion benefit to complete the compilation process.

[0022] Optionally, M is one tenth of all candidate solutions, and N is one tenth of M.

[0023] Based on the same technical concept, the present invention also provides an AI compiler model compilation multi-level performance evaluation system, comprising:

[0024] AI compiler, used to generate several candidate solutions for each operator, each candidate solution has a different intermediate expression;

[0025] The IR level evaluation module is used to perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer.

[0026] An instruction-level evaluation module, used to compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions therefrom, where N is a positive integer and is less than the M;

[0027] The actual measurement level evaluation module is used to decompile the N candidates to generate binary instructions and import them into the chip for actual measurement, and obtain the optimal running time and corresponding running instructions according to the actual measurement results.

[0028] Optionally, the AI ​​compiler model compilation multi-level performance evaluation system also includes a pruning module, which is used to generate a number of candidate solutions for each operator and then filter out candidate solutions that do not meet performance requirements based on hardware information.

[0029] Optionally, the AI ​​compiler model compilation multi-level performance evaluation system also includes a matching module, which is used to match the description information of the original operator with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated; if the match fails, it is directly skipped.

[0030] Based on the same technical concept, the present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed, it can implement the multi-level performance evaluation method for compiling an AI compiler model as described above.

[0031] The AI ​​compiler model compilation multi-level performance evaluation method and system provided by the present invention has at least one of the following beneficial effects:

[0032] 1) By passing the partial optimal solutions selected at each level to the next level for more accurate evaluation, the efficiency of IR-level evaluation and the accuracy of instruction-level evaluation and actual measurement-level evaluation can be fully utilized, and the optimal operator automatic generation method can be found on the basis of greatly reducing the compilation time. Compared with the IR-level or instruction-level time evaluation, the accuracy is higher. Compared with the instruction-level or actual measurement-based evaluation methods, it can not only greatly reduce the compilation time, but also effectively solve the selection problem of multiple backend implementations;

[0033] 2) It can significantly improve the efficiency of model compilation and deployment as well as function and performance verification, thereby improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0035] Figure 1A flowchart of a multi-level performance evaluation method for AI compiler model compilation provided by one embodiment of the present invention;

[0036] Figure 2 A structural block diagram of a multi-level performance evaluation system for compiling an AI compiler model provided by an embodiment of the present invention.

[0037] in:

[0038] 100-AI compiler; 200-IR-level evaluation module; 300-instruction-level evaluation module; 400-measurement-level evaluation module; 500-pruning module; 600-matching module. DETAILED DESCRIPTION

[0039] As described in the background technology, among the current mainstream time evaluation methods, the time evaluation model based on the IR level has high operating efficiency but large errors. The time evaluation scheme based on the instruction level or actual measurement will lead to too long compilation time. The compilation time of the scheme based on reinforcement learning is still relatively long. The scheme based on training neural networks for time evaluation will lead to maintenance costs and poor robustness. In addition, none of the above schemes can solve the problem of selecting multiple backend implementations.

[0040] Based on this, the present invention aims to provide a multi-level performance evaluation method, system and readable storage medium for AI compiler model compilation, which selects some optimal solutions level by level and passes them to the next level for more accurate evaluation. It can fully utilize the efficiency based on IR level evaluation, and fully utilize the accuracy based on instruction level evaluation and actual measurement level evaluation. It can find the optimal operator automatic generation method on the basis of greatly reducing the compilation time. Compared with IR-level or instruction-level time evaluation, it has higher accuracy. Compared with instruction-level or actual measurement-based evaluation methods, it can not only greatly reduce the compilation time, but also effectively solve the selection problem of multiple back-end implementations.

[0041] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the same or similar conditions as the effects that can be produced by the present invention and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0042] As used in the present invention, the singular forms "one", "an", and "the" include plural objects, unless the content clearly indicates otherwise. As used in the present invention, the term "or" is generally used in a sense that includes "and / or", unless the content clearly indicates otherwise. As used in the present invention, the term "several" is generally used in a sense that includes "at least one", unless the content clearly indicates otherwise. As used in the present invention, the term "at least two" is generally used in a sense that includes "two or more", unless the content clearly indicates otherwise. In addition, the terms "first", "second", and "third" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", and "third" may explicitly or implicitly include one or at least two of the features.

[0043] In the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] Please refer to Figure 1 , an embodiment of the present invention provides an AI compiler model compilation multi-level performance evaluation method, comprising the following steps:

[0045] S1. Use the AI ​​compiler to generate several candidate solutions for each operator, each candidate solution has a different intermediate representation (IR);

[0046] S2. Perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer.

[0047] S3, compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions, where N is a positive integer and less than the M;

[0048] S3. Decompile the N candidates to generate binary instructions and import them into the chip for actual measurement, and obtain the optimal running time and corresponding running instructions according to the actual measurement results.

[0049] The AI ​​compiler model compilation multi-level performance evaluation method provided in this embodiment selects some of the optimal solutions at each level and passes them to the next level for more accurate evaluation. Compared with the time evaluation based on the IR level or instruction level, it has higher accuracy. Moreover, compared with the evaluation method based on the instruction level or actual measurement, it can not only greatly reduce the compilation time, but also effectively solve the selection problem of multiple backend implementations, significantly improve the efficiency of model compilation deployment and function and performance verification, thereby improving production efficiency.

[0050] First, execute S1, and use the AI ​​compiler to generate several candidate solutions for each operator. Each candidate solution has a different intermediate expression, and the intermediate expression includes the input, output, splitting, and mathematical logic description of the operator. In this embodiment, the candidate solution includes the splitting of the operator at different storage levels in different dimensions and the pipeline settings of the different storage levels. The intermediate expression (IR) can be multi-level, and a level of IR is generated for each subsequent compilation step.

[0051] Preferably, before executing S2, after generating several candidate solutions for each operator, candidate solutions with substandard performance are filtered out according to the hardware information. This step is equivalent to a pruning operation, that is, directly filtering out some solutions with poor performance according to the hardware information to improve evaluation efficiency.

[0052] Then execute S2 to perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer. This step aims to perform time evaluation on candidate solutions based on the IR level and select the best M candidate solutions to enter the next level for more accurate evaluation.

[0053] Then, S3 is executed to compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions, where N is a positive integer and is less than M. This step is intended to perform time evaluation on the candidate solutions based on the instruction level, and select the best N candidate solutions to enter the next level for more accurate evaluation. In this embodiment, the running instructions include but are not limited to assembly instructions or other hardware instructions.

[0054] In this embodiment, M is preferably one tenth of all candidate solutions, and N is preferably one tenth of M, but the present application is not limited thereto. For example, after filtering out candidate solutions that do not meet the performance standards according to hardware information, 10,000 candidate solutions are generated. After S2 performs time evaluation at the IR level, 1,000 candidate solutions are screened out and enter S3 for instruction-level time evaluation, and then 100 candidate solutions are screened out and enter S4 for actual measurement evaluation.

[0055] Before executing S4, the AI ​​compiler model compilation multi-level performance evaluation method further includes:

[0056] According to the description information of the original operator (which can be understood as the IR of the first-level neural network node description), it is matched with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated. If the match fails, it is skipped directly. This step is intended to solve the selection problem of multiple backend implementations. Since the operators in the operator library are all compiled in advance, the description information of the original operator is matched with the operator in the existing operator library. If the match is successful, the matched operator and the corresponding instruction information can be used as a candidate solution, and enter the next level together with the N candidate solutions screened by S3 for actual measurement and evaluation. In this way, the advantages and disadvantages of the operators in the operator library and other candidate solutions can be directly evaluated during the compilation process.

[0057] Finally, S4 is executed to decompile the N candidates to generate binary instructions and import them into the chip for actual measurement, and the optimal running time and corresponding running instructions are obtained according to the actual measurement results, so as to obtain the optimal operator implementation.

[0058] In this embodiment, the running instruction may be an assembly instruction or a binary instruction.

[0059] Preferably, for a network segment including multiple operators, a time evaluation is performed for each operator in the network segment, and the optimal running time and corresponding running instructions of each operator are obtained according to the fusion benefit to complete the compilation process. This embodiment takes a simple network segment as an example to further illustrate the use method of the multi-level performance evaluation of AI compiler model compilation provided in this embodiment. For example, a simple network segment is provided, including three nodes A, B and C (operators or fusion operators), where A outputs to B, and B outputs to C. All possible fusion results of the network include:

[0060] 1) None of them are integrated, and the final network structure is still A, B and C;

[0061] 2) Only A and B are fused, and the final network structure is AB and C;

[0062] 3) Only B and C are fused, and the final network structure is A and BC;

[0063] 4) After all are integrated, the final network structure contains only one integrated node ABC.

[0064] Therefore, the operators that need to be evaluated in total include: A, B, C, AB, BC, ABC. For each operator, the time evaluation and final code generation are performed according to the multi-level performance evaluation method of AI compiler model compilation proposed in the embodiment of the present invention, and the final selection can be made according to the fusion benefit, thereby completing the compilation process of the entire network segment.

[0065] The following will use a specific example to illustrate the technical effect of the multi-level performance evaluation method for AI compiler model compilation proposed in an embodiment of the present invention.

[0066] In the existing solutions based on IR to build time evaluation models, it is usually implemented in the following way. Taking a simple vector addition using a one-level pipeline as an example, a block of data is copied from DDR (double rate synchronous dynamic random access memory) to the chip, assuming that this process is L. Then the data is vector added on the chip, assuming that this process is C. After the calculation is completed, the calculation structure needs to be written back to DDR from the chip, assuming that this process is S. Since L, C and S can be performed in a pipeline manner, after the data is divided into blocks, L, C and S can be performed at the same time, so the maximum time consumption of L, C and S is the running time of the data block. The final evaluation time can be obtained by counting the running time of all blocks. Of course, the actual process will involve the selection of multiple levels of pipelines and the splitting of various dimensions, and the final model will be much more complicated. Obviously, this model is an ideal model. The evaluation of C may be relatively accurate, but it is difficult to accurately evaluate L and S because the instantaneous bandwidth of data transmission is related to many factors, such as the size of the data, the starting address, the step length of the data copy, etc. At the same time, the pipeline programming model also involves data synchronization. In short, there are many influencing factors. In order to make the evaluation accurate, a lot of hardware details need to be exposed at this level, which destroys the hierarchy of software design and is difficult to achieve accurate evaluation. The instruction-level time evaluation method is very similar to the IR-based time evaluation method, except that it can present more detailed execution details at the instruction level.

[0067] As the operator expression layer, the IR layer cannot perceive the implementation details of the hardware, and it is difficult to achieve the accuracy of time evaluation. Based on this, the present invention no longer pursues the absolute time accuracy of operator time evaluation at the IR level. Through a large number of test statistics, it is found that although the time cannot be accurately evaluated when the time evaluation model (cost model) is constructed at the IR level (the average error is about 30% compared with the real time), it is very effective in selecting the optimal interval. Taking 100 results as an example, the top 10 statistically calculated by the time evaluation model is more than 95% accurate compared with the real top 10. This is because the ideal model lacks only the hardware execution details, but the quality of the splitting and pipeline construction is consistent with the actual execution efficiency of the hardware in terms of trend.

[0068] For AI chips, a simple vector add statement at the upper level may generate dozens of hardware instructions when translated into the underlying instructions, so the compilation time of instruction-based time evaluation will increase by more than 10 times compared with IR-based time evaluation. At the same time, instruction-level time evaluation is very close to the hardware execution process, usually with an accuracy of more than 90%, and its selection of the optimal interval is also more accurate. Taking 100 results as an example, the top 10 statistically calculated by this layer of time evaluation model is more than 98% accurate compared with the real top 10.

[0069] During the actual test run, the instructions corresponding to the operator need to be passed to the development board, then run and count the time, and then send the time result back. When counting the running time, in order to avoid the influence of hardware preheating and running fluctuations, it is usually repeated more than a hundred times and then the average value is taken. At the same time, data transmission between different devices is very time-consuming. Therefore, the compilation time of the time evaluation based on actual measurement will increase by more than 10 times compared with the time evaluation based on the instruction level, and the compilation time will increase by more than 100 times compared with the time evaluation based on the IR level.

[0070] Based on this, the multi-level performance evaluation method proposed in the present invention will select part (such as 10%) of the optimal solutions step by step and pass them to the next level for more accurate evaluation. Assume that the number of solution spaces of a certain operator is 10,000 (usually the solution space is much larger than this data). According to the above analysis of the time-consuming relationship of time evaluation models at different levels, assuming that the time consumption based on the IR level is 1, the time consumption based on the instruction level is 10, and the time consumption based on actual measurement is 100. The total time consumption of the time evaluation based on the IR level is 10,000, and the absolute time accuracy is about 70%. The total time consumption based on the instruction level is 100,000, and the absolute time accuracy exceeds 90%. The total time consumption based on actual measurement is 1,000,000, and the absolute time accuracy is 100%. The total time consumption of the multi-level performance evaluation method proposed by the present invention is as follows: the total time consumption of the IR-level time evaluation is 10,000, the total time consumption of selecting 10% of the solutions based on the instruction-level time evaluation is 1000*10, and the total time consumption of selecting another 10% of the solutions based on the actual measurement is 100*100, so the total time consumption of the whole process is 30,000, and the corresponding absolute time accuracy exceeds 95%*98%, i.e., 93.1%. If an operator that has been implemented in the operator library is encountered, it only needs to be measured as one of the candidate solutions of the actual measurement evaluation scheme, and the optimal solution can be selected, and the compilation time will not be affected too much.

[0071] Through the above analysis, it can be seen that the multi-level performance evaluation method proposed in the present invention has improved the accuracy of the evaluation method based on the IR level by one level, and fully meets the engineering requirements. Compared with the time evaluation method based on the instruction level, the time consumption is only 30% of that, and compared with the evaluation method based on actual measurement, the time consumption is only 3%. Under the condition of meeting the performance accuracy requirements, the compilation time is greatly reduced, and the selection problem of multiple backend implementations can be effectively solved.

[0072] Based on the same invention concept, please refer to Figure 2 The embodiment of the present invention further proposes an AI compiler model compilation multi-level performance evaluation system, comprising:

[0073] The AI ​​compiler 100 is used to generate a number of candidate solutions for each operator, each candidate solution having a different intermediate expression;

[0074] The IR level evaluation module 200 is used to perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer;

[0075] An instruction-level evaluation module 300 is used to compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions, where N is a positive integer and is less than the M;

[0076] The actual measurement level evaluation module 400 is used to decompile the N candidates to generate binary instructions and import them into the chip for actual measurement, and obtain the optimal running time and corresponding running instructions according to the actual measurement results.

[0077] In this embodiment, the candidate solutions include the splitting of the operator into different storage levels in different dimensions and the pipeline settings of the different storage levels. The intermediate representation (IR) can be multi-level, and each subsequent compilation step generates a level of IR.

[0078] Preferably, the AI ​​compiler model compilation multi-level performance evaluation system also includes a pruning module 500, which is used to generate a number of candidate solutions for each operator and then filter out candidate solutions whose performance does not meet the requirements according to hardware information.

[0079] Preferably, the AI ​​compiler model compilation multi-level performance evaluation system also includes a matching module 600, which is used to match the description information of the original operator with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated, and if the match fails, it is directly skipped. Since the operators in the operator library are all compiled in advance, the description information of the original operator is matched with the operator in the existing operator library. If the match is successful, the matched operator and the corresponding instruction information can be used as a candidate solution, and the N candidate solutions screened in step S3 can be entered into the next level for more accurate evaluation. In this way, the advantages and disadvantages of the operators in the operator library and other candidate solutions can be directly evaluated during the compilation process, solving the selection problem of multiple backend implementations.

[0080] Based on the same inventive concept, an embodiment of the present invention also proposes a readable storage medium on which a computer program is stored. When the computer program is executed, it can implement the multi-level performance evaluation method for compiling an AI compiler model as described above.

[0081] The readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device, such as but not limited to an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof. The more specific example (non-exhaustive list) of the readable storage medium includes: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer program described herein can be downloaded from the readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer program from the network and forwards the computer program for storage in the readable storage medium in each computing / processing device. The computer program for performing the operation of the present invention can be a run instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by utilizing the state information of a computer program to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions to implement various aspects of the present invention.

[0082] Here, various aspects of the present invention are described with reference to the flowchart and / or block diagram of the method, system and computer program product according to the embodiment of the present invention. It should be understood that each square frame of the flowchart and / or block diagram and the combination of the square frames in the flowchart and / or block diagram can be realized by a computer program. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so as to produce a machine, so that these programs, when executed by the processor of a computer or other programmable data processing device, produce a device for realizing the function / action specified in one or more square frames in the flowchart and / or block diagram. These computer programs can also be stored in a readable storage medium, and these computer programs make the computer, programmable data processing device and / or other equipment work in a specific way, so that the readable storage medium storing the computer program includes a manufactured product, which includes instructions for realizing various aspects of the function / action specified in one or more square frames in the flowchart and / or block diagram.

[0083] The computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0084] In summary, the embodiments of the present invention provide a multi-level performance evaluation method and system for AI compiler model compilation, which passes the partial optimal solutions selected level by level to the next level for more accurate evaluation, and can fully utilize the efficiency based on IR-level evaluation, and can fully utilize the accuracy based on instruction-level evaluation and actual measurement-level evaluation, and can find the optimal operator automatic generation method on the basis of greatly reducing the compilation time. Compared with the IR-level or instruction-level time evaluation, the accuracy is higher. Compared with the instruction-level or actual measurement-based evaluation method, it can not only greatly reduce the compilation time, but also effectively solve the selection problem of multiple back-end implementations. Therefore, the multi-level performance evaluation method and system for AI compiler model compilation provided by the embodiments of the present invention can significantly improve the efficiency of model compilation deployment and function and performance verification, thereby improving production efficiency.

[0085] The above description is only a description of the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any changes and modifications made by a person skilled in the art in the field of the present invention based on the above disclosure are within the scope of protection of the present invention. Obviously, a person skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-level performance evaluation method for AI compiler model compilation, characterized in that: include: Use the AI ​​compiler to generate several candidate solutions for each operator, each with a different intermediate expression; Perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer; Compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions, where N is a positive integer and is less than the M; The N candidates are decompiled to generate binary instructions and imported into the chip for actual measurement, and the optimal running time and corresponding running instructions are obtained according to the actual measurement results.

2. The AI ​​compiler model compilation multi-level performance evaluation method according to claim 1, characterized in that: The candidate solutions include splitting of the operator into different storage levels in different dimensions and pipeline settings of the different storage levels.

3. The AI ​​compiler model compilation multi-level performance evaluation method according to claim 1, characterized in that: After generating several candidate solutions for each operator, candidate solutions that do not meet the performance requirements are filtered out based on hardware information.

4. The AI ​​compiler model compilation multi-level performance evaluation method according to claim 1, characterized in that: Before decompilation of the N candidates to generate binary instructions, the AI ​​compiler model compilation multi-level performance evaluation method further includes: According to the description information of the original operator, it is matched with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated. If the match fails, it is directly skipped.

5. The AI ​​compiler model compilation multi-level performance evaluation method according to claim 1, characterized in that: For a network segment including multiple operators, a time evaluation is performed for each operator in the network segment, and the optimal running time and corresponding running instructions of each operator are obtained according to the fusion benefit to complete the compilation process.

6. The AI ​​compiler model compilation multi-level performance evaluation method according to claim 1, characterized in that: The M is one tenth of all candidate solutions, and the N is one tenth of the M.

7. A multi-level performance evaluation system for AI compiler model compilation, characterized in that: include: AI compiler, used to generate several candidate solutions for each operator, each candidate solution has a different intermediate expression; The IR level evaluation module is used to perform time evaluation on the intermediate expressions corresponding to all candidate solutions and select the best M candidate solutions, where M is a positive integer. An instruction-level evaluation module, used to compile the intermediate expressions corresponding to the M candidate solutions to generate corresponding assembly instructions and perform time evaluation, and select the best N candidate solutions therefrom, where N is a positive integer and is less than the M; The actual measurement level evaluation module is used to decompile the N candidates to generate binary instructions and import them into the chip for actual measurement, and obtain the optimal running time and corresponding running instructions according to the actual measurement results.

8. The AI ​​compiler model compilation multi-level performance evaluation system according to claim 7, characterized in that: The AI ​​compiler model compilation multi-level performance evaluation system also includes a pruning module, which is used to generate several candidate solutions for each operator and filter out candidate solutions whose performance does not meet the requirements according to hardware information.

9. The AI ​​compiler model compilation multi-level performance evaluation system according to claim 7, characterized in that: The AI ​​compiler model compilation multi-level performance evaluation system also includes a matching module, which is used to match the description information of the original operator with the existing operators in the existing operator library. If the match is successful, one more candidate solution is generated, and if the match fails, it is directly skipped.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, it can implement the AI ​​compiler model compilation multi-level performance evaluation method according to any one of claims 1-6.

Images