A cross-platform tensor program performance prediction method

Abstract

The present invention discloses a cross-platform tensor program performance prediction method, which relates to the technical field of deep learning compilation optimization, and comprises: defining the overall execution cost of a tensor program; when the tensor program is not executed on a source platform, directly extracting static program features that are independent of hardware from the LLVM IR of the tensor program; when the tensor program is executed on the source platform, extracting dynamic program features that are independent of hardware; based on a reuse profile of the tensor program and cache parameters of a target platform, calculating the memory access instruction cost, and obtaining the computing instruction cost by executing a micro-benchmark test of target hardware on the target platform; inputting the task graph, the memory access instruction cost and the computing instruction cost of each basic block into a performance predictor, and obtaining the predicted execution cost of the tensor program on the target platform; the tensor program performance prediction method avoids the huge overhead of performance measurement on the target platform.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning compilation optimization, and in particular to a cross-platform tensor program performance prediction method. Background Art

[0002] Deep learning has achieved unprecedented success in many fields, but its huge amount of computation also brings great challenges to the implementation of the model. How to make full use of existing computing resources, maximize the inference speed of the model, and improve the real-time performance of the model in the actual production environment is one of the hot issues in the current development of deep learning.

[0003] Empirical search methods are widely used to optimize deep learning reasoning, and typical representative works include template-based autoTVM and template-independent Ansor. These methods generate many equivalent tensor programs through certain strategies, then evaluate their performance and finally select the tensor program with the best performance.

[0004] In recent years, using learning-based methods to predict tensor program performance has become increasingly popular. Although this method can simplify the development of performance prediction models and achieve high prediction accuracy, it relies on large-scale performance data (millions) of target hardware. Since programs and hardware on different platforms are bound, cross-platform performance tests cannot be used interchangeably, which increases the cost of tensor program performance prediction.

[0005] Some existing technologies use transfer learning methods to alleviate this problem, such as Tenset, TLP, Moss, etc., but they still require a considerable amount of performance data (tens of thousands). The existing technology (the paper named "PPT-Multicore: performance prediction of OpenMP applications using reuse profiles and analytical modeling", authored by Atanu Barai et al.) proposed a framework called PPT-Multicore to achieve performance prediction for OpenMP parallel programs. This framework is not suitable for tensor programs generated by deep learning compilers, and does not consider the scenario of predicting program performance across platforms. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a cross-platform tensor program performance prediction method, which avoids the huge overhead of performance measurement on the target platform.

[0007] The present invention proposes a cross-platform tensor program performance prediction method, which realizes fast performance prediction of tensor programs by reusing hardware-independent program features in tensor programs on a target platform. The tensor program performance prediction method comprises the following steps:

[0008] Step 1: Define the overall execution cost Cost(P) of the tensor program P;

[0009]

[0010] Where n is the number of functions in the tensor program P, and m i is the number of basic blocks in the ith function, Cost(BB ij ) and ECount(BB ij ) represent the jth basic block BB in the i-th function respectively. ij The execution cost and effective execution times;

[0011] Step 2: When the tensor program P is not executed on the source platform, hardware-independent static program features are directly extracted from the LLVM IR of the tensor program P, where the static program features include a control flow graph CFG of each function and a task graph of each basic block;

[0012] Step 3: When executing the tensor program P on the source platform, extract dynamic program features that are independent of hardware, where the dynamic program features include the effective execution count of each basic block and the reuse profile of the tensor program, and add the effective execution count of each basic block to the control flow graph CFG of each function to improve the control flow graph CFG of each function;

[0013] Step 4: Based on the reuse profile of the tensor program and the cache parameters of the target platform, the memory access instruction cost is calculated, and the computing instruction cost is obtained by executing a microbenchmark test on the target hardware on the target platform;

[0014] Step 5: Input the task graph, memory access instruction cost and calculation instruction cost of each basic block into the pipeline simulator, predict the execution cost for each basic block, and obtain the predicted execution cost of the tensor program P on the target platform based on the predicted execution cost of each basic block and the improved control flow graph CFG of each function.

[0015] Furthermore, in step 1, the basic block BB ij The execution cost Cost(BB ij ) is defined as follows:

[0016] Cost(BB ij )=Pipeline(Tg(BB ij ), Cost(Insts))

[0017] Among them, Tg(BB ii ) represents the basic block BB ij The task graph describes the instructions in the basic block and the data dependencies between the instructions in the form of a task graph; Cost(Insts) represents the set of execution costs of various instructions on the target platform, and Pipeline represents the pipeline simulator, which calculates the execution cost of a given basic block based on the input task graph and the execution cost of each instruction.

[0018] Furthermore, in step 1, the basic block BB ij The effective execution times ECount(BB ij ) is defined as follows:

[0019]

[0020] Among them, Count(F i ) represents the function F i The number of executions, Count(BB ij ) indicates that each time the function F is executed i Basic Block BB ij The number of executions of threads indicates that if the function F i is the number of threads used for parallel execution, ifF i parallel means that if the function F i is executed in parallel, ifF i serial means if the function F i It is executed serially.

[0021] Furthermore, in step 2, the generation process of static program features is as follows:

[0022] Perform control flow analysis on the LLVM IR of the tensor program P to obtain the control flow graph CFG of each function;

[0023] Perform data dependency analysis on the LLVM IR of the tensor program P to obtain the task graph of each basic block.

[0024] Furthermore, the control flow analysis is performed on the LLVM IR of the tensor program P to obtain the control flow graph CFG of each function, which specifically includes:

[0025] Traverse each function in the LLVM IR of the tensor program P;

[0026] For each function, traverse each basic block in it, record the relationship between the current basic block and the predecessor basic block and the successor basic block in the form of a graph, and obtain the control flow graph CFG of each function.

[0027] Furthermore, data dependency analysis is performed on the LLVM IR of the tensor program P to obtain the task graph of each basic block, which specifically includes:

[0028] Traverse each instruction in each basic block, and for each instruction, obtain its output variable;

[0029] Check all subsequent instructions of the current instruction. If the output variable is an operand of the subsequent instruction, add the subsequent instruction to the child list of the current instruction.

[0030] Repeat the above traversal process. After the traversal is completed, the relationship between each instruction and each instruction in its child list is recorded in the form of a graph to obtain the task graph of each basic block.

[0031] Furthermore, in step three, the generation process of dynamic program features is as follows:

[0032] Based on the reuse tracker, the tensor program P is tracked during execution to obtain the reuse profile of the tensor program;

[0033] The effective execution count of each basic block is obtained by counting instrumentation while the tensor program P is running.

[0034] Furthermore, in step 4, the memory access instruction cost is generated as follows: the reuse profile of the tensor program and the cache parameters of the target platform are input into the memory analysis unit to calculate the memory access instruction cost.

[0035] Furthermore, in step 5, the predicted execution cost of each basic block and the predicted execution cost of the tensor program P on the target platform are generated as follows:

[0036] Based on the task graph, memory access instruction cost and calculation instruction cost of each basic block, a pipeline simulator is used to predict the execution cost of each basic block;

[0037] The predicted execution cost of the tensor program P is obtained based on the predicted execution cost of each basic block, the control flow graph CFG annotated with the effective execution count of each basic block, and the number of available threads on the target platform.

[0038] The advantages of a cross-platform tensor program performance prediction method provided by the present invention are: the core is to decouple program features from hardware features, and reuse hardware-independent program features on multiple platforms to achieve fast tensor program performance prediction. Based on the given target platform parameterized hardware features, tensor program performance is predicted for the target platform by combining pre-collected hardware-independent tensor program features, avoiding the huge overhead of performance measurement on the target platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0040] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0041] like Figure 1 As shown, a cross-platform tensor program performance prediction method proposed in the present invention realizes fast performance prediction of tensor programs based on reusing hardware-independent program features in tensor programs on a target platform. The tensor program performance prediction method includes the following steps:

[0042] Step 1: Define the overall execution cost Cost(P) of the tensor program P;

[0043]

[0044] Where n is the number of functions in the tensor program P, and m i is the number of basic blocks in the ith function, Cost(BB ij ) and ECount(BB ij ) represent the jth basic block BB in the i-th function respectively. ij The execution cost and effective execution times;

[0045] Formal definition of performance modeling: Determine the definition of the overall execution cost of the tensor program, the definition of the execution cost of the basic block, and the definition of the effective execution number of the basic block.

[0046] Basic Block BB ij The execution cost Cost(BB ij ) is defined as follows:

[0047] Cost(BB ij )=Pipeline(Tg(BB ij ), Cost(Insts))

[0048] Among them, Tg(BB ij ) represents the basic block BB ij The task graph describes the instructions in the basic block and the data dependencies between the instructions in the form of a task graph; Cost(Insts) represents the set of execution costs of various instructions on the target platform, and Pipeline represents the pipeline simulator, which calculates the execution cost of a given basic block based on the input task graph and the execution cost of each instruction.

[0049] In step 1, the basic block BB ij The effective execution times ECount(BB ij ) is defined as follows:

[0050]

[0051] Among them, Count(F i ) represents the function F i The number of executions, Count(BB ij ) indicates that each time the function F is executed i Basic Block BB ij The number of executions of threads indicates that if the function F i is the number of threads used for parallel execution, ifF i parallel means that if the function F i is executed in parallel, ifF i serial means if the function F i It is executed serially.

[0052] Step 2: When the tensor program P is not executed on the source platform, a static feature extractor is used to directly extract hardware-independent static program features from the LLVM IR of the tensor program P, where the static program features include a control flow graph CFG of each function and a task graph of each basic block;

[0053] Perform control flow analysis on the LLVM IR of the tensor program P to obtain the control flow graph CFG of each function, specifically: traverse each function in the LLVM IR of the tensor program P;

[0054] For each function, traverse each basic block in it, record the relationship between the current basic block and the predecessor basic block and the successor basic block in the form of a graph, and obtain the control flow graph CFG of each function.

[0055] Perform data dependency analysis on the LLVM IR of the tensor program P to obtain the task graph of each basic block. Specifically, traverse each instruction in each basic block, and for each instruction, obtain its output variable; check all subsequent instructions of the current instruction, and if the output variable is the operand of the subsequent instruction, add the subsequent instruction to the child list of the current instruction; repeat the above traversal process, and after the traversal is completed, record the relationship between each instruction and each instruction in its respective child list in the form of a graph to obtain the task graph of each basic block.

[0056] Step 3: When executing the tensor program P on the source platform, a dynamic feature extractor is used to extract dynamic program features that are independent of hardware. The dynamic program features include the effective execution count of each basic block and the reuse profile of the tensor program. The effective execution count of each basic block is added to the control flow graph CFG of each function to improve the control flow graph CFG of each function.

[0057] Based on the reuse tracker, the reuse profile of the tensor program P is obtained by tracking the execution of the tensor program, that is, using the existing memory tracking analysis tool to generate a reuse profile when the program is executed. The effective execution count of each basic block is obtained when the tensor program P is running by inserting a counting instrument, that is, before the program runs, a counting instrument is inserted in each basic block; when the program runs, the counting instrument counts the execution times of each basic block and writes it to a file.

[0058] Step 4: In the hardware feature extractor, the memory access instruction cost is calculated based on the reuse profile of the tensor program and the cache parameters of the target platform, and the computing instruction cost is obtained by executing the micro-benchmark test of the target hardware on the target platform;

[0059] The reuse profile of the tensor program and the cache parameters of the target platform are input into the memory analysis unit to calculate the memory access instruction cost, specifically: according to the cache size and cache associativity of the target platform, the cache hit rate is calculated for each reuse distance in the reuse profile; according to the frequency and cache hit rate of each reuse distance in the reuse profile, the cache hit rate of the entire tensor program is calculated; according to the cache hit rate of the entire tensor program and the memory access overhead of each level of storage structure of the target platform, the memory access instruction cost is calculated.

[0060] Step 5. In the performance predictor, the task graph, memory access instruction cost and calculation instruction cost of each basic block are input into the existing pipeline simulator to predict the execution cost for each basic block. Based on the predicted execution cost of each basic block and the improved control flow graph CFG of each function, the predicted execution cost of the tensor program P on the target platform is obtained.

[0061] The predicted execution cost of each basic block and the predicted execution cost of the tensor program P on the target platform are generated as follows: according to the task graph, memory access instruction cost and calculation instruction cost of each basic block, the execution cost of each basic block is predicted using a pipeline simulator; according to the predicted execution cost of each basic block, the control flow graph CFG annotated by the effective execution count of each basic block and the number of available threads on the target platform, the predicted execution cost of the tensor program P is obtained.

[0062] Through steps one to three: 1) Tensor program performance modeling: By decoupling program features from hardware features, tensor program performance is modeled, solving the problem of deep coupling between existing cost models and hardware platforms, and achieving fast tensor program performance prediction on the target platform. 2) General program feature representation: By extracting hardware-independent program features, program features can be directly extended to other hardware platforms without re-collection, solving the problem of high overhead in measuring tensor programs on the target platform, and achieving the effect of one-time collection and repeated use. 3) Parameterized hardware feature representation: By describing hardware in a parameterized way, the complexity of hardware modeling is simplified, and lightweight hardware modeling and performance prediction are achieved.

[0063] Therefore, the core of this embodiment is to decouple program features from hardware features and reuse hardware-independent program features on multiple platforms to achieve fast tensor program performance prediction. Based on the given target platform parameterized hardware features, tensor program performance is predicted for the target platform by combining pre-collected hardware-independent tensor program features, avoiding the huge overhead of performance measurement on the target platform.

[0064] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cross-platform tensor program performance prediction method, characterized in that: Based on reusing hardware-independent program features in tensor programs on a target platform, a fast performance prediction of tensor programs is achieved. The tensor program performance prediction method includes the following steps: Step 1: Define the overall execution cost Cost(P) of the tensor program P; Where n is the number of functions in the tensor program P, and m i is the number of basic blocks in the ith function, Cost(BB ij ) and ECount(BB ij ) represent the jth basic block BB in the i-th function respectively. ij The execution cost and effective execution times; Basic Block BB ij The effective execution times ECount(BB ij ) is defined as follows: Among them, Count(F i ) represents the function F i The number of executions, Conunt(BB ij ) indicates that each time the function F is executed i Basic Block BB ij The number of executions of threads indicates that if the function F i is the number of threads used for parallel execution, if F i parallel means that if the function F i is executed in parallel, if F i serial means that if the function F i It is executed serially; Step 2: When the tensor program P is not executed on the source platform, static program features that are independent of hardware are directly extracted from the LLVMIR of the tensor program P, where the static program features include a control flow graph CFG of each function and a task graph of each basic block; Step 3: When executing the tensor program P on the source platform, extract dynamic program features that are independent of hardware, where the dynamic program features include the effective execution count of each basic block and the reuse profile of the tensor program, and add the effective execution count of each basic block to the control flow graph CFG of each function to improve the control flow graph CFG of each function; Step 4: Based on the reuse profile of the tensor program and the cache parameters of the target platform, the memory access instruction cost is calculated, and the computing instruction cost is obtained by executing a microbenchmark test on the target hardware on the target platform; Step 5: Input the task graph, memory access instruction cost and calculation instruction cost of each basic block into the pipeline simulator, predict the execution cost for each basic block, and obtain the predicted execution cost of the tensor program P on the target platform based on the predicted execution cost of each basic block and the improved control flow graph CFG of each function.

2. The cross-platform tensor program performance prediction method according to claim 1, characterized in that: In step 1, the basic block BB ij The execution cost Cost(BB ij ) is defined as follows: Cost(BB ij )=Pipeline(Tg(BB ij ),Cost(Insts)) Among them, Tg(BB ij ) represents the basic block BB ij The task graph describes the instructions in the basic block and the data dependencies between the instructions in the form of a task graph; Cost(Insts) represents the set of execution costs of various instructions on the target platform, and Pipeline represents the pipeline simulator, which calculates the execution cost of a given basic block based on the input task graph and the execution cost of each instruction.

3. The cross-platform tensor program performance prediction method according to claim 1, characterized in that: In step 2, the generation process of static program features is as follows: Perform control flow analysis on the LLVM IR of the tensor program P to obtain the control flow graph CFG of each function; Perform data dependency analysis on the LLVM IR of the tensor program P to obtain the task graph of each basic block.

4. The cross-platform tensor program performance prediction method according to claim 3, characterized in that: When performing control flow analysis on the LLVM IR of the tensor program P, the control flow graph CFG of each function is obtained, which specifically includes: Iterate over each function in the LLVMIR of the tensor program P; For each function, traverse each basic block in it, record the relationship between the current basic block and the predecessor basic block and the successor basic block in the form of a graph, and obtain the control flow graph CFG of each function.

5. The cross-platform tensor program performance prediction method according to claim 3, characterized in that: Perform data dependency analysis on the LLVM IR of the tensor program P to obtain the task graph of each basic block, including: Traverse each instruction in each basic block, and for each instruction, obtain its output variable; Check all subsequent instructions of the current instruction. If the output variable is an operand of the subsequent instruction, add the subsequent instruction to the child list of the current instruction. Repeat the above traversal process. After the traversal is completed, the relationship between each instruction and each instruction in its child list is recorded in the form of a graph to obtain the task graph of each basic block.

6. The cross-platform tensor program performance prediction method according to claim 1, characterized in that: In step 3, the generation process of dynamic program features is as follows: Based on the reuse tracker, the tensor program P is tracked during execution to obtain the reuse profile of the tensor program; The effective execution count of each basic block is obtained by counting instrumentation while the tensor program P is running.

7. The cross-platform tensor program performance prediction method according to claim 1, characterized in that: In step 4, the memory access instruction cost is generated as follows: the reuse profile of the tensor program and the cache parameters of the target platform are input into the memory analysis unit to calculate the memory access instruction cost.

8. The cross-platform tensor program performance prediction method according to claim 1, characterized in that: In step 5, the predicted execution cost of each basic block and the predicted execution cost of the tensor program P on the target platform are generated as follows: Based on the task graph, memory access instruction cost and calculation instruction cost of each basic block, a pipeline simulator is used to predict the execution cost of each basic block; The predicted execution cost of the tensor program P is obtained based on the predicted execution cost of each basic block, the control flow graph CFG annotated with the effective execution count of each basic block, and the number of available threads on the target platform.

Images