A GPU performance evaluation method for deep learning model deployment

Abstract

The present application relates to the field of GPU performance evaluation, and discloses a GPU performance evaluation method for deep learning model deployment, wherein a T-Profiler framework is proposed, two-stage analysis technology, namely "kernel function category analysis" and "kernel function aggregation analysis", is introduced, and the multi-level performance of the model after deployment can be obtained semi-automatically; through the two-stage design, the evaluation tool can be adapted to various deep learning compilers and various target hardware; experiments show that, compared with the original evaluation system, the T-Profiler framework can adapt to various application scenarios with low code, and the performance bottleneck and module performance of the tested model can be quickly statistically analyzed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of GPU performance evaluation, and particularly relates to a GPU performance evaluation method for deep learning model deployment. BACKGROUND

[0002] With the rapid development of deep learning algorithms, many new algorithm models with excellent precision performance have huge parameter quantity and computation quantity. When deployed to actual application scenarios of hardware (usually GPU) platforms, there are challenges such as long computation time delay, limited computation and storage resources, and the like, which are difficult to meet the requirements of real-time performance and resources in actual application scenarios, and limit the landing application of deep learning models. In order to cope with the above challenges, the industry usually uses deep learning compilers to realize the optimization of models on GPU platforms, in order to realize fast and efficient model deployment.

[0003] Deep learning compilers generally use automatic or semi-automatic methods to complete the generation of target code from deep learning computation graphs to various underlying hardware, thereby reducing the cost of computation graph deployment, operator development, and the like. The compilation and optimization of the computation graph usually uses optimization strategies such as complex operator splitting, operator fusion, and kernel function selection based on input shape. For example, complex operators such as SoftMax and LayerNorm are usually split into a combination of a set of algebraically equivalent basic operators. The computation graph composed of these basic operators is further divided into multiple subgraphs according to the operator fusion strategy and the shape of the input, and each subgraph is executed by a kernel function generated by the compiler or a called operator library.

[0004] The above-mentioned compilation and optimization strategies usually disrupt the original structure of the deep learning model, resulting in a non-one-to-one correspondence between the kernel functions actually executed on the GPU and the operators in the original model. When deploying the model generated by the compiler, if the model inference speed does not meet the expectations, it is necessary to collect the computation time delay of the model execution and further locate the model performance bottleneck. In addition, a single deep learning model is usually composed of multiple same or different operation modules, each module being composed of a group of operators. When designing a hardware-friendly algorithm, it is necessary to replace the modules according to the computation time delay of different modules.

[0005] The existing related performance evaluation tools mainly have two types, one type is the embedded evaluation tool of deep learning inference framework, such as PyTorch Profiler, etc., the evaluator inserts time counting before and after the execution of the operator through code instrumentation to obtain the calculation delay of the operator. This kind of tool is bound to the framework, and cannot be applied to other deep learning compiler deployed models. Another type is the evaluator provided by the hardware manufacturer, such as Nsight System of Nvdia, etc., which obtains the calculation delay of the kernel function through the underlying driver, and can give the kernel function execution timeline and the performance data inside each kernel function. With these data, it is difficult to establish a corresponding relationship between the kernel function and the original model operator, and the calculation delay of each operator and different modules of the original model cannot be obtained.

[0006] In summary, for the performance evaluation of deep learning model deployment, there is currently no tool that can give the calculation delay of each operator and different modules in the original model, which is not conducive to the targeted optimization of the deployed test model by algorithm engineers on the target hardware. SUMMARY

[0007] To solve the above technical problems, the present application provides a GPU performance evaluation method for deep learning model deployment.

[0008] To solve the above technical problems, the present application adopts the following technical solutions:

[0009] A GPU performance evaluation method for deep learning model deployment, the deployment version of the deep learning model on the deep learning inference framework is called the test model, and the GPU performance evaluation method specifically includes the following steps:

[0010] Step 1: Obtain the kernel function execution timeline of the test model, and the kernel function execution timeline is related to the deep learning inference framework relied on by the test model;

[0011] Step 2, kernel function category analysis:

[0012] Define the name recognition rule of the basic kernel function, decompose and type identify the kernel function name in the kernel function execution timeline according to the name recognition rule, construct a basic kernel function symbol sequence irrelevant to GPU and deep learning inference framework, and save the corresponding kernel function name and calculation delay of the kernel function in each basic kernel function symbol of the basic kernel function symbol sequence;

[0013] Step 3, kernel function aggregation analysis:

[0014] According to the structure description of the measured model and the aggregation rule, each basic kernel function is aggregated from bottom to top into a hierarchical performance structure tree according to the structure of "operator-module-encoding layer block"; wherein the operator includes a plurality of basic kernel functions, the module includes a plurality of operators, and the encoding layer block includes a plurality of modules.

[0015] Further, in the kernel function category analysis in step two, the following steps are included:

[0016] Data preprocessing: all kernel function names called in the measured model execution process obtained from the kernel function execution timeline are spliced together, and the line breaks are used to separate the kernel function names; the calculation delay of the kernel function is saved in an array, and the array index corresponding to each calculation delay value is uniquely determined by the line number of the kernel function name, and the obtained array is recorded as the calculation delay array;

[0017] Kernel function name recognition: define the name recognition rule of the kernel function, and the name recognition rule includes the basic kernel function regular expression capable of generating the basic kernel function corresponding symbol from the kernel function name; in the kernel function category analysis, all kernel function names satisfying the basic kernel function regular expression are returned to a basic kernel function corresponding symbol, and the line number of the corresponding basic kernel function is also saved; all basic kernel function corresponding symbols decomposed from the kernel function names in the kernel function execution timeline are saved in a symbol sequence in order, and the basic kernel function symbol sequence is obtained.

[0018] Further, the operator, the module and the encoding layer block are collectively referred to as the model structure, the basic kernel function is the substructure of the operator, the operator is the substructure of the module, and the module is the substructure of the encoding layer block.

[0019] In the kernel function aggregation analysis in step three, the following steps are included:

[0020] Kernel function aggregation: the basic kernel functions are aggregated into the model structure tree according to the hierarchy of "operator-module-encoding layer block";

[0021] Model structure delay statistics: the model structure tree is postorder traversed, and for any model structure A in the traversal process: the calculation delay of all substructures of the model structure A is first calculated, if there is no operator fusion phenomenon between each substructure, that is, the line numbers of the basic kernel functions included in any two substructures are different, then the calculation delay of the model structure A is the sum of the calculation delays of all substructures of the model structure A; if there is an operator fusion phenomenon between any two substructures, the calculation delay of the model structure A is the sum of the calculation delays of all substructures of the model structure A minus the calculation delay corresponding to the basic kernel functions with the same line number.

[0022] Data output: for each model structure on the model structure tree, a model structure performance record is generated to obtain the performance structure tree; the model structure performance record includes the name of the model structure, the path from the root of the model structure tree to the model structure, the model structure information description, the normal form of the model structure and the calculation delay data of the model structure.

[0023] Further, the model structure name is the type of the model structure; the path from the root of the model structure tree to the model structure is used to identify the position of the model structure in the measured model; the model structure information includes the serial number of all sub-structures of the model structure and the number of kernel functions; the normal form of the model structure is used to describe the reduction composition of the model structure; and the calculation delay data of the model structure contains the actual execution calculation delay of the model structure and the calculation delay of the operator execution.

[0024] Compared with the prior art, the beneficial technical effects of the present application are:

[0025] Deep learning has been widely applied in computer vision, natural language processing, business recommendation and other fields. For deep learning model deployment, existing evaluation tools cannot obtain the execution performance of the original model operators and modules after the model is deployed by a deep learning compiler. The present application proposes a T-Profiler framework, which introduces two-stage analysis technology, i.e. "kernel function category analysis" and "kernel function aggregation analysis", to semi-automatically obtain the multi-level performance of the model after deployment. Through the two-stage design, the evaluation tool can be adapted to multiple deep learning compilers and multiple target hardware. Experiments show that compared with the original evaluation system, the T-Profiler framework can adapt to multiple application scenarios with low code, and realize rapid statistical analysis of the performance bottleneck and module performance of the measured model. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The figure is the architecture diagram of the T-Profiler framework in the present application;

[0027] Figure 2 The figure is the visualization diagram of the T-Profiler framework evaluating the performance of the Swin Transformer deployment model module in the present application;

[0028] Figure 3 The figure is the visualization diagram of the T-Profiler framework evaluating the performance of the Swin Transformer deployment model module in the present application; DETAILED DESCRIPTION

[0029] A preferred embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0030] The deep learning model before deployment is called the original model, and the version after deployment is called the measured model.

[0031] As Figure 1 shown, the GPU performance evaluation method based on the T-Profiler framework in the present application mainly includes three parts: evaluation executor, kernel function category analysis, and kernel function aggregation analysis.

[0032] The evaluation executor calls an existing general GPU performance evaluation tool to obtain the kernel function execution timeline of the measured model under the current deep learning inference framework (such as TensorRT, TVM, etc.), wherein the kernel function is related to the deep learning inference framework relied on by the measured model.

[0033] The kernel function category analysis decomposes and identifies the kernel function name in the kernel function execution timeline output by the evaluation executor according to the configured basic kernel function name identification rule, constructs a basic kernel function notation sequence independent of hardware and deep learning inference framework, and saves the corresponding kernel function name, calculation delay, etc. in the notation.

[0034] The kernel function aggregation analysis aggregates each basic kernel function from bottom to top into a hierarchical performance structure tree according to the structure description and aggregation rule of the measured model in the structure of “operator-module-encoding layer block”.

[0035] 1. Evaluation executor

[0036] The T-Profiler framework supports obtaining the kernel function execution timeline of the measured model from various evaluation tools, for example:

[0037] Through a general GPU-based evaluation tool such as Nsight System, and using the GUI interface to copy the table data of the kernel function execution timeline;

[0038] Through a deep learning inference framework-based evaluator such as PyTorch Profiler, the kernel function execution timeline of the measured model deployed in the deep learning inference framework is counted, and the [torch.profile.profiler.export_chrome_trace] method is used to export the json data of the kernel function execution timeline;

[0039] For NVIDIA GPU, the CUPTI evaluation interface is used to record the kernel function call timestamp.

[0040] 2. Kernel function category analysis

[0041] The input of the kernel function category analysis part is the kernel function execution timeline of the measured model output by the evaluation executor, and the output is a basic kernel function token sequence. When analyzing the kernel function category, first, the name recognition rule of the basic kernel function is defined, the corresponding kernel function category parser is generated for the name recognition rule through the ANTLR tool, and the type of each basic kernel function is obtained. Here, the type of the basic kernel function is consistent with the ONNX operator type, which can guarantee the completeness of the basic kernel function type and can support most deep learning models. The specific steps of kernel function category analysis are as follows:

[0042] Data preprocessing: used for preprocessing the kernel function execution timeline file of the input measured model; specifically, all kernel function names called in the execution process of the measured model are spliced together, and the kernel function names are separated by line feeds; the calculation delay of the kernel function is saved in an array, and the array index corresponding to each calculation delay value is uniquely determined by the line number of the kernel function name, and the obtained array is recorded as the calculation delay array.

[0043] Kernel function name recognition: there are fusion kernel functions in the kernel function, and the fusion kernel function is composed of multiple basic kernel functions; the name of the fusion kernel function is decomposed into the name of the basic kernel function by using lexical analysis; specifically, first, the name recognition rule of the kernel function is defined, and the name recognition rule describes how to generate the token corresponding to the basic kernel function from the kernel function name; during the analysis process, all kernel function names that meet the regular expression of the basic kernel function will return a corresponding token, and the line number of the corresponding basic kernel function will also be saved. After analysis, all kernel function names in the kernel function execution timeline are decomposed into the tokens of the basic kernel functions, which are saved in a token sequence in order, and the basic kernel function token sequence is obtained.

[0044] The basic kernel function token sequence obtained by the above analysis process is independent of the compiler and specific hardware.

[0045] 3. Kernel function aggregation analysis

[0046] The input of the kernel function aggregation analysis part is the basic kernel function token sequence and the calculation delay array of the kernel function, and the output is an abstract syntax tree, and each syntax node in the abstract syntax tree corresponds to a model structure, and the model structure is generally an operator, a module or a coding layer structure. The kernel function aggregation parser defines the aggregation rule of the basic kernel function according to the model structure of the measured model, and generates the aggregation parser through the ANTLR tool to execute the specific analysis process. According to the development characteristics of the current deep learning model structure, the model structure is positioned as three levels of operators, modules and coding layer blocks, so that the corresponding model structure description naturally contains the following three levels from bottom to top:

[0047] The first level is a basic kernel function sequence description of an operator: each operator is generally composed of one or more basic kernel functions, i.e., the basic kernel function is a substructure of the operator. Since the basic kernel functions such as "Add" and "Div" have multiple inputs and outputs, for a complex operator containing these basic kernel functions, the topological order of the execution of the internal basic kernel functions can not be unique. Therefore, when defining the basic kernel function structure of the operator, all possible topological orders of the execution of the basic kernel functions need to be enumerated.

[0048] The second level is a sequence description of the operators of a module: each module is generally composed of multiple operators, i.e., the operator is a substructure of the module. Similarly to the first level, when there are multiple inputs and outputs of an operator, there can be a case where the topological order of the execution of the operators of the module is not unique, and therefore all possible topological orders of the execution of the operators need to be enumerated.

[0049] The third level is a sequence description of the modules in the encoding layer block: with the development of the design level of deep learning algorithms, there are generally encoding layers and encoding blocks on the module in the deep learning model, the encoding layers and the encoding blocks are collectively referred to as the encoding layer block, and the encoding layer block is generally composed of several modules, i.e., the module is a substructure of the encoding layer block. Therefore, the structure of this level needs to be reasonably described. Similarly to the first level, all possible topological orders of the execution of the modules need to be enumerated according to the number of internal module inputs and outputs.

[0050] The specific aggregation and analysis steps are as follows:

[0051] The aggregation process of the kernel function name: according to the model structure described above, the basic kernel functions are aggregated into different model structures by using grammar analysis, and the entire model structure is organized into a tree structure. The input of this step is a sequence of basic kernel function symbols, and the output is a model structure tree (i.e., a syntax tree obtained by aggregation and analysis).

[0052] Delay statistics of the model structure: the input of this step is the model structure tree generated by the aggregation process and the calculation delay array of the kernel function, and the output is a sequence of model structure performance records. The present application uses postorder traversal on the model structure tree, and for any model structure, the calculation delay of all substructures is first calculated. If there is no operator fusion phenomenon between each substructure, i.e., the line numbers of the basic kernel functions included in any two substructures are different, the calculation delay of the model structure is the sum of the calculation delays of all substructures; otherwise, the calculation delay of the model structure is the sum of the calculation delays of the substructures minus the time delays corresponding to the basic kernel functions with the same line numbers.

[0053] Data output: Finally, for each model structure (i.e. all non-leaf nodes on the model structure tree), a model structure performance record is generated, and finally a performance structure tree is obtained. The format of the model structure performance record is as follows: <model structure name, path from the root of the model structure tree to the model structure, model structure information description, model structure regular expression, model structure computation delay data>. Among them, the model structure name is the type of the model structure, such as operator-module-encoding layer block; the path from the root of the model structure tree to the model structure outlines the position of the model structure in the measured model; the model structure information description gives the data and description information used for display and visualization of the model structure, including the sequence id of all substructure corresponding records, how many kernel functions are contained, the model structure regular expression retains which substructures and basic operators the model structure is reduced from, and the model structure delay data contains the actual execution (including the idle time of the operator execution gap caused by data preparation, memory allocation, etc. System overhead) and the computation delay of the operator execution.

[0054] The analysis process is only related to the model structure.

[0055] Embodiments

[0056] The T-Profiler framework structure is shown in Figure 1 The execution flow of the GPU performance evaluation method based on the T-Profiler framework is as follows:

[0057] Select the target deep learning compiler or deep learning computing system, and deploy the deployment version of the deep learning model according to the specified evaluation configuration parameters, that is, the measured model.

[0058] Call the evaluation executor to track the execution process of the measured model, and obtain the kernel function execution timeline;

[0059] Call the kernel function category analysis process, read the related rules in the given kernel function category description file, perform the analysis of the framework-related kernel function categories on the obtained kernel function execution timeline, and obtain the basic kernel function token sequence;

[0060] Call the kernel function aggregation analysis process, read the related rules in the given model structure description file, and perform the analysis of the framework-independent module category on the input basic kernel function token sequence to obtain the model hierarchical performance structure tree data.

[0061] Figure 1The English terms appearing in the middle are all special terms. Among them, PyTorch, ONNX, TensorRT, Faster Transformer in the model deployment version are the names of related deep learning systems, NVIDIANsight, VTune, Perf in the evaluation tool are the names of performance evaluation tools on GPU and CPU, dense, GEMM and other terms appearing in the kernel function category description represent the names of kernel functions in the deep learning model inference process, and Model, Encoder, MHA and other terms in the model structure definition represent the model structure.

[0062] Figure 2 The visualization results of the module latency of the Swin Transformer image classification model deployed based on TensorRT (TRT), Faster Transformer (FT) and TVM using the T-Profiler framework are shown. The T-Profiler framework can count the latency data of different modules according to the model structure description file of the deep learning model.

[0063] Figure 3 The visualization results of the operator latency of the Swin Transformer image classification model with different computing scales (Large: large scale, Base: general scale, Small: small scale, Tiny: very small scale) deployed based on TensorRT using the T-Profiler framework are shown. The T-Profiler framework can count the latency data of the operator according to the kernel function category description file of the deep learning model.

[0064] Existing tools for performance evaluation of the measured model inference process suitable for compilation optimization systems, such as NsightSystem, can only give kernel function execution timeline data. If you want to get evaluation results such as Figure 2 and Figure 3 , you generally need to classify and count according to artificial experience. This artificial analysis method is difficult to apply in the following scenarios:

[0065] To ensure evaluation stability, the model inference needs to be repeated several times, and the statistical analysis results need to be averaged;

[0066] In the iterative stage of deep learning model development, the model structure usually only has some minor changes, and after deployment, a lot of adjustments to the artificial analysis rules are still needed;

[0067] The size of the deep learning model is very large, and the number of kernel functions called during execution is huge, so the artificial analysis complexity is too large.

[0068] In contrast, the present application lists the regular expressions of the basic operators of the convolutional network and the common Transformer series model in TensorRT and the corresponding names of the TVM compiler, and realizes them as the basic kernel function name identification rules in the lexical description file. In addition, the present application lists the common complex operators such as LayerNorm, SoftMax, GELU and Patch Merging, the common operation modules such as Bottleneck, MLP and Attention, and the aggregation rules of the common encoding layers and encoding blocks such as Transformer Encoder and CNNBackbone, and realizes the overall aggregation rules of the common convolutional neural network and the Transformer series model based on the combination of these basic rules. Based on these rules, low-code adaptation of various target hardware, deep learning compilers and deep learning model structures can be realized, thereby greatly saving the time of manual analysis and improving the accuracy of statistical analysis.

[0069] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, but that the application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, and the scope of the application is defined by the appended claims rather than the above description, and it is intended to include all changes falling within the meaning and scope of the equivalent elements of the claims, and no figure reference in the claims should be considered as limiting the claims.

[0070] Furthermore, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A GPU performance evaluation method for deep learning model deployment, wherein a deployed version of a deep learning model on a deep learning inference framework is referred to as a tested model, and the GPU performance evaluation method specifically comprises the following steps: Step 1: obtaining a kernel function execution timeline of the tested model, wherein the kernel function execution timeline is related to a deep learning inference framework on which the tested model depends; Step 2: performing kernel function category analysis, including: data preprocessing: concatenating all kernel function names called in the execution process of the tested model obtained from the kernel function execution timeline, with line breaks between the kernel function names; saving the calculation delay of the kernel function in an array, with the array index corresponding to each calculation delay value being uniquely determined by the line number of the kernel function name, to obtain a calculation delay array; kernel function name identification: defining a name identification rule for the kernel function, which includes a basic kernel function regular expression capable of generating a basic kernel function corresponding symbol from the kernel function name; during kernel function category analysis, all kernel function names satisfying the basic kernel function regular expression are returned with a basic kernel function corresponding symbol, and the line number of the corresponding basic kernel function is also saved; the basic kernel function corresponding symbols decomposed from all kernel function names in the kernel function execution timeline are saved in a symbol sequence in order, to obtain a basic kernel function symbol sequence; Step 3: performing kernel function aggregation analysis, including: kernel function name aggregation: aggregating the basic kernel functions into a model structure tree according to the hierarchy of "operator-module-encoding layer block", wherein the operator includes multiple basic kernel functions, the module includes multiple operators, and the encoding layer block includes multiple modules; model structure delay statistics: performing post-order traversal on the model structure tree, and for any model structure A in the traversal process: first, statistics the calculation delay of all child structures of the model structure A; if there is no operator fusion phenomenon between any two child structures, i.e., the line numbers of the basic kernel functions included in any two child structures are different, then the calculation delay of the model structure A is the sum of the calculation delays of all child structures of the model structure A; if there is an operator fusion phenomenon between any two child structures, then the calculation delay of the model structure A is the sum of the calculation delays of all child structures of the model structure A minus the calculation delay of the basic kernel functions with the same line number; data output: generating a model structure performance record for each model structure on the model structure tree, to obtain a performance structure tree; the model structure performance record includes the name of the model structure, the path from the root of the model structure tree to the model structure, the model structure information description, the regular expression of the model structure, and the calculation delay data of the model structure.

2. The GPU performance evaluation method for deep learning model deployment according to claim 1, wherein, The operator, the module, and the encoding layer block are collectively referred to as the model structure, the basic kernel function is the substructure of the operator, the operator is the substructure of the module, and the module is the substructure of the encoding layer block.

3. The GPU performance evaluation method for deep learning model deployment according to claim 1, wherein: The model structure name is the type of the model structure; the path from the root of the model structure tree to the model structure is used to identify the position of the model structure in the tested model; the model structure information includes the sequence number and the number of kernel functions of all child structures of the model structure. The normal form of the model structure is used to describe the reduction composition of the model structure; and the calculation delay data of the model structure comprises a calculation delay actually executed by the model structure and a calculation delay executed by an operator. The normal form of the model structure is used to describe the reduction composition of the model structure; and the calculation delay data of the model structure comprises a calculation delay actually executed by the model structure and a calculation delay executed

Images