Compilation optimization method for deep learning models with some constant parameters

Abstract

The present invention relates to the field of deep learning technology, and discloses a compilation optimization method for a deep learning model whose some parameters are constants; the present invention solves the problem that it is difficult to fully utilize tensor shape API information for constant optimization in traditional compilation technology. Through external type inference and constant quantization replacement, the present invention can accurately identify and optimize constant parameters, and the optimized tiling parameters are embedded in the kernel function, which solves the problem of unnecessary computational overhead at runtime, and ultimately achieves the effect of improving the execution efficiency of the kernel function. The present invention effectively reduces redundant calculations and useless codes through optimization steps such as constant propagation, dead code deletion, and common subexpression optimization, solves the problem of low kernel function execution efficiency, and ultimately achieves the effect of accelerating the reasoning process of the deep learning model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a compilation optimization method for a deep learning model whose some parameters are constants. Background Art

[0002] In the training and reasoning process of deep learning models, the computational efficiency and performance optimization of operators are crucial. Dynamically shaped operators refer to operators with input or output tensors whose shapes are unknown at compile time. Traditional compilation technologies often face many challenges when dealing with such dynamically shaped operators. These challenges mainly stem from the difficulty for the compiler to fully understand and utilize the specific tensor shape information in the operator input and output tensors, thus limiting further exploration of the optimization space.

[0003] First, for the analysis and optimization of the operator host side (such as CPU) code, traditional compilation technology cannot use the static information in the tensor shape of the dynamic shape operator for fine-grained optimization. The host side code needs to call the tensor shape API to obtain the input and output tensor shapes of the operator, so as to calculate the optimal scheduling optimization (such as tiling size, etc.) when the kernel function is executed, and pass it to the device side code. When facing dynamic shape operators, some scheduling optimization parameters can be calculated using the static information in the shape. Traditional compilation technology ignores this point, resulting in the system needing to frequently calculate and pass these constant parameters that could have been determined at compile time during runtime, thereby increasing unnecessary computing overhead and reducing the execution efficiency of the overall system.

[0004] Secondly, the performance optimization of kernel functions executed on the device side (such as accelerators such as NPU) is also limited by the limitations of traditional compilers. Since the scheduling optimization strategy needs to be passed from the host side, traditional compilers often find it difficult to identify and utilize the constant information in kernel functions when processing them, making it impossible to deeply optimize kernel functions. For example, common compiler optimization techniques such as constant propagation, dead code removal, and common subexpression optimization are not ideal for kernel function optimization. This leads to a large number of redundant scalar calculations and useless code in kernel functions. Since the scalar computing power of the accelerator is poor compared to the vector and matrix computing power, the execution efficiency of the kernel function will be significantly reduced.

[0005] In summary, existing compilation technologies have obvious deficiencies in dynamic shape operators, especially in the use of static information in the tensor shape API on the host side and constant optimization of kernel functions on the device side. These deficiencies limit the performance of dynamic shape deep learning models in complex scenarios and hinder the further development and application of deep learning technology. Therefore, there is an urgent need for a new technology that can fully utilize the static information of tensor shapes and the constant information in scheduling optimization parameters for deep optimization, so as to improve the execution efficiency and performance of deep learning models. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a compilation optimization method for deep learning models whose some parameters are constants. The core idea is to collect constraint information between variables by analyzing the program data flow of the Host side code of the deep learning operator, determine the value information of the constant parameters of the Device side kernel function by symbolic solution, and introduce the determined constant parameter value information into the compilation process of the kernel function, thereby allowing the compiler to fully propagate and optimize the kernel function and improve the execution efficiency of the kernel function.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A compilation optimization method for a deep learning model with some parameters being constants, comprising:

[0009] Host-side code analysis and constant replacement, including:

[0010] External type inference: In the host side code of the deep learning model, the shape of the input tensor is queried through the tensor shape API. Through external type inference technology, it is inferred whether the call point of each tensor shape API returns a constant value. The call point of the tensor shape API that is inferred to return a constant value is replaced with a constant value, that is, the constant value is directly used instead of the return value of the tensor shape API.

[0011] Constant propagation and optimization: After constant replacement is completed, constant propagation, dead code deletion, and common sub-expression optimization are performed on the host side code. In the optimized host side code, a certain constant tiling parameter and its corresponding value are generated, and the constant tiling parameter and its corresponding value are saved in the kernel function on the device side.

[0012] Device-side code constant optimization, including:

[0013] Kernel function optimization: During the device-side code compilation process, the kernel function is compiled and optimized using the constant tiling parameters and corresponding values ​​passed from the host side, including constant propagation, dead code removal, and common sub-expression optimization.

[0014] Optimized kernel function execution: The optimized device-side code is compiled into a binary package, and the kernel function in the optimized device-side code is called at runtime to improve the compilation efficiency of the deep learning model.

[0015] Compared with the prior art, the beneficial technical effects of the present invention are:

[0016] (1) Code analysis of host-side calculation of tiling parameters: By performing program data flow analysis on the host-side code of the operator, the present invention solves the problem that it is difficult to fully utilize tensor shape API information for constant optimization in traditional compilation technology. Through external type inference and constant replacement, the present invention can accurately identify and optimize constant parameters. The optimized tiling parameters are embedded in the kernel function, solving the problem of unnecessary computational overhead at runtime, and ultimately achieving the effect of improving the execution efficiency of the kernel function.

[0017] (2) Device-side code constant optimization: By adopting constant optimization technology for device-side kernel functions, the present invention solves the problem that traditional compilers are difficult to fully utilize constant information to optimize the kernel function execution path. This method effectively reduces redundant calculations and useless code through optimization steps such as constant propagation, dead code deletion, and common subexpression optimization, solves the problem of low kernel function execution efficiency, and ultimately achieves the effect of accelerating the inference process of deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of a deep learning framework and the compilation optimization method of the present invention. DETAILED DESCRIPTION

[0019] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0020] 1. Definitions of concepts related to the present invention:

[0021] (1) Kernel function tiling parameters: refers to multiple runtime parameters involved in the operator execution process. The calculation logic of these tiling parameters is defined in the host side code and determined according to the runtime shape. These parameters include but are not limited to:

[0022] Loop block size: refers to the size of the data block processed each time during the calculation process to optimize the efficiency of parallel computing.

[0023] Data computation format: refers to how data is arranged or stored during computation, such as whether a specific matrix multiplication format or data compression form is used.

[0024] Memory access buffer size: defines the buffer size used by the kernel function in memory access operations to balance memory bandwidth and computation requirements.

[0025] Number of pipeline stages: refers to the number of pipeline stages used by the kernel function during execution, which involves the decomposition and execution order of tasks on the GPU or other hardware to optimize the parallelism of computing and data transfer.

[0026] (2) Tensor shape API: It is a method used in deep learning frameworks to query or calculate the shape information of tensors in different dimensions. These APIs are usually called in the host side code to obtain the dimension size of the input data and determine the tiling parameters of the kernel function based on the dimension size of the input data.

[0027] 2.Host side code analysis and constant replacement:

[0028] External type inference: In the host side code of the deep learning model, the shape information of the input tensor is queried through the tensor shape API. Through external type inference technology, the call point of each tensor shape API is analyzed to determine whether it returns a constant value. For API calls inferred as constant types, these call points are replaced with constants, that is, the constant value is directly used instead of the API return value.

[0029] Constant propagation and optimization: After constant replacement is completed, further data flow analysis and optimization operations are performed. Specific steps include but are not limited to constant propagation (further propagating known constant values ​​in the code), dead code removal (removing redundant code caused by constants), and common subexpression optimization (reducing repeated calculations) and other data flow optimizations. Ultimately, after these optimizations, the Host-side code will generate certain constant tiling parameters and their values, and this information will be saved in the kernel function on the Device side.

[0030] 3. Constant optimization of device side code:

[0031] Kernel function optimization: During the code compilation process on the device side, a series of compilation optimization operations are performed on the kernel function using the constant tiling parameter values ​​passed down from the host side. These operations include but are not limited to constant propagation, dead code removal, common sub-expression optimization and other data flow optimizations.

[0032] Optimized kernel function execution: The device-side code that has undergone the above optimization processing is compiled into an efficient binary package. These optimized kernel functions are called at runtime, which improves the execution efficiency of the entire deep learning model.

[0033] Example:

[0034] For example, BERT, a classic model in the field of natural language processing, needs to achieve the highest possible average inference speed while processing various possible input sequence lengths. In implementation, each operator in BERT needs to calculate the Tiling parameters required by the kernel function on the Host side and pass them to the kernel function on the Device side. The tensor shape of the model input is defined as (batch size, sequence length, word embedding length). Although only the sequence length cannot be determined, in theory some Tiling parameters can be calculated from the sizes of the two dimensions of the determined batch size and word embedding length, but all Tiling parameters are still treated as variables, which results in the loss of a lot of constant information when compiling the kernel function.

[0035] For the above case, the present invention analyzes and replaces the static information of the shape API of the Host side code with constants, determines the value of the constant Tiling parameter, and further introduces it into the optimization process of the kernel function, thereby achieving the purpose of optimizing the execution efficiency of the Host side code and the Device side kernel function.

[0036] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0037] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A compilation optimization method for a deep learning model with some parameters being constants, characterized in that: include: Host-side code analysis and constant replacement, including: External type inference: In the host side code of the deep learning model, the shape of the input tensor is obtained through the tensor shape API query. Through the external type inference technology, it is inferred whether the call point of each tensor shape API returns a constant value. The call point of the tensor shape API that is inferred to return a constant value is replaced with a constant value, that is, the constant value is directly used instead of the return value of the tensor shape API. Constant propagation and optimization: After constant replacement is completed, constant propagation, dead code deletion, and common sub-expression optimization are performed on the host side code. In the optimized host side code, a certain constant tiling parameter and its corresponding value are generated, and the constant tiling parameter and its corresponding value are saved in the kernel function on the device side. Device-side code constant optimization, including: Kernel function optimization: During the device-side code compilation process, the kernel function is compiled and optimized using the constant tiling parameters and corresponding values ​​passed from the host side, including constant propagation, dead code removal, and common sub-expression optimization. Optimized kernel function execution: The optimized device-side code is compiled into a binary package, and the kernel function in the optimized device-side code is called at runtime to improve the compilation efficiency of the deep learning model.

Images