Model Inference Optimization Method, Apparatus, Device, Medium, and Program Product

Abstract

The present application discloses a method, device, equipment, medium and program product for optimizing model inference, relating to the field of computer technology, including: obtaining the network characteristics of at least one network layer included in the target model; generating kernel code adapted to the at least one network layer according to the network characteristics; the kernel code represents the calculation method of the network layer in the current inference process of the target model; configuring kernel parameters based on the hardware resources of the target device to be used for inferring the target model; the kernel parameters refer to the relevant parameters for the execution of the kernel code on the target device; compiling the kernel code to generate an executable kernel file; and invoking the kernel file to perform inference on the target model. By generating kernel code adapted to each network layer based on the network characteristics and flexibly setting the kernel parameters according to the hardware resources, the technical problems of poor flexibility of fixed code and low utilization rate of hardware resources are solved, and the technical effects of improving the model inference efficiency and resource utilization rate are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a method, device, equipment, medium, and program product for optimizing model inference. Background Art

[0002] With the wide application of deep learning models, the performance of model inference has become one of the key bottlenecks. In related technologies, due to the large differences in the characteristics of different network layers (such as convolutional layers, fully connected layers, etc.), general optimization methods are difficult to fully meet the requirements of each network layer, resulting in problems such as insufficient utilization of Graphics Processing Unit (GPU) resources and limited computing speed. In addition, traditional methods usually rely on fixed kernel codes or general optimization strategies provided by frameworks, lacking flexibility and pertinence, and unable to fully exploit the potential of hardware. Therefore, there is an urgent need for a technology that can dynamically generate adapted kernel codes according to the characteristics of network layers to improve model inference efficiency and hardware resource utilization. Summary of the Invention

[0003] The present application provides a method, device, equipment, medium, and program product for optimizing model inference to at least solve the problems of poor flexibility of fixed codes and low utilization of hardware resources in related technologies.

[0004] The present application provides a method for optimizing model inference, including:

[0005] Obtaining the network characteristics of at least one network layer included in the target model;

[0006] Generating kernel codes adapted to at least one network layer according to the network characteristics; wherein, the kernel codes represent the calculation method of the network layer in the current inference process of the target model;

[0007] Configuring kernel parameters based on the hardware resources of the target device for which the target model is to be inferred; wherein, the kernel parameters refer to the relevant parameters for the execution of the kernel codes on the target device;

[0008] Compiling the kernel codes to generate executable kernel files;

[0009] Invoking the kernel files to perform inference on the target model.

[0010] The present application further provides a device for optimizing model inference, including:

[0011] An obtaining module, configured to obtain the network characteristics of at least one network layer included in the target model;

[0012] A kernel code generation module, configured to generate kernel codes adapted to at least one network layer according to the network characteristics; wherein, the kernel codes represent the calculation method of the network layer in the current inference process of the target model;

[0013] A kernel parameter configuration module, configured to configure kernel parameters based on the hardware resources of a target device for which target model inference is to be performed; wherein, the kernel parameters refer to the relevant parameters for the execution of kernel code on the target device.

[0014] A kernel code compilation module, configured to compile the kernel code to generate an executable kernel file.

[0015] A model inference module, configured to call the kernel file to perform inference on the target model.

[0016] This application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above model inference optimization methods when executing the computer program.

[0017] This application further provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any of the above model inference optimization methods when executed by a processor.

[0018] This application further provides a computer program product, including a computer program, and the computer program implements the steps of any of the above model inference optimization methods when executed by a processor.

[0019] Through this application, kernel code adapted to each network layer is generated based on the network characteristics of each network layer included in the model, and the kernel parameters in the kernel code are flexibly configured according to the hardware resources of the device for which model inference is performed. Therefore, the technical problems of poor flexibility of fixed code and low utilization rate of hardware resources can be solved, and the technical effects of improving model inference efficiency and resource utilization rate are achieved. Description of the Drawings

[0020] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of this application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic flowchart of a model inference optimization method provided by an embodiment of this application;

[0022] Figure 2 It is a schematic flowchart of a model inference optimization method provided by an embodiment of this application;

[0023] Figure 3 It is a schematic flowchart of a model inference optimization method provided by an embodiment of this application;

[0024] Figure 4 Flow schematic diagram of a model inference optimization method provided by an embodiment of the present application;

[0025] Figure 5 Flow schematic diagram of a model inference optimization method provided by an embodiment of the present application;

[0026] Figure 6 Flow schematic diagram of a model inference optimization device provided by an embodiment of the present application. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0028] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0030] Currently, with the wide application of deep learning models in fields such as image recognition, natural language processing, and autonomous driving, inference performance has become one of the key metrics. In particular, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer models are widely used in production environments, leading to a significant increase in their computational requirements. To address this, using high-performance CUDA (Compute Unified Device Architecture) kernels to accelerate neural network inference has become a common practice in the industry. CUDA is a parallel computing platform and programming model developed by NVIDIA that allows developers to utilize the powerful computing capabilities of GPUs (Graphics Processing Units) for general computing tasks through standard programming languages such as C, C++, and Python. However, traditional CUDA kernel codes are mostly designed in a general way and are difficult to be deeply optimized according to the characteristics of different network layers, resulting in problems such as insufficient GPU resource utilization and limited computing speed.

[0031] To address this problem, various optimization methods have been proposed, including manually adjusting loop unrolling factors, thread configurations, etc., in an attempt to improve performance by manually optimizing CUDA kernel codes. However, manual adjustment and optimization not only rely on developers' experience but also require a large amount of debugging time. At the same time, changes in model structure and data size often require re-adjusting parameters, resulting in high maintenance costs. Additionally, although automated CUDA optimization modules can significantly reduce the development difficulty, existing automatic optimizations mainly rely on compile-time static analysis and lack dynamic tuning means combined with network structure characteristics, making it difficult to handle complex and diverse network inference tasks.

[0032] Current research on the combination and optimization of CUDA kernels and deep learning models mainly falls into the following categories:

[0033] Compiler-level optimizations. Deep learning compilers such as TVM (Tensor Virtual Machine) and TensorRT optimize CUDA kernel codes by parsing model structures, optimizing operator combinations, and automatically selecting loop unrolling factors. In static optimizations, compilers can select and optimize kernel configurations based on hardware characteristics.

[0034] Manual kernel optimizations. Many deep learning frameworks allow developers to write their own CUDA kernel codes to achieve specific optimization goals. Developers can manually adjust loop unrolling factors, configure the number of threads, etc., according to the network layer structure.

[0035] Empirical-based automatic optimization modules: Some optimization modules attempt to combine empirical rules and specific hardware characteristics to automatically generate optimized configurations. For example, they select the degree of parallelism and unrolling factor based on heuristic methods or use known best practices for code generation.

[0036] However, the optimization process of such methods at the compiler level is usually carried out during the model compilation stage, lacking runtime feedback and unable to make dynamic adjustments to changes during execution (such as changes in input data distribution or hardware environment differences).

[0037] Manual kernel optimization: This method relies on programming experience, is cumbersome to operate, and requires manual adjustment every time the model or data changes, which not only affects efficiency but also increases maintenance costs.

[0038] The adaptability of empirical-based automatic optimization modules is limited and often unable to effectively adapt to the characteristics of different network layers and hardware, making it difficult to achieve true adaptive optimization.

[0039] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the model inference optimization method depends, the specific application environment architecture or specific hardware architecture is described here.

[0040] Embodiments of the present application provide a model inference optimization method, and the method is described in detail in combination with the execution process of the model inference optimization method. Specifically, see Figure 1 , Figure 1 is a schematic flowchart of a model inference optimization method provided by an embodiment of the present application, which specifically includes the following steps as Figure 1 shown:

[0041] S101. Obtain the network characteristics of at least one network layer included in the target model.

[0042] It can be understood that the target model refers to a deep learning model to be inferred or optimized for training. The target model can be any type of neural network model, such as a convolutional neural network, a recurrent neural network, etc. The target model includes at least one network layer, and these network layers are the basic computational units of the model, responsible for performing specific mathematical operations, such as convolutional layers, fully connected layers, pooling layers, normalization layers, etc.

[0043] It is understandable that by extracting the characteristic parameters of each network layer, the network characteristics of each network layer can be obtained. The network characteristics refer to the key attributes that describe the functions and behaviors of the network layer, and are used to characterize the role of the network layer in the model and its requirements for hardware resources. That is, the key attributes of specific network layers in the deep learning model are extracted. Among them, the network characteristics include the convolution kernel size, the number of channels, the type of activation function, etc. For example, for a convolutional layer, the obtained network characteristics include "layer type: convolutional layer, input dimension: 224×224×3, output dimension: 224×224×64, convolution kernel size: 3×3, number of filters: 64". The above characteristic parameters can be extracted from the metadata of the model or the actual data during the dynamic inference process, and then the extracted characteristic parameters are recorded to obtain the network characteristics, so as to provide data support for the subsequent generation of specialized kernel code.

[0044] Optionally, before obtaining the network characteristics of at least one network layer included in the target model, the method further includes:

[0045] Loading and initializing a pre-built kernel generation and optimization module; wherein, the kernel generation and optimization module is used to generate and optimize kernel code; configuring the environment parameters required by the kernel generation and optimization module; loading a compiler; wherein, the compiler is used to compile the kernel code.

[0046] It is understandable that before performing model inference optimization or obtaining network characteristics, the kernel generation and optimization module is loaded and initialized. The kernel generation and optimization module (hereinafter referred to as the module) is a pre-built module that can automatically generate and optimize CUDA kernel code. Specifically, this module is responsible for generating adapted kernel code according to the network characteristics and optimizing the generated kernel code to improve performance. In addition, the module can be an independently developed tool or a plug-in embedded in an existing deep learning framework. Among them, the module initialization process includes allocating memory resources, loading configuration files, and setting default parameters, etc. After initializing the module, configure the environment parameters required for the module to run. The environment parameters include target hardware information, task requirement information, and optimization strategies, etc., such as CUDA version, compiler path, temporary file storage directory, etc. Among them, the environment parameters determine the behavior and output results of the module. Subsequently, by dynamically adjusting the environment parameters, different hardware platforms and task requirements can be adapted. After completing the environment configuration of the module, load the CUDA compiler (such as nvcc or NVRTC) to ensure that the compilation environment is ready. The compiler is used to convert the generated kernel code into a binary file executable by the target device. Subsequently, run the module and load the initial configuration. The initial configuration is a predefined set of initial kernel parameters used to initially generate CUDA code.

[0047] Optionally, after completing the construction of the kernel generation and optimization module, the method further includes:

[0048] Modularize the kernel generation and optimization module to decouple it from the learning framework of the target model; generate the application programming interface of the kernel generation and optimization module to integrate the kernel generation and optimization module into the learning frameworks of other models through the application programming interface.

[0049] Understandably, after the construction of the module, through modular design, the module is separated from the learning framework of the target model, that is, the module is decoupled from the deep learning framework of the target model, making it an independent functional module. Subsequently, this module can run independently or be integrated into other models without relying on the deep learning framework of a specific model. Then, provide a set of standardized application programming interfaces (Application Programming Interface, API) for the module so that the deep learning frameworks of other models can directly call this module. Specifically, the application programming interface can provide a network feature extraction interface, a kernel code generation interface, and a compilation and execution interface, etc. Among them, the network feature extraction interface allows the external deep learning framework to transfer model structure information (network features), the kernel code generation interface can receive the network features and return the optimized kernel code, and the compilation and execution interface supports compiling the generated kernel code into an executable file for the model to call. This way of adopting modular design and API interface design enables this module to be seamlessly integrated into the existing deep learning framework, which can not only reduce the workload of developers but also continuously improve the inference performance through the kernel configuration caching mechanism.

[0050] S102. Generate kernel code adapted to at least one network layer according to the network features.

[0051] Among them, the kernel code represents the calculation method of the network layer in the current inference process of the target model.

[0052] Understandably, based on the above S101, CUDA or other GPU computing kernel codes adapted to each network layer are automatically generated according to the captured network characteristics to ensure that the generated kernel codes can adapt to the specific computing requirements of the network layer. Specifically, a template engine or code generation tool is used to dynamically generate kernel codes according to network characteristics; different code generation strategies are adopted for different types of network layers (such as convolutional layers, fully connected layers, etc.); the kernel codes contain specific computing logics and optimization strategies, such as thread allocation, memory access patterns, etc. Among them, the generated kernel codes not only implement the basic computing functions of the network layer, but also reflect the specific implementation details in the current inference process, that is, the kernel codes involve at least three aspects of content: computing logic, optimization strategy, and hardware strategy. The computing logic includes convolutional operations, matrix multiplication, etc., the optimization strategy includes parallelism setting, shared memory allocation, data chunking strategy, etc., and the hardware strategy can be configured in real time according to the hardware resources of the following target device.

[0053] Optionally, generating kernel codes adapted to at least one network layer according to network characteristics can be specifically implemented through the following steps:

[0054] Calculate the similarity between at least one network layer according to network characteristics, and determine at least one network layer pair according to the similarity; generate kernel codes adapted to any network layer according to the network characteristics of any network layer in at least one network layer pair; cache the kernel codes for reuse by the remaining network layers in at least one network layer pair except any network layer.

[0055] Understandably, before generating kernel codes adapted to each network layer, calculate the similarity between each network layer according to the analyzed network characteristics (such as input and output dimensions, convolutional kernel size, activation function type, etc.), that is, calculate the similar layers between network layers. For example, if the target model includes 3 convolutional layers, the 3 convolutional layers can be understood as similar layers, and the 3 convolutional layers can be defined as a network layer pair to generate shared kernel codes for the network layer pair. Specifically, distance measurement methods (such as Euclidean distance, cosine similarity) can be used to compare the numerical differences of network characteristics, and other possible methods will not be elaborated. After determining the network layer pair, for any network layer in the network layer pair, such as the first convolutional layer among the above 3 convolutional layers, generate adapted kernel codes based on its network characteristics to ensure that the generated kernel codes can adapt to the specific computing requirements of the first convolutional layer. Subsequently, store the generated kernel codes in the cache for reuse by other network layers in the same network layer pair. During subsequent inference processes, if the same network layer is encountered, directly load the already generated kernel codes from the cache. This way of directly reusing kernel codes for similar layers avoids repeated generation and compilation of kernel codes, significantly reducing the computing overhead. Subsequently, optimized configuration information can also be cached for direct reuse by similar layers to further improve the inference performance.

[0056] Optionally, the network characteristics include the tensor sizes of each network layer. Generating kernel code adapted to at least one network layer according to the network characteristics can be specifically implemented through the following steps:

[0057] Calculate and set the loop unrolling factor according to the tensor size; generate kernel code adapted to each network layer based on the loop unrolling factor.

[0058] It can be understood that the tensor size refers to the dimensional information of the input and output tensors of the network layer. For example, the input size (height, width, number of channels) of the convolutional layer or the length of the input vector of the fully connected layer, etc. The tensor size directly affects the scale of the computing task and the memory access pattern. Calculate an appropriate loop unrolling factor according to the tensor size. For example, a higher unrolling factor can be selected for larger tensors to reduce the loop overhead, and a lower unrolling factor can be selected for smaller tensors to avoid resource waste. Among them, the loop unrolling factor is an optimization technique that improves computing efficiency by reducing the number of loop iterations. By reasonably setting the loop unrolling factor, the execution efficiency of the kernel code can be optimized. Subsequently, using the calculated loop unrolling factor, generate kernel code optimized for a specific network layer. Specifically, the loop can be explicitly unrolled in the kernel code to reduce the execution overhead of branch instructions. This automatic and intelligent optimization method captures the Tensor (tensor) size and dynamically adjusts the loop unrolling factor, automatically generating and compiling the optimal CUDA kernel code, thereby significantly improving the efficiency of model inference.

[0059] S103. Configure kernel parameters based on the hardware resources of the target device for which the target model inference is to be performed.

[0060] Among them, the kernel parameters refer to the relevant parameters for the execution of the kernel code on the target device.

[0061] It is understandable that, based on the above S102, the hardware resources of the target device for which the target model inference is to be performed are obtained, such as the GPU architecture, the number of CUDA cores, the size of the shared memory, etc. Subsequently, according to the hardware resources and the extracted characteristic parameters, the kernel parameters such as the parallelism of the kernel and the shared memory configuration are automatically set, that is, the hardware policy in the above kernel code is set to ensure that the kernel code can make full use of the hardware characteristics of the target device and maximize the computing efficiency and resource utilization rate. Among them, the kernel parameters are specific settings that control the execution behavior of the kernel code on the target device and directly affect the performance. For example, for the convolutional layer, the appropriate thread block size and shared memory allocation policy are selected according to the hardware resources, the size of the convolutional kernel, and the number of channels. Among them, the thread block size defines the number of threads in each thread block, and usually values such as 256 and 512 are selected to adapt to the hardware parallel ability. The shared memory allocation allocates an appropriate shared memory size for each thread block, which can optimize the data access mode.

[0062] Optionally, the network characteristics include at least one operation executable by the network layer. After configuring the kernel parameters based on the hardware resources of the target device for which the target model inference is to be performed, the method further includes:

[0063] Adding at least one operation code corresponding to the at least one operation to the kernel code; wherein, the at least one operation includes an activation operation and a normalization operation.

[0064] It is understandable that the network characteristics not only include static information such as the tensor size, but also cover the specific operations that the network layer can perform (such as activation functions, normalization, etc.). Inserting code snippets related to specific operations (such as activation or normalization operations) into the generated kernel code to expand the functions of the kernel. Specifically, according to the operation type of the network layer, the corresponding operation code template is selected, and the operation code is embedded into the core computing logic of the kernel code to ensure its seamless integration, that is, common operations such as activation and normalization are fused into the kernel code. By fusing multiple operations, the multiple startup overheads of common operations can be effectively reduced, and the execution efficiency can be improved. For example, for the convolutional layer, the ReLU code and the normalization code are embedded after the convolutional operation to form the complete kernel code. Compared with the method of separately starting the ReLU code and the normalization code in the related technology (that is, starting at least 2 times), the method of fusing the ReLU code and the normalization code into the kernel code (that is, starting 1 time) not only reduces the number of startups, but also speeds up the computing speed.

[0065] Optionally, at least one network layer includes a convolutional layer, and the network characteristics include the size of the convolutional kernel, the number of input and output channels, and / or the type of activation function of the convolutional layer. Configuring the kernel parameters based on the hardware resources of the target device for which the target model inference is to be performed can be specifically implemented through the following steps:

[0066] Configure kernel parameters for the convolutional layer according to the network characteristics of the convolutional layer and the hardware resources of the target device for which the target model inference is to be performed; wherein, the kernel parameters include parallelism, thread block size, and / or shared memory size.

[0067] Understandably, dynamically adjusting the kernel parameters according to the network characteristics of the convolutional layer and the hardware resources of the target device to adapt to the inference task of the convolutional layer can maximize the utilization rate of the hardware resources of the target device and improve the inference efficiency of the convolutional layer.

[0068] S104. Compile the kernel code to generate an executable kernel file.

[0069] Understandably, based on S103 above, the generated kernel code is converted into an executable file adapted to the target device through a compiler, that is, the kernel code compiled using a compiler (such as nvcc) to generate an executable kernel file. Among them, the compilation process includes selecting compilation options adapted to the hardware architecture of the target device and applying optimization strategies. The kernel file generated after compilation is a binary file or intermediate representation file that can be directly loaded and executed by the target device, such as in the format of.cubin or.ptx.

[0070] S105. Invoke the kernel file to perform inference on the target model.

[0071] Understandably, based on S104 above, the compiled kernel file is loaded onto the target device (such as a GPU), and the parameters required for target model inference are set, including input data, output buffer, and thread block layout, etc. Subsequently, the inference task of the target model is completed by invoking the kernel function in the kernel file. This way of using the optimized kernel code to efficiently execute model inference effectively improves the computing performance and resource utilization rate.

[0072] Optionally, before invoking the kernel file to perform inference on the target model, the method further includes:

[0073] Modify the calculation method of the target model so that at least one network layer included in the target model invokes the corresponding kernel file during the current inference process.

[0074] Understandably, before invoking the kernel file for inference, modify the calculation method of the target model so that it invokes the generated kernel file during the inference process to replace the original calculation method. The new calculation method represented by the kernel file will ensure that the calculation of each network layer uses the optimized kernel code. Subsequently, during the actual inference process, the calculation of each network layer of the model will invoke the optimized kernel file. At this time, the inference performance is improved through the calculation method optimized by the generated and compiled kernel code.

[0075] Exemplarily, see Figure 2 ,Figure 2 The figure is a schematic flowchart of a model inference optimization method provided by an embodiment of the present application, which can be understood as a specialized kernel generation process based on the characteristics of network layers. Specifically:

[0076] 1) Capture the characteristics of network layers; 2) Automatically generate CUDA kernel code that highly matches the layer structure according to the network characteristics; 3) Automatically set the unrolling factor, parallelism, and shared memory configuration according to the hardware resources and network characteristics; 4) Incorporate operations such as activation and normalization into the kernel code; 5) Automatically generate CUDA kernel code and save it as a ".cu" file.

[0077] It can be understood that after capturing the characteristics of network layers, CUDA kernel code that highly matches the network layer structure is automatically generated. According to the characteristics of each network layer, parameters such as the loop unrolling factor, parallelism, and shared memory configuration are automatically set, and operations such as activation functions and normalization are incorporated into the kernel code, thereby reducing the overhead caused by multiple kernel launches. Through this specialized kernel generation method, adapted CUDA kernels can be obtained for different network layers, significantly improving the model inference efficiency. For other specific implementation descriptions, refer to the above embodiments and will not be elaborated here.

[0078] The model inference optimization method provided by the present application adopts a specialized kernel generation method, enabling each network layer operation to customize the best configuration according to its computational characteristics, improving the kernel execution efficiency, and reducing GPU resource waste. By automatically adjusting the loop unrolling factor and thread configuration, the kernel code can maximize the utilization of the GPU's computing power, improve the operation speed, and also reduce the development difficulty and cost, simplifying the kernel code writing and optimization. The modular design of kernel code generation and optimization can conveniently integrate the module into the existing deep learning framework. Subsequently, users do not need to deeply master the CUDA programming details and can achieve efficient inference optimized for network layer characteristics through simple API calls, which not only reduces the development threshold but also improves the applicability and generality of model inference.

[0079] Based on the above embodiments, Figure 3 The figure is a schematic flowchart of a model inference optimization method provided by an embodiment of the present application. After calling the kernel file to perform inference on the target model, the method further includes the following steps as Figure 3 shown below:

[0080] S301. Obtain the performance data of the target device during the current inference process of the target model.

[0081] Among them, the performance data includes execution time and / or memory occupancy data.

[0082] It is understandable that performance data is collected during model inference, for example, the execution time and memory occupancy of the kernel code are monitored in real time during inference, and the execution time and memory occupancy data are recorded layer by layer for each network layer. Among them, the execution time refers to the time required for the kernel code to run to completion on the target device and is used to evaluate the inference efficiency. The memory occupancy data refers to the memory usage of the target device, including video memory allocation, shared memory usage, etc., and is used to optimize resource management.

[0083] S302. Generate at least one execution policy configuration according to the performance data.

[0084] Among them, the execution policy configuration includes the configuration of parallelism and thread blocks.

[0085] It is understandable that, based on the above S301, at least one execution policy configuration is generated according to the collected performance data, and the execution policy configuration covers the parallelism configuration and / or the thread block configuration. Specifically, a neighborhood search strategy can be used to generate configuration combinations of different execution policies. Among them, the parallelism configuration refers to dynamically adjusting the granularity of task decomposition according to the execution time and hardware resource utilization rate in the performance data to optimize the allocation of computing resources. The thread block configuration refers to setting an appropriate thread block size and grid layout in combination with the memory occupancy data and the hardware characteristics of the target device (such as the number of CUDA cores, the capacity of shared memory, etc.) to improve the kernel execution efficiency.

[0086] S303. Determine the target execution policy configuration with the best execution effect among at least one execution policy configuration, and adjust the kernel parameters of the kernel code based on the target execution policy configuration.

[0087] It is understandable that, based on the above S302, by evaluating the generated multiple execution policy configurations, one with the best execution effect is selected as the target execution policy configuration. For example, an optimization algorithm can be used to screen out the configuration combination with the best execution effect. Subsequently, the kernel parameters in the target execution policy configuration are applied to the kernel code, and the kernel parameters are dynamically adjusted to adapt to the requirements of the inference task and improve the execution efficiency of the kernel code on the target device, such as adjusting the parallelism and the thread block size.

[0088] It is understandable that through a real-time performance feedback mechanism, the module can automatically select the kernel configuration with the best performance and cache these configurations for the next inference. This not only reduces the overhead of repeated optimization but also ensures that each inference can quickly reach the best performance state, achieving continuous performance improvement.

[0089] Optionally, after calling the kernel file to perform inference on the target model, the method further includes:

[0090] Adjust the kernel parameters of the kernel code according to the real-time hardware data of the target device; wherein, the hardware data includes the change data of the hardware environment and / or data distribution.

[0091] Understandably, in addition to tuning according to the monitored performance data, the kernel parameters can also be dynamically adjusted according to the change data of the hardware environment and / or data distribution to ensure that the inference efficiency is always in the best state. This way of optimizing according to the real-time changing data of the hardware can be understood as optimizing on the whole, which can not only make full use of the hardware resources, but also reduce the memory access conflicts.

[0092] Optionally, after the inference of the target model is completed, the method further includes:

[0093] Obtain the best kernel code after the target model completes the current inference process, and obtain the configuration parameters corresponding to at least one network layer calculation according to the best kernel code; generate a dynamic link library based on the configuration parameters, so that the target model can call the dynamic link library in the next inference process.

[0094] Understandably, after one or more rounds of inference optimization are completed, select the kernel code with the best performance in a certain round as the best kernel code, and the specific evaluation method is not limited. Subsequently, extract the configuration parameters related to each network layer calculation from the best kernel code, that is, obtain the best configuration parameters corresponding to each network layer calculation. The best configuration parameters can provide an efficient configuration reference for the next inference process and avoid regenerating and optimizing the kernel code repeatedly. Subsequently, generate a dynamic link library based on the best configuration parameters. For example, use a compiler to package the configuration parameters and related kernel code into a dynamic link library file (such as a.so file). The dynamic link library contains the optimized kernel code and its configuration parameters required for the target model inference. The model will directly call this dynamic link library in the subsequent inference process without regenerating or compiling, avoiding the repeated optimization process, which not only reduces the preparation time before inference, but also improves the overall inference efficiency. Accordingly, the model optimization process is completed and efficient inference can be performed.

[0095] Exemplarily, Figure 4 FIG. is a schematic flow chart of a model inference optimization method provided by an embodiment of the present application, which can be specifically understood as a CUDA kernel dynamic tuning process based on performance data. Specifically:

[0096] 1) Monitor the execution time and memory occupancy; 2) Analyze the performance data; 3) Use the neighborhood search strategy to generate configuration combinations with different unfolding factors and thread block sizes; 4) Compare the execution results of different configuration groups and select the best configuration combination; 5) Cache the optimized configuration for direct reuse by similar layers.

[0097] It is understandable that during the inference process, the execution time and memory occupancy of the CUDA kernel code are monitored in real time, and the parameter configuration is dynamically adjusted through performance feedback to optimize the execution efficiency. The execution time is recorded layer by layer, a neighborhood search strategy is adopted to generate multiple configuration combinations, and the best configuration combination is selected to adapt to different hardware and data distributions. At the same time, the optimized configuration is cached for reuse by similar layers, further improving the inference performance. For other detailed descriptions, refer to the above embodiments and will not be elaborated here.

[0098] The model inference optimization method provided by this application adopts a dynamic optimization mechanism to obtain real-time performance feedback of model inference. By monitoring indicators such as the execution time and memory bandwidth occupancy of the kernel code corresponding to each network layer, configuration parameters such as the loop unrolling factor and thread block size are automatically adjusted to adapt to different hardware and data conditions. This dynamic optimization mechanism ensures that the model can always achieve the optimal inference performance in different hardware environments and network structures, solving the problem that traditional static optimization schemes cannot handle real-time changes.

[0099] Based on the above embodiments, Figure 5 FIG. is a schematic flowchart of a model inference optimization method provided by an embodiment of this application, which specifically includes the following steps:

[0100] 1) Load and initialize the kernel generation and optimization module; 2) Configure the environment for the module and load the compiler; 3) Capture the network characteristics of each network layer included in the model; 4) Automatically generate adapted kernel code according to the network characteristics; 5) Configure kernel parameters such as parallelism and memory occupancy, and fuse common operations such as activation and normalization into the kernel code; 6) Compile the generated kernel code, generate a dynamic link library, and replace the original calculation method in the model; 7) Dynamically adjust the kernel parameters through performance feedback (such as execution time, memory occupancy, etc.).

[0101] It is understandable that for the specific implementation descriptions of the above steps, refer to the above embodiments and will not be elaborated here.

[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0103] An embodiment of this application also provides a model inference optimization device, Figure 6 FIG. is a schematic flowchart of a model inference optimization device provided by an embodiment of this application. The model inference optimization device 600 includes an acquisition module 601, a kernel code generation module 602, a kernel parameter configuration module 603, a kernel code compilation module 604, and a model inference module 605, where:

[0104] An acquisition module 601, configured to acquire network characteristics of at least one network layer included in a target model;

[0105] A kernel code generation module 602, configured to generate kernel code adapted to at least one network layer according to the network characteristics; wherein, the kernel code characterizes the calculation method of the network layer in the current inference process of the target model;

[0106] A kernel parameter configuration module 603, configured to configure kernel parameters based on the hardware resources of a target device for which the target model is to be inferred; wherein, the kernel parameters refer to the relevant parameters for the execution of the kernel code on the target device;

[0107] A kernel code compilation module 604, configured to compile the kernel code to generate an executable kernel file;

[0108] A model inference module 605, configured to call the kernel file to perform inference on the target model.

[0109] Wherein, the network characteristics include at least one operation executable by the network layer.

[0110] Optionally, the model inference optimization device 600 is further configured to:

[0111] Add at least one operation code corresponding to at least one operation to the kernel code;

[0112] Wherein, the at least one operation includes an activation operation and / or a normalization operation.

[0113] Optionally, the kernel code generation module 602 is configured to:

[0114] Calculate the similarity between at least one network layer according to the network characteristics, and determine at least one pair of network layers according to the similarity;

[0115] Generate kernel code adapted to any one of the network layers according to the network characteristics of any one of the at least one pair of network layers;

[0116] Cache the kernel code for the remaining network layers in the at least one pair of network layers other than any one of the network layers to reuse the kernel code.

[0117] Optionally, the model inference optimization device 600 further includes a tuning module 606, and the tuning module 606 is configured to:

[0118] Acquire performance data of the target device in the current inference process of the target model; wherein, the performance data includes execution time and / or memory occupancy data;

[0119] Generate at least one execution policy configuration according to the performance data; wherein, the execution policy configuration includes the configuration of parallelism and thread blocks;

[0120] Determine the target execution policy configuration with the best execution effect in at least one execution policy configuration, and adjust the kernel parameters of the kernel code based on the target execution policy configuration.

[0121] Optionally, the tuning module 606 is further configured to:

[0122] Adjust the kernel parameters of the kernel code according to the real-time hardware data of the target device;

[0123] Wherein, the hardware data includes the change data of the hardware environment and / or data distribution.

[0124] Optionally, the model inference optimization device 600 is further configured to:

[0125] Obtain the best kernel code after the target model completes the current inference process, and obtain the configuration parameters corresponding to the calculations of at least one network layer according to the best kernel code;

[0126] Generate a dynamic link library based on the configuration parameters, so that the target model can call the dynamic link library in the next inference process.

[0127] Wherein, the network characteristics include the tensor sizes of each network layer.

[0128] Optionally, the kernel code generation module 602 is configured to:

[0129] Calculate and set the loop unrolling factor according to the tensor size;

[0130] Generate kernel code adapted to each network layer based on the loop unrolling factor.

[0131] Wherein, at least one network layer includes a convolutional layer, and the network characteristics include the convolutional kernel size, the number of input and output channels, and / or the activation function type of the convolutional layer.

[0132] Optionally, the kernel parameter configuration module 603 is configured to:

[0133] Configure kernel parameters for the convolutional layer according to the network characteristics of the convolutional layer and the hardware resources of the target device for which the target model is to be inferred;

[0134] Wherein, the kernel parameters include parallelism, thread block size, and / or shared memory size.

[0135] Optionally, the model inference optimization device 600 is further configured to:

[0136] Load and initialize a pre-built kernel generation and optimization module; wherein, the kernel generation and optimization module is used to generate and optimize kernel code;

[0137] Configure the environment parameters required by the kernel generation and optimization module;

[0138] Load a compiler; wherein, the compiler is used to compile the kernel code.

[0139] Optionally, the model inference optimization device 600 is further configured to:

[0140] Modularize the kernel generation and optimization module to decouple the kernel generation and optimization module from the learning framework of the target model;

[0141] Generate an application programming interface for the kernel generation and optimization module to integrate the kernel generation and optimization module into the learning frameworks of other models through the application programming interface.

[0142] Optionally, the model inference optimization device 600 is further configured to:

[0143] Modify the calculation method of the target model so that the target model includes each network layer calling the corresponding kernel file during the current inference process.

[0144] For the description of the features in the embodiments corresponding to the model inference optimization device, reference can be made to the relevant descriptions in the embodiments corresponding to the model inference optimization method, which will not be elaborated here one by one.

[0145] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above model inference optimization method embodiments.

[0146] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above model inference optimization method embodiments when running.

[0147] In an exemplary embodiment, the above computer-readable storage medium may include: various media such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc that can store computer programs.

[0148] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above model inference optimization method embodiments.

[0149] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above model inference optimization method embodiments.

[0150] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0151] The above has introduced in detail a model inference optimization provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for optimizing model inference, characterized in that, Including: Obtain the network characteristics of at least one network layer included in the target model; wherein, the network characteristics are used to characterize the role of the network layer in the target model and the requirements of the network layer for hardware resources; Generate kernel code adapted to the at least one network layer according to the network characteristics; wherein, the kernel code represents the computing logic, optimization strategy, and hardware strategy of the network layer in the current inference process of the target model; the kernel code can be reused by similar layers of the network layer, and the similar layers are determined according to the network characteristics; Configure kernel parameters based on the hardware resources of the target device for which the target model is to be inferred, so as to configure the hardware strategy in real time; wherein, the kernel parameters refer to the relevant parameters for the execution of the kernel code on the target device; Compile the kernel code to generate an executable kernel file; Call the kernel file to perform inference on the target model.

2. The method according to claim 1, characterized in that, The network characteristics include at least one operation executable by the network layer. After configuring the kernel parameters based on the hardware resources of the target device for which the target model is to be inferred, the method further includes: Add at least one operation code corresponding to the at least one operation to the kernel code; Wherein, the at least one operation includes an activation operation and / or a normalization operation.

3. The method according to claim 1, wherein The generating kernel code adapted to the at least one network layer according to the network characteristics includes: Calculate the similarity between the at least one network layer according to the network characteristics, and determine at least one pair of network layers according to the similarity; Generate kernel code adapted to any one of the network layers according to the network characteristics of any one of the at least one pair of network layers; Cache the kernel code for the remaining network layers in the at least one pair of network layers other than the any one of the network layers to reuse the kernel code.

4. The method according to claim 1, wherein After calling the kernel file to perform inference on the target model, the method further includes: Obtain the performance data of the target device in the current inference process of the target model; wherein, the performance data includes execution time and / or memory occupancy data; Generate at least one execution policy configuration according to the performance data; wherein, the execution policy configuration includes the configuration of parallelism and / or thread blocks; Determine the target execution policy configuration with the best execution effect among the at least one execution policy configuration, and adjust the kernel parameters of the kernel code based on the target execution policy configuration.

5. The method according to any one of claims 1 or 4, characterized in that, After calling the kernel file to perform inference on the target model, the method further includes: Adjust the kernel parameters of the kernel code according to the real-time hardware data of the target device; Wherein, the hardware data includes change data of the hardware environment and / or data distribution.

6. The method according to claim 1, wherein After completing the inference of the target model, the method further includes: Obtain the best kernel code of the target model after completing the current inference process, and obtain the configuration parameters corresponding to the calculation of the at least one network layer according to the best kernel code; Generate a dynamic link library based on the configuration parameters, so that the target model can call the dynamic link library in the next inference process.

7. The method according to claim 1, characterized in that, The network characteristics include the tensor sizes of each network layer. Generating kernel code adapted to the at least one network layer according to the network characteristics includes: Calculating and setting a loop unrolling factor according to the tensor size; Generating kernel code adapted to each network layer based on the loop unrolling factor.

8. The method according to claim 1, wherein The at least one network layer includes a convolutional layer. The network characteristics include the convolutional kernel size, the number of input and output channels, and / or the activation function type of the convolutional layer. Configuring kernel parameters based on the hardware resources of the target device for performing inference of the target model includes: Configuring kernel parameters for the convolutional layer according to the network characteristics of the convolutional layer and the hardware resources of the target device for performing inference of the target model; Wherein, the kernel parameters include parallelism, thread block size, and / or shared memory size.

9. The method according to claim 1, wherein Before obtaining the network characteristics of at least one network layer included in the target model, the method further includes: Loading and initializing a pre-constructed kernel generation and optimization module; wherein, the kernel generation and optimization module is used to generate and optimize kernel code; Configuring environment parameters required by the kernel generation and optimization module; Loading a compiler; wherein, the compiler is used to compile the kernel code.

10. The method according to claim 9, wherein The method further includes: Performing modular design on the kernel generation and optimization module to decouple the kernel generation and optimization module from the learning framework of the target model; Generating an application programming interface for the kernel generation and optimization module to integrate the kernel generation and optimization module into the learning frameworks of other models through the application programming interface.

11. The method according to claim 1, wherein Before calling the kernel file to perform inference of the target model, the method further includes: Modifying the calculation method of the target model so that at least one network layer included in the target model calls the corresponding kernel file during the current inference process.

12. An apparatus for optimizing model inference, characterized in that, Including: An acquisition module, configured to acquire the network characteristics of at least one network layer included in the target model; wherein, the network characteristics are used to characterize the role of the network layer in the target model and the requirements of the network layer for hardware resources; A kernel code generation module, configured to generate kernel code adapted to the at least one network layer according to the network characteristics; wherein, the kernel code represents the calculation logic, optimization strategy, and hardware strategy of the network layer during the current inference process of the target model; the kernel code can be reused by similar layers of the network layer, and the similar layers are determined according to the network characteristics; A kernel parameter configuration module, configured to configure kernel parameters based on the hardware resources of the target device for performing inference of the target model to configure the hardware strategy in real time; wherein, the kernel parameters refer to the relevant parameters for the kernel code to execute on the target device; A kernel code compilation module, configured to compile the kernel code to generate an executable kernel file; A model inference module, configured to call the kernel file to perform inference of the target model.

13. An electronic device, characterized in that, Including: A memory, configured to store a computer program; A processor, configured to implement the steps of the model inference optimization method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the steps of the model inference optimization method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the model inference optimization method according to any one of claims 1 to 11 are implemented.

Images