Model reasoning optimization method, device, equipment, medium and program product

Abstract

The invention discloses a model reasoning optimization method and device, equipment, a medium and a program product, and relates to the technical field of computers, and the method comprises the steps: obtaining the network characteristics of at least one network layer included in a target model; generating kernel codes matched with the at least one network layer according to the network characteristics; the kernel code represents the calculation mode of the network layer in the current reasoning process of the target model; kernel parameters are configured based on hardware resources of the target device to be subjected to target model reasoning; the kernel parameters refer to related parameters executed by the kernel codes on the target equipment; compiling the kernel code to generate an executable kernel file; and calling the kernel file to perform reasoning of the target model. According to the technical scheme, the kernel codes matched with all the network layers are generated based on the network characteristics, the kernel parameters are flexibly set according to the hardware resources, the technical problems that fixed codes are poor in flexibility and the hardware resource utilization rate is low are solved, and the technical effect of improving the model reasoning efficiency and the resource utilization rate is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model reasoning optimization method, device, equipment, medium and program product. Background Art

[0002] With the widespread application of deep learning models, model reasoning performance has become one of the key bottlenecks. In related technologies, due to the large differences in the characteristics of different network layers (such as convolutional layers, fully connected layers, etc.), general optimization methods are difficult to fully adapt to the requirements of each network layer, resulting in insufficient utilization of graphics processing unit (GPU) resources and limited computing speed. In addition, traditional methods usually rely on fixed kernel codes or general optimization strategies provided by frameworks, which lack flexibility and pertinence and cannot fully tap the potential of hardware. Therefore, there is an urgent need for a technology that can dynamically generate adaptive kernel code based on the characteristics of the network layer to improve model reasoning efficiency and hardware resource utilization. Summary of the invention

[0003] The present application provides a model reasoning optimization method, apparatus, device, medium and program product to at least solve the problems of poor flexibility of fixed codes and low utilization of hardware resources in related technologies.

[0004] This application provides a model reasoning optimization method, including: Acquire network characteristics of at least one network layer included in the target model; Generate kernel code adapted to at least one network layer according to the network characteristics; wherein the kernel code represents the calculation method of the network layer in the current reasoning process of the target model; Configure kernel parameters based on hardware resources of the target device to be used for target model inference; wherein the kernel parameters refer to parameters related to the execution of kernel code on the target device; Compile the kernel code to generate an executable kernel file; Call the kernel file to perform inference on the target model.

[0005] The present application also provides a model reasoning optimization device, comprising: An acquisition module, used for acquiring network characteristics of at least one network layer included in the target model; A kernel code generation module, used to generate a kernel code adapted to at least one network layer according to network characteristics; wherein the kernel code represents the calculation method of the network layer in the current reasoning process of the target model; A kernel parameter configuration module, used to configure kernel parameters based on hardware resources of a target device to be used for target model inference; wherein kernel parameters refer to parameters related to execution of kernel code on the target device; The kernel code compilation module is used to compile the kernel code and generate an executable kernel file; The model inference module is used to call the kernel file to perform inference on the target model.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned model reasoning optimization methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned model reasoning optimization methods are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned model reasoning optimization methods when executed by a processor.

[0009] Through this application, kernel code adapted to each network layer is generated based on the network characteristics of each network layer included in the model, and the kernel parameters in the kernel code are flexibly configured according to the hardware resources of the device performing model inference. Therefore, the technical problems of poor flexibility of fixed code and low utilization of hardware resources can be solved, and the technical effect of improving model reasoning efficiency and resource utilization can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A flowchart of a model reasoning optimization method provided in an embodiment of the present application; Figure 2 A flowchart of a model reasoning optimization method provided in an embodiment of the present application; Figure 3 A flowchart of a model reasoning optimization method provided in an embodiment of the present application; Figure 4 A flowchart of a model reasoning optimization method provided in an embodiment of the present application; Figure 5 A flowchart of a model reasoning optimization method provided in an embodiment of the present application; Figure 6 A flow chart of a model reasoning optimization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0014] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0015] At present, deep learning models are widely used in image recognition, natural language processing, autonomous driving and other fields, and reasoning performance has become one of the key indicators. In particular, convolutional neural networks (CNN), recurrent neural networks (RNN) and transformer models are widely used in production environments, and their computing requirements have increased significantly. To this end, using high-performance CUDA (Compute Unified Device Architecture) kernels to accelerate neural network reasoning has become a common practice in the industry. CUDA is a parallel computing platform and programming model developed by NVIDIA that allows developers to use the powerful computing power of GPUs (Graphics Processing Units) for general computing tasks through standard programming languages ​​(such as C, C++, Python, etc.). However, traditional CUDA kernel codes are mostly general-purpose designs, and it is difficult to deeply optimize the characteristics of different network layers, resulting in problems such as insufficient GPU resource utilization and limited computing speed.

[0016] To address this problem, a variety of optimization methods have been proposed, including manually adjusting loop unrolling factors, thread configuration, and other methods, in an attempt to improve performance by manually optimizing CUDA kernel code. However, manual optimization not only relies on the developer's experience, but also requires a lot of debugging time. At the same time, changes in model structure and data size often require re-adjustment of parameters, which has high maintenance costs. In addition, although the automated CUDA optimization module can greatly reduce the difficulty of development, the existing automatic optimization mainly relies on compile-time static analysis, lacks dynamic tuning methods combined with network structure characteristics, and is difficult to cope with complex and diverse network inference tasks.

[0017] At present, the research on the optimization of CUDA kernel and deep learning model can be divided into the following categories: Compiler-level optimization, such as TVM (Tensor Virtual Machine), TensorRT and other deep learning compilers optimize CUDA kernel code by analyzing model structure, optimizing operator combinations, automatically selecting loop unrolling factors, etc. In static optimization, the compiler can select and optimize kernel configurations based on hardware characteristics.

[0018] Manual kernel optimization: Many deep learning frameworks allow developers to write CUDA kernel code to achieve specific optimization goals. Developers can manually adjust the loop unrolling factor, configure the number of threads, etc. according to the network layer structure.

[0019] Experience-based automatic optimization modules,Some optimization modules try to combine empirical rules and specific hardware characteristics,,to automatically generate optimized configurations, such as selecting parallelism and,unrolling factors based on heuristics, or using known best practices,for code generation.

[0020] However, the optimization process of compiler-level optimization methods is usually performed during the model compilation phase, lacking feedback during execution and unable to make dynamic adjustments to changes during execution (such as changes in input data distribution or differences in hardware environment).

[0021] Manual kernel optimization relies on programming experience, is cumbersome to operate, and requires manual adjustments every time the model or data changes, which affects efficiency and increases maintenance costs.

[0022] The adaptability of experience-based automatic optimization modules is limited and they often cannot effectively adapt to the characteristics of different network layers and hardware, making it difficult to achieve true adaptive optimization.

[0023] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the model reasoning optimization method depends, the specific application environment architecture or specific hardware architecture is described here.

[0024] The embodiment of the present application provides a model reasoning optimization method, and the method is described in detail in combination with the execution process of the model reasoning optimization method. Figure 1 , Figure 1 A flow chart of a model reasoning optimization method provided in an embodiment of the present application, specifically including the following steps: Figure 1 The following steps are shown: S101. Obtain network characteristics of at least one network layer included in a target model.

[0025] It is understandable that the target model refers to the deep learning model to be inferred or trained and optimized, and the target model can be any type of neural network model, such as convolutional neural network, recurrent neural network, etc. The target model includes at least one network layer, which is the basic computing unit of the model and is responsible for performing specific mathematical operations, such as convolutional layer, fully connected layer, pooling layer, normalization layer, etc.

[0026] It is understandable that the characteristic parameters of each network layer are extracted to obtain the network characteristics of each network layer. Network characteristics refer to the key attributes that describe the functions and behaviors of the network layer, which are used to characterize the role of the network layer in the model and its demand for hardware resources, that is, to extract the key attributes of a specific network layer in the deep learning model, where network characteristics include convolution kernel size, number of channels, activation function type, etc. For example, for the convolution layer, the obtained network characteristics include "layer type: convolution layer, input dimension: 224×224×3, output dimension: 224×224×64, convolution kernel size: 3×3, number of filters: 64". The above characteristic parameters can be extracted through the metadata of the model or the actual data in the dynamic reasoning process, and then the extracted characteristic parameters are recorded to obtain the network characteristics, so as to provide data support for the subsequent generation of specialized kernel code.

[0027] Optionally, before obtaining the network characteristics of at least one network layer included in the target model, the method further includes: Load and initialize a pre-built kernel generation and optimization module; wherein the kernel generation and optimization module is used to generate and optimize kernel code; configure environmental parameters required by the kernel generation and optimization module; load a compiler; wherein the compiler is used to compile the kernel code.

[0028] It is understandable that before performing model inference optimization or obtaining network characteristics, the kernel generation and optimization module is loaded and initialized. The kernel generation and optimization module (hereinafter referred to as the module) is a pre-built module that can automatically generate and optimize CUDA kernel code. Specifically, the module is responsible for generating adaptive kernel code according to network characteristics and optimizing the generated kernel code to improve performance. In addition, the module can be an independently developed tool or a plug-in embedded in an existing deep learning framework. Among them, the module initialization process includes allocating memory resources, loading configuration files, and setting default parameters. After initializing the module, the environment parameters required for the module to run are configured. The environment parameters include target hardware information, task requirement information, and optimization strategies, such as CUDA version, compiler path, temporary file storage directory, etc., among which the environment parameters determine the behavior and output results of the module. By dynamically adjusting the environment parameters later, different hardware platforms and task requirements can be adapted. After completing the environment configuration of the module, load the CUDA compiler (such as nvcc or NVRTC) to ensure that the compilation environment is ready, wherein the compiler is used to convert the generated kernel code into a binary file executable by the target device. Subsequently, the module is run and the initial configuration is loaded, which is a predefined set of initial kernel parameters for preliminary CUDA code generation.

[0029] Optionally, after completing the kernel generation and the construction of the optimization module, the method further includes: The kernel generation and optimization module is modularly designed to decouple the kernel generation and optimization module from the learning framework of the target model; an application programming interface of the kernel generation and optimization module is generated to integrate the kernel generation and optimization module into the learning framework of other models through the application programming interface.

[0030] Understandably, after completing the construction of the module, the module is separated from the learning framework of the target model through modular design, that is, the module and the deep learning framework of the target model are decoupled to make it an independent functional module. The module can then run independently or be integrated into other models without relying on the deep learning framework of a specific model. Subsequently, a set of standardized application programming interfaces (APIs) are provided for the module so that the deep learning frameworks of other models can directly call the module. Specifically, the application interface can provide a network feature extraction interface, a kernel code generation interface, and a compilation and execution interface, among which the network feature extraction interface allows the external deep learning framework to pass model structure information (network features), the kernel code generation interface can receive network features and return optimized kernel code, and the compilation and execution interface supports compiling the generated kernel code into an executable file for model calling. This modular design and API interface design method enables the module to be seamlessly integrated into the existing deep learning framework, which can not only reduce the workload of developers, but also continuously improve the reasoning performance through the kernel configuration cache mechanism.

[0031] S102: Generate kernel code adapted to at least one network layer according to network characteristics.

[0032] Among them, the kernel code represents the calculation method of the network layer during the current inference process of the target model.

[0033] It is understandable that, based on the above S101, the kernel code of CUDA or other GPU calculations adapted to each network layer is automatically generated according to the captured network characteristics to ensure that the generated kernel code can adapt to the specific computing requirements of the network layer. Specifically, a template engine or code generation tool is used to dynamically generate kernel code according to network characteristics; different code generation strategies are adopted for different types of network layers (such as convolutional layers, fully connected layers, etc.); the kernel code contains specific computing logic and optimization strategies, such as thread allocation, memory access mode, etc. Among them, the generated kernel code not only realizes the basic computing function of the network layer, but also reflects its specific implementation details in the current reasoning process, that is, the kernel code involves at least three aspects such as computing logic, optimization strategy and hardware strategy. The computing logic includes convolution operations, matrix multiplication, etc., and the optimization strategy includes parallelism setting, shared memory allocation, data blocking strategy, etc. The hardware strategy can be configured in real time according to the hardware resources of the target device described below.

[0034] Optionally, a kernel code adapted to at least one network layer is generated according to network characteristics, which can be implemented by the following steps: The similarity between at least one network layer is calculated according to the network characteristics, and at least one network layer pair is determined according to the similarity; the kernel code adapted to any network layer is generated according to the network characteristics of any network layer in at least one network layer pair; the kernel code is cached so that the remaining network layers except any network layer in at least one network layer pair can reuse the kernel code.

[0035] It is understandable that before generating kernel codes adapted to each network layer, the similarity between each network layer is calculated based on the analysis of network characteristics (such as input and output dimensions, convolution kernel size, activation function type, etc.), that is, similar layers between network layers are calculated. For example, if the target model includes 3 convolution layers, the 3 convolution layers can be understood as similar layers, and the 3 convolution layers can be defined as a network layer pair, so as to generate shared kernel codes for the network layer pair. Specifically, distance measurement methods (such as Euclidean distance and cosine similarity) can be used to compare the numerical differences of network characteristics. Other possible methods are not described in detail. After determining the network layer pair, for any network layer in the network layer pair, such as the first convolution layer in the above three convolution layers, an adapted kernel code is generated based on its network characteristics to ensure that the generated kernel code can adapt to the specific computing requirements of the first convolution layer, and then the generated kernel code is stored in the cache for reuse by other network layers in the same network layer pair. In the subsequent reasoning process, if the same network layer is encountered, the generated kernel code can be directly loaded from the cache. This method of directly reusing kernel code in similar layers avoids repeated generation and compilation of kernel code, significantly reducing computing overhead. The optimized configuration information can also be cached for direct reuse by similar layers, further improving reasoning performance.

[0036] Optionally, the network characteristics include the tensor size of each network layer. The kernel code adapted to at least one network layer is generated according to the network characteristics, which can be specifically implemented by the following steps: According to the size of the tensor, the loop expansion factor is calculated and set; based on the loop expansion factor, the kernel code adapted to each network layer is generated.

[0037] Understandably, tensor size refers to the dimensional information of the input and output tensors of the network layer, such as the input size (height, width, number of channels) of the convolutional layer or the input vector length of the fully connected layer. The tensor size directly affects the scale of the computing task and the memory access pattern. The appropriate loop unrolling factor is calculated according to the tensor size. For example, a higher unrolling factor can be selected for larger tensors to reduce loop overhead, and a lower unrolling factor can be selected for smaller tensors to avoid resource waste. Among them, the loop unrolling factor is an optimization technology that improves computing efficiency by reducing the number of loop iterations. By setting the loop unrolling factor reasonably, the execution efficiency of the kernel code can be optimized. Subsequently, the calculated loop unrolling factor is used to generate kernel code optimized for a specific network layer. Specifically, the loop can be displayed in the kernel code to reduce the execution overhead of branch instructions. This automatic and intelligent optimization method automatically generates and compiles the optimal CUDA kernel code by capturing the Tensor size and dynamically adjusting the loop unrolling factor, thereby significantly improving the efficiency of model inference.

[0038] S103: configure kernel parameters based on hardware resources of the target device on which target model inference is to be performed.

[0039] The kernel parameters refer to the parameters related to the execution of the kernel code on the target device.

[0040] It is understandable that, based on the above S102, the hardware resources of the target device to be used for target model inference are obtained, such as GPU architecture, number of CUDA cores, shared memory size, etc. Subsequently, according to the hardware resources and the extracted characteristic parameters, the kernel parameters such as the parallelism of the kernel and the shared memory configuration are automatically set, that is, the hardware strategy in the above kernel code is set to ensure that the kernel code can make full use of the hardware characteristics of the target device and maximize computing efficiency and resource utilization. Among them, the kernel parameters are specific settings that control the execution behavior of the kernel code on the target device, which directly affects the performance. For example, for the convolution layer, the appropriate thread block size and shared memory allocation strategy are selected according to the hardware resources, convolution kernel size and number of channels. Among them, the thread block size defines the number of threads in each thread block, and usually selects values ​​such as 256 and 512 to adapt to the hardware parallel capability. The shared memory allocation allocates a suitable shared memory size to each thread block, which can optimize the data access mode.

[0041] Optionally, the network characteristic includes at least one operation executable by the network layer, and after configuring the kernel parameters based on the hardware resources of the target device to be subjected to the target model inference, the method further includes: At least one operation code corresponding to the at least one operation is added to the kernel code; wherein the at least one operation includes an activation operation and a normalization operation.

[0042] It is understandable that network characteristics not only include static information such as tensor size, but also cover specific operations that the network layer can perform (such as activation function, normalization, etc.). Insert code snippets related to specific operations (such as activation or normalization operations) into the generated kernel code to expand the functionality of the kernel. Specifically, according to the operation type of the network layer, select the corresponding operation code template, embed the operation code into the core calculation logic of the kernel code, and ensure its seamless integration, that is, integrate common operations such as activation and normalization into the kernel code. By integrating multiple operations, the multiple startup overhead of common operations can be effectively reduced and the execution efficiency can be improved. For example, for the convolution layer, the ReLU code and the normalization code are embedded after the convolution operation to form a complete kernel code. Compared with the method of starting the ReLU code and the normalization code separately in the related art (i.e., starting at least 2 times), the method of integrating the ReLU code and the normalization code into the kernel code (i.e., starting once) not only reduces the number of startups, but also speeds up the calculation speed.

[0043] Optionally, at least one network layer includes a convolutional layer, and the network characteristics include a convolution kernel size, a number of input and output channels, and / or an activation function type of the convolutional layer. Configuring kernel parameters based on hardware resources of a target device on which target model inference is to be performed can be specifically implemented by the following steps: According to the network characteristics of the convolution layer and the hardware resources of the target device to be used for target model inference, kernel parameters are configured for the convolution layer; wherein the kernel parameters include parallelism, thread block size and / or shared memory size.

[0044] It is understandable that dynamically adjusting the kernel parameters to adapt to the inference tasks of the convolution layer according to the network characteristics of the convolution layer and the hardware resources of the target device can maximize the utilization of the hardware resources of the target device and improve the inference efficiency of the convolution layer.

[0045] S104: Compile the kernel code to generate an executable kernel file.

[0046] It is understandable that, based on the above S103, the generated kernel code is converted into an executable file adapted to the target device by a compiler, that is, the generated kernel code is compiled using a compiler (such as nvcc) to generate an executable kernel file. Among them, the compilation process includes selecting compilation options adapted to the hardware architecture of the target device and applying optimization strategies. The kernel file generated after the compilation is completed is a binary file or intermediate representation file that can be directly loaded and executed by the target device, such as a .cubin or .ptx format.

[0047] S105. Call the kernel file to perform inference on the target model.

[0048] It is understandable that, based on the above S104, the compiled kernel file is loaded onto the target device (such as GPU), and the parameters required for the target model reasoning are set, including input data, output buffer, and thread block layout. Subsequently, the reasoning task of the target model is completed by calling the kernel function in the kernel file. This method of efficiently executing model reasoning using optimized kernel code effectively improves computing performance and resource utilization.

[0049] Optionally, before calling the kernel file to perform inference on the target model, the method further includes: Modify the calculation method of the target model so that at least one network layer included in the target model calls the corresponding kernel file during the current reasoning process.

[0050] Understandably, before calling the kernel file for inference, modify the calculation method of the target model so that it calls the generated kernel file during the inference process, replacing the original calculation method. The new calculation method represented by the kernel file will ensure that the calculation of each network layer uses the optimized kernel code. Later in the actual inference process, the calculation of each network layer of the model will call the optimized kernel file. At this time, the inference performance is improved by the calculation method optimized by the generated and compiled kernel code.

[0051] For example, see Figure 2 , Figure 2 The flowchart of a model reasoning optimization method provided in the embodiment of the present application can be understood as a specialized kernel generation process based on network layer characteristics, specifically: 1) Capture network layer characteristics; 2) Automatically generate CUDA kernel code that is highly consistent with the layer structure based on the network characteristics; 3) Automatically set the expansion factor, parallelism and shared memory configuration based on hardware resources and network characteristics; 4) Integrate operations such as activation and normalization into the kernel code; 5) Automatically generate CUDA kernel code and save it as a ".cu" file.

[0052] It is understandable that after capturing the characteristics of the network layer, the CUDA kernel code that is highly consistent with the network layer structure is automatically generated. According to the characteristics of each network layer, the loop expansion factor, parallelism, and shared memory configuration parameters are automatically set, and the activation function, normalization and other operations are integrated into the kernel code, thereby reducing the overhead caused by multiple kernel startups. Through this specialized kernel generation method, different network layers can obtain adapted CUDA kernels, significantly improving the efficiency of model reasoning. For other specific implementation instructions, please refer to the above embodiments, which will not be repeated here.

[0053] The model reasoning optimization method provided by this application adopts a specialized kernel generation method so that each network layer operation can be customized to the best configuration according to its computing characteristics, improve kernel execution efficiency, and reduce GPU resource waste. By automatically adjusting the loop expansion factor and thread configuration, the kernel code can maximize the use of the GPU's computing power, improve the operation speed, reduce the development difficulty and cost, and simplify kernel code writing and optimization. The modular design of kernel code generation and optimization can easily integrate the module into the existing deep learning framework. Subsequent users do not need to deeply master the details of CUDA programming. They can achieve efficient reasoning optimized for network layer characteristics through simple API calls, which not only lowers the development threshold, but also improves the applicability and versatility of model reasoning.

[0054] Based on the above embodiments, Figure 3 A flow chart of a model reasoning optimization method provided in an embodiment of the present application, after calling the kernel file to reason about the target model, the method further includes: Figure 3 The following steps are shown: S301. Obtain performance data of a target device during a current inference process of a target model.

[0055] The performance data includes execution time and / or memory usage data.

[0056] Understandably, performance data is collected during / in the process of model inference, such as real-time monitoring of kernel code execution time and memory usage during inference, and recording execution time and memory usage data for each network layer. Execution time refers to the time required for kernel code to run on the target device, which is used to evaluate inference efficiency. Memory usage data refers to the memory usage of the target device, including video memory allocation, shared memory usage, etc., to optimize resource management.

[0057] S302: Generate at least one execution strategy configuration according to the performance data.

[0058] The execution strategy configuration includes the configuration of parallelism and thread blocks.

[0059] It is understandable that, based on the above S301, at least one execution strategy configuration is generated according to the collected performance data, and the execution strategy configuration covers parallelism configuration and / or thread block configuration. Specifically, a neighborhood search strategy can be used to generate configuration combinations of different execution strategies. Among them, parallelism configuration refers to dynamically adjusting the granularity of task decomposition according to the execution time and hardware resource utilization in the performance data to optimize the allocation of computing resources. Thread block configuration refers to setting the appropriate thread block size and grid layout in combination with memory usage data and the hardware characteristics of the target device (such as the number of CUDA cores, shared memory capacity, etc.), thereby improving kernel execution efficiency.

[0060] S303: Determine a target execution policy configuration with the best execution effect in at least one execution policy configuration, and adjust kernel parameters of the kernel code based on the target execution policy configuration.

[0061] It is understandable that, based on the above S302, by evaluating the generated multiple execution strategy configurations, one of them with the best execution effect is selected as the target execution strategy configuration, for example, an optimization algorithm can be used to screen out the configuration combination with the best execution effect. Subsequently, the kernel parameters in the target execution strategy configuration are applied to the kernel code, and the kernel parameters are dynamically adjusted to adapt to the inference task requirements, so as to improve the execution efficiency of the kernel code on the target device, such as adjusting the degree of parallelism and thread block size.

[0062] Understandably, through the real-time performance feedback mechanism, the module can automatically select the kernel configuration with the best performance and cache these configurations for the next reasoning. This not only reduces the overhead of repeated optimization, but also ensures that each reasoning can quickly reach the best performance state, achieving continuous performance improvement.

[0063] Optionally, after calling the kernel file to perform inference on the target model, the method further includes: According to the real-time hardware data of the target device, kernel parameters of the kernel code are adjusted; wherein the hardware data includes the change data of the hardware environment and / or data distribution.

[0064] It is understandable that in addition to tuning based on the monitored performance data, kernel parameters can also be dynamically adjusted based on the changing data of the hardware environment and / or data distribution to ensure that the inference efficiency is always in the best state. This method of optimizing based on real-time hardware changes can be understood as an overall optimization, which can not only make full use of hardware resources, but also reduce memory access conflicts.

[0065] Optionally, after completing the inference of the target model, the method further includes: Obtain the best kernel code after the target model completes the current reasoning process, and obtain configuration parameters corresponding to at least one network layer calculation according to the best kernel code; generate a dynamic link library based on the configuration parameters so that the target model calls the dynamic link library in the next reasoning process.

[0066] It is understandable that after completing one or more rounds of reasoning optimization, the kernel code with the best performance in one round is selected as the best kernel code, and the specific evaluation method is not limited. Subsequently, the configuration parameters related to the calculation of each network layer are extracted from the best kernel code, that is, the best configuration parameters corresponding to the calculation of each network layer are obtained. The best configuration parameters can provide an efficient configuration reference for the next reasoning process, avoiding repeated generation and optimization of kernel code. Subsequently, a dynamic link library is generated based on the best configuration parameters, for example, the configuration parameters and related kernel codes are packaged into a dynamic link library file (such as a .so file) using a compiler. The dynamic link library contains the optimized kernel code and its configuration parameters required for the target model reasoning. The model will directly call this dynamic link library in the subsequent reasoning process without regeneration or compilation, avoiding repeated optimization processes, not only reducing the preparation time before reasoning, but also improving the overall reasoning efficiency. Based on this, the model optimization process is completed, and efficient reasoning can be performed.

[0067] For example, Figure 4 A flow chart of a model reasoning optimization method provided in an embodiment of the present application can be specifically understood as a CUDA kernel dynamic tuning process based on performance data, specifically: 1) Monitor execution time and memory usage; 2) Analyze performance data; 3) Use neighborhood search strategy to generate configuration combinations with different expansion factors and thread block sizes; 4) Compare the execution results of different configuration groups and select the best configuration combination; 5) Cache optimized configurations for direct reuse by similar layers.

[0068] It is understandable that the execution time and memory usage of the CUDA kernel code are monitored in real time during the inference process, and the parameter configuration is dynamically adjusted through performance feedback to optimize the execution efficiency. The execution time is recorded layer by layer, and a variety of configuration combinations are generated using a neighborhood search strategy, and the best configuration combination is selected to adapt to different hardware and data distributions. At the same time, the cache optimization configuration is reused by similar layers to further improve the inference performance. For other detailed descriptions, please refer to the above embodiment, which will not be repeated here.

[0069] The model reasoning optimization method provided in this application adopts a dynamic optimization mechanism to obtain real-time performance feedback of model reasoning. By monitoring the execution time of the kernel code corresponding to each network layer, memory bandwidth occupancy and other indicators, it automatically adjusts configuration parameters such as loop expansion factor and thread block size to adapt to different hardware and data conditions. This dynamic optimization mechanism ensures that the model can always achieve the optimal reasoning performance under different hardware environments and network structures, solving the problem that traditional static optimization solutions cannot cope with real-time changes.

[0070] Based on the above embodiments, Figure 5 A flow chart of a model reasoning optimization method provided in an embodiment of the present application specifically includes the following steps: 1) Load and initialize the kernel generation and optimization module; 2) Configure the environment for the module and load the compiler; 3) Capture the network characteristics of each network layer included in the model; 4) Automatically generate adaptive kernel code based on the network characteristics; 5) Configure kernel parameters such as parallelism and memory usage, and integrate common operations such as activation and normalization into the kernel code; 6) Compile the generated kernel code, generate a dynamic link library, and replace the original calculation method in the model; 7) Dynamically adjust kernel parameters through performance feedback (such as execution time, memory usage, etc.).

[0071] It is understandable that the specific implementation instructions of the above steps can be found in the above embodiments and will not be repeated here.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0073] The embodiment of the present application also provides a model reasoning optimization device, Figure 6 A flow chart of a model reasoning optimization device provided in an embodiment of the present application. The model reasoning optimization device 600 includes an acquisition module 601, a kernel code generation module 602, a kernel parameter configuration module 603, a kernel code compilation module 604 and a model reasoning module 605, wherein: An acquisition module 601 is used to acquire network characteristics of at least one network layer included in the target model; A kernel code generation module 602 is used to generate a kernel code adapted to at least one network layer according to network characteristics; wherein the kernel code represents the calculation method of the network layer in the current reasoning process of the target model; A kernel parameter configuration module 603 is used to configure kernel parameters based on the hardware resources of the target device to be subjected to target model inference; wherein the kernel parameters refer to the parameters related to the execution of the kernel code on the target device; The kernel code compiling module 604 is used to compile the kernel code and generate an executable kernel file; The model inference module 605 is used to call the kernel file to perform inference on the target model.

[0074] The network characteristic includes at least one operation executable by the network layer.

[0075] Optionally, the model reasoning optimization device 600 is further used for: adding at least one operation code corresponding to at least one operation to the kernel code; Wherein, at least one operation includes an activation operation and / or a normalization operation.

[0076] Optionally, the kernel code generation module 602 is used to: Calculating similarity between at least one network layer according to the network characteristics, and determining at least one network layer pair according to the similarity; generating kernel code adapted to any network layer according to the network characteristics of any network layer in at least one network layer pair; Kernel codes are cached for reuse by remaining network layers except any one of the network layers in at least one network layer pair.

[0077] Optionally, the model reasoning optimization device 600 further includes a tuning module 606, and the tuning module 606 is used to: Obtaining performance data of the target device during the current inference process of the target model; wherein the performance data includes execution time and / or memory usage data; Generate at least one execution strategy configuration according to the performance data; wherein the execution strategy configuration includes configuration for parallelism and thread blocks; A target execution policy configuration with the best execution effect is determined in at least one execution policy configuration, and kernel parameters of the kernel code are adjusted based on the target execution policy configuration.

[0078] Optionally, the tuning module 606 is further used for: Adjust kernel parameters of kernel code according to real-time hardware data of target device; The hardware data includes change data of the hardware environment and / or data distribution.

[0079] Optionally, the model reasoning optimization device 600 is further used for: Obtaining the best kernel code after the target model completes the current reasoning process, and obtaining configuration parameters corresponding to at least one network layer calculation according to the best kernel code; A dynamic link library is generated based on the configuration parameters, so that the target model calls the dynamic link library in the next reasoning process.

[0080] Among them, the network characteristics include the tensor size of each network layer.

[0081] Optionally, the kernel code generation module 602 is used to: Calculate and set the loop unrolling factor based on the tensor size; Based on the loop unrolling factor, kernel code adapted to each network layer is generated.

[0082] Among them, at least one network layer includes a convolutional layer, and the network characteristics include a convolution kernel size, a number of input and output channels and / or an activation function type of the convolutional layer.

[0083] Optionally, the kernel parameter configuration module 603 is used to: Configure kernel parameters for the convolution layer according to the network characteristics of the convolution layer and the hardware resources of the target device to be used for target model inference; The kernel parameters include parallelism, thread block size and / or shared memory size.

[0084] Optionally, the model reasoning optimization device 600 is further used for: Load and initialize a pre-built kernel generation and optimization module; wherein the kernel generation and optimization module is used to generate and optimize kernel code; Configure the environment parameters required for kernel generation and optimization modules; Load the compiler; the compiler is used to compile the kernel code.

[0085] Optionally, the model reasoning optimization device 600 is further used for: Modularize the kernel generation and optimization modules to decouple them from the learning framework of the target model. Generate an application program interface for the kernel generation and optimization module to integrate the kernel generation and optimization module into the learning framework of other models through the application program interface.

[0086] Optionally, the model reasoning optimization device 600 is further used for: Modify the calculation method of the target model so that the target model includes each network layer and calls the corresponding kernel file during the current inference process.

[0087] For the description of the features in the embodiment corresponding to the model reasoning optimization device, please refer to the relevant description of the embodiment corresponding to the model reasoning optimization method, which will not be repeated here.

[0088] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned model reasoning optimization method embodiments.

[0089] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned model reasoning optimization method embodiments when running.

[0090] In an exemplary embodiment, the computer-readable storage medium may include: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0091] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned model reasoning optimization method embodiments are implemented.

[0092] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned model reasoning optimization method embodiments are implemented.

[0093] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] The above is a detailed introduction to a model reasoning optimization provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A model reasoning optimization method, characterized in that: include: Acquire network characteristics of at least one network layer included in the target model; Generate a kernel code adapted to the at least one network layer according to the network characteristics; wherein the kernel code represents the calculation method of the network layer in the current reasoning process of the target model; Configure kernel parameters based on hardware resources of the target device on which the target model inference is to be performed; wherein the kernel parameters refer to parameters related to the execution of the kernel code on the target device; Compiling the kernel code to generate an executable kernel file; The kernel file is called to perform reasoning on the target model.

2. The method according to claim 1, characterized in that The network characteristic includes at least one operation executable by the network layer. After configuring kernel parameters based on hardware resources of a target device to perform the target model inference, the method further includes: adding at least one operation code corresponding to the at least one operation to the kernel code; The at least one operation includes an activation operation and / or a normalization operation.

3. The method according to claim 1, characterized in that The step of generating a kernel code adapted to the at least one network layer according to the network characteristics comprises: Calculating similarity between the at least one network layer according to the network characteristics, and determining at least one network layer pair according to the similarity; Generate kernel code adapted to any network layer according to the network characteristics of any network layer in the at least one network layer pair; The kernel code is cached so that the remaining network layers except any one of the network layers in the at least one network layer pair can reuse the kernel code.

4. The method according to claim 1, characterized in that After calling the kernel file to perform reasoning on the target model, the method further includes: Acquire performance data of the target device during the current inference process of the target model; wherein the performance data includes execution time and / or memory usage data; Generating at least one execution strategy configuration according to the performance data; wherein the execution strategy configuration includes configuration for parallelism and / or thread blocks; A target execution policy configuration with the best execution effect is determined in the at least one execution policy configuration, and kernel parameters of the kernel code are adjusted based on the target execution policy configuration.

5. The method according to any one of claims 1 or 4, characterized in that: After calling the kernel file to perform reasoning on the target model, the method further includes: Adjusting kernel parameters of the kernel code according to real-time hardware data of the target device; The hardware data includes change data of hardware environment and / or data distribution.

6. The method according to claim 1, characterized in that After completing the reasoning of the target model, the method further includes: Obtaining an optimal kernel code after the target model completes the current reasoning process, and obtaining configuration parameters corresponding to the at least one network layer calculation according to the optimal kernel code; A dynamic link library is generated based on the configuration parameters, so that the target model calls the dynamic link library in the next reasoning process.

7. The method according to claim 1, characterized in that The network characteristics include the tensor size of each network layer, and the generating of kernel code adapted to the at least one network layer according to the network characteristics includes: Calculate and set a loop unrolling factor based on the tensor size; Based on the loop unrolling factor, kernel codes adapted to the network layers are generated.

8. The method according to claim 1, characterized in that The at least one network layer includes a convolution layer, the network characteristics include a convolution kernel size, a number of input and output channels, and / or an activation function type of the convolution layer, and the kernel parameters are configured based on hardware resources of a target device on which the target model inference is to be performed, including: Configuring kernel parameters for the convolution layer according to network characteristics of the convolution layer and hardware resources of a target device on which the target model inference is to be performed; The kernel parameters include parallelism, thread block size and / or shared memory size.

9. The method according to claim 1, characterized in that: Before obtaining the network characteristics of at least one network layer included in the target model, the method further includes: Loading and initializing a pre-built kernel generation and optimization module; wherein the kernel generation and optimization module is used to generate and optimize kernel code; Configuring the environment parameters required for the kernel generation and optimization module; Loading a compiler; wherein the compiler is used to compile the kernel code.

10. The method according to claim 9, characterized in that The method further comprises: Modularizing the kernel generation and optimization module to decouple the kernel generation and optimization module from the learning framework of the target model; Generate an application program interface for the kernel generation and optimization module, so as to integrate the kernel generation and optimization module into a learning framework of other models through the application program interface.

11. The method according to claim 1, characterized in that: Before calling the kernel file to perform reasoning on the target model, the method further includes: The calculation method of the target model is modified so that at least one network layer included in the target model calls a corresponding kernel file during the current reasoning process.

12. A model reasoning optimization device, characterized in that: include: An acquisition module, used for acquiring network characteristics of at least one network layer included in the target model; A kernel code generation module, used to generate a kernel code adapted to the at least one network layer according to the network characteristics; wherein the kernel code represents the calculation method of the network layer in the current reasoning process of the target model; A kernel parameter configuration module, used to configure kernel parameters based on hardware resources of a target device to be used for target model inference; wherein the kernel parameters refer to parameters related to execution of the kernel code on the target device; A kernel code compiling module, used to compile the kernel code to generate an executable kernel file; The model inference module is used to call the kernel file to perform inference on the target model.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the model reasoning optimization method as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model reasoning optimization method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model reasoning optimization method according to any one of claims 1 to 11 are implemented.

Images