An AI compiler automatic tuning method and electronic device

Abstract

The present application relates to the technical field of compiler optimization and artificial intelligence, and discloses an AI compiler automatic tuning method and electronic equipment, which comprises the following steps: generating a structured prompt word for representing model information according to an AI model to be deployed to a target hardware; inputting the structured prompt word into a pre-trained hybrid expert large language model to obtain a tensor program and a compilation optimization strategy, wherein a gating network of the hybrid expert large language model is used to distribute the structured prompt word to an expert module matched with the target hardware. The purpose of the present application is to replace the traditional search-type tuning process with the code understanding and generation capabilities of the hybrid expert large language model, and to realize the continuous evolution of the compilation tuning model by combining the real running feedback of the hardware, so as to greatly shorten the deployment and tuning cycle of the AI model on the target hardware.

Description

Technical Field

[0001] This invention relates to the fields of compiler optimization and artificial intelligence technology, specifically to an automatic tuning method and electronic device for an AI compiler. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale and structural complexity of AI models continue to grow. During the deployment of AI models, compilers need to convert high-level model representations into executable code for the target hardware and improve the model's performance on the target hardware through compilation optimization techniques.

[0003] However, when faced with compilation optimization tasks on different hardware platforms, existing AI compilers typically require a large amount of computing resources to repeatedly perform trial calculations and evaluations in a huge scheduling search space, resulting in a long optimization cycle. Summary of the Invention

[0004] The purpose of this invention is to replace the traditional search-based tuning process with the code understanding and generation capabilities of a hybrid expert large language model, and to achieve continuous evolution of the compilation tuning model by combining real hardware operation feedback, thereby significantly shortening the deployment and tuning cycle of AI models on target hardware. This invention proposes an automatic tuning method for AI compilers and an electronic device.

[0005] In a first aspect, embodiments of the present invention provide an automatic tuning method for an AI compiler, comprising: Based on the AI ​​model to be deployed to the target hardware, generate structured prompts to represent model information; The structured prompts are input into a pre-trained hybrid expert large language model to obtain a tensor program and a compilation optimization strategy. The gating network of the hybrid expert large language model is used to distribute the structured prompts to expert modules that match the target hardware. The expert modules are used to generate the tensor program and the compilation optimization strategy based on the structured prompts. The tensor program is subjected to hardware performance evaluation to obtain the operating data of the target hardware, and a reward feedback signal is generated based on the operating data. The hybrid expert large language model is optimized based on the reward feedback signal. The compilation optimization strategy and the tensor program generated by the optimized hybrid expert large language model are sent to the compiler backend so that the compiler backend can generate the executable file for the target hardware.

[0006] In one embodiment, the structured prompt word includes a target hardware identifier; the gating network is used to determine a routing weight based on the structured prompt word, and distribute the structured prompt word to the expert module corresponding to the target hardware identifier based on the routing weight.

[0007] In one embodiment, the expert module includes at least one of a GPU expert module, an NPU expert module, and a RISC-V expert module; the GPU expert module is used to generate the tensor program containing CUDA code optimized for thread blocks, shared memory, and parallel scheduling based on the structured prompts; the NPU expert module is used to generate the tensor program conforming to the operator execution mode of the target hardware based on the structured prompts; and the RISC-V expert module is used to generate the tensor program containing code optimized for vector access, vector computation, and data arrangement based on the structured prompts.

[0008] In one embodiment, generating structured prompts to represent model information based on the AI ​​model to be deployed to the target hardware includes: performing intermediate representation transformation on the AI ​​model to obtain a multi-level information representation (IR); extracting semantic features from the multi-level IR; and structurally encapsulating the semantic features to obtain the structured prompts.

[0009] In one embodiment, the semantic features include at least one of operator topology, tensor shape, data flow dependency, and target hardware identifier.

[0010] In one embodiment, generating a reward feedback signal based on the operational data includes: constructing a reward model based on the operational data; and performing reinforcement learning based on the reward model to obtain the reward feedback signal.

[0011] In one embodiment, the runtime data includes at least one of the following: runtime of the tensor program, compilation success rate, hardware resource utilization, memory access efficiency, and operator parallelism; the step of performing reinforcement learning based on the reward model to obtain the reward feedback signal includes: inputting the runtime data into the reward model to calculate the reward score; when the tensor program has compilation errors or hardware constraint violations, the reward model generates a negative reward feedback signal.

[0012] In one embodiment, the tensor program performs hardware performance evaluation to obtain runtime data of the target hardware, including: deploying the tensor program to the corresponding target hardware and obtaining the runtime data generated by the target hardware performing micro-benchmark tests; and / or, performing performance simulation on the target hardware based on the tensor program and the hardware performance model to generate the runtime data.

[0013] In one embodiment, the hybrid expert large language model is pre-trained based on pre-training data, which includes tensor program samples, compilation optimization strategy samples, and hardware performance annotation samples under different hardware backends. Different expert modules are trained based on the pre-training data of their respective hardware backends.

[0014] In a second aspect, embodiments of the present invention provide an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the AI ​​compiler automatic tuning method as described in any embodiment of the first aspect.

[0015] Compared with the prior art, the technical solution of this application has the following beneficial technical effects: This application applies a hybrid expert large language model to the tensor program generation stage of an AI compiler. By utilizing a gating network to automatically route to the corresponding expert module based on target hardware information, it can adaptively generate tensor programs and compilation optimization strategies for different hardware backends, achieving automatic code generation across hardware architectures. Simultaneously, this application constructs a reward model using real-world runtime data from the target hardware and optimizes the hybrid expert large language model through reinforcement learning, forming a closed-loop tuning mechanism. This allows the compiler tuning model to self-evolve end-to-end, reducing the computational resource consumption of large-scale searches, significantly shortening the deployment and tuning cycle of the AI ​​model on the target hardware, and reducing reliance on manually written high-performance operator libraries. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the AI ​​compiler automatic tuning method provided in the embodiments of the present invention; Figure 2 A schematic diagram illustrating the composition of semantic features provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the gated routing and expert module architecture of the hybrid expert big language model provided in an embodiment of the present invention; Figure 4 A schematic diagram of the reward feedback and model optimization closed-loop framework provided in this embodiment of the invention; Figure 5 This is a schematic diagram illustrating the composition of pre-training data provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] According to a first aspect of this application, embodiments of this application provide an automatic tuning method for an AI compiler. This method can be executed by an electronic device. Exemplarily, the electronic device executing the method provided in this application embodiment can be a server, workstation, computing device with compilation capabilities, etc. Figure 1 As shown, the main steps include: S110 generates structured prompts to represent model information based on the AI ​​model to be deployed to the target hardware.

[0019] Understandably, AI models contain a wealth of information related to compilation and optimization, such as the types of operators, the connections between operators, and the tensor dimensions of each operator's inputs and outputs. To pass this information to subsequent processing stages, it's necessary to extract key information from the AI ​​model and organize it into structured cue words. Compared to free text descriptions, structured cue words can convey the model's structural information more accurately and completely, thereby reducing information loss.

[0020] In some embodiments, the AI ​​model is parsed by a compiler front-end to extract operator information, tensor dimension information, data transfer relationships between operators, and relevant information about the target hardware. This information is then organized and encapsulated according to a preset structured format to obtain structured prompts. For example, the AI ​​model can be a model file in ONNX, TensorFlow, or PyTorch format; this application embodiment does not limit the specific format of the AI ​​model. The structured prompts can be encapsulated in JSON, XML, or other structured data formats.

[0021] For example, taking the ResNet50 model, information such as the operator type of the Conv2D operator, the shape of the input tensor (e.g., [1,64,56,56]), the shape of the weight tensor (e.g., [64,64,3,3]), stride, and padding, as well as the identification information of the target hardware (e.g., "NPU"), can be extracted from the model and encapsulated into structured prompts in JSON format. Similarly, for the YOLOv5s model, information related to multiple operators such as Conv, Pool, and Concat can be extracted and assembled into structured prompts.

[0022] S120, the structured prompts are input into a pre-trained hybrid expert large language model to obtain a tensor program and a compilation optimization strategy, wherein the gating network of the hybrid expert large language model is used to distribute the structured prompts to expert modules that match the target hardware, and the expert modules are used to generate the tensor program and the compilation optimization strategy based on the structured prompts.

[0023] Understandably, a hybrid expert large language model is a large language model architecture comprising multiple expert modules and a gating network. Different expert modules correspond to different hardware backends, and each expert module learns the code generation rules and compilation optimization strategies for its corresponding hardware backend during the pre-training phase. The gating network is responsible for routing structured prompts to the expert module matching the target hardware based on information in the input structured prompts. This hybrid expert architecture enables code generation for multiple hardware backends within a unified model, avoiding the overhead of training a separate model for each hardware.

[0024] When structured prompts are input into the hybrid expert large language model, the gating network determines the matching degree between each expert module and the current input based on the information contained in the structured prompts, and distributes the structured prompts to the expert module with the highest matching degree, i.e., the expert module that matches the target hardware. After receiving the structured prompts, this expert module generates a tensor program adapted to the target hardware based on the code generation knowledge for the corresponding hardware architecture learned during the pre-training phase. Simultaneously, this expert module also outputs a compiler optimization strategy. This strategy guides the compiler on which optimization operations to perform during code generation and the order in which these optimization operations are executed.

[0025] For example, the compilation optimization strategy can be a Pass sequence, such as ["op fusion", "loop tiling", "vectorize", "buffer hoisting"].

[0026] For example, when the target hardware information carried in the structured prompt is "NPU", the gating network distributes the structured prompt to the expert module corresponding to the NPU. The expert module outputs a tensor program adapted to the NPU and a compilation optimization strategy. Through this dynamic routing mechanism, automatic switching between different hardware backends can be achieved without manual intervention, and there is no need to traverse a large-scale scheduling search space. A tensor program and compilation optimization strategy can be output in a single inference.

[0027] S130, perform hardware performance evaluation on the tensor program to obtain the operating data of the target hardware, and generate a reward feedback signal based on the operating data.

[0028] The tensor program generated by the hybrid expert large language model needs to be validated on the target hardware to evaluate its actual performance. Obtaining real-world performance data through hardware performance evaluation and generating reward feedback signals based on this data can provide objective performance metrics for subsequent model optimization.

[0029] S140, optimize the hybrid expert large language model according to the reward feedback signal, and send the compilation optimization strategy and tensor program generated by the optimized hybrid expert large language model to the compiler backend so that the compiler backend can generate the executable file for the target hardware.

[0030] Understandably, the reward feedback signal reflects the actual performance of the tensor program generated by the hybrid expert large language model on the target hardware. By using the reward feedback signal to optimize the hybrid expert large language model, the model can adjust its code generation strategy based on the actual hardware performance, gradually generating tensor programs with better performance and implementing compilation optimization strategies.

[0031] In some embodiments, human feedback-based reinforcement learning (RLHF) can be used to fine-tune the hybrid expert large language model, for example, using the Proximal Policy Optimization (PPO) algorithm as the policy optimization algorithm. After accumulating a batch of reward samples from hardware operation, policy fine-tuning is performed to update the network parameters of the hybrid expert large language model.

[0032] For example, 500 sets of NPU operator hardware running reward samples can be collected to perform PPO strategy fine-tuning. After fine-tuning, for the same Conv2D operator input, the average inference time of the tensor program generated by the hybrid expert large language model can be reduced from 2.8ms to 2.0ms, and the compilation pass rate can be increased from 72% to 91%. The model relies on real hardware feedback for iterative optimization and does not require manual annotation of the optimal scheduling scheme.

[0033] Next, the optimized compiler optimization strategy and tensor program generated from the hybrid expert large language model are sent to the compiler backend. After receiving the tensor program and compiler optimization strategy, the compiler backend can further perform operations such as lowering, code generation, linking, and optimization, ultimately generating the executable file for the target hardware.

[0034] It should be noted that steps S110 to S140 above can be executed iteratively, meaning the above process can be repeated, allowing the hybrid expert large language model to continuously learn compilation optimization experience under different hardware backends. Through multiple iterations, the hybrid expert large language model can continuously accumulate compilation optimization experience, thereby enabling it to quickly generate adaptation solutions when new models, new operators, or new hardware platforms emerge, shortening deployment and tuning time.

[0035] For example, when faced with a brand-new custom operator, traditional solutions require engineers to manually develop scheduling code for 2 to 3 days; however, by using the method provided in this application, the relevant information of the operator is input, and after 3 to 5 rounds of closed-loop iteration, a runnable high-performance tensor program can be output in a total time of 10 to 20 minutes.

[0036] In some embodiments, the structured prompt word includes a target hardware identifier; the gating network is used to determine the routing weight based on the structured prompt word, and distribute the structured prompt word to the expert module corresponding to the target hardware identifier based on the routing weight.

[0037] Understandable, such as Figure 3 As shown, the target hardware identifier is information in the structured prompts that indicates the type of target hardware the AI ​​model needs to be deployed on, such as "GPU," "NPU," or "RISC-V vector processor." After receiving the structured prompts, the gating network can identify the target hardware identifier and, combined with other information in the prompts, calculate the routing weights for each expert module. The routing weights reflect the degree of matching between each expert module and the current input. Based on this, the gating network distributes the structured prompts to the expert module with the highest routing weight, i.e., the expert module corresponding to the target hardware identifier.

[0038] It should be noted that the method for calculating route weights can be designed according to actual needs. For example, the softmax function can be used to normalize the output of the gated network to obtain the route weights.

[0039] In some embodiments, the expert module includes at least one of a GPU expert module, an NPU expert module, and a RISC-V expert module.

[0040] The GPU expert module generates tensor programs containing optimized CUDA code for thread blocks, shared memory, and parallel scheduling based on structured prompts. Understandably, GPU hardware architectures possess massively parallel computing capabilities, and their programming models are typically based on a hierarchical structure of thread blocks and grids, utilizing shared memory to accelerate data exchange between threads. During pre-training, the GPU expert module learns code generation paradigms specific to the GPU architecture, enabling it to generate optimized CUDA code tailored to the GPU's thread hierarchy, shared memory access patterns, and parallel scheduling strategies. For example, for matrix multiplication operators, the GPU expert module can generate CUDA code that includes appropriate thread block partitioning, shared memory block loading, and parallel reduction operations.

[0041] The NPU expert module generates tensor programs that conform to the operator execution pattern of the target hardware based on structured prompts. It is understood that NPUs (Neural Processing Units) typically have dedicated operator execution units and specific data arrangement requirements, and their computational patterns differ significantly from those of general-purpose GPUs. The NPU expert module can generate tensor programs that conform to the NPU hardware operator execution pattern to fully utilize the NPU's hardware acceleration capabilities. For example, for convolution operators, the NPU expert module can generate tensor programs that conform to the NPU's on-chip cache management and operator scheduling patterns.

[0042] The RISC-V expert module generates tensor programs containing optimized code for vector access, vector computation, and data arrangement based on structured prompts. As can be understood, the RISC-V vector processor achieves data parallel processing through a vector instruction set, and its programming paradigm emphasizes vectorized data access and computation operations. The RISC-V expert module can generate code optimized for the vector instruction set, including vector load / store operations, vector computation operations, and data arrangement optimization. For example, for the element-wise addition operator, the RISC-V expert module can generate optimized code that utilizes vector registers for batch data loading, vector addition computation, and batch data storage.

[0043] It should be noted that the type and number of expert modules can be expanded according to the actual hardware backends that need to be supported. For example, it can also include FPGA accelerator expert modules, edge AI chip expert modules, etc.

[0044] In some embodiments, step S110 generates structured prompt words to represent model information based on the AI ​​model to be deployed to the target hardware, including: performing intermediate representation transformation on the AI ​​model to obtain multi-level IR; extracting semantic features from the multi-level IR; and encapsulating the semantic features in a structured manner to obtain structured prompt words.

[0045] Understandably, intermediate representation (IR) transformation is one of the core functions of a compiler frontend. By converting AI models into multi-level IR, key information such as operator information, data flow relationships, and tensor shapes in the model can be expressed in a unified format, facilitating subsequent feature extraction and analysis. Multi-level IR can express model information at different levels of abstraction, thus providing a rich source of information for semantic feature extraction.

[0046] For AI models in ONNX format, the onnx-mlir front-end conversion tool can be used to convert them to MLIR (Multi-Level Intermediate Representation) high-level IR. For AI models in TensorFlow format, the tensorflow-mlir front-end conversion tool can be used. Of course, the specific format of the multi-level IR can be selected according to actual needs.

[0047] After obtaining the multi-level information retrieval (IR), semantic features are extracted from it. Semantic features are key information in the multi-level IR that reflects the model structure and computational characteristics. Then, the extracted semantic features are encapsulated according to a predefined structured format to obtain structured cue words.

[0048] It should be noted that the specific format of the structured encapsulation can be designed according to actual needs, such as using JSON format, key-value pair format, etc., and this application embodiment does not limit it in this regard.

[0049] In some embodiments, such as Figure 2 As shown, semantic features include at least one of operator topology, tensor shape, data flow dependency, and target hardware identifier.

[0050] Understandably, operator topology describes the connections and execution order between operators in an AI model, reflecting the overall computational graph structure of the model. Tensor shape describes the dimensional information of the input and output tensors of each operator, such as batch size, number of channels, height, and width of the input tensor, providing a basis for determining appropriate data partitioning and parallelization strategies. Data flow dependency describes the data transfer relationships between operators, reflecting execution constraints and data reuse opportunities. Target hardware identifier indicates the type of target hardware the AI ​​model needs to be deployed on, providing a basis for expert routing in gating networks.

[0051] For example, for the Conv2D operator in the ResNet50 model, the semantic features that can be extracted include: information on the predecessor and successor operators of the operator in the operator topology, tensor shape as input tensor [1,64,56,56] and weight tensor [64,64,3,3], data flow dependency as the data transfer relationship between the operator and the predecessor operator, and target hardware identifier as "NPU".

[0052] It should be noted that semantic features may also include other information such as memory access patterns, operator types, and model task types. The specific types of semantic features included can be selected according to actual needs.

[0053] In some embodiments, generating a reward feedback signal based on the running data in step S130 includes: constructing a reward model based on the running data; and performing reinforcement learning based on the reward model to obtain the reward feedback signal.

[0054] The reward model is the core component that transforms runtime data into reward feedback signals. It calculates reward scores based on multiple dimensions of the runtime data, providing quantitative feedback signals for the optimization of hybrid expert large language models. By constructing a reward model and performing reinforcement learning based on it, hardware performance evaluation results can be transformed into signals that can be used to update model parameters, enabling automatic model optimization.

[0055] When constructing a reward model, corresponding weights and scoring rules can be set according to the importance of each indicator in the runtime data. For example, runtime can be set as the primary scoring indicator, with a higher reward score for shorter runtime; successful compilation can be used as a basic scoring condition, with a basic reward score awarded for successful compilation; and hardware resource utilization can be used as an auxiliary scoring indicator, with no points deducted if resource utilization is within a reasonable range, and points deducted if it exceeds the hardware limit.

[0056] Based on this, reinforcement learning is performed using the constructed reward model to obtain reward feedback signals. During the reinforcement learning process, the reward model calculates reward scores based on the running data, and these reward scores serve as reward feedback signals to guide the parameter update direction of the hybrid expert large language model.

[0057] In some embodiments, such as Figure 4 As shown, the runtime data includes at least one of the following: runtime of the tensor program, compilation success rate, hardware resource utilization, memory access efficiency, and operator parallelism. Reinforcement learning is performed based on a reward model to obtain reward feedback signals, including: inputting the runtime data into the reward model to calculate the reward score; and generating a negative reward feedback signal when the tensor program has compilation errors or hardware constraint violations.

[0058] Execution time reflects the execution speed of a tensor program on the target hardware and is a core performance indicator. Compilation success rate reflects the syntactic and semantic correctness of the generated code and is a fundamental guarantee of the usability of the tensor program. Hardware resource utilization reflects the efficiency of the tensor program in utilizing hardware computing and storage resources. Memory access efficiency reflects the degree of optimization of data access patterns and directly affects the actual runtime performance of the tensor program. Operator parallelism reflects the level of parallelization of computational tasks and affects the utilization of hardware computing resources. These indicators comprehensively evaluate the quality of tensor programs from different dimensions.

[0059] After the runtime data is input into the reward model, the model comprehensively evaluates various indicators according to preset scoring rules and outputs a quantified reward score. For example, for a Conv2D tensor program, a successful compilation earns a base score of +30, an inference time of 2.1ms corresponds to +60 points, and on-chip SRAM usage of 180KB does not exceed the hardware limit; therefore, the overall reward score is 82 points. A higher reward score indicates better performance of the tensor program.

[0060] Accordingly, when a tensor program exhibits compilation errors or violates hardware constraints, the reward model generates a negative reward feedback signal. For example, if the generated tensor program fails to compile, a negative reward of -50 can be output; similarly, if the tensor program's hardware resource usage exceeds hardware limits (e.g., on-chip SRAM usage exceeds available hardware capacity), a negative reward can also be applied. This negative reward feedback signal guides the hybrid expert large language model to avoid generating similar erroneous code or code that violates hardware constraints during subsequent optimization processes, thereby gradually improving the quality of code generation and hardware adaptability.

[0061] It should be noted that the specific calculation rules, score range, and specific values ​​of positive and negative rewards can be set according to actual needs.

[0062] In some embodiments, step S130, where the tensor program performs hardware performance evaluation to obtain runtime data of the target hardware, includes: deploying the tensor program to the corresponding target hardware and obtaining runtime data generated by the target hardware performing micro-benchmark tests; and / or, performing performance simulation on the target hardware based on the tensor program and the hardware performance model to generate runtime data.

[0063] Understandably, hardware performance evaluation can be achieved in two ways. The first is to actually deploy the tensor program to the target hardware and perform micro-benchmarking to obtain real-world runtime data. Micro-benchmarking is a performance testing method for a single operator or a small-scale computational task, capable of accurately measuring metrics such as the execution time and resource consumption of the tensor program on the target hardware. For example, the tensor program can be deployed to the target NPU to perform micro-benchmarking to collect runtime data such as operator inference time per run and on-chip SRAM usage.

[0064] The second approach involves using a hardware performance model to simulate the target hardware's performance when direct access is unavailable, generating simulated runtime data. A hardware performance model is a software tool that models the performance characteristics of the target hardware, such as its computing power, storage bandwidth, and latency. It can predict the performance of tensor programs without actually running the hardware. For example, when the target hardware is a new chip that is not yet in mass production, its hardware performance model can be used for performance simulation.

[0065] It should be noted that the two methods can also be used in combination to improve the accuracy and flexibility of the evaluation. For example, preliminary screening can be performed using hardware performance models, and then the selected candidate tensor programs can be deployed to the target hardware for micro-benchmarking to obtain accurate runtime data. The specific evaluation method used can be chosen based on actual conditions.

[0066] In some embodiments, such as Figure 5 As shown, the hybrid expert big language model is pre-trained based on pre-training data, which includes tensor program samples, compilation optimization strategy samples, and hardware performance annotation samples under different hardware backends. Different expert modules are trained based on the pre-training data of their respective hardware backends.

[0067] The pre-training phase is fundamental for the hybrid expert large language model to acquire code generation capabilities. By collecting high-quality training data from different hardware backends, the hybrid expert large language model is pre-trained, enabling each expert module to learn the code generation rules and compilation optimization strategies for its corresponding hardware architecture.

[0068] Tensor program samples are publicly available high-performance tensor program code under different hardware backends, such as open-source high-performance CUDA code under GPU hardware backends, operator implementation code under NPU hardware backends, and vectorized code under RISC-V vector processors. Compiler optimization strategy samples are combinations of compiler optimization passes under different hardware backends, such as pass sequences for GPUs and pass sequences for NPUs. Hardware performance annotation samples are data that annotates the runtime performance of tensor programs on corresponding hardware, such as performance metrics like runtime and resource consumption.

[0069] The following is a complete engineering implementation case study to further illustrate the automatic tuning method of the AI ​​compiler provided in this application.

[0070] For example, let's take the deployment of a YOLOv5s INT8 quantization model to an NPU as an example: First, step S110 is executed, the YOLOv5s ONNX model is input into the compiler front end, the AI ​​model is transformed into an intermediate representation to obtain the MLIR high-level IR, the semantic features in the multi-level IR are extracted, including the operator topology, tensor shape, data flow dependency, etc. of operators such as Conv, Pool, and Concat, the target hardware identifier is set to NPU, and the semantic features are encapsulated in a structured manner to obtain structured prompt words.

[0071] Next, step S120 is executed, where structured prompts are input into the pre-trained hybrid expert large language model. The gating network determines the routing weights based on the target hardware identifier in the structured prompts and distributes the structured prompts to the NPU expert module. The NPU expert module outputs the tensor programs of each operator in batches and outputs the entire set of compilation optimization strategies.

[0072] Accordingly, step S130 is executed, the tensor program is deployed to the target NPU to perform micro-benchmark tests, and runtime data such as runtime, compilation status, and SRAM usage of each operator are obtained. A reward model is constructed based on the runtime data, and reinforcement learning is performed based on the reward model to obtain reward feedback signals. Negative rewards are applied to abnormal generation results such as memory access out of bounds.

[0073] Based on this, step S140 is executed, where the PPO strategy is fine-tuned on the hybrid expert large language model according to the reward feedback signal, and the network parameters of the expert module are updated. This process is repeated 2 to 4 times to continuously improve the code generation quality. Finally, the optimal tensor program generated by the optimized hybrid expert large language model and the compilation optimization strategy are sent to the compiler backend, which generates the NPU executable firmware.

[0074] In this implementation case, traditional AutoSchedule search tuning of the entire model requires 22 hours of GPU search time; using the method provided in this application, the entire tuning closed loop can be completed in just 16 minutes; the model inference throughput is improved by 47% compared to the compiler's default implementation, and no manual operators need to be written.

[0075] According to a second aspect of this application, this application provides an electronic device comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the AI ​​compiler automatic tuning method as described in any of the above embodiments.

[0076] In some embodiments, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The processor may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA).

[0077] Regarding the electronic devices in the above embodiments, the specific manner in which the processor performs operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automatic tuning method for an AI compiler, characterized in that, include: Based on the AI ​​model to be deployed to the target hardware, generate structured prompts to represent model information; The structured prompts are input into a pre-trained hybrid expert large language model to obtain a tensor program and a compilation optimization strategy. The gating network of the hybrid expert large language model is used to distribute the structured prompts to expert modules that match the target hardware. The expert modules are used to generate the tensor program and the compilation optimization strategy based on the structured prompts. The tensor program is subjected to hardware performance evaluation to obtain the operating data of the target hardware, and a reward feedback signal is generated based on the operating data. The hybrid expert large language model is optimized based on the reward feedback signal. The compilation optimization strategy and the tensor program generated by the optimized hybrid expert large language model are sent to the compiler backend so that the compiler backend can generate the executable file for the target hardware.

2. The AI ​​compiler automatic tuning method according to claim 1, characterized in that, The structured prompt includes a target hardware identifier; the gating network is used to determine a routing weight based on the structured prompt and distribute the structured prompt to the expert module corresponding to the target hardware identifier based on the routing weight.

3. The AI ​​compiler automatic tuning method according to claim 2, characterized in that, The expert module includes at least one of a GPU expert module, an NPU expert module, and a RISC-V expert module; The GPU expert module is used to generate the tensor program containing CUDA code with optimizations for thread blocks, shared memory, and parallel scheduling based on the structured prompts. The NPU expert module is used to generate the tensor program that conforms to the operator execution mode of the target hardware based on the structured prompts. The RISC-V expert module is used to generate the tensor program containing vector access, vector computation, and data layout optimization code based on the structured prompts.

4. The AI ​​compiler automatic tuning method according to claim 1, characterized in that, The step of generating structured prompts to represent model information based on the AI ​​model to be deployed to the target hardware includes: The AI ​​model is subjected to intermediate representation transformation to obtain multi-level IR; Extract semantic features from the multi-level IR; The semantic features are encapsulated in a structured manner to obtain the structured prompt words.

5. The AI ​​compiler automatic tuning method according to claim 4, characterized in that, The semantic features include at least one of operator topology, tensor shape, data flow dependency, and target hardware identifier.

6. The AI ​​compiler automatic tuning method according to claim 1, characterized in that, The step of generating a reward feedback signal based on the operational data includes: A reward model is constructed based on the aforementioned operational data; Reinforcement learning is performed based on the reward model to obtain the reward feedback signal.

7. The AI ​​compiler automatic tuning method according to claim 6, characterized in that, The runtime data includes at least one of the following: runtime of the tensor program, compilation success rate, hardware resource utilization, memory access efficiency, and operator parallelism. The reinforcement learning based on the reward model to obtain the reward feedback signal includes: The operational data is input into the reward model to calculate the reward score; When the tensor program has compilation errors or hardware constraint violations, the reward model generates a negative reward feedback signal.

8. The AI ​​compiler automatic tuning method according to claim 1, characterized in that, The tensor program performs hardware performance evaluation to obtain the operating data of the target hardware, including: Deploy the tensor program to the corresponding target hardware and obtain the runtime data generated by the micro-benchmark test performed on the target hardware; and / or, Based on the tensor program and hardware performance model, the target hardware is subjected to performance simulation to generate the runtime data.

9. The AI ​​compiler automatic tuning method according to claim 1, characterized in that, The hybrid expert large language model is pre-trained based on pre-training data, which includes tensor program samples, compilation optimization strategy samples, and hardware performance annotation samples under different hardware backends. Different expert modules are trained based on the pre-training data of their respective hardware backends.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the AI ​​compiler automatic tuning method as described in any one of claims 1 to 9 when executing the computer program.

Images