Automatic operator optimization method, system and equipment based on multi-stage search and local optimization and medium
By employing a multi-stage search and local optimization method, the optimal scheduling strategy is generated and selected, solving the problems of high search cost and poor cross-platform generalization in operator optimization. This achieves efficient adaptive optimization of operator performance and supports the continuous evolution of multi-architecture inference frameworks.
Patent Information
- Application Number
- CN202511677280.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-02-27
AI Technical Summary
Existing technologies face challenges in operator optimization, such as high search costs, poor cross-platform generalization, and high coupling between model features and hardware behavior. These challenges make it difficult to achieve efficient search and adaptive optimization of performance modeling, thus affecting the continuous evolution of multi-architecture inference frameworks and high-performance computing systems.
A multi-stage search and local optimization method is adopted. By abstracting the characteristics of operators and hardware, multiple sets of candidate scheduling schemes are generated. An adaptive cost prediction model is used to quickly narrow the problem solution space, and local fine-grained search is used to improve the optimization accuracy and generate the optimal scheduling strategy.
It achieves operator performance adaptive optimization that balances efficient search, performance modeling, and versatility, supports the continuous evolution of multi-architecture inference frameworks and high-performance computing systems, shortens search time, and improves optimization accuracy.
Smart Images
Figure CN121579013A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to the technical field of operator optimization, in particular to an operator automatic optimization method, system, device and medium based on multi-stage search and local optimization. BACKGROUND
[0002] With the continuous improvement of the scale and complexity of deep learning models, the calculation optimization at the operator level has become a key link for improving the performance of model inference and training. Traditional operator optimization mainly relies on manual experience and fixed templates, which is difficult to adapt to the needs of heterogeneous hardware, multi-operator fusion and new compilation backends. Especially in GPU, NPU and ASIC architectures, different thread scheduling, shared memory layout, vectorization granularity and cache utilization strategies will significantly affect the performance. To cope with the complexity of this multi-dimensional design space, research has gradually shifted to automatic operator optimization technology. This kind of method is usually based on search and modeling mechanism, through the automatic generation of candidate scheduling schemes, performance evaluation and gradual optimization, to achieve efficient mapping on different hardware. The core is to guide the scheduling search by machine learning or heuristic algorithm, so that the system can automatically explore the optimal computation graph partitioning, loop transformation, memory access mode and parallel strategy at the operator level. However, the existing technology still faces problems such as high search cost, poor cross-platform generalization, high coupling degree of model features and hardware behavior, etc.
[0003] Therefore, how to realize adaptive optimization of operator performance with efficient search, performance modeling and generalization, and further support the continuous evolution of multi-architecture inference framework and high-performance computing system is a technical problem to be solved at present. SUMMARY
[0004] The technical task of the present application is to provide an operator automatic optimization method, system, device and medium based on multi-stage search and local optimization, to solve the problem of how to realize adaptive optimization of operator performance with efficient search, performance modeling and generalization, and further support the continuous evolution of multi-architecture inference framework and high-performance computing system.
[0005] The technical task of the present application is realized in the following way: an operator automatic optimization method based on multi-stage search and local optimization, which is to automatically generate multiple groups of candidate scheduling schemes, quickly narrow down the problem solving space through wide-area search, and improve the optimization accuracy by using local optimization, to realize adaptive optimization of operator performance; the specific implementation is as follows:
[0006] Abstractly represent the operator: abstractly represent the target operator as operator feature information and hardware feature representation;
[0007] Operator optimization template generation: abstract representation of target operator is converted into parameterized rules, which are used to define adjustable dimensions of operator computation on different levels of loops, parallelism and memory; each type of parameterized rule is composed of several adjustable parameters, which are used to guide code generation and scheduling strategy selection; by combining parameterized rules at different levels, a number of candidate operator optimization templates are automatically generated, each of which represents a specific optimization implementation;
[0008] Sample evaluation and adaptive cost prediction model learning: based on the generated operator optimization templates, sample evaluation is performed on the operator optimization templates, and an adaptive cost prediction model is established;
[0009] Local fine search: for each operator optimization template, all adjustable parameters are taken as a vector, and the rest of the parameters are fixed except one parameter, and the descending direction is explored along the corresponding parameter dimension. If a better parameter combination is found in any dimension, the current combination is updated and the process continues. If all dimensions cannot be improved, the process is terminated.
[0010] Output and integration: after optimization, the final scheduling description and its performance indicators are output, and the operator implementation embedded in the model inference framework is generated, and the specific model inference is performed using the inference framework.
[0011] As preferred, the operator feature information includes the dimension attributes of input and output tensors, the calculation type, and the dependency topology relationship between operators; among them, the dimension attributes of input and output tensors include batch size, channel number and height / width dimension; the calculation type includes matrix multiplication, convolution and normalization; the dependency topology relationship between operators includes loop nesting structure and data reuse dependency;
[0012] Hardware feature information includes thread parallelism of computing unit, shared memory and register level, cache block size, vector instruction width and memory bandwidth.
[0013] As preferred, the parameter rule includes the following types:
[0014] Loop partitioning rule: for each loop dimension in the operator (such as batch dimension N, channel dimension C, spatial dimension H / W, etc.), a loop splitting factor is defined, which is: the loop for i in [0, N) is split into two loops for i0 in [0, N / s) and for i1 in [0, s); where s is the splitting factor, which can take multiple candidate values (such as 2, 4, 8, etc.), which is used to control the block size or parallel granularity;
[0015] Loop rearrangement rule: allow different loop dimension to adjust execution order, such as (i,j,k) can be rearranged to (j,k,i), the order of different loop dimension affects cache locality and memory access pattern;
[0016] Parallel mapping rule: assign execution position for each layer of loop, such as outer loop is mapped to thread block, inner loop is mapped to thread, different mapping method is defined by mapping parameter, used to control parallel division of thread hierarchy;
[0017] Memory layout rule: define storage location of intermediate tensor (such as register, shared memory, global memory) and its access step or storage alignment; memory layout rule indicates usage strategy of different storage hierarchy through parameter, balance memory bandwidth and delay.
[0018] As a preferred, adaptive cost prediction model is used to predict operator delay without actual execution, thereby shortening search time and improving scheduling quality; specifically as follows:
[0019] Sample collection and performance evaluation: for each operator optimization template, select several representative parameter combinations (such as different loop division factors, thread mapping strategies, memory allocation schemes, etc.) from the operator optimization template parameterization space, use the code generator to automatically generate the corresponding low-level execution code, and actually run on the target hardware platform, measure the real execution time, and the measured performance indicators and operator optimization template parameters together constitute the training sample;
[0020] Adaptive cost prediction model construction and learning mechanism: adaptive cost prediction model is a performance prediction module based on machine learning, used to quickly estimate the execution cost of operator template without actual running;
[0021] Template screening based on adaptive cost prediction model: after training, adaptive cost prediction model is used to evaluate operator optimization templates that have not been executed, generate multiple representative parameter combinations for each operator optimization template, and input them into the adaptive cost prediction model to obtain the predicted execution time, calculate the comprehensive score, and sort the operator optimization templates according to the score, and select the top candidate operator optimization templates.
[0022] More preferably, the adaptive cost prediction model adopts a neural network model based on multilayer perceptron.
[0023] An operator automatic optimization system based on multi-stage search and local optimization, the system comprises:
[0024] An abstract representation operator module is configured to abstractly represent a target operator as operator feature information and hardware feature representation; the operator feature information comprises dimension attributes of input and output tensors, a calculation type, and a dependency topology relationship between operators; the dimension attributes of the input and output tensors comprise batch size, channel number, and height and width dimensions; the calculation type comprises matrix multiplication, convolution, and normalization; and the dependency topology relationship between operators comprises a loop nesting structure and a data reuse dependency; and the hardware feature information comprises thread parallelism of a calculation unit, a shared memory and register hierarchy, a cache block size, a vector instruction width, and a memory access bandwidth.
[0025] An operator optimization template generation module is configured to convert the abstract representation of the target operator into parameterized rules, and the parameterized rules are used to define adjustable dimensions of operator calculation on loops, parallelism, and different memory levels; each type of parameterized rule is composed of a plurality of adjustable parameters, and the adjustable parameters are used to guide code generation and scheduling strategy selection; a plurality of candidate operator optimization templates are automatically generated by combining the parameterized rules at different levels to arrange and combine loop dimensions, parallel mapping, and memory strategies; and each operator optimization template represents a specific optimization implementation manner.
[0026] A sample evaluation and adaptive cost prediction model learning module is configured to sample and evaluate the operator optimization templates on the basis of the generated operator optimization templates, and establish an adaptive cost prediction model.
[0027] A local fine search module is configured to, for each operator optimization template, fix all adjustable parameters as a vector, fix the remaining parameters except one parameter, and probe the descending direction along the corresponding parameter dimension; if a better parameter combination is found in any dimension, the current combination is updated and the process continues; if all dimensions cannot be improved, the process is terminated.
[0028] An output and integration module is configured to output a final scheduling description and its performance indicators after optimization, and generate an operator implementation manner embedded into a model inference framework, and use the inference framework to perform inference on a specific model.
[0029] As a preferred, the parameter rules comprise the following types:
[0030] A loop division rule: for each loop dimension (such as batch dimension N, channel dimension C, and spatial dimension H / W) in the operator, a loop splitting factor is defined, specifically: a loop for i in [0, N) is split into two loops for i0 in [0, N / s) and for i1 in [0, s); wherein s is the splitting factor, and the splitting factor can take a plurality of candidate values (such as 2, 4, 8, etc.), which is used to control the block size or parallel granularity;
[0031] Loop rearrangement rule: allows different loop dimensions to adjust the execution order, such as (i,j,k) can be rearranged to (j,k,i), the order of different loop dimensions affects cache locality and memory access patterns;
[0032] Parallel mapping rule: assign an execution position to each layer of loop, such as outer loop is mapped to thread block and inner loop is mapped to thread, different mapping methods are defined by mapping parameters, which are used to control the parallel division of thread hierarchy;
[0033] Memory layout rule: defines the storage location of intermediate tensor (such as register, shared memory, global memory) and its access step or storage alignment; memory layout rule indicates the usage strategy of different storage levels through parameters, balancing memory bandwidth and delay.
[0034] More preferably, the adaptive cost prediction model is used to predict the operator delay without actual execution, thereby shortening the search time and improving the scheduling quality; the adaptive cost prediction model is constructed as follows:
[0035] (1) Sample collection and performance evaluation: for each operator optimization template, select several representative parameter combinations (such as different loop division factors, thread mapping strategies, memory allocation schemes, etc.) from the operator optimization template parameterization space, use the code generator to automatically generate the corresponding low-level execution code, and run it on the target hardware platform to measure its real execution time; the measured performance indicators and operator optimization template parameters together constitute the training sample;
[0036] (2) Adaptive cost prediction model construction and learning mechanism: the adaptive cost prediction model is a performance prediction module based on machine learning, which is used to quickly estimate the execution cost of the operator template without actual running; wherein, the adaptive cost prediction model adopts a neural network model based on multilayer perception;
[0037] (3) Template screening based on adaptive cost prediction model: after training, the adaptive cost prediction model is used to evaluate the operator optimization templates that have not been executed, generate multiple representative parameter combinations for each operator optimization template, and input them into the adaptive cost prediction model to obtain the predicted execution time, calculate the comprehensive score, and sort the operator optimization templates according to the score, and select the top candidate operator optimization templates.
[0038] An electronic device, comprising: a memory and at least one processor;
[0039] Wherein, the memory stores a computer program;
[0040] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the operator automatic optimization method based on multi-stage search and local optimization as described above.
[0041] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program can be executed by a processor to implement the operator automatic optimization method based on multi-stage search and local optimization as described above.
[0042] The operator automatic optimization method based on multi-stage search and local optimization, system, device and medium of the application have the following advantages:
[0043] (I) The application automatically generates multiple groups of candidate scheduling schemes, and then realizes adaptive optimization of operator performance with efficient search, performance modeling and general-purpose performance by using rough optimization operator templates and fine optimization operator scheduling strategies, and supports continuous evolution of multi-architecture inference framework and high-performance computing system.
[0044] (II) The application automatically generates multiple groups of candidate scheduling schemes, and then realizes adaptive optimization of operator performance by using wide-area search to quickly narrow down the problem solving space and using local optimization to improve optimization accuracy, which has high innovation value and practical value. BRIEF DESCRIPTION OF DRAWINGS
[0045] The application will be further described below with reference to the accompanying drawings.
[0046] ATTACHMENT Figure 1 The flowchart of the operator automatic optimization method based on multi-stage search and local optimization. DETAILED DESCRIPTION
[0047] The operator automatic optimization method based on multi-stage search and local optimization, system, device and medium of the application will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] Embodiment 1:
[0049] As shown in the accompanying drawings, Figure 1 The embodiment provides an operator automatic optimization method based on multi-stage search and local optimization, which realizes adaptive optimization of operator performance by using wide-area search to quickly narrow down the problem solving space and using local optimization to improve optimization accuracy after automatically generating multiple groups of candidate scheduling schemes; the specific implementation is as follows:
[0050] S1, abstractly representing an operator: abstractly representing a target operator as operator feature information and hardware feature information; wherein the operator feature information includes dimension attributes of input and output tensors, a calculation type, and a dependency topology relationship between operators; wherein the dimension attributes of the input and output tensors include batch size, channel number, and height and width dimensions; the calculation type includes matrix multiplication, convolution, and normalization; and the dependency topology relationship between operators includes a loop nesting structure and a data reuse dependency; and the hardware feature information includes thread parallelism of a calculation unit, a shared memory and register hierarchy, a cache block size, a vector instruction width, and a memory access bandwidth;
[0051] S2, generating an operator optimization template: converting the abstract representation of the target operator into a parameterized rule, the parameterized rule being used to define adjustable dimensions of operator calculation on a loop, parallelism, and different memory levels; each type of parameterized rule is composed of a plurality of adjustable parameters, and the adjustable parameters are used to guide code generation and scheduling strategy selection; by combining the parameterized rules at different levels, the loop dimensions, parallel mapping, and memory strategies are arranged and combined to automatically generate a plurality of candidate operator optimization templates; each operator optimization template represents a specific optimization implementation mode;
[0052] S3, sample evaluation and adaptive cost prediction model learning: based on the generated operator optimization template, the operator optimization template is sampled and evaluated, and an adaptive cost prediction model is established;
[0053] S4, local fine search: for each operator optimization template, all adjustable parameters are taken as a vector, and the rest of the parameters are fixed except one parameter, and the descending direction is explored along the corresponding parameter dimension; if a better parameter combination is found in any dimension, the current combination is updated and continued; if all dimensions cannot be improved, it is terminated;
[0054] S5, output and integration: after optimization, the final scheduling description and its performance indicators are output, and an operator implementation mode embedded into a model inference framework is generated, and the inference framework is used to perform specific model inference.
[0055] The parameter rules in step S2 of the embodiment include the following types:
[0056] ① Loop division rule: for each loop dimension (such as batch dimension N, channel dimension C, and spatial dimension H / W) in the operator, a loop splitting factor is defined, specifically: the loop for i in [0, N) is split into two loops for i0 in [0, N / s) and for i1 in [0, s); wherein s is the splitting factor, which can take a plurality of candidate values (such as 2, 4, 8, etc.), and is used to control the block size or parallel granularity;
[0057] (ii) Loop rearrangement rule: allows different loop dimensions to adjust the execution order, such as (i, j, k) can be rearranged to (j, k, i), the order of different loop dimensions affects the cache locality and memory access pattern;
[0058] (iii) Parallel mapping rule: assigns an execution position to each layer of loop, such as the outer loop is mapped to the thread block, and the inner loop is mapped to the thread, different mapping methods are defined by mapping parameters, which are used to control the parallel division of thread hierarchy;
[0059] (iv) Memory layout rule: defines the storage location of intermediate tensors (such as registers, shared memory, global memory) and its access step or storage alignment; Memory layout rule indicates the usage strategy of different storage levels through parameters, balancing memory bandwidth and delay.
[0060] The adaptive cost prediction model in step S3 of the embodiment is used to predict the operator delay without actual execution, thereby shortening the search time and improving the scheduling quality; the specific process is as follows:
[0061] S301, sample collection and performance evaluation: for each operator optimization template, select several representative parameter combinations (such as different loop division factors, thread mapping strategies, memory allocation schemes, etc.) from the operator optimization template parameterization space, use the code generator to automatically generate the corresponding low-level execution code, and run it on the target hardware platform to measure the real execution time, and the measured performance indicators and operator optimization template parameters together constitute the training sample;
[0062] S302, adaptive cost prediction model construction and learning mechanism: the adaptive cost prediction model is a performance prediction module based on machine learning, which is used to quickly estimate the execution cost of the operator template without actual running; wherein the adaptive cost prediction model adopts a neural network model based on multilayer perception;
[0063] S303, template screening based on adaptive cost prediction model: after training, the adaptive cost prediction model is used to evaluate the operator optimization templates that have not been executed, generate multiple representative parameter combinations for each operator optimization template, and input them into the adaptive cost prediction model to obtain the predicted execution time, calculate the comprehensive score, and sort the operator optimization templates according to the score, and select a part of the candidate operator optimization templates with the highest score.
[0064] Embodiment 2:
[0065] The embodiment provides an operator automatic optimization system based on multi-stage search and local optimization, which comprises:
[0066] An abstract representation operator module is configured to abstractly represent a target operator as operator feature information and hardware feature representation; wherein the operator feature information comprises dimension attribute of input and output tensors, computation type, and dependency topology relationship between operators; wherein the dimension attribute of input and output tensors comprises batch size, channel number, and height and width dimensions; the computation type comprises matrix multiplication, convolution, and normalization; and the dependency topology relationship between operators comprises cyclic nested structure and data reuse dependency; and the hardware feature information comprises thread parallelism of a computing unit, shared memory and register hierarchy, cache block size, vector instruction width, and memory access bandwidth;
[0067] An operator optimization template generation module is configured to convert the abstract representation of the target operator into parameterized rules, and the parameterized rules are used to define adjustable dimensions of operator computation on a loop, parallelism, and different memory levels; each type of parameterized rule is composed of a plurality of adjustable parameters, and the adjustable parameters are used to guide code generation and scheduling strategy selection; by combining the parameterized rules at different levels, a plurality of candidate operator optimization templates are automatically generated by arranging and combining the loop dimensions, parallel mapping, and memory strategies; and each operator optimization template represents a specific optimization implementation mode;
[0068] A sample evaluation and adaptive cost prediction model learning module is configured to sample and evaluate the operator optimization templates on the basis of the generated operator optimization templates, and establish an adaptive cost prediction model;
[0069] A local fine search module is configured to, for each operator optimization template, fix all adjustable parameters as a vector, fix the remaining parameters except one parameter, and probe the descending direction along the corresponding parameter dimension; if a better parameter combination is found in any dimension, the current combination is updated and the process continues; if all dimensions cannot be improved, the process is terminated;
[0070] An output and integration module is configured to output the final scheduling description and performance indicators after optimization, and generate an operator implementation mode embedded into a model inference framework, and use the inference framework to perform inference on a specific model.
[0071] The parameter rules in the embodiment include the following types:
[0072] ① Loop division rule: for each loop dimension (such as batch dimension N, channel dimension C, and spatial dimension H / W) in the operator, a loop splitting factor is defined, specifically: the loop for i in [0, N) is split into two loops for i0 in [0, N / s) and for i1 in [0, s); wherein s is the splitting factor, and the splitting factor can take a plurality of candidate values (such as 2, 4, 8, etc.), which is used to control the block size or parallel granularity;
[0073] (ii) Loop rearrangement rule: allows different loop dimensions to adjust the execution order, such as (i,j,k) can be rearranged to (j,k,i), and the order of different loop dimensions affects cache locality and memory access patterns;
[0074] (iii) Parallel mapping rule: assigns an execution position to each layer of loop, such as outer loop is mapped to thread block and inner loop is mapped to thread, and different mapping methods are defined by mapping parameters to control parallel division of thread hierarchy;
[0075] (iv) Memory layout rule: defines the storage location of intermediate tensor (such as register, shared memory, global memory) and its access step or storage alignment; memory layout rule indicates the usage strategy of different storage levels through parameters to balance memory bandwidth and delay.
[0076] The adaptive cost prediction model in the embodiment is used to predict the operator delay without actual execution, thereby shortening the search time and improving the scheduling quality; the adaptive cost prediction model is constructed as follows:
[0077] (1) Sample collection and performance evaluation: for each operator optimization template, select several representative parameter combinations (such as different loop division factors, thread mapping strategies, memory allocation schemes, etc.) from the operator optimization template parameterization space, use the code generator to automatically generate the corresponding low-level execution code, and actually run on the target hardware platform to measure the real execution time, and the measured performance indicators and operator optimization template parameters together constitute the training sample;
[0078] (2) Adaptive cost prediction model construction and learning mechanism: the adaptive cost prediction model is a performance prediction module based on machine learning, which is used to quickly estimate the execution cost of the operator template without actual running; wherein, the adaptive cost prediction model adopts a neural network model based on multilayer perception;
[0079] (3) Template screening based on adaptive cost prediction model: after training, the adaptive cost prediction model is used to evaluate the operator optimization templates that have not been executed, generate multiple representative parameter combinations for each operator optimization template, input the adaptive cost prediction model, obtain the predicted execution time, calculate the comprehensive score, sort the operator optimization templates according to the score, and select a part of candidate operator optimization templates with the highest score.
[0080] Embodiment 3:
[0081] The embodiment also provides an electronic device, comprising: a memory and a processor;
[0082] The memory stores computer execution instructions.
[0083] The processor executes computer-executable instructions stored in the memory such that the processor performs the operator auto-optimization method based on multi-stage search and local optimization in any of the embodiments of the present application.
[0084] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.
[0085] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.
[0086] Embodiment 4:
[0087] The embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor to make the processor execute the operator auto-optimization method based on multi-stage search and local optimization in any of the embodiments of the present application. Specifically, a system or device provided with a storage medium can be provided, and software program codes realizing the functions of any of the above embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.
[0088] In this case, the program codes read from the storage medium can realize the functions of any of the above embodiments, and thus the program codes and the storage medium storing the program codes constitute a part of the present application.
[0089] The storage medium for providing the program codes includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program codes can be downloaded from a server computer via a communication network.
[0090] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.
[0091] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.
[0092] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for automatic optimization of operators based on multi-stage search and local optimization, characterized in that, The method is to quickly narrow down the problem solving space through wide-area search after automatically generating multiple groups of candidate scheduling schemes, and to improve optimization accuracy by using local optimization to realize adaptive optimization of operator performance; Specifically as follows: Abstract representation of operators: abstractly represent the target operator as operator feature information and hardware feature representation; Generate operator optimization templates: convert the abstract representation of the target operator into parameterized rules, which are used to define the adjustable dimensions of the operator calculation at different levels of loops, parallelism, and memory; Each type of parameterized rule consists of several adjustable parameters, which are used to guide code generation and scheduling strategy selection; By combining parameterized rules at different levels, the loop dimensions, parallel mapping, and memory strategies are arranged and combined to automatically generate several candidate operator optimization templates; Each operator optimization template represents a specific optimization implementation; Sample evaluation and adaptive cost prediction model learning: based on the generated operator optimization templates, sample and evaluate the operator optimization templates, and establish an adaptive cost prediction model; Local fine search: for each operator optimization template, all adjustable parameters are treated as a vector, and the remaining parameters are fixed except for one parameter, which is fixed. Explore the descending direction along the corresponding parameter dimension. If a better parameter combination is found in any dimension, update the current combination and continue. If all dimensions cannot be improved, terminate; Output and integration: after optimization, output the final scheduling description and its performance indicators, and generate the operator implementation embedded in the model inference framework to perform specific model inference using the inference framework.
2. The multi-stage search and local optimization based operator auto-optimization method of claim 1, wherein, Operator feature information includes the dimension attributes of input and output tensors, computation types, and dependency topology relationships between operators; Among them, the dimension attributes of input and output tensors include batch size, channel number, and height and width dimensions; The computation type includes matrix multiplication, convolution, and normalization; The dependency topology relationship between operators includes loop nesting structure and data reuse dependency; Hardware feature information includes thread parallelism of computing units, shared memory and register levels, cache block size, vector instruction width, and memory bandwidth.
3. The multi-stage search and local optimization based operator auto-optimization method of claim 1, wherein, Parameter rules include the following types: Loop partitioning rule: for each loop dimension in the operator, define the loop splitting factor, specifically: split the loop for i in [0, N) into two loops for i0 in [0, N / s) and for i1 in [0, s); Where s is the splitting factor, which can take multiple candidate values to control block size or parallel granularity; Loop rearrangement rule: allows different loop dimensions to adjust the execution order, and the order of different loop dimensions affects cache locality and memory access mode; Parallel mapping rule: assign an execution position to each loop level, and different mapping methods are defined by mapping parameters to control thread-level parallel partitioning; Memory layout rule: defines the storage location of intermediate tensors and their access step or storage alignment method; Memory layout rules use parameters to indicate the usage strategy of different storage levels to balance memory bandwidth and delay.
4. The multi-stage search and local optimization based operator auto-optimization method of claim 1, wherein, The adaptive cost prediction model is used to predict the operator delay without actual execution, thereby shortening the search time and improving the scheduling quality; Specifically as follows: Sample collection and performance evaluation: For each operator optimization template, several representative parameter combinations are selected from the parameterization space of the operator optimization template. The code generator automatically generates the corresponding low-level execution code and runs it on the target hardware platform to measure its actual execution time. The measured performance indicators and the operator optimization template parameters together constitute the training samples. Adaptive cost prediction model construction and learning mechanism: The adaptive cost prediction model is a performance prediction module based on machine learning, which is used to quickly estimate the execution cost of operator templates without actual running. Template selection based on adaptive cost prediction model: After training, the adaptive cost prediction model is used to evaluate operator optimization templates that have not yet been executed. Multiple representative parameter combinations are generated for each operator optimization template and input into the adaptive cost prediction model to obtain the predicted execution time and calculate the comprehensive score. The operator optimization templates are sorted according to the score, and the candidate operator optimization templates with the highest scores are selected.
5. The method of claim 4, wherein, The adaptive cost prediction model uses a neural network model based on a multilayer perceptron.
6. An operator auto-optimization system based on multi-stage search and local optimization, characterized in that, The system includes: The abstract representation operator module is used to abstractly represent the target operator as operator feature information and hardware feature representation. The operator feature information includes the dimensional attributes of the input and output tensors, the computation type, and the dependency topology between operators. Specifically, the dimensional attributes of the input and output tensors include batch size, number of channels, and height and width dimensions; the computation type includes matrix multiplication, convolution, and normalization; the dependency topology between operators includes nested loop structures and data reuse dependencies; and the hardware feature information includes the thread parallelism of the computation unit, shared memory and register hierarchy, cache block size, vector instruction width, and memory access bandwidth. The operator optimization template generation module is used to convert the abstract representation of the target operator into parameterized rules. The parameterized rules are used to define the adjustable dimensions of operator computation at different levels of loop, parallel, and memory. Each type of parameterized rule consists of several adjustable parameters, which are used to guide code generation and scheduling strategy selection. By combining parameterized rules at different levels, the loop dimension, parallel mapping, and memory strategy are arranged and combined to automatically generate several candidate operator optimization templates. Each operator optimization template represents a specific optimization implementation method. The sample evaluation and adaptive cost prediction model learning module is used to sample and evaluate the operator optimization template based on the generated operator optimization template, and to establish an adaptive cost prediction model. The local fine-grained search module is used to optimize the template for each operator. It treats all adjustable parameters as a vector, fixes all parameters except one, and explores the descent direction along the corresponding parameter dimension. If a better parameter combination is found in any dimension, the current combination is updated and the process continues; if no improvement can be made in any dimension, the process terminates. The output and integration module is used to output the final scheduling description and its performance indicators after optimization, and to generate operator implementations that are embedded in the model inference framework, so as to use the inference framework to perform inference on the specific model.
7. The multi-stage search and local optimization based operator auto-optimization system of claim 6, wherein, Parameter rules include the following types: Loop partitioning rule: For each loop dimension in the operator, define a loop splitting factor, specifically: split the loop for iin[0,N) into two loops for i0 in[0,N / s) and for i1 in[0,s); where s is the splitting factor, which can take multiple candidate values to control the block size or parallel granularity. Loop reordering rules: allow different loop dimensions to adjust the execution order, and the order of different loop dimensions affects cache locality and memory access patterns; Parallel mapping rules: Execution positions are allocated to each loop level. Different mapping methods are defined by mapping parameters, which are used to control the parallel partitioning of thread levels. Memory layout rules: Define the storage location of intermediate tensors and their access stride or storage alignment method; memory layout rules indicate the usage strategy of different storage levels through parameters, balancing memory access bandwidth and latency.
8. The multi-stage search and local optimization based operator auto-optimization system of claim 6 or 7, wherein, The adaptive cost prediction model is used to predict operator latency without actual execution, thereby shortening search time and improving scheduling quality. The specific construction process of the adaptive cost prediction model is as follows: (1) Sample collection and performance evaluation: For each operator optimization template, several representative parameter combinations are selected from the parameterization space of the operator optimization template. The corresponding low-level execution code is automatically generated by the code generator and actually run on the target hardware platform. The actual execution time is measured. The measured performance indicators and the operator optimization template parameters together constitute the training samples. (2) Adaptive cost prediction model construction and learning mechanism: The adaptive cost prediction model is a performance prediction module based on machine learning, which is used to quickly estimate the execution cost of the operator template without actual operation; among them, the adaptive cost prediction model adopts a neural network model based on multilayer perceptron; (3) Template selection based on adaptive cost prediction model: After training, the adaptive cost prediction model is used to evaluate the operator optimization templates that have not yet been executed. Multiple sets of representative parameter combinations are generated for each operator optimization template and input into the adaptive cost prediction model to obtain the predicted execution time and calculate the comprehensive score. The operator optimization templates are sorted according to the score, and the candidate operator optimization templates with the highest scores are selected.
9. An electronic device, comprising: include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the operator automatic optimization method based on multi-stage search and local optimization as described in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the operator automatic optimization method based on multi-stage search and local optimization as described in any one of claims 1 to 5.