Multi-operator optimization scheduling method, electronic device and storage medium

Abstract

This invention relates to the field of artificial intelligence technology, providing a multi-operator optimization scheduling method, electronic device, and storage medium. The method includes: acquiring performance data of the model to be optimized on target computing hardware; calculating the overall model speedup ratio based on the execution time proportion of each operator in the performance data, and sorting them to obtain an operator optimization priority queue; sequentially executing optimization iterations on each operator in the operator optimization priority queue as a target operator; acquiring the performance test results of the target operator in the current iteration cycle and evaluating them based on a preset termination criterion; if the preset termination criterion is met, stopping the optimization iteration of the target operator and scheduling the next operator to continue the optimization iteration. This invention achieves automated scheduling that balances global performance and single-operator computing resource utilization by prioritizing based on the overall model speedup ratio and automatically controlling the iteration termination of operators using a preset termination criterion, resulting in reasonable allocation of computing resources and improved overall optimization efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more particularly to a hardware computing performance optimization and intelligent scheduling technology, specifically to a multi-operator optimization scheduling method, electronic device, and storage medium. Background Technology

[0002] With the widespread application of deep learning models, the optimization and scheduling techniques of computing hardware (such as Graphics Processing Units, GPUs) are crucial for improving model running efficiency. Currently, optimization of GPU operators mainly relies on manual tuning, automatic tuning frameworks, or compiler optimization. For example, engineers manually adjust low-level parameters based on experience, or use automatic tuning tools to search for optimal configurations for individual operators, or use compilers to perform operator fusion at the macroscopic computational graph level.

[0003] However, the aforementioned optimization methods have certain limitations in practical applications. Specifically, these technologies are usually limited to local tuning of a single operator or macroscopic structural optimization, lacking a comprehensive approach to computational resources from the perspective of overall model performance. This can easily lead to excessive time being spent on operators that contribute little to global acceleration. Secondly, current tuning processes often rely heavily on manual intervention to assess the optimization status and manage the tuning progress, lacking a systematic automatic evaluation and tuning mechanism. This results in limited overall optimization efficiency and makes it difficult to adapt to the needs of large-scale, continuous tuning. Summary of the Invention

[0004] This invention provides a multi-operator optimization scheduling method, electronic device, and storage medium to address the shortcomings of related technologies, such as the lack of a global perspective in operator optimization leading to unreasonable resource allocation, and the high dependence on manual intervention resulting in low systematization and optimization efficiency.

[0005] This invention provides a multi-operator optimization scheduling method, comprising: Obtain the performance data of the model to be optimized on the target computing hardware. Calculate the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data. Sort the operators according to the total speedup ratio of the model to obtain the operator optimization priority queue. According to the order of each operator in the operator optimization priority queue, each operator is taken as the target operator, and optimization iteration is performed on the target operator; Obtain the performance test results of the target operator in the current iteration cycle, and evaluate the performance test results based on a preset termination criterion; If the evaluation result meets the preset termination criterion, the optimization iteration of the target operator is stopped, and the next operator is scheduled from the operator optimization priority queue as the new target operator to continue the optimization iteration.

[0006] According to a multi-operator optimization scheduling method provided by the present invention, the preset termination criterion is a multi-dimensional termination criterion, and the evaluation of the performance test results based on the preset termination criterion includes: The first evaluation result is obtained by evaluating whether the number of consecutive backtracking iterations in the optimization process has reached a preset threshold. The second evaluation result is obtained by evaluating whether the actual execution performance corresponding to the performance test result reaches the preset peak ratio of the theoretical peak performance of the target computing hardware. The third evaluation result is obtained by assessing whether the current cumulative optimization time of the target operator exceeds a preset time threshold. The fourth evaluation result is obtained by evaluating whether the actual speedup ratio corresponding to the performance test result reaches the preset speedup ratio threshold. The fifth evaluation result is obtained by evaluating whether the total speedup of the next operator in the operator optimization priority queue is higher than the speedup benefit brought by continuing to optimize the target operator. If any one of the first evaluation result, the second evaluation result, the third evaluation result, the fourth evaluation result, and the fifth evaluation result is yes, then the preset termination criterion is satisfied.

[0007] According to a multi-operator optimization scheduling method provided by the present invention, the step of performing optimization iteration on the target operator includes: The code of the target operator is edited according to a preset optimization strategy to obtain the optimized operator; The optimized operator is subjected to correctness verification and performance benchmark testing to obtain verification results and current performance data. The current performance data is then compared with the baseline performance data to obtain performance comparison results. Based on the verification results and the performance comparison results, the optimized operator is subjected to version retention or version rollback operations.

[0008] According to a multi-operator optimization scheduling method provided by the present invention, the step of performing version retention or version rollback operations on the optimized operator based on the verification results and the performance comparison results includes: If the verification result is passed and the performance comparison result is a performance improvement, then the code modification of the optimized operator is submitted through the version control system, and the status of the target operator in the current iteration cycle is updated to be retained; If the verification result is a failure or the performance comparison result is no performance improvement, the rollback operation of the version control system is triggered, and the code state of the target operator before the current iteration cycle is restored.

[0009] According to a multi-operator optimization scheduling method provided by the present invention, before the step of editing the code of the target operator according to a preset optimization strategy, the method further includes: Count the number of iterations that the target operator has executed; Based on the preset number of iteration cycles, a corresponding preset optimization strategy is selected from the candidate strategy set, which includes a large-range parameter adjustment strategy, a systematic parameter scanning strategy, and a fine-tuning strategy. If the evaluation result indicates a plateau and the actual execution performance corresponding to the performance test result does not reach the preset peak ratio, the preset optimization strategy will be switched to the underlying implementation refactoring strategy.

[0010] According to a multi-operator optimization scheduling method provided by the present invention, the step of calculating the total model speedup ratio corresponding to each operator based on the execution time ratio of each operator in the performance data, and sorting the operators according to the total model speedup ratio to obtain an operator optimization priority queue includes: For each operator in the model to be optimized, the total speedup ratio of the model corresponding to the operator is calculated based on the execution time ratio of the operator and the preset single operator speedup ratio. All operators are sorted in descending order according to the overall speedup ratio of the model to obtain the initial priority queue; According to the order of the operators in the initial priority queue, the cumulative model speedup brought about by simultaneously optimizing the first few operators is calculated in turn; When the cumulative model speedup reaches the target speedup threshold, the target truncation position is determined based on the number of operators optimized simultaneously, and the operators in the initial priority queue that are located before the target truncation position constitute the operator optimization priority queue.

[0011] According to a multi-operator optimization scheduling method provided by the present invention, after the steps of obtaining the performance test results of the target operator in the current iteration cycle and evaluating the performance test results based on a preset termination criterion, the method further includes: Write the current progress information of the optimization iteration, the current state information of the target operator, and the optimal speedup updated based on the performance test results into the progress record file; The configuration data for the current iteration cycle and the performance test results are appended to the experimental log file in a structured format. The progress record file and the experiment log file are both used to restore the state of the optimization iteration when the session is interrupted.

[0012] According to a multi-operator optimization scheduling method provided by the present invention, after the step of scheduling the next operator from the operator optimization priority queue as a new target operator to continue performing optimization iterations, the method further includes: After all operators in the operator optimization priority queue have completed optimization iterations, all optimized operators are replaced in the model to be optimized to obtain the optimized model. The optimized model is used to perform end-to-end inference computation on the target computing hardware, and the output results and total latency of the end-to-end inference computation are compared with the reference implementation to determine the overall computational correctness and end-to-end speedup of the optimized model. An aggregate report is generated based on the end-to-end speedup ratio, the optimization time of each operator, and the performance test results.

[0013] The present invention also provides a multi-operator optimized scheduling apparatus, comprising: The model analysis unit is used to obtain the performance data of the model to be optimized on the target computing hardware, calculate the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data, and sort the operators according to the total speedup ratio of the model to obtain the operator optimization priority queue. The operator optimization unit is used to select each operator as a target operator according to the order of the operators in the operator optimization priority queue, and to perform optimization iteration on the target operators. The performance evaluation unit is used to obtain the performance test results of the target operator in the current iteration cycle and evaluate the performance test results based on a preset termination criterion. An optimization scheduling unit is configured to stop the optimization iteration of the target operator if the evaluation result meets the preset termination criterion, and schedule the next operator from the operator optimization priority queue as a new target operator to continue the optimization iteration.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the multi-operator optimization scheduling method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-operator optimization scheduling method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the multi-operator optimization scheduling methods described above.

[0017] The multi-operator optimization scheduling method, electronic device, and storage medium provided by this invention acquire performance data of the model to be optimized on the target computing hardware, calculate the overall speedup of the model based on the execution time proportion of each operator in the performance data, and then construct an operator optimization priority queue. This allows the system to prioritize the allocation of computing resources to the operators that contribute the most to the overall performance from a global model perspective, thereby effectively avoiding unnecessary time wasted on low-impact local operators. Furthermore, this invention sequentially performs optimization iterations on each operator as the target operator according to the order of the operators in the operator optimization priority queue, and introduces a preset termination criterion to dynamically evaluate the performance test results in the current iteration cycle. When the evaluation result meets the preset termination criterion, the current optimization is automatically stopped and the system is scheduled to switch to the next operator. This solves the scheduling problem of when to stop or switch during the optimization process without relying on manual intervention and effectively avoids the waste of time and resources on hardware bottleneck operators. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the multi-operator optimization scheduling method provided by the present invention; Figure 2 This is a schematic diagram of the overall architecture of the multi-operator optimization scheduling system provided by the present invention; Figure 3 This is a schematic diagram of the single-operator optimized state machine provided by the present invention; Figure 4 This is a schematic diagram of the single-operator optimization iteration process provided by the present invention; Figure 5 This is a schematic diagram of the structure of the multi-operator optimization scheduling device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] With the widespread application of deep learning models, computational performance optimization and intelligent scheduling technologies for computing hardware play a crucial role in improving model running efficiency. Here, computing hardware can refer to GPUs, general-purpose graphics processing units (GPGPUs), neural network processing units (NPUs), etc. Taking GPUs as an example, current GPU operator optimization primarily relies on the following related technologies during the execution of deep learning models on GPUs: The first type is manual tuning, where engineers analyze and optimize operators one by one based on their personal experience, and try to improve computing performance by manually adjusting underlying parameters such as tile size, thread configuration, and memory access mode.

[0022] The second type is automatic tuning frameworks, such as Triton's autotune decorator, TVM (Tensor Virtual Machine)'s AutoTVM, or AutoScheduler. These frameworks can find the optimal configuration by searching the parameter space. However, these tools only work on a single operator and do not consider the model as a whole.

[0023] The third type is compiler optimization, such as deep learning compilers like TVM and XLA (Accelerated Linear Algebra), which mainly perform operator fusion and scheduling optimization at the computation graph level, but do not involve internal implementation tuning of operators.

[0024] However, the aforementioned technologies have certain drawbacks in practical applications. First, the automatic tuning frameworks only operate independently for individual operators, without considering the operator's weight in the overall model execution time; while compiler optimization, although focusing on the global picture, does not involve internal implementation tuning of operators. This means that the optimization process does not prioritize and allocate resources based on the operator's contribution to end-to-end latency, easily leading to a significant amount of time being spent optimizing low-impact operators.

[0025] Secondly, the current performance optimization process heavily relies on the continuous involvement of engineers, requiring manual decisions on when to stop tuning, when to switch optimization targets, and when to roll back code. The lack of systematic and objective judgment criteria leads to unreasonable resource allocation, often resulting in over-investment in bottleneck operators that are difficult to overcome, or under-investment in easily optimizable operators.

[0026] Furthermore, the related optimization work is usually done in a single or fragmented manner, and the optimization progress cannot be continuously recorded and restored across work sessions. When encountering environmental interruptions or when a session ends, each restart will lead to repetitive work, which greatly reduces the efficiency and systematic nature of GPU operator optimization.

[0027] To address this, the present invention provides a multi-operator optimization scheduling method. This method obtains the execution time percentage of each operator on the target computing hardware (such as a GPU) through model-level performance profiling, calculates the total speedup ratio of the model corresponding to each operator, prioritizes the multi-operators based on the total speedup ratio, performs autonomous closed-loop optimization iterations on the operators according to the priority order, and controls resource allocation by combining multi-dimensional termination criteria, thereby overcoming the above-mentioned defects.

[0028] It should be noted that the execution entity of the method provided by this invention can be a multi-operator optimization scheduling system, electronic device, computer device, or server, or a multi-operator optimization scheduling logic platform integrated into the above-mentioned physical devices. For ease of description, the multi-operator scheduling optimization system will be used as the execution entity in the following embodiments. The multi-operator optimization scheduling method provided by this invention is mainly aimed at performance tuning of deep learning models running on computing hardware such as GPUs, and aims to automatically complete the performance optimization closed loop of the entire model under unattended conditions.

[0029] Figure 1 This is a flowchart illustrating the multi-operator optimization scheduling method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S10: Obtain the performance data of the model to be optimized on the target computing hardware. Calculate the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data. Sort the operators according to the total speedup ratio of the model to obtain the operator optimization priority queue.

[0030] Specifically, the model to be optimized refers to the deep learning network or graph computing model that needs performance acceleration, while the target computing hardware mainly refers to hardware platforms with large-scale concurrent processing capabilities such as GPUs, GPGPUs, and NPUs.

[0031] The system first enters the analysis phase, using hardware performance profiling tools (such as torch.profiler for GPUs) to perform end-to-end analysis of the model to be optimized, thereby collecting performance data of the model running on the target computing hardware. Here, the collected performance data may include basic status indicators such as the execution time percentage of each underlying hardware operator, whether the operator is a compute-bound or memory-bound bottleneck, and the total latency of the model.

[0032] After obtaining the aforementioned performance data, to overcome the shortcomings of related technologies that only optimize single operators and ignore the overall model performance gain, this embodiment of the invention introduces Amdahl's Law as a mapping basis from operator-level acceleration to model-level acceleration. Specifically, the system extracts the proportion of each operator's execution time in the overall GPU time (i.e., the percentage of execution time) from the performance data. And under the unified assumption of preset acceleration conditions (e.g., assuming that a single operator can obtain...), Double acceleration, Based on the premise that the value can be 2), calculate the overall speedup of the model that each operator can bring to the overall model after optimization (i.e., Here, the overall model speedup refers to the factor by which the total time taken for end-to-end inference or runtime of the entire model to be optimized is improved relative to the theoretical performance before optimization, assuming that only a specific single operator is locally accelerated. From a global perspective, the overall model speedup accurately quantifies the true contribution (i.e., Amdahl influence) of a single operator's local acceleration to the overall model performance. Its calculation logic follows the Amdahl's Law formula as follows: In the above formula, This represents the proportion of the operator's execution time before optimization to the total model execution time (such as GPU time), i.e., the percentage of the operator's execution time. This represents the local speedup that the operator can independently achieve after local optimization. For example, the system can uniformly assume that all operators can achieve the same speedup after optimization. Acceleration, such as The value is 2.

[0033] The above formula shows the percentage of time spent. The higher the value of the operator, the greater its contribution to end-to-end performance. By calculating the Amdahl influence (i.e., the overall speedup of the model) of each operator, the system can sort all operators in the model in descending order according to the value of this influence, thereby generating an operator optimization priority queue that reflects the expected global optimization benefit.

[0034] Step S20: According to the order of each operator in the operator optimization priority queue, each operator is taken as the target operator, and optimization iteration is performed on the target operator.

[0035] Specifically, after determining the global optimization priority, i.e., obtaining the operator optimization priority queue, the system enters a fully autonomous closed-loop optimization phase. In this phase, the system schedules resources according to the top-down order of the operator optimization priority queue. The operator at the head of the queue or currently scheduled is determined as the current target operator.

[0036] For each target operator, the system automatically triggers optimization iterations within its corresponding independent state machine. These optimization iterations refer to systematically adjusting the underlying implementation of the operator, such as block size, thread configuration, and memory access mode parameters, and performing a cyclical process of code editing, testing, and feedback. This process is entirely handled by the system without human intervention, thus eliminating the high cost of continuous manual intervention required in traditional tuning.

[0037] Step S30: Obtain the performance test results of the target operator in the current iteration cycle, and evaluate the performance test results based on a preset termination criterion.

[0038] Specifically, since improving operator performance typically requires multiple parameter combinations and strategy adjustments, each modification constitutes a current iteration cycle. Within this cycle, the system performs benchmark tests on the modified and compiled operator to obtain its actual time consumption and throughput performance in that cycle, thus obtaining objective performance test results.

[0039] To effectively address the issue of over-investing in bottleneck operators or under-investing in easily optimizable operators, the system pre-configures a termination criterion within its state machine orchestration. This termination criterion is a multi-dimensional set of state judgment standards, which may include composite conditions such as hardware physical limit approximation, time budget thresholds, and continuous failure monitoring. It provides the system with a systematic basis for deciding when to abandon the current exploration. The system dynamically compares and evaluates the performance test results of each round against the termination criterion to accurately determine whether the optimization of the current target operator has reached its optimal level.

[0040] Step S40: If the evaluation result satisfies the preset termination criterion, then stop the optimization iteration of the target operator, and schedule the next operator from the operator optimization priority queue as the new target operator to continue the optimization iteration.

[0041] Specifically, when the system detects that the evaluation results of the above dimensions meet the preset termination criteria, it indicates that the current optimization of the target operator has reached a bottleneck, such as reaching the predetermined speedup expectation or approaching the hardware computing power limit, and continuing to allocate computing resources to it will be meaningless. At this time, the system's state machine will automatically transition, immediately stopping the optimization iteration of the current target operator, and marking the scheduling status of the operator as completed or skipped. Subsequently, the system extracts the next high Amdahl influence operator from the operator optimization priority queue according to the index, uses it as the new target operator, and restarts a new round of independent optimization iterations until the queue coverage is completed or the overall speedup is achieved.

[0042] The method provided in this invention acquires the performance data of the model to be optimized on the target computing hardware, calculates the overall speedup of the model based on the execution time ratio of each operator in the performance data, and then constructs an operator optimization priority queue. This allows the system to prioritize the allocation of computing resources to the operators that contribute the most to the overall performance from a global model perspective, thereby effectively avoiding unnecessary time wasted on low-impact local operators. Furthermore, this invention sequentially optimizes each operator as a target operator according to the order of the operators in the operator optimization priority queue, and introduces a preset termination criterion to dynamically evaluate the performance test results in the current iteration cycle. When the evaluation result meets the preset termination criterion, the current optimization is automatically stopped and the process is switched to the next operator. This solves the scheduling problem of when to stop or switch during the optimization process without relying on manual intervention and effectively avoids ineffective time and resource investment on hardware bottleneck operators.

[0043] Based on the above embodiments, in step S10, calculating the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data, and sorting the operators according to the total speedup ratio to obtain an operator optimization priority queue, includes: Step S11: For each operator in the model to be optimized, calculate the total speedup ratio of the model corresponding to the operator based on the execution time ratio of the operator and the preset single operator speedup ratio; Step S12: Sort all operators in descending order according to the total speedup ratio of the model to obtain the initial priority queue; Step S13: Calculate the cumulative model speedup ratio brought about by simultaneously optimizing the first few operators in the order of the operators in the initial priority queue. Step S14: When the cumulative model speedup reaches the target speedup threshold, determine the target cutoff position based on the number of operators optimized simultaneously, and form the operator optimization priority queue with the operators in the initial priority queue that are located before the target cutoff position.

[0044] Specifically, after obtaining end-to-end profiling data (i.e., performance data), it is assumed that the model contains... The system extracts the nth operator. The percentage of execution time for each operator And under the unified assumption, it achieves a preset single-operator speedup ratio (such as... Under the premise of (multiple speedups), the speedup mapping that the optimized operator would bring to the entire model is accurately calculated using Amdahl's law. The formula for calculating the overall speedup ratio of the model is as follows: in, Indicates the first The overall speedup of the model for each operator is calculated using the formula above. The system quantifies the Amdahl influence of each operator and sorts all operators in the model from largest to smallest based on this value, forming a basic recommendation processing order, i.e., the initial priority queue.

[0045] To evaluate the overall performance optimization gains from a holistic model perspective, the system further performs a cumulative impact analysis. This assumes that the top-ranked components in the queue are optimized simultaneously. All operators (all reaching) (Speed ​​increase), the system calculates its aggregated cumulative model speedup ratio. The specific calculation formula is as follows: The system pre-sets a desired end-to-end speedup threshold. Following the cumulative calculation process described above, the system searches for the minimum value that just brings the cumulative speedup to that threshold. Value, that The value represents the target truncation position. The system will then... Each operator is extracted to generate an operator optimization priority queue to guide subsequent autonomous scheduling, while low-impact operators after the truncation position are directly discarded.

[0046] The method provided in this invention calculates the total model speedup ratio corresponding to each operator and the cumulative model speedup ratio of multiple operators, and uses a threshold truncation mechanism to remove low-impact operators. This not only determines the number of recommended operators for optimization at the global level, but also determines the stopping boundary for model-level performance optimization. Thus, while ensuring that the overall end-to-end model speedup meets the target, it maximizes the saving of computational overhead and time cycle for subsequent automated optimization.

[0047] Based on any of the above embodiments, step S20, which involves performing optimization iteration on the target operator, includes: Step S21: Edit the code of the target operator according to the preset optimization strategy to obtain the optimized operator; Step S22: Perform correctness verification and performance benchmark testing on the optimized operator to obtain verification results and current performance data, and compare the current performance data with the baseline performance data to obtain performance comparison results; Step S23: Based on the verification results and the performance comparison results, perform version retention or version rollback operations on the optimized operator.

[0048] Specifically, for each target operator, in a single experimental loop (also known as a single optimization iteration, such as a closed-loop process lasting approximately 90 seconds), the system first invokes a preset optimization strategy. This preset optimization strategy refers to a set of rules built into the system to guide modifications to the underlying logic of the operator, such as adjusting block size, changing thread configurations, and modifying memory access modes. The system automatically generates the corresponding parameter combinations based on this strategy and automatically edits and rewrites the source code of the target operator without manual intervention, thereby compiling and generating the optimized operator to be verified.

[0049] After code editing is complete, it's crucial to ensure that performance improvements are based on accurate calculations. Therefore, the system first performs a preliminary correctness verification between the optimized operator's output and a reference standard implementation (such as native PyTorch output) to ensure that any performance improvements haven't introduced computational errors, resulting in a pass or fail verification result. Based on this, the system performs a performance benchmark test on the operator, running it on the target computing hardware and collecting its current performance data (such as execution latency and throughput). Subsequently, the system extracts the best performance record retained from before optimization or the previous iteration as baseline performance data and compares the two to determine whether the code modification has brought positive benefits, thus obtaining a quantified performance comparison result.

[0050] After obtaining the above two key results, the system's orchestrator state machine will use them as the basis for decision-making. By comprehensively judging whether the verification is passed and whether the performance is improved, it will decide whether to perform version retention or version rollback for the code changes generated in this round, thus completing the single operator experiment in a closed loop.

[0051] The method provided in this invention transforms the tedious process of manually rewriting, compiling, testing, and comparing traditionally relied upon by engineers into an automated single-operator experimental loop process. By introducing pre-testing for correctness verification before performance testing, it not only improves the trial-and-error iteration efficiency of the underlying operator code but also avoids the risk of calculation errors in model inference due to blindly pursuing performance, thus ensuring the reliability and effectiveness of the optimization direction.

[0052] Based on any of the above embodiments, step S23 specifically includes: Step S231: If the verification result is passed and the performance comparison result is a performance improvement, then submit the code modification of the optimized operator through the version control system, and update the state of the target operator in the current iteration cycle to retain. Step S232: If the verification result is a failure or the performance comparison result is no performance improvement, then the rollback operation of the version control system is triggered, and the code state of the target operator before the current iteration cycle is restored.

[0053] Specifically, when the system determines that the verification result is passed and the performance comparison result shows a performance improvement, it indicates that the currently generated optimized operator not only meets the requirement of correct calculation results but also outperforms the previous baseline record in terms of time consumption. At this point, it can be determined that this iteration is a successful exploration. The system will automatically call the underlying version control system (such as Git) to commit and solidify the code changes, ensuring that this experiment has traceability and reproducibility. At the same time, the current state of the target operator is marked as retained in the internal state machine, and the corresponding optimal baseline data is updated to establish a higher starting point for the next round of optimization exploration.

[0054] Conversely, if the verification result fails or the performance comparison shows no improvement during testing, it indicates a deviation in the operator's output (such as decreased calculation accuracy or errors), or that although the result is correct, the time consumption has not decreased or even performance has degraded. In this case, the system determines that the optimization exploration is invalid. At this time, the system will trigger the undo or checkout function of the version control system to perform an automated rollback operation, restoring the underlying code of the operator to the state before the start of this iteration, in order to eliminate the traces of this failed edit and thus avoid polluting the original high-quality codebase.

[0055] The method provided in this invention integrates the version control system into the underlying logic of the multi-operator autonomous optimization scheduling process, providing secure and traceable code-level protection for every unattended code edit and performance trial and error. This enables the system's iteration process to not only automatically accumulate successful optimization experience, but also to recover when faced with optimization failures, thereby enhancing the robustness of the system's continuous operation.

[0056] Based on any of the above embodiments, before step S21, the method further includes: Count the number of iterations that the target operator has executed; Based on the preset number of iteration cycles, a corresponding preset optimization strategy is selected from the candidate strategy set, which includes a large-range parameter adjustment strategy, a systematic parameter scanning strategy, and a fine-tuning strategy. If the evaluation result indicates a plateau and the actual execution performance corresponding to the performance test result does not reach the preset peak ratio, the preset optimization strategy will be switched to the underlying implementation refactoring strategy.

[0057] Specifically, in order to achieve efficient exploration within the parameter search space of a single operator, this embodiment of the invention also provides an adaptive adjustment process for the preset optimization strategy before editing the code of the target operator according to the preset optimization strategy. Specifically, since the optimization of each operator is a continuous trial-and-error process, the system records in real time the number of experiments the current target operator has undergone, i.e., the number of iteration cycles executed.

[0058] Subsequently, based on the preset number of iterations within a certain range, the system can select a corresponding preset optimization strategy from the candidate strategy set. Here, the candidate strategy set is a collection of tuning methods built into the system for different optimization stages, and the preset number of iterations is used to divide the lifecycle of the exploration. For example, when the number of iterations (i.e., the number of iteration cycles) is in the initial stage of 0-10 iterations, the system selects a large-range parameter adjustment strategy (also known as an aggressive strategy), which will significantly adjust the block size and try different execution modes; when it is in the middle stage of 11-30 iterations, the system switches to a systematic parameter scanning strategy (also known as a focused strategy), which performs systematic parameter traversal within a local space; when the number of experiments reaches more than 30 iterations, the system adopts a fine-tuning strategy (also known as a refined strategy), which fine-tunes and combines the successfully verified technical features.

[0059] In actual optimization, if the orchestrator detects that the current optimization has plateaued (e.g., due to multiple consecutive rollbacks), and the actual execution performance of the operator is still significantly lower than the hardware peak performance (e.g., performance is less than 50% of the theoretical peak), it indicates that the current direction may be trapped in a local optimum. At this point, the system will trigger an aggressive shift, switching the strategy to a low-level implementation refactoring strategy. This involves rewriting the operator logic from the ground up using a fundamentally different implementation method, thereby breaking the current performance bottleneck.

[0060] Furthermore, if the orchestrator detects that the current optimization has also plateaued, but the actual execution performance corresponding to the performance test results has reached a high percentage of the preset peak performance, such as reaching or exceeding 80% of the theoretical peak performance of the hardware, it indicates that the current operator performance is approaching the physical limits of the hardware. At this time, the system will trigger the acceptance policy, that is, it will no longer attempt to refactor the underlying code, but will directly retain the currently obtained optimal code implementation, and trigger the system's termination criteria to end the exploration of the operator, thus avoiding the waste of computing power.

[0061] The method provided in this invention introduces an adaptive strategy switching mechanism based on the number of iteration cycles and performance status, which can dynamically adjust the search direction according to the optimization progress, thereby improving the flexibility and success rate of single operator search space traversal and avoiding the technical problem that traditional automatic tuning frameworks are prone to getting stuck in local bottlenecks and wasting computing power.

[0062] Based on any of the above embodiments, the preset termination criterion is a multi-dimensional termination criterion. Accordingly, in step S30, evaluating the performance test results based on the preset termination criterion includes: The first evaluation result is obtained by evaluating whether the number of consecutive backtracking iterations in the optimization process has reached a preset threshold. The second evaluation result is obtained by evaluating whether the actual execution performance corresponding to the performance test result reaches the preset peak ratio of the theoretical peak performance of the target computing hardware. The third evaluation result is obtained by assessing whether the current cumulative optimization time of the target operator exceeds a preset time threshold. The fourth evaluation result is obtained by evaluating whether the actual speedup ratio corresponding to the performance test result reaches the preset speedup ratio threshold. The fifth evaluation result is obtained by evaluating whether the total speedup of the next operator in the operator optimization priority queue is higher than the speedup benefit brought by continuing to optimize the target operator. If any one of the first evaluation result, the second evaluation result, the third evaluation result, the fourth evaluation result, and the fifth evaluation result is yes, then the preset termination criterion is satisfied.

[0063] Specifically, in the autonomous optimization loop, to systematically determine when to abandon the optimization of the current target operator, the system orchestrator performs multiple-dimensional state checks in parallel after each operator experiment loop. Specifically, the system evaluates whether any one of the following termination conditions is met: First, the system evaluates whether the number of consecutive rollbacks in the optimization iterations has reached a preset threshold, thus obtaining the first evaluation result. This step is equivalent to plateau detection. The system monitors the effectiveness of code modifications. When it finds that modifications to the target operator have failed to bring performance improvements multiple times and have been rolled back, for example, when the number of consecutive rollbacks reaches 5, it indicates that the current optimization strategy for that operator may have been exhausted and has fallen into a local optimum plateau.

[0064] Secondly, the evaluation results determine whether the actual execution performance corresponding to the performance test results reaches a preset peak percentage of the theoretical peak performance of the target computing hardware, thus obtaining a second evaluation result. Here, the preset peak percentage refers to the degree to which the operator's measured throughput or computing power utilization approaches the hardware limit. For example, when the operator's actual execution performance reaches more than 90% of the GPU's theoretical peak performance, it means that the hardware limit has been reached, and the room for further optimization is extremely small; forced optimization is not very meaningful.

[0065] Third, the evaluation assesses whether the current cumulative optimization time of the target operator exceeds a preset time threshold, yielding a third evaluation result. This criterion is a global protection mechanism based on time budgeting, setting a hard time limit for each operator, such as a system default time limit of 120 minutes for a single operator. This time-limited blocking effectively prevents the system from endlessly consuming computing power on a single abnormal or difficult-to-optimize operator, thus preventing excessive occupation of global resources.

[0066] Fourth, evaluate whether the actual speedup corresponding to the performance test results reaches the preset speedup threshold to obtain the fourth evaluation result. After several iterations, if the actual speedup of the operator has reached the preset speedup threshold, such as reaching a speedup of more than or equal to 2 times the baseline speedup, it indicates that significant performance gains have been achieved on the current operator, and the optimization can be terminated at this time.

[0067] Fifth, the overall speedup of the next operator in the operator optimization priority queue is evaluated to determine whether it exceeds the speedup benefit of continuing to optimize the target operator, thus obtaining the fifth evaluation result. This step is based on the dynamic diminishing returns detection of Amdahl's Law. The system compares in real time the potential performance improvement from continuing to optimize the current target operator with the Amdahl influence of the next operator in the scheduling queue that is awaiting optimization. If the theoretical influence of the next operator is found to be higher, it indicates that the system's computing resources should be reallocated to the next operator to obtain a greater performance improvement.

[0068] Under the aforementioned multi-dimensional monitoring and state machine operation, the system can perform full evaluations of the multiple termination conditions in parallel, or it can adopt a sequential judgment logic. As long as the evaluation finds that any of the above termination conditions are met, that is, any of the above evaluation results is true, the preset termination criterion can be determined, and the system will then trigger a state transition, ending the optimization iteration of the current target operator. It should be understood that, regardless of whether parallel evaluation or sequential judgment is used, as long as the evaluation finds that any of the above termination conditions are met, the evaluation of other termination conditions can be stopped, and a state transition can be directly triggered.

[0069] The method provided in this invention introduces the above-mentioned multi-dimensional termination criterion evaluation mechanism, enabling the system to switch scheduling targets when it discovers exhaustive strategies, approaches hardware limits, or timeouts. This allows the limited computing resources to be allocated to the operator with the greatest potential for profit at the global level, achieving a balance between single operator tuning and overall model performance improvement.

[0070] Based on any of the above embodiments, after step S30, the method further includes: Write the current progress information of the optimization iteration, the current state information of the target operator, and the optimal speedup updated based on the performance test results into the progress record file; The configuration data for the current iteration cycle and the performance test results are appended to the experimental log file in a structured format. The progress record file and the experiment log file are both used to restore the state of the optimization iteration when the session is interrupted.

[0071] It should be noted that, in order to prevent progress loss due to unexpected power outages, process crashes, or manual pauses during the fully automated unattended optimization process, this embodiment of the invention introduces a dual-file state persistence and recovery mechanism.

[0072] Specifically, the current progress information of the optimization iteration, the current state information of the target operator, and the optimal speedup updated based on the performance test results are first written to the progress log file. This progress log file (e.g., migration_progress.md file) is used to save a snapshot of the state at a macro level. The system will save the current optimization stage, the last specific action executed, the currently active target operator, the best speedup achieved, the number of experiments executed, the current blocking factors, and the next plan, as well as other progress and state information, to this file in real time.

[0073] Secondly, the configuration data and performance test results of the current iteration cycle are appended to the experiment log file in a structured format. Here, the experiment log file (e.g., results.tsv file, i.e., tab-delimited value file) is mainly responsible for storing the underlying data at the micro-level. The system will append the specific parameter configuration data used in this round of experiment and the measured performance latency and other results data to this log file in a structured format, line by line, to form a detailed record of each experiment.

[0074] Understandably, both progress log files and experiment log files can be used to restore the state of optimization iterations when a session is interrupted. In other words, whether it is a macro-level progress log file or a micro-level experiment log file, either file can independently support the system to restore to its complete state before the interruption, ensuring the continuity of optimization across sessions.

[0075] The method provided in this embodiment of the invention, through the above-mentioned dual-file persistence mechanism, not only provides detailed structured knowledge accumulation for subsequent optimization schemes, but also achieves seamless continuity across work sessions, solving the technical problem that each restart in the traditional optimization process leads to repetitive work and wasted time.

[0076] Based on any of the above embodiments, after step S40, the method further includes: Step S50: After all operators in the operator optimization priority queue have completed optimization iterations, replace all optimized operators in the model to be optimized to obtain the optimized model. Step S60: Perform end-to-end inference computation on the target computing hardware using the optimized model, and compare the output results and total latency of the end-to-end inference computation with the reference implementation to determine the overall computational correctness and end-to-end speedup of the optimized model. Step S70: Generate an aggregate report based on the end-to-end speedup ratio, the optimization time of each operator, and the performance test results.

[0077] Specifically, after the automated optimization loop is completed, the system needs to perform a final end-to-end integration verification. Specifically, when the scheduling queue (i.e., the operator optimization priority queue) is cleared, it means that all the set target operators have undergone system optimization. At this point, the system replaces the basic operators at the corresponding positions in the original model with self-tuned high-performance operator code, thereby assembling the optimized model.

[0078] The high efficiency of a single operator does not guarantee the absolute reliability of the assembled model. Therefore, the system runs a complete inference task on the GPU from the perspective of the entire model. The system extracts the model output at this point and compares it with the results of the native framework (such as the reference implementation PyTorch) to verify the overall computational correctness. At the same time, it collects the actual total latency of the model at this point and calculates the actual end-to-end speedup ratio by combining it with the baseline latency before optimization. It can also compare this with the theoretical speedup ratio predicted based on Amdahl's Law.

[0079] After successful verification, the system automatically aggregates all logs and performance data, outputting a detailed aggregate report. This report includes baseline performance comparisons, speedup ratios, number of experiments, code retention rate, and the measured end-to-end speedup ratio for the entire model for each operator. It also displays time investment analysis and margin analysis for each operator, indicating which operators may still have optimization potential.

[0080] The method provided in this invention, through the above-mentioned integrated verification and report generation mechanism, ensures that no deep performance optimization will introduce calculation errors that affect business operations. At the same time, the automatically generated aggregated report compresses the model-level optimization cycle from several weeks to several hours, avoiding optimization quality fluctuations caused by differences in engineer experience.

[0081] Based on any of the above embodiments Figure 2 This is a schematic diagram of the overall architecture of the multi-operator optimization scheduling system provided by the present invention, as shown below. Figure 2 As shown, the multi-operator optimization scheduling system provided in this embodiment of the invention is divided into three pipeline stages in its overall architecture: stage A (model analysis and optimization plan generation), stage B (fully autonomous optimization), and stage C (integration verification).

[0082] In Phase A, the system primarily performs model analysis and optimization plan generation. First, the system invokes the model profiler to collect end-to-end performance data on the model to be optimized on the target computing hardware (e.g., GPU). The collected data is then input into the operator sorter. At this stage, the system calculates the GPU time percentage for each operator and passes it to the optimization plan generator. This generator uses Amdahl's Law to calculate the overall model speedup ratio (i.e., Amdahl influence) for each operator and generates an operator optimization priority queue in descending order. During this phase, the system allows for a final window of manual intervention, where users can manually determine the desired range of operators to optimize and the target speedup threshold.

[0083] Once the optimization plan is confirmed, the system enters unattended phase B. The core component of this phase is the orchestrator state machine, which is responsible for scheduling underlying resources and controlling... Figure 3 The lifetimes of each operator are shown. Figure 3 This is a schematic diagram of the single-operator optimized state machine provided by the present invention, as shown below. Figure 3 As shown, when an operator is added to the optimization queue, its initial state is marked as pending. As the scheduling process progresses, when the operator is selected and optimization begins, its state transitions to optimizing. The target operator in the optimizing state will continue to be optimized... Figure 4 In the single-operator experimental loop shown, if the orchestrator evaluation finds that the loop meets the termination condition (e.g., performance peak has been reached or the strategy has been exhausted), the operator state will change to "done"; otherwise, if an unforeseen system error or underlying architecture limitation prevents optimization, it will change to "skipped".

[0084] Figure 4 This is a schematic diagram of the single-operator optimization iteration process provided by the present invention, as shown below. Figure 4As shown, the operator in the optimizing state will execute Figure 4 The single-operator experimental closed loop is shown. In one experimental cycle, the system will perform the following steps: Step S1, Assumption: The system first determines the direction of this optimization based on performance data (also known as Roofline data, which is the performance theoretical model data that reflects the computation memory access ratio and the upper limit of hardware computing power).

[0085] Step S2, Edit: Subsequently, based on the aforementioned assumptions, the system performs a focused, automated modification to the kernel function code of the operator.

[0086] Step S3, Commit: Before conducting the actual test, the system calls the version control tool to execute git commit, recording a snapshot of the code for this experiment.

[0087] Step S4, Run: The system performs correctness verification and performance benchmark tests on the modified code.

[0088] Step S5, Analysis: After the test is completed, the system automatically extracts the generated performance test results and correctness verification results.

[0089] Step S6, Decision: The system makes a decision based on the extracted results. If the correctness verification fails, the code rollback command `git reset` is triggered, discarding the erroneous code. If the correctness verification passes but there is no performance improvement, the `git reset` rollback mechanism is also triggered to avoid code redundancy. Only when the correctness verification passes and the performance improvement is ≥1% (1% here is an example threshold) will the system determine it as a valid optimization and retain it.

[0090] Step S7, Recording and Checking: Regardless of the outcome of the above determination, the complete status of this experiment will be recorded in detail in the results.tsv log file (i.e., Figure 2 (The experiment recorder component in the middle). After recording is completed, the orchestrator checks the termination condition. If the termination condition has not been met, it sends a CONTINUE instruction to return to the first step and continue to propose new hypotheses. If the current operator has met the termination criterion, it issues a NEXT instruction to switch to the next operator. When all operators in the queue have been processed, it issues a DONE instruction to drive the system to jump out of stage B and enter stage C.

[0091] Finally, the system transitioned to Figure 2Phase C of the process. In this phase, the system integrates all optimal operator variants that have undergone multiple iterations and been retained into the original model, performing comprehensive end-to-end verification to ensure the correctness of local optimizations from a global perspective and the improvement in macroscopic performance. After verification, the system compiles all optimization trajectories, performance gains, and state statistics of each operator, generating and outputting a visualized aggregate report.

[0092] This invention, through the construction of a three-stage closed-loop architecture encompassing model analysis, autonomous optimization iteration, and integrated verification, and embedding an operator lifecycle state machine and a single operator experimental loop process, not only achieves seamless integration of GPU operators from theoretical sorting to code modification, but also endows the system with high trial-and-error efficiency and self-correction capabilities without manual intervention. This ensures that, within limited computing resources and time, the performance limits of deep learning models can be accurately identified, and maximum end-to-end acceleration can be safely and reliably obtained.

[0093] Furthermore, it should be noted that the multi-operator optimization scheduling method provided in this embodiment of the invention is not an abstract mathematical calculation rule, but a technical solution for system-level optimization of the underlying computing resource allocation and computing power scheduling mechanism of specific target computing hardware (such as GPU, GPGPU, etc.). By automatically scheduling and reconstructing the underlying machine instructions and hardware resource calling logic of operators, this method can solve the technical problems such as hardware computing power bottlenecks and high inference latency faced by deep learning models in practical industrial applications, and can be widely applied to the following complex technical scenarios with extremely high real-time and computing power requirements.

[0094] For example, in the field of autonomous driving, the computing resources of onboard computing platforms (such as onboard GPUs or dedicated autonomous driving chips) are extremely limited, yet they need to process massive amounts of environmental perception data from multiple cameras, millimeter-wave radars, and lidars in real time and in parallel. The end-to-end inference latency of perception models (such as 3D object detection models) directly determines the vehicle's braking response distance and driving safety. By applying the method provided in this invention, the massive number of operators in the onboard perception model can be optimized at the hardware level based on Amdahl's law, enabling the system to automatically identify and prioritize the bottleneck operators that occupy the longest processing time in the onboard computing unit, avoiding wasting computing power on non-critical operators. Ultimately, the end-to-end inference latency of the perception model on the onboard edge computing hardware is significantly compressed, meeting the physical requirements of autonomous driving for ultra-low latency (such as within 30 milliseconds).

[0095] For example, in machine vision defect detection scenarios on industrial production lines, such as automated production lines for semiconductor wafer inspection and new energy battery surface defect inspection, industrial cameras capture images at extremely high frame rates. This requires the backend industrial vision inspection model to complete defect identification from a single high-definition industrial image within milliseconds to tens of milliseconds; otherwise, production line congestion or missed detections will occur. By utilizing the method of this invention, the defect detection model can be automatically optimized and scheduled unattended on the factory's edge server hardware before deployment on the production line. By adaptively adjusting the block size and memory access mode of each operator under specific memory bandwidth, the theoretical peak performance of the edge computing server is fully utilized, maximizing the utilization rate of the hardware computing units, thereby improving the defect detection throughput on the production line and meeting the industrial demands of continuous high-speed production line operations.

[0096] The multi-operator optimization scheduling device provided by the present invention will be described below. The multi-operator optimization scheduling device described below can be referred to in correspondence with the multi-operator optimization scheduling method described above.

[0097] Based on any of the above embodiments Figure 5 This is a schematic diagram of the structure of the multi-operator optimization scheduling device provided by the present invention, as shown below. Figure 5 As shown, the device includes: The model analysis unit 510 is used to acquire the performance data of the model to be optimized on the target computing hardware, calculate the total speedup ratio of the model corresponding to each operator according to the execution time ratio of each operator in the performance data, and sort the operators according to the total speedup ratio of the model to obtain the operator optimization priority queue. The operator optimization unit 520 is used to select each operator as a target operator according to the order of each operator in the operator optimization priority queue, and to perform optimization iteration on the target operator; The performance evaluation unit 530 is used to obtain the performance test results of the target operator in the current iteration cycle, and evaluate the performance test results based on a preset termination criterion. The optimization scheduling unit 540 is used to stop the optimization iteration of the target operator if the evaluation result meets the preset termination criterion, and to schedule the next operator from the operator optimization priority queue as a new target operator to continue the optimization iteration.

[0098] The apparatus provided in this invention acquires performance data of the model to be optimized on the target computing hardware, calculates the overall speedup ratio of the model based on the execution time proportion of each operator in the performance data, and then constructs an operator optimization priority queue. This allows the system to prioritize the allocation of computing resources to the operators that contribute the most to the overall performance from a global model perspective, thereby effectively avoiding unnecessary time wasted on low-impact local operators. Simultaneously, this invention executes optimization iterations on each operator in the order of the operator optimization priority queue, treating each operator as the target operator. A preset termination criterion is introduced in the current iteration cycle to dynamically evaluate the performance test results. When the evaluation result meets the preset termination criterion, the current optimization is automatically stopped and the process switches to the next operator. This solves the scheduling problem of when to stop or switch during the optimization process without relying on manual intervention and effectively avoids ineffective time and resource investment on hardware bottleneck operators.

[0099] Based on any of the above embodiments, the preset termination criterion is a multi-dimensional termination criterion, and the performance evaluation unit is specifically used for: The first evaluation result is obtained by evaluating whether the number of consecutive backtracking iterations in the optimization process has reached a preset threshold. The second evaluation result is obtained by evaluating whether the actual execution performance corresponding to the performance test result reaches the preset peak ratio of the theoretical peak performance of the target computing hardware. The third evaluation result is obtained by assessing whether the current cumulative optimization time of the target operator exceeds a preset time threshold. The fourth evaluation result is obtained by evaluating whether the actual speedup ratio corresponding to the performance test result reaches the preset speedup ratio threshold. The fifth evaluation result is obtained by evaluating whether the total speedup of the next operator in the operator optimization priority queue is higher than the speedup benefit brought by continuing to optimize the target operator. If any one of the first evaluation result, the second evaluation result, the third evaluation result, the fourth evaluation result, and the fifth evaluation result is yes, then the preset termination criterion is satisfied.

[0100] Based on any of the above embodiments, the operator optimization unit includes: The editing subunit is used to edit the code of the target operator according to a preset optimization strategy to obtain the optimized operator; The running subunit is used to perform correctness verification and performance benchmark testing on the optimized operator respectively, obtain verification results and current performance data, and compare the current performance data with the baseline performance data to obtain performance comparison results; The determination subunit is used to perform version retention or version rollback operations on the optimized operator based on the verification results and the performance comparison results.

[0101] Based on any of the above embodiments, the determination subunit is specifically used for: If the verification result is passed and the performance comparison result is a performance improvement, then the code modification of the optimized operator is submitted through the version control system, and the status of the target operator in the current iteration cycle is updated to be retained; If the verification result is a failure or the performance comparison result is no performance improvement, the rollback operation of the version control system is triggered, and the code state of the target operator before the current iteration cycle is restored.

[0102] Based on any of the above embodiments, the operator optimization unit further includes a policy determination subunit, which is used for: Count the number of iterations that the target operator has executed; Based on the preset number of iteration cycles, a corresponding preset optimization strategy is selected from the candidate strategy set, which includes a large-range parameter adjustment strategy, a systematic parameter scanning strategy, and a fine-tuning strategy. If the evaluation result indicates a plateau and the actual execution performance corresponding to the performance test result does not reach the preset peak ratio, the preset optimization strategy will be switched to the underlying implementation refactoring strategy.

[0103] Based on any of the above embodiments, the model analysis unit is specifically used for: For each operator in the model to be optimized, the total speedup ratio of the model corresponding to the operator is calculated based on the execution time ratio of the operator and the preset single operator speedup ratio. All operators are sorted in descending order according to the overall speedup ratio of the model to obtain the initial priority queue; According to the order of the operators in the initial priority queue, the cumulative model speedup brought about by simultaneously optimizing the first few operators is calculated in turn; When the cumulative model speedup reaches the target speedup threshold, the target truncation position is determined based on the number of operators optimized simultaneously, and the operators in the initial priority queue that are located before the target truncation position constitute the operator optimization priority queue.

[0104] Based on any of the above embodiments, the device further includes a session persistence and recovery unit, the session persistence and recovery unit being used for: Write the current progress information of the optimization iteration, the current state information of the target operator, and the optimal speedup updated based on the performance test results into the progress record file; The configuration data for the current iteration cycle and the performance test results are appended to the experimental log file in a structured format. The progress record file and the experiment log file are both used to restore the state of the optimization iteration when the session is interrupted.

[0105] Based on any of the above embodiments, the device further includes an integrated verification unit, the integrated verification unit being used for: After all operators in the operator optimization priority queue have completed optimization iterations, all optimized operators are replaced in the model to be optimized to obtain the optimized model. The optimized model is used to perform end-to-end inference computation on the target computing hardware, and the output results and total latency of the end-to-end inference computation are compared with the reference implementation to determine the overall computational correctness and end-to-end speedup of the optimized model. An aggregate report is generated based on the end-to-end speedup ratio, the optimization time of each operator, and the performance test results.

[0106] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a multi-operator optimization scheduling method. The method includes: acquiring performance data of the model to be optimized on the target computing hardware; calculating the total speedup ratio of the model corresponding to each operator according to the execution time ratio of each operator in the performance data; sorting the operators according to the total speedup ratio of the model to obtain an operator optimization priority queue; taking each operator as a target operator according to the order of the operators in the operator optimization priority queue, and performing optimization iteration on the target operator; acquiring the performance test results of the target operator in the current iteration cycle, and evaluating the performance test results based on a preset termination criterion; if the evaluation result meets the preset termination criterion, stopping the optimization iteration of the target operator, and scheduling the next operator from the operator optimization priority queue as a new target operator to continue the optimization iteration.

[0107] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-operator optimization scheduling method provided by the above methods. The method includes: acquiring performance data of the model to be optimized on the target computing hardware; calculating the total speedup ratio of the model corresponding to each operator according to the execution time ratio of each operator in the performance data; sorting the operators according to the total speedup ratio of the model to obtain an operator optimization priority queue; taking each operator as a target operator according to the order of the operators in the operator optimization priority queue, and performing optimization iteration on the target operator; acquiring the performance test result of the target operator in the current iteration cycle, and evaluating the performance test result based on a preset termination criterion; if the evaluation result meets the preset termination criterion, stopping the optimization iteration of the target operator, and scheduling the next operator from the operator optimization priority queue as a new target operator to continue the optimization iteration.

[0109] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the multi-operator optimization scheduling method provided by the above methods. The method includes: acquiring performance data of the model to be optimized on the target computing hardware; calculating the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data; sorting the operators according to the total speedup ratio of the model to obtain an operator optimization priority queue; taking each operator as a target operator according to the order of the operators in the operator optimization priority queue, and performing optimization iteration on the target operator; acquiring the performance test result of the target operator in the current iteration cycle, and evaluating the performance test result based on a preset termination criterion; if the evaluation result satisfies the preset termination criterion, stopping the optimization iteration of the target operator, and scheduling the next operator from the operator optimization priority queue as a new target operator to continue the optimization iteration.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-operator optimization scheduling method, characterized in that, include: Obtain the performance data of the model to be optimized on the target computing hardware. Calculate the total speedup ratio of the model corresponding to each operator based on the execution time ratio of each operator in the performance data. Sort the operators according to the total speedup ratio of the model to obtain the operator optimization priority queue. According to the order of each operator in the operator optimization priority queue, each operator is taken as the target operator, and optimization iteration is performed on the target operator; Obtain the performance test results of the target operator in the current iteration cycle, and evaluate the performance test results based on a preset termination criterion; If the evaluation result meets the preset termination criterion, the optimization iteration of the target operator is stopped, and the next operator is scheduled from the operator optimization priority queue as the new target operator to continue the optimization iteration.

2. The multi-operator optimization scheduling method according to claim 1, characterized in that, The preset termination criterion is a multi-dimensional termination criterion, and the evaluation of the performance test results based on the preset termination criterion includes: The first evaluation result is obtained by evaluating whether the number of consecutive backtracking iterations in the optimization process has reached a preset threshold. The second evaluation result is obtained by evaluating whether the actual execution performance corresponding to the performance test result reaches the preset peak ratio of the theoretical peak performance of the target computing hardware. The third evaluation result is obtained by assessing whether the current cumulative optimization time of the target operator exceeds a preset time threshold. The fourth evaluation result is obtained by evaluating whether the actual speedup ratio corresponding to the performance test result reaches the preset speedup ratio threshold. The fifth evaluation result is obtained by evaluating whether the total speedup of the next operator in the operator optimization priority queue is higher than the speedup benefit brought by continuing to optimize the target operator. If any one of the first evaluation result, the second evaluation result, the third evaluation result, the fourth evaluation result, and the fifth evaluation result is yes, then the preset termination criterion is satisfied.

3. The multi-operator optimization scheduling method according to claim 1, characterized in that, The optimization iteration on the target operator includes: The code of the target operator is edited according to a preset optimization strategy to obtain the optimized operator; The optimized operator is subjected to correctness verification and performance benchmark testing to obtain verification results and current performance data. The current performance data is then compared with the baseline performance data to obtain performance comparison results. Based on the verification results and the performance comparison results, the optimized operator is subjected to version retention or version rollback operations.

4. The multi-operator optimization scheduling method according to claim 3, characterized in that, The step of performing version retention or version rollback operations on the optimized operator based on the verification results and the performance comparison results includes: If the verification result is passed and the performance comparison result is a performance improvement, then the code modification of the optimized operator is submitted through the version control system, and the status of the target operator in the current iteration cycle is updated to be retained; If the verification result is a failure or the performance comparison result is no performance improvement, the rollback operation of the version control system is triggered, and the code state of the target operator before the current iteration cycle is restored.

5. The multi-operator optimization scheduling method according to claim 3, characterized in that, Before the step of editing the code of the target operator according to the preset optimization strategy, the method further includes: Count the number of iterations that the target operator has executed; Based on the preset number of iteration cycles, a corresponding preset optimization strategy is selected from the candidate strategy set, which includes a large-range parameter adjustment strategy, a systematic parameter scanning strategy, and a fine-tuning strategy. If the evaluation result indicates a plateau and the actual execution performance corresponding to the performance test result does not reach the preset peak ratio, the preset optimization strategy will be switched to the underlying implementation refactoring strategy.

6. The multi-operator optimization scheduling method according to claim 1, characterized in that, The step involves calculating the overall model speedup ratio for each operator based on the execution time percentage of each operator in the performance data, and then sorting the operators according to the overall model speedup ratio to obtain an operator optimization priority queue, including: For each operator in the model to be optimized, the total speedup ratio of the model corresponding to the operator is calculated based on the execution time ratio of the operator and the preset single operator speedup ratio. All operators are sorted in descending order according to the overall speedup ratio of the model to obtain the initial priority queue; According to the order of the operators in the initial priority queue, the cumulative model speedup brought about by simultaneously optimizing the first few operators is calculated in turn; When the cumulative model speedup reaches the target speedup threshold, the target truncation position is determined based on the number of operators optimized simultaneously, and the operators in the initial priority queue that are located before the target truncation position constitute the operator optimization priority queue.

7. The multi-operator optimization scheduling method according to any one of claims 1 to 6, characterized in that, After the steps of obtaining the performance test results of the target operator in the current iteration cycle and evaluating the performance test results based on a preset termination criterion, the method further includes: Write the current progress information of the optimization iteration, the current state information of the target operator, and the optimal speedup updated based on the performance test results into the progress record file; The configuration data for the current iteration cycle and the performance test results are appended to the experimental log file in a structured format. The progress record file and the experiment log file are both used to restore the state of the optimization iteration when the session is interrupted.

8. The multi-operator optimization scheduling method according to any one of claims 1 to 6, characterized in that, After the step of scheduling the next operator from the operator optimization priority queue as the new target operator to continue the optimization iteration, the method further includes: After all operators in the operator optimization priority queue have completed optimization iterations, all optimized operators are replaced in the model to be optimized to obtain the optimized model. The optimized model is used to perform end-to-end inference computation on the target computing hardware, and the output results and total latency of the end-to-end inference computation are compared with the reference implementation to determine the overall computational correctness and end-to-end speedup of the optimized model. An aggregate report is generated based on the end-to-end speedup ratio, the optimization time of each operator, and the performance test results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multi-operator optimized scheduling method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-operator optimization scheduling method as described in any one of claims 1 to 8.

Images