Operator dynamic scheduling method and device for large model inference

Abstract

The application provides an operator dynamic scheduling method and device for large model inference, electronic equipment and computer program product, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring input tensor characteristics of a current inference step, wherein the input tensor characteristics comprise batch size and sequence length; collecting real-time resource state data of a hardware device for executing the current inference step; for a current fusion candidate region, calculating total costs of each candidate path respectively through a preset cost estimation model, wherein the candidate paths comprise fusion paths and non-fusion paths, and the total cost is determined according to the input tensor characteristics, the real-time resource state data and an implementation mode of the current fusion candidate region; comparing the total costs of each candidate path, and selecting a candidate path with the minimum total cost as an execution path of the current fusion candidate region.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, electronic device, and computer program product for dynamic scheduling of operators for large model inference. Background Technology

[0002] In large language model inference, fusion operator techniques are widely used to reduce kernel startup and intermediate memory access overhead. However, fusion is not always optimal: with very small input sizes, startup and compilation overhead cannot be amortized; with very large input sizes, register pressure, shared memory contention, and GPU memory pressure intensify, leading to performance degradation. Existing technologies employ a static strategy of always fusion or never fusion, fixing decisions once and for all during compilation or deployment. This approach cannot flexibly adjust based on dynamically changing input shapes and real-time resource states, and lacks a mechanism to correct decisions based on historical execution data. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and computer program product for dynamic operator scheduling for large model inference, which can solve the above-mentioned problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for dynamic operator scheduling for large model inference, the method comprising the following steps: Obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length; Collect real-time resource status data of the hardware device executing the current inference step; For the current fusion candidate region, the total cost of each candidate path is calculated using a preset cost estimation model. The candidate paths include fusion paths and non-fusion paths. The total cost is determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region. Compare the total cost of each candidate path and select the candidate path with the lowest total cost as the execution path for the current fusion candidate region.

[0005] Secondly, embodiments of this application provide an operator dynamic scheduling device for large model inference, the device comprising the following: The input feature extraction module is used to obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length; The resource status acquisition module is used to collect real-time resource status data of the hardware device executing the current inference step; The cost estimation module is used to calculate the total cost of each candidate path for the current fusion candidate region using a preset cost estimation model. The candidate paths include fusion paths and non-fusion paths. The total cost is determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region. The path selection module is used to compare the total cost of each candidate path and select the candidate path with the minimum total cost as the execution path of the current fusion candidate region.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the operator dynamic scheduling method for large model inference as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, implement the steps of the operator dynamic scheduling method for large model inference as described in the first aspect.

[0008] This application's embodiments use the batch size and sequence length of the current inference step as decision-making criteria, enabling scheduling decisions to adjust in real time according to the input shape of each inference request. This allows for the selection of a more suitable path for different input scales, improving average latency and tail latency performance. By collecting real-time resource status data of the hardware device executing the current inference step, such as video memory, bandwidth, and computing unit usage, cost estimation considers not only input characteristics but also the actual availability of the current hardware. This allows scheduling decisions to adapt to dynamic scenarios such as multi-task sharing and resource contention, avoiding performance degradation or execution failure due to insufficient resources. By incorporating input tensor features, real-time resource status data, and operator implementation methods into the determination of the total cost, candidate paths can be comprehensively evaluated from multiple dimensions such as computational load, memory access overhead, and resource availability. Compared to single-dimensional judgments, this achieves more comprehensive and accurate path selection. Furthermore, by comparing fused and non-fused paths as candidates and selecting the optimal one through cost comparison, non-fused paths can ensure normal request completion, improving system robustness and inference stability, especially when fused paths may experience memory overflow or instability due to resource constraints. Scheduling decisions are made automatically before each operator is executed, eliminating the need for manual threshold setting and model recompilation or redeployment due to changes in business traffic. This reduces operational costs and enhances the service's adaptability to traffic fluctuations. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating an operator dynamic scheduling method for large model inference provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another operator dynamic scheduling method for large model inference provided in this application embodiment; Figure 3 This is a system architecture diagram of a dynamic scheduling method for large model inference runtime provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the performance boundary between the fusion path and the non-fusion path provided in the embodiments of this application; Figure 5 This is a schematic diagram of the offline modeling and online scheduling collaboration process provided in the embodiments of this application; Figure 6 This is a schematic diagram of the runtime dynamic scheduling process provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of an operator dynamic scheduling device for large model inference provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] For ease of understanding, the terminology used in this application will be explained as follows.

[0014] Fusion operators refer to the implementation method of combining multiple consecutive operators or multiple tensor operations into a single or fewer kernel executions to reduce intermediate memory access and kernel startup overhead.

[0015] Non-fusion operators refer to the implementation method of calling existing high-performance libraries or frameworks to execute native operators one by one according to the granularity of the original operators.

[0016] Runtime dynamic scheduling refers to selecting a path based on real-time input and device status when an inference request arrives after the model has been deployed, rather than fixing the execution plan only during compilation or offline stages.

[0017] Resource profiling refers to the statistical description of resource characteristics of target hardware, such as video memory capacity, bandwidth utilization, streaming multiprocessor utilization, register pressure, and cache hit status.

[0018] A strategy table refers to structured data based on the shape, resource, and cost mapping relationship obtained from offline analysis, used to quickly infer the cost of different execution paths at runtime.

[0019] In recent years, large language models have demonstrated excellent performance in tasks such as text generation, question answering, code generation, and multimodal understanding, but their inference costs remain high. To reduce latency and improve throughput, the industry has widely adopted fusion operator technology, which combines multiple originally independent computational steps into one or a few kernels for execution, thereby reducing kernel startup overhead and intermediate data access overhead.

[0020] However, fusion is not always optimal. When the input batch size is small and the sequence length is short, the startup cost, just-in-time compilation cost, or fixed kernel overhead of the fused kernel may not be amortized. When the input size is large or resources are scarce, the fused kernel may experience performance degradation due to factors such as increased register pressure, shared memory contention, increased GPU memory pressure, and decreased thread utilization, or even trigger unstable rollbacks. Existing systems often adopt a static strategy of always fusion or never fusion, which makes it difficult to achieve optimal execution results under different shapes, stages, and hardware conditions.

[0021] For example, a method for joint optimization of a computation graph based on a neural network takes the computation graph as input. First, a graph embedding neural network is used to extract the structural features of the computation graph. Then, a policy neural network generates three types of mutually coupled optimization decisions on the embedding space: device placement, operator scheduling priority, and operator fusion grouping. Based on the generated decisions, the node distribution, execution order, and fusion groups of the computation graph are reconstructed, and the output is an optimized computation graph that can be jointly executed on multiple devices such as CPU / GPU / TPU, so as to reduce cross-device data movement and intermediate tensor I / O and improve the execution efficiency of the neural network. However, in this method, device placement, operator scheduling, and fusion grouping decisions are generated once by the policy neural network during the compilation / deployment phase, while fusion grouping is relatively fixed during the execution phase. Its optimization objective is mainly focused on the static structure of the computation graph itself, without incorporating the real-time input tensor features and real-time hardware resource pressure during large model inference into the decision framework. When the actual operating environment deviates significantly from the graph pattern seen during offline training, the fusion decision may no longer be optimal. Furthermore, it lacks a mechanism for switching between fused and non-fused paths during runtime and lacks explicit penalty terms in the cost function for risks such as path switching, memory overflow, and just-in-time compilation cold start.

[0022] For example, an automatic operator fusion method for computation graphs transforms the optimal operator fusion problem into an optimal control data search problem. The control data covers fusion feasibility, data partitioning strategies, pipeline patterns, and weight residency. Fusion searches are performed separately for L1 / L2 multi-level on-chip storage layers, and two strategies, adjacent operator fusion search and block structure fusion search, are combined to expand the range of fusionable operator combinations. The weight residency decision is modeled as a knapsack problem, and the shortest cumulative execution time before fusion and the shortest execution time after fusion are used as fusion benefit evaluation indicators. Different control data samples are automatically generated for specific hardware, and the optimal fusion scheme is searched. Finally, a fixed set of fused operator implementations is output on the hardware. However, this method revolves around determining the optimal fusion scheme during the offline search phase. The fusion configuration is essentially fixed after deployment, and its search process relies on a complete analysis, search, and evaluation pipeline, making it impossible to re-trigger at low cost during online operation. This method focuses on the storage / splitting / resident parameters of a single operator or local subgraph on specific hardware, and does not provide a mechanism for secondary optimization between the overall path of the fusion operator and the native path of the non-fusion operator at runtime. When large model inference is running online, the difference between the real-time batch and sequence length is huge, and the KV Cache state, SM occupancy rate, and GPU memory availability will change dynamically with requests. The optimal fusion scheme obtained from offline search may actually be inferior to the non-fusion implementation in certain input intervals.

[0023] For example, a computation graph generation method for deep learning compilers can build a unified intermediate computation graph across different deep learning frameworks and AI chips: first, the original model is converted into a TVM high-level intermediate representation, and then mapped to a unified computation graph; for operators not supported by the target AI chip, they are broken down into sub-operators of the smallest granularity and executed on the AI ​​chip or host CPU / GPU according to preset rules; calibration parameters are read by reading calibration files, and the computation layer names are matched to generate the final calibrated computation graph that can be deployed on the target chip, achieving one-time modeling and multi-terminal deployment. However, this method is essentially a static computation graph generation scheme during the compilation / deployment phase: whether operators are split and on which type of hardware they are executed are determined during the deployment phase, without involving a dynamic optimization decision mechanism between the overall path of fused operators and the overall path of non-fused operators during the inference runtime; it does not consider the significant differences in the optimal implementation of operators between the pre-filling and decoding stages in large model inference, nor does it use real-time hardware resource profiles as the basis for operator-level path selection; when the actual traffic after deployment differs significantly from the offline assumptions, this method has neither online cost estimation capabilities nor online feedback updates, making it difficult to maintain optimal performance in long-term operation.

[0024] In addition, there is a GPU scheduling optimization method for deep learning inference service systems. This method is designed for online inference service scenarios where multiple models share a single GPU. It includes a throughput demand prediction module and a throughput adjustment module based on feedback control. In each scheduling cycle, the prediction module periodically estimates the throughput capacity of each model when running independently. The feedback control module dynamically adjusts the resource usage ratio and batch processing parameters of each model on the GPU based on the deviation between the actual observed throughput and the target throughput, so as to meet the different latency and throughput SLAs of each model as much as possible in multi-tenant scenarios. However, the scheduling object of this method is the allocation of GPU resources at the model / service level, rather than the specific implementation path of key operators within a single model. Its scheduling variables are throughput and resource quotas, and it does not incorporate the input tensor shape features and operator-level cost estimation into the decision. It lacks the ability to handle the phenomenon that the optimal implementation of key operators such as attention and feedforward networks will switch between fused and non-fused implementations in large-scale model inference under different batches, different sequence lengths, and different KV cache states.

[0025] In summary, the main problems with existing technologies are: First, static fusion strategies cannot cover situations that are widespread in large model inference, such as dynamic batches, dynamic sequence lengths, switching between pre-filling and decoding stages, and changes in KV Cache state.

[0026] Second, existing solutions typically consider "fusion" as the better option by default, ignoring issues such as the dominance of startup and compilation overhead under extremely small input scales, and the increased competition for registers / shared memory and increased graphics memory pressure under extremely large input scales, which leads to a decrease in average latency, tail latency, or stability.

[0027] Third, existing solutions often lack a mechanism to incorporate real-time hardware resource status into the decision-making loop. For example, they do not uniformly convert factors such as idle video memory, bandwidth utilization, streaming multiprocessor utilization, register pressure, and cache status into scheduling costs.

[0028] Fourth, existing solutions often lack a complete closed-loop system from offline analysis and online estimation to feedback updates, making it difficult to continuously correct strategy errors during long-term service operation.

[0029] Therefore, this application aims to propose a dynamic scheduling method for large model inference runtime. It establishes a cost estimation mechanism based on input shape and resource state between fused and non-fused paths, and selects the better path with low runtime overhead. At the same time, it improves average latency, tail latency, resource utilization and inference stability, so as to solve the problem that the performance of fused operator paths and non-fused operator paths is not fixed under different input shapes and hardware resource states. The method can realize a scheduling scheme that dynamically selects operator execution paths based on input tensor characteristics, cache state and hardware resource constraints near runtime or during runtime.

[0030] The following is in conjunction with the appendix Figures 1 to 8 This application provides a detailed description of an operator dynamic scheduling method, apparatus, electronic device, and computer program product for large model inference, through specific embodiments and application scenarios.

[0031] Figure 1 This application illustrates an embodiment of an operator dynamic scheduling method for large model inference, which can be executed by an electronic device, including servers and / or terminal devices. In other words, the method can be executed by software or hardware installed on the server and / or terminal device, and includes the following steps: Step 110: Obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length.

[0032] In the reasoning process of a large language model, each reasoning step corresponds to one forward computation operation. For example, in the pre-filling stage, the reasoning step corresponds to a complete forward computation of the entire input prompt word; in the decoding stage, the reasoning step corresponds to a forward computation to generate a single output token.

[0033] The input tensor features refer to the attribute information of the input tensors involved in the computation of the current inference step. Input tensor features include at least batch size and sequence length. Batch size represents the number of requests or samples processed in parallel in the current inference step. In large language model inference, the batch size changes dynamically with the arrival and completion of requests. The system can use continuous batch processing technology, counting the number of currently active requests as the batch size before the start of each inference step. Sequence length represents the number of tokens corresponding to each request in the current inference step. In the pre-filling stage, the sequence length is the token length of the input prompt word; in the decoding stage, the sequence length is usually 1.

[0034] In this step, the system can directly read the shape information of the current input tensor from the tensor metadata of the deep learning framework, extract the specific values ​​of the batch size dimension and sequence length dimension, form the input tensor features, and use them as input parameters for subsequent cost estimation.

[0035] Step 120: Collect real-time resource status data of the hardware device executing the current inference step.

[0036] In online service environments for large language model inference, the resource status of hardware devices is not constant. Multiple inference tasks or other computational tasks may run simultaneously on the same hardware device, causing the availability of video memory, computing resources, and bandwidth to change dynamically over time. Therefore, it is necessary to collect the current hardware resource usage status in real time when making scheduling decisions at each inference step.

[0037] The real-time resource status data refers to the usage of various resources on the hardware device at the current inference step execution time, including but not limited to: free video memory, streaming multiprocessor utilization, register utilization, bandwidth utilization, and cache utilization. Specifically, free video memory represents the available video memory capacity on the device at the current moment, used to assess whether candidate path execution will trigger insufficient video memory. Streaming multiprocessor utilization reflects the busyness of the device's computing cores. Register utilization and bandwidth utilization are used to characterize the tension of register resources and data bandwidth resources, respectively. Cache utilization reflects the hit rate and utilization efficiency of the on-chip cache.

[0038] In this step, the system queries the current resource usage data through the hardware runtime interface, and summarizes this data into real-time resource status data, which serves as the basis for resource risk penalties and resource corrections in subsequent cost estimation.

[0039] Step 130: For the current fusion candidate region, calculate the total cost of each candidate path using a preset cost estimation model. The candidate paths include fusion paths and non-fusion paths. The total cost is determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region.

[0040] Generally, in large language model inference, the same operator or group of operators often has multiple implementations. The fusion path combines multiple consecutive computational operations into a single kernel execution, reducing kernel startup overhead and intermediate data access overhead. The non-fusion path calls existing high-performance libraries or framework native operators one by one at the original operator granularity, resulting in lower startup overhead when the input size is small.

[0041] The cost estimation model can be a pre-built computational model based on offline performance profiling data, used to estimate the execution cost of each candidate path according to the current input conditions and resource status. In the offline phase, the system executes each candidate path on the target hardware for preset input shape sampling points, measuring its performance data such as latency, peak memory usage, and bandwidth utilization, forming a benchmark profile library as the data foundation for the cost estimation model.

[0042] In this step, the system inputs the input tensor features obtained in step 110, the real-time resource status data collected in step 120, and the implementation method of the current fusion candidate region into the cost estimation model. The cost estimation model obtains the baseline execution cost of each candidate path under the corresponding input shape by querying the baseline profile library, then corrects the baseline cost based on the current resource status, and simultaneously evaluates risk factors such as path switching, resource insufficiency, and just-in-time compilation cold start, finally outputting the total cost of each candidate path. Candidate paths include fused paths and non-fused paths.

[0043] Step 140: Compare the total cost of each candidate path and select the candidate path with the lowest total cost as the execution path of the current fusion candidate region.

[0044] After calculating the total cost of each candidate path in step 130, the final execution path needs to be determined from multiple candidate paths. The fused path usually performs best in the intermediate input size range because the kernel startup overhead is effectively diluted and intermediate memory accesses are reduced; the non-fused path is more stable at very small input sizes and can avoid unnecessary kernel startup overhead and compilation overhead.

[0045] In this step, the system compares the total costs of the fused path and the non-fused path, selecting the path with the lowest total cost as the execution path for the current fusion candidate region. The lowest total cost indicates that, given the current input features and resource conditions, this path exhibits the best overall performance in terms of execution latency, resource consumption, execution risk, and stability. After determining the execution path, the system invokes the corresponding kernel or operator sequence to execute the computational task for the current inference step.

[0046] In this embodiment, by using the batch size and sequence length of the current inference step as the basis for decision-making, the scheduling decision can be adjusted in real time according to the input shape of each inference request. This allows for the selection of a path more suitable for the current shape under different input scales, improving average latency and tail latency performance. By collecting real-time resource status data of the hardware device executing the current inference step, such as video memory, bandwidth, and computing unit usage, the cost estimation not only considers input characteristics but also reflects the actual availability of the current hardware. This enables the scheduling decision to adapt to dynamic scenarios such as multi-task sharing and resource contention, avoiding performance degradation or execution failure due to insufficient resources. By incorporating input tensor features, real-time resource status data, and operator implementation methods into the determination of the total cost, each candidate path can be comprehensively evaluated from multiple dimensions such as computational load, memory access overhead, and resource availability. Compared to single-dimensional judgment, this achieves a more comprehensive and accurate path selection. Furthermore, by simultaneously considering fused and non-fused paths as candidates and selecting the optimal one through cost comparison, the non-fused path can ensure normal request completion, improving the robustness and inference stability of the system, especially when fused paths may experience memory overflow or instability due to resource constraints. Scheduling decisions are made automatically before each operator is executed, eliminating the need for manual threshold setting and model recompilation or redeployment due to changes in business traffic. This reduces operational costs and enhances the service's adaptability to traffic fluctuations.

[0047] In yet another exemplary embodiment, the method of this embodiment may further include the following specific steps: In the offline phase, for preset input shape sampling points, the execution latency and resource consumption of each candidate path are measured to generate a benchmark profile library; and / or in the online phase, based on the latency data fed back after actual execution, the cost correction parameters of the corresponding shape interval in the benchmark profile library are updated using exponential moving average.

[0048] Before deploying the inference service, the system needs to perform offline performance profiling on the target hardware to establish performance benchmark data for each candidate path under different input conditions. Since the input shape of a large language model (e.g., batch size and sequence length) changes dynamically in actual operation, and there are significant differences in latency performance between fused and non-fused paths under different input shapes, it is necessary to obtain performance data covering multiple input shapes through offline measurement to provide a data foundation for runtime cost estimation.

[0049] The preset input shape sampling points can refer to a combination of typical values ​​pre-selected within the range of batch size and sequence length. The selection principles for sampling points can include: setting denser sampling points in regions with drastic performance changes, such as the minimum input region, the performance inflection point region, and the maximum input region, to improve the accuracy of subsequent interpolation estimation; and appropriately reducing sampling points in regions with gradual performance changes to reduce the overhead of offline profiling.

[0050] Each candidate path can refer to different implementations of the same operator or operator group, including fused paths and non-fused paths. During offline profiling, the system places the current hardware device in a stable operating environment and executes the kernel or operator sequence corresponding to each candidate path for each set of sampling points, measuring performance data such as latency, peak memory usage, bandwidth utilization, and register usage for each execution. To reduce the impact of random noise on the measurement results, the system performs multiple repeated measurements for each set of sampling points and uses the median or the average value after removing extreme values ​​as the final performance measurement value of each candidate path at that sampling point.

[0051] The benchmark profile library can be a structured dataset built based on offline measurement results, used to store the input shape features, corresponding performance measurements, and resource consumption data of each candidate path at each sampling point. The benchmark profile library can be stored using a multi-dimensional grid data structure to enable quick querying of performance data of neighboring sampling points based on the current input shape at runtime.

[0052] Through the above steps, the feature data, measurement values ​​and statistical results of each sampling point can be organized and written into the benchmark profile library, which can be used as the source of benchmark cost in the runtime cost estimation model.

[0053] During the continuous operation of the inference service, the actual runtime latency may deviate from the offline measurement value due to changes in the hardware environment, increased resource contention, or fluctuations in system load. If only the baseline data from offline measurements is relied upon without any correction, the accuracy of the cost estimation model will gradually decrease with increasing runtime. Therefore, it is necessary to dynamically correct the data in the baseline profile library based on the actual execution results during the online phase.

[0054] The latency data refers to the actual time consumed by the system in executing the selected path during each inference step. After each execution, the system records the actual latency of that path and the corresponding input shape as feedback information. This feedback information reflects the true performance of each candidate path in the current operating environment.

[0055] The exponential moving average is a method for progressively updating statistics over time. The new estimate is obtained by a weighted average of the old estimate and the current observation, with the current observation having a pre-defined weight of a constant less than 1. When using the exponential moving average for updates, the most recent observation has the greatest impact on the estimate, while the influence of earlier observations decays exponentially. This allows the estimate to gradually adjust to changes in the environment without causing drastic fluctuations due to a single outlier.

[0056] The cost correction parameters can be correction values ​​stored in a benchmark image library, each corresponding to a specific shape interval. After execution, the shape interval to which the input shape of the current inference step belongs is determined, and the current cost correction parameters for that interval are obtained. Then, the observed actual latency is compared with the offline benchmark value, and the cost correction parameters for that interval are updated using the exponential moving average formula. The updated cost correction parameters are used for subsequent inference steps within the same shape interval, enabling the cost estimation model to gradually approximate the true latency distribution over long-term operation, thus improving the accuracy of scheduling decisions.

[0057] In this embodiment, during each runtime cost estimation, the baseline estimated cost of each candidate path is first obtained by querying the baseline profile library based on the current input shape. Then, the current cost correction parameter for that shape interval is read, and the baseline estimated cost is superimposed with the correction parameter to serve as the estimated latency of each candidate path in the current inference step. Through the synergy of offline baseline and online correction, the hardware-level performance baseline obtained through actual measurements in the offline profiling phase can be preserved, while the estimated value can be continuously adjusted based on real feedback during online operation. This allows the scheduling decision to possess both the high accuracy of the offline phase and the adaptive capability of the online phase.

[0058] In yet another exemplary embodiment, based on step 130 of the above embodiment, for the current fusion candidate region, the total cost of each candidate path is calculated respectively using a preset cost estimation model. The method of this embodiment may further include the following specific steps: When calculating the total cost of each candidate path, the total cost is also adjusted according to the batch size and sequence length, wherein: when the batch size is less than a first threshold and the sequence length is less than a second threshold, the total cost of the fusion path is increased; when the batch size is greater than or equal to the first threshold and less than or equal to a third threshold, and the sequence length is greater than or equal to the second threshold and less than or equal to a fourth threshold, the total cost of the fusion path is decreased; when the batch size is greater than the third threshold and the sequence length is greater than the fourth threshold, the total cost of the fusion path is increased.

[0059] In large language model inference, the performance of the fusion path exhibits non-monotonic characteristics as the input size changes. When the input size is small, the fusion path combines multiple operators into a single kernel for execution, resulting in relatively high kernel startup and just-in-time (JIT) compilation overheads that cannot be amortized through computation. In this case, the non-fusion path performs better because it requires no additional compilation and its startup overhead is distributed. When the input size is in the intermediate range, the advantages of the fusion path—reducing the number of kernel startups and intermediate tensor memory accesses—are fully utilized, and its latency is typically lower than that of the non-fusion path. When the input size is too large, the fusion path experiences performance degradation due to increased register pressure, intensified shared memory contention, and increased GPU memory pressure. In this case, the non-fusion path is more suitable.

[0060] Based on the above characteristics, this embodiment adjusts the total cost of fused and non-fused paths according to batch size and sequence length when the cost estimation model calculates the total cost of each candidate path. The purpose of the adjustment is to make the total cost output by the cost estimation model more accurately reflect the actual execution cost of each candidate path under the current input scale, and guide the path selection unit to select a better execution path.

[0061] The first, second, third, and fourth thresholds can be boundary values ​​determined after performance measurements of the target hardware and model during the offline profiling phase. The first and third thresholds define the minimum and maximum boundaries of the batch size, while the second and fourth thresholds define the minimum and maximum boundaries of the sequence length. The first threshold is less than the third threshold, and the second threshold is less than the fourth threshold. The specific values ​​of these thresholds vary depending on the hardware platform and model structure, and are set by those skilled in the art based on the offline profiling method.

[0062] Specifically, the adjustment method is as follows: When the batch size is less than the first threshold and the sequence length is less than the second threshold, it indicates that the current input size is in a very small input range, and the execution efficiency of the fused path is usually lower than that of the non-fused path. In this case, the cost estimation model adds a positive value to the baseline total cost of the fused path to suppress the selection of the fused path and make the scheduler tend to choose the non-fused path.

[0063] When the batch size is greater than or equal to the first threshold and less than or equal to the third threshold, and the sequence length is greater than or equal to the second threshold and less than or equal to the fourth threshold, it indicates that the current input size is in the middle range, and the execution efficiency of the fusion path is generally better than that of the non-fusion path. In this case, the cost estimation model reduces the baseline total cost of the fusion path by a positive value to increase the probability that the fusion path will be selected.

[0064] When the batch size exceeds the third threshold and the sequence length exceeds the fourth threshold, it indicates that the current input scale is in the maximum range. The fusion path may experience performance degradation or even execution anomalies due to resource contention and storage pressure. In this case, the cost estimation model adds a positive value to the baseline total cost of the fusion path to suppress the selection of the fusion path and make the scheduler tend to choose the non-fusion path.

[0065] In this embodiment, through the above-mentioned segmented adjustment, the preset cost estimation model can reduce the competitiveness of the fusion path in the minimum and maximum input intervals, and improve the competitiveness of the fusion path in the intermediate input interval, so that the scheduler can select a better execution path under different input scales.

[0066] In yet another exemplary embodiment, the current fusion candidate region includes at least one or a combination of attention operators, feedforward network operators, or normalization operators; the hardware device includes at least one or a combination of GPU, NPU, and CPU.

[0067] The attention operator can refer to one or more computational operations involved in the attention module, including but not limited to attention core computation, masking, softmax operation, value aggregation, and their combinations. The feedforward network operator can refer to one or more computational operations involved in the feedforward network module, including but not limited to linear transformations, activation functions, and their combinations. The normalization operator can refer to one or more computational operations involved in the normalization module, including but not limited to layer normalization and root mean square normalization, and their combinations with adjacent operators.

[0068] In large language models, different network modules have different computational characteristics and memory access patterns, thus each may have multiple implementations. The attention module is responsible for handling the dependencies between positions in the input sequence; its core computation involves multiple consecutive operations such as matrix multiplication, masking, softmax normalization, and value aggregation. The feedforward network module is responsible for performing nonlinear transformations on the attention output; its core computation involves consecutive operations such as linear transformations, activation functions, and linear transformations. The normalization module is responsible for normalizing the mean and variance of the tensor to stabilize the training and inference processes. The dynamic scheduling method provided in this application can be applied to the above-mentioned modules.

[0069] The aforementioned operators can be used individually as scheduling objects, or as a group of operators. For example, the fusion path of the attention operator can combine attention core computation, masking, softmax, and value aggregation into a single kernel for execution, while its non-fusion path executes the library functions one by one according to the original operator granularity of each operation. The fusion path of the feedforward network operator can combine linear transformation, activation function, and linear transformation into a single kernel for execution, while its non-fusion path executes the operations one by one according to the original operator granularity.

[0070] Furthermore, the dynamic scheduling method provided in this application is not limited to a single hardware platform. Graphics Processing Units (GPUs) possess massively parallel computing capabilities, suitable for computationally intensive operations such as matrix multiplication and convolution in large language model inference. Neural Processing Units (NPUs) are specifically designed for deep learning tasks, improving energy efficiency through optimized systolic arrays and low-precision computation. Central Processing Units (CPUs), as general-purpose computing units, can be used for small-scale inference tasks or work in conjunction with the aforementioned accelerators. The aforementioned hardware can be used individually or in combination to form a heterogeneous computing platform. The method of this application can be deployed on any of the aforementioned hardware platforms or combinations thereof, achieving dynamic optimization of fused and non-fused paths under different hardware architectures.

[0071] Figure 2 This illustration shows a flowchart of another operator dynamic scheduling method for large model inference provided by an embodiment of this application. This method can be executed by an electronic device, which may include a server and / or terminal device, etc. In other words, the method can be executed by software or hardware installed on the server and / or terminal device, and includes the following steps: Step 210: Obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length.

[0072] Step 220: Collect real-time resource status data of the hardware device executing the current inference step.

[0073] Steps 210 and 220 can be found above. Figure 1 The specific descriptions of steps 110 and 120 in the illustrated embodiment are provided, and they achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0074] Step 230, based on step 130 of the above embodiment, for the current fusion candidate region, the total cost of each candidate path is calculated respectively using a preset cost estimation model. The method of this embodiment may further include the following specific steps: The total cost is also determined based on at least one of path switching cost, resource risk penalty, and just-in-time compilation penalty; the path switching cost is used to characterize the additional overhead incurred in switching from the previously executed path to the current candidate path; the resource risk penalty is used to characterize the probability that executing the current candidate path under the real-time resource status data will result in insufficient resources; the just-in-time compilation penalty is used to characterize the additional compilation overhead required for the first execution when the kernel corresponding to the current candidate path is not cached.

[0075] The path switching cost refers to the additional overhead incurred when the execution path selected in the current inference step differs from the path executed in the previous inference step, due to factors such as changes in cached content, reconfiguration of hardware resources, or kernel loading. Specifically, in multiple consecutive inference steps, if the scheduler frequently switches between different paths, each switch may lead to recompilation of the computation graph, reloading of the kernel, or invalidation of cached content, thus introducing additional latency overhead. By introducing path switching cost, frequent oscillations of candidate paths can be suppressed, ensuring that the scheduler only performs path switching when the switching benefit outweighs the switching cost, thereby improving the stability of the inference service and reducing unnecessary overhead.

[0076] The resource risk penalty term refers to the penalty value corresponding to the probability that executing a candidate path may fail or experience a sharp performance drop due to insufficient resources, given the current real-time resource status data. During the operation of the inference service, hardware resources such as video memory, registers, and bandwidth are in a dynamic state. Some candidate paths, such as fusion paths, may consume a lot of video memory or register resources during execution. When the device currently has insufficient idle video memory or high register pressure, executing this path may trigger video memory overflow or register overflow, leading to inference request failure or abnormally increased latency. By introducing a resource risk penalty term, the above risks can be quantified to prevent the scheduler from selecting high-resource-consuming paths when resources are scarce, thus ensuring the stability and reliability of the inference service.

[0077] The just-in-time (JIT) compilation penalty can refer to the additional compilation overhead incurred when the kernel corresponding to the current candidate path is not yet cached by the system and undergoes JIT compilation during the first execution. In large language model inference, the implementation of some fused paths relies on JIT compilation technology. This technology requires compiling the intermediate representation into machine code executable by the target hardware during the first execution, a compilation process that may take tens of milliseconds or even longer, far exceeding the execution latency of the operator itself. If the scheduler selects a path that is not cached, it will significantly increase the latency of the current inference step. By introducing the JIT compilation penalty, the scheduler can weigh the compilation overhead against the execution benefits during the cold start phase or when facing uncached paths. When the compilation overhead is too high, it tends to choose cached non-fused paths or other available paths, avoiding the deterioration of tail latency caused by JIT compilation.

[0078] Step 240: Compare the total cost of each candidate path and select the candidate path with the lowest total cost as the execution path of the current fusion candidate region.

[0079] Step 240 can be found above. Figure 1 The specific description of step 140 in the illustrated embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0080] In this embodiment, by introducing at least one of path switching cost, resource risk penalty, and just-in-time compilation penalty into the total cost calculation, the total cost not only reflects the execution latency of the path itself, but also comprehensively reflects the historical impact of path switching, the risk constraints of the current resource status, and the cold start cost of just-in-time compilation, making the scheduling decision more comprehensive and robust.

[0081] In yet another exemplary embodiment, the input tensor features further include at least one of hidden dimension, data type, KV cache status, and broadcast identifier; the real-time resource status data includes at least one of video memory free space, streaming multiprocessor utilization, register utilization, bandwidth utilization, and cache utilization.

[0082] For input tensor features, the hidden dimension represents the channel width or hidden layer size of each layer in the model. This dimension directly affects the computational cost of matrix multiplication and the memory usage of intermediate activation tensors. The data type represents the numerical precision format used by the input tensor; different data types result in different computational speeds and memory usage on the same hardware. The key-value cache state indicates whether the key-value cache is hit, its length, and its fragmentation status in the current inference step. This state directly affects the computational efficiency and memory usage of the attention module. The broadcast flag indicates whether tensor broadcasting exists in the current computational operation. Broadcasting operations may cause intermediate tensor size inflation, thus affecting memory usage and computational cost. These feature dimensions can further describe the input characteristics of the current inference step from different perspectives, providing more comprehensive input information for the cost estimation model.

[0083] For real-time resource status data, the amount of free video memory represents the available video memory capacity on the device at the current moment, used to assess whether candidate path execution will trigger video memory shortage. Streaming multiprocessor utilization reflects the busyness of the device's computing cores, used to determine whether the device currently has sufficient computing resources to execute candidate paths. Register utilization reflects the scarcity of register resources, which is particularly important for fused paths, as fused paths typically require more register resources. Bandwidth utilization reflects the busyness of video memory or on-chip data paths, used to assess the performance of memory-intensive paths under current bandwidth pressure. Cache utilization reflects the efficiency of on-chip cache usage, affecting the estimation of memory access latency. These resource dimensions can describe the real-time status of the current hardware device from aspects such as computing, storage, and bandwidth, providing more accurate resource constraint information for cost estimation models.

[0084] Specifically, in practical implementation, based on computational resource overhead and accuracy requirements, some or all of the aforementioned input tensor feature dimensions and some or all of the aforementioned real-time resource status data dimensions can be selected for scheduling decisions. Selecting only some dimensions can reduce the overhead of feature acquisition and cost calculation, but the scheduling accuracy may be slightly lower than using all dimensions. Using all dimensions can obtain the most comprehensive decision information, but the acquisition and computation overhead is relatively high. The specific implementation method can be flexibly determined according to the resource constraints and performance requirements of the specific application scenario.

[0085] In yet another exemplary embodiment, a dynamic scheduling method for large model inference runtime is specifically illustrated. This method can be deployed on a heterogeneous computing platform consisting of GPUs, NPUs, CPUs, or combinations thereof, and is applicable to inference critical paths where attention modules, feedforward network modules, normalization modules, residual connection modules, and other optional fused / non-fused implementations coexist.

[0086] See Figure 3 The overall architecture of this method includes: an input request and tensor information extraction module, a resource awareness module, a policy decision-making module, an operator execution path library, an offline profiling library, and an online feedback module. The system first parses the input features corresponding to the current request, such as batch size, sequence length, hidden dimension, data type, whether it is broadcasting, and KV cache status. Then, combined with resource features such as real-time GPU memory availability, bandwidth pressure, streaming multiprocessor utilization, and register pressure, it queries the policy model and outputs a fusion path or a non-fusion path. After execution, the measured latency and anomaly information are written back to the online feedback module for correcting subsequent decisions.

[0087] It should be noted that the core objective of the embodiments in this application is not simply to pursue a higher degree of fusion, but rather to determine when fusion is superior and when non-fusion is superior. See also Figure 4In the range of extremely small input sizes, startup and compilation overhead account for a relatively high proportion, and the latency of non-fusion paths is often more stable. In the intermediate range, fusion paths can reduce operator launch and intermediate memory accesses, and are generally better. In the extremely large range, fusion paths may cause latency increases due to resource contention, in which case non-fusion paths are more suitable. The embodiments of this application differ from existing technologies that only optimize the fusion operator itself, but form a runtime-oriented strategy selection mechanism.

[0088] The key variables and formulas are defined as follows: The input feature vector in formula (1) is defined as follows: in, This indicates the batch size (i.e., the number of requests or samples participating in inference in a single scheduling). This indicates the sequence length (i.e., the maximum number of valid tokens in the pre-padding stage, and can usually be considered as 1 in the decoding stage). This represents the hidden dimension (i.e., the channel width or hidden size of the model layer). Represents data type, It indicates the KV cache status (including cache hit status, cache length, fragmentation status, etc.). This indicates whether additional tensor features such as broadcasting exist.

[0089] The resource state vector in formula (2) is defined as follows: in, This indicates the available video memory (i.e., the available video memory capacity of the device at the time of scheduling). This indicates the streaming multiprocessor utilization rate (i.e., SM utilization rate, which reflects the busy or idle state of the device's computing cores). Indicates register pressure (used to characterize register contention that may occur during the fusion path). This indicates bandwidth pressure (used to characterize memory or on-chip bandwidth bottlenecks). This indicates the level of L2 cache utilization.

[0090] Formula (3) defines the total cost function for any path p as follows: in, This represents the estimated execution delay for path p. Indicates the cost of path switching. This indicates a penalty for insufficient video memory or resource overflow. This indicates a penalty for just-in-time compilation or cold start.

[0091] Formula (4) defines the runtime scheduling decision as follows:

[0092] That is, to choose the path with the lowest total cost between the fused path and the unfused path.

[0093] Formula (5) uses bilinear interpolation to estimate the time delay for input shapes that are not precisely covered by offline samples:

[0094] in, and To enclose the nearest sampling points of the current input shape, For the corresponding weights.

[0095] To enhance long-term operational stability, formula (6) uses an exponential moving average to update the observation delay of a certain path: in, Indicates the first After the next reasoning step is executed, the path The exponential moving average time delay estimate; This indicates the path after the previous reasoning step was executed. The exponential moving average time delay estimate; Indicates the first Path in the next reasoning step The actual observation delay; This represents the weighting coefficient, which ranges from 0 to 1 and is used to control the proportion of historical estimates and current observations in the update.

[0096] See Figure 5 First, offline performance profiling is performed on the target hardware. At least one key dimension combination is selected as the sampling dimension, such as batch size B, sequence length S, hidden dimension H, number of heads, data type, or whether it is in the pre-padding / decoding stage. For each group of sampling points, the fused path and the non-fused path are executed separately, and their execution latency, peak memory usage, bandwidth utilization, streaming multiprocessor utilization, register usage, and abnormal conditions are measured.

[0097] In a preferred embodiment, offline profiling includes preheating, repeated measurements, and median or extreme value removal averaging to reduce the impact of random noise. In regions of drastic performance variation, such as minimum input regions, inflection point regions, and maximum input regions, denser sampling points can be used to improve interpolation accuracy.

[0098] In the offline phase, the features and measurements corresponding to each sampling point are written into a policy table or a profile database. The policy table can be two-dimensional or multi-dimensional grid data, or an estimator obtained through tree models, linear models, lightweight neural networks, or piecewise rule fitting. In the preferred embodiment of this disclosure, a scheme combining rule-based thresholding and bilinear interpolation is adopted to balance accuracy and runtime overhead.

[0099] See Figure 6 The runtime scheduling and execution process specifically includes the following steps: Step 301: Receive the current operator call request. The operator call may come from the attention module, feedforward network module, normalization module, or other pre-registered candidate scheduling points.

[0100] Step 302: Extract input metadata. The system reads information such as the current batch size, sequence length, hidden dimension, data type, broadcast relationship, KV cache length, and whether it is in the pre-filling or decoding stage, and forms the input feature vector. x .

[0101] Step 303: Collect real-time resource status. The system reads the current free video memory, video memory fragmentation level, bandwidth utilization, streaming multiprocessor utilization, register usage, cache hit status, etc., to form a resource status vector. r .

[0102] Step 304: Estimate the cost of candidate paths. The scheduler calls the policy function to estimate the cost of fused paths and non-fused paths respectively. If the resource risk of a candidate path is higher than a threshold, for example, if the predicted peak memory usage exceeds the safety boundary, then a risk penalty is imposed on it or it is directly determined to be unselectable.

[0103] Step 305: Select the path with the lowest total cost to execute the corresponding kernel or operator sequence. Fusion paths can call pre-compiled fused operators, JIT-generated operators, or fused operators from the operator library provided by the hardware vendor. Non-fused paths can call general high-performance libraries or the framework's native JIT execution path or dynamic graph execution path (eager path).

[0104] Step 306: Record execution feedback. The system saves information such as the path selection, measured latency, throughput, peak resource usage, abnormal events, and whether a path switch occurred in this round. For input intervals where prediction errors frequently occur, the online feedback weight can be increased, the threshold adjusted, or the profile database revised.

[0105] Regarding path cost estimation and penalty term design, in a preferred embodiment, the estimated delay... It can consist of two parts: one is an offline interpolation estimation based solely on the input shape; the other is an online correction term based on the current resource pressure.

[0106] Specifically, It can be represented as: in, Representing a path Based solely on input shape The offline interpolation estimation delay is obtained by looking up a table in the offline-generated baseline image library; Representing a path Based on the current resource status The online correction item is used to express additional corrections to the path cost when the bandwidth pressure is too high, the free video memory is insufficient, or the register pressure is too high.

[0107] Path switching cost It can be used to suppress excessively frequent path oscillations. For example, when adjacent requests have similar input shapes but the policy switches back and forth between fused and non-fused paths due to minor noise, the switching cost can play a smoothing role.

[0108] Risk penalty This is used to evaluate the probability that a candidate path will trigger insufficient video memory, register overflow, compilation failure, or long-tail latency anomalies under the current resource conditions. This penalty term can be given by explicit rules or obtained through offline statistical learning.

[0109] Just-in-time compilation penalty This factor is used to account for the additional latency that fused kernels, which are not yet warmed up or cached, may introduce during the first execution. By incorporating this factor into the total cost, the non-fused path can be favored during the cold start phase, while the benefits of the fused path are gradually released after warm-up.

[0110] Regarding online feedback and adaptive updates, if a device frequently selects a high-resource-consuming path under high load, the relevant risk penalty will be increased.

[0111] In a preferred embodiment, online updates may only update lightweight statistics, such as exponential moving average latency, outlier rate, and upper bound of memory peak values, while retraining complex models is performed in infrequent offline maintenance cycles. This maintains sufficiently low runtime overhead while allowing the strategy to gradually adapt to changes in business traffic and hardware environment.

[0112] Example 1: In the attention module, a fusion path is defined for "attention core computation + masking + softmax + value aggregation," and a non-fusion path is defined for the implementation based on general matrix multiplication and stepwise softmax. When the batch size is small and the sequence length is short, the scheduler selects the non-fusion path; when the batch size and sequence length are in the middle range, the scheduler selects the fusion path; when the sequence length is extremely large and the memory pressure exceeds the threshold, the scheduler selects the non-fusion path.

[0113] Example 2: In the feedforward network module, a fused path is defined for "linear transformation + activation function + linear transformation", and a non-fused path is defined for the implementation based on operator-by-operator calls from a high-performance library. If the current device bandwidth pressure is high but computing resources are sufficient, the fused path achieves better performance by reducing intermediate memory accesses; if the current register pressure is high or there are other concurrent tasks, the non-fused path may be better.

[0114] Example 3: For the common S=1 scenario in the decoding stage, several graph execution instances or kernel caches can be pre-generated for the common batch size. When a common shape is hit, the fast path is taken directly, while the general scheduling logic is called for shapes that are not hit.

[0115] Corresponding to the operator dynamic scheduling method for large model inference provided in the above embodiments, based on the same technical concept, this application also provides an operator dynamic scheduling device for large model inference. See Figure 7 The device 400 includes an input feature extraction module 410, a resource status acquisition module 420, a cost estimation module 430, and a path selection module 440.

[0116] The input feature extraction module 410 is used to obtain the input tensor features of the current inference step, the input tensor features including batch size and sequence length; the resource status acquisition module 420 is used to acquire real-time resource status data of the hardware device executing the current inference step; the cost estimation module 430 is used to calculate the total cost of each candidate path for the current fusion candidate region using a preset cost estimation model, the candidate paths including fused paths and non-fused paths, the total cost being determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region; and the path selection module 440 is used to compare the total costs of each candidate path and select the candidate path with the minimum total cost as the execution path of the current fusion candidate region.

[0117] In yet another exemplary embodiment, the cost estimation module 430 includes: The benchmark lookup submodule is used to query the offline-generated benchmark profile library based on the input tensor features to obtain the benchmark estimated cost of each candidate path; The resource correction submodule is used to generate resource pressure correction coefficients based on real-time resource status data; A penalty item generation submodule is used to generate penalty items for each candidate path, wherein the penalty items include at least one of path switching cost, resource risk penalty items, and just-in-time compilation penalty items; and The cost synthesis submodule is used to weight and sum the baseline estimated cost, resource pressure correction coefficient, and penalty term to obtain the total cost of each candidate path.

[0118] It should be noted that the operator dynamic scheduling device for large model inference provided in this application embodiment and the operator dynamic scheduling method for large model inference provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned operator dynamic scheduling method for large model inference, and the repeated parts will not be described again.

[0119] In yet another exemplary embodiment, a runtime dynamic scheduling apparatus includes an input feature extraction unit, a resource status acquisition unit, a policy estimation unit, a path selection unit, an execution control unit, and an online feedback update unit. The input feature extraction unit generates an input feature vector; the resource status acquisition unit generates a resource status vector; the policy estimation unit calculates the total cost of each candidate path based on the input feature vector and the resource status vector; the path selection unit selects the path with the minimum total cost; the execution control unit triggers corresponding fusion and non-fusion operators; and the online feedback update unit corrects subsequent decisions based on measured results.

[0120] Corresponding to the operator dynamic scheduling method for large model inference provided in the above embodiments, based on the same technical concept, this application also provides an electronic device for executing the above method. Figure 8 To illustrate the structure of an electronic device according to various embodiments of this application, as shown in the following diagrams... Figure 8 As shown. Electronic device 500 can vary considerably due to differences in configuration or performance, and may include one or more processors 510 and memory 520. Memory 520 may store one or more application programs or data. Memory 520 may be temporary or persistent storage. The application programs stored in memory 520 may include one or more modules (not shown), each module may include a series of computer-executable instructions for the electronic device. Furthermore, processor 510 may be configured to communicate with memory 520 and execute the series of computer-executable instructions stored in memory 520 on the electronic device.

[0121] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, implement the steps of the operator dynamic scheduling method for large model inference as described above.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems, devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0128] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0132] It should be understood that the training and prediction processes of the AI ​​models involved in the various embodiments of this specification all adhere to multiple legal and compliant principles, including legal data sources, compliant data content, compliant data governance, compliant training objectives and schemes, compliant training processes, compliant training environments and tools, and compliant ethical verification of training results, and comply with the requirements of Article 5 of the Patent Law. Among them: Data source legitimacy: All datasets used for AI model training were obtained through legal means, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Publicly authorized data originates from compliant data sources that comply with relevant open-source agreements, with complete copyright attribution and authorization scope clearly marked, and no unauthorized open-source code or data reuse. Data authorized by partners has been subject to formal data usage agreements, clearly defining the scope, duration, and confidentiality obligations, and possessing a complete authorization chain. For self-collected data involving personal information, strict informed consent procedures have been followed, and anonymization processes (including but not limited to field masking, feature anonymization, and differential privacy technology) have been implemented to remove personally identifiable information, fully complying with relevant laws and regulations.

[0133] Data content compliance: The AI ​​model's dataset undergoes multiple screenings and cleaning processes to remove all non-compliant information and ensures that there is no illegal acquisition or use of genetic resources. For data in sensitive fields (such as healthcare and finance), an additional privacy-preserving computation module (including federated learning and secure multi-party computation technologies) is used to ensure that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0134] Data governance norms: A complete data traceability system is established during the AI ​​model training process to automatically record the source, collection time, annotation process, cleaning rules, and permission allocation of training data, generating traceable compliance reports to ensure that the data is verifiable throughout its entire lifecycle. The dataset annotation process for AI models is completed by a professional human R&D team, clearly defining the proportion of human creative contributions and avoiding reliance on AI-generated data that has not undergone substantial human modification, thus meeting the examination requirements for "human main contributions" in AI patent applications.

[0135] Training objectives and plans are compliant: The training objectives, training schemes, and final output results of the AI ​​model do not violate any mandatory provisions of laws or administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal or criminal activities, infringing on privacy, or disrupting public safety. They strictly adhere to the ethical principle of "intelligent for good".

[0136] Training process compliance: A closed-loop training framework is adopted to ensure compliance and controllability of the training process. The specific process is as follows: First, training samples are obtained through compliant data sources. After the aforementioned data cleaning and desensitization, they are input into the neural network model to generate preliminary training results. Second, an expert system is introduced to verify the preliminary results. Based on preset rules and human expert experience, the feasibility of the results is evaluated, and outputs that may pose ethical risks or compliance hazards are corrected (such as removing decision-making logic that violates public order and good morals, and adjusting model parameters that do not comply with safety regulations). Finally, the loss function weights are dynamically optimized based on expert system feedback to strengthen the model's learning of compliant results, avoid overfitting errors or non-compliant labels, and form a closed-loop control of "data input - model training - expert verification - parameter optimization - result feedback" to ensure that the entire training process complies with the ethical review requirements of relevant laws.

[0137] Training environment and tool compliance: AI model training is implemented using nationally licensed chips and a compliant training platform. All open-source frameworks and components used in the training process have obtained their corresponding licenses, and copyright statements and patent citation information are fully retained, with no instances of infringement or reuse. The training environment is built using virtual devices (containers / virtual machines) with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. Furthermore, through access control and operation log recording, risks such as data leakage and parameter tampering during training are prevented, ensuring the security and compliance of the training process.

[0138] Training results ethical verification compliance: After the model is trained, it undergoes additional third-party ethical compliance assessment and algorithm filing review to verify that the model output does not violate social morality or harm public interests. For potentially sensitive scenarios, a dedicated result verification mechanism is established to ensure that the model always complies with relevant laws and regulations in practical applications.

[0139] In summary, the data and training process used in the AI ​​model in this specification strictly comply with relevant regulations and do not violate any laws, social ethics, public interests, or regulations on the use of genetic resources. Therefore, it fully meets the compliance requirements for patent authorization.

Claims

1. A method for dynamic operator scheduling for large model inference, characterized in that, The method includes the following steps: Obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length; Collect real-time resource status data of the hardware device executing the current inference step; For the current fusion candidate region, the total cost of each candidate path is calculated using a preset cost estimation model. The candidate paths include fusion paths and non-fusion paths. The total cost is determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region. Compare the total cost of each candidate path and select the candidate path with the lowest total cost as the execution path for the current fusion candidate region.

2. The method according to claim 1, characterized in that, It also includes the following steps: In the offline phase, for preset input shape sampling points, the execution latency and resource consumption of each candidate path are measured to generate a benchmark profile library; and / or During the online phase, based on the latency data fed back after actual execution, the cost correction parameters for the corresponding shape interval in the benchmark image library are updated using an exponential moving average.

3. The method according to claim 1, characterized in that, When calculating the total cost of each candidate path, the total cost is also adjusted according to the batch size and sequence length, wherein: When the batch size is less than the first threshold and the sequence length is less than the second threshold, the total cost of the fusion path is increased. When the batch size is greater than or equal to the first threshold and less than or equal to the third threshold, and the sequence length is greater than or equal to the second threshold and less than or equal to the fourth threshold, the total cost of the fusion path is reduced. When the batch size is greater than the third threshold and the sequence length is greater than the fourth threshold, the total cost of the fusion path is increased.

4. The method according to claim 1, characterized in that, The current fusion candidate region includes at least one or a combination of attention operators, feedforward network operators, or normalization operators; the hardware device includes at least one or a combination of GPU, NPU, and CPU.

5. The method according to claim 1, characterized in that, The total cost is also determined based on at least one of path switching cost, resource risk penalty, and just-in-time compilation penalty. The path switching cost is used to characterize the additional overhead incurred in switching from the previously executed path to the current candidate path; The resource risk penalty term is used to characterize the probability that executing the current candidate path will result in insufficient resources under the real-time resource status data; The just-in-time compilation penalty term is used to characterize the additional compilation overhead required for the first execution when the kernel corresponding to the current candidate path is not cached.

6. The method according to claim 1, characterized in that, The input tensor features also include at least one of hidden dimension, data type, KV cache status, and broadcast identifier; the real-time resource status data includes at least one of video memory free space, streaming multiprocessor utilization, register utilization, bandwidth utilization, and cache utilization.

7. An operator dynamic scheduling device for large model inference, characterized in that, The device includes the following: The input feature extraction module is used to obtain the input tensor features of the current inference step, wherein the input tensor features include batch size and sequence length; The resource status acquisition module is used to collect real-time resource status data of the hardware device executing the current inference step; The cost estimation module is used to calculate the total cost of each candidate path for the current fusion candidate region using a preset cost estimation model. The candidate paths include fusion paths and non-fusion paths. The total cost is determined based on the input tensor features, real-time resource status data, and the implementation method of the current fusion candidate region. as well as The path selection module is used to compare the total cost of each candidate path and select the candidate path with the minimum total cost as the execution path of the current fusion candidate region.

8. The apparatus according to claim 7, characterized in that, The cost estimation module includes: The benchmark lookup submodule is used to query the offline-generated benchmark profile library based on the input tensor features to obtain the benchmark estimated cost of each candidate path; The resource correction submodule is used to generate resource pressure correction coefficients based on real-time resource status data; A penalty item generation submodule is used to generate penalty items for each candidate path, wherein the penalty items include at least one of path switching cost, resource risk penalty items, and just-in-time compilation penalty items; and The cost synthesis submodule is used to weight and sum the baseline estimated cost, resource pressure correction coefficient, and penalty term to obtain the total cost of each candidate path.

9. An electronic device, characterized in that, It includes a processor, a memory, a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, implement the steps of the method as described in any one of claims 1 to 6.

Images