Request batch scheduling method based on global optimization and two-dimensional load awareness

Abstract

The invention discloses a request batch scheduling method based on global optimization and two-dimensional load awareness, which comprises the following steps: firstly constructing a two-dimensional request feature vector, then defining a two-dimensional joint time delay model, then carrying out global batch optimization on a plurality of P instances, and finally formulating a scheduling-routing collaborative feedback mechanism. According to the method, cross-P global batch optimization is realized for the first time, a traditional P-by-P greedy strategy is broken through, and the average time delay is minimized from a system-level perspective.

Description

Technical Field

[0001] This invention belongs to the field of large model technology, specifically relating to a request batch scheduling method based on global optimization and dual-dimensional load awareness. Background Technology

[0002] In large-scale hybrid expert (MoE) model inference deployments, the industry commonly adopts an EP (Expert-Parallel) architecture, where expert sub-networks are distributed across devices, coupled with separate prefill / decode (P / D) scheduling to improve throughput. The operator-level performance bottlenecks are mainly concentrated in two modules: Attention module: O(L^2) computation and key-value cache usage during the Prefill phase; MoE Expert Module: Dynamically routes and activates some experts, and its computational overhead is strongly correlated with the distribution of expert load.

[0003] Current mainstream inference engines (such as vLLM and SGLang) employ the following scheduling strategies in multi-P and multi-D scenarios: 1. Greedy batching per P: The scheduler constructs a batch of requests independently for each Prefill instance (P), usually sorted by similar sequence length or arrival time, without optimizing global request allocation across multiple P instances; 2. Single-dimensional load balancing: When batching, only minimize the Attention latency variance within the batch (such as balancing the total number of tokens), completely ignoring the expert load skew that may be caused by the batch of requests after MoE routing; 3. Decoupling of static routing and scheduling: MoE routing decisions are completed within the model. The scheduler cannot predict which experts will be activated for a certain batch of requests, resulting in high-load experts being repeatedly and centrally invoked.

[0004] The above strategy leads to the following problems: 1. Overall average latency is not optimal: Locally optimal batching cannot guarantee that the total execution time of global P instances is minimized; 2. Frequent expert hotspots: A batch of requests with similar semantics (such as all being code generation) leads to a high degree of overlap among the Top-K experts, causing GPU memory bandwidth bottlenecks or competition for computing units; 3. Large fluctuations in resource utilization: Some P instances are blocked due to expert overload, while the remaining P instances are idle and waiting, reducing system throughput.

[0005] Therefore, there is an urgent need for a two-dimensional (Attention + MoE) joint batching method that is globally optimized across P for multi-P and multi-D architectures. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a request batch scheduling method based on global optimization and dual-dimensional load awareness. First, a dual-dimensional request feature vector is constructed. Then, a dual-dimensional joint latency model is defined. Next, global batch optimization is performed across multiple P instances. Finally, a scheduling-routing collaborative feedback mechanism is established. This invention is the first to achieve cross-P global batch optimization, breaking through the traditional P-by-P ​​greedy strategy and minimizing average latency from a system-level perspective.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Construct a two-dimensional request feature vector; For each pending Prefill request ,extract: Attention dimensional features: sequence length ; MoE Dimensional Features: Semantic Complexity With the activation distribution of prediction experts ; Indicates the total number of routing experts; Step 2: Define a two-dimensional joint delay model; For any batch of requests Its estimated execution latency on instance P is:

[0008] in e i,e For the request For experts e activation intensity, p ≥2, l ∈[0,1] represents adjustable weights.

[0009] Step 3: Global batch optimization across multiple P instances; Allocate the set R of all pending Prefill requests to K instances P, forming K batches. B 1,..., B K Solve the following optimization problem:

[0010] This optimization problem is solved using a heuristic clustering algorithm to minimize the global average latency. Step 4: Scheduling-routing collaborative feedback mechanism; After actual implementation, the actual workload of each expert will be collected to dynamically adjust the next round. Predictive models and l Weights are used to form a closed-loop optimization.

[0011] Preferably, the estimated expert activation distribution is obtained by predicting the Top-K expert IDs and their weights using a lightweight routing simulator.

[0012] Preferably, the heuristic clustering algorithm is an improved K-means++ or integer programming approximation.

[0013] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the above-described request batching scheduling method.

[0014] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described request batching scheduling method.

[0015] A chip includes a processor for calling and running a computer program from a memory, causing a device on which the chip is installed to perform the aforementioned request batching scheduling method.

[0016] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described request batching scheduling method.

[0017] The beneficial effects of this invention are as follows: 1. First-ever implementation of cross-P global batch optimization: Breaking through the traditional greedy strategy of proceeding step by step, minimizing average latency from a system-level perspective; 2. Dual-dimensional load joint modeling: Simultaneously constrains the load of Attention and MoE to avoid the trap of "Attention being balanced but expert being overloaded"; 3. Supports dynamic adaptation: Adjusts batching strategies online through a feedback mechanism to adapt to different business traffic patterns; 4. Low overhead and high benefit: The routing simulator adds only <2% extra computation, but brings significant QoS improvement; 5. Compatible with existing EP architecture: No need to modify the MoE model or underlying communication library, only enhance the scheduler logic. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] A request batch scheduling method based on global optimization and dual-dimensional load awareness includes the following steps: 1. Input request queue; Multiple pending prefill requests enter the scheduling system, each containing the original input text.

[0021] 2. Feature extraction module; For each request: 1) Calculate the sequence length (L); 2) Generate semantic complexity score (S) using a lightweight semantic encoder (such as fine-tuned Sentence-T5); 3) Input to the MoE routing simulator (simplified Top-K gated network), output the predicted expert activation distribution vector e=[ e 1 , e 2 ,..., e E ].

[0022] 3. Two-dimensional load modeling module; 1) Construct the joint feature vector for each request: [ L ,e]; 2) Define the batch delay evaluation function T batch B ), merging Attention term (L 2 ) and MoE term (expert load Lp norm).

[0023] 4. Global batch optimization module; 1) Treat all requests as points to be clustered; 2) Employ an improved K-means++ or integer programming approximation algorithm to minimize the average of all batches. T Batch is the target; 3) Output K request batches B 1, B 2,..., B K Distributed to K Prefill instances (P1–P K ).

[0024] 5. Execution and Feedback Module; 1) Each Prefill instance executes its corresponding batch in parallel; 2) After execution, collect metrics such as actual expert calculation time, video memory bandwidth, and KV cache usage; 3) Update the routing simulator parameters and latency model weights. l This is used for the next round of scheduling.

[0025] 6. Decode request scheduling (optional parallel path); Decode requests are scheduled based on the remaining steps and expert popularity, following a similar principle, but they are not included in the global Prefill batching.

[0026] Example 1: Global batch scheduling in online services; The system has 4 Prefill instances (P1–P4) and 20 new requests to be processed, including short question and answer, long paper abstracts, code generation, etc.

[0027] Step 1: For each request, run a lightweight routing simulator (such as a simplified version of Top-2 MoE) and output the predicted expert activation vector e. i ; Step 2: Calculate the two-dimensional distance for each pair of requests:

[0028] Step 3: Use the improved K-means++ algorithm to minimize the total To achieve this, the 20 requests were divided into 4 batches; Step 4: Assign each batch to P1–P4 for parallel execution; Results: Compared to greedy batching by P (sorted by length), the average latency was reduced by 29%, and the standard deviation of the expert's maximum load decreased by 63%.

[0029] Example 2: Dynamic weight adaptive scenario; During continuous reasoning, the system detected that a certain type of expert (such as "mathematical reasoning expert") was frequently overloaded.

[0030] Step 1: Monitor the actual calculation time and memory bandwidth utilization of each expert; Step 2: If the workload of an expert exceeds the threshold, automatically increase the weight of the MoE dimension (1- ); Step 3: Subsequent batching tends to distribute semantically similar requests more evenly, even at the cost of sacrificing a small amount of Attention balance; Results: The system remained stable under sudden surges of semantically homogeneous requests, and P99 latency fluctuations were reduced by 45%.

Claims

1. A request batch scheduling method based on global optimization and dual-dimensional load awareness, characterized in that, Includes the following steps: Step 1: Construct a two-dimensional request feature vector; For each pending Prefill request ,extract: Attention dimensional features: sequence length ; MoE Dimensional Features: Semantic Complexity With the activation distribution of prediction experts ; Indicates the total number of routing experts; Step 2: Define a two-dimensional joint delay model; For any batch of requests Its estimated execution latency on instance P is: in e i,e For the request For experts e activation intensity, p ≥2, λ ∈[0,1] represents adjustable weights; Step 3: Global batch optimization across multiple P instances; Allocate the set R of all pending Prefill requests to K instances P, forming K batches. B 1,..., B K Solve the following optimization problem: This optimization problem is solved using a heuristic clustering algorithm to minimize the global average latency. Step 4: Scheduling-routing collaborative feedback mechanism; After actual implementation, the actual workload of each expert will be collected to dynamically adjust the next round. Predictive models and λ Weights are used to form a closed-loop optimization.

2. The request batch scheduling method based on global optimization and dual-dimensional load awareness according to claim 1, characterized in that, The predicted expert activation distribution is obtained by predicting the Top-K expert IDs and their weights using a lightweight routing simulator.

3. The request batch scheduling method based on global optimization and dual-dimensional load awareness according to claim 1, characterized in that, The heuristic clustering algorithm is an improved K-means++ or integer programming approximation.

4. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

6. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 3.

Images