A distributed large model inference optimization method and device based on dynamic micro-batch scheduling

Abstract

A distributed large model inference optimization method and device based on dynamic micro-batch scheduling, the method comprising: (1) system initialization, establishing a large model distributed inference pipeline; (2) evaluating the computing power and network state of each node and summarizing to the head node; (3) determining the number of Micro-Batch and the scheduling quota of each Micro-Batch according to the request distribution, the computing power of each node and the network state; (4) using the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch in turn and start execution. The present application effectively solves the serious pipeline bubble problem in the distributed large model inference system by dynamically adjusting the number of Micro-Batch, significantly improves the GPU utilization and system throughput, and also improves the key indicators such as TTFT (first token latency) and TPOT (token interval latency) in the field of large model inference. The present application has good adaptability and can adaptively adjust the dynamic scheduling strategy under different hardware devices, network conditions and request loads, is suitable for different distributed large model deployment scenarios and has wide application value.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed computing, and in particular to a distributed large-model inference optimization method and apparatus based on dynamic micro-batch scheduling. Technical Background

[0002] Large Language Models (LLMs) have been widely used to build various intelligent question-answering systems due to their powerful capabilities in natural language processing tasks. With the continuous expansion of parameter scale and the rise of sparse architectures (such as MoE), models with hundreds of billions (100B) parameters have gradually become mainstream. Such massive models far exceed the memory capacity of a single GPU and must be implemented through distributed deployment of multiple GPUs.

[0003] Large language models are typically composed of multiple stacked decoder layers with identical structures. Pipeline parallelism is a common distributed deployment strategy that divides the model into layers, with different GPUs responsible for computing consecutive segments of a layer. Intermediate results are transferred between GPUs over a network to complete the entire forward inference process. Compared to tensor parallelism, pipeline parallelism transfers less data, making it more suitable for low-cost deployment on consumer devices with limited network bandwidth.

[0004] To improve the computational efficiency of pipelined parallelism, the industry generally adopts Micro-Batch technology, which splits input requests into multiple micro-batches, allowing different GPUs to process different batches of data in parallel, thereby reducing pipeline cavitation and improving equipment utilization.

[0005] In existing large-scale model inference frameworks, the number of micro-batches is typically set to be equal to the number of pipeline stages (i.e., the number of nodes). In data center scenarios, due to the high-bandwidth, low-latency RDMA network environment, intermediate data transmission overhead is low, and this strategy can effectively utilize GPU performance. However, in consumer-grade deployment environments, the network bandwidth between nodes is limited, latency is high, and transmission overhead is significant. If the number of micro-batches equals the number of nodes, after a node completes the last micro-batch of the current round, the first micro-batch of the new round cannot arrive in time due to the intermediate data transmission overhead. The node must wait for a long time, during which the GPU is completely idle, thus reducing overall resource utilization and system throughput.

[0006] Therefore, in bandwidth-constrained deployment scenarios, how to alleviate pipeline cavitation, improve GPU utilization and end-to-end throughput performance has become a key issue that large model inference systems urgently need to address. Summary of the Invention

[0007] To address the aforementioned shortcomings of the prior art, this invention proposes a distributed large-model inference optimization method and apparatus based on dynamic micro-batch scheduling. The core of this invention lies in dynamically calculating and adjusting the number of micro-batches and the scheduling strategy based on performance, network monitoring data, and real-time request load, thereby resolving the severe pipeline cavitation problem in distributed large-model inference systems.

[0008] The first aspect of this invention proposes a distributed large-scale model inference optimization method based on dynamic micro-batch scheduling, comprising the following steps:

[0009] Step 1: System initialization, establishing a large-scale distributed inference pipeline;

[0010] Step 2: Assess the computing power and network status of each node and summarize them to the head node;

[0011] Step 3: Determine the number of Micro-Batches and the scheduling quota for each Micro-Batch based on the request distribution, the computing power of each node, and the network status;

[0012] Step 4: Use the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch sequentially and start execution.

[0013] Step 1 specifically includes:

[0014] Step 1.1: Start the distributed large model inference engine on multiple nodes, with each node deploying several adjacent consecutive model layers;

[0015] Step 1.2: Establish connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the deployment model layer order. A dedicated gRPC connection for intermediate tensor transmission is established between adjacent nodes.

[0016] Step 2 specifically includes:

[0017] Step 2.1: During system initialization, each node simulates and constructs request batches of different sizes and performs calculations, evaluates the computation latency of the node under different conditions, and aggregates the data to the head node;

[0018] Step 2.2: Periodically assess the network status of each node, including connection bandwidth and RTT with neighboring nodes, and summarize them to the head node;

[0019] Step 2.3: The head node records the aggregated performance and network data in a hash table.

[0020] Step 3 specifically includes:

[0021] Step 3.1: Statistically analyze the request distribution in the current system request queue, including: the total number of tokens P for requests in the Prefill stage and the total number of tokens D for requests in the Decode stage;

[0022] Step 3.2: Execute the dynamic micro-batch scheduling algorithm and return a Micro-Batch scheduling strategy that can minimize pipeline bubbles.

[0023] Step 3.2.1: Set the global maximum scheduling quota M for each Micro-Batch and the maximum number U of Micro-Batches.

[0024] Step 3.2.2: Assume the number of nodes constituting the pipeline is N, then set the initial number of Micro-Batches to N, set the number of additional Micro-Batches to be constructed as I, and the total number of Micro-Batches in the system as N + I. Start executing the subsequent steps from I = 0.

[0025] Step 3.2.3: Calculate the scheduling quota E for each Micro-Batch under the current request distribution as follows:

[0026]

[0027] If E < M, then set the scheduling quota L for each Micro-batch in this round to E; otherwise, set L = M.

[0028] Step 3.2.4: Evenly schedule the Decode stage requests to the current N + I Micro-Batches. The number of Decode tokens assigned to each Micro-Batch is:

[0029]

[0030] After that, the remaining Prefill token scheduling quota for each Micro-Batch is:

[0031] L P = L - L D (3)

[0032] Perform block processing on the Prefill stage requests and fill them into the remaining scheduling quotas of each Micro-Batch.

[0033] Step 3.2.5: The inference calculation latency of the constructed Micro-Batches on each node can be queried from the hash table, denoted as T comp ; According to the node network conditions recorded in the table, calculate the latency T for the intermediate tensor data corresponding to the Micro-Batch to be transmitted to the next nodetrans for:

[0034]

[0035] Where num_token represents the total number of tokens in Micro-Batch, hidden_size and dtype represent the hidden layer dimension and numerical type, which are related to the model itself, and bandwidth and RTT represent the network bandwidth and round-trip time between the current node and the next node;

[0036] Step 3.2.6: If for any node i, the following holds true:

[0037]

[0038] If the computational delay of the additional I Micro-Batches can overlap the pipeline cavitation between the current node and the next node, then accept the current value I, return the number of Micro-Batches as N+I, and the current Micro-Batch construction strategy. If this condition is not met, then set I = I+1 and re-execute from step 3.2.3. If I = U and (5) is still not met, then return the number of Micro-Batches as U.

[0039] Step 4 specifically includes:

[0040] Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in Step 3.2, including the number of Micro-Batches and the construction strategy, use the Continuous Batching and Chunked Prefill strategies to schedule requests and construct Micro-Batches;

[0041] Step 4.2: Submit the Micro-Batch sequentially to the head node inference engine to begin execution in the pipeline.

[0042] A second aspect of the present invention provides a distributed large model inference apparatus based on dynamic micro-batch scheduling, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the distributed large model inference optimization method based on dynamic micro-batch scheduling of the present invention.

[0043] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the distributed large-model inference optimization method based on dynamic micro-batch scheduling of the present invention.

[0044] The core of this invention lies in dynamically calculating and adjusting the number of Micro-Batches and scheduling strategies based on performance, network monitoring data, and real-time request load, thereby solving the severe pipeline cavitation problem in distributed large-model inference systems.

[0045] The innovation of this invention lies in dynamically adjusting the number of Micro-Batches during runtime based on performance, network monitoring data, and request load. Unlike traditional solutions that fix the number of Micro-Batches to the number of nodes, dynamically adjusting the number of Micro-Batches can fill pipeline bubbles caused by intermediate data transmission overhead under different load and network conditions, thereby improving resource utilization and end-to-end throughput performance.

[0046] The working principle of this invention is as follows: based on the real-time request load of the system and the performance and network monitoring data of each node, the number of Micro-Batches that minimizes pipeline bubbling and the request scheduling strategy for each Micro-Batch are calculated. In a distributed large-model inference scenario with pipelined parallelism, because the computation time of the additionally constructed Micro-Batch overlaps with some intermediate data transmission overhead, the first Micro-Batch of the next round can arrive before the last Micro-Batch of the current round is completed. Nodes can continuously perform calculations for multiple rounds, thereby significantly improving system resource utilization and throughput. The advantages of this invention are: by dynamically adjusting the number of Micro-Batches, it effectively solves the serious pipeline bubbling problem in distributed large-model inference systems, significantly improves GPU utilization and system throughput, and also improves key indicators such as TTFT (first token latency) and TPOT (token-to-token latency) in the field of large-model inference. This invention has good adaptability and can adaptively adjust the dynamic scheduling strategy under different hardware devices, network conditions, and request loads. It is suitable for different distributed large-model deployment scenarios and has broad application value. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the method of the present invention.

[0049] Figure 2 This is the overall system architecture diagram of the present invention.

[0050] Figures 3(a)-3(c) Figure 3(a) is a schematic diagram of the pipeline cavitation problem and the iterative process of the dynamic Micro-Batch scheduling algorithm of the present invention; Figure 3(b) is a schematic diagram of the Micro-Batch scheduling algorithm under the traditional scheme; Figure 3(c) is a schematic diagram of the iterative process of the dynamic Micro-Batch scheduling algorithm, where I = 1; Figure 3(c) is a schematic diagram of the iterative process of the dynamic Micro-Batch scheduling algorithm, where I = 2. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0052] Example 1

[0053] This invention proposes a distributed large model inference optimization method based on dynamic micro-batch scheduling. Its core lies in dynamically calculating and adjusting the number of micro-batches and scheduling strategies according to performance, network monitoring data and real-time request load, so as to solve the serious pipeline cavitation problem in distributed large model inference systems.

[0054] like Figure 1 The method of the present invention specifically includes the following steps:

[0055] Step 1: System initialization, establishing a large-scale distributed inference pipeline;

[0056] Step 1.1: Start the distributed large model inference engine on multiple nodes. Each node deploys several adjacent consecutive model layers. The overall system architecture diagram is as follows. Figure 2 As shown. Specifically, the Qwen3-32B model is deployed using 3 servers, each equipped with an NVIDIA V100 GPU, with an average bandwidth of 1Gbps and an average latency of 20ms between nodes;

[0057] Step 1.2: Establish connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the deployment model layer order. A dedicated gRPC connection for intermediate tensor transmission is established between adjacent nodes.

[0058] Step 2: Assess the computing power and network status of each node and summarize them to the head node;

[0059] Step 2.1: During system initialization, each node simulates the construction of request batches of different sizes and calculates, evaluates the computing latency of the node under different conditions, and summarizes the data to the head node;

[0060] Step 2.2: Regularly evaluate the network status of each node, including the connection bandwidth and RTT with adjacent nodes, and summarize to the head node;

[0061] Step 2.3: The head node records the summarized performance and network data in a hash table. <​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0072] After this, the remaining Prefill token allocation for each Micro-Batch is:

[0073] L P =LL D (3)

[0074] The Prefill phase requests are processed in chunks and filled into the remaining scheduling quota of each Micro-Batch;

[0075] Step 3.2.5: The inference computation latency of the constructed Micro-Batch on each node can be obtained by looking up the hash table, denoted as T. comp Based on the node network conditions recorded in the table, calculate the time delay T for transmitting the intermediate tensor data of the Micro-Batch to the next node. trans for:

[0076]

[0077] Where num_token represents the total number of tokens in Micro-Batch, hidden_size and dtype represent the hidden layer dimension and numerical type, which are related to the model itself, and bandwidth and RTT represent the network bandwidth and round-trip time between the current node and the next node;

[0078] Step 3.2.6: If for any node i, the following holds true:

[0079]

[0080] If the computational delay of the additional I Micro-Batches can overlap the pipeline cavitation between the current node and the next node, then the current value I is accepted, and the number of Micro-Batches is N+I, along with the current Micro-Batch construction strategy is returned. The condition is shown in part (c) of Figure 3, where the pipeline cavitation has been filled by the additional Micro-Batches, significantly improving system resource utilization. If the condition is not met, then I = I+1, and execution is restarted from step 3.2.3. If I = U and (5) is still not met, then the number of Micro-Batches is returned as U.

[0081] Step 4: As shown in Figure 3, the Micro-Batch is scheduled and executed sequentially using the Continuous Batching and Chunked Prefill strategies.

[0082] Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in Step 3.2, including the number of Micro-Batches and the construction strategy, use the Continuous Batching and Chunked Prefill strategies to schedule requests and construct Micro-Batches;

[0083] Step 4.2: Submit the Micro-Batch sequentially to the head node inference engine to begin execution in the pipeline.

[0084] Example 2

[0085] This embodiment relates to a distributed large model inference device based on dynamic micro-batch scheduling, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the distributed large model inference optimization method based on dynamic micro-batch scheduling in Embodiment 1.

[0086] Example 3

[0087] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the distributed large model inference optimization method based on dynamic micro-batch scheduling of Embodiment 1.

[0088] Example 4

[0089] This embodiment relates to an optimization method for a large-scale distributed intelligent question-answering system based on a distributed large-model inference optimization method with dynamic micro-batch scheduling, according to the present invention. The method focuses on optimizing the inference efficiency of multi-turn human-computer dialogue in a large-scale distributed intelligent question-answering system, and includes the following steps:

[0090] Step 1: System initialization, establishing a distributed intelligent question-answering system inference pipeline based on a large model;

[0091] Step 2: Assess the computing power and network status of each node and summarize them to the head node;

[0092] Step 3: Determine the number of Micro-Batches and the scheduling quota for each Micro-Batch based on the request distribution, the computing power of each node, and the network status;

[0093] Step 4: Use the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch sequentially and start execution.

[0094] Step 5: Based on the aforementioned distributed large model inference method, provide support for human-computer multi-turn dialogue with contextual understanding capabilities through command-line tools or graphical interfaces (WebUI).

[0095] Specifically, it includes:

[0096] Step 5.1: Use the API Key to authenticate with the API server in the distributed inference system and enter the question-and-answer session. The session is compatible with the standard interface of the OpenAI Chat Completions API, supports inference parameters such as max_tokens and temperature, and also supports specifying human-machine role identifiers in the messages field of the message body to implement system prompts.

[0097] Step 5.2: Interact with the large model in the session through command-line tools or the WebUI graphical interface, and select the interaction method based on the specific model, such as text-based question answering, image understanding, or image generation.

[0098] Step 5.3: Each round of user input is automatically associated with the session history cache to construct a complete input request body. This body is then input into the distributed inference pipeline of the large model built in Step 1, where steps 2-4 are executed in an autoregressive manner for inference. After the request inference is completed, the response result is returned to the user, and the dialogue record for that round is cached.

[0099] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A distributed large-scale model inference optimization method based on dynamic micro-batch scheduling, comprising the following steps: Step 1: System initialization, establishing a large-scale distributed inference pipeline; Step 2: Assess the computing power and network status of each node and summarize them to the head node; Step 3: Determine the number of Micro-Batches and the scheduling quota for each Micro-Batch based on the request distribution, the computing power of each node, and the network status; Specifically, it includes: Step 3.1: Analyze the request distribution in the current system request queue, including the total number of tokens for requests in the Prefill phase. The total number of tokens requested during the Decode phase ; Step 3.2: Execute the dynamic micro-batch scheduling algorithm to return a micro-batch scheduling strategy that minimizes pipeline cavitation; the steps are as follows: Step 3.2.1: Set the global maximum scheduling quota for each Micro-Batch and the maximum number of Micro-Batch ; Step 3.2.2: Assume the number of nodes forming the pipeline is... Then set the initial Micro-Batch size to Set the number of additional Micro-Batches that need to be constructed to [number]. The total number of Micro-Batch in the system is ,from Begin executing the subsequent steps in sequence; Step 3.2.3: Calculate the scheduling quota E for each Micro-Batch under the current request distribution: like Then set the scheduling quota for each Micro-batch in this round. Otherwise set ; Step 3.2.4: Average scheduling of requests during the Decode phase to the current... There are 1 micro-batch, and the number of decoded tokens allocated to each micro-batch is: After this, the remaining Prefill token allocation for each Micro-Batch is: The Prefill phase requests are processed in chunks and filled into the remaining scheduling quota of each Micro-Batch; Step 3.2.5: The inference computation latency of the constructed Micro-Batch on each node can be obtained by looking up the hash table, denoted as... Based on the node network conditions recorded in the table, calculate the latency for transmitting the intermediate tensor data corresponding to the Micro-Batch to the next node. for: in, This represents the total number of tokens in the Micro-Batch. and The hidden layer dimension and numerical type are related to the model itself. and This represents the network bandwidth and round-trip latency between the current node and the next node; Step 3.2.6: If for any node All conditions are met: Indicates additional construction If the computation latency of a micro-batch can overlap the pipeline cavitation between the current node and the next node, then the current value is accepted. The returned Micro-Batch size is and the current Micro-Batch construction strategy; if not satisfied, then let Re-execute from step 3.2.3; if and If the condition is still not met, then return the number of Micro-Batch. ; Step 4: Use the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch sequentially and start execution.

2. The distributed large-model inference optimization method based on dynamic micro-batch scheduling as described in claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Start the distributed large model inference engine on multiple nodes, with each node deploying several adjacent consecutive model layers; Step 1.2: Establish connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the deployment model layer order. A dedicated gRPC connection for intermediate tensor transmission is established between adjacent nodes.

3. The distributed large-model inference optimization method based on dynamic micro-batch scheduling as described in claim 1, characterized in that, Step 2 specifically includes: Step 2.1: During system initialization, each node simulates and constructs request batches of different sizes and performs calculations, evaluates the computation latency of the node under different conditions, and aggregates the data to the head node; Step 2.2: Periodically assess the network status of each node, including connection bandwidth and RTT with neighboring nodes, and summarize them to the head node; Step 2.3: The head node records the aggregated performance and network data in a hash table.

4. The distributed large-model inference optimization method based on dynamic micro-batch scheduling as described in claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in Step 3.2, including the number of Micro-Batches and the construction strategy, use the Continuous Batching and Chunked Prefill strategies to schedule requests and construct Micro-Batches; Step 4.2: Submit the Micro-Batch sequentially to the head node inference engine to begin execution in the pipeline.

5. A distributed large-model inference device based on dynamic micro-batch scheduling, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the distributed large model inference optimization method based on dynamic micro-batch scheduling as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements a distributed large model inference optimization method based on dynamic micro-batch scheduling as described in any one of claims 1-4.

Images