Distributed large model reasoning optimization method and device based on dynamic micro-batch scheduling

Abstract

The invention discloses a distributed large model reasoning optimization method and device based on dynamic micro-batch scheduling, and the method comprises the steps: (1) initializing a system, and building a large model distributed reasoning assembly line; (2) evaluating the computing power and the network state of each node and summarizing the computing power and the network state to a head node; (3) determining the number of Micro-Batches and the scheduling quota of each Micro-Batch according to the request distribution condition, the computing power of each node and the network state; and (4) sequentially scheduling the Micro-Batch by adopting a Control Batch strategy and a Chunked Prefill strategy, and starting to execute the Micro-Batch. According to the method, by dynamically adjusting the number of Micro-Batch, the serious assembly line cavitation problem in a distributed large model reasoning system is effectively solved, the GPU utilization rate and the system throughput are remarkably improved, and meanwhile key indexes such as TTFT (first token time delay) and TPOT (inter-token time delay) in the large model reasoning field are also improved. The method has good adaptability, can adaptively adjust the dynamic scheduling strategy under different hardware equipment, network conditions and request loads, is suitable for different distributed large model deployment scenes, and has wide application value.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and distributed computing, and in particular to a distributed large model reasoning optimization method and device based on dynamic micro-batch scheduling. Technical Background

[0002] Large language models (LLMs) have been widely used in building various intelligent question-answering systems due to their powerful capabilities in natural language processing. With the continued expansion of parameter sizes and the rise of sparse architectures such as MoE, models with hundreds of billions of parameters (100B) have become mainstream. Such large models far exceed the memory capacity of a single GPU and must be implemented through a distributed deployment across multiple GPUs.

[0003] Large language models are typically constructed from multiple decoder layers stacked together with identical architectures. Pipeline parallelism is a commonly used distributed deployment strategy. It partitions the model layer by layer, assigning computations to different GPUs for successive layers. Intermediate results are then transferred between the GPUs over the network to complete the entire forward inference. Compared to tensor parallelism, pipeline parallelism transfers smaller amounts of data, making it more suitable for low-cost deployment on consumer devices with limited network bandwidth.

[0004] To improve the computing efficiency of pipeline parallelism, the industry generally adopts Micro-Batch technology, which splits input requests into multiple micro-batches (Micro-Batches), allowing different GPUs to process different batches of data in parallel, thereby reducing pipeline cavitation and improving equipment utilization.

[0005] In existing large-model inference frameworks, the number of Micro-Batches is typically set to be equal to the number of pipeline stages (i.e., the number of nodes). In data center scenarios, this strategy can effectively leverage GPU performance due to the high-bandwidth, low-latency RDMA network environment, where intermediate data transmission overhead is low. However, in consumer-grade deployment environments, the network bandwidth between nodes is limited, latency is high, and transmission overhead is significant. If the number of Micro-Batches is equal to the number of nodes, after a node completes the last Micro-Batch of the current round, the first Micro-Batch of the new round cannot arrive in time due to the intermediate data transmission overhead. The node must wait for a long time, during which the GPU is completely idle, thereby reducing overall resource utilization and system throughput.

[0006] Therefore, in bandwidth-constrained deployment scenarios, how to alleviate pipeline cavitation, improve GPU utilization and end-to-end throughput performance has become a key issue that needs to be urgently addressed in large-model inference systems. Summary of the Invention

[0007] To address the aforementioned shortcomings of the prior art, the present invention proposes a distributed large-model inference optimization method and apparatus based on dynamic micro-batch scheduling. The core of this invention is to dynamically calculate and adjust the number of micro-batches and the scheduling strategy based on performance, network monitoring data, and real-time request load, thereby addressing the serious pipeline cavitation problem in distributed large-model inference systems.

[0008] The first aspect of the present invention provides a distributed large model inference optimization method based on dynamic micro-batch scheduling, comprising the following steps:

[0009] Step 1: Initialize the system and establish a large model distributed inference pipeline;

[0010] Step 2: Evaluate the computing power and network status of each node and summarize them to the head node;

[0011] Step 3: Determine the number of micro-batches and the scheduling quota for each micro-batch based on the request distribution, computing power of each node, and network status;

[0012] Step 4: Use the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch in sequence and start execution.

[0013] Among them, step 1 specifically includes:

[0014] Step 1.1: Start a distributed large model inference engine on multiple nodes, deploying several adjacent consecutive model layers on each node;

[0015] Step 1.2: Establish connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the order of the deployed model layers, and establishes gRPC connections between adjacent nodes specifically for intermediate tensor transmission.

[0016] Among them, step 2 specifically includes:

[0017] Step 2.1: During system initialization, each node simulates and computes request batches of different sizes, evaluates the node's computational latency under different conditions, and aggregates the data to the head node.

[0018] Step 2.2: Regularly evaluate the network status of each node, including the connection bandwidth and RTT with neighboring nodes, and summarize them to the head node;

[0019] Step 2.3: The head node records the aggregated performance and network data in a hash table.

[0020] Among them, step 3 specifically includes:

[0021] Step 3.1: Statistically analyze the request distribution in the current system request queue, including: the total number of tokens P for requests in the Prefill stage and the total number of tokens D for requests in the Decode stage;

[0022] Step 3.2: Execute the dynamic micro-batch scheduling algorithm and return a Micro-Batch scheduling strategy that can minimize pipeline bubbles.

[0023] Step 3.2.1: Set the global maximum scheduling quota M for each Micro-Batch and the maximum number U of Micro-Batches.

[0024] Step 3.2.2: Assume the number of nodes constituting the pipeline is N. Set the initial number of Micro-Batches to N, the number of additional Micro-Batches to be constructed to I, and the total number of Micro-Batches in the system to N + I. Start executing the subsequent steps from I = 0.

[0025] Step 3.2.3: Calculate the scheduling quota E for each Micro-Batch under the current request distribution as follows:

[0026]

[0027] If E < M, set the scheduling quota L for each Micro-batch in this round to E; otherwise, set L = M.

[0028] Step 3.2.4: Evenly schedule the Decode stage requests to the current N + I Micro-Batches. The number of Decode tokens assigned to each Micro-Batch is:

[0029]

[0030] After that, the remaining Prefill token scheduling quota for each Micro-Batch is:

[0031] L P = L - L D (3)

[0032] Perform block processing on the Prefill stage requests and fill them into the remaining scheduling quotas of each Micro-Batch.

[0033] Step 3.2.5: The inference calculation latency of the constructed Micro-Batches on each node can be queried from the hash table and is denoted as T comp ; According to the node network conditions recorded in the table, calculate the latency T for the intermediate tensor data corresponding to the Micro-Batch to be transmitted to the next nodetrans for:

[0034]

[0035] Among them, num_token represents the total number of tokens in the micro-batch, hidden_size and dtype represent the hidden layer dimension and value type, which are related to the model itself, and bandwidth and RTT represent the network bandwidth and round-trip delay between the current node and the next node;

[0036] Step 3.2.6: If for any node i:

[0037]

[0038] If the computational latency of the additional I micro-batches can overlap the pipeline bubbles between this node and the next node, the current value of I is accepted, and the number of micro-batches is returned to N + I, along with the current micro-batch construction strategy. If this is not satisfied, set I = I + 1 and repeat the process from step 3.2.3. If I = U and (5) is still not satisfied, return the number of micro-batches to U.

[0039] Among them, step 4 specifically includes:

[0040] Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in step 3.2, including the number of micro-batches and the construction strategy, the continuous batching and chunked prefill strategies are used to schedule requests and construct micro-batches.

[0041] Step 4.2: Submit the micro-batches sequentially to the head node inference engine and start execution in the pipeline.

[0042] A second aspect of the present invention provides a distributed large model inference device based on dynamic micro-batch scheduling, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the distributed large model inference optimization method based on dynamic micro-batch scheduling of the present invention.

[0043] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the distributed large model inference optimization method based on dynamic micro-batch scheduling of the present invention.

[0044] The core of the present invention is to dynamically calculate and adjust the number of Micro-Batch and scheduling strategy according to performance, network monitoring data and real-time request load, so as to solve the serious pipeline cavitation problem in distributed large-model inference systems.

[0045] The innovation of this invention lies in dynamically adjusting the number of micro-batches at runtime based on performance, network monitoring data, and request load. Unlike traditional solutions that fix the number of micro-batches to the number of nodes, dynamically adjusting the number of micro-batches can fill pipeline voids caused by intermediate data transmission overhead under varying load and network conditions, improving resource utilization and end-to-end throughput.

[0046] The working principle of the present invention is: based on the real-time request load of the system and the performance and network monitoring data of each node, the number of micro-batches that can minimize pipeline voids and the request scheduling strategy for each micro-batch are calculated. In the distributed large-scale model reasoning scenario with parallel pipelines, since the calculation time of the additionally constructed micro-batches overlaps part of the intermediate data transmission overhead, the first micro-batch of the next round can arrive before the calculation of the last micro-batch of the current round is completed, and the node can perform multiple rounds of calculations continuously, thereby significantly improving system resource utilization and throughput. The advantages of the present invention are: by dynamically adjusting the number of micro-batches, it effectively solves the serious pipeline void problem in the distributed large-scale model reasoning system, significantly improves GPU utilization and system throughput, and also improves key indicators such as TTFT (first token delay) and TPOT (inter-token delay) in the field of large-scale model reasoning. The present invention has good adaptability and can adaptively adjust the dynamic scheduling strategy under different hardware devices, network conditions and request loads. It is suitable for different distributed large-scale model deployment scenarios and has wide application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the examples of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 It is a flow chart of the method of the present invention.

[0049] Figure 2 It is the overall system architecture diagram of the present invention.

[0050] Figure 3(a)-Figure 3(c) 3(a) is a schematic diagram of the iterative process of the pipeline cavitation problem and the dynamic Micro-Batch scheduling algorithm of the present invention, wherein FIG3(b) is a schematic diagram of the iterative process of the dynamic Micro-Batch scheduling algorithm, where I=1; FIG3(c) is a schematic diagram of the iterative process of the dynamic Micro-Batch scheduling algorithm, where I=2. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] Example 1

[0053] This paper proposes a distributed large-model inference optimization method based on dynamic micro-batch scheduling. The core of the method is to dynamically calculate and adjust the number of micro-batches and scheduling strategies according to performance, network monitoring data, and real-time request load, thereby solving the serious pipeline cavitation problem in distributed large-model inference systems.

[0054] like Figure 1 , the method of the present invention specifically comprises the following steps:

[0055] Step 1: Initialize the system and establish a large model distributed inference pipeline;

[0056] Step 1.1: Start the distributed large model inference engine on multiple nodes. Each node deploys several adjacent continuous model layers. The overall system architecture is as follows: Figure 2 Specifically, the Qwen3-32B model was deployed on three servers, each equipped with an NVIDIA V100 GPU. The average bandwidth between nodes was 1 Gbps, and the average latency was 20 ms.

[0057] Step 1.2: Establish connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the order of the deployed model layers, and establishes gRPC connections between adjacent nodes specifically for intermediate tensor transmission.

[0058] Step 2: Evaluate the computing power and network status of each node and summarize them to the head node;

[0059] Step 2.1: During system initialization, each node simulates the construction of request batches of different sizes and calculates, evaluates the computing latency of the node under different conditions, and aggregates the data to the head node;

[0060] Step 2.2: Regularly evaluate the network status of each node, including the connection bandwidth and RTT with adjacent nodes, and aggregate them to the head node;

[0061] Step 2.3: The head node records the aggregated performance and network data in a hash table.

[0062] Step 3: Determine the number of Micro-Batches and the scheduling quota for each Micro-Batch according to the request distribution, the computing power of each node, and the network status;

[0063] Step 3.1: Statistically analyze the request distribution in the current system request queue, including: the total number of tokens P of requests in the Prefill stage and the total number of tokens D of requests in the Decode stage;

[0064] Step 3.2: Execute the dynamic micro-batch scheduling algorithm and return a Micro-Batch scheduling strategy that can minimize the pipeline bubbles.

[0065] Step 3.2.1: Set the global maximum scheduling limit M for each Micro-Batch and the maximum number U of Micro-Batches;

[0066] Step 3.2.2: Assume that the number of nodes constituting the pipeline is N, then set the initial number of Micro-Batches to N. In this embodiment, the initial number of Micro-Batches is 3. As shown in part (a) of Figure 3, there is a serious pipeline bubble phenomenon in the system. Set the number of additional Micro-Batches to be constructed as I, and the total number of Micro-Batches in the system is N + I. Start executing the subsequent steps from I = 0;

[0067] Step 3.2.3: Calculate the scheduling limit E for each Micro-Batch under the current request distribution as follows:

[0068]

[0069] If W < M, then set the scheduling limit L for each Micro-batch in this round to E, otherwise set L = M;

[0070] Step 3.2.4: Evenly schedule the Decode stage requests to the current N + I Micro-Batches, and the number of Decode tokens allocated to each Micro-Batch is:

[0071]

[0072] After that, the remaining Prefill token scheduling quota for each Micro-Batch is:

[0073] L P =LL D (3)

[0074] The Prefill stage requests are divided into blocks and filled into the remaining scheduling quota of each Micro-Batch;

[0075] Step 3.2.5: The inference computation delay of the constructed Micro-Batch on each node can be obtained by querying the hash table, denoted as T comp ; According to the node network status recorded in the table, calculate the delay T of the intermediate tensor data corresponding to Micro-Batch to the next node trans For:

[0076]

[0077] Among them, num_token represents the total number of tokens in the micro-batch, hidden_size and dtype represent the hidden layer dimension and value type, which are related to the model itself, and bandwidth and RTT represent the network bandwidth and round-trip delay between the current node and the next node;

[0078] Step 3.2.6: If for any node i:

[0079]

[0080] If the computational latency of the additionally constructed I micro-batches can overlap the pipeline bubble between this node and the next node, the current value of I is accepted, and the number of micro-batches is returned to N+I, along with the current micro-batch construction strategy. The situation that meets the conditions is shown in part (c) of Figure 3. The pipeline bubble has been filled with the additional micro-batches, and the system resource utilization rate is significantly improved. If it is not satisfied, set I=I+1 and re-execute from step 3.2.3. If I=U and (5) is still not satisfied, return the number of micro-batches to U.

[0081] Step 4: As shown in Figure 3, the Continuous Batching and Chunked Prefill strategies are used to schedule Micro-Batches in sequence and start execution.

[0082] Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in step 3.2, including the number of micro-batches and the construction strategy, the continuous batching and chunked prefill strategies are used to schedule requests and construct micro-batches.

[0083] Step 4.2: Submit the micro-batches sequentially to the head node inference engine and start execution in the pipeline.

[0084] Example 2

[0085] This embodiment relates to a distributed large model inference device based on dynamic micro-batch scheduling, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the distributed large model inference optimization method based on dynamic micro-batch scheduling of Example 1.

[0086] Example 3

[0087] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the distributed large model inference optimization method based on dynamic micro-batch scheduling of Example 1 is implemented.

[0088] Example 4

[0089] This embodiment relates to a large-scale distributed intelligent question-answering system optimization method using a distributed large-model reasoning optimization method based on dynamic micro-batch scheduling of the present invention. The method optimizes the reasoning efficiency of multi-round human-computer dialogue in the large-scale distributed intelligent question-answering system, and includes the following steps:

[0090] Step 1: Initialize the system and establish a distributed intelligent question-answering system reasoning pipeline based on a large model;

[0091] Step 2: Evaluate the computing power and network status of each node and summarize them to the head node;

[0092] Step 3: Determine the number of micro-batches and the scheduling quota for each micro-batch based on the request distribution, computing power of each node, and network status;

[0093] Step 4: Use the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batch in sequence and start execution.

[0094] Step 5: Based on the aforementioned distributed large model reasoning method, support for human-computer multi-round dialogue with contextual association capabilities is provided through command line tools or graphical interface (WebUI).

[0095] Specifically include:

[0096] Step 5.1: Use your API key to authenticate with the distributed inference system's API server and enter a Q&A session. This session is compatible with the OpenAI Chat Completions API standard, supports inference parameters such as max_tokens and temperature, and supports specifying a human-role identifier in the messages field of the message body to implement system prompts.

[0097] Step 5.2: Interact with the large model in a session through command-line tools or the WebUI graphical interface. Depending on the specific model, select an interaction method such as text-based question answering, image understanding, or image generation.

[0098] Step 5.3: The user's input for each round is automatically associated with the conversation history cache to construct a complete input request body. This body is then fed into the large model distributed inference pipeline built in Step 1. Steps 2-4 are then looped through inference in an autoregressive manner. After the request inference is complete, the response result is returned to the user, and the conversation record for that round is cached.

[0099] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A distributed large model inference optimization method based on dynamic micro-batch scheduling, comprising the following steps: Step 1: System initialization, establishing a distributed inference pipeline for the large model; Step 2: Evaluating the computing capabilities and network status of each node and summarizing them to the head node; Step 3: Determining the number of Micro-Batches and the scheduling quota for each Micro-Batch according to the request distribution, computing power of each node, and network status; Step 4: Using the Continuous Batching and Chunked Prefill strategies to schedule Micro-Batches in sequence and start execution.

2. A distributed large model inference optimization method based on dynamic micro-batch scheduling according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1: Starting a distributed large model inference engine on multiple nodes, and deploying several adjacent continuous model layers on each node; Step 1.2: Establishing connections between nodes. Each node reports its own network identification information to the head node. The head node organizes the pipeline arrangement according to the order of the deployed model layers, and establishes a gRPC connection specifically for intermediate tensor transmission between adjacent nodes.

3. The distributed large model inference optimization method based on dynamic micro-batch scheduling according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: During system initialization, each node simulates and constructs request batches of different sizes and computes, evaluates the computing latency of the node under different conditions, and summarizes the data to the head node; Step 2.2: Regularly evaluating the network status of each node, including the connection bandwidth and RTT with adjacent nodes, and summarizing them to the head node; Step 2.3: The head node records the summarized performance and network data in a hash table.

4. A distributed large model inference optimization method based on dynamic micro-batch scheduling according to claim 1, characterized in that: Step 3 specifically includes: Step 3.1: Statistically analyzing the request distribution in the current system request queue, including: the total number of tokens P in requests during the Prefill stage and the total number of tokens D in requests during the Decode stage; Step 3.2: Executing a dynamic micro-batch scheduling algorithm to return a Micro-Batch scheduling strategy that can minimize pipeline bubbles.

5. The distributed large model inference optimization method based on dynamic micro-batch scheduling according to claim 1 is characterized in that: Step 3.2 specifically includes: Step 3.2.1: Setting the global maximum scheduling quota M for each Micro-Batch and the maximum number U of Micro-Batches; Step 3.2.2: Assuming the number of nodes forming the pipeline is N, setting the initial number of Micro-Batches as N, setting the number of additional Micro-Batches to be constructed as I, and the total number of Micro-Batches in the system as N + I. Starting from I = 0, execute the subsequent steps in sequence; Step 3.2.3: Calculating the scheduling quota E for each Micro-Batch under the current request distribution as: If E < M, set the scheduling quota L for each Micro-batch in this round as L = E, otherwise set L = M; Step 3.2.4: Evenly scheduling Decode stage requests to the current N + I Micro-Batches. The number of Decode tokens allocated to each Micro-Batch is: Thereafter, the remaining Prefill token scheduling quota for each Micro-Batch is: L P =LL D (3) The Prefill stage requests are divided into blocks and filled into the remaining scheduling quota of each Micro-Batch; Step 3.2.5: The inference computation delay of the constructed Micro-Batch on each node can be obtained by querying the hash table, denoted as T comp ; According to the node network status recorded in the table, calculate the delay T of the intermediate tensor data corresponding to Micro-Batch to the next node trans for: Among them, num_token represents the total number of tokens in the micro-batch, hidden_size and dtype represent the hidden layer dimension and value type, which are related to the model itself, and bandwidth and RTT represent the network bandwidth and round-trip delay between the current node and the next node; Step 3.2.6: If for any node i: If the computational latency of the additional I micro-batches can overlap the pipeline bubbles between this node and the next node, the current value of I is accepted, and the number of micro-batches is returned to N + I, along with the current micro-batch construction strategy. If this is not satisfied, set I = I + 1 and repeat the process from step 3.2.

3. If I = U and (5) is still not satisfied, return the number of micro-batches to U.

6. A distributed large model inference optimization method based on dynamic micro-batch scheduling as claimed in claim 1, characterized in that: Step 4 specifically includes: Step 4.1: Based on the results returned by the dynamic micro-batch scheduling algorithm in step 3.2, including the number of micro-batches and the construction strategy, the continuous batching and chunked prefill strategies are used to schedule requests and construct micro-batches. Step 4.2: Submit the micro-batches sequentially to the head node inference engine and start execution in the pipeline.

7. A distributed large model inference device based on dynamic micro-batch scheduling, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a distributed large model reasoning optimization method based on dynamic micro-batch scheduling according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a distributed large model reasoning optimization method based on dynamic micro-batch scheduling according to any one of claims 1 to 6 is implemented.

Images