Cloud-edge-terminal collaborative dynamic batch processing scheduling system for large language model reasoning

Abstract

The application discloses a cloud edge end cooperative dynamic batch processing scheduling system for large language model reasoning, and relates to the technical field of dynamic batch processing; a reasoning request issued by a user side is input to an output length prediction module, a pre-trained output length prediction model is used to predict the number of tokens predicted when the reasoning request is reasoned on a large language model, a resource consumption estimation module establishes a resource consumption model for each reasoning request based on the predicted number of tokens and the parameters of a server module to estimate batch feasibility, a dynamic batch processing scheduling module constructs a batch according to the batch feasibility and repeatedly constructs the next batch, and the server module executes reasoning calculation according to the constructed batch and feeds back the reasoning calculation result to the user side; the cloud edge end cooperative dynamic batch processing scheduling system for large language model reasoning can realize the highest resource utilization rate and meet the quality of service constraints of all requests.

Description

Technical Field

[0001] This invention relates to the field of dynamic batch processing technology, and in particular to a cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference. Background Technology

[0002] With the rapid development of generative artificial intelligence, Large Language Models (LLMs) have been widely applied in various scenarios such as intelligent customer service, code generation, and content creation. These applications are typically deployed on cloud servers to provide users with high-quality content output. However, the existing cloud deployment model struggles to meet requirements such as low latency and privacy security. Therefore, in recent years, many applications have attempted to deploy within cloud, edge, and client collaborative computing frameworks to meet the different users' needs for Quality of Service (QoS). Existing large model inference scheduling technologies suffer from the following problems: Request heterogeneity leads to low resource utilization: Inference requests in LLM are highly heterogeneous, with huge differences between the length of the input text and the more critical, unknown output text. Traditional scheduling methods, such as first-in-first-out (FIFO), are prone to "head-of-line blocking," where a computationally intensive and time-consuming request will block the processing of a large number of short requests that follow, resulting in an increase in the average system response time and idle computing resources such as GPUs.

[0003] Unknown output length leads to blind scheduling decisions: The time required for LLM inference and GPU memory (especially the memory used to store key-value pairs, KV Cache) are strongly correlated with the number of output tokens. During scheduling, due to the unknown output length, the dynamic batch scheduling module cannot accurately estimate the resource consumption of each request, resulting in the inability to make optimal batching decisions. This may lead to overly conservative batch building and insufficient GPU utilization, or overly aggressive batch building, resulting in memory overflow or request timeout during inference, which seriously affects service stability.

[0004] Static batch processing strategies have poor adaptability: To improve processing efficiency, the industry generally adopts batch processing technology. However, static batch processing (i.e., fixed batch size) cannot adapt to dynamically changing request load and request characteristics. In scenarios with high concurrency or diverse request types, fixed-size batches are difficult to balance high throughput and low latency, leading to resource waste or service quality degradation.

[0005] Therefore, how to comprehensively consider traditional scheduling algorithms and combine them with the novel characteristics of LLM inference requests to maximize system utilization while meeting diverse service quality constraints for users is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0006] The purpose of this invention is to provide a cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference. By introducing an output length prediction mechanism and combining it with an innovative jump-type dynamic batch processing strategy, it can construct batch processing tasks with the highest resource utilization and satisfy all request service quality constraints in each scheduling cycle, thereby significantly improving the system's throughput and request success rate.

[0007] To achieve the above objectives, this invention provides a cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference, including a request receiving module, an output length prediction module, a resource consumption estimation module, a dynamic batch processing scheduling module, and a server module. The request receiving module is responsible for inputting the inference request sent by the user side into the output length prediction module; The output length prediction module uses a pre-trained output length prediction model to predict the number of tokens predicted when inferring on the large language model LLM for inference requests issued by the user. The resource consumption estimation module uses the predicted number of tokens and the parameters of the server module to build a full-link resource consumption estimation model for each inference request to estimate batch feasibility. The dynamic batch processing scheduling module performs jump-style dynamic batch processing based on the batch feasibility of the resource consumption estimation module, outputs the constructed batch, and repeats the construction of the next batch. The server module performs inference calculations based on the constructed batches and feeds back the inference calculation results to the user.

[0008] Preferably, the request receiving module receives inference requests from the user side, including input text and user-defined quality of service constraints. The quality of service constraints include a preference constraint for routing requests to specific processing nodes and an end-to-end latency constraint.

[0009] Preferably, the output length prediction model in the output length prediction module is set to BERT based on the Transformer architecture, and the request receiving module inputs the input text in the inference request to the output length prediction module.

[0010] Preferably, the batch feasibility estimated by the resource consumption estimation module includes end-to-end latency constraints and server memory usage; the end-to-end latency constraints include uplink communication latency, computation inference latency, and downlink communication latency; The resource consumption estimation module, based on the Transformer architecture of the Large Language Model (LLM), quantitatively calculates the memory usage and inference latency of the batch to be constructed. The specific details are as follows: Obtain the hardware parameters of the server module and the architecture parameters of the large language model. The hardware parameters of the server module include GPU computing power. Video memory capacity Uplink bandwidth Downlink bandwidth The architectural parameters of a large language model include the number of layers. Hidden layer dimension Attention count Dimensions of each head Parameter precision in bytes ; Establish a full-link memory consumption model: For a batch containing multiple inference requests, calculate its total memory usage. Total video memory usage Including static model weight memory and dynamic key-value pair cache memory ,Right now ; Perform a feasibility assessment of video memory: If If so, it is determined that the batch can be used in video memory. If so, it is determined that the batch cannot be used for video memory purposes; Establish an end-to-end latency consumption model: end-to-end latency Uplink communication delay Inference computation delay and downlink communication latency Composition, that is : Then, the batch feasibility is determined based on the end-to-end latency constraints and server memory usage. If it is feasible, the batch feasibility is output as passed; if it is not feasible, the batch feasibility is output as rejected.

[0011] Preferred total video memory usage The calculations include the following: The formula for calculating the static model weight memory, i.e., the fixed memory required to load the LLM model parameters, is as follows: The total number of architectural parameters of a large language model; The contents of the dynamic key-value cache memory are calculated as follows: For each inference request in the batch Based on the length of its input text and the number of predicted tokens obtained from the output length prediction module The maximum dynamic key-value cache memory usage generated during inference is calculated. The total dynamic memory usage of the batch is the sum of the dynamic key-value cache memory usage of all inference requests. The calculation formula is as follows: ; Here, coefficient 2 represents the two vector matrices Key and Value. Represents the length of the entire sequence.

[0012] Preferably, end-to-end latency The calculation process is as follows: Calculated based on the amount of input text data and uplink bandwidth. Calculated based on the predicted number of tokens and downlink bandwidth. ; Based on the inference characteristics of Transformer, the inference process is divided into an initial stage and an autoregressive stage, and the inference computation latency is... Output length of all inference requests within the batch The relationship is non-linear, and the processor's computing speed is... The estimation was performed using a multinomial regression model: ; in, This represents the computational cost of processing the initial input phase. This represents the linear computational cost of multiplying the model weight matrix when predicting the number of tokens. This indicates that the computational cost of the attention mechanism increases quadratically with the sequence length during the autoregressive phase. These are the preset constant coefficients related to the specific model architecture.

[0013] Preferably, the constraints for determining batch feasibility in the resource consumption estimation module are as follows: After adding inference requests, will the total memory usage of the batch exceed the available memory limit of each server in the server module? After adding inference requests, does the estimated completion time of all inference requests within the batch still meet their respective quality of service constraints? If all conditions are met, the batch feasibility is output as passed; if any condition is not met, the batch feasibility is output as rejected.

[0014] Preferably, the specific working process of the dynamic batch processing scheduling module is as follows: S1. Place all pending inference requests into a waiting queue and sort them in ascending order according to the urgency of each inference request's deadline. S2. Starting from the head of the waiting queue of S1, each sorted inference request is traversed sequentially. For the currently traversed inference request, the dynamic batch processing scheduling module performs a tentative addition operation to simulate adding the inference request to the current batch being built. After S3 and S2 are tentatively added to the simulation, the dynamic batch processing scheduling module uses the resource consumption estimation module to recalculate the total inference time and total memory usage of the entire batch and check whether the constraints are still met after adding the inference request. S4. If all constraints in S3 are met, the inference request is officially added to the current batch. If any condition is not met, the inference request is skipped and kept in the waiting queue. The process of repeating S2 and S3 continues to iterate through the next inference request in the queue until the waiting queue is exhausted or the current batch reaches its computing capacity limit. S5. Once a batch is built, it is immediately sent to the server module, while the dynamic batch processing scheduling module continues to build the next batch.

[0015] Preferably, when the dynamic batch processing scheduling module constructs a batch during its specific operation, if the batch being constructed is an empty set and the first inference request added does not meet the quality of service constraints, then the inference request is directly removed from the waiting queue, and a new inference request is extracted from the waiting queue to repeat the above verification process.

[0016] Therefore, the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference, which adopts the above-mentioned content, has the following advantages compared with the prior art: 1. This application significantly improves service quality assurance capabilities and request success rate. By predicting resource consumption and modeling end-to-end latency, it identifies requests that may time out during the scheduling phase and prioritizes allocating resources to requests that can be completed. Simulation experiments show that under high load, the request success rate of this application is improved by 150% and 29.6% compared to no batch processing and static batch processing, respectively. 2. This application significantly improves system throughput and GPU utilization. The skip-type batch processing mechanism allows the dynamic batch processing scheduling module to skip large requests that do not temporarily meet resource constraints and prioritize the processing of subsequent small requests, effectively avoiding head-of-line blocking. This allows the GPU to be filled as much as possible in each cycle, thereby maximizing concurrent processing capabilities and overall throughput. 3. This application enhances the stability and robustness of the system. The prediction-based resource assessment effectively reduces the risk of task failure due to resource overruns (especially GPU memory overflow), making the entire inference service system more stable and reliable in the face of dynamic and sudden requests.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is the overall framework diagram of the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference of this invention; Figure 2 This is a flowchart of the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to the present invention; Figure 3This is a dynamic batch processing effect diagram of the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to the present invention. Detailed Implementation

[0019] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0020] Example like Figures 1-3 As shown, the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference of the present invention includes a request receiving module, an output length prediction module, a resource consumption estimation module, a dynamic batch processing scheduling module, and a server module.

[0021] The request receiving module is responsible for inputting the inference request sent by the user side into the output length prediction module; The inference request issued by the user includes the input text and user-defined quality of service constraints. The quality of service constraints include the preference constraint for routing the request to a specific processing node and the end-to-end latency constraint.

[0022] The output length prediction module uses a pre-trained output length prediction model to predict the number of tokens predicted when inferring on the large language model LLM for inference requests issued by the user. The output length prediction model is set to the BERT (Bidirectional Encoder Representations from Transformers) model based on the Transformer architecture. The request receiving module inputs the input text in the inference request to the output length prediction module, and achieves accurate prediction of the output length by extracting the deep semantic features of the input text.

[0023] The resource consumption estimation module uses the predicted number of tokens and the parameters of the server module to build a full-link resource consumption estimation model for each inference request to estimate batch feasibility. The batch feasibility estimation module estimates by the resource consumption estimation module includes end-to-end latency constraints and server memory usage; end-to-end latency constraints include uplink communication latency, computation and inference latency, and downlink communication latency; The resource consumption estimation module, based on the Transformer architecture of the Large Language Model (LLM), quantitatively calculates the memory usage and inference latency of the batch to be constructed. The specific details are as follows: Obtain the hardware parameters of the server module and the architecture parameters of the large language model. The hardware parameters of the server module include GPU computing power. Video memory capacity Uplink bandwidth Downlink bandwidth The architectural parameters of a large language model include the number of layers. Hidden layer dimension Attention count Dimensions of each head Parameter precision in bytes ; Establish a full-link memory consumption model: For a batch containing multiple inference requests, calculate its total memory usage. Total video memory usage Including static model weight memory and dynamic key-value pair cache memory ,Right now ; Total video memory usage The calculations include the following: The formula for calculating the static model weight memory, i.e., the fixed memory required to load the LLM model parameters, is as follows: The total number of architectural parameters of a large language model; The contents of the dynamic key-value cache memory are calculated as follows: For each inference request in the batch Based on the length of its input text and the number of predicted tokens obtained from the output length prediction module The maximum dynamic key-value cache memory usage generated during inference is calculated. The total dynamic memory usage of the batch is the sum of the dynamic key-value cache memory usage of all inference requests. The calculation formula is as follows: ; Here, coefficient 2 represents the two vector matrices Key and Value. Represents the length of the entire sequence; Perform a feasibility assessment of video memory: If If so, it is determined that the batch can be used in video memory. If so, it is determined that the batch cannot be used for video memory purposes; Establish an end-to-end latency consumption model: end-to-end latency Uplink communication delay Inference computation delay and downlink communication latency Composition, that is : End-to-end latency The calculation process is as follows: Calculated based on the amount of input text data and uplink bandwidth. Calculated based on the predicted number of tokens and downlink bandwidth. ; Based on the inference characteristics of Transformer, the inference process is divided into an initial stage and an autoregressive stage, and the inference computation latency is... Output length of all inference requests within the batch The relationship is non-linear, and the processor's computing speed is... The estimation was performed using a multinomial regression model: ; in, This represents the computational cost of processing the initial input phase. This represents the linear computational cost of multiplying the model weight matrix when predicting the number of tokens. This indicates that the computational cost of the attention mechanism increases quadratically with the sequence length during the autoregressive phase. These are preset constant coefficients related to the specific model architecture; Then, the batch feasibility is determined based on the end-to-end latency constraints and server memory usage. If it is feasible, the batch feasibility is output as passed; if it is not feasible, the batch feasibility is output as rejected. The constraints for determining the feasibility of a batch are as follows: After adding inference requests, will the total memory usage of the batch exceed the available memory limit of each server in the server module? After adding inference requests, does the estimated completion time of all inference requests within the batch still meet their respective quality of service constraints? If all conditions are met, the batch feasibility is output as passed; if any condition is not met, the batch feasibility is output as rejected.

[0024] The dynamic batch processing scheduling module performs jump-style dynamic batch processing based on the batch feasibility of the resource consumption estimation module, outputs the constructed batch, and repeats the construction of the next batch. The specific working process of the dynamic batch processing scheduling module is as follows: S1. Place all pending inference requests into a waiting queue and sort them in ascending order according to the urgency of each inference request's deadline. S2. Starting from the head of the waiting queue of S1, each sorted inference request is traversed sequentially. For the currently traversed inference request, the dynamic batch processing scheduling module performs a tentative addition operation to simulate adding the inference request to the current batch being built. After S3 and S2 are tentatively added to the simulation, the dynamic batch processing scheduling module uses the resource consumption estimation module to recalculate the total inference time and total memory usage of the entire batch and check whether the constraints are still met after adding the inference request. S4. If all constraints in S3 are met, the inference request is officially added to the current batch. If any condition is not met, the inference request is skipped and kept in the waiting queue. The process of repeating S2 and S3 continues to iterate through the next inference request in the queue until the waiting queue is exhausted or the current batch reaches its computing capacity limit. S5. Once a batch is built, it is immediately sent to the server module, while the dynamic batch processing scheduling module continues to build the next batch. When the dynamic batch processing scheduling module is building a batch, if the batch being built is an empty set and the first inference request added does not meet the quality of service constraints, then the inference request is directly removed from the waiting queue, and a new inference request is extracted from the waiting queue to repeat the above verification process. The server module performs inference calculations based on the constructed batches and feeds back the inference calculation results to the user.

[0025] In the specific implementation process, such as Figure 2 As shown, this illustrates the testing process of the system in a simulation environment. In each time slot, based on the node... j Request queue ,ask Input length Arrival time Delay constraints The node's resource parameters, such as its computing power. Existing capacity (i.e., node) j Maximum capacity ), and hyperparameter cycle period and the time slot length allocated to the calculation within the period To perform computational processing within this time slot, during the scheduling and execution phase, the dynamic batch processing scheduling module employs a skip-style dynamic batch construction strategy within each computation time slot, first initializing the current batch. and temporary queue The system is set to an empty set and scans all arriving requests within the current time slot. For each inference request, the dynamic batch scheduling module attempts to add it to the current batch, based on the start time of this round. Arrival time The time that has passed in this round Uplink time Downlink overhead time Real-time evaluation of the overall temporary batch after its addition Feasibility under constraints of computational latency, memory usage, and end-to-end latency; if adding a request still satisfies all resource and latency constraints, then the request is included in the current batch, and the next request is considered; if adding a request causes any constraints to be violated, then the request is skipped and not included in the current batch. In practice, this is manifested as temporarily storing the data in a queue. And after this round of scanning is completed, the queue will be... The elements in the queue are put back into the waiting queue. middle.

[0026] After a complete scan is completed, if the current batch If the request is not empty, the dynamic batch processing scheduling module immediately submits the batch for inference computation and continues to attempt to build subsequent batches within the same time slot. If the head element does not meet the quality of service constraints, the request is directly removed from the queue. The dynamic batch processing scheduling module executes the above batch building process in a loop within each computation time slot until the time slot ends or all pending requests are scheduled. Through this skip-style dynamic batch processing algorithm, the system effectively improves batch utilization and overall system throughput while ensuring resource and latency constraints.

[0027] For example, REF _Ref216885944 \h \ MERGEFORMAT Figure 3 As shown, the performance of the skip-based dynamic batching strategy is demonstrated under different arrival rates. It can be observed that when the arrival rate is low, the processor can effectively handle the arriving requests, and the performance of the three strategies is similar. However, when the arrival rate increases, the performance of no batching decreases the most rapidly. Since no batching needs to repeatedly read different requests, its performance is poor. The inflection point of performance degradation for static batching is later. Dynamic batching, because it uses output length prediction information, can select the largest possible batch size for processing while ensuring user service quality constraints. This significantly improves the throughput of the entire system and results in a higher request completion rate.

[0028] Therefore, the cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference described above adopts the above-mentioned content. By introducing an output length prediction mechanism and combining it with an innovative jump-type dynamic batch processing strategy, it can construct batch processing tasks with the highest resource utilization and satisfy all request service quality constraints in each scheduling cycle, thereby significantly improving the system's throughput and request success rate.

[0029] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference, characterized by: It includes a request receiving module, an output length prediction module, a resource consumption estimation module, a dynamic batch processing scheduling module, and a server module; The request receiving module is responsible for inputting the inference request sent by the user side into the output length prediction module; The output length prediction module uses a pre-trained output length prediction model to predict the number of tokens predicted when inferring on the large language model LLM for inference requests issued by the user. The resource consumption estimation module uses the predicted number of tokens and the parameters of the server module to build a full-link resource consumption estimation model for each inference request to estimate batch feasibility. The dynamic batch processing scheduling module performs jump-style dynamic batch processing based on the batch feasibility of the resource consumption estimation module, outputs the constructed batch, and repeats the construction of the next batch. The server module performs inference calculations based on the constructed batches and feeds back the inference calculation results to the user.

2. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 1, characterized in that: The request receiving module receives inference requests from the user side, including input text and user-defined quality of service constraints. The quality of service constraints include the tendency to route the request to a specific processing node and the end-to-end latency constraints.

3. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 2, characterized in that: The output length prediction model in the output length prediction module is set to BERT based on the Transformer architecture. The request receiving module inputs the input text in the inference request to the output length prediction module.

4. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 1, characterized in that: The batch feasibility estimation module estimates by the resource consumption estimation module includes end-to-end latency constraints and server memory usage; end-to-end latency constraints include uplink communication latency, computation and inference latency, and downlink communication latency; The resource consumption estimation module, based on the Transformer architecture of the Large Language Model (LLM), quantitatively calculates the memory usage and inference latency of the batch to be constructed. The specific details are as follows: Obtain the hardware parameters of the server module and the architecture parameters of the large language model. The hardware parameters of the server module include GPU computing power. Video memory capacity Uplink bandwidth Downlink bandwidth The architectural parameters of a large language model include the number of layers. Hidden layer dimension Attention count Dimensions of each head Parameter precision in bytes ; Establish a full-link memory consumption model: For a batch containing multiple inference requests, calculate its total memory usage. Total video memory usage Including static model weight memory and dynamic key-value pair cache memory ,Right now ; Perform a feasibility assessment of video memory usage: If If so, it is determined that the batch can be used in video memory. If so, it is determined that the batch cannot occupy video memory. Establish an end-to-end latency consumption model: end-to-end latency Uplink communication delay Inference computation delay and downlink communication latency Composition, that is : Then, the batch feasibility is determined based on the end-to-end latency constraints and server memory usage. If it is feasible, the batch feasibility is output as passed; if it is not feasible, the batch feasibility is output as rejected.

5. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 4, characterized in that: Total video memory usage The calculations include the following: The formula for calculating the static model weight memory, i.e., the fixed memory required to load the LLM model parameters, is as follows: The total number of architectural parameters of a large language model; The contents of the dynamic key-value cache memory are calculated as follows: For each inference request in the batch Based on the length of its input text and the number of predicted tokens obtained from the output length prediction module The maximum dynamic key-value cache memory usage generated during inference is calculated. The total dynamic memory usage of the batch is the sum of the dynamic key-value cache memory usage of all inference requests. The calculation formula is as follows: ； Here, coefficient 2 represents the two vector matrices Key and Value. Represents the length of the entire sequence.

6. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 4, characterized in that: End-to-end latency The calculation process is as follows: Calculated based on the amount of input text data and uplink bandwidth. Calculated based on the predicted number of tokens and downlink bandwidth. ; Based on the inference characteristics of Transformer, the inference process is divided into an initial stage and an autoregressive stage, and the inference computation latency is... Output length of all inference requests within the batch The relationship is non-linear, and the processor's computing speed is... The estimation was performed using a multinomial regression model: ； in, This represents the computational cost of processing the initial stage of input. This represents the linear computational cost of multiplying the model weight matrix when predicting the number of tokens. This indicates that the computational cost of the attention mechanism increases quadratically with the sequence length during the autoregressive phase. These are the preset constant coefficients related to the specific model architecture.

7. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 4, characterized in that: The constraints for determining batch feasibility in the resource consumption estimation module are as follows: After adding inference requests, will the total memory usage of the batch exceed the available memory limit of each server in the server module? After adding inference requests, does the estimated completion time of all inference requests within the batch still meet their respective quality of service constraints? If all conditions are met, the batch feasibility is output as passed; if any condition is not met, the batch feasibility is output as rejected.

8. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 1, characterized in that: The specific working process of the dynamic batch processing scheduling module is as follows: S1. Place all pending inference requests into a waiting queue and sort them in ascending order according to the urgency of each inference request's deadline. S2. Starting from the head of the waiting queue of S1, each sorted inference request is traversed sequentially. For the currently traversed inference request, the dynamic batch processing scheduling module performs a tentative addition operation to simulate adding the inference request to the current batch being built. After S3 and S2 are tentatively added to the simulation, the dynamic batch processing scheduling module uses the resource consumption estimation module to recalculate the total inference time and total memory usage of the entire batch and check whether the constraints are still met after adding the inference request. S4. If all constraints in S3 are met, the inference request is officially added to the current batch. If any condition is not met, the inference request is skipped and kept in the waiting queue. The process of repeating S2 and S3 continues to iterate through the next inference request in the queue until the waiting queue is exhausted or the current batch reaches its computing capacity limit. S5. Once a batch is built, it is immediately sent to the server module, while the dynamic batch processing scheduling module continues to build the next batch.

9. The cloud-edge-device collaborative dynamic batch processing scheduling system for large language model inference according to claim 8, characterized in that: In the specific operation of the dynamic batch processing scheduling module, when constructing a batch, if the batch being constructed is an empty set and the first inference request added does not meet the quality of service constraints, the inference request is directly removed from the waiting queue, and a new inference request is extracted from the waiting queue to repeat the above verification process.

Images