A Hybrid Request Queue Scheduling Method for Service Level Objective (SLO) Awareness in Intelligent Agent Systems

Abstract

This invention discloses a hybrid request queue scheduling method for SLO awareness in intelligent agent systems, belonging to the field of intelligent agent service systems. Based on the constantly changing online request load of different agents, it fully utilizes task characteristics and global information during request processing to improve system resource utilization while maximizing the satisfaction of request latency targets. The invention first fine-tunes the DistillBERT-base based on the Alpaca dataset to predict the length of low-latency requests similar to simple chat requests. Then, it designs a waiting queue and its management module to calculate the priority of requests in the queue for scheduling decisions by the global scheduler. Next, it designs an SLO monitor to record the latency information of requests in the system and feeds it back to the global scheduler to coordinate the scheduling frequency of each queue. Finally, it conducts experimental verification using the Alpaca question-answering dataset and the application of rStar in complex mathematical problems as request load. Through the synergistic effect of the various parts of the SLO-MRQS framework, the request SLO achievement rate and resource utilization can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent service systems, and more particularly to an intelligent agent service system that needs to handle requests for different Service Level Objects (SLOs). Background Technology

[0002] Intelligent agents, as advanced artificial intelligence entities, refer to systems capable of autonomously perceiving their environment, making decisions, and executing actions. With the rapid development of Large Language Model (LLM) technology, a powerful cognitive and reasoning foundation has been provided for the development of intelligent agents. Current intelligent agent applications include not only simple chatbots but also analytical agents that need to handle complex planning requests. As the core decision engine of intelligent agents, LLM enhances reasoning capabilities through methods such as thought chains, cue learning, and Monte Carlo Tree Search, providing effective decision support for the interaction between intelligent agents and their external environment. In particular, Monte Carlo Tree Search (MCTS), as a heuristic search method, when combined with LLM, effectively compensates for the limitations of LLM in complex reasoning tasks by systematically executing four core steps: selection, expansion, simulation, and backtracking to construct a search tree.

[0003] Existing mainstream service systems (such as vLLM and SGLang) have limitations in request scheduling. These systems primarily aim to optimize throughput, employing techniques like continuous batch processing and PagedAttention to significantly improve throughput and reduce operating costs. However, they fail to consider the differences in Service Level Objectives (SLOs) and processing logic among different agents, resulting in low overall SLO achievement rates and resource utilization. Due to the autoregressive generative nature of the Transformer architecture, the output length of requests for large language models cannot be obtained in advance. This leads to unpredictable online workloads related to large language models, with unpredictable request arrival times and quantities, as well as unpredictable processing time. This multi-layered unpredictability poses a greater challenge to the scheduling of agent service systems. Regarding scheduling strategies, existing systems primarily use a first-come, first-served (FFS) scheduling strategy, which is prone to head-of-line blocking. Because these systems optimize on a request-by-request basis, ignoring the dependencies between multiple model calls within a single request and global request information, complex requests may further block simpler requests. Furthermore, different requests have different service level objectives; deploying multiple clusters to handle requests with different SLOs increases enterprise operating costs. Existing systems do not consider how to handle requests with different SLOs (Solution Time Limits). For example, real-time agent applications that need to strictly meet latency requirements (such as chatbots) and analytical tasks that can tolerate longer response times. If there is a lack of effective strategies at the scheduling level, it will lead to severe head-of-line blocking, resulting in insufficient SLO achievement rate.

[0004] The above problems can be solved by designing relevant scheduling strategies, but currently, mainstream service systems have not systematically integrated SLO awareness into scheduling decisions. As agent application scenarios become increasingly complex, agent requests often involve multiple steps, multiple model calls with dependencies, and simple requests requiring rapid responses. Existing scheduling frameworks cannot effectively manage these requests with mixed SLOs, resulting in low SLO achievement rates and a decline in overall system service quality. Therefore, there is an urgent need for a hybrid request queue scheduling method that can perceive SLO requirements and understand the internal call dependencies of requests to improve the resource utilization efficiency and service quality assurance capabilities of agent service systems. Summary of the Invention

[0005] This invention addresses the problems existing in the aforementioned background technology by disclosing a hybrid request queue scheduling method based on SLO awareness for intelligent agent systems. By fully utilizing task characteristics and global information during request processing, and based on the continuous changes in online requests from different agents, this method improves system resource utilization while maximizing the satisfaction of request latency targets. The invention includes the following steps:

[0006] Step 1: Train a length prediction model based on the DistillBERT-base model and the Alpaca dataset. The dataset contains several requests. ,ask With input length The actual output length is Since predicting the exact length of the request's output is difficult, we predict the range within which the request's output length might fall. Let the number of ranges be . The maximum value of the output length is Then the range of each interval is ,Exceed Divide into the last interval.

[0007] Step 2: Design the SLO-MRQS (SLO-aware Mixed-Request Queue Scheduling) framework. The SLO-MRQS framework consists of three parts: a waiting queue, an SLO monitor, and a global scheduler. The waiting queue is responsible for receiving requests and calculating request priorities. The SLO monitor is responsible for recording SLO violation information for requests in the system. The global scheduler adjusts the scheduling probability of the queues based on the feedback from the waiting queue and the SLO monitor, ensuring the system's SLO achievement rate and resource utilization.

[0008] 1) Waiting queue. Assume the service system has a waiting queue at time intervals. Internally received A request, i.e., a set of requests. Because different agents send different requests, including low-latency requests similar to chat requests, and complex planning requests requiring the MCTS method, the latter involves calling the model multiple times to make a more reasonable decision. The SLO requirements for these different requests are drastically different, with the latter being significantly more lenient than the former. SLO-MRQS sets up a waiting queue. Each queue receives requests with similar SLOs (Solution Time Limits). Each queue has a queue management module that calculates request priorities based on global latency information and schedules the highest-priority requests. Taking into account both time urgency and processing overhead, the request priority calculation formula is as follows:

[0009]

[0010] in Represents the current timestamp. Representative request The timestamp of entering the system. Representing the processing of requests Expenses, and These are parameters used to control the weights of each component.

[0011] When the queue contains low-latency requests similar to chat requests, the specific priority calculation formula is as follows:

[0012]

[0013] in This indicates the deadline for generating the first word element of the request. This indicates that in step 1, the predicted length model corresponds to the request. The predicted interval markings.

[0014] When the queue contains complex planning requests that require the use of the MCTS method, the specific priority calculation formula is as follows:

[0015]

[0016] in That is, request The This call This indicates the end-to-end time at which the entire request is processed. Indicates the relative overhead of the call.

[0017] 2) SLO Monitor. Assume the user's reading speed is... The unit is seconds per word. For the request The Middle The deadline for each word element includes the time of requesting entry into the system, the deadline for generating the first word element, and the deadline for the user to read the word element. The specific formula is as follows:

[0018]

[0019] When processing complex planning requests from analytical agents, the primary latency metric is end-to-end latency. Its end-to-end deadline is This includes the time the request enters the system and the deadline for processing the entire request. The specific formula is as follows:

[0020]

[0021] SLO monitor settings queue monitoring targets , This is the monitoring threshold. When... Exceed In such cases, the SLO monitor records violations and reports them to the global scheduler as a basis for scheduling. In online chat applications, if the word output speed cannot match the user's reading speed, output pauses will significantly impact user satisfaction. Specifically, when there are simple chat requests in the queue, the monitoring metric is... This is designed to monitor the smoothness of output for each requested token. When handling complex planning requests, the focus is on whether the request can be completed on time; the main latency metric is end-to-end latency, and the monitoring metric is... The specific formula is as follows:

[0022]

[0023]

[0024] in Indicates the first The first request The completion time of each word element Indicates the first The completion time of each request.

[0025] 3) Global Scheduler. The SLO-MRQS framework uses a global scheduler to coordinate all request waiting queues based on task characteristics and global information fed back from the waiting queues and the SLO monitor. The optimization goal of the global scheduler is to maximize the number of requests that fulfill SLOs. Definition For the request Throughput under SLO conditions, expressed in units of tokens. Indicates a request Whether it is scheduled into the system, Indicates scheduling, This indicates no scheduling. This is an indicator function; it is 1 if the condition is true, and 0 otherwise.

[0026]

[0027] The scheduling process needs to consider the system's video memory capacity, and the proportion of video memory occupied by running requests needs to be maintained at a certain level. To maintain system stability, the model is designed to address the following optimization problem:

[0028]

[0029]

[0030] The optimization problem described above can be reduced to a 0-1 knapsack problem with double boundary constraints, hence it is NP-hard. SLO-MRQS employs a heuristic algorithm to satisfy request latency requirements as much as possible while maximizing request throughput. The global scheduler first initializes the base weights of each waiting queue and calculates the scheduling probability of the queue based on these weights. Then, based on the global latency information fed back by the SLO monitor, it calculates the weights of each waiting queue to ensure that requests with different SLOs can all receive scheduling opportunities.

[0031] Initialize the waiting queue The base weight is ,in For queue The base scheduling probability is 1, and the sum of the scheduling probabilities of all queues is 1.

[0032]

[0033] It is a queue In time The achievement rate of the monitoring target is calculated using the following formula:

[0034]

[0035] in, For queue In the monitoring window The request set within, It is a queue The monitoring target This is the monitoring threshold.

[0036] The specific formula for determining the violation score is as follows:

[0037]

[0038] Queue In time The degree of violation, It is the expected achievement rate.

[0039] In time ,queue The weighting is dynamically adjusted based on the violation score, using the following formula:

[0040]

[0041] in, This is the gain coefficient for violations. The threshold for triggering violations, To restore the trigger threshold, This is the time interval since the last adjustment.

[0042] queue The actual scheduling probability is adjusted through dynamic weights, as shown in the following formula:

[0043]

[0044] To translate probabilities into actual scheduling behavior, a deterministic scheduling algorithm is designed to generate a sequence of queues that satisfy a specific scheduling order based on a given probability distribution. The input includes the probability distribution. and the number of times the monitoring window is scheduled The output is a scheduling sequence. Each element Indicates the first The queue number for this scheduling. First, calculate the target number of scheduling iterations for each queue based on the probability distribution. (like The remaining attempts are then allocated proportionally, prioritizing queues with higher probabilities, and an array of remaining attempts is constructed. and initialize the current queue pointer. and empty result sequence Then the main polling scheduling loop begins, processing each scheduling operation. ,from Start the loop to find the first remaining number of times. queue index , queue Add to the result sequence Update remaining attempts And move the pointer to the next queue. This process ensures the probability ratio of matching schedule sequences by controlling the fixed polling order and remaining number of polls. At the same time, it generates a unique and definite scheduling order.

[0045] Step 3: Inject a mixed request load into the system for application verification. The Alpaca question-and-answer dataset is used as a low-latency, pause-sensitive chat request load. rStar utilizes the MCTS algorithm to solve complex mathematical problems as a complex planning request load with multiple calls, and gamma distribution simulates request arrival. Through the synergistic effect of the various parts of the SLO-MRQS framework, the request SLO achievement rate and resource utilization can be effectively improved. Attached Figure Description

[0046] Figure 1 This is a flowchart of the present invention;

[0047] Figure 2 This is the system architecture diagram of the present invention. Detailed Implementation

[0048] like Figure 2 As shown, the present invention includes the following steps:

[0049] Step 1: Using DistillBERT-base as the base model, fine-tune it on the Alpaca dataset to achieve request length prediction. The dataset contains several requests. ,ask With input length The actual output length is Since predicting the exact length of the request's output is difficult, we predict the range within which the request's output length might fall. Let the number of ranges be . The maximum value of the output length is Then the range of each interval is ,Exceed Divide into the last interval. Analyze the length distribution of the Alpaca dataset, and let... The value is 1024. This was explored through experiments. The optimal value of is determined so that the accuracy of the predicted length model reaches the ideal value. After obtaining the value, request a predicted length. The value is 1 to Integers between [a certain range].

[0050] Step 2: Design the SLO-MRQS (SLO-aware Mixed-Request Queue Scheduling) framework. The SLO-MRQS framework consists of three parts: a waiting queue, an SLO monitor, and a global scheduler. The waiting queue is responsible for receiving requests and calculating request priorities. The SLO monitor is responsible for recording SLO violation information for requests in the system. The global scheduler adjusts the scheduling probability of the queues based on the feedback from the waiting queue and the SLO monitor, ensuring the system's SLO achievement rate and resource utilization.

[0051] 1) Waiting queue. Assume the service system has a waiting queue at time intervals. Internally received A request, i.e., a set of requests. Because different agents send different requests, including low-latency requests similar to chat requests, and complex planning requests requiring the MCTS method, the latter involves multiple model calls to make a more reasonable decision. The SLOs (Standardized Time Limits) vary significantly between these requests, with the latter being considerably more lenient than the former. SLO-MRQS sets up a waiting queue. Each queue receives requests with similar SLOs (Solution Time Limits). Each queue has a queue management module that calculates request priorities based on global latency information and schedules the highest-priority requests. The priority calculation formula is as follows, taking into account both time urgency and processing overhead:

[0052]

[0053] in Represents the current timestamp. Representative request The timestamp of entering the system. Representing the processing of requests Expenses, and The parameter is used to control the weight of each part. Taking a simple chat request and a complex planning request as examples, the different queue management modules calculate the request priority as follows.

[0054] When the queue contains low-latency requests similar to chat requests, the specific priority calculation formula is as follows:

[0055]

[0056] in This indicates the deadline for generating the first word element of the request. This indicates that in step 1, the predicted length model corresponds to the request. The predicted interval markings.

[0057] When the queue contains complex planning requests that require the use of the MCTS method, the specific priority calculation formula is as follows:

[0058]

[0059] in That is, request The This call This indicates the end-to-end time for processing the entire request. Taking the rStar algorithm as an example, the model call details for expanding different child nodes in the MCTS algorithm are shown in the table below. This represents the relative scaling overhead. A higher number of child nodes indicates more model calls required for the next step of scaling, resulting in greater latency overhead.

[0060]

[0061] 2) SLO Monitor. Assume the user's reading speed is... The unit is seconds per word, and it was set to 0.15 seconds per word in the experiment. For the request The Middle The deadline for each word element includes the time of requesting entry into the system, the deadline for generating the first word element, and the deadline for the user to read the word element. The specific formula is as follows:

[0062]

[0063] When processing complex planning requests from analytical agents, the primary latency metric is end-to-end latency. Its end-to-end deadline is This includes the time the request enters the system and the deadline for processing the entire request. The specific formula is as follows:

[0064]

[0065] SLO monitor settings queue monitoring targets , This is the monitoring threshold. When... Exceed In such cases, the SLO monitor records violations and reports them to the global scheduler as a basis for scheduling. In online chat applications, if the word output speed cannot match the user's reading speed, output pauses significantly impact user satisfaction. Specifically, when there are simple chat requests in the queue, the monitoring metric is... This is designed to monitor the smoothness of output for each requested token. When handling complex planning requests, the focus is on whether the request can be completed on time; the main latency metric is end-to-end latency, and the monitoring metric is... The specific formula is as follows:

[0066]

[0067]

[0068] Once a chat request term is generated, it enters a buffer awaiting output. Because pauses significantly impact user satisfaction, the SLO monitor streams request terms based on the user's reading speed. Outputting chat request terms immediately after generation creates pressure for subsequent terms; any pauses in outputting later terms will decrease user satisfaction.

[0069] 3) Global Scheduler. The SLO-MRQS framework uses a global scheduler to coordinate various request queues based on task characteristics and global information fed back from the waiting queues and SLO monitors. The optimization goal of the global scheduler is to maximize the number of requests that fulfill SLOs. Definition For the request Throughput under SLO conditions, expressed in units of tokens. Indicates a request Whether it is scheduled into the system, Indicates scheduling, This indicates no scheduling. This is an indicator function; it is 1 if the condition is true, and 0 otherwise.

[0070]

[0071] The scheduling process needs to consider the system's video memory capacity, and the proportion of video memory occupied by running requests needs to be maintained at a certain level. To maintain system stability. When the system requests more than [a certain amount of video memory] If the request is less than a certain threshold, then the incoming request will be stopped to prevent it from affecting currently running requests. At that time, requests are allowed to enter. In summary, the model is the following optimization problem:

[0072]

[0073]

[0074] The optimization problem described above can be reduced to a 0-1 knapsack problem with double boundary constraints, hence it is NP-hard. SLO-MRQS employs a heuristic algorithm to satisfy request latency requirements as much as possible while maximizing resource utilization. The global scheduler first initializes the basic weights of each waiting queue and calculates the scheduling probability of the queue based on these weights. Then, based on the global latency information fed back by the SLO monitor, it calculates the weights of each waiting queue to ensure that requests with different SLOs can all receive scheduling opportunities.

[0075] Initialize the waiting queue The base weight is ,in For queue The base scheduling probability is 1, and the sum of the scheduling probabilities of all queues is 1.

[0076]

[0077] It is a queue In time The rate of achievement of monitoring targets For queue In the monitoring window The request set within, It is a queue The monitoring target The monitoring threshold is calculated using the following formula:

[0078]

[0079] The score for the violation represents the queue. In time The degree of violation, This is the expected achievement rate, and the specific formula is as follows:

[0080]

[0081] In time ,queue Weights are dynamically adjusted based on violation scores:

[0082]

[0083] in, This is the gain coefficient for violations. The threshold for triggering violations, To restore the trigger threshold, This is the time interval since the last adjustment.

[0084] queue The actual scheduling probability is adjusted through dynamic weights, as shown in the following formula:

[0085]

[0086] To translate probabilities into actual scheduling behavior, a deterministic scheduling algorithm is designed to generate a sequence of queues that satisfy a specific scheduling order based on a given probability distribution. The input includes the probability distribution. and the number of times the monitoring window is scheduled The output is a scheduling sequence. Each element Indicates the first The queue number for this scheduling. First, calculate the target number of scheduling iterations for each queue based on the probability distribution. (like The remaining attempts are then allocated proportionally, prioritizing queues with higher probabilities, and an array of remaining attempts is constructed. and initialize the current queue pointer. and empty result sequence Then the main polling scheduling loop begins, processing each scheduling operation. ,from Start the loop to find the first remaining number of times. queue index , queue Add to the result sequence Update remaining attempts And move the pointer to the next queue. This process ensures the probability ratio of matching schedule sequences by controlling the fixed polling order and remaining number of polls. At the same time, it generates a unique and definite scheduling order.

[0087] Once a request is completed, the global scheduler immediately schedules new requests from the waiting queue into the system. After each monitoring window, if a SLO violation occurs, the global scheduler adjusts the scheduling frequency between the waiting queues according to the algorithm described above. When chat request tokens are generated, they are streamed into the buffer, awaiting guidance from the SLO monitor for streamed output.

[0088] Step 3: Inject a mixed request load into the system for application verification. The Alpaca question-and-answer dataset is used as a low-latency, pause-sensitive chat request load. rStar utilizes the MCTS algorithm to solve complex mathematical problems as a complex planning request load with multiple calls, and gamma distribution simulates request arrival. Through the synergistic effect of the various parts of the SLO-MRQS framework, the request SLO achievement rate and resource utilization can be effectively improved.

Claims

1. A hybrid request queue scheduling method for service level objectives (SLOs) of intelligent agent systems, characterized in that, include: Step 1: Based on the DistillBERT-base model and the Alpaca dataset, train a classification model to predict the range of request output length, where the output length refers to the number of tokens in the request response. Step 2: Based on the predicted length model obtained in Step 1, design the SLO-aware hybrid request queue scheduling framework SLO-MRQS, which includes three parts: a waiting queue, an SLO monitor, and a global scheduler. The waiting queue is responsible for receiving requests and calculating request priorities. The SLO monitor is responsible for recording SLO violation information of requests in the system. The global scheduler adjusts the scheduling probability of the queue based on the feedback from the waiting queue and the SLO monitor to ensure the system's SLO achievement rate and resource utilization. Step 3: Based on the SLO-MRQS framework designed in Step 2, inject mixed request load into the system for application verification; Step 2, in which the global scheduler adjusts the scheduling probability of the queues based on the feedback from the waiting queues and the SLO monitor, also includes the following steps: Initialize the waiting queue The basic weight is ,in For queue The base scheduling probability is 1, and the sum of the scheduling probabilities of all queues is 1. in, This indicates the total number of queues. Representative queue The basic weights; It is a queue In time The rate of achievement of monitoring targets For queue In the monitoring window The request set within, It is a queue The monitoring target The monitoring threshold is defined by the following formula: in, This is an indicator function; it returns 1 if the condition is true, and 0 otherwise. The score for the violation represents the queue. In time The degree of violation, This is the expected achievement rate, and the specific formula is as follows: In time ,queue The weighting is dynamically adjusted based on the violation score, using the following formula: in, This is the gain coefficient for violations. The threshold for triggering violations, To restore the trigger threshold, The time interval since the last adjustment; queue The actual scheduling probability is adjusted through dynamic weights, as shown in the following formula: in, Representing time The weight of queue j after the update.

2. The hybrid request queue scheduling method for SLO awareness in intelligent agent systems according to claim 1, characterized in that, The waiting queue Each queue receives requests with similar SLOs (Solution Time Limits). Each queue has a queue management module that calculates request priorities based on global latency information and schedules the highest-priority request. The priority calculation formula is as follows, taking into account both time urgency and processing overhead: in Represents the current timestamp. Representative request The timestamp of entering the system. Representative request Service level goals Representative handles the request Expenses, and The parameter is used to control the weight of each part; when the queue contains low-latency requests similar to chat requests, the specific priority calculation formula is as follows: in This indicates the deadline for generating the first word element of the request. This indicates that in step 1, the predicted length model corresponds to the request. The predicted interval label; when the queue contains complex planning requests that require the use of the Monte Carlo Tree Search (MCTS) method, the specific priority calculation formula is as follows: in This indicates the end-to-end time at which the entire request is processed. That is, request The This call, using This indicates the relative scaling overhead. The more scaling nodes there are, the more times the model will be called in the next step, resulting in greater latency overhead.

3. The hybrid request queue scheduling method for SLO awareness in intelligent agent systems according to claim 1, characterized in that, The SLO monitor sets the user's reading speed to... The unit is seconds per word; For the request The Middle The deadline for each word element includes the time of requesting entry into the system, the deadline for generating the first word element, and the deadline for the user to read the word element. The specific formula is as follows: in This indicates the deadline for generating the first word element of the request. Representative request The timestamp of entering the system; while when processing complex planning requests from analytical agents, the main latency metric is end-to-end latency. Its end-to-end deadline is This includes the time the request enters the system and the deadline for processing the entire request. The specific formula is as follows: in This indicates the end-to-end time at which the entire request is processed. Representative request The timestamp of entering the system; SLO monitor settings queue monitoring targets , For monitoring thresholds; when Exceed At that time, the SLO monitor will record violations and report them to the global scheduler for scheduling decisions; specifically, when there are simple chat requests in the queue, the monitoring metric is... The purpose is to monitor the smoothness of output for each requested term; when handling complex planning requests, the focus is on whether the request can be completed on time, i.e., the main latency metric is end-to-end latency, and the monitoring metric is... The specific formula is as follows: in This represents the completion time of the j-th word element in the i-th request. This represents the completion time of the i-th request. For the request The deadline for the j-th word element. For the request The end-to-end deadline.

4. The hybrid request queue scheduling method for SLO awareness in intelligent agent systems according to claim 1, characterized in that, The global scheduler, in order to convert probabilities into actual scheduling behavior, designs a deterministic scheduling algorithm that generates a queue sequence that satisfies a specific scheduling order based on a given probability distribution; the input includes the probability distribution. and the number of times the monitoring window is scheduled The output is a scheduling sequence. Each element Indicates the first The queue number for this scheduling; first, calculate the target scheduling count for each queue based on the probability distribution. ,like The remaining attempts will be allocated proportionally, prioritizing queues with higher probabilities, because... Since it rounds down, the total number of scheduling times for all queues will not exceed [a certain number]. ,like If the remaining number of times is 0, then according to Allocate scheduling attempts and construct an array of remaining attempts. and initialize the current queue pointer. and empty result sequence ; Then the main polling scheduling loop begins, and each scheduling operation is performed... ,from Start the loop to find the first remaining number of times. Given queue index i, add queue i+1 to the result sequence. Update remaining attempts And move the pointer to the next queue. This process ensures the matching probability ratio of the scheduling sequence by controlling the fixed polling order and the remaining number of polls. At the same time, it generates a unique and definite scheduling order.

Images