Methods, apparatus, devices, and media for inference request scheduling based on reinforcement learning

Abstract

This application relates to the field of artificial intelligence technology and discloses a method, apparatus, device, and medium for scheduling inference requests based on reinforcement learning. Applied to an inference cluster routing and scheduling system, the method includes: determining a target model instance matching the scheduled inference request based on the current state space used to characterize the system's operation; receiving the results returned by the target model instance after processing the inference request through a pre-filling stage and a decoding stage, and returning them to the user; wherein the reinforcement learning model places the evaluation expectation value and action selection in the target network and the main network, respectively; the main network outputs the expectation values ​​of each action based on the current state space and selects the action with the highest expectation value; the target network evaluates the target expectation value after executing the selected action and updates the network parameters to complete the training of the reinforcement learning model. The technical solution provided by this application can balance the node load balancing of model instances and the KV cache hit rate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for scheduling inference requests based on reinforcement learning. Background Technology

[0002] With the increasing application of Large Scale Language Models (LLM), the efficiency and low latency of inference services have become core challenges. The efficiency of LLM inference not only depends on the full utilization of computing resources but is also constrained by factors such as key-value cache matching rate, load balancing, and the adaptability of computing resources. The key-value cache matching rate directly affects inference efficiency; proper utilization of cached data can reduce computational load and improve response speed. Load balancing ensures the efficient allocation of computing resources and memory bandwidth, avoiding excessive resource consumption and increased latency.

[0003] However, existing coarse-grained scheduling algorithms ignore task differences and struggle to cope with complex computing resources, and traditional heuristic algorithms fail to achieve optimal results when dealing with diverse environments.

[0004] Therefore, how to balance the node load balancing of model instances and the KV cache hit rate to avoid inefficiency and increased latency is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for scheduling inference requests based on reinforcement learning, which achieves the technical effect of balancing node load and KV cache hit rate of model instances to avoid inefficiency and increased latency.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include:

[0007] In a first aspect, embodiments of this application provide an inference request scheduling method based on reinforcement learning, applied to an inference cluster routing scheduling system, the method comprising:

[0008] Get the inference request submitted by the user;

[0009] Based on the current state space used to characterize the system's operation, a target model instance matching the inference request is determined through a pre-built reinforcement learning model.

[0010] The system receives the result returned by the target model instance after processing the inference request through the pre-filling stage and the decoding stage, and returns the result to the user.

[0011] The reinforcement learning model places the evaluation expectation value and the action selection in the target network and the main network, respectively. The main network outputs the expectation value of each action according to the current state space and selects the action with the largest expectation value. The target network evaluates the target expectation value after executing the selected action and updates the network parameters to complete the training of the reinforcement learning model.

[0012] This embodiment provides a reinforcement learning-based inference request scheduling method. By intelligently scheduling inference requests through a reinforcement learning model, the system can dynamically select appropriate target model instances to process inference requests based on the current state space, thereby achieving precise resource allocation and load balancing. This intelligent scheduling strategy optimizes the allocation of computing resources according to task requirements and system load, avoiding unnecessary resource waste or computational bottlenecks. By rationally allocating computing resources, the system can avoid overload, reduce latency, and improve overall system efficiency. When processing inference requests, the system can efficiently utilize computing resources, ensuring low-latency responses and avoiding inefficiencies caused by resource contention or improper processing, thus improving the system's response speed and processing capacity.

[0013] In one implementation, the reinforcement learning model is trained in the following manner:

[0014] Obtain the node state representing the queue length of each model instance at different stages, the request state representing the input sequence length of the inference request, and the KV cache distribution state to construct the state space;

[0015] A set of actions that can be taken at each time step is defined as the action space; wherein, the set of actions includes scheduling the inference request to the corresponding model instance operation and not taking the scheduling operation;

[0016] Based on the state space and the action space, a reinforcement learning model is trained, the optimal action in the action space is selected according to the state space, and the target expected value after executing the optimal action is determined.

[0017] The main network selects the expected value corresponding to the optimal action. Based on the loss function between the expected value and the target expected value, the network parameters are updated by minimizing the loss function until a preset condition is met, thus obtaining a trained reinforcement learning model.

[0018] This embodiment constructs a state space by acquiring the node state, request state, and KV cache distribution state of each model instance. This state information helps the system understand the current computational load and cache utilization. Next, an action space is defined, containing operations such as scheduling requests to the most suitable model instance or not scheduling them. Through reinforcement learning training based on the state and action spaces, the system can select the optimal action according to the current state and predict the expected value after executing the action. Finally, by minimizing the loss function (i.e., the difference between the predicted expected value and the target expected value), the network parameters are updated, resulting in a well-trained reinforcement learning model. This model can optimize the KV cache matching rate, the reasonable allocation of load, and the adaptability of computing resources according to actual conditions, thereby improving system performance, reducing latency, and ensuring efficient inference processing.

[0019] In one implementation, the reward function in the reinforcement learning model is constructed as follows:

[0020] Obtain the token processing per second metric, the first response time metric, and the total processing time metric for the request scheduling of the reinforcement learning model;

[0021] The reward function is constructed based on the token processing per second metric, the first response time metric, and the total processing time metric.

[0022] In one implementation, the training process of the reinforcement learning model further includes:

[0023] Construct an experience replay buffer; wherein, the experience replay buffer is used to store experience data, the experience data including the current state, the action taken, the reward value obtained, the next state, and whether it is completed;

[0024] At each time step, the experience data is stored in the experience playback buffer;

[0025] During training, a batch of the experience data is randomly sampled from the experience replay buffer;

[0026] The network parameters of the reinforcement learning model are updated using the empirical data obtained from sampling.

[0027] This embodiment introduces an experience replay buffer, allowing the reinforcement learning model to store data such as the current state, actions taken, and rewards obtained at each time step. This data is then randomly sampled during training to update the reinforcement learning model. Combined with reinforcement learning, this optimizes key-value cache matching, load balancing, and computing resource adaptation, effectively improving service efficiency and response speed. The experience replay buffer enhances data utilization and gradient update stability, cache matching optimization reduces latency, load balancing ensures reasonable resource allocation, and computing resource adaptation guarantees efficient use of computing resources. Overall, by comprehensively optimizing multiple aspects, not only is system throughput improved, but performance bottlenecks are also effectively avoided, achieving efficient and stable inference services.

[0028] In one implementation, the training process of the reinforcement learning model further includes:

[0029] A greedy strategy is constructed by randomly selecting an action with a first preset probability as the exploration phase and selecting the optimal action in the current state with a second preset probability as the utilization phase; wherein the sum of the first preset probability and the second preset probability is 1.

[0030] At each time step, an action is selected based on the current state and the greedy strategy.

[0031] By adjusting the value of the first preset probability, the process shifts from the exploration phase to the utilization phase, thereby increasing the probability of selecting the optimal action and optimizing the training of the reinforcement learning model.

[0032] This embodiment uses an exploration phase to help the system randomly select different actions, thereby discovering potential optimization paths, such as improving cache hit rate and avoiding cache misses. The exploitation phase relies on known optimal strategies to ensure the system's stability in terms of computational resource allocation and load balancing. As the exploration probability gradually decreases, computational resources and load allocation can be dynamically adjusted based on real-time data, improving computational efficiency, reducing unnecessary resource waste, and thus avoiding inefficiency and increased latency. Through this dynamic strategy adjustment, reinforcement learning not only optimizes the training process but also achieves efficient resource utilization and performance improvement.

[0033] In one implementation, the training process of the reinforcement learning model further includes:

[0034] Initialize the weights of the target network so that the weights of the target network are consistent with the weights of the main network;

[0035] After each time step or each training batch, the weights of the target network are updated according to a pre-set soft update method.

[0036] This embodiment effectively improves stability and resource utilization efficiency by optimizing and updating the target network during reinforcement learning. The target network serves as a stable benchmark, avoiding fluctuations caused by frequent updates to the main network. This is similar to using a stable caching strategy to improve matching efficiency when optimizing the key-value cache matching rate. The optimization mechanism gradually brings the weights of the target network closer to those of the main network, facilitating dynamic adjustment of load distribution and optimization of computing resource allocation. This ensures load balancing and efficient resource utilization while reducing latency and inefficiency, thereby improving overall computational performance.

[0037] In one implementation, the pre-set soft update method is:

[0038] The first change value is determined based on the preset first update ratio and the weight of the main network;

[0039] The second change value is determined based on the preset second update ratio and the weights of the target network;

[0040] The first change value and the second change value are added together to obtain the updated weights of the target network;

[0041] Wherein, the sum of the first update ratio and the second update ratio is 1.

[0042] This embodiment adjusts the weight updates of the main network and the target network according to a preset update ratio, enabling a smooth transition and avoiding instability caused by excessive fluctuations. This update strategy controls the frequency of cache updates, avoiding over-reliance on any single cache and improving the overall matching rate and system stability. Simultaneously, this smooth update method also facilitates load balancing, preventing overload of a single computing node and ensuring reasonable resource allocation, thereby optimizing the adaptation of computing resources. Ultimately, the system reduces inefficiencies and increased latency, improving resource utilization and overall training performance during the computation process.

[0043] Secondly, embodiments of this application provide a reinforcement learning-based inference request scheduling apparatus, applied to an inference cluster routing scheduling system, the apparatus comprising:

[0044] The inference request acquisition unit is used to acquire inference requests submitted by users.

[0045] The model instance unit is used to determine the target model instance that matches the inference request by scheduling, based on the current state space used to characterize the system's operation, through a pre-built reinforcement learning model.

[0046] The processing result return unit is used to receive the result returned by the target model instance after processing the inference request through the pre-filling stage and the decoding stage in sequence, and return the result to the user;

[0047] The reinforcement learning model places the evaluation expectation value and the action selection in the target network and the main network, respectively. The main network outputs the expectation value of each action according to the current state space and selects the action with the largest expectation value. The target network evaluates the target expectation value after executing the selected action and updates the network parameters to complete the training of the reinforcement learning model.

[0048] Thirdly, embodiments of this application provide a computer device, including:

[0049] The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the aforementioned reinforcement learning-based inference request scheduling method.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the reinforcement learning-based inference request scheduling method described above. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a reinforcement learning-based inference request scheduling method provided in an embodiment of this application;

[0053] Figure 2 A flowchart illustrating the reinforcement learning model training method provided in the embodiments of this application;

[0054] Figure 3 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of the first performance optimization method;

[0055] Figure 4 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of a second performance optimization method;

[0056] Figure 5 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of a third performance optimization method;

[0057] Figure 6A flowchart illustrating a pre-configured soft update method provided in this application embodiment;

[0058] Figure 7 A block diagram of a reinforcement learning-based inference request scheduling device provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] As Large Scale Language Models (LLM) are increasingly applied across various services, the efficiency and low latency of inference services have become core challenges. The efficiency of LLM inference depends not only on the full utilization of computing resources but also on multiple factors such as the matching rate of the key-value (KV) cache, load balancing, and the adaptability of computing resources. Firstly, the matching rate of the KV cache is a crucial factor affecting inference service efficiency. The LLM inference process relies on an autoregressive algorithm to generate each token, which requires frequent access to the KV cache. Properly utilizing existing cached data can effectively reduce computational load and improve inference response speed. Therefore, improving the KV cache hit rate and fully utilizing existing cache resources is essential for enhancing inference service efficiency.

[0062] Secondly, load balancing is another key factor in improving the efficiency of inference services. LLM inference tasks typically consist of two main phases: the prefill phase and the decode phase, which differ in their computational patterns. The prefill phase is usually computationally intensive, while the decode phase relies more on memory bandwidth. Proper load balancing not only ensures efficient allocation of computing resources and memory bandwidth but also avoids excessive resource consumption in any one phase, thereby reducing overall computational latency. Most existing scheduling algorithms are based on coarse-grained load balancing methods, simply making decisions based on the overall load of cluster nodes. However, this approach often ignores the inherent characteristics of the tasks. For example, prefill requests and decode requests have different computational requirements, and long response generation requests can easily lead to queue accumulation, resulting in uneven cluster resource utilization and increased response latency.

[0063] Furthermore, while traditional heuristic scheduling algorithms can empirically find a balance between load balancing and cache matching in specific environments, their insufficient adaptability often prevents them from achieving optimal results when dealing with complex and diverse computing resources. For example, the algorithm in the Mooncake paper sets a scheduling threshold for the KV cache, but this value lacks universality, and the algorithm executes processes such as KV cache transfer and queue waiting sequentially, failing to effectively utilize parallel computing resources. Therefore, it has poor adaptability in diverse environments.

[0064] Therefore, how to balance the node load balancing of model instances and the KV cache hit rate to avoid inefficiency and increased latency is a technical problem that urgently needs to be solved.

[0065] To address the aforementioned technical problems, according to an embodiment of this application, an embodiment of an inference request scheduling method based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0066] This embodiment provides a reinforcement learning-based inference request scheduling method. Figure 1 A flowchart of a reinforcement learning-based inference request scheduling method provided in this application embodiment is shown below. Figure 1 As shown, this process, applied to an inference cluster routing and scheduling system, includes the following steps:

[0067] Step S1: Obtain the inference request submitted by the user.

[0068] Step S3: Based on the current state space used to characterize the system's operation, determine the target model instance that matches the scheduling inference request through a pre-built reinforcement learning model.

[0069] Step S5: Receive the result returned by the target model instance after processing the inference request through the pre-filling stage and the decoding stage, and return the result to the user.

[0070] Specifically, users submit inference requests through a standard API interface. These requests can be various types of inference tasks, such as natural language processing and image recognition. The request scheduler (Router / Scheduler) in the inference cluster routing and scheduling system receives the inference requests submitted by users. The request scheduler adds the received inference requests to a waiting queue. The queue status is represented by w. qtThis represents the number of inference requests waiting to be processed in the queue at time t. The request scheduler schedules inference requests based on a reinforcement learning model. The reinforcement learning model (such as Double DQN) selects the optimal action 'a' based on the current state space 's', i.e., which model instance is chosen to handle the current inference request. After receiving an inference request, the model instance first undergoes a pre-filling phase, loading data and initializing model parameters. After the pre-filling phase, the model instance enters the decoding phase, performing the actual inference computation and generating the output result. After processing, the model instance returns the result to the request scheduler, which then returns the result to the user through a standard API interface. Furthermore, each model instance is distributed across different nodes and does not affect others.

[0071] In this embodiment, the reinforcement learning-based inference request scheduling method deployment uses a P / D group (Prefill / Decode group) as an inference engine unit responsible for processing inference requests, i.e., model instances. Each P / D group includes: a Prefill phase (initialization part for processing inference requests) and a Decode phase (actual inference computation part for processing inference requests). Each P / D group acts as an independent inference engine, providing services externally. Each P / D group provides interface services through the OpenAI standard API. Users can submit inference requests to the P / D group through the standard API interface. The P / D group can receive and process inference requests and return results. These interface services enable the P / D group to interact with external systems (such as user applications, schedulers, etc.). The backend of the inference engine can provide services using dynamo+vllm (or dynamo+sglang). dynamo is a framework for optimizing and executing deep learning models. vllm is a library for efficiently processing large language models (LLM). sglang is a library for graphics and geometry processing. The request scheduler is responsible for collecting the status of multiple P / D groups, including node status, request status, computing load metrics (tokens processed per second, first response time, and total processing time), and KV cache distribution status. Based on the collected status information, the request scheduler uses a reinforcement learning model to determine which P / D group to assign newly arriving inference requests. In some cases, the request scheduler's scheduling may involve KV cache transfer: if the intermediate result required by an inference request (cached in the KV cache) is not in the currently assigned P / D group, the request scheduler can use the kv transfer command to transfer the required KV cache data from one P / D group to another. This ensures that requests can efficiently utilize cached data, reduce redundant computations, and improve overall performance.

[0072] In this reinforcement learning model, the expected value and the action selection are placed in the target network and the main network, respectively. The main network outputs the expected value of each action based on the current state space and selects the action with the largest expected value. The target network evaluates the target expected value after executing the selected action and updates the network parameters to complete the training of the reinforcement learning model.

[0073] Specifically, reinforcement learning models optimize scheduling strategies through the collaborative work of a main network and a target network. The main network outputs the expected value of each action based on the current state space and selects the action with the highest expected value as the optimal action. This process, called action selection, aims to find the action most likely to bring the maximum expected reward in the current state. The target network evaluates the target expected value after executing the optimal action. Its weights are periodically updated from the main network, but at a low frequency, to maintain stability and effectively avoid the problem of overestimating the Q-value of the main network, thus improving training stability. By calculating the loss function between the target expected value and the main network's predicted Q-value and updating the main network parameters, the model continuously optimizes the scheduling strategy to complete the training of the reinforcement learning model.

[0074] This embodiment provides a reinforcement learning-based inference request scheduling method. By intelligently scheduling inference requests through a reinforcement learning model, the system can dynamically select appropriate target model instances to process inference requests based on the current state space, thereby achieving precise resource allocation and load balancing. This intelligent scheduling strategy optimizes the allocation of computing resources according to task requirements and system load, avoiding unnecessary resource waste or computational bottlenecks. By rationally allocating computing resources, the system can avoid overload, reduce latency, and improve overall system efficiency. When processing inference requests, the system can efficiently utilize computing resources, ensuring low-latency responses and avoiding inefficiencies caused by resource contention or improper processing, thus improving the system's response speed and processing capacity.

[0075] Figure 2 A flowchart illustrating the reinforcement learning model training method provided in this application embodiment, which may include the following steps:

[0076] Step S31: Obtain the node state representing the queue length of each model instance at different stages, the request state representing the input sequence length of the inference request, and the KV cache distribution state to construct the state space.

[0077] Specifically, in the inference cluster routing and scheduling system, a model instance is an independently running copy of the model, responsible for handling inference requests and generating responses. A model instance contains a complete model structure, such as a Transformer architecture, and loads the model's weights and parameters, which determine the model's behavior. Each model instance can handle inference independently; requests are processed through a prefill and decode phase, and the results are returned to the user. They can run in parallel, improving the throughput and response speed of the inference cluster routing and scheduling system.

[0078] Node state s n This describes the queue length for each model instance during the pre-filling and decoding phases. Specifically: the queue length during the pre-filling phase is denoted by X. t ={x1,x2,…,x m} represents, where x i Y represents the number of tokens waiting to be processed in the pre-filling phase for the i-th model instance. The queue length in the decoding phase is represented by Y. t ={y1,y2,…,y m} indicates that, where y i is the number of tokens waiting to be processed during the decoding phase for the i-th model instance. m represents the total number of model instances in the system.

[0079] Request status s r The length of the input sequence for the inference request is described by x. n express.

[0080] KV cache distribution status s kv The distribution of the KV cache is described, specifically including: the maximum length of the adaptation prefix using bestLen. prefix This indicates the maximum length of the current inference request that matches the cached prefix. The optimal node is chosen: bestInstance. matched This indicates the node containing the model instance that best matches the current inference request. These two values ​​are calculated using the function BestPrefixMatch(P, block_keys): parameter P represents the address of the model instance pool. Parameter block... keys The key value representing the current inference request.

[0081] The node state s defined above n Request status s r and KV cache distribution state s kv Combined, they form the state space s of the reinforcement learning model, specifically represented as: s = {s n ∪s r ∪s kv}

[0082] Step S33: Determine the set of actions that can be taken at each time step as the action space; wherein, the set of actions includes scheduling the inference request to the corresponding model instance operation and not taking the scheduling operation.

[0083] Specifically, in reinforcement learning models, the Agent is the entity responsible for making decisions. The Agent's task is to decide how to allocate the user's inference request to different model instances, or to choose not to perform any action. The action space refers to the set of all possible actions the Agent can take at each time step. At each time step t, the Agent needs to decide how to process the user's inference request i. At any given time t, the Agent has the following choices:

[0084] First, schedule inference requests to the corresponding model instance operations:

[0085] The agent can choose to schedule inference request i to any one of the m model instances. Each model instance is represented by an index, such as 1, 2, ..., m.

[0086] For example, if m=3, the Agent can choose to schedule inference requests to model instance 1, model instance 2, or model instance 3.

[0087] Second, choose not to perform a scheduling operation:

[0088] The agent can also choose not to schedule inference requests to any model instance, but instead temporarily hold the inference requests for later processing. This action is represented by index 0.

[0089] The action space A can be represented as: A={0,1,2,…,m}, where 0 indicates that the Agent chooses not to perform a scheduling operation, and 1 to m indicate that the Agent schedules the inference request to the 1st to the mth model instance.

[0090] Step S35: Based on the state space and action space, train the reinforcement learning model, select the optimal action in the action space according to the state space, and determine the expected value of the target after executing the optimal action.

[0091] Specifically, a reinforcement learning model comprises two networks: a main network (or online network) and a target network. The main network, with weights θ, is responsible for selecting the optimal action 'a' from the action space given the state void 's', and continuously updates it to learn the expected value (Q), i.e., Q(s,a;θ). The target network has weights of... The weights of the target network are used to evaluate the expected value (Q), i.e., Q(s,a;θ′). The weights θ are periodically updated from the main network, but at a low frequency to maintain stability.

[0092] Specifically, in a given next state space s next Next, the main network is used to calculate the expected value (Q) of the prediction for each possible action a, and the action a with the largest expected value is selected. max As the optimal action: a max =max a {s next In step θ}, the optimal action is selected through the main network, but the Q-value of the main network is not used directly. Then, based on the optimal action a... max and the next state space s next Calculate Q on the target network next_max =Q(s next ,a max Therefore, the target expected value Q is obtained. target =r+γ×Q next_max Where r is the reward value, and γ is the discount factor, which is usually set to 0.95-0.99 and is used to control the importance of future rewards.

[0093] In a preferred embodiment, the reward function in the reinforcement learning model is constructed as follows: obtain the token processing per second metric, the first response time metric, and the total processing time metric for the reinforcement learning model's request scheduling; and construct the reward function based on the token processing per second metric, the first response time metric, and the total processing time metric.

[0094] Specifically, in reinforcement learning models, the reward function is the key mechanism guiding the agent to learn the desired behavior. The reward function directly affects: learning efficiency: the speed and effectiveness of the agent's learning; policy quality: whether the action chosen by the agent is optimal; and task achievement: whether the agent can effectively complete the task objective.

[0095] In this embodiment, the three key metrics for optimizing inference are:

[0096] Reduce the TTFT (Time to First Token) metric: the time from when a request arrives at the system to when the first output token is generated.

[0097] Reduce the TPOT (Total Processing Time) metric: the total time from when a request arrives at the system to when all output is generated.

[0098] Improve TPS (Tokens Per Second): the system's throughput, which is the number of tokens processed per unit of time.

[0099] To achieve the above optimization objectives, the reward function is designed in the following form:

[0100]

[0101] Where r is the reward value; v tps To reinforce the learning model, the metric for the number of tokens processed per second during request scheduling; v ttft To reinforce the learning model, the first response time metric for request scheduling is used; v tpot The total processing time metric for requesting scheduling is used to reinforce the learning model. The reward function is designed to be controlled within the range (0,1). This means that when the system performance is very good (high TPS, low TTFT, low TPOT), the reward value r is close to 1; when the system performance is very poor, the reward value r is close to 0.

[0102] Step S37: Determine the expected value of the prediction corresponding to the optimal action selected by the main network. Based on the loss function between the expected value of the prediction and the target expected value, update the network parameters by minimizing the loss function until the preset conditions are met, and obtain the trained reinforcement learning model.

[0103] Specifically, the loss function L(θ) is obtained by comparing the target expected value Q. target The expected value of the prediction Q(s,a;θ) obtained from the main network prediction is: L(θ)=E[(Q target -Q(s,a;θ)) 2 This loss function measures the difference between the predicted expected value Q(s,a;θ) and the target expected value Q. target The difference between them is used to update the weights θ of the main network by minimizing this loss function.

[0104] This embodiment constructs a state space by acquiring the node state, request state, and KV cache distribution state of each model instance. This state information helps the system understand the current computational load and cache utilization. Next, an action space is defined, containing operations such as scheduling requests to the most suitable model instance or not scheduling them. Through reinforcement learning training based on the state and action spaces, the system can select the optimal action according to the current state and predict the expected value after executing the action. Finally, by minimizing the loss function (i.e., the difference between the predicted expected value and the target expected value), the network parameters are updated, resulting in a well-trained reinforcement learning model. This model can optimize the KV cache matching rate, the reasonable allocation of load, and the adaptability of computing resources according to actual conditions, thereby improving system performance, reducing latency, and ensuring efficient inference processing.

[0105] Figure 3 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of a first performance optimization method, which may include the following steps:

[0106] Step S101: Construct an experience replay buffer; wherein, the experience replay buffer is used to store experience data, including the current state, the action taken, the reward value obtained, the next state, and whether it is completed.

[0107] Step S103: At each time step, the experience data is stored in the experience playback buffer.

[0108] Step S105: During the training process, a batch of empirical data is randomly sampled from the empirical replay buffer.

[0109] Step S107: Update the network parameters of the reinforcement learning model using the sampled empirical data.

[0110] In reinforcement learning model training, the agent's behavior is continuous, and the choice of the current state and action often depends on the previous state and action. This temporal dependence can lead to unstable gradient updates during training. By storing experience data in an experience replay buffer and randomly sampling it during training, this temporal dependence can be broken, making training more stable. Each piece of experience data (current state, action taken, reward value obtained, next state, and completion status, etc.) can be used multiple times during training, rather than being discarded after a single use. This method can make full use of limited data and improve training efficiency. Furthermore, random sampling can reduce fluctuations during training, making network updates smoother and thus improving training stability.

[0111] Specifically, an experience replay buffer is created to store the agent's experiences. Experiences include the current state, the action taken, the reward received, the next state, and whether the task is completed. At each time step t, the agent takes action a, receives reward r, and transitions to the next state s. next This information is then stored in an experience replay buffer. During training, a batch of experience data is randomly sampled from the experience replay buffer to update the network parameters of the reinforcement learning model, breaking the temporal correlation of the data and avoiding the impact of continuous data on training.

[0112] This embodiment introduces an experience replay buffer, allowing the reinforcement learning model to store data such as the current state, actions taken, and rewards obtained at each time step. This data is then randomly sampled during training to update the reinforcement learning model. Combined with reinforcement learning, this optimizes key-value cache matching, load balancing, and computing resource adaptation, effectively improving service efficiency and response speed. The experience replay buffer enhances data utilization and gradient update stability, cache matching optimization reduces latency, load balancing ensures reasonable resource allocation, and computing resource adaptation guarantees efficient use of computing resources. Overall, by comprehensively optimizing multiple aspects, not only is system throughput improved, but performance bottlenecks are also effectively avoided, achieving efficient and stable inference services.

[0113] Figure 4 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of a second performance optimization method, which may include the following steps:

[0114] Step S201: Randomly select an action with a first preset probability as the exploration phase, and select the optimal action in the current state with a second preset probability as the utilization phase, thus constructing a greedy strategy; wherein the sum of the first preset probability and the second preset probability is 1.

[0115] Step S203: At each time step, select an action based on the current state and the greedy strategy.

[0116] Step S205: By adjusting the value of the first preset probability, the model is shifted from the exploration phase to the utilization phase, thereby increasing the probability of selecting the optimal action and optimizing the training of the reinforcement learning model.

[0117] Specifically, in reinforcement learning model training, the agent needs to find a balance between the exploration and exploitation phases. The exploration phase involves the agent randomly selecting an action with a first pre-set probability ε to discover new states and rewards, thereby learning the characteristics of the environment. Too many exploration phases can lead to low training efficiency because frequent random selections may prevent the agent from effectively utilizing the learned knowledge. In this case, the system's performance may be unstable. The exploitation phase involves the agent selecting the action that the current policy considers optimal with a second pre-set probability 1-ε to maximize cumulative rewards. Staying in the exploitation phase too early or for too long may result in the agent lacking sufficient adaptation to changes in the environment, potentially getting trapped in local optima and failing to discover the global optimum. To find a suitable balance between the exploration and exploitation phases, the value of ε is typically gradually decreased over time, i.e., the probability of the exploration phase is reduced, and the probability of the exploitation phase is increased. This helps the agent transition from the exploration phase to the exploitation phase, allowing the agent to primarily rely on the learned policy for decision-making, avoiding getting trapped in local optima, and optimizing the training of the reinforcement learning model.

[0118] This embodiment uses an exploration phase to help the system randomly select different actions, thereby discovering potential optimization paths, such as improving cache hit rate and avoiding cache misses. The exploitation phase relies on known optimal strategies to ensure the system's stability in terms of computational resource allocation and load balancing. As the exploration probability gradually decreases, computational resources and load allocation can be dynamically adjusted based on real-time data, improving computational efficiency, reducing unnecessary resource waste, and thus avoiding inefficiency and increased latency. Through this dynamic strategy adjustment, reinforcement learning not only optimizes the training process but also achieves efficient resource utilization and performance improvement.

[0119] Figure 5 The training process of the reinforcement learning model provided in the embodiments of this application also includes a flowchart of a third performance optimization method, which may include the following steps:

[0120] Step S301: Initialize the weights of the target network so that the weights of the target network are consistent with the weights of the main network.

[0121] Step S303: After each time step or each training batch, update the weights of the target network according to a pre-set soft update method.

[0122] Specifically, the target network serves to provide a stable benchmark for evaluating the target expectation value Q of selecting the optimal action 'a' in the action space. target Weights of the target network The weights are periodically updated from the main network's weights θ, but at a low frequency. This design reduces fluctuations during training and improves training stability. Specifically, the initial weights of the target network... The weights θ, set as the weights of the main network, are used to progressively update the weights of the target network, rather than completely replicating the weights of the main network, thereby improving training stability. The soft update formula for the target network weights is defined. The weights are updated using the following formula: θ′←τθ+(1-τ)θ′, where τ is a small positive number (e.g., 0.001) representing the update ratio. Soft updates are performed after each time step or training batch, updating the target network weights according to the above formula. This update method is smoother than completely replicating the weights of the main network, reducing fluctuations during training.

[0123] This embodiment effectively improves stability and resource utilization efficiency by optimizing and updating the target network during reinforcement learning. The target network serves as a stable benchmark, avoiding fluctuations caused by frequent updates to the main network. This is similar to using a stable caching strategy to improve matching efficiency when optimizing the key-value cache matching rate. The optimization mechanism gradually brings the weights of the target network closer to those of the main network, facilitating dynamic adjustment of load distribution and optimization of computing resource allocation. This ensures load balancing and efficient resource utilization while reducing latency and inefficiency, thereby improving overall computational performance.

[0124] Figure 6 A flowchart of a pre-configured soft update method provided in an embodiment of this application, the process may include the following steps:

[0125] Step S3031: Determine the first change value based on the preset first update ratio and the weight of the main network.

[0126] Step S3033: Determine the second change value based on the preset second update ratio and the weights of the target network.

[0127] Step S3035: Add the first change value and the second change value to obtain the updated weights of the target network.

[0128] The sum of the first update ratio and the second update ratio is 1.

[0129] Specifically, the first change value represents the contribution of the main network weights to the target network weights: Δθ = τθ, where τ is a preset first update ratio, usually a small positive number (e.g., 0.001), and θ is the weight of the main network. The second change value represents the contribution of the target network weights themselves: Δθ′ = (1-τ)θ′, where 1-τ is a preset second update ratio. Let θ' be the weights of the target network. Add the first and second changes to obtain the updated weights of the target network: θ′←τθ+(1-τ)θ′. The weights of the target network are updated incrementally, rather than completely replicating the weights of the main network. This update method is smoother, reduces fluctuations during training, and improves training stability.

[0130] This embodiment adjusts the weight updates of the main network and the target network according to a preset update ratio, enabling a smooth transition and avoiding instability caused by excessive fluctuations. This update strategy controls the frequency of cache updates, avoiding over-reliance on any single cache and improving the overall matching rate and system stability. Simultaneously, this smooth update method also facilitates load balancing, preventing overload of a single computing node and ensuring reasonable resource allocation, thereby optimizing the adaptation of computing resources. Ultimately, the system reduces inefficiencies and increased latency, improving resource utilization and overall training performance during the computation process.

[0131] Accordingly, please refer to Figure 7 A block diagram of a reinforcement learning-based inference request scheduling device provided in this application embodiment, applied to an inference cluster routing scheduling system, the device includes:

[0132] The inference request acquisition unit 101 is used to acquire the inference request submitted by the user;

[0133] Model instance determination unit 103 is used to determine the target model instance that matches the scheduling inference request based on the current state space used to characterize the system operation, through a pre-built reinforcement learning model.

[0134] The processing result return unit 105 is used to receive the result returned by the target model instance after processing the inference request through the pre-filling stage and the decoding stage in sequence, and return the result to the user.

[0135] In this reinforcement learning model, the expected value and the action selection are placed in the target network and the main network, respectively. The main network outputs the expected value of each action based on the current state space and selects the action with the largest expected value. The target network evaluates the target expected value after executing the selected action and updates the network parameters to complete the training of the reinforcement learning model.

[0136] In some alternative implementations, the reinforcement learning model is trained in the following manner:

[0137] Obtain the node state representing the queue length of each model instance at different stages, the request state representing the input sequence length of the inference request, and the KV cache distribution state to construct the state space;

[0138] The action space is defined as the set of actions that can be taken at each time step; the action set includes scheduling the inference request to the corresponding model instance operation and not taking the scheduling operation.

[0139] Based on the state space and action space, a reinforcement learning model is trained, the optimal action in the action space is selected according to the state space, and the expected value of the target after executing the optimal action is determined.

[0140] The expected value of the prediction corresponding to the optimal action selected by the main network is determined. Based on the loss function between the expected value of the prediction and the expected value of the target, the network parameters are updated by minimizing the loss function until the preset conditions are met, and a trained reinforcement learning model is obtained.

[0141] In some alternative implementations, the reward function in the reinforcement learning model is constructed as follows:

[0142] Obtain metrics for the number of tokens processed per second, the first response time, and the total processing time for the reinforcement learning model request scheduling.

[0143] A reward function is constructed based on metrics such as the number of tokens processed per second, the first response time, and the total processing time.

[0144] In some alternative implementations, the training process of the reinforcement learning model also includes:

[0145] Construct an experience replay buffer; the experience replay buffer is used to store experience data, which includes the current state, the action taken, the reward value obtained, the next state, and whether it is completed;

[0146] At each time step, the experience data is stored in the experience playback buffer;

[0147] During training, a batch of empirical data is randomly sampled from the empirical replay buffer;

[0148] The network parameters of the reinforcement learning model are updated using the empirical data obtained from sampling.

[0149] In some alternative implementations, the training process of the reinforcement learning model also includes:

[0150] A greedy strategy is constructed by randomly selecting an action with a first preset probability as the exploration phase and selecting the optimal action in the current state with a second preset probability as the utilization phase; wherein the sum of the first preset probability and the second preset probability is 1.

[0151] At each time step, select an action based on the current state and the greedy strategy;

[0152] By adjusting the value of the first preset probability, shifting from the exploration phase to the utilization phase, the probability of selecting the optimal action is increased, thereby optimizing the training of the reinforcement learning model.

[0153] In some alternative implementations, the training process of the reinforcement learning model also includes:

[0154] Initialize the weights of the target network to make them consistent with the weights of the main network;

[0155] After each time step or each training batch, the weights of the target network are updated according to a pre-set soft update method.

[0156] In some optional implementations, the pre-configured soft update method is as follows:

[0157] The first change value is determined based on the preset first update ratio and the weight of the main network;

[0158] The second change value is determined based on the preset second update ratio and the weights of the target network;

[0159] The first and second change values ​​are added together to obtain the updated weights of the target network.

[0160] The sum of the first update ratio and the second update ratio is 1.

[0161] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0162] In this embodiment, the inference request scheduling device based on reinforcement learning is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0163] Please see Figure 8 , Figure 8 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 8As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0164] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0165] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0166] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0168] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0169] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0170] The apparatus and units described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0171] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0172] Those skilled in the art will understand that the embodiments of this application can be provided as methods or apparatus. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0178] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0179] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A reasoning request scheduling method based on reinforcement learning, characterized in that, The method is applied to an inference cluster routing scheduling system, and the method comprises the following steps: An inference request submitted by a user is acquired; A target model instance matched with the inference request is determined by a pre-constructed reinforcement learning model based on a current state space used to represent a current system running condition; wherein the reinforcement learning model is obtained by training in the following manner: node states used to represent queue lengths of each model instance at different stages, request states used to represent input sequence lengths of inference requests, and KV cache distribution states are acquired, and a state space is constructed; a set of actions that can be taken at each time step is determined as an action space; wherein the set of actions comprises an operation of scheduling an inference request to a corresponding model instance and an operation of not taking scheduling; a reinforcement learning model is trained based on the state space and the action space, an optimal action in the action space is selected according to the state space, and a target expected value after the optimal action is executed is determined; a prediction expected value corresponding to the optimal action is selected by a main network, and network parameters are updated by minimizing a loss function between the prediction expected value and the target expected value until a preset condition is met, so as to obtain a trained reinforcement learning model; A result returned by the target model instance after the inference request is processed in a pre-padding stage and a decoding stage in sequence is received, and the result is returned to the user; In the reinforcement learning model, an evaluation expected value and a selected action are placed in a target network and a main network respectively, the main network outputs expected values of each action according to the current state space and selects an action with a maximum expected value, the target network evaluates a target expected value after the selected action is executed, and network parameters are updated to complete training of the reinforcement learning model.

2. The method of claim 1, wherein, The reinforcement learning model is trained in the following manner: An indicator of a number of tokens processed per second, an indicator of a first response time, and an indicator of a total processing time are acquired; The reward function is constructed based on the number of tokens processed per second, the first response time, and the total processing time.

3. The method of claim 1, wherein, The training process of the reinforcement learning model further comprises the following steps: An experience replay buffer is constructed; wherein the experience replay buffer is used to store experience data, and the experience data comprises a current state, a selected action, an obtained reward value, a next state, and whether completion is achieved; At each time step, the experience data is stored in the experience replay buffer; During the training process, a batch of experience data is randomly sampled from the experience replay buffer; The network parameters of the reinforcement learning model are updated using the sampled experience data.

4. The method of claim 1, wherein, The training process of the reinforcement learning model further comprises the following steps: A greedy strategy is constructed by randomly selecting an action as an exploration stage with a first preset probability and selecting an optimal action under a current state as a utilization stage with a second preset probability; wherein the sum of the first preset probability and the second preset probability is 1; At each time step, an action is selected according to a current state and the greedy strategy. The first preset probability is adjusted to adjust from the exploration stage to the utilization stage to improve the probability of selecting the optimal action and optimize the training of the reinforcement learning model.

5. The method of claim 1, wherein, The training process of the reinforcement learning model further comprises: initializing the weights of the target network to make the weights of the target network consistent with the weights of the main network; after each time step or each training batch, updating the weights of the target network according to a preset soft update method.

6. The method of claim 5, wherein, The preset soft update method is: determining a first change value according to a preset first update ratio and the weights of the main network; determining a second change value according to a preset second update ratio and the weights of the target network; adding the first change value and the second change value to obtain the updated weights of the target network; wherein the sum of the first update ratio and the second update ratio is 1.

7. A reinforcement learning based inference request scheduling apparatus, characterized by, Applied to an inference cluster routing scheduling system, the device comprises: an inference request acquisition unit configured to acquire an inference request submitted by a user; a model instance determination unit configured to determine a target model instance matched with the inference request by scheduling based on a current state space currently used to represent a system running condition and a pre-constructed reinforcement learning model; wherein the reinforcement learning model is obtained by training in the following manner: acquiring node states used to represent queue lengths of each model instance at different stages, request states used to represent input sequence lengths of inference requests, and KV cache distribution states, constructing a state space; determining a set of actions that can be taken at each time step as an action space; wherein the set of actions comprises scheduling an inference request to a corresponding model instance operation and not taking a scheduling operation; training a reinforcement learning model based on the state space and the action space, selecting an optimal action in the action space according to the state space, and determining a target expected value after executing the optimal action; determining a predicted expected value corresponding to the optimal action selected by a main network, and updating network parameters by minimizing a loss function between the predicted expected value and the target expected value until a preset condition is met, to obtain a trained reinforcement learning model; a processing result return unit configured to receive results returned by the target model instance after processing the inference request in sequence through a pre-padding stage and a decoding stage, and return the results to the user; wherein the reinforcement learning model places an evaluation expected value and a selected action in a target network and a main network respectively, the main network outputs expected values of each action according to the current state space and selects an action with the maximum expected value, the target network evaluates a target expected value after executing the selected action, and updates network parameters to complete the training of the reinforcement learning model.

8. A computer device, comprising: comprise: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the inference request scheduling method based on reinforcement learning according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to execute the inference request scheduling method based on reinforcement learning in any one of claims 1 to 6.

Images