Reinforcement learning training method and device for large model inference request scheduling

Abstract

The application relates to the technical field of artificial intelligence, and discloses a reinforcement learning training method and device for large model inference request scheduling, wherein the method comprises the following steps: constructing a four-tuple data set and determining a benchmark value; performing normalization processing on offline data and online data in training data respectively by using the benchmark value, to obtain normalized offline data and normalized online data; taking the normalized offline data as input of a large model to be trained, performing initialization training by using offline reinforcement learning, and obtaining a trained initialization large model; mixing the normalized offline data and the normalized online data to form a mixed training set, taking the mixed training set as input of the initialization large model, performing fine tuning by using online reinforcement learning, and obtaining a trained target large model. According to the technical scheme provided by the application, reinforcement learning can be used to optimize a large model inference scheduling strategy in a power heterogeneity and load diversity environment, so that the performance is improved and the delay is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reinforcement learning training method and apparatus for scheduling inference requests for large models. Background Technology

[0002] With the rapid deployment of domestically produced GPUs and the ever-expanding scale of large models, inference services exhibit characteristics such as heterogeneous computing power, diverse workloads, latency sensitivity, and throughput priority, making traditional heuristic scheduling methods insufficient to meet performance requirements. Reinforcement learning (RL), as an adaptive optimization method, theoretically possesses potential, but faces multiple challenges in production environments: First, data scarcity and cold start issues, coupled state space under high concurrency leading to training fluctuations, and experience replay failing to fully reflect complex inference processes; second, sparse and delayed reward signals, insufficient end-to-end linear metric feedback, low policy exploration efficiency, and susceptibility to local optima; third, significant differences in computational constraints between the Prefill and Decode stages, making unified scheduling difficult to optimize simultaneously; and fourth, limited policy generalization capabilities, prone to catastrophic forgetting during hardware or model updates, inability to transfer old policies, requiring retraining on new hardware with uncertain performance.

[0003] Therefore, how to efficiently use reinforcement learning to optimize large model inference scheduling strategies in environments with heterogeneous computing power and diverse loads, in order to improve performance and reduce latency, is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a reinforcement learning training method and apparatus for scheduling large model inference requests, which achieves the technical effect of efficiently optimizing large model inference scheduling strategies using reinforcement learning in environments with heterogeneous computing power and diverse loads, thereby improving performance and reducing latency.

[0005] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a reinforcement learning training method for scheduling inference requests in large models, the method comprising: Based on the collected training data containing instance running data and user inference requests, a quadruple dataset is constructed, and a baseline value matching each feature dimension in the quadruple dataset is determined; wherein, each quadruple represents the transfer data of an inference request during the scheduling process. Using the benchmark value, the offline data and online data in the training data are normalized respectively to obtain normalized offline data and normalized online data; The normalized offline data is used as input to the large model to be trained, and offline reinforcement learning is used for initial training to obtain the trained initial large model. The normalized offline data and the normalized online data are mixed to form a hybrid training set, which is used as input to the initial large model. Online reinforcement learning is then used for fine-tuning to obtain the trained target large model.

[0006] This embodiment provides a reinforcement learning training method for scheduling inference requests in large models. It constructs a four-tuple dataset using collected training data and normalizes the data using a benchmark value. Then, offline reinforcement learning is used for model initialization to ensure the model can effectively learn inference scheduling strategies based on historical data. Next, the normalized offline data is mixed with normalized online data, and online reinforcement learning is used to fine-tune the model, ensuring it can adapt to changes in real-time load. In this process, offline data provides a good foundation for model startup, while online fine-tuning ensures the model can quickly adapt to dynamic environments during actual operation, thereby optimizing inference performance, reducing latency, and improving the overall efficiency of the model. This method not only efficiently utilizes computing resources but also enhances the model's generalization ability, ensuring good performance under varying hardware and load conditions.

[0007] In one implementation, the quadruple is constructed as follows: According to the time interval, the instance features of pre-filled instances and decoded instances are extracted from the collected instance running data, and concatenated with the request length of the user inference request to form the state space at the current moment and the state space at the next moment. The sequence number of the pre-filled instance and the sequence number of the decoded instance are encoded into a single action number to form an action space; A reward function is formed based on the long-term reward used to characterize the end-to-end performance after the inference request scheduling is completed and the immediate reward used to characterize the health performance of the inference request scheduling node, obtained from the training data collection. The current state space, the action space, the reward function, and the next state space are combined to construct a quadruple.

[0008] This embodiment constructs the state spaces for the current and next time steps by extracting instance features from pre-filled and decoded instances and concatenating them with the request length of the user's inference request. An action space is constructed by encoding the sequence numbers of pre-filled and decoded instances as single action numbers, ensuring the model can select the optimal solution from multiple feasible scheduling actions, avoiding resource waste and scheduling conflicts. Furthermore, a reward function is designed based on long-term and immediate rewards from training data acquisition. The long-term reward focuses on the end-to-end performance of the inference task, while the immediate reward measures the health status of nodes, thereby optimizing the scheduling strategy and ensuring the model's stability and efficiency. Finally, by combining the current state space, action space, reward function, and next time step state space into a quadruple, decisions can be progressively adjusted and optimized based on model feedback, ultimately achieving efficient scheduling of large-scale inference tasks, significantly improving inference performance and reducing latency.

[0009] In one implementation, the step of extracting instance features of pre-filled instances and decoded instances from the collected instance runtime data at time intervals, and concatenating them with the request length of the user inference request to form the state space at the current time and the state space at the next time, includes: Extract the current-time instance features of the pre-filled instance and the decoded instance from the collected instance running data; wherein, the instance features include the sequence number of each instance, the number of tokens to be processed, the video memory cache utilization rate, the inference stage node, and the number of prefix cache hit tokens; The current instance features are concatenated with the length of the user's inference request to form the current state space; Extract the next-time instance features of the pre-filled instance and the decoded instance from the collected instance runtime data; The next-time instance features are concatenated with the request length of the user's inference request to form the state space for the next time step.

[0010] This embodiment, through feature extraction and concatenation, can obtain the real-time running status of instances at the current and next time moments, and combine this information with the request length of the user's inference request to form an accurate state space. These state spaces provide the necessary input for reinforcement learning algorithms, helping the model dynamically optimize inference scheduling strategies based on current and future task requirements. In environments with heterogeneous computing power and diverse loads, it can accurately adjust the allocation of computing resources, ensuring that each instance receives the most suitable computing resources, thereby avoiding resource waste and latency, improving inference performance, and reducing latency in the overall inference process.

[0011] In one implementation, the long-term reward, which characterizes the end-to-end performance after inference request scheduling is completed, and the immediate reward, which characterizes the health performance of the inference request scheduling node, obtained from the training data, form a reward function, including: Based on the first token latency, per-token latency, and throughput extracted from the instance runtime data, the long-term reward is determined; The instant reward is determined based on the instance characteristics and the request length; The long-term reward function and the immediate reward function are weighted and summed to form the reward function.

[0012] This embodiment extracts key metrics such as first-token latency, per-token latency, and throughput from instance runtime data, and then defines a long-term reward. The long-term reward focuses on global efficiency, optimizing overall performance during inference and ensuring the model provides efficient service across different environments. Next, based on the instance's real-time load and request length, the model calculates an immediate reward to help adjust resource allocation and task scheduling, thereby improving short-term inference efficiency. Finally, a weighted sum of the long-term and immediate rewards forms a comprehensive reward function that balances short-term adjustments with long-term goals, optimizing resource usage, reducing inference latency, and improving throughput and system stability. This method ensures that even in heterogeneous computing environments, strategies can be dynamically adjusted based on real-time feedback, thereby improving adaptability, robustness, and overall inference performance.

[0013] In one implementation, determining the instant reward based on the instance characteristics and the request length includes: Extract the number of unprocessed tokens, video memory cache occupancy rate, and prefix cache hit tokens for the pre-filled instance from the instance features, and determine the immediate reward for the pre-filled node in combination with the request length; Extract the number of unprocessed tokens and the video memory cache usage rate for the decoding instance from the instance features to determine the instant reward for the decoding node; The instant reward is determined by averaging the instant reward of the pre-filled node and the instant reward of the decoded node.

[0014] This embodiment extracts key features from pre-filled and decoded nodes, such as the number of tokens to be processed, GPU memory cache utilization, and the number of tokens hit in the prefix cache, and combines these features with the request length to determine the immediate reward. These immediate rewards help the model dynamically adjust resource allocation and load balancing, thereby avoiding computational bottlenecks and wasted GPU memory resources. By comprehensively considering the immediate rewards of pre-filled and decoded nodes, reinforcement learning can intelligently optimize resource usage at each stage, ensuring efficient and stable inference processes and ultimately reducing overall latency. In this way, not only is the utilization rate of computing resources improved, but rapid response under load changes is also possible, ensuring the smooth execution of large model inference tasks.

[0015] In one implementation, determining the benchmark value that matches each feature dimension in the quadruple dataset includes: For each feature dimension in the quadruple dataset, the mean and standard deviation corresponding to each feature dimension are determined using offline data in the training data; The mean and standard deviation corresponding to each feature dimension are determined as the benchmark values ​​that match each feature dimension.

[0016] This embodiment standardizes the input data by calculating the mean and standard deviation of each feature dimension in the offline data and using them as benchmark values. This technique eliminates differences in feature scale, enabling reinforcement learning models to obtain consistent and stable data input under different computing power devices and load conditions, thereby accurately assessing the scheduling requirements of each task. In environments with heterogeneous computing power and diverse loads, these benchmark values ​​provide reliable data references for reinforcement learning, enabling it to dynamically select the optimal scheduling strategy, improve the inference performance of large models, effectively reduce latency, and enhance the stability and convergence efficiency of the algorithm.

[0017] In one implementation, the ratio of the normalized offline data to the normalized online data in the mixed training set is 2:1.

[0018] Secondly, embodiments of this application provide a reinforcement learning training apparatus for scheduling inference requests in large models, the apparatus comprising: The dataset construction unit is used to construct a quadruple dataset based on the collected training data containing instance running data and user inference requests, and to determine the baseline value that matches each feature dimension in the quadruple dataset; wherein, each quadruple represents the transfer data of an inference request during the scheduling process; The data normalization unit is used to normalize the offline data and online data in the training data using the benchmark value, respectively, to obtain normalized offline data and normalized online data. The offline reinforcement learning unit is used to take the normalized offline data as input to the large model to be trained, and use offline reinforcement learning to perform initial training to obtain a trained initial large model. The online reinforcement learning unit is used to mix the normalized offline data and the normalized online data to form a mixed training set, which is then used as input to the initial large model. Online reinforcement learning is used for fine-tuning to obtain the trained target large model.

[0019] Thirdly, embodiments of this application provide a computer device, including: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the reinforcement learning training method for large model inference request scheduling described above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the reinforcement learning training method for scheduling inference requests for large models as described above. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a reinforcement learning training method for scheduling inference requests in large models, provided as an embodiment of this application; Figure 2 A flowchart illustrating the construction method of quadruplets provided in this application embodiment; Figure 3 A flowchart of step S11 provided in an embodiment of this application; Figure 4 A flowchart of step S15 provided in an embodiment of this application; Figure 5 A flowchart of step S153 provided in an embodiment of this application; Figure 6 A flowchart of step S1 provided in the embodiments of this application; Figure 7 This application provides a reinforcement learning training device for scheduling inference requests for large models. Figure 8This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] With the rapid deployment of domestically produced GPUs and the ever-expanding scale of large models, the demands of inference services are becoming increasingly complex, exhibiting characteristics such as "heterogeneous computing power, diverse loads, latency sensitivity, and throughput priority." In this environment, traditional inference scheduling methods, especially heuristic rules, can no longer meet the growing performance requirements, necessitating the introduction of more advanced technologies for optimization. Reinforcement learning (RL, an adaptive optimization method that learns the optimal policy through trial and error by interacting with the environment) holds great promise in theory, but its practical implementation in production-grade inference systems still faces numerous challenges.

[0025] 1. Data quality and cold start issues The effectiveness of reinforcement learning largely depends on high-quality data. However, in real-world environments, especially large-scale inference tasks, interaction costs are high and effective samples are scarce. This has led to the persistent "cold start" problem. In this context, while experience replay mechanisms can accumulate historical data, many samples fail to effectively reflect the complexity of the inference process due to the high variance inherent in model inference. Particularly when handling high-concurrency requests, the state-space coupling of some requests can lead to ineffective state transitions, resulting in drastic fluctuations during training.

[0026] 2. The sparsity and delay issues of reward signals In reinforcement learning, the design of reward signals is crucial for policy learning. However, in existing inference scheduling systems, reward signals are often merely linearly weighted by end-to-end metrics such as TTFT (total time from request to generation of the first token), TPOT (total time from request to completion of all token generation), and TPS (number of requests processed per second). This approach lacks timely feedback on intermediate processes. Therefore, reinforcement learning suffers from low policy exploration efficiency, slow convergence, and is prone to getting trapped in local optima in this scenario.

[0027] 3. Scheduling differences in the Prefill / Decode stages Existing one-size-fits-all scheduling strategies assume that the Prefill and Decode phases can be executed sequentially. However, in practice, especially in a PD-separated architecture, this assumption often fails. The computational limitations of the Prefill phase and the constraints of the KV Cache (a cache storing intermediate computation results to accelerate the model decoding process) subject it to the dual constraints of computation and caching, while the Decode phase is more limited by computational capabilities. Without differentiation, traditional scheduling methods will prevent the simultaneous optimization of TTFT and TPOT, thus impacting overall throughput.

[0028] 4. The Challenge of Generalization Ability A significant problem with reinforcement learning methods is that policy networks often suffer from "catastrophic forgetting" when chip microarchitecture or model structure changes, meaning they cannot effectively adapt to new hardware or model configurations. This is particularly true during the rapid iteration of domestically produced GPUs, where older policies cannot be directly transferred to new hardware. This necessitates retraining the policy from scratch after chip updates, which not only lengthens the adaptation period but also may prevent the expected performance levels from being achieved on the new hardware.

[0029] In summary, how to efficiently use reinforcement learning to optimize large model inference scheduling strategies in environments with heterogeneous computing power and diverse loads, in order to improve performance and reduce latency, is a technical problem that urgently needs to be solved.

[0030] To address the aforementioned technical problems, according to an embodiment of this application, a reinforcement learning training method embodiment for scheduling inference requests for large models is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a reinforcement learning training method for scheduling inference requests in large models. Figure 1 A flowchart illustrating a reinforcement learning training method for scheduling inference requests in large models, as provided in this application embodiment, is shown below. Figure 1 As shown, the process includes the following steps: Step S1: Based on the collected training data containing instance running data and user inference requests, construct a quadruple dataset and determine the baseline value that matches each feature dimension in the quadruple dataset; wherein, each quadruple represents the transfer data of an inference request during the scheduling process.

[0032] Specifically, based on the collected training data, which includes instance runtime data and user inference requests, a four-tuple dataset can be constructed. Each four-tuple represents the transition data of an inference request during the scheduling process. The four-tuple contains the current state, action, reward, and the state at the next time step, comprehensively representing the dynamic changes of inference requests under diverse computing resources and load environments. By analyzing the feature dimensions of each four-tuple, matching benchmark values ​​can be determined. These benchmark values ​​serve as reference standards to help the model evaluate and optimize scheduling strategies.

[0033] Step S3: Using the baseline value, normalize the offline and online data in the training data respectively to obtain normalized offline data and normalized online data.

[0034] Specifically, for each sample in the training data, the original value V of a certain feature is transformed into a normalized value V0. norm The calculation formula is: Among them, V μ V is the mean of this feature; σ This represents the standard deviation of the feature.

[0035] In this way, the values ​​of all features will fall within a similar range, avoiding the disproportionate impact of some feature values ​​having large ranges (e.g., load or computational resources) on the model learning process. Through this process, a normalized offline dataset D can be generated. offline_norm .

[0036] When real-time online data D online When collected, to ensure consistency with offline data, the mean V calculated from the offline data must be used. μ Sum of standard deviation V σ The online data is normalized. Specifically, for each sample in the online data, it is processed according to the same normalization formula to obtain the normalized online data D. online_norm .

[0037] Over time, the collected online data will be continuously added to the normalized online data D. online_norm Each time new online data arrives, it is processed according to a normalization standard, resulting in new normalized offline and online data. By continuously expanding and updating the data, the model can be trained and optimized on a larger scale of historical and real-time data.

[0038] Step S5: Use the normalized offline data as input to the large model to be trained, and use offline reinforcement learning for initial training to obtain the trained initial large model.

[0039] Specifically, Offline Reinforcement Learning (Offline RL) is a reinforcement learning method in which model training relies entirely on pre-collected historical data, without requiring real-time environmental feedback. This differs from traditional Online Reinforcement Learning (Online RL), which requires the model to be trained through interaction with the environment (real-time feedback). In Offline Reinforcement Learning, normalized offline data is used as input to the large model to be trained. This dataset contains data samples from different scenarios, covering various scheduling transition information for inference requests. Using this data, the reinforcement learning model can learn decision patterns in multiple scenarios, thereby making reasonable scheduling decisions under different circumstances. The model guides scheduling decisions by estimating the Q-value (the expected cumulative reward for each state-action pair). The Q-value estimation is based on historical experience with offline data. At this stage, the model does not rely on real-time environmental feedback but is trained by calculating the reward of known data. Offline Reinforcement Learning optimizes strategies to make the trained model perform more stably in multiple scenarios, avoiding the dilemma of training from scratch for each new scenario.

[0040] Step S7: Mix the normalized offline data and the normalized online data to form a mixed training set, and use it as input to initialize the large model. Fine-tune the model using online reinforcement learning to obtain the trained target large model.

[0041] Specifically, although offline reinforcement learning initializes the model using historical data, this does not mean the learning process stops. In fact, the model after initial training usually requires further online learning to optimize its policy. Preferably, a mixed training set is formed by mixing normalized offline data and normalized online data in a 2:1 ratio and used as input for initializing the large model. Online reinforcement learning can fully utilize the diversity of offline data and the timeliness of online data to fine-tune the initial large model. This fine-tuning process helps the model gradually adapt to environmental changes in new scenarios, especially when facing new "computing power-model" combinations, allowing for better policy adjustments and improved generalization ability.

[0042] It should be noted that the model here refers to a large language model. In the context of a large language model, instance execution data typically refers to the data generated, processed, and transmitted during model execution. This data can include the text input to the model (e.g., questions or instructions), the output generated by the model (e.g., answers or suggestions), and any intermediate computational steps and stored content involved in the model's execution.

[0043] Therefore, combining offline reinforcement learning and online reinforcement learning to form a two-stage training framework can not only shorten the cold start time, but also accelerate the generalization ability of the model, enabling it to quickly adapt to different "computing power-model" combinations and ensure its stable and efficient performance in changing real-world environments.

[0044] It should be noted that this embodiment deploys multiple inference instances in the target domestic GPU cluster: the number of pre-filled instances is m, and the number of decoding instances is n. The backend engine can be an open-source engine such as sglang or vllm, providing inference service support. To cover diverse state spaces, a hybrid scheduler is selected for data collection: the hybrid scheduling strategy includes three sub-strategies: "random," "round-robin," and "shortest queue." A sampling ratio of 5:2:3 is used to ensure the collection of diverse data from different scenarios.

[0045] To avoid long-tail bias present in real business traffic and ensure uniform distribution of training data.

[0046] The generation rules are as follows: Input length: between [4,096,32,768] tokens, using uniform random sampling with a certain step size.

[0047] Output length: Also within the range of [4,096,32,768] tokens, uniformly randomized.

[0048] Request rate (RPS): Within the range of [1, 64] requests / second, a uniform random distribution is adopted, conforming to the Poisson arrival process.

[0049] Perturbation mechanism: Every 256 requests constitute a round. When generating the next round, there is an 80% probability that the input / output length of the previous round will be randomly perturbed by ±10%; there is a remaining 20% ​​probability that the original length will be maintained.

[0050] This mechanism is used to simulate a user's "retry + rewrite" behavior, increasing the diversity of data.

[0051] Total data volume: At least 500,000 requests will be generated to ensure the sufficiency and breadth of the data.

[0052] Ultimately, this embodiment reduces TTFT by 30% and cold start time by 20% compared to random and polling schemes, thereby improving the overall utilization of computing power and the SLO of the inference system.

[0053] This embodiment provides a reinforcement learning training method for scheduling inference requests in large models. It constructs a four-tuple dataset using collected training data and normalizes the data using a benchmark value. Then, offline reinforcement learning is used for model initialization to ensure the model can effectively learn inference scheduling strategies based on historical data. Next, the normalized offline data is mixed with normalized online data, and online reinforcement learning is used to fine-tune the model, ensuring it can adapt to changes in real-time load. In this process, offline data provides a good foundation for model startup, while online fine-tuning ensures the model can quickly adapt to dynamic environments during actual operation, thereby optimizing inference performance, reducing latency, and improving the overall efficiency of the model. This method not only efficiently utilizes computing resources but also enhances the model's generalization ability, ensuring good performance under varying hardware and load conditions.

[0054] Figure 2 A flowchart illustrating the construction method of a quadruple provided in this application embodiment, the process may include the following steps: Step S11: Extract instance features of pre-filled instances and decoded instances from the collected instance running data according to time intervals, and concatenate them with the request length of the user inference request to form the state space at the current time and the state space at the next time.

[0055] Specifically, by extracting instance features of pre-filled and decoded instances from the collected instance execution data at time intervals, detailed state information of each instance at the current moment can be obtained. These instance features contain the computational steps and resource consumption experienced by the instance during inference, effectively reflecting the execution status of the instance. These instance features are concatenated with the request length of the user's inference request to form the state space at the current moment, providing the model with a comprehensive perspective and helping it accurately assess the needs of the current environment and task. This not only helps the model optimize the scheduling strategy of inference tasks in dynamically changing environments but also enables efficient task resource allocation, reduces latency, and improves overall performance. By continuously acquiring and updating the state space, the model can respond to different loads and computational demands in real time, thereby ensuring the efficient execution and optimized scheduling of large model inference tasks.

[0056] Step S13: Encode the sequence number of the pre-filled instance and the sequence number of the decoded instance into a single action number to form an action space.

[0057] Specifically, by encoding the indices of pre-filled instances and decoded instances into a single action number (A), an action space with multiple discrete actions is formed. Specifically, this is encoded and mapped using the following formula: A = i*n + j, where A is the action number, ranging from 0 to (m×n-1); i ∈ [0, m-1] is the pre-filled instance indices; and j ∈ [0, n-1] is the decoded instance.

[0058] This formula maps the combination of prefilled and decoded instances to a single integer number (A). Assuming there are (m=4) prefilled instances and (n=4) decoded instances, a 16-dimensional action space is obtained. Each action number (A) represents a specific scheduling decision, indicating which prefilled instance and which decoded instance should be selected for processing at the current moment.

[0059] Step S15: Based on the long-term reward used to characterize the end-to-end performance after the inference request scheduling is completed and the immediate reward used to characterize the health performance of the inference request scheduling node, obtained from the training data collection, a reward function is formed.

[0060] Specifically, the reward function consists of two parts: long-term reward and immediate reward. The long-term reward, based on the training data, aims to characterize the backend-to-end performance after the inference request scheduling is completed. This part of the reward reflects the efficiency and quality of the entire inference process, typically involving factors such as the overall task completion time, inference accuracy, and resource consumption, and embodies the model's long-term optimization goals. Specifically, the long-term reward can be obtained through a comprehensive evaluation of multiple scheduling instances, helping the model understand the long-term impact of different scheduling strategies on overall performance, thereby gradually adjusting its scheduling strategy.

[0061] Immediate rewards characterize the health and performance of inference request scheduling nodes, typically focusing on real-time feedback and considering factors such as node resource utilization, load balancing, and latency. This reward reflects in real-time whether a node experiences performance bottlenecks or failures during inference, guiding timely scheduling adjustments to ensure the health of each node and the stable operation of the system. Immediate rewards are usually obtained after each inference request scheduling execution and are a direct driver of short-term decisions, ensuring the model can respond quickly and perform necessary optimizations in dynamically changing environments.

[0062] By combining long-term and immediate rewards to form a comprehensive reward function, the model can comprehensively consider overall performance and the health of local nodes during reinforcement learning, optimizing the scheduling strategy. Long-term rewards ensure overall performance improvement after the inference task is completed, while immediate rewards guarantee that the model can promptly identify and address potential issues such as uneven load distribution at each scheduling decision. This reward function design effectively guides the reinforcement learning model to converge quickly to the optimal solution, avoiding slow convergence in the early stages of training due to sparse rewards or local optima.

[0063] Step S17: Combine the current state space, action space, and reward function with the state space of the next time step to construct a quadruple.

[0064] Specifically, the current state space provides the model with a comprehensive description of the environment, encompassing information such as resource usage, load status, and task progress, helping the model accurately understand the current operating environment. The action space defines all possible actions the model can take in a specific state, determining how the model interacts with the environment and triggers corresponding changes. The reward function quantifies the effect of each action, evaluating whether the model's decisions are conducive to optimizing the objective, thereby guiding the model towards the optimal policy. After executing an action, the environment changes, forming the state space for the next moment, serving as feedback on the model's behavior. The construction of the four-tuple provides the foundation for the model to learn and adjust its policy, enabling reinforcement learning to progressively optimize the decision-making process in complex and dynamic environments.

[0065] This embodiment constructs the state spaces for the current and next time steps by extracting instance features from pre-filled and decoded instances and concatenating them with the request length of the user's inference request. An action space is constructed by encoding the sequence numbers of pre-filled and decoded instances as single action numbers, ensuring the model can select the optimal solution from multiple feasible scheduling actions, avoiding resource waste and scheduling conflicts. Furthermore, a reward function is designed based on long-term and immediate rewards from training data acquisition. The long-term reward focuses on the end-to-end performance of the inference task, while the immediate reward measures the health status of nodes, thereby optimizing the scheduling strategy and ensuring the model's stability and efficiency. Finally, by combining the current state space, action space, reward function, and next time step state space into a quadruple, decisions can be progressively adjusted and optimized based on model feedback, ultimately achieving efficient scheduling of large-scale inference tasks, significantly improving inference performance and reducing latency.

[0066] Figure 3 The flowchart for step S11 provided in the embodiments of this application may include the following steps: Step S111: Extract the current-time instance features of the pre-filled instance and the decoded instance from the collected instance running data; wherein, the instance features include the sequence number of each instance, the number of tokens to be processed, the video memory cache utilization rate, the inference stage node, and the number of tokens hit by the prefix cache.

[0067] Specifically, instance features are extracted from the collected instance runtime data to distinguish between prefill and decode instances. Instance features include: the instance ID (used to differentiate between instances); the number of tokens to be processed (waiting_queue_length), reflecting the current instance's computational load; a higher number of tokens means the instance requires more computational resources (such as GPU memory and computing power) for inference; GPU cache usage (gpu_kv_cache_usage), indicating the current instance's GPU cache usage, helping to infer resource allocation and load; high GPU cache usage means more GPU memory is needed to store and compute data, which may affect the performance of other instances. Excessive GPU cache usage may require cache optimization or adjustments to computational resources to balance the load; and the inference stage (infer_stage), indicating the current instance's inference stage, which could be either the "prefill" or "decode" stage. This helps to distinguish different stages of the inference task, thus enabling different optimizations to be made in subsequent inference processes; the prefix cache hit count (prefix_kv_cache_match_length) reflects the number of tokens that the instance hits in the prefix cache. If the cache hit rate is high, it indicates that the caching effect is good during the inference process, which can reduce computation and storage overhead and improve the overall inference efficiency.

[0068] Step S113: Concatenate the instance features at the current moment with the request length of the user's inference request to form the state space at the current moment.

[0069] Specifically, the state space at the current moment is formed by concatenating the instance features at the current moment and the request length of the user's inference request: Where S is the state space; i = m + n; m is the number of pre-filled instances; n is the number of decoded instances; and input_token_length is the length of the user's inference request.

[0070] Step S115: Extract the next-time instance features of the pre-filled instance and the decoded instance from the collected instance running data.

[0071] Specifically, the method for obtaining instance features and state features at the next moment (e.g., 1 second later) is the same as in step S111, but it targets the state information at a future moment. Since there may be state updates or changes during the inference process, the instance features at the next moment may differ from those at the current moment. Therefore, extracting instance features at the next moment can help the model better predict future inference states, thereby optimizing resources.

[0072] Step S117: Concatenate the instance features of the next time step with the request length of the user's inference request to form the state space of the next time step.

[0073] Specifically, the formation of the state space S' at the next moment is the same as in step S113, and will not be repeated here.

[0074] This embodiment, through feature extraction and concatenation, can obtain the real-time running status of instances at the current and next time moments, and combine this information with the request length of the user's inference request to form an accurate state space. These state spaces provide the necessary input for reinforcement learning algorithms, helping the model dynamically optimize inference scheduling strategies based on current and future task requirements. In environments with heterogeneous computing power and diverse loads, it can accurately adjust the allocation of computing resources, ensuring that each instance receives the most suitable computing resources, thereby avoiding resource waste and latency, improving inference performance, and reducing latency in the overall inference process.

[0075] Figure 4 The flowchart for step S15 provided in the embodiments of this application may include the following steps: Step S151: Determine the long-term reward based on the first token delay, per-token delay, and throughput extracted from the instance runtime data.

[0076] Specifically, First Token Latency (TTFT) represents the latency at which the model generates the first token when the inference task begins. A lower TTFT is better, indicating a faster model response. Per Token Latency (TPOT) represents the latency for generating each token during inference. Lower latency indicates higher inference efficiency. Throughput (TPS) refers to the number of tokens processed per unit of time; higher throughput indicates stronger processing capabilities.

[0077] Long-term reward R l The settlement is only performed once after all inference requests have been completed; the lower the better. Where TTFT, TPOT, and TPS represent the measured values ​​after the inference request is completed; α1, α2, and α3 are 0.3, 0.5, and 0.2, respectively.

[0078] Therefore, the smaller the TTFT / TPOT, the lower the R l The larger the value; the larger the TPS, the higher the R l Also bigger.

[0079] Step S153: Determine the immediate reward based on instance characteristics and request length.

[0080] Specifically, key features are first extracted from the input instances, such as the number of tokens to be processed, GPU memory cache utilization, and the number of tokens hit by the prefix cache. These features directly reflect the computational load and resource utilization of the current node. Combined with the request length (i.e., the size of the input data) and inference requirements, the complexity of the task and resource pressure are further assessed. Through comprehensive analysis of these factors, an immediate reward is assigned to each node (e.g., pre-filling nodes and decoding nodes) as the basis for optimization decisions. The immediate reward not only considers the efficiency of computational resource utilization but also dynamically adjusts based on task execution, ensuring that the model can make real-time scheduling decisions based on changes in load and resources. This feature- and request-length-based immediate reward mechanism helps the model balance load, optimize resource allocation, thereby improving inference efficiency, reducing latency, and maintaining high performance under different environmental conditions.

[0081] Step S155: Perform a weighted summation of the long-term reward function and the immediate reward function to form the reward function.

[0082] Specifically, long-term rewards focus on the model's ultimate optimization goal, representing the long-term efficiency of the inference process, while immediate rewards provide immediate feedback based on the current state, helping the model adjust its strategy to avoid undesirable optimization directions. This hierarchical reward design of "long-term-immediate" allows for a better balance between short-term adjustments and long-term goals in inference tasks, thereby improving overall performance.

[0083] The overall reward function combines long-term and immediate rewards through a weighted summation: Among them, R l Represents long-term rewards, serving as the long-term goal for ultimate optimization; R s α represents the immediate reward, which helps prevent the model from going astray through immediate feedback; α is the weight of the immediate reward and β is the weight of the long-term reward, which are 0.7 and 0.3 respectively.

[0084] This embodiment extracts key metrics such as first-token latency, per-token latency, and throughput from instance runtime data, and then defines a long-term reward. The long-term reward focuses on global efficiency, optimizing overall performance during inference and ensuring the model provides efficient service across different environments. Next, based on the instance's real-time load and request length, the model calculates an immediate reward to help adjust resource allocation and task scheduling, thereby improving short-term inference efficiency. Finally, a weighted sum of the long-term and immediate rewards forms a comprehensive reward function that balances short-term adjustments with long-term goals, optimizing resource usage, reducing inference latency, and improving throughput and system stability. This method ensures that even in heterogeneous computing environments, strategies can be dynamically adjusted based on real-time feedback, thereby improving adaptability, robustness, and overall inference performance.

[0085] Figure 5 The flowchart for step S153 provided in the embodiments of this application may include the following steps: Step S1531: Extract the number of unprocessed tokens, video memory cache occupancy rate, and prefix cache hit tokens for the pre-filled instance from the instance features, and determine the immediate reward for the pre-filled node in combination with the request length.

[0086] Specifically, in the pre-filling nodes (i.e., the input token preprocessing stage), it is necessary to maintain load balancing while utilizing the prefix cache as efficiently as possible. The immediate reward for the pre-filling nodes is determined by R. sp The following factors determine this: Wherein, waiting_queue_length is the number of pending tokens in the instance; waiting_queue_length_max is the maximum queue capacity (constant) (user-defined, such as 32768); gpu_kv_cache_usage is the GPU memory cache usage rate in the instance; prefix_kv_cache_match_length is the number of prefix cache hit tokens in the instance; and input_token_length is the request length.

[0087] Step S1533: Extract the number of unprocessed tokens and the video memory cache usage rate for the decoding instance from the instance features, and determine the instant reward for the decoding node.

[0088] Specifically, at the decoding node (i.e., the token generation phase), only load balancing is considered. The immediate reward R for the decoding node here... sd It is determined by the following factors: Step S1535: Take the average of the instant reward of the pre-filled node and the instant reward of the decoded node to determine the instant reward.

[0089] Specifically, the final instant reward is the average of the instant rewards of the pre-filled nodes and the instant rewards of the decoded nodes, forming a comprehensive instant reward R. s : By combining two-stage instant rewards, resource allocation can be adaptively adjusted during inference, optimizing task processing efficiency and response time.

[0090] This embodiment extracts key features from pre-filled and decoded nodes, such as the number of tokens to be processed, GPU memory cache utilization, and the number of tokens hit in the prefix cache, and combines these features with the request length to determine the immediate reward. These immediate rewards help the model dynamically adjust resource allocation and load balancing, thereby avoiding computational bottlenecks and wasted GPU memory resources. By comprehensively considering the immediate rewards of pre-filled and decoded nodes, reinforcement learning can intelligently optimize resource usage at each stage, ensuring efficient and stable inference processes and ultimately reducing overall latency. In this way, not only is the utilization rate of computing resources improved, but rapid response under load changes is also possible, ensuring the smooth execution of large model inference tasks.

[0091] Figure 6 The flowchart for step S1 provided in the embodiments of this application may include the following steps: Step S191: For each feature dimension in the quadruple dataset, determine the mean and standard deviation corresponding to each feature dimension using offline data from the training data.

[0092] Specifically, for each feature dimension in the quadruple dataset, the mean V of that feature dimension across all samples is calculated. μ Sum of standard deviation V σ Specifically: Mean V μ : This represents the average value of this feature dimension across all samples.

[0093] Standard deviation V σ This indicates the distribution of the values ​​for this feature dimension in the offline data sample, i.e., the degree of dispersion of the values. The larger the standard deviation, the greater the fluctuation of the data; the smaller the standard deviation, the more concentrated the data.

[0094] For example, in a quadruple dataset with 5 samples, the values ​​of the feature `gpu_kv_cache_usage` are [0.3, 0.5, 0.4, 0.2, 0.6]. Then, the mean V of the feature `gpu_kv_cache_usage` is... μIt is 0.4, which is calculated using the standard deviation formula to obtain V. σ =0.1414.

[0095] Step S193: Determine the mean and standard deviation corresponding to each feature dimension as the benchmark value that matches each feature dimension.

[0096] Specifically, once the mean and standard deviation of each feature dimension are calculated, they become the benchmark values ​​for that feature dimension.

[0097] This embodiment standardizes the input data by calculating the mean and standard deviation of each feature dimension in the offline data and using them as benchmark values. This technique eliminates differences in feature scale, enabling reinforcement learning models to obtain consistent and stable data input under different computing power devices and load conditions, thereby accurately assessing the scheduling requirements of each task. In environments with heterogeneous computing power and diverse loads, these benchmark values ​​provide reliable data references for reinforcement learning, enabling it to dynamically select the optimal scheduling strategy, improve the inference performance of large models, effectively reduce latency, and enhance the stability and convergence efficiency of the algorithm.

[0098] Accordingly, please refer to Figure 7 A block diagram of a reinforcement learning training apparatus for scheduling inference requests for large models, provided in an embodiment of this application, is shown. The apparatus includes: Dataset construction unit 101 is used to construct a quadruple dataset based on the collected training data containing instance running data and user inference requests, and to determine the baseline value that matches each feature dimension in the quadruple dataset; wherein, each quadruple represents the transfer data of an inference request during the scheduling process. The data normalization unit 103 is used to normalize the offline data and online data in the training data using the benchmark value to obtain normalized offline data and normalized online data. The offline reinforcement learning unit 105 is used to take normalized offline data as input to the large model to be trained, and use offline reinforcement learning for initial training to obtain a trained initial large model. The online reinforcement learning unit 107 is used to mix normalized offline data and normalized online data to form a mixed training set, which is used as input to initialize the large model. The model is then fine-tuned using online reinforcement learning to obtain the trained target large model.

[0099] In some optional implementations, the dataset construction unit 101 is as follows: According to the time interval, the instance features of pre-filled instances and decoded instances are extracted from the collected instance running data, and concatenated with the request length of the user inference request to form the state space at the current time and the state space at the next time. The sequence number of the pre-filled instance and the sequence number of the decoded instance are encoded into a single action number to form an action space; A reward function is formed based on the long-term reward used to characterize the end-to-end performance after inference request scheduling is completed, obtained from the training data, and the immediate reward used to characterize the health performance of the inference request scheduling node. By combining the current state space, action space, and reward function with the state space of the next time step, a quadruple is constructed.

[0100] In some optional implementations, instance features of pre-filled instances and decoded instances are extracted from the collected instance runtime data at time intervals, and concatenated with the request length of the user inference request to form the state space at the current time and the state space at the next time, including: Extract the current-time instance features of the pre-filled instance and the decoded instance from the collected instance running data; the instance features include the sequence number of each instance, the number of tokens to be processed, the video memory cache utilization rate, the inference stage node, and the number of tokens hit by the prefix cache; The current instance features are concatenated with the length of the user's inference request to form the current state space; Extract the next-time instance features of the pre-filled instance and the decoded instance from the collected instance runtime data; The next-time instance features are concatenated with the length of the user's inference request to form the state space for the next time step.

[0101] In some optional implementations, a reward function is formed based on the long-term reward used to characterize the end-to-end performance after inference request scheduling is completed, obtained from training data collection, and the immediate reward used to characterize the health performance of the inference request scheduling node, including: Long-term rewards are determined based on the first token latency, per-token latency, and throughput extracted from instance runtime data. The immediate reward is determined based on instance characteristics and request length; The long-term reward function and the immediate reward function are weighted and summed to form the reward function.

[0102] In some alternative implementations, the immediate reward is determined based on instance characteristics and request length, including: Extract the number of unprocessed tokens, video memory cache utilization, and prefix cache hit tokens for pre-filled instances from instance features, and determine the immediate reward for pre-filled nodes based on the request length; Extract the number of unprocessed tokens and the video memory cache usage rate from the instance features for the decoding instance to determine the immediate reward for the decoding node; The instant reward is determined by averaging the instant rewards of the pre-filled nodes and the decoded nodes.

[0103] In some optional implementations, the dataset construction unit 101 includes: For each feature dimension in the quadruplet dataset, the mean and standard deviation of each feature dimension are determined using offline data from the training data. The mean and standard deviation corresponding to each feature dimension are determined as the benchmark values ​​that match each feature dimension.

[0104] In some alternative implementations, the ratio of normalized offline data to normalized online data in the mixed training set is 2:1.

[0105] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0106] In this embodiment, a reinforcement learning training device for scheduling inference requests for large models is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0107] Please see Figure 8 , Figure 8 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0108] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0109] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0110] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0112] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0113] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0114] The apparatus and units described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0115] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0116] Those skilled in the art will understand that the embodiments of this application can be provided as methods or apparatus. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0123] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A reinforcement learning training method for scheduling inference requests in large models, characterized in that, The method includes: Based on the collected training data containing instance runtime data and user inference requests, a quadruple dataset is constructed, and a baseline value matching each feature dimension in the quadruple dataset is determined. Each quadruple represents the transition data of an inference request during the scheduling process. The construction of the quadruples includes: extracting instance features of pre-filled instances and decoded instances from the collected instance runtime data at time intervals, and concatenating them with the request length of the user inference request to form the current state space and the next state space; and forming a reward function combination based on the long-term reward used to characterize the end-to-end performance after inference request scheduling and the immediate reward used to characterize the health performance of the inference request scheduling node, obtained from the training data. The method for obtaining the long-term reward includes: determining the long-term reward based on the first token delay, per-token delay, and throughput extracted from the instance running data; The method for obtaining the instant reward includes: extracting the number of unprocessed tokens, video memory cache utilization rate, and prefix cache hit tokens for the pre-filled instance from the instance features, and determining the instant reward for the pre-filled node in combination with the request length; extracting the number of unprocessed tokens and video memory cache utilization rate for the decoding instance from the instance features, and determining the instant reward for the decoding node; and averaging the instant reward for the pre-filled node and the instant reward for the decoding node to determine the instant reward. Using the benchmark value, the offline data and online data in the training data are normalized respectively to obtain normalized offline data and normalized online data; The normalized offline data is used as input to the large model to be trained, and offline reinforcement learning is used for initial training to obtain a trained initial large model. The normalized offline data and the normalized online data are mixed to form a hybrid training set, which is used as input to the initial large model. Online reinforcement learning is then used for fine-tuning to obtain the trained target large model.

2. The method according to claim 1, characterized in that, The construction methods for the quadruple also include: The sequence number of the pre-filled instance and the sequence number of the decoded instance are encoded into a single action number to form an action space; The current state space, the action space, the reward function, and the next state space are combined to construct a quadruple.

3. The method according to claim 1, characterized in that, The step of extracting instance features of pre-filled instances and decoded instances from the collected instance runtime data according to time intervals, and concatenating them with the request length of the user inference request to form the state space at the current time and the state space at the next time, includes: Extract the current-time instance features of the pre-filled instance and the decoded instance from the collected instance running data; wherein, the instance features include the sequence number of each instance, the number of tokens to be processed, the video memory cache utilization rate, the inference stage node, and the number of prefix cache hit tokens; The current instance features are concatenated with the length of the user's inference request to form the current state space; Extract the next-time instance features of the pre-filled instance and the decoded instance from the collected instance runtime data; The next-time instance features are concatenated with the request length of the user's inference request to form the state space for the next time step.

4. The method according to claim 1, characterized in that, The long-term reward, used to characterize the end-to-end performance after inference request scheduling is completed, and the immediate reward, used to characterize the health performance of the inference request scheduling node, obtained from the training data, form a reward function combination, including: The long-term reward and the immediate reward are weighted and summed to form a reward function combination.

5. The method according to claim 1, characterized in that, The determination of the benchmark value that matches each feature dimension in the quadruple dataset includes: For each feature dimension in the quadruple dataset, the mean and standard deviation corresponding to each feature dimension are determined using offline data in the training data; The mean and standard deviation corresponding to each feature dimension are determined as the benchmark values ​​that match each feature dimension.

6. The method according to claim 1, characterized in that, The ratio of normalized offline data to normalized online data in the mixed training set is 2:

1.

7. An apparatus for implementing the reinforcement learning training method for large model inference request scheduling as described in any one of claims 1-6, characterized in that, The device includes: The dataset construction unit is used to construct a quadruple dataset based on the collected training data containing instance running data and user inference requests, and to determine a baseline value that matches each feature dimension in the quadruple dataset; wherein, each quadruple represents the transfer data of an inference request during the scheduling process; The data normalization unit is used to normalize the offline data and online data in the training data using the benchmark value, respectively, to obtain normalized offline data and normalized online data. The offline reinforcement learning unit is used to take the normalized offline data as input to the large model to be trained, and use offline reinforcement learning to perform initial training to obtain a trained initial large model. The online reinforcement learning unit is used to mix the normalized offline data and the normalized online data to form a mixed training set, which is then used as input to the initial large model. Online reinforcement learning is used for fine-tuning to obtain the trained target large model.

8. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the reinforcement learning training method for large model inference request scheduling as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the reinforcement learning training method for large model inference request scheduling as described in any one of claims 1 to 6.

Images