A large language model inference request scheduling method and system for a multi-level heterogeneous environment

Abstract

The application provides a large language model inference request scheduling method and system for a multi-level heterogeneous environment. In view of the complex scene of heterogeneous quality of service, model heterogeneity and hardware resource heterogeneity of inference request, the method first constructs a performance affinity index table of the quantitative "request-(model-hardware)" matching relationship through offline benchmarking. Secondly, based on the hardware performance of each inference instance and the model copy parameter scale deployed by it, a weight coefficient is determined, a virtual token counter is constructed using the weight coefficient, the user's counter value is updated by combining the token usage weight, and a fairness priority score is calculated. The system dynamically monitors the resource state of each inference engine instance. When there is idle resource, the fairness priority and affinity of the candidate request are comprehensively scored, the request with the highest score is selected for scheduling, the optimal matching of the request and the resource is realized, and the trade-off between the overall inference efficiency of the system and the service quantity fairness among users is achieved.

Description

Technical Field

[0001] This application relates to the fields of inference performance optimization and heterogeneous resource scheduling, and in particular to a method and system for scheduling inference requests for large language models in multi-level heterogeneous environments. Background Technology

[0002] With the rapid development of natural language processing and related technologies, large language models (LLMs), represented by ChatGPT and Llama, have demonstrated outstanding capabilities in fields such as natural language processing, code generation, and intelligent question answering. Generative AI applications are rapidly penetrating various industries. User demand for these computationally intensive services is growing exponentially, with increasingly stringent and diverse requirements for service response latency and quality. Due to the limitations of terminal devices in terms of memory capacity and computing power, deploying large model inference services in the cloud or edge server clusters has become the mainstream solution.

[0003] On the actual inference service supply side, limited by the high cost and supply shortage of high-end computing chips (such as NVIDIA A100 / H100), service providers often use heterogeneous clusters composed of computing chips of different generations and performance levels (such as A100, RTX 3090, Ascend 910B, etc.) to support services. Meanwhile, to accommodate different levels of user needs, the system typically deploys model replicas with different parameter sizes (such as 7B, 70B) and different quantization precisions (such as FP16, INT4). This multi-level heterogeneous environment, encompassing "application request layer QoS heterogeneity," "model layer deployment heterogeneity," and "computing resource layer hardware heterogeneity," significantly reduces deployment costs and improves resource flexibility, but also poses a significant challenge to the scheduling of inference tasks.

[0004] However, current mainstream large-model inference scheduling methods have two significant drawbacks when dealing with such complex heterogeneous scenarios: First, existing scheduling strategies struggle to effectively perceive and leverage the performance affinity differences between models and heterogeneous hardware. The LLM inference process includes a computationally intensive pre-filling phase and a memory-intensive decoding phase, and the performance differences between different hardware tiers in handling these two types of workloads are not linear. Traditional scheduling algorithms (such as round-robin, random, or scheduling based solely on available GPU memory) often blindly distribute long input tasks to inference nodes with weaker computing power, causing these nodes to enter a prolonged "dead" state due to computational bottlenecks, leading to severe head-of-line blocking and ultimately dragging down the overall system throughput.

[0005] Secondly, existing fairness scheduling algorithms fail to establish a precise measurement mechanism that adapts to the value of heterogeneous resources. Traditional fairness scheduling typically assumes that the underlying resources are homogeneous. However, in a heterogeneous environment, the cost and value of using a high-end graphics card for one second is far higher than that of using a low-end graphics card. If only the number of tokens is used as a single billing dimension, users using low-end resources to process simple tasks will pay the same "virtual cost" as users using high-end resources to process complex tasks. This clearly violates the principle of substantive fairness in a multi-tenant environment and may even lead to high-end resources being occupied by low-value requests for a long period of time.

[0006] Therefore, there is an urgent need to design an LLM inference scheduling method for multi-level heterogeneous environments. This method not only needs to have a deep understanding of the characteristics of the "request-model-hardware" triple and be able to achieve optimal task matching based on affinity to improve system efficiency, but also should establish a fairness mechanism that can uniformly measure the consumption of heterogeneous resources, maximizing the overall inference performance of the system while ensuring fairness in multi-user services. Summary of the Invention

[0007] To address the limitations of existing large-model inference scheduling technologies in multi-level heterogeneous environments—namely, the difficulty in simultaneously considering performance affinity differences among different hardware and substantive fairness among multiple users—this application provides a method and system for fair scheduling of large language model inference requests in multi-level heterogeneous environments. This method constructs an offline affinity profile to accurately perceive the compatibility between requests and heterogeneous hardware, and introduces a Virtual Token Counter (VTC) mechanism based on differentiated weights. While ensuring user fairness, it monitors the operating load, concurrency status, and remaining GPU memory of each heterogeneous computing node, and executes task orchestration decisions based on hybrid scoring to achieve optimal allocation and scheduling of multi-user requests, thereby maximizing the overall inference efficiency of the system.

[0008] This application provides a method for scheduling large language model inference requests in a multi-level heterogeneous environment, including the following steps: S0. System initialization and affinity table construction: Construct a multi-level heterogeneous large language model inference service system; Based on offline benchmark stress testing, construct a performance affinity index table to quantify the matching relationship between "request-(model-hardware)"; Based on the index table, determine the weight coefficient of each inference engine instance. S1. Construct a candidate request set for inference engine instances: The system receives user requests in real time and maintains a global waiting queue; parses the QoS constraint information carried in the request; filters out inference engine instances that meet the constraint requirements based on the constraint information, and adds the user request to the candidate request set of the corresponding inference engine instance. S2. Calculate the fairness score of candidate requests: When it is detected that there are available resources for a certain inference engine instance, calculate the fairness priority score of each candidate request using a virtual token counter for the corresponding candidate request set. S3. Perform affinity score calculation and comprehensive scoring: Query the index table to obtain the affinity score of each candidate request to the target inference engine instance; calculate the comprehensive scheduling score of each candidate request based on the affinity score and the fairness priority score; S4. Issue a request and update the status: Based on the comprehensive scheduling score, issue the request with the highest score to the target inference engine for execution, and remove the request from the global queue; update the system resource status and user counter value, and return to step S1.

[0009] Preferably, the multi-level heterogeneous inference service system includes three bottom-up heterogeneous features: application request layer QoS heterogeneity (covering different accuracy baselines and model size requirements), model layer heterogeneity (deploying model copies with different parameter amounts and quantization versions), and computing resource layer heterogeneity (composed of chips with different performance characteristics, computing power generation differences, and memory bandwidth).

[0010] Preferably, the large language model reasoning service system in the multi-level heterogeneous environment includes three layers of bottom-up heterogeneous features: Heterogeneous computing resource layer: It consists of chips with different performance characteristics, computing power generation, video memory bandwidth and hardware instruction set architecture. The chips include various models of graphics processing units (GPUs) and domestic computing acceleration chips. Model layer heterogeneity: It includes several types of large models, a basic large language model for each type, and multiple different quantized version copies, as well as model copies with different parameter counts; Application request layer QoS heterogeneity: including QoS requirements for inference requests. The QoS requirements include minimum requirements for inference accuracy and requirements for model parameter size. The inference accuracy requirements cover multiple quantization accuracies including FP16, BF16, INT8 and INT4, and the model parameter size requirements cover different parameter size levels including 0.6B, 7B, 13B and 70B.

[0011] Preferably, the process of constructing the index table in step S0 includes the following sub-steps: S01. Load Scenarios Classification Definition: Based on the differences in computational characteristics between the prefill and decoding stages in the large language model inference process, and with preset input sequence length thresholds and output sequence length thresholds, inference requests are classified into four typical load scenarios: Short Input & Short Output (SS); Short Input & Long Output (SL); Long Input & Short Output (LS); and Long Input & Long Output (LL). S02, Full-combination offline benchmark stress test: For each "model-hardware" execution pair in the system, perform offline benchmark stress tests under the four typical load scenarios to obtain the average stable throughput rate under near full load conditions; S03. Relative Performance Score Calculation: Select the highest-performing execution pair in the system as the global benchmark, and calculate the ratio of the measured throughput rate of each target execution pair under a specific load scenario to the benchmark value to obtain the relative performance score. ; S04. Index table construction: Using the relative performance score as the affinity score, establish a global performance affinity index table with "request-(model-hardware)" as the joint key value for real-time viewing in step S3.

[0012] Preferably, the method for calculating and filtering the relative performance score in step S03 is as follows: Measurement process: In an offline testing environment, the maximum stable throughput rate of each (model-hardware) combination was recorded under four inference request load types (SS, SL, LS, LL) using concurrent execution until the GPU memory was close to saturation. The unit is the number of requests processed per second; Calculation process: Set the throughput rate of the baseline (model-hardware) combination in the corresponding scenario as follows: Calculate the relative performance score of the remaining combinations in the corresponding scenarios. Each relative performance score is denoted as the affinity of each (model-hardware) combination in the corresponding scenario. ; Offline quantization filtering: If measured Less than the preset threshold Then, the affinity of the combination in the corresponding scenario will be added to the index table. Forced marking as invalid or minimal value enables automatic avoidance of inefficient matching such as "weak computing power nodes processing long pre-filled tasks" during the online scheduling phase.

[0013] Preferably, the process of selecting inference engine instances that meet the constraints and generating a candidate request set in step S1 specifically includes the following sub-steps: S11, Request QoS Resolution and Feasibility Domain Construction: When a new inference request is received... At that time, the Quality of Service (QoS) vector carried by the device is parsed to extract the minimum accuracy requirement. With minimum model parameter size ; Traverse all deployed inference service instances in the system and filter those that meet the requirements and All (model-hardware) combinations constitute the exclusive execution feasible domain of this request. , will request Synchronously add it to the "candidate list for execution" of all inference engines that meet its feasible domain conditions, and send the request. Its feasible domain marker is added to the global waiting queue; S12. Spare space triggering based on concurrency threshold: Each inference engine presets the maximum number of concurrent executions during the system startup phase. The scheduler monitors the current number of active requests for each engine in real time. When the target inference engine is detected The state satisfies When the engine has available computing power, the task scheduling process for that engine is triggered. S13. Candidate Request Set Generation: In response to the trigger signal in step S12, the scheduler directly reads the target inference engine. The "candidate list to be executed" maintained in step S11 is directly used as the current candidate request set. No further traversal and filtering operations are performed to minimize scheduling latency and proceed to the subsequent scoring stage; Preferred minimum accuracy requirement and minimum model parameter size Dynamically adjustable; when the system detects that the load exceeds a preset threshold, the scheduler adjusts the load according to a preset strategy. or The value of expands the feasible domain of the request in exchange for a higher hardware affinity score and system throughput.

[0014] Preferably, the joint determination logic of the weighted VTC billing weight coefficient in step S2 is as follows: the incremental resource consumption of a user is recorded as the total amount of tokens requested multiplied by the comprehensive billing weight; wherein, the comprehensive billing weight is jointly determined by the specification weight (based on the model parameter scale and quantification accuracy) and the value coefficient (based on the hardware asset value and measured computing power). This mechanism achieves a substantial measurement of the multi-tenant service volume acquisition capability in a heterogeneous resource pool by imposing higher virtual billing values ​​on requests using high-value computing power resources and high-specification model resources.

[0015] Preferably, in the Virtual Token Counter (VTC) algorithm described in step S2, the user's real-time VTC value is the fairness indicator. VTC Consumption Calculation: The user's incremental resource consumption is recorded as... ;in, According to the model The specification weights are determined by the parameter size, quantization accuracy, and corresponding computational complexity. According to hardware The value coefficient is determined by the asset value, the architectural generation difference, and the measured computing power TFLOPS; This represents the total number of input and output tokens for this request.

[0016] Preferably, the fairness decision in step S2 also includes a VTC level alignment strategy: the system monitors the minimum VTC value of all active users in real time; if a user in an inactive state (i.e., currently has no pending requests) re-initiates a request and their current VTC value is lower than the minimum value, then the user is forced to be raised to the minimum value to prevent them from obtaining an unjustified high priority through extremely low billing values.

[0017] Preferably, the comprehensive scheduling scoring algorithm in step S3 specifically adopts the following dynamic weighting formula: The formula for calculating the requested integrated scheduling score is as follows: ; In the above formula, The final overall scheduling score requested. Prioritize scores based on fairness; Affinity score; weighting coefficient and These are hyperparameters preset before scheduling, used to ensure fairness for users while also taking into account the overall inference efficiency of the system.

[0018] Preferably, the step S4 of updating the corresponding user's VTC value specifically executes the following two-stage settlement logic: (1) Pre-deduction stage: Before the request is issued, the estimated resource consumption is calculated based on the number of input tokens in the request and the maximum number of output tokens preset by the system, and the pre-deduction is performed in the user's VTC; (2) Settlement correction phase: When the inference engine completes the generation task and returns the result, obtain the actual number of output tokens generated; calculate the difference between the actual resource consumption and the estimated amount, and perform correction operations on the user's VTC to refund the excess and supplement the deficiency.

[0019] The pre-selected step S3 and S4 also includes a non-blocking scheduling method based on a two-layer architecture, including: Real-time status feedback mechanism: Each lower-level execution worker reports the current number of active requests for its model replica instance to the scheduler in real time. When the number is lower than the maximum concurrency set at startup, it is determined that the instance has the physical conditions to receive new requests. Penetration matching and sequential processing logic: When the scheduler traverses the global queue, it performs sequential best-preference matching; if the highest priority request is currently being evaluated... If the required specific (model-hardware) pair is full, the scheduler will not block the queue, but will continue to retrieve subsequent requests from the queue. ; Fairness-based anti-starvation lockout mechanism: If subsequent requests... It can be executed on other model instances with available resources, and the scheduler allows it to be distributed across multiple instances; meanwhile, requests that are skipped... In the next scheduling cycle, it will obtain a higher fairness weight score through the VTC mechanism, ensuring that it is prioritized for processing as soon as there is a vacancy in the target resource, thereby achieving a dynamic balance between global throughput and request timeliness.

[0020] This application also provides a large language model inference request scheduling system for multi-level heterogeneous environments, characterized by comprising: The initialization and affinity table construction module is used to build a multi-level heterogeneous large language model inference service system. Based on offline benchmark stress testing, it constructs a performance affinity index table of "request-(model-hardware)" matching relationship and determines the weight coefficient of each inference engine instance based on the index table. The request receiving and candidate set construction module is used to receive user requests in real time and maintain a global waiting queue, parse the QoS constraint information carried in the request, filter out inference engine instances that meet the constraint requirements based on the constraint information, and add the user request to the candidate request set of the inference engine instance. The fairness score calculation module is used to calculate the fairness priority score of each candidate request for the corresponding candidate request set when it is detected that there are available resources for a certain inference engine instance, using a virtual token counter. The comprehensive score calculation module is used to query the performance affinity index table, obtain the affinity score of each candidate request to the target inference engine instance, and calculate the comprehensive scheduling score of each candidate request based on the affinity score and the fairness priority score. The request scheduling and status update module is used to send the request with the highest score to the target inference engine for execution based on the comprehensive scheduling score, remove the request from the global queue, update the system resource status and user counter value, and trigger the request receiving and candidate set construction module to continue processing subsequent requests.

[0021] This application leverages the affinity relationship between "request load characteristics, model specifications, and hardware capabilities" to fully exploit the performance advantages of different tasks on heterogeneous GPUs, thus avoiding the blocking of computationally intensive long tasks on nodes with weak computing power. Furthermore, by introducing a VTC mechanism with differentiated weights and a two-stage correction strategy, a fairness metric adapted to heterogeneous resources is established.

[0022] This application features strong heterogeneous adaptability, high fairness guarantee, and high resource utilization, and can be widely applied to large model inference service clusters in the cloud and at the edge. Attached Figure Description

[0023] Figure 1 is a flowchart of a large language model inference request scheduling method for multi-level heterogeneous environments provided in this application; Figure 2 is a flowchart illustrating the request affinity index creation and matching method based on offline performance profiling provided in this application; Figure 3 is a flowchart illustrating the VTC billing and two-stage settlement correction method based on differentiated weights provided in this application. Figure 4 is a schematic diagram of the structure of a large language model inference request scheduling system for a multi-level heterogeneous environment provided in this application. Detailed Implementation

[0024] This application discloses a method and system for scheduling large language model inference requests in a multi-level heterogeneous environment. By constructing a "request-(model-hardware)" feasible domain, introducing a virtual token counter (VTC) mechanism based on differentiated weight coefficients, and combining it with an affinity matching algorithm, this application features high fairness assurance, high utilization of heterogeneous resources, and strong adaptability, and can be widely applied to large model inference service clusters.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in specific embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In one embodiment, this application provides a method for scheduling large language model inference requests in a multi-level heterogeneous environment, characterized in that the method includes the following steps: S0. System initialization and affinity table construction: Construct a multi-level heterogeneous large language model inference service system; Based on offline benchmark stress testing, construct a performance affinity index table to quantify the matching relationship between "request-(model-hardware)"; Based on the index table, determine the weight coefficient of each inference engine instance. S1. Construct a candidate request set for inference engine instances: The system receives user requests in real time and maintains a global waiting queue; parses the QoS constraint information carried in the request; filters out inference engine instances that meet the constraint requirements based on the constraint information, and adds the user request to the candidate request set of the corresponding inference engine instance. S2. Calculate the fairness score of candidate requests: When it is detected that there are available resources for a certain inference engine instance, calculate the fairness priority score of each candidate request using a virtual token counter for the corresponding candidate request set. S3. Perform affinity score calculation and comprehensive scoring: Query the index table to obtain the affinity score of each candidate request to the target inference engine instance; calculate the comprehensive scheduling score of each candidate request based on the affinity score and the fairness priority score; S4. Issue a request and update the status: Based on the comprehensive scheduling score, issue the request with the highest score to the target inference engine for execution, and remove the request from the global queue; update the system resource status and user counter value, and return to step S1.

[0027] In one embodiment, as shown in Figure 1, a flowchart of a large language model inference request scheduling method for a multi-level heterogeneous environment according to this application is presented. The specific steps include: Step 100 (System Initialization and Profile Construction): Deploy the triple heterogeneous inference service system; based on offline benchmark stress testing, construct a "request-(model-hardware)" affinity performance index table covering four request load scenarios; configure the VTC billing weight coefficient jointly determined by hardware computing power and model specifications.

[0028] Step 101 (Request Reception and Queue Maintenance): Receive user inference requests in real time and maintain a global waiting queue; simultaneously monitor the status of all inference engines, parse the QoS requirements of the requests (such as accuracy and model size), and pre-add the requests to the waiting queues of eligible inference engines (i.e., build a candidate list).

[0029] It should be noted that this application supports both 'rigid contract' and 'flexible contract' service modes. In the embodiment oriented towards flexible contracts (Best-effort), the QoS resolution in step 101 has a dynamic relaxation mechanism. The system monitors the global load status in real time (such as queue length or average waiting time). When the system load is detected to exceed a preset warning threshold, the scheduler triggers a QoS degradation strategy: reducing the minimum precision requirement of the request. (e.g., FP16) Downgrade to a lower precision (e.g., INT8), or reduce the minimum model parameter size. (e.g., 72B) downgraded to a smaller scale (e.g., 13B). Through this dynamic adjustment, the system can expand the 'executable feasible domain' of the request, thereby matching more inference engines with idle resources in subsequent affinity matching (step 105), trading a moderate loss of accuracy for an increase in overall system throughput and a reduction in response latency.

[0030] Step 102 (Empty Space and Task Detection): Determine if there is an inference engine in the system with an empty space (i.e., the current concurrency is less than the maximum threshold) and there are requests waiting to be executed in the waiting queue of that engine; if not, return to step 101 to continue receiving and monitoring; if yes, proceed to step 103.

[0031] Step 103 (Target Locking and Candidate Acquisition): Select the inference engine with available slots as the target engine for this scheduling, and directly obtain the current request waiting queue of the target engine (as a set of candidate requests).

[0032] Step 104 (Candidate Set Request VTC Fairness Score): For requests in the above candidate queue, obtain the real-time VTC value of the user to which the request belongs; perform a counter boosting operation on the reactivated user, and determine the user's fairness score based on the VTC value.

[0033] Step 105 (Target Node Request Affinity Calculation and Comprehensive Score): Based on the current target inference engine's node and request information, consult the index table established in Step 100 to calculate the affinity score of candidate requests; combine fairness and affinity to calculate a weighted comprehensive scheduling score.

[0034] Step 106 (Optimal Task Distribution and Closed-Loop Feedback): Select the request with the highest total score, distribute it to the target inference engine for execution, and remove it from the relevant queues; update the status of the engine and the corresponding user VTC counter value; return to step 100 or 101 to continue monitoring resource availability.

[0035] In one embodiment, as shown in Figure 2, the specific process of establishing the performance affinity index includes the following steps: Step 200 (Load Classification Definition): Based on the difference in characteristics between Prefill (computation-intensive) and Decoding (memory-intensive) in large model inference, preset input and output length thresholds are used to classify requests into four scenarios: short input short output (SS), short input long output (SL), long input short output (LS), and long input long output (LL). Step 201 (Full Combination Offline Load Testing): Traverse each "model-hardware" execution pair in the system and perform offline benchmark tests in the above four scenarios, recording the maximum stable throughput rate (Req / s) when resources are saturated. Step 202 (Relative Performance Score Calculation): Select the highest-performing combination within the system as the global benchmark. Calculate the ratio of the measured throughput rate of the target combination in a specific scenario to the benchmark value to obtain the relative performance score. ; Step 203 (Inefficiency Filtering): Determine if the relative performance score is less than a preset threshold. (e.g., 0.05); if yes, mark the affinity of the combination as invalid to avoid inefficient matching (e.g., T4 graphics card handling long input tasks); if no, retain the score; Step 204 (Index Construction): The filtered relative performance scores are used to build a global affinity index table for real-time reference during the online scheduling phase (corresponding to S3 in Figure 1).

[0036] In one embodiment, as shown in Figure 3, the VTC billing and two-phase settlement correction method based on differentiated weights includes the following steps: Step 300 (Heterogeneous Weight Determination): Jointly determine the comprehensive cost weight coefficient per unit token. Specifically, the specification weight is determined based on the model's parameter scale and quantization accuracy, and the value coefficient is determined based on the hardware's asset value and measured computing power (TFLOPS); the product of these two factors is the comprehensive cost weight for processing a unit token. Step 301 (Water Level Alignment): Before scheduling, monitor the status of the user initiating the request. If the user is inactive and re-initiates the request, check if their VTC is lower than the minimum value for currently active users; if so, force their VTC to be raised to that minimum value to prevent them from obtaining unjustified excessively high priority. Step 302 (Phase 1: Pre-deduction): Before the request is issued, based on the number of input tokens in the request and the system's preset maximum number of output tokens (Max_New_Tokens), combined with the weights in Step 300, the estimated resource consumption is calculated and pre-deducted from the user's VTC account; Step 303 (Second Stage: Settlement Correction): When the inference engine completes the generation task and returns the result, obtain the actual number of output tokens generated; calculate the difference between the actual resource consumption and the estimated amount; Step 304 (Account Update): Update the user account based on the difference. If the actual consumption is less than the estimate, the difference will be refunded to the user's VTC account; if it is more than the estimate, the difference will be deducted, thus achieving accurate billing with "refund for overpayment and supplement for underpayment" in heterogeneous environments.

[0037] In one embodiment, as shown in Figure 4, a large language model inference request scheduling system for multi-level heterogeneous environments is provided, characterized by comprising: 41 Initialization and affinity table construction module, used to build a multi-level heterogeneous large language model inference service system. Based on offline benchmark stress test, it constructs a performance affinity index table of "request-(model-hardware)" matching relationship, and determines the weight coefficient of each inference engine instance based on the index table; 42 Request Receiving and Candidate Set Construction Module, used to receive user requests in real time and maintain a global waiting queue, parse the QoS constraint information carried in the request, filter out inference engine instances that meet the constraint requirements based on the constraint information, and add the user request to the candidate request set of the inference engine instance; 43. Fairness score calculation module, used to calculate the fairness priority score of each candidate request for the corresponding candidate request set when it is detected that there are available resources for a certain inference engine instance; 44. The comprehensive score calculation module is used to query the performance affinity index table, obtain the affinity score of each candidate request to the target inference engine instance, and calculate the comprehensive scheduling score of each candidate request based on the affinity score and the fairness priority score. The 45 request scheduling and status update module is used to send the request with the highest score to the target inference engine for execution based on the comprehensive scheduling score, remove the request from the global queue, update the system resource status and user counter value, and trigger the request receiving and candidate set construction module to continue processing subsequent requests.

[0038] This application discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as: constructing a heterogeneous affinity index; receiving user inference requests in real time and parsing QoS constraints; filtering the feasible domain of requests based on dynamic resource status; calculating user fairness priority based on a VTC mechanism with differentiated weights; performing multi-objective integrated scheduling in combination with affinity scores; and performing a two-stage VTC account settlement and correction.

[0039] This application provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the methods provided in the above-described method embodiments.

[0040] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0042] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0043] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] Numerous specific details are set forth in this specification. However, it will be understood that embodiments of the invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification. Similarly, it should be understood that, in order to simplify the disclosure of this invention and aid in understanding one or more aspects of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. It should be noted that, without conflict, embodiments and features in these embodiments can be combined with each other.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for scheduling inference requests for large language models in multi-level heterogeneous environments, characterized in that, The method includes the following steps: S0. System initialization and affinity table construction: Construct a multi-level heterogeneous large language model inference service system; Based on offline benchmark stress testing, construct a performance affinity index table to quantify the matching relationship between "request-(model-hardware)"; Based on the index table, determine the weight coefficient of each inference engine instance. S1. Construct a candidate request set for inference engine instances: The system receives user requests in real time and maintains a global waiting queue; it parses the QoS constraint information carried in the requests. Based on the constraint information, inference engine instances that meet the constraint requirements are selected, and the user request is added to the candidate request set of the corresponding inference engine instance. S2. Calculate the fairness score of candidate requests: When it is detected that there are available resources for a certain inference engine instance, calculate the fairness priority score of each candidate request using a virtual token counter for the corresponding candidate request set. S3. Perform affinity score calculation and comprehensive scoring: Query the index table to obtain the affinity score of each candidate request to the target inference engine instance; Based on the affinity score and the fairness priority score, calculate the comprehensive scheduling score for each candidate request; S4. Send the request and update the status: Based on the comprehensive scheduling score, send the request with the highest score to the target inference engine for execution, and remove the request from the global queue at the same time; Update the system resource status and user counter values, then return to step S1.

2. The method according to claim 1, characterized in that, The large language model reasoning service system in the multi-level heterogeneous environment includes three layers of bottom-up heterogeneous features: Heterogeneous computing resource layer: It consists of chips with different performance characteristics, computing power generation, video memory bandwidth and hardware instruction set architecture. The chips include various models of graphics processing units (GPUs) and domestic computing acceleration chips. Model layer heterogeneity: It includes several types of large models, a basic large language model for each type, and multiple different quantized version copies, as well as model copies with different parameter counts; Application request layer QoS heterogeneity: including QoS requirements for inference requests. The QoS requirements include minimum requirements for inference accuracy and requirements for model parameter size. The inference accuracy requirements cover multiple quantization accuracies including FP16, BF16, INT8 and INT4, and the model parameter size requirements cover different parameter size levels including 0.6B, 7B, 13B and 70B.

3. The method according to claim 1, characterized in that, The process of constructing the index table in step S0 includes the following sub-steps: S01. Load Scenarios Classification Definition: Based on the differences in computational characteristics between the prefill and decoding stages in the large language model inference process, and with preset input sequence length thresholds and output sequence length thresholds, inference requests are classified into four typical load scenarios: Short Input & Short Output (SS); Short Input & Long Output (SL); Long Input & Short Output (LS); and Long Input & Long Output (LL). S02, Full-combination offline benchmark stress test: For each "model-hardware" execution pair in the system, perform offline benchmark stress tests under the four typical load scenarios to obtain the average stable throughput rate under near full load conditions; S03. Relative Performance Score Calculation: Select the highest-performing execution pair in the system as the global benchmark, and calculate the ratio of the measured throughput rate of each target execution pair under a specific load scenario to the benchmark value to obtain the relative performance score. ; S04. Index table construction: Using the relative performance score as the affinity score, a global performance affinity index table is established with "request-(model-hardware)" as the joint key value, for real-time viewing in step S3.

4. The method according to claim 3, characterized in that, The specific method for calculating and filtering the relative performance score in step S03 is as follows: Measurement process: In an offline testing environment, the maximum stable throughput rate of each (model-hardware) combination was recorded under four inference request load types (SS, SL, LS, LL) using concurrent execution until the GPU memory was close to saturation. The unit is the number of requests processed per second; Calculation process: Set the throughput rate of the baseline (model-hardware) combination in the corresponding scenario as follows: Calculate the relative performance score of the remaining combinations in the corresponding scenarios. Each relative performance score is denoted as the affinity of each (model-hardware) combination in the corresponding scenario. ; Offline quantization filtering: If measured Less than the preset threshold Then, the affinity of the combination in the corresponding scenario will be added to the index table. Forced marking as invalid or minimum value enables automatic avoidance of inefficient matching during the online scheduling phase.

5. The method according to claim 1, characterized in that, The process of selecting inference engine instances that meet the constraints and generating a candidate request set as described in step S1 specifically includes the following sub-steps: S11, Request QoS Resolution and Feasibility Domain Construction: When a new inference request is received... At that time, the Quality of Service (QoS) vector carried by the device is parsed to extract the minimum accuracy requirement. With minimum model parameter size ; Traverse all deployed inference service instances in the system and filter those that meet the requirements and All (model-hardware) combinations constitute the exclusive execution feasible domain of this request. , will request Synchronously add it to the "candidate list for execution" of all inference engines that meet its feasible domain conditions, and send the request. Its feasible domain marker is added to the global waiting queue; S12. Spare space triggering based on concurrency threshold: Each inference engine presets the maximum number of concurrent executions during the system startup phase. ; The scheduler monitors the current number of active requests for each engine in real time. When the target inference engine is detected The state satisfies When the engine has available computing power, the task scheduling process for that engine is triggered. S13. Candidate Request Set Generation: In response to the trigger signal in step S12, the scheduler directly reads the target inference engine. The "candidate list to be executed" that was maintained in step S11 is directly used as the current candidate request set. No further traversal and filtering operations are performed to minimize scheduling latency and proceed to the subsequent scoring stage; Among them, the minimum accuracy requirement and minimum model parameter size Dynamically adjustable; when the system detects that the load exceeds a preset threshold, the scheduler adjusts the load according to a preset strategy. or The value of expands the feasible domain of the request in exchange for a higher hardware affinity score and system throughput.

6. The method according to claim 1, characterized in that, In step S2, the Virtual Token Counter (VTC) algorithm uses the user's real-time VTC value as a fairness indicator. VTC Consumption Calculation: The user's incremental resource consumption is recorded as... ;in, According to the model The specification weights are determined by the parameter size, quantization accuracy, and corresponding computational complexity. According to hardware The value coefficient is determined by the asset value, the architectural generation difference, and the measured computing power TFLOPS; This represents the total number of input and output tokens for this request. Specifically, the user counter value is subject to a two-stage settlement logic: Pre-deduction stage: Before the request is issued, based on the number of input tokens and the expected output length of the request, the estimated resource consumption is calculated according to the VTC consumption calculation formula and pre-deducted from the user's VTC; Settlement Correction Phase: When the inference engine completes the generation task and returns the results, obtain the actual number of output tokens generated; calculate the difference between the actual resource consumption and the estimated amount, and perform replenishment or additional deduction operations on the user's VTC; This also includes a VTC level alignment and boosting strategy: The system monitors the minimum VTC of all currently active users in real time. ; If the user is inactive Re-initiate the request and its current Then a mandatory order This is to prevent users from obtaining unfairly high priority in heterogeneous resource allocation through extremely low billing values ​​due to long-term non-consumption of resources.

7. The method according to claim 6, characterized in that, Step S3 describes a comprehensive scheduling scoring algorithm that specifically employs the following dynamic weighting formula: The formula for calculating the requested integrated scheduling score is as follows: ; In the above formula, The final overall scheduling score requested. The score is based on fairness priority. Affinity score; Weighting coefficient and These are hyperparameters preset before scheduling, used to ensure fairness for users while also taking into account the overall inference efficiency of the system.

8. The method according to claim 1, characterized in that, Steps S3 and S4 also include a non-blocking scheduling method based on a two-layer architecture, including: Real-time status feedback mechanism: Each lower-level execution worker reports the current number of active requests for its model replica instance to the scheduler in real time. When the number is lower than the maximum concurrency set at startup, it is determined that the instance has the physical conditions to receive new requests. Penetration matching and sequential processing logic: When the scheduler traverses the global queue, it performs sequential best-preference matching; if the highest priority request is currently being evaluated... If the required specific (model-hardware) pair is full, the scheduler will not block the queue, but will continue to retrieve subsequent requests from the queue. ; Fairness-based anti-starvation lockout mechanism: If subsequent requests... It can be executed on other model instances with available resources, and the scheduler allows it to be distributed across multiple instances; meanwhile, requests that are skipped... In the next scheduling cycle, it will obtain a higher fairness weight score through the VTC mechanism, ensuring that it is prioritized for processing as soon as there is a vacancy in the target resource, thereby achieving a dynamic balance between global throughput and request timeliness.

9. A large language model inference request scheduling system for multi-level heterogeneous environments as described in any one of claims 1-8, characterized in that, include: The initialization and affinity table construction module is used to build a multi-level heterogeneous large language model inference service system. Based on offline benchmark stress testing, it constructs a performance affinity index table of "request-(model-hardware)" matching relationship and determines the weight coefficient of each inference engine instance based on the index table. The request receiving and candidate set construction module is used to receive user requests in real time and maintain a global waiting queue, parse the QoS constraint information carried in the request, filter out inference engine instances that meet the constraint requirements based on the constraint information, and add the user request to the candidate request set of the inference engine instance. The fairness score calculation module is used to calculate the fairness priority score of each candidate request for the corresponding candidate request set when it is detected that there are available resources for a certain inference engine instance, using a virtual token counter. The comprehensive score calculation module is used to query the performance affinity index table, obtain the affinity score of each candidate request to the target inference engine instance, and calculate the comprehensive scheduling score of each candidate request based on the affinity score and the fairness priority score. The request scheduling and status update module is used to send the request with the highest score to the target inference engine for execution based on the comprehensive scheduling score, remove the request from the global queue, update the system resource status and user counter value, and trigger the request receiving and candidate set construction module to continue processing subsequent requests.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-9.

Images