Large language model heterogeneous computing power resource allocation scaling decision-making method and system and medium

Abstract

The invention provides a large language model heterogeneous computing power resource configuration scaling decision-making method and system and a medium. The method comprises the steps of obtaining a candidate strategy set corresponding to a to-be-processed task; the candidate strategy set comprises a plurality of candidate scaling operation strategies corresponding to the to-be-processed task; according to the model scale of a large language model corresponding to the to-be-processed task, the resource configuration corresponding to each candidate scaling operation strategy and the load feature of the to-be-processed task, predicting to obtain the service improvement amount of executing each candidate scaling operation strategy to the large language model; according to the service promotion amount and the execution cost of each candidate scaling operation strategy, obtaining a strategy score of each candidate scaling operation strategy; according to the highest strategy score and a preset hysteresis threshold value, the effectiveness of the target scaling operation strategy corresponding to the highest strategy score is judged; and if the target scaling operation strategy is effective, adjusting the computing power resource configuration according to the target scaling operation strategy, so that the large language model processes the to-be-processed task according to the adjusted computing power resource configuration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of adjusting computing power resource configuration, and in particular to a heterogeneous computing power resource configuration scaling decision method and system for large language models and a medium. BACKGROUND

[0002] In recent years, large language models (LLMs) have made breakthrough progress in natural language processing, and their inference services have been widely applied to online chat, content generation, document summarization, and other scenarios. The LLM inference process has significant stage characteristics, typically divided into a prefilling (Prefill) stage and a decoding (Decode) stage. The prefilling stage is responsible for parallel processing of all tokens of user input text, and its computational load is biased towards compute-intensive, which can effectively utilize the large-scale parallel computing capabilities of hardware such as GPUs. The decoding stage is responsible for generating output text one token at a time, as the generation of each new token depends on all previous tokens, this process has inherent seriality, and its performance bottleneck is often limited by memory bandwidth rather than raw computing power.

[0003] This difference in appeal for hardware resource computing and memory bandwidth poses a huge challenge to the resource configuration of LLM inference services. Using a fixed hardware configuration (e.g., a single model of GPU) cannot simultaneously optimize two key service level objective (SLO) indicators: Time To First Token (TTFT) and Time Per Output Token (TPOT). TTFT mainly reflects the efficiency of the prefilling stage, while TPOT reflects the efficiency of the decoding stage. Under a fixed hardware configuration, facing dynamic changes in workloads (such as fluctuations in request concurrency and input / output length), service providers often face a dilemma: either over-provision resources to meet SLOs under peak load, resulting in idle resources during off-peak periods and high total cost of ownership (TCO); or under-provision to save costs, resulting in TTFT or TPOT exceeding SLO thresholds during load peaks, causing a decline in user experience and potential service breach risks.

[0004] To cope with dynamic loads, the prior art generally uses an autoscaling mechanism. Traditional autoscaling is mostly reactive or based on fixed threshold strategies, such as increasing service instance replicas (i.e., horizontal scaling) when CPU utilization exceeds 80%. Some more advanced methods introduce predictive scaling by analyzing historical load patterns to reserve resources in advance to cope with foreseeable load peaks. However, these methods mainly focus on "when" and "how much" to scale, and fail to effectively solve the problem of "how" to scale in a heterogeneous computing environment. Heterogeneous computing environments contain multiple types of computing resources, such as different types of CPUs, GPUs, and specialized AI accelerators (such as NPUs, TPUs, etc.), each with its own strengths and weaknesses in performance, cost, and power consumption. Existing scaling strategies lack a unified decision framework and cannot prospectively compare two fundamentally different scaling paths when making scaling decisions: one is horizontal scaling, which increases or decreases the number of instances of the same type, and the other is heterogeneous scaling, which migrates the service to another instance of a different type or specification.

[0005] In addition, existing technologies often ignore the transition cost (Transition Cost) caused by the operation itself when making scaling decisions. For example, in the process of heterogeneous scaling, new and old instances may need to run in parallel for a period of time to achieve smooth migration, resulting in additional instance overlap costs. At the same time, during the service switching process, the response time will temporarily deteriorate due to cold start, cache unpreheat, etc., causing the application performance index (Apdex) to decline. The implicit loss caused by this service degradation to the business (such as user loss) is usually not quantified and considered in the cost.

[0006] Some existing technologies provide a query routing method based on reinforcement learning, which selects routing between multiple different LLM models through mechanisms such as multi-armed bandit, PPO, etc., to balance answer quality, resource consumption, and delay. However, the core of this technology is model selection and request routing at the application layer, not the elastic scaling of heterogeneous computing resources at the infrastructure layer that the present invention focuses on. Moreover, its decision mechanism relies on learning from after-the-fact rewards, lacking explicit and economic prediction and measurement of various costs (especially transition costs) before performing scaling actions. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings and deficiencies in the prior art and provide a large language model heterogeneous computing resource configuration scaling decision method, system and medium.

[0008] The first aspect of the embodiments of the present application provides a large language model heterogeneous computing resource configuration scaling decision method, comprising: obtaining a candidate strategy set corresponding to a to-be-processed task; the candidate strategy set comprises a plurality of candidate scaling operation strategies corresponding to the to-be-processed task; predict service improvement amounts of the large language model when each of the candidate scaling operation strategies is executed according to the model size of the large language model corresponding to the to-be-processed task, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the to-be-processed task; obtain execution costs of each of the candidate scaling operation strategies; obtain strategy scores of each of the candidate scaling operation strategies according to the service improvement amounts and the execution costs; determine effectiveness of a target scaling operation strategy corresponding to the highest strategy score according to the highest strategy score and a preset hysteresis threshold; if the target scaling operation strategy is effective, adjust the computing resource configuration according to the target scaling operation strategy, and cause the large language model to process the to-be-processed task according to the adjusted computing resource configuration.

[0009] As an implementation manner, before the step of obtaining the candidate strategy set corresponding to the to-be-processed task, the method further includes: if a real-time service index parameter of the large language model is greater than or equal to a preset index threshold, or if a predicted load change amount of the large language model when processing the to-be-processed task is greater than a preset change threshold, obtain the candidate strategy set; wherein operation types of multiple candidate scaling operation strategies in the candidate strategy set include horizontal stretching operation and heterogeneous scaling operation.

[0010] As an implementation manner, the step of predicting service improvement amounts of the large language model when each of the candidate scaling operation strategies is executed according to the model size of the large language model corresponding to the to-be-processed task, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the to-be-processed task includes: input the large language model corresponding to the to-be-processed task, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the to-be-processed task into a service prediction model to obtain predicted service data of the large language model corresponding to each of the candidate scaling operation strategies; obtain service improvement amounts of the large language model when each of the candidate scaling operation strategies is executed according to current service data of the large language model and each of the predicted service data.

[0011] As an implementation manner, the step of obtaining the execution cost of each of the candidate scaling operation strategies includes: obtain a direct cost of the corresponding candidate scaling operation strategy according to a leasing cost and an energy consumption cost of a newly added or replaced instance copy; obtain a transition cost of the corresponding candidate scaling operation strategy according to an instance overlap cost, a service degradation cost, and a migration overhead cost. The execution cost is obtained according to the direct cost and the transition cost.

[0012] As an implementation form, the step of obtaining the strategy score of each candidate scaling operation strategy according to the service improvement amount and the execution cost comprises: The strategy score is obtained according to the quotient of the service improvement amount of each candidate scaling operation strategy and the corresponding execution cost.

[0013] As an implementation form, the preset hysteresis threshold comprises an upper limit hysteresis threshold. If the target scaling operation strategy is valid, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the to-be-processed task according to the adjusted computing resource configuration comprises: If the target scaling operation strategy is valid, and the corresponding strategy score is greater than the upper limit hysteresis threshold, the computing resource configuration is adjusted according to the expansion / upwards heterogeneous switching strategy.

[0014] As an implementation form, the preset hysteresis threshold comprises a lower limit hysteresis threshold; the lower limit hysteresis threshold is less than the upper limit hysteresis threshold. If the target scaling operation strategy is valid, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the to-be-processed task according to the adjusted computing resource configuration comprises: If the target scaling operation strategy is valid, and the corresponding strategy score is less than the lower limit hysteresis threshold, the computing resource configuration is adjusted according to the contraction / downwards heterogeneous switching strategy.

[0015] As an implementation form, the preset hysteresis threshold comprises an upper limit hysteresis threshold and a lower limit hysteresis threshold; the lower limit hysteresis threshold is less than the upper limit hysteresis threshold. After the step of judging the validity of the target scaling operation strategy corresponding to the highest strategy score according to the highest strategy score and the preset hysteresis threshold, the method further comprises: If the target scaling operation strategy is less than or equal to the upper limit hysteresis threshold, and greater than or equal to the lower limit hysteresis threshold, the current computing resource configuration of the large language model is maintained.

[0016] Compared to existing technologies, the heterogeneous computing resource allocation scaling decision method for large language models in this application can predict the service improvement of the large language model by executing each candidate scaling operation strategy based on the large language model corresponding to the task to be processed, the resource allocation corresponding to each candidate scaling operation strategy, and the load characteristics of the task to be processed. It then combines the execution cost of each candidate scaling operation strategy to obtain a strategy score for each strategy. Based on the highest strategy score and a preset hysteresis threshold, it determines the effectiveness of the target scaling operation strategy corresponding to the highest strategy score. If the target scaling operation strategy is effective, it adjusts the computing resource allocation according to the target scaling operation strategy, enabling the large language model to process the task to be processed according to the adjusted computing resource allocation. This method allows for the selection of the target scaling operation strategy with the largest service improvement based on the service improvement and execution cost, thus improving the service performance of the large language model while controlling execution costs.

[0017] A second aspect of this application provides a large language model heterogeneous computing resource allocation scaling decision system, including: The candidate strategy acquisition module is used to acquire a set of candidate strategies for a corresponding task to be processed; the set of candidate strategies includes multiple candidate scaling operation strategies for the corresponding task to be processed. The service improvement acquisition module is used to predict the service improvement of the large language model by executing each of the candidate scaling operation strategies based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed. The execution cost acquisition module is used to acquire the execution cost of each of the candidate scaling operation strategies; The strategy score acquisition module is used to acquire the strategy score of each of the candidate scaling operation strategies based on the service improvement amount and the execution cost. The strategy effectiveness acquisition module is used to determine the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold. The resource configuration adjustment module is used to adjust the computing power resource configuration according to the target scaling operation strategy if the target scaling operation strategy is effective, so that the large language model can process the task to be processed according to the adjusted computing power resource configuration.

[0018] Compared to existing technologies, the heterogeneous computing resource allocation scaling decision system for large language models in this application can predict the service improvement of the large language model by executing each candidate scaling operation strategy based on the large language model corresponding to the task to be processed, the resource allocation corresponding to each candidate scaling operation strategy, and the load characteristics of the task to be processed. It then combines the execution cost of each candidate scaling operation strategy to obtain a strategy score for each strategy. Based on the highest strategy score and a preset hysteresis threshold, it determines the effectiveness of the target scaling operation strategy corresponding to the highest strategy score. If the target scaling operation strategy is effective, it adjusts the computing resource allocation according to the target scaling operation strategy, enabling the large language model to process the task to be processed according to the adjusted computing resource allocation. This allows for the selection of the target scaling operation strategy with the higher service improvement and execution cost to adjust the computing resource allocation, which is beneficial for improving the service performance of the large language model while controlling execution costs.

[0019] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the large language model heterogeneous computing resource allocation scaling decision method as described above.

[0020] To provide a clearer understanding of this application, the specific embodiments of this application will be described below in conjunction with the accompanying drawings. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a method for heterogeneous computing resource allocation scaling decision-making for large language models, according to one embodiment of this application.

[0022] Figure 2 This is a first schematic diagram of a heterogeneous computing resource allocation scaling decision system for a large language model according to an embodiment of this application.

[0023] Figure 3 This is a second schematic diagram of a large language model heterogeneous computing resource allocation scaling decision system according to an embodiment of this application.

[0024] Figure 4 This is a schematic diagram of an economic benefit evaluation model according to an embodiment of this application.

[0025] 100. Large Language Model Heterogeneous Computing Power Resource Allocation Scaling Decision System; 101. Candidate Strategy Acquisition Module; 102. Service Improvement Acquisition Module; 103. Execution Cost Acquisition Module; 104. Strategy Score Acquisition Module; 105. Strategy Effectiveness Acquisition Module; 106. Resource Allocation Adjustment Module. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0028] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" as used herein can be interpreted as "when," "when," or "in response to determination."

[0029] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0030] Please see Figure 1 This is a flowchart of the large language model heterogeneous computing power resource allocation scaling decision method according to the first embodiment of this application. The method includes:

[0031] S1: Obtain a set of candidate strategies for the corresponding task to be processed; the set of candidate strategies includes multiple candidate scaling operation strategies for the corresponding task to be processed;

[0032] That is, the relationship between any candidate scaling operation policy a and the candidate policy set S is a∈S

[0033] S2: Based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed, predict the service improvement of the large language model by executing each of the candidate scaling operation strategies.

[0034] The resource configuration includes the instance type, number of replicas, inference framework, quantization precision, etc., corresponding to the candidate scaling operation strategy; the load characteristics of the task to be processed include the input length, concurrency, batch size, expected output length, etc.; the model scale includes the number of model parameters, architecture, etc.

[0035] S3: Obtain the execution cost of each of the candidate scaling operation strategies;

[0036] S4: Based on the service improvement amount and the execution cost, obtain the strategy score for each of the candidate scaling operation strategies;

[0037] S5: Determine the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold;

[0038] The preset hysteresis threshold includes an upper hysteresis threshold and a lower hysteresis threshold; wherein the lower hysteresis threshold is less than the upper hysteresis threshold; the upper hysteresis threshold is a high-water level that triggers resource increases (homogeneous expansion or upward heterogeneous switching), and it is set to a value significantly greater than the cost-benefit balance point (e.g., R=1), such as θ_up = 1.5; the lower hysteresis threshold is a low-water level that triggers resource decreases (homogeneous shrinkage or downward heterogeneous switching), and it is set to a value that clearly indicates that resources are being continuously wasted, usually a negative value, such as θ_down = -0.5.

[0039] After determining the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold, the method further includes:

[0040] If the target scaling operation strategy is less than or equal to the upper limit hysteresis threshold and greater than or equal to the lower limit hysteresis threshold, the current computing resource configuration of the large language model is maintained.

[0041] S6: If the target scaling operation strategy is effective, adjust the computing power resource configuration according to the target scaling operation strategy, so that the large language model can process the task to be processed according to the adjusted computing power resource configuration.

[0042] In a feasible embodiment, before step S1: obtaining the candidate strategy set corresponding to the task to be processed, the following is included:

[0043] If the real-time service metric parameter of the large language model is greater than or equal to a preset metric threshold, or if the predicted load change of the large language model in processing the task to be processed is greater than a preset change threshold, the candidate strategy set is obtained; wherein, the operation types of the multiple candidate scaling operation strategies in the candidate strategy set include horizontal scaling operation and heterogeneous scaling operation.

[0044] The real-time service metrics parameters are Service Level Objectives (SLO) metrics, including Time To First Token (TTFT) and Time Per Output Token (TPOT). TTFT mainly reflects the efficiency of the pre-filling stage, while TPOT reflects the efficiency of the decoding stage.

[0045] Horizontal scaling is used to increase or decrease the number of instances of the same type, while heterogeneous scaling is used to migrate services to instances of a different type or size.

[0046] In a feasible embodiment, step S2: predicting the service improvement of the large language model by executing each of the candidate scaling operation strategies based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed, includes:

[0047] S21: Input the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed into the service prediction model to obtain the prediction service data of the large language model corresponding to each of the candidate scaling operation strategies.

[0048] The service prediction model can be implemented in the following ways:

[0049] Analysis Model: Based on a deep understanding of the LLM inference process and hardware microarchitecture, a mathematical formula is established to describe the relationship between performance and input parameters.

[0050] Experience Mapping / Lookup Table: A large performance database is built through comprehensive offline benchmarking of various hardware and load combinations. During online prediction, prediction results are quickly obtained through table lookup or interpolation.

[0051] Machine learning models: Lightweight regression models (such as gradient boosting trees or neural networks) are trained using historical running data to predict performance.

[0052] The forecast service data represents the predicted SLO satisfactory throughput.

[0053] S22: Based on the current service data of the large language model and each of the predicted service data, obtain the service improvement amount of the large language model by executing each of the candidate scaling operation strategies.

[0054] The current service data is the SLO satisfactory throughput of the large language model based on the current computing resource configuration. Specifically, the SLO satisfactory throughput... It can be obtained through the following formula:

[0055]

[0056] in, To achieve satisfactory throughput for SLO, For total throughput, SLO Completion Rate represents the percentage of SLO requests that are fulfilled.

[0057] The service improvement is obtained using the following formula:

[0058]

[0059] in, Candidate scaling operation strategy The service volume improvement Candidate scaling operation strategy Predictive service data, This is the current service data.

[0060] In a feasible embodiment, step S3: obtaining the execution cost of each of the candidate scaling operation strategies includes:

[0061] S31: Based on the rental cost and energy consumption cost of adding or replacing instance replicas, obtain the direct cost of the corresponding candidate scaling operation strategy;

[0062] The rental cost is the incremental hardware rental fee incurred when adding or replacing instances in executing candidate scaling operation strategy a. For example, adding an instance replica costs the instance's unit-time price multiplied by the evaluation window duration; replacing with a higher-specification instance costs the price difference between the old and new instances multiplied by the evaluation window duration.

[0063] Energy cost is the additional power consumption cost incurred when executing candidate scaling operation strategy a. It is calculated based on the instance's rated power, actual power consumption model under load, evaluation duration, and electricity price (further considerations may include data center PUE value, carbon emission factor, etc.).

[0064] The direct costs are the sum of rental costs and energy costs.

[0065] S32: Based on instance overlap cost, service degradation cost, and migration overhead cost, obtain the transition cost of the corresponding candidate scaling operation strategy;

[0066] Among them, the instance overlap cost is the double resource cost incurred when new and old instances run in parallel during the transition period when a smooth migration strategy such as blue-green deployment or rolling updates is adopted.

[0067] The service degradation cost is the amount by which the application performance index (Apdex) decreases (ΔApdex) during the transition window due to temporary deterioration in response time distribution caused by service restarts, model cold starts, etc. This decrease can be converted into corresponding economic losses using a pre-established business value mapping function to obtain the service degradation cost.

[0068] Migration overhead costs are the resource and human costs consumed in performing the migration operation itself, such as the overhead of running automated scripts, data transfer, model loading, and warm-up.

[0069] S33: The execution cost is obtained based on the direct cost and the transition cost.

[0070] The execution cost is the sum of the direct cost and the transition cost.

[0071] In a feasible embodiment, step S4: obtaining the strategy score of each of the candidate scaling operation strategies based on the service improvement amount and the execution cost, includes:

[0072] The strategy score is obtained by dividing the service improvement amount of each candidate scaling operation strategy by the corresponding execution cost.

[0073] The strategy score is obtained using the following formula:

[0074]

[0075] in, Candidate scaling operation strategy The strategy score mentioned above, i.e., the benefit / cost ratio, Candidate scaling operation strategy The service volume improvement Candidate scaling operation strategy The aforementioned execution cost.

[0076] In one feasible embodiment, the preset hysteresis threshold includes an upper hysteresis threshold;

[0077] S6: If the target scaling operation strategy is effective, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the task to be processed according to the adjusted computing resource configuration, includes:

[0078] S61: If the target scaling operation strategy is effective and the corresponding strategy score is greater than the upper limit hysteresis threshold, adjust the computing power resource configuration according to the expansion / upward heterogeneous switching strategy.

[0079] In one feasible embodiment, the preset hysteresis threshold includes a lower hysteresis threshold; the lower hysteresis threshold is less than the upper hysteresis threshold;

[0080] S6: If the target scaling operation strategy is effective, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the task to be processed according to the adjusted computing resource configuration, includes:

[0081] S62: If the target scaling operation strategy is effective and the corresponding strategy score is less than the lower limit hysteresis threshold, adjust the computing resource configuration according to the scaling down / down heterogeneous switching strategy.

[0082] The expansion / upward heterogeneous switching strategy includes expansion actions and upward heterogeneous actions. The expansion action is homogeneous expansion, and the upward heterogeneous action is switching to a heterogeneous computing power with a higher configuration. The shrinkage / downward heterogeneous switching strategy includes shrinkage actions and downward heterogeneous actions. The shrinkage action is homogeneous shrinkage, and the downward heterogeneous action is switching to a heterogeneous computing power with a lower configuration.

[0083] Compared to existing technologies, the heterogeneous computing resource allocation scaling decision method for large language models in this application can predict the service improvement of the large language model by executing each candidate scaling operation strategy based on the large language model corresponding to the task to be processed, the resource allocation corresponding to each candidate scaling operation strategy, and the load characteristics of the task to be processed. It then combines the execution cost of each candidate scaling operation strategy to obtain a strategy score for each strategy. Based on the highest strategy score and a preset hysteresis threshold, it determines the effectiveness of the target scaling operation strategy corresponding to the highest strategy score. If the target scaling operation strategy is effective, it adjusts the computing resource allocation according to the target scaling operation strategy, enabling the large language model to process the task to be processed according to the adjusted computing resource allocation. This method allows for the selection of the target scaling operation strategy with the largest service improvement based on the service improvement and execution cost, thus improving the service performance of the large language model while controlling execution costs.

[0084] Please see Figure 2 The second embodiment of this application provides a large language model heterogeneous computing resource allocation scaling decision system 100, including:

[0085] The candidate strategy acquisition module 101 is used to acquire a set of candidate strategies for a corresponding task to be processed; the set of candidate strategies includes multiple candidate scaling operation strategies for the corresponding task to be processed.

[0086] The service improvement acquisition module 102 is used to predict the service improvement of the large language model by executing each of the candidate scaling operation strategies based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed.

[0087] The execution cost acquisition module 103 is used to acquire the execution cost of each of the candidate scaling operation strategies;

[0088] The strategy score acquisition module 104 is used to acquire the strategy score of each of the candidate scaling operation strategies based on the service improvement amount and the execution cost.

[0089] The strategy effectiveness acquisition module 105 is used to determine the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold.

[0090] The resource configuration adjustment module 106 is used to adjust the computing power resource configuration according to the target scaling operation strategy if the target scaling operation strategy is effective, so that the large language model can process the task to be processed according to the adjusted computing power resource configuration.

[0091] It should be noted that the large language model heterogeneous computing power resource allocation scaling decision system 100 provided in the second embodiment of this application is only illustrated by the above-mentioned division of functional modules when executing the large language model heterogeneous computing power resource allocation scaling decision method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the large language model heterogeneous computing power resource allocation scaling decision system 100 provided in the second embodiment of this application and the large language model heterogeneous computing power resource allocation scaling decision method of the first embodiment of this application belong to the same concept, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0092] Please see Figure 3 The large language model heterogeneous computing resource allocation scaling decision system of this application can also be implemented in the form of other functional modules:

[0093] The computing resource allocation scaling decision system includes an SLO monitoring module, a workload analysis module, an economic benefit assessment model (i.e., an SLO-ROI engine), a heterogeneous resource management module, a dynamic scaling controller, and an inference execution module. The functions of each module are as follows:

[0094] The SLO monitoring module is directly connected to the inference execution module and is responsible for collecting performance metrics for each inference request in real time with high accuracy. Core metrics include TTFT and TPOT. Simultaneously, this module summarizes and statistically analyzes macro-level performance metrics such as total request latency, system throughput (req / s), and request queue length. The SLO monitoring module compares the real-time monitored metrics with preset SLO thresholds. When it detects a decline in the SLO achievement rate approaching the default boundary, it generates an alarm and sends a scaling decision trigger signal to the economic benefit assessment model and the heterogeneous resource management module.

[0095] The workload analysis module receives inference requests from users and analyzes them before processing. This module extracts key load characteristics, such as input sequence length, number of concurrent requests, batch size, and the user's expected output sequence length. Based on this, it makes a preliminary estimate of the resource consumption of a single request and predicts the overall system load trend over a future period. The prediction results are then used as input to the economic benefit evaluation model to more accurately assess the performance under different scaling strategies.

[0096] The Heterogeneous Resource Management Module is the system's "brain" and "central nervous system." This module maintains a resource template library that defines all available heterogeneous computing resources, including different models of GPUs (such as NVIDIA A100, H100, Muxi Cloud C500, Ascend 910B), CPUs, dedicated accelerators, etc., and associates detailed specifications (such as memory capacity, computing power, memory bandwidth), cost information (hourly unit price, rated power consumption), applicable inference frameworks (such as TensorRT-LLM, vLLM), and quantization precision (such as FP16, INT8) for each resource. The Heterogeneous Resource Management Module receives the final decision (optimal scaling strategy) from the economic benefit evaluation model, translates it into specific instructions, and drives the dynamic scaling controller to execute them.

[0097] Inference Execution Module: This module actually performs the LLM model inference task. Based on instructions from the Heterogeneous Resource Management Module, it prepares the inference environment on the specified computing instance, loads the model, performs weight warm-up, and then performs pre-filling and decoding operations. During inference, this module generates detailed performance events (such as "First Token Generated," "Next Token Generated") and sends these timestamped events back to the SLO monitoring module as a source of performance data collection.

[0098] Dynamic scaling controller: The execution-level controller, typically implemented based on an existing cloud-native orchestration system (such as Kubernetes). It receives commands from the heterogeneous resource management module (e.g., "add 2 B-type GPU instances to service A" or "migrate service C from D-type GPU to E-type GPU") and is responsible for calling the underlying IaaS / PaaS interfaces to complete specific operations such as instance creation, destruction, network configuration, and rolling updates of services.

[0099] Economic Benefit Evaluation Model (SLO-ROI Engine): The core innovative component of this invention, its internal structure is as follows Figure 4 As shown, the model receives real-time performance data from the SLO monitoring module, load characteristics from the workload analysis module, and cost parameters from the heterogeneous resource management module. It then quantitatively evaluates candidate scaling strategies and outputs decision recommendations based on the SLO-ROI metric.

[0100] To further illustrate the technical solution of this application, the following examples are used for explanation:

[0101] Example 1: Real-time chat application

[0102] Business characteristics: Users are extremely sensitive to first response time (TTFT) and have strict SLO requirements (e.g., TTFT < 1 second). Users cannot tolerate significant lag or service interruption during the conversation.

[0103] Decision-making process: When the system detects a sudden surge in concurrent user requests, a decision-making process is triggered. Candidate strategies include "increasing the number of copies of existing GPU instances" (horizontal scaling) and "switching to a new, more powerful GPU instance" (heterogeneous scaling). For heterogeneous scaling, although the new GPU offers better steady-state performance, it may bring greater costs. However, its transition costs Very high – Service switching requires restarting the service, and even a few seconds of interruption can lead to a large number of real-time session interruptions or a severe degradation in experience. The corresponding Apdex degradation cost is judged to be extremely high by the model. In contrast, horizontal scaling can seamlessly expand capacity by adding instance replicas, with transition costs that are almost zero.

[0104] Decision Outcome: Based on SLO-ROI calculations, in this scenario, the score for horizontal scaling operations is almost always higher than that for heterogeneous scaling operations. Therefore, the system prioritizes adding replicas for expansion. When the load decreases, the system safely reclaims excess instances based on the hysteresis threshold, provided the ROI of the scaling-down scheme is below θdown (the lower limit of the hysteresis threshold). In other words, in scenarios with high real-time requirements, the method of this invention tends to choose the horizontal scaling path to ensure service continuity.

[0105] Example 2: Long document summary batch processing task

[0106] Service characteristics: Users are sensitive to the total task completion time, but not to the time-to-flight delay (TTFT) of a single task. The service can tolerate brief, planned interruptions.

[0107] Decision-making process: A decision-making process is triggered when the system receives a large number of offline document summarization tasks. Candidate strategies include "increasing the number of replicas of existing GPU instances" (horizontal scaling) and "switching to GPU instances with larger memory and higher throughput" (heterogeneous scaling). For heterogeneous scaling, since the business is not sensitive to brief interruptions, the transition cost is low. The cost of service degradation is very low. Furthermore, the new high-performance GPUs can significantly improve batch processing throughput, bringing huge benefits. In contrast, while horizontal scaling can also improve overall processing capacity, it is limited by the performance bottleneck of a single instance. Growth is approximately linear, and cost growth is also approximately linear.

[0108] Decision Result: In this scenario, the heterogeneous switching strategy typically exhibits a significantly higher benefit / cost ratio than horizontal scaling, meaning the score for horizontal scaling is lower than that for heterogeneous scaling. Therefore, the system will choose to perform heterogeneous switching to achieve better long-term capital efficiency. As can be seen from the above embodiments, the method of the present invention can adaptively make the most economical and reasonable choice among different scaling paths based on the characteristics of the business load, taking into account both the stability of SLO and long-term cost-effectiveness.

[0109] The third embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the large language model heterogeneous computing resource allocation scaling decision method as described above.

[0110] The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function selected in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for scaling decision-making in heterogeneous computing resource allocation for large language models, characterized in that, include: Obtain a set of candidate strategies for the corresponding task to be processed; the set of candidate strategies includes multiple candidate scaling operation strategies for the corresponding task to be processed. Based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed, the service improvement of the large language model by executing each of the candidate scaling operation strategies is predicted. Obtain the execution cost of each of the candidate scaling operation strategies; Based on the service improvement amount and the execution cost, obtain the strategy score for each of the candidate scaling operation strategies; The effectiveness of the target scaling operation strategy corresponding to the highest strategy score is determined based on the highest strategy score and the preset hysteresis threshold. If the target scaling operation strategy is effective, the computing resource configuration is adjusted according to the target scaling operation strategy, so that the large language model processes the task to be processed according to the adjusted computing resource configuration.

2. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, Before the step of obtaining the candidate strategy set for the corresponding task to be processed, the following steps are included: If the real-time service metric parameter of the large language model is greater than or equal to a preset metric threshold, or if the predicted load change of the large language model in processing the task to be processed is greater than a preset change threshold, the candidate strategy set is obtained; wherein, the operation types of the multiple candidate scaling operation strategies in the candidate strategy set include horizontal scaling operation and heterogeneous scaling operation.

3. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, The step of predicting the service improvement of the large language model by executing each candidate scaling operation strategy based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each candidate scaling operation strategy, and the load characteristics of the task to be processed includes: The large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed are input into the service prediction model to obtain the prediction service data of the large language model corresponding to each of the candidate scaling operation strategies. Based on the current service data of the large language model and the prediction service data of each, the service improvement of the large language model by executing each candidate scaling operation strategy is obtained.

4. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, The step of obtaining the execution cost of each of the candidate scaling operation strategies includes: The direct cost of the corresponding candidate scaling operation strategy is obtained based on the rental cost and energy consumption cost of adding or replacing instance replicas. Based on instance overlap cost, service degradation cost, and migration overhead cost, the transition cost of the corresponding candidate scaling operation strategy is obtained; The execution cost is obtained based on the direct cost and the transition cost.

5. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, The step of obtaining the strategy score for each candidate scaling operation strategy based on the service improvement amount and the execution cost includes: The strategy score is obtained by dividing the service improvement amount of each candidate scaling operation strategy by the corresponding execution cost.

6. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, The preset hysteresis threshold includes an upper hysteresis threshold; If the target scaling operation strategy is effective, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the task to be processed according to the adjusted computing resource configuration, includes: If the target scaling operation strategy is effective and the corresponding strategy score is greater than the upper limit hysteresis threshold, the computing resource configuration is adjusted according to the expansion / upward heterogeneous switching strategy.

7. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 6, characterized in that, The preset hysteresis threshold includes a lower hysteresis threshold; the lower hysteresis threshold is less than the upper hysteresis threshold. If the target scaling operation strategy is effective, the step of adjusting the computing resource configuration according to the target scaling operation strategy, so that the large language model processes the task to be processed according to the adjusted computing resource configuration, includes: If the target scaling operation strategy is effective and the corresponding strategy score is less than the lower limit hysteresis threshold, the computing resource configuration is adjusted according to the scaling down / down heterogeneous switching strategy.

8. The method for scalable decision-making of heterogeneous computing power resource allocation for large language models according to claim 1, characterized in that, The preset hysteresis threshold includes an upper hysteresis threshold and a lower hysteresis threshold; wherein, the lower hysteresis threshold is less than the upper hysteresis threshold; After determining the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold, the following steps are included: If the target scaling operation strategy is less than or equal to the upper limit hysteresis threshold and greater than or equal to the lower limit hysteresis threshold, the current computing resource configuration of the large language model is maintained.

9. A scalable decision-making system for heterogeneous computing power resource allocation for large language models, characterized in that, include: The candidate strategy acquisition module is used to acquire a set of candidate strategies for a corresponding task to be processed; the set of candidate strategies includes multiple candidate scaling operation strategies for the corresponding task to be processed. The service improvement acquisition module is used to predict the service improvement of the large language model by executing each of the candidate scaling operation strategies based on the model size of the large language model corresponding to the task to be processed, the resource configuration corresponding to each of the candidate scaling operation strategies, and the load characteristics of the task to be processed. The execution cost acquisition module is used to acquire the execution cost of each of the candidate scaling operation strategies; The strategy score acquisition module is used to acquire the strategy score of each of the candidate scaling operation strategies based on the service improvement amount and the execution cost. The strategy effectiveness acquisition module is used to determine the effectiveness of the target scaling operation strategy corresponding to the highest strategy score based on the highest strategy score and the preset hysteresis threshold. The resource configuration adjustment module is used to adjust the computing power resource configuration according to the target scaling operation strategy if the target scaling operation strategy is effective, so that the large language model can process the task to be processed according to the adjusted computing power resource configuration.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the large language model heterogeneous computing power resource allocation scaling decision method as described in any one of claims 1 to 8.

Images