A vGPU-based SLO-aware model scaling method and system in a computing power network

Abstract

The application provides a vGPU-based SLO-aware model expansion and contraction method and system in a computing power network, which comprises the following steps: a Prometheus-based monitoring system is designed and deployed, load indexes on the GPU side and the instance side in a Kubernetes cluster are collected by using the monitoring system, and monitoring indexes are obtained; an output length predictor based on a fine-tuned BERT-base lightweight model is constructed, and a joint prediction model of first Token delay, Token-to-Token delay and end-to-end delay is constructed in combination with the real-time state of the system, so that delay prediction results are obtained; based on the monitoring indexes, a two-stage scheduling strategy is used to realize two-level load balancing of the GPU level and the instance level; based on the monitoring indexes and the delay prediction results, an SLO-aware elastic expansion and contraction mechanism is designed, and an expansion and contraction strategy is dynamically selected according to an SLO violation risk. The application realizes the optimized balance between operation cost and service performance.

Description

Technical Field

[0001] This invention belongs to the fields of cloud-native and artificial intelligence technologies, specifically relating to a scaling method and system for a vGPU-based SLO-aware model in computing power networks. In scenarios based on virtual GPU (vGPU) technology, with Service Level Objective (SLO) as the core constraint, and focusing on latency metrics such as First Token Latency (TTFT) and Inter-Token Latency (TBT), an optimal resource allocation strategy and reliability assurance mechanism are designed to achieve efficient on-demand scheduling of computing power, reduce GPU fragmentation, improve GPU resource utilization efficiency, and lower the overall system operating cost. Background Technology

[0002] In recent years, applications driven by deep learning models such as Large Language Models (LLM) have grown rapidly. Due to their enormous computational demands, these models are typically deployed in Model-as-a-Service (MAAS) systems. These systems manage clusters of accelerators such as GPUs and configure an appropriate number of service instances, including GPUs, for each deployed model. Automatic scaling of models is a promising solution. Through automatic scaling, MAAS systems can configure the number of instances based solely on the relatively stable average demand of each service model over a long period, effectively improving resource utilization. When sudden traffic surges, the system automatically adds new instances, preventing request queuing due to insufficient instance replicas and thus avoiding SLO (Solution Time Limit) violations.

[0003] However, LLM service load is significantly unpredictable, posing a serious challenge to efficient resource scaling. This unpredictability manifests itself in two main aspects: First, the unpredictability of request length. LLM requests can vary by orders of magnitude in the number of input and output tokens, ranging from a short question-and-answer session to generating tens of thousands of tokens, resulting in drastically different computational requirements. Second, the unpredictability of requests over time. User traffic often exhibits dramatic fluctuations and sudden bursts, making reliable predictions based on historical data difficult. The interplay of these two uncertainties—request length and temporal dynamics—makes scaling strategies relying solely on traditional threshold metrics insufficient to achieve an effective balance between resource utilization efficiency and service SLO guarantees.

[0004] Existing scaling methods typically trigger scaling based on thresholds set according to GPU utilization, request volume, and concurrency, which is a reactive, post-event adjustment that is prone to SLO (Solution Time Limit) violations. Furthermore, resource allocation generally employs a full-card strategy, where a single instance occupies one or more GPU cards, fragmenting available computing power into numerous seemingly idle but actually difficult-to-load GPU memory fragments, resulting in low GPU resource utilization.

[0005] To address the challenges of dynamic workloads in MAAS systems, this invention proposes a vGPU-based SLO-aware scaling method for computing power networks. This method employs a two-stage scheduling mechanism of "card selection first, instance selection second" to achieve dual-level load balancing at the GPU and instance levels; it establishes an SLO-aware scaling strategy, selecting different scaling methods based on SLO violation risks; and it adopts a full-card-first scaling-down mechanism to maximize the release of available computing power. By significantly improving resource utilization and reducing GPU fragmentation while ensuring service quality, it achieves an optimized balance between system operating costs and service performance. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a scaling method and system for a vGPU-based SLO-aware model in a computing power network. The aim is to significantly reduce GPU fragmentation and improve GPU utilization while minimizing resource overhead costs, based on the constantly changing load of LLM services in the MAAS system, while ensuring that TTFT and TBT meet SLO requirements.

[0007] To achieve the above objectives, the present invention provides the following solution: A scaling method for a vGPU-based SLO-aware model in a computing power network, the method comprising: Design and deploy a Prometheus-based monitoring system to collect load metrics on the GPU side and instance side of the Kubernetes cluster and obtain monitoring metrics. Construct an output length predictor based on a fine-tuned BERT-base lightweight model, and combine it with the real-time system status to build a joint prediction model for the first token delay, inter-token delay, and end-to-end delay, and obtain the delay prediction results. Based on monitoring metrics, a two-stage scheduling strategy is used for request distribution. Based on monitoring metrics and latency prediction results, an SLO-aware elastic scaling mechanism is designed. Combining request distribution results, the scaling strategy is dynamically selected according to the SLO violation risk.

[0008] Preferably, the two-stage scheduling strategy includes: a GPU card selection strategy and an instance selection strategy; GPU card selection strategy: Calculate the overall GPU load index and select the GPU card with the lowest overall GPU load index as the optimal physical GPU card for new requests; Instance selection strategy: Based on the optimal physical GPU card, calculate the overall scheduling cost, select the instance with the lowest overall scheduling cost as the optimal instance, and schedule new requests to the optimal instance on the optimal physical GPU card for inference.

[0009] Preferably, the method for calculating the GPU overall load index includes: ; in, This refers to the overall GPU load index. This represents the normalized GPU computational utilization, and , For GPU computing utilization; Indicates the video memory usage rate; This represents the normalized bandwidth utilization, and , For bandwidth utilization; the weighting coefficients satisfy... .

[0010] Preferably, the method for calculating the overall scheduling cost includes: ; in, To cover overall scheduling costs; For normalized queuing load costs, , For example At any moment The number of queued requests, This is the threshold for the number of people queuing; For normalized operating load costs, , For example At any moment Number of requests being processed For example At any moment The number of concurrent requests; Normalized arrival rate pressure cost, , Representation of instances At any moment Request delivery rate This is the maximum allowed arrival rate threshold; For normalized KV cache pressure cost, , Representation of instances At any moment The percentage of KV cache usage; For TTFT performance cost, To represent an instance At any moment The first token delay of P90 For TTFT's service level objectives, TTFT prioritizes Token latency; For TBT performance cost, Representation of instances At any moment P90 Token latency, The service level target for TBT is the inter-Token latency; For end-to-end performance cost, Representation of instances At any moment P90 end-to-end latency, The service level objective is to achieve end-to-end latency; the weighting coefficients satisfy... .

[0011] Preferably, a method for designing an SLO-aware elastic scaling mechanism based on monitoring metrics and latency prediction results, and dynamically selecting scaling strategies according to SLO violation risks based on request distribution results, includes: Based on monitoring indicators and latency prediction results, a scaling-up / scaling trigger mechanism is established; Based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling up is triggered, a risk-based SLO-aware scaling-up strategy is designed, and differentiated scaling up is implemented according to the SLO violation risk level. Based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling-up is triggered, a cost-priority scaling-up strategy is designed according to the principle of "quickly releasing the entire GPU card", prioritizing the reclamation of GPU card resources sparsely populated in the instance.

[0012] Preferably, the methods for establishing a scaling-up / scaling trigger mechanism based on monitoring indicators and latency prediction results include: Establish an early warning system for SLO (Solution Loading) violation risks based on latency prediction results, triggering capacity expansion assessments; Establish capacity pressure alerts based on real-time instance load status to trigger expansion assessments; Establish a cost-optimized scaling-down trigger mechanism based on resource utilization efficiency to trigger scaling-down assessment.

[0013] Preferably, based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling is triggered, a risk-based SLO-aware scaling strategy is designed. The methods for implementing differentiated scaling based on the SLO violation risk level include: The SLO violation risk level is comprehensively assessed using multiple dimensions of indicators, and the calculation formula is as follows: ; in, SLO violation risk level; and These are the predicted values ​​for TTFT and TBT, respectively. Differentiated expansion based on SLO violation risk level: like The scenario is assessed as low-risk. Under low-risk scenarios, when the existing GPU card's memory availability and overall load index meet the following conditions, a same-card expansion strategy is adopted. This involves using vGPU technology to create new instances on the existing GPU card, achieving low-cost capacity expansion: and ; in, This refers to the remaining video memory of the existing GPU card. The amount of video memory required to deploy a single new instance; like If an application is assessed as a high-risk scenario, or if the existing GPU card's memory is insufficient, a new GPU card will be selected from the set of available GPU cards for capacity expansion. The new GPU card must meet the following conditions: and and ; in, This represents the total video memory capacity of the newly added GPU card.

[0014] Preferably, based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling-up is triggered, a cost-priority scaling-up strategy is designed according to the principle of "quickly releasing the entire GPU card," and methods for prioritizing the reclamation of sparse GPU card resources in instances include: A two-layer selection mechanism is designed based on the principle of "quickly releasing the entire GPU card": During the GPU card selection phase, instances are selected for sparse GPU card deployment by calculating a scaling-down priority score: ; in, Rate the priority; Indicates the number of instances deployed on the GPU card; Maximum instance capacity; weighting coefficients satisfy ; Select the GPU card with the highest priority score for scaling down as the target GPU card for scaling down. During the instance selection phase, a multi-metric evaluation method is used to calculate the scaling-down score for each instance on the target scaling-down GPU card to select the scaling-down target: ; in, Score for reduced capacity; The minimum effective request arrival rate threshold is preset for the system; the weighting coefficients satisfy... ; The instance with the highest scaling-up score is selected as the scaling-up target to achieve rapid recycling and consolidation of GPU resources.

[0015] The present invention also provides a scaling system for a vGPU-based SLO perception model in a computing power network. The system is used to implement the aforementioned method and includes: a data acquisition module, a prediction module, a scheduling module, and a scaling module. The data collection module is used to design and deploy a Prometheus-based monitoring system. The system collects load metrics from the GPU side and instance side of the Kubernetes cluster to obtain monitoring metrics. The prediction module is used to construct an output length predictor based on a fine-tuned BERT-base lightweight model, and combine it with the real-time system status to build a joint prediction model for the first token delay, inter-token delay, and end-to-end delay, and obtain the delay prediction results. The scheduling module is used to distribute requests based on monitoring metrics and employs a two-stage scheduling strategy. The scaling module is used to design an SLO-aware elastic scaling mechanism based on monitoring metrics and latency prediction results. It dynamically selects scaling strategies based on SLO violation risks, combined with request distribution results.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention discloses a vGPU-based SLO-aware model scaling method and system for computing power networks. It can handle the online inference request load of large language models (LLMs) on Model-as-a-Service (MAAS) platforms. While ensuring that the first-to-token latency (TTFT) and inter-token latency (TBT) meet the Service Level Objective (SLO), it leverages the advantages of virtual GPUs (vGPUs) and combines scheduling and scaling strategies to reduce GPU fragmentation, improve GPU utilization, and minimize resource overhead costs. First, a latency prediction framework based on a lightweight BERT model is used to achieve forward prediction of request load. Second, a two-stage scheduling strategy of "card selection first, instance selection later" is designed to achieve dual-level load balancing at the GPU and instance levels. Finally, an SLO-aware elastic scaling mechanism is established, dynamically selecting scaling strategies based on real-time risk assessment. In low-risk situations, a same-card scaling strategy is used to save resources; in high-risk situations, a strategy of deploying new GPU cards is used to avoid SLO violations caused by model instance resource contention. This is combined with a cost-priority scaling strategy, prioritizing the release of sparsely populated GPU card resources. This method effectively improves GPU utilization and reduces GPU resource fragmentation while ensuring service quality, thus achieving an optimal balance between operating costs and service performance. Attached Figure Description

[0017] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the scaling method for the SLO perception model based on vGPU in the computing power network according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the system architecture of an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Example 1 This invention provides a scaling method for a vGPU-based SLO-aware model in a computing power network, comprising: A scaling method for a vGPU-based SLO-aware model in a computing power network, the method comprising: Design and deploy a Prometheus-based monitoring system to collect load metrics on the GPU side and instance side of the Kubernetes cluster and obtain monitoring metrics. Construct an output length predictor based on a fine-tuned BERT-base lightweight model, and combine it with the real-time system status to build a joint prediction model for the first token delay, inter-token delay, and end-to-end delay, and obtain the delay prediction results. Based on monitoring metrics, a two-stage scheduling strategy is used for request distribution. Based on monitoring metrics and latency prediction results, an SLO-aware elastic scaling mechanism is designed. Combining request distribution results, the scaling strategy is dynamically selected according to the SLO violation risk.

[0022] like Figure 1 , Figure 2 As shown, the specific implementation process of the present invention is as follows: Step 1: Pull GPU and instance load metrics based on Prometheus, design and deploy a monitoring system based on Prometheus. Kubernetes is the underlying infrastructure and implementation platform for resource management and scheduling. Build a MAAS system based on Kubernetes and its ecosystem, and use Prometheus to collect load metrics on the GPU side and instance side in the Kubernetes cluster.

[0023] 1) Deploy the Prometheus monitoring component. Deploy the Prometheus monitoring stack in the Kubernetes cluster and configure it to automatically capture monitoring metrics through service discovery. This includes: load metrics related to the vLLM inference service, and GPU hardware metrics obtained through components such as DCGM Exporter.

[0024] 2) GPU and instance metric data collection. In the Kubernetes cluster, PromQL is used at fixed sampling intervals. Regularly collect load monitoring metrics from both the "instance-physical GPU" layers to construct a unified system status view for request scheduling and scaling decisions. Define the set of running instances as... N is the number of instances, and the physical GPU set is... M represents the number of physical GPUs.

[0025] Each timestamp The instance-side indicator sequence is defined as: in, For example At any moment Request delivery rate For example At any moment Number of requests running (being processed) For example At any moment Number of queued requests For example At any moment Concurrent request volume (i.e., instance) At any moment Maximum concurrent request processing capacity) For example At any moment KV Cache utilization percentage; latency metrics , and These are the P90 first-token latency, P90 inter-token latency, and P90 end-to-end latency, calculated using the P90 quantile value.

[0026] Each timestamp The GPU-side metric sequence is defined as follows: in, For GPU computing utilization, and This refers to the amount of video memory used and the amount of free memory. For bandwidth utilization, This represents the video memory usage rate.

[0027] The unified view of Kubernetes cluster status is represented as follows: in, For instance-side metrics set, This is a set of metrics for the GPU side.

[0028] By establishing the association between instances and physical GPUs through metadata such as Pod tags and GPU indexes / UUIDs, and synchronously outputting global resource views and instance-level detailed views, a complete real-time data foundation is provided for the scheduling strategy of "selecting the GPU first and then the instance" and scaling decisions.

[0029] Step 2: Constructing a joint predictor of "output length-delay" based on a fine-tuned BERT-base lightweight model. This step aims to build a joint prediction framework that can predict the output length of a request and combine it with the real-time state of the system to achieve advance prediction of TTFT, TBT and end-to-end delay.

[0030] This step addresses the unpredictability and latency fluctuations of LLM inference request output length. The framework first predicts the request output length by fine-tuning the BERT-base model, and then, combined with the real-time system status, establishes an accurate latency prediction model, providing forward-looking guidance for subsequent scheduling and scaling decisions.

[0031] 1) Construct an output length predictor based on a fine-tuned BERT-base lightweight model, which predicts the "request text → output length". First, construct training samples based on real question-and-answer logs (fields must include: input text, number of input tokens). Actual number of output tokens The preprocessed text is used for fine-tuning training. BERT-base is used as the text encoder, employing a regression task, and the L1 loss function is used for lightweight fine-tuning. The predicted output length is then calculated. This serves as the direct input for time delay estimation.

[0032] 2) Latency Prediction (TTFT, TBT, End-to-End). Based on the real-time GPU and instance load metrics obtained from Prometheus in step 1 and the predicted output length obtained using the BERT-based lightweight model in the previous step. We construct an online estimation link, which involves building a joint prediction model for the latency of the first token, the latency between tokens, and the end-to-end latency, performing latency decomposition and estimation, and obtaining latency prediction results.

[0033] First, use the TBT (P90) of the nearest window of the instance to calculate the throughput. , This represents the 90th percentile of the inter-Token latency for the instance within the most recent time window; if missing, it is calibrated online using lightweight regression. Therefore, .

[0034] Then, based on the two key components of queuing and prefilling, we obtain... .

[0035] Among them, linear approximation For pre-filled delay, where , It is a constant.

[0036] Therefore, according to and The estimated end-to-end latency is : This step enables proactive prediction of latency indicators by fully utilizing the characteristics of the request and the real-time status of the system, providing a core basis for subsequent SLO assurance and resource decisions.

[0037] Based on GPU and instance load metrics obtained from Prometheus within the Kubernetes cluster and the output length predicted by the BERT-based lightweight model, TTFT, TBT, and end-to-end latency are predicted. This provides data support for subsequent SLO violation risk assessment and serves as input signals for subsequent request scheduling and scaling decisions.

[0038] Step 3: Request Scheduling Strategy Design. To fully leverage the fine-grained resource advantages of vGPUs and improve GPU resource utilization, this method employs a two-stage scheduling strategy for request distribution: first, the most suitable physical GPU card is selected globally within the system; then, the most suitable instance is selected from that card to receive inference requests. This process forms a scheduling decision mechanism of "selecting the GPU card first, then the instance," achieving two-level load balancing at the GPU and instance levels. This effectively avoids local hotspots, improves overall system throughput, and reduces the risk of SLO violations.

[0039] The two-stage scheduling strategy includes: GPU card selection strategy and instance selection strategy; 1) GPU card selection strategy: Calculate the overall GPU load index and select the GPU card with the lowest overall GPU load index as the optimal physical GPU card for new requests; The GPU card selection strategy is to choose the physical GPU card with the lightest overall load and the most abundant resources for each new request, ensuring that this card can provide stable computing power support for all instances on it. For physical GPU ensembles... Each GPU card in At any moment Calculate the "GPU Overall Load Index" The higher the index, the heavier the load, and the less suitable it is to be selected.

[0040] Specifically, methods for calculating the overall GPU load index include: in, This represents the normalized GPU computational utilization, and , For GPU computing utilization; Indicates the video memory usage rate; This represents the normalized bandwidth utilization, and , For bandwidth utilization; the weighting coefficients satisfy... These coefficients need to be optimized based on workload characteristics. For memory-intensive workloads like LLM, this invention sets... The weight is the highest.

[0041] choose GPU card with the lowest value It is the optimal physical GPU card.

[0042] 2) Instance selection strategy: Based on the optimal physical GPU card, calculate the overall scheduling cost, select the instance with the lowest overall scheduling cost as the optimal instance, and schedule new requests to the optimal instance on the optimal physical GPU card for inference.

[0043] Instance selection strategy on the selected target GPU card From its set of running instances, the instance most likely to quickly process new requests is selected. (For GPUs) Each instance on At any moment Calculate a "comprehensive scheduling cost". This cost takes into account the instance's current load status, historical performance, and resource usage.

[0044] in, For normalized queuing load costs, , For example At any moment The number of queued requests, This is the threshold for the number of people queuing; For normalized operating load costs, , For example At any moment Number of requests being processed For example Maximum concurrent request processing capacity; Normalized arrival rate pressure cost, , Representation of instances At any moment Request delivery rate This is the maximum allowed arrival rate threshold; For normalized KV cache pressure cost, , Representation of instances At any moment The percentage of KV cache usage; For TTFT performance cost, To represent an instance At any moment The first token delay of P90 For TTFT's service level objectives, TTFT prioritizes Token latency; For TBT performance cost, Representation of instances At any moment P90Token latency, The service level target for TBT is the inter-Token latency; For end-to-end performance cost, Representation of instances At any moment P90 end-to-end latency, The service level objective is to achieve end-to-end latency; the weighting coefficients satisfy... .

[0045] choose The instance with the smallest value This is the optimal instance. Ultimately, requests will be scheduled to the GPU. Examples on To reason.

[0046] Step 4: SLO-Aware Scaling / Scaling Strategy Design. Based on the monitoring metrics obtained in Step 1 and the latency prediction results obtained in Step 2, this step designs a proactive SLO-aware elastic scaling / scaling mechanism to maximize resource utilization and reduce overall operating costs while ensuring service quality. This includes the following steps: 1) Establish a scaling-up / scaling trigger mechanism based on monitoring indicators and latency prediction results.

[0047] Based on the monitoring indicators in step 1 and the latency prediction results in step 2, multi-dimensional and forward-looking scaling-up and scaling-down trigger conditions are established.

[0048] First, establish an SLO violation risk warning system based on latency prediction results to trigger a capacity expansion assessment: when Exceed 70% of the threshold or Exceed When the performance reaches 70%, a capacity expansion assessment is triggered. Simultaneously, a historical performance baseline trigger condition is set; when the actual monitored performance reaches 70%, a capacity expansion assessment is triggered. Exceeding its 80% of the threshold, or Exceeding its When the threshold reaches 80%, a capacity expansion assessment is also triggered. For the service level target of TTFT, For TBT's service level objectives.

[0049] Secondly, establish capacity pressure alerts based on the real-time load status of instances to trigger expansion assessments: When the instance's KV cache utilization More than 85%, or request delivery rate Exceeding the preset maximum threshold 80%, or the number of requests currently in operation. Exceeding the instance's concurrency limit When the capacity reaches 70%, any condition met will trigger a capacity expansion assessment to address potential performance bottlenecks.

[0050] Finally, establish a cost-optimized scaling-down trigger mechanism based on resource utilization efficiency to trigger scaling-down evaluation: A scaling-down assessment is triggered when the instance load simultaneously meets the following three conditions: KV Cache utilization. The percentage is below 30%, and the request delivery rate is below the preset minimum threshold. 50% of the total, and the number of running requests Below the instance concurrency limit 30%.

[0051] Through the combination of the above-mentioned multi-level triggering mechanisms, this invention can proactively intervene before SLO violations occur, realizing the transformation from passive response to proactive prevention.

[0052] 2) Based on the scaling-up / scaling trigger mechanism and the request distribution results in step 3, after scaling up is triggered, a risk-based SLO-aware scaling-up strategy is designed, and differentiated scaling up is implemented according to the SLO violation risk level.

[0053] This step, based on the SLO violation risk level assessment results, adopts a differentiated expansion plan to achieve a balance between precise resource allocation and cost control.

[0054] By comprehensively assessing the SLO violation risk level through multi-dimensional indicators, a risk classification system is established to calculate the SLO violation risk level. : in, and These are the predicted values ​​for TTFT and TBT, respectively. Differentiated expansion based on SLO violation risk level: like The scenario is assessed as low-risk. Under low-risk scenarios, when the existing GPU card (target GPU card) has sufficient memory and the overall load index meets the following conditions, a same-card expansion strategy is adopted. This involves using vGPU technology to create new instances on the existing GPU card, achieving low-cost capacity expansion: and in, This refers to the remaining video memory of the existing GPU card. The amount of video memory required to deploy a single new instance; like The scenario is assessed as high-risk, and the existing GPU card has insufficient video memory in high-risk scenarios. When ), from the set of idle GPU cards that are not currently deployed with any instances ( Select the comprehensive load index. The smallest card will be used as a new GPU card for capacity expansion. The new GPU card is: in, This represents the set of GPU cards in the cluster that have never had any instances deployed.

[0055] The new GPU card must meet the following conditions: and and in, The total video memory capacity and computational utilization of the newly added GPU card. And video memory usage This avoids contention between instances and ensures SLO.

[0056] 3) Based on the scaling-up / scaling trigger mechanism and the request distribution results from step 3, after scaling-up is triggered, a cost-priority scaling-up strategy is designed according to the principle of "quickly releasing the entire GPU card" to prioritize the reclamation of GPU card resources sparse in the instance.

[0057] A two-layer selection mechanism is designed based on the principle of "quickly releasing the entire GPU card": During the GPU card selection phase, GPU cards with sparse instance deployment and low overall utilization are selected for scaling down by calculating a scaling-down priority score: in, Rate the priority; Indicates the number of instances deployed on the GPU card; Maximum instance capacity; weighting coefficients satisfy ,and ; The GPU card with the highest scaling priority score is selected as the target scaling GPU card. After determining the target scaling GPU card, the instance with the lightest load on that card is selected for scaling down, thus entering the instance selection stage.

[0058] During the instance selection phase, a multi-metric evaluation method is used to calculate the scaling-down score for each instance on the target scaling-down GPU card to select the scaling-down target: in, Score for reduced capacity; The minimum effective request arrival rate threshold is preset for the system; the weighting coefficients satisfy... ; The instance with the highest scaling-up score is selected as the scaling-up target to achieve rapid recycling and consolidation of GPU resources.

[0059] This strategy effectively releases and consolidates computing resources by focusing on reclaiming low-load instances and consolidating sparse GPU cards. While ensuring service stability, it also provides the system with the resilience to cope with future load growth.

[0060] In summary, this invention discloses a vGPU-based SLO-aware scaling method for computing power networks. This method can handle the online inference request load of large language models (LLMs) on Model-as-a-Service (MAAS) platforms. While ensuring that the first-to-token latency (TTFT) and inter-token latency (TBT) meet the Service Level Objective (SLO), it leverages the advantages of virtual GPUs (vGPUs) and combines scheduling and scaling strategies to reduce GPU fragmentation, improve GPU utilization, and minimize resource overhead costs. First, a latency prediction framework based on a lightweight BERT model is used to achieve forward prediction of request load. Second, a two-stage scheduling strategy of "card selection first, instance selection later" is designed to achieve dual-level load balancing at the GPU and instance levels. Finally, an SLO-aware elastic scaling mechanism is established, dynamically selecting scaling strategies based on real-time risk assessment. In low-risk situations, a same-card scaling strategy is used to save resources; in high-risk situations, a strategy of deploying new GPU cards is used to avoid SLO violations caused by instance resource contention. This is combined with a cost-priority scaling strategy to prioritize the release of sparse GPU card resources in instances. This method effectively improves GPU utilization and reduces GPU resource fragmentation while ensuring service quality, thus achieving an optimal balance between operating costs and service performance.

[0061] Example 2 Based on the same inventive concept, the present invention also provides a scaling system for a vGPU-based SLO perception model in a computing power network, used to implement the methods described in the foregoing embodiments. The system includes: a data acquisition module, a prediction module, a scheduling module, and a scaling module. The data collection module is used to design and deploy a Prometheus-based monitoring system. The system collects load metrics from the GPU side and instance side of the Kubernetes cluster to obtain monitoring metrics. The prediction module is used to construct an output length predictor based on a fine-tuned BERT-base lightweight model, and combine it with the real-time system status to build a joint prediction model for the first token delay, inter-token delay, and end-to-end delay, and obtain the delay prediction results. The scheduling module is used to distribute requests based on monitoring metrics and employs a two-stage scheduling strategy. The scaling module is used to design an SLO-aware elastic scaling mechanism based on monitoring metrics and latency prediction results. It dynamically selects scaling strategies based on SLO violation risks, combined with request distribution results.

[0062] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A scaling method for a vGPU-based SLO-aware model in a computing power network, characterized in that, The method includes: Design and deploy a Prometheus-based monitoring system to collect load metrics on the GPU side and instance side of the Kubernetes cluster and obtain monitoring metrics. An output length predictor based on a fine-tuned BERT-based lightweight model is constructed. Combined with the real-time system status, a joint prediction model is built for the first token delay, inter-token delay, and end-to-end delay to obtain delay prediction results, including: A lightweight BERT-base model is used to build an output length predictor that converts request text to output length. First, training samples are constructed based on real question-and-answer logs and preprocessed for fine-tuning. Then, BERT-base is used as the text encoder, employing a regression task and an L1 loss function for lightweight fine-tuning to predict the output length. , as a direct input for time delay estimation; First Token Latency (TTFT), Inter-Token Latency (TBT), and End-to-End Latency Prediction: Based on real-time GPU and instance load metrics obtained from Prometheus within the Kubernetes cluster and the predicted output length. To construct an online and implementable estimation process, a joint prediction model is built for the latency of the first token, the latency between tokens, and the end-to-end latency. This model is then used for latency decomposition and estimation to obtain the latency prediction results. First, use the TBT of the nearest window instance to calculate the throughput. , This represents the 90th percentile of the inter-Token latency for the instance within the most recent time window; if missing, it is calibrated online using lightweight regression. Therefore, ,in, For GPU computing utilization, This refers to the video memory usage rate. For example KV Cache utilization percentage For example The number of concurrent requests; Then, based on the two key components of queuing and prefilling, we obtain... : ； in, For example Number of queued requests For the number of input tokens, a linear approximation For pre-filled delay, where , It is a constant; Therefore, according to and The estimated end-to-end latency is : ； Based on monitoring metrics, a two-stage scheduling strategy is adopted for request distribution. The two-stage scheduling strategy includes: GPU card selection strategy and instance selection strategy. GPU card selection strategy: Calculate the overall GPU load index and select the GPU card with the lowest overall GPU load index as the optimal physical GPU card for new requests; Instance selection strategy: Based on the optimal physical GPU card, calculate the overall scheduling cost, select the instance with the lowest overall scheduling cost as the optimal instance, and schedule new requests to the optimal instance on the optimal physical GPU card for inference. Based on monitoring metrics and latency prediction results, an SLO-aware elastic scaling mechanism is designed. Combining request distribution results, a scaling strategy is dynamically selected based on the SLO violation risk, including: Establish a scaling-up / scaling trigger mechanism based on monitoring indicators and latency prediction results; Based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling is triggered, a risk-based SLO-aware scaling strategy is designed, implementing differentiated scaling based on the SLO violation risk level: The SLO violation risk level is comprehensively assessed using multiple dimensions of indicators, and the calculation formula is as follows: ； in, SLO violation risk level, and These are the predicted values ​​for TTFT and TBT, respectively. To represent an instance At any moment The first token delay of P90 Representation of instances At any moment P90Token latency, For the service level target of TTFT, For TBT's service level objectives; Differentiated expansion based on SLO violation risk level: like The scenario is assessed as low-risk. Under low-risk scenarios, when the existing GPU card's memory availability and overall load index meet the following conditions, a same-card expansion strategy is adopted. This involves using vGPU technology to create new instances on the existing GPU card, achieving low-cost capacity expansion: and ; in, This refers to the remaining video memory of the existing GPU card. The amount of video memory required to deploy a single new instance, For GPU card At any moment GPU overall load index; like If an application is assessed as a high-risk scenario, or if the existing GPU card's memory is insufficient, a new GPU card will be selected from the set of available GPU cards for capacity expansion. The new GPU card must meet the following conditions: and and ; in, This represents the total video memory capacity of the newly added GPU card. Based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling-up is triggered, a cost-priority scaling-up strategy is designed according to the principle of "quickly releasing the entire GPU card", prioritizing the reclamation of GPU card resources sparsely populated in the instance.

2. The method according to claim 1, characterized in that, Methods for calculating the overall GPU load index include: ； in, This refers to the overall GPU load index. This represents the normalized GPU computational utilization, and , For GPU computing utilization; Indicates the video memory usage rate; This represents the normalized bandwidth utilization, and , For bandwidth utilization; the weighting coefficients satisfy... .

3. The method according to claim 2, characterized in that, Methods for calculating the overall scheduling cost include: ； in, To cover overall scheduling costs; For normalized queuing load costs, , For example At any moment The number of queued requests, This is the threshold for the number of people queuing; For normalized operating load costs, , For example At any moment Number of requests being processed For example At any moment The number of concurrent requests; Normalized arrival rate pressure cost, , Representation of instances At any moment Request delivery rate This is the maximum allowed arrival rate threshold; For normalized KV cache pressure cost, , Representation of instances At any moment The percentage of KV cache usage; For TTFT performance cost, To represent an instance At any moment The first token delay of P90 For TTFT's service level objectives, TTFT prioritizes Token latency; For TBT performance cost, Representation of instances At any moment P90Token latency, The service level target for TBT is the inter-Token latency; For end-to-end performance cost, Representation of instances At any moment P90 end-to-end latency, The service level objective is end-to-end latency; the weighting coefficients satisfy... .

4. The method according to claim 1, characterized in that, Methods for establishing scaling-up / scaling trigger mechanisms based on monitoring metrics and latency prediction results include: Establish an early warning system for SLO (Solution Loading) violation risks based on latency prediction results, triggering capacity expansion assessments; Establish capacity pressure alerts based on real-time instance load status to trigger expansion assessments; Establish a cost-optimized scaling-down trigger mechanism based on resource utilization efficiency to trigger scaling-down assessment.

5. The method according to claim 1, characterized in that, Based on the scaling-up / scaling trigger mechanism and request distribution results, after scaling-up is triggered, a cost-priority scaling-up strategy is designed according to the principle of "quickly releasing the entire GPU card." Methods that prioritize the reclamation of sparse GPU card resources in instances include: A two-layer selection mechanism is designed based on the principle of "quickly releasing the entire GPU card": During the GPU card selection phase, instances are selected for sparse GPU card deployment by calculating a scaling-down priority score: ； in, Rate the priority; Indicates the number of instances deployed on the GPU card; Maximum instance capacity; weighting coefficients satisfy ; Select the GPU card with the highest priority score for scaling down as the target GPU card for scaling down. During the instance selection phase, a multi-metric evaluation method is used to calculate the scaling-down score for each instance on the target scaling-down GPU card to select the scaling-down target: ； in, Score for reduced capacity; The minimum effective request arrival rate threshold is preset for the system; the weighting coefficients satisfy... ; The instance with the highest scaling-up score is selected as the scaling-up target to achieve rapid recycling and consolidation of GPU resources.

6. A scaling system for a vGPU-based SLO-aware model in a computing power network, the system being used to implement the method described in any one of claims 1-5, characterized in that, The system includes: a data acquisition module, a prediction module, a scheduling module, and a scaling module; The data collection module is used to design and deploy a Prometheus-based monitoring system. The system collects load metrics from the GPU side and instance side of the Kubernetes cluster to obtain monitoring metrics. The prediction module is used to construct an output length predictor based on a fine-tuned BERT-base lightweight model, and combine it with the real-time system status to build a joint prediction model for the first token delay, inter-token delay, and end-to-end delay, and obtain the delay prediction results. The scheduling module is used to distribute requests based on monitoring metrics and employs a two-stage scheduling strategy. The scaling module is used to design an SLO-aware elastic scaling mechanism based on monitoring metrics and latency prediction results. It dynamically selects scaling strategies based on SLO violation risks, combined with request distribution results.

Images