Pod scaling method, device, equipment, storage medium and computer program product

Abstract

The application provides a Pod capacity expansion and reduction method, device, equipment, storage medium and computer program product. The method is applied to a capacity expansion and reduction cluster. The capacity expansion and reduction cluster is deployed with a customized controller with a control function, a calculator with a calculation function and a collector with a collection function. The method comprises the following steps: listening to attribute information of a Pod corresponding to a reasoning service through the controller; acquiring the attribute information through the collector, and collecting running index data of the Pod based on the attribute information, wherein the running index data comprises one or more of the following: throughput index, load state index and cache index; acquiring the running index data through the calculator, and determining to perform a capacity expansion operation or a capacity reduction operation on the Pod corresponding to the reasoning service based on the running index data.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a Pod scaling method, system, storage medium, and computer program product. Background Technology

[0002] Currently, Kubernetes (K8s) has become the preferred runtime environment and platform for an increasing number of large-scale model inference services. K8s provides powerful elastic scheduling, resource management, and automatic scaling capabilities. Typically, scaling solutions rely on a single metric, which often fails to accurately reflect the real-time data of the inference service. This can easily lead to Pod scaling not meeting actual needs, and current metric collection often relies on third-party plugins, resulting in significant operational costs. Summary of the Invention

[0003] In view of this, embodiments of this application provide a Pod scaling method, system, storage medium, and computer program product, which can reduce operation and maintenance costs, and the Pod scaling can meet actual needs.

[0004] The technical solution of this application is implemented as follows: In a first aspect, this application provides a Pod scaling method applied to a scaling cluster, wherein a custom controller with control functions, a calculator with calculation functions, and a collector with data collection functions are deployed in the scaling cluster, and the method includes: The controller monitors the attribute information of the Pods corresponding to the deployed inference service. The collector acquires attribute information and, based on this attribute information, collects Pod runtime metrics data, which include one or more of the following: throughput metrics, load status metrics, and cache metrics. The calculator is used to obtain operational metric data. Based on this data, it is determined whether to perform scaling up or scaling down operations on the Pods corresponding to the deployed inference service. Secondly, this application provides a capacity expansion / contraction device, the device comprising: The controller is used to listen to the attribute information of the Pod corresponding to the deployed inference service; The collector is used to obtain attribute information and, based on the attribute information, collect the Pod's runtime metrics data. The runtime metrics data includes one or more of the following: throughput metrics, load status metrics, and cache metrics. The calculator is used to obtain operational metric data and, based on this data, determine whether to perform scaling up or scaling down operations on the Pods corresponding to the deployed inference service.

[0005] Thirdly, this application provides a scaling device, the device comprising: a processor and a memory; the processor implements the above-described Pod scaling method when executing a running program stored in the memory.

[0006] Fourthly, this application provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described Pod scaling method.

[0007] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described Pod scaling method.

[0008] This application provides a method, apparatus, device, storage medium, and computer program product for scaling up and down Pods. The method includes: monitoring the attribute information of Pods corresponding to the deployed inference service through a controller; acquiring the attribute information through a collector, and collecting Pod operational metric data based on the attribute information, wherein the operational metric data includes one or more of the following: throughput metrics, load status metrics, and cache metrics; acquiring the operational metric data through a calculator, and determining whether to perform a scaling up or scaling down operation on the Pods corresponding to the deployed inference service based on the operational metric data. Using the above implementation scheme, when scaling up or down Pods, it is necessary to continuously monitor the attribute information of the Pods corresponding to the deployed inference service, and collect operational metric data of the Pods corresponding to the monitored attribute information. Since the operational metric data contains multiple types of metric data, it is possible to analyze the real-time operational data of the Pods deploying the inference service from multiple perspectives. Therefore, when scaling up or down based on the operational metric data, the scaling up or down decision can be more aligned with actual needs. Furthermore, by directly collecting data and performing corresponding calculations through a custom controller, calculator, and collector deployed in the scaling up / down cluster, data collection and corresponding calculation processing are performed without the need for third-party plugins, reducing operational costs. Attached Figure Description

[0009] Figure 1 This is a schematic flowchart of a Pod scaling method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of an overall architecture for scaling up and down. Figure 3 This is a schematic diagram of the composition structure of a capacity expansion / contraction device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the composition structure of a capacity expansion / contraction device provided in an embodiment of this application. Detailed Implementation

[0010] To gain a more detailed understanding of the features and technical content of the embodiments of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The accompanying drawings are for reference only and are not intended to limit the embodiments of this application.

[0011] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first / second / third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0013] Currently, Kubernetes (K8s) has become the preferred runtime environment and platform for an increasing number of large-scale model inference services. Kubernetes provides powerful elastic scheduling, resource management, and automatic scaling capabilities, efficiently supporting high-concurrency model inference and the utilization of heterogeneous hardware resources. When a user initiates an inference request (such as text, images, etc.), the request first reaches the inference service gateway. The gateway, as a unified entry point, is responsible for accessing, parsing, and performing preliminary verification. Then, the gateway load balances the request to the inference service Pods. The model in the Pod loads the requested content, performs forward inference computation, generates the result, and then returns the inference result to the gateway. The gateway then encapsulates the response and returns it to the client.

[0014] When request traffic increases, the current number of Pod backend instances is insufficient to handle all requests. The main ways to dynamically scale up and down the number of inference service backend instances are as follows: Method 1: Based on Kubernetes' Horizontal Pod Autoscaler (HPA), this method scales up and down the number of backend instances by monitoring the Queries Per Second (QPS) metric related to the inference service in the monitoring gateway. QPS represents the number of query requests the system can process per second, and is an important performance indicator measuring the server's or system's request processing capacity per unit of time.

[0015] Method 2: Configure the inference service as a Knative Service. Based on the Knative Pod Autoscaler (KPA) component, it can scale up and down using QPS (Requests Per Second, RPS). At the same time, the "KPA panic window" mechanism can respond more sensitively to instantaneous traffic.

[0016] It is evident that there is no Layer 7 gateway specifically designed for large-model inference scenarios among relevant technologies. The common implementation approach is to optimize mainstream gateways (such as Nginx / Envoy) for large-model scenarios. Currently, most mainstream gateways automatically scale the number of backend instance replicas based on the QPS of user requests. This solution, which dynamically determines the timing of scaling based on QPS metrics, is suitable for short, uniform, and stateless lightweight service scenarios, and performs well for inference services where resource consumption has a linear relationship. However, it is not suitable for large-model inference service scenarios.

[0017] In large-scale model inference service scenarios, static deployment can easily lead to resource waste or performance bottlenecks due to the high resource consumption, latency sensitivity, and significant traffic fluctuations. Dynamic scaling, on the other hand, automatically adjusts computing resources based on real-time load, ensuring low latency and high availability during peak periods while reducing costs during off-peak periods. Especially in online inference, timely scaling can prevent request queuing and tail latency increases; while in offline batch processing scenarios, reasonable scaling helps improve resource utilization. Therefore, possessing efficient and accurate elasticity is a key foundation for building a stable, economical, and sustainable large-scale model service system.

[0018] Currently, the technical solutions in related technologies still have the following main technical problems when facing scenarios of large model inference services: 1. QPS is essentially designed for stateless microservices, and the approach of measuring request rate using QPS is not entirely applicable to large model inference request scenarios. In large model inference request scenarios, large model inference services are characterized by slow, uneven, and expensive requests. The overhead of a single inference request for a large language model varies greatly. Different input context lengths or generated outputs can lead to vastly different GPU computation and memory usage. However, QPS only counts the number of requests and does not distinguish the difficulty of the requests. This may result in a situation where the QPS is small, but the context length is very large, meaning that the actual inference service load is already very high. Therefore, using only the QPS metric cannot accurately reflect the load of requests on large model inference services.

[0019] 2. Among related technologies, dynamic scaling primarily relies on Kubernetes HPA features or Knative components. For Kubernetes HPA, third-party components such as Prometheus and its adapter are required to acquire monitoring data in real time. For Knative components, they also need to be installed and deployed in the cluster. For users, this approach is cumbersome, with a high learning curve and significant ongoing maintenance costs.

[0020] Based on the above description, the embodiments of this application mainly address the two problems mentioned above by designing a Kubernetes controller to implement automatic elastic scaling of large model inference services. Furthermore, for large model inference service scenarios, in addition to monitoring QPS metrics, multiple evaluation metrics are considered to determine the timing of inference service scaling. Moreover, the collection of all monitoring metrics does not rely on any third-party components, reducing user learning and maintenance costs.

[0021] The detailed technical solution of this application is as follows: This application provides a method for scaling up or down a Pod, such as... Figure 1 As shown, this method is applied to a scaling cluster, in which a custom controller with control functions, a calculator with calculation functions, and a collector with data acquisition functions are deployed. The method may include: S101. Listen to the attribute information of the Pod corresponding to the deployed inference service through the controller.

[0022] In this embodiment, the controller, calculator, and collector deployed in the scaling cluster are all custom-developed based on actual needs and deployed in the K8s cluster.

[0023] In this embodiment, the controller is implemented based on the K8s interface and runs as a container in the K8s cluster.

[0024] In this embodiment, the calculator and the data collector can also be deployed in the same way as the controller and run in the K8s cluster.

[0025] In this embodiment of the application, the controller is responsible for monitoring all inference service Pods in the cluster, thereby obtaining the attribute information of each Pod. The attribute information of the Pod includes the Pod name, namespace, Pod IP, the address and port number of the metrics interface exposed by the Pod, etc.

[0026] In this embodiment of the application, after the controller listens to and obtains the attribute information of all inference service Pods, it saves the obtained attribute information of each Pod to a module named Datastore.

[0027] It should be noted that the data in the Datastore module is stored only in the computer's memory and does not rely on external storage.

[0028] It should be noted that the Datastore module is exposed through an interface, allowing other modules (such as the collector) to query and write data.

[0029] In one embodiment of this application, after the controller listens to the attribute information of the Pod corresponding to the deployed inference service, the method further includes: When the lifecycle of the Pod corresponding to the deployed inference service changes, the attribute information of the updated Pod corresponding to the deployed inference service is monitored so that the collector can collect the updated Pod's runtime metric data based on the updated Pod's attribute information.

[0030] In this embodiment, the controller monitors the Pods that deploy the inference service in real time. When the lifecycle of the Pods that deploy the inference service changes (Pod ends running, Pod fails, etc.), once the controller detects that the Pods that deploy the inference service have changed, it can promptly obtain the updated Pod attribute information (i.e., the updated Pod's name, namespace, Pod IP, the address and port number of the metrics interface exposed by the Pod, etc.). After obtaining the updated Pod attribute information, the controller updates the relevant Pod information maintained in the Datastore accordingly to ensure that the Pod information maintained in the Datastore is always up-to-date.

[0031] It should be noted that the controller continuously monitors the Pods corresponding to the inference service. Once the lifecycle of a Pod changes, the controller can also collect the latest Pod attribute information in a timely manner. This ensures the accuracy of the subsequent collection of operational metric data by the collector, making the timing of scaling up and down more in line with the actual situation.

[0032] S102. Obtain attribute information through the collector, and collect Pod's running metric data based on the attribute information. The running metric data includes one or more of the following: throughput metrics, load status metrics, and cache metrics.

[0033] In this embodiment of the application, throughput metrics such as request processing efficiency are used to characterize the number of requests processed by a Pod per unit time, and may include the QPS metric.

[0034] In this embodiment of the application, the load status indicator, such as the number of requests waiting for the pod to process, represents the number of requests currently waiting for the pod to process (i.e., the number of requests that are queuing up, denoted as num_requests_waiting).

[0035] In this embodiment of the application, cache metrics such as cache utilization rate represent the proportion of key-value cache used in the Pod.

[0036] In this embodiment of the application, the controller stores the attribute information of the Pods that deploy the inference service in the Datastore. Then the collector (also known as the metrics collector) can query the Pods to be collected from the attribute information of the Pods stored in the Datastore module. That is, before the collector performs collection, it needs to determine which Pods to collect and where to collect them.

[0037] In this embodiment of the application, after the collector obtains the relevant information of the Pod to be collected, it periodically collects (the collection interval is configurable) the running metric data (also known as metrics running metric data) of each inference service Pod.

[0038] In this embodiment of the application, in order to more accurately describe the current load of the inference service, in addition to the QPS metric of the inference service requests (represented as request_success_total), the collector also collects the key-value (KV) cache utilization rate (i.e., gpu_cache_usage_perc) and the number of requests waiting in the queue for each inference service Pod.

[0039] It should be noted that the above-mentioned multiple metrics can provide the calculator with a more accurate description of the real-time load of the inference service.

[0040] In this embodiment of the application, after the collector collects the above-mentioned running indicator data corresponding to each inference service Pod, the collector writes the collected running indicator data into the Datastore module. It can be understood that the Datastore module can not only store the attribute information of each Pod that the controller listens to and obtains, but also store the running indicator data collected by the collector (or the monitoring indicator data collected by the collector).

[0041] In one implementation of this application, when there are multiple Pods, each Pod corresponds to an attribute information. The collector collects the Pod's operating metric data based on the attribute information. This can also be achieved in the following way: multiple monitoring threads are created based on the number of Pods; at preset intervals, multiple monitoring threads collect the operating metric data corresponding to each Pod in parallel based on the attribute information corresponding to each Pod.

[0042] In this embodiment of the application, since there may be multiple Pods for the inference service, this embodiment of the application uses a concurrent approach to request the metrics interface (i.e. the interface exposed by the Pod) of each Pod to obtain the QPS metric, KVCache utilization and the number of requests waiting in the queue for each Pod.

[0043] In this embodiment of the application, after the Pod is created, the collector can create the same number of monitoring threads as the number of Pods, that is, each Pod corresponds to a separate thread to collect the Pod's running metric data.

[0044] In this embodiment of the application, multiple threads can run concurrently. The collector can collect the QPS metric, KV Cache utilization, and number of requests waiting in the queue for each Pod in multiple Pods at preset time intervals based on the attribute information of the Pods, and then write the data to the DataStore module.

[0045] It should be noted that the interval range for each preset time interval can be selected according to the actual situation, and no specific limitation is made in this embodiment.

[0046] S103. Obtain the operational metric data through a calculator, and based on the operational metric data, determine whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployment of the inference service.

[0047] In this embodiment of the application, the calculator can specifically obtain the QPS metric, KV Cache utilization, and number of queued requests for each inference service Pod collected by the collector from the DataStore module through a subscription method, and perform real-time calculations.

[0048] In this embodiment, the calculator subscribes to data update events in the DataStore module associated with the controller. Once the metrics of a Pod in the DataStore module are updated, the calculator can detect the change in real time and obtain the new metrics, including QPS, KV Cache utilization, and the number of requests waiting in the queue.

[0049] In this embodiment, the calculator obtains metrics such as QPS, KV Cache utilization, and the number of requests waiting in the queue for each inference service Pod in the DataStore module. Then, based on the user-configured metrics and corresponding calculation rules, it determines whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployed inference service. The specific implementation is as follows: it determines whether the throughput, load status, and cache metrics included in the running metrics data meet the corresponding preset scaling-up or scaling-down conditions, and obtains a confirmation result; based on the confirmation result, it determines whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployed inference service.

[0050] In this embodiment of the application, the user can pre-set calculation rules (i.e., scaling up or down rules) for each indicator parameter in the running indicator data. For example, if the throughput indicator is greater than a certain threshold, scaling up is required. For example, if the QPS is greater than 80%, scaling up is required. If the load status indicator is greater than a certain threshold, scaling up is required. For example, if the number of requests waiting in the queue is greater than 20, scaling up is required. If the cache indicator is greater than a certain threshold, scaling up is required. For example, if the KV Cache utilization rate is greater than 85%, scaling up is required.

[0051] It should be noted that the calculation rules set are used to determine whether each indicator meets the conditions for expansion. In this embodiment of the application, the expansion calculation rules are to set corresponding thresholds for each indicator parameter.

[0052] In this embodiment, if the QPS is greater than the QPS expansion threshold, the expansion condition is determined to be met; if the KVCache utilization rate is greater than the KVCache utilization rate expansion threshold, the expansion condition is determined to be met; if the number of requests waiting in the queue is greater than the number of waiting requests expansion threshold, the expansion condition is determined to be met.

[0053] In this embodiment of the application, if none of the above expansion conditions are met, it means that expansion is not required at this time. Therefore, it can be determined whether to reduce the size under the current circumstances based on the current Pod's operating metric data.

[0054] In this embodiment of the application, the user can also pre-set calculation rules for each parameter in the running metric data. For example, if the throughput metric is less than a certain threshold, the system will scale down. For example, if the QPS is less than 40%, the system will scale down. If the load status metric is less than a certain threshold, the system will scale down. For example, if the number of requests waiting in the queue is less than 5, the system will scale down. If the cache metric is less than a certain threshold, the system will scale down. For example, if the KV Cache utilization is less than 45%, the system will scale down.

[0055] It should be noted that the calculation rules set are used to determine whether each indicator meets the conditions for scaling down. In this embodiment of the application, the scaling down calculation rules are to set corresponding thresholds for each indicator parameter.

[0056] In this embodiment, if the QPS is less than the QPS scaling-down threshold, the scaling-down condition is determined to be met; if the KVCache utilization rate is less than the KVCache utilization rate scaling-down threshold, the scaling-down condition is determined to be met; if the number of requests waiting in the queue is less than the number of waiting requests scaling-down threshold, the scaling-down condition is determined to be met.

[0057] In this embodiment of the application, the above implementation process is that the user pre-sets a judgment rule for each running indicator parameter. After the calculator obtains the QPS of the Pod, the number of requests waiting in the queue, and the KV Cache utilization rate, the obtained QPS of the Pod, the number of requests waiting in the queue, and the KV Cache utilization rate are compared with the preset expansion threshold and shrinkage threshold respectively to determine whether each indicator meets the expansion condition or shrinkage condition, and finally obtain the comparison result of each running indicator parameter with the corresponding threshold.

[0058] In this embodiment of the application, after determining whether each indicator parameter meets the conditions for scaling up or down based on the above implementation method, it is also necessary to combine the comparison results to further determine whether it is necessary to perform scaling up or scaling down operations on the Pod that deploys the inference service.

[0059] It should be noted that if the QPS, KV Cache utilization, and the number of requests waiting in the queue do not meet the conditions for scaling up or down, then it means that neither scaling up nor down is necessary.

[0060] In this application embodiment, when further determining whether to expand or shrink capacity based on the comparison results of each operating indicator parameter and threshold obtained above, it can be determined whether to perform expansion or shrinkage operation in the current state according to the expansion and shrinkage rules preset by the user.

[0061] In one embodiment of this application, based on the confirmation result, it is determined whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployed inference service. Specifically, this can be achieved in the following ways: if the confirmation result shows that the throughput metric, load status metric, and cache metric all meet the corresponding preset scaling-up conditions, it is determined to perform a scaling-up operation on the Pod corresponding to the deployed inference service; or, if the confirmation result shows that the throughput metric, load status metric, and cache metric all meet the corresponding preset scaling-down conditions, it is determined to perform a scaling-down operation on the Pod corresponding to the deployed inference service.

[0062] In this embodiment, the user-preset scaling rules stipulate that scaling operations will only be triggered if QPS, KV Cache utilization, and the number of queued requests all meet certain scaling conditions. Specifically, scaling operations are triggered only when QPS exceeds a QPS scaling threshold, KV Cache utilization exceeds a KV Cache utilization scaling threshold, and the number of queued requests exceeds a waiting request scaling threshold. In other words, scaling operations can only be performed if the comparison between each performance metric and its corresponding preset threshold meets the scaling conditions.

[0063] In this embodiment of the application, a scaling-down operation is triggered only if the QPS is less than the QPS scaling-down threshold, the KV Cache utilization rate is less than the KV Cache utilization scaling-down threshold, and the number of requests waiting in the queue is less than the number of waiting requests scaling-down threshold. That is, the scaling-down operation can only be performed if the comparison result of each running indicator parameter with the corresponding preset threshold meets the scaling-down condition.

[0064] In one embodiment of this application, based on the confirmation result, determining whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployed inference service can also be achieved in the following ways: if the confirmation result shows that any one of the throughput metric, load status metric, and cache metric meets the corresponding preset scaling-up condition, then determine to perform a scaling-up operation on the Pod corresponding to the deployed inference service; if the confirmation result shows that any one of the throughput metric, load status metric, and cache metric meets the corresponding preset scaling-down condition, then determine to perform a scaling-down operation on the Pod corresponding to the deployed inference service.

[0065] In this embodiment, the user-preset scaling rules can also trigger a scaling operation if any one of the following conditions is met: QPS, KV Cache utilization, or the number of requests waiting in the queue. Specifically, a scaling operation can be triggered if QPS exceeds a QPS scaling threshold, KV Cache utilization exceeds a KV Cache utilization scaling threshold, or the number of requests waiting in the queue exceeds a waiting request scaling threshold. That is, if any one of the comparison results between each running metric parameter and its corresponding preset threshold meets the scaling condition, a scaling operation can be performed.

[0066] In this embodiment of the application, if the QPS is less than the QPS reduction threshold, the KV Cache utilization rate is less than the KV Cache utilization reduction threshold, or the number of requests waiting in the queue is less than the number of waiting requests reduction threshold, the reduction operation will be triggered. That is, if any one of the comparison results of each running indicator parameter with the corresponding preset threshold meets the reduction condition, the reduction operation can be performed.

[0067] The above implementation can be summarized as follows: When configuring dynamic scaling strategies (rules), the calculator supports setting one or more metrics that may affect the timing of scaling up or down: 1. Users can define multiple conditional expressions based on system metrics for a scaling-up or scaling-down action, including QPS, KV Cache utilization, and the number of requests waiting in the queue. This can be understood as setting corresponding scaling-up and scaling-down conditional expressions for each metric in advance.

[0068] 2. Users must also pre-specify an "ANDOR" logical operator for this dynamic scaling strategy.

[0069] If "AndOr=And", then the corresponding scaling action will only be triggered when all conditional expressions within the scaling strategy meet the conditions.

[0070] If "AndOr=Or", then the corresponding scaling action will be triggered when any conditional expression within the scaling strategy is satisfied.

[0071] In this embodiment, the method provides users with the ability to precisely control the scaling-up and scaling-down triggering logic. Depending on the business scenario and business needs, users can flexibly choose whether to require all key indicators to reach the threshold before executing the scaling-up and scaling-down action (more stringent, preventing false triggering) or to execute the scaling-up and scaling-down action as soon as any key indicator reaches the threshold (more sensitive, faster response).

[0072] In one embodiment of this application, after determining whether to perform a scaling-up or scaling-down operation on the Pods corresponding to the deployed inference service based on operational metric data, the method further includes: determining the number of Pods to be added or removed from the deployed inference service; and performing a corresponding number of scaling-up or scaling-down operations on the Pods corresponding to the deployed inference service based on the number of Pods.

[0073] In this embodiment, when determining to perform a scaling operation on the Pods deploying the inference service based on the obtained QPS, the number of requests waiting in the queue, and the KV Cache utilization, the calculator can also determine the number of Pods that need to be added based on the current Pod's QPS, the number of requests waiting in the queue, and the KV Cache utilization. That is, it can be understood that the calculator can combine the Pod's load operation status, such as the load currently being higher than a certain value, in this case determine the accurate number of Pods that need to be scaled up, directly output the corresponding number of Pods that need to be scaled up, and trigger the corresponding increase in the number of Pods for the inference service based on the number of Pods to be scaled up.

[0074] It should be noted that the above applies to situations with a large load.

[0075] In this embodiment, when determining to perform a scaling-down operation on the Pods deploying the inference service based on the obtained QPS, the number of requests waiting in the queue, and the KV Cache utilization, the calculator can also determine the number of Pods that need to be reduced based on the current Pod's QPS, the number of requests waiting in the queue, and the KV Cache utilization. That is, it can be understood that the calculator can combine the Pod's load operation status, such as the load currently being lower than a certain value, in this case determine the accurate number of Pods that need to be scaled down, directly output the corresponding number of Pods that need to be scaled down, and trigger a corresponding reduction in the number of Pods deployed for the inference service based on the number of Pods reduced.

[0076] It should be noted that the above applies to situations with relatively low load.

[0077] It is understood that the Pod scaling method provided in this application requires continuous monitoring of the attribute information of the Pods deploying inference services during Pod scaling. Based on the monitored attribute information, the method collects the runtime metric data of the Pods corresponding to the attribute information. Since the runtime metric data includes various types of metrics, it allows for analysis of the real-time runtime data of the Pods deploying inference services from multiple perspectives. This makes scaling decisions more aligned with actual needs when scaling based on runtime metric data. Furthermore, by using custom controllers, calculators, and collectors deployed in the scaling cluster, data collection and corresponding calculations are performed directly, eliminating the need for third-party plugins and reducing operational costs. Based on the above embodiments, this application also provides an overall architecture diagram of a Pod scaling method, such as... Figure 2 As shown, the functionalities of each module are implemented as follows: The controller, calculator, and collector in this embodiment are implemented based on the K8s interface and run as containers in the K8s cluster. Figure 2 The overall architecture diagram includes three main parts: collector, calculator, and controller.

[0078] The collector is responsible for obtaining the attribute information of the Pods that the controller listens to and stores from the Datastore, and collecting the corresponding runtime metrics data for each inference service Pod based on the attribute information.

[0079] The controller is responsible for listening to the attribute information of all inference services and all Pods contained in each inference service, including Pod IP information, metrics interface information, etc., and writing the attribute information of the monitored Pods into the Datastore for the collector to query the Pod information to be collected.

[0080] The calculator is responsible for obtaining the runtime metrics data of the inference service's Pods collected and written by the collector from the Datastore, calculating whether the current inference service needs to be scaled up or down based on the runtime metrics data collected by the collector, and controlling the scaling up or down of the inference service based on the scaling up or down results.

[0081] Based on the above embodiments, this application presents a Kubernetes controller that provides basic data support for dynamic scaling by real-time monitoring of dynamic changes in inference service resources and periodically collecting metrics. This avoids the installation, deployment, and maintenance of third-party dependent components, reducing deployment and maintenance costs. Secondly, instead of simply using the QPS metric of the inference service to calculate the timing of scaling and the number of replicas, specific optimizations have been made for large model scenarios. The QPS metric (request_success_total), KV Cache utilization metric (gpu_cache_usage_perc), and the number of queued requests metric (num_requests_waiting) are used to comprehensively evaluate the current load of the inference service, thereby determining whether scaling is necessary.

[0082] Compared with related technologies, the embodiments of this application have the following technical advantages: 1. A lightweight architecture with zero dependencies eliminates the need to deploy third-party components such as Prometheus. It directly collects metrics through a self-developed controller, significantly reducing system complexity and operational costs. Low resource overhead avoids the GC pressure and memory consumption of traditional monitoring stacks, making it particularly suitable for large-scale model scenarios sensitive to GPU resources.

[0083] 2. Deep monitoring of large-scale model-specific metrics: It not only determines the timing of scaling up or down based on request traffic, but also uses other core bottleneck metrics of the large-scale model to improve the accuracy of scaling up or down decisions.

[0084] The technical solutions of this application embodiment can achieve the following effects in practical applications: 1. Cost optimization: Precisely match resources and load to avoid resource idleness, improve GPU utilization, and reduce costs.

[0085] 2. Business continuity assurance: Second-level response to traffic peaks, reducing the risk of service degradation. 3. Improved Operation and Maintenance Efficiency: Automation replaces manual scaling up and down, reducing manpower investment in operation and maintenance and allowing focus on high-value innovation.

[0086] The technical solution of this application embodiment can ultimately be implemented in the large model inference service function of the mobile cloud intelligent computing platform. Through intelligent scheduling, load prediction, elastic scaling, and resource utilization optimization, it largely solves the problems of serious resource waste, untimely response, and high maintenance costs in traditional large model inference services. With the continuous growth of demand for large model inference, mobile cloud is expected to leverage this application embodiment to form an inference service capability with high cost-effectiveness and high reliability.

[0087] Based on the above embodiments, another embodiment of this application provides a capacity expansion / contraction device 1, such as... Figure 3 As shown, the expansion / contraction device 1 includes: Controller 10 is used to monitor the attribute information of the Pod corresponding to the deployed inference service; Collector 11 is used to obtain attribute information and collect Pod's running metric data based on the attribute information. The running metric data includes one or more of the following: throughput metrics, load status metrics, and cache metrics. Calculator 12 is used to obtain operational metric data and, based on the operational metric data, determine whether to perform scaling up or scaling down operations on the Pod corresponding to the deployed inference service.

[0088] In one embodiment, the calculator 12 is further configured to determine whether the throughput, load status, and cache metrics included in the operational metrics data meet the corresponding preset scaling-up or scaling-down conditions, and obtain a confirmation result; based on the confirmation result, it is determined to perform scaling-up or scaling-down operations on the Pod corresponding to the deployment of the inference service. In one embodiment, the calculator 12 is further configured to determine to perform a scaling-up operation on the Pod corresponding to the deployment of the inference service if the confirmation result shows that the throughput indicator, the load status indicator, and the cache indicator all meet the corresponding preset scaling-up conditions; or, if the confirmation result shows that the throughput indicator, the load status indicator, and the cache indicator all meet the corresponding preset scaling-down conditions, determine to perform a scaling-down operation on the Pod corresponding to the deployment of the inference service.

[0089] In one embodiment, the calculator 12 is further configured to determine, when the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset expansion condition, to perform an expansion operation on the Pod corresponding to the deployment of the inference service; and to determine, when the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset shrinkage condition, to perform a shrinkage operation on the Pod corresponding to the deployment of the inference service.

[0090] In one embodiment, there are multiple Pods, and each Pod corresponds to one attribute information.

[0091] In one embodiment, the collector 11 is further configured to create multiple monitoring threads equal to the number of Pods; at preset time intervals, based on the attribute information corresponding to each Pod, the multiple monitoring threads collect the running indicator data corresponding to each of the multiple Pods in parallel.

[0092] In one embodiment, the controller 10 is further configured to listen to the updated attribute information of the Pod corresponding to the deployed inference service when the lifecycle of the Pod changes, so as to collect the updated Pod's running metric data with the collector based on the updated Pod's attribute information.

[0093] In one embodiment, the calculator 12 is further configured to determine the number of Pods that are deployed with the inference service to be increased or decreased; based on the number, to perform a corresponding number of scaling up or down operations on the Pods that are deployed with the inference service.

[0094] This application provides a scaling device that uses a controller to monitor the attribute information of the Pods corresponding to the deployed inference service; a collector to acquire the attribute information and, based on the attribute information, collects the Pod's operational metrics data, which includes one or more of the following: throughput metrics, load status metrics, and cache metrics; and a calculator to acquire the operational metrics data and, based on the operational metrics data, determines whether to perform a scaling up or scaling down operation on the Pods corresponding to the deployed inference service. Therefore, this scaling device continuously monitors the attribute information of the Pods corresponding to the deployed inference service during Pod scaling up or down, and collects operational metrics data corresponding to the monitored attribute information. Since the operational metrics data includes multiple types of metrics, it allows for analysis of the real-time operational data of the Pods deploying the inference service from multiple perspectives. This makes scaling up or down decisions more aligned with actual needs. Furthermore, by using a custom controller, calculator, and collector deployed in the scaling up / down cluster, data collection and corresponding calculations are performed directly, eliminating the need for third-party plugins and reducing operational costs.

[0095] Figure 4 This is a schematic diagram of the composition of a capacity expansion / reduction device 2 provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 4 As shown, the scaling device 2 in this embodiment includes a processor 20, a memory 21, and a communication bus 22.

[0096] In specific embodiments, the controller 10, data acquisition unit 11, and calculator 12 can be implemented by a processor 20 located on the scaling device 2. The processor 20 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the processor function can also be other types, and this application does not impose specific limitations.

[0097] In this embodiment, the communication bus 22 is used to realize the connection communication between the processor 20 and the memory 21; when the processor 20 executes the running program stored in the memory 21, it implements the following Pod scaling method: The controller listens to the attribute information of the Pods corresponding to the deployed inference service; the collector obtains the attribute information and collects the Pod's operation metrics data based on the attribute information. The operation metrics data includes one or more of the following: throughput metrics, load status metrics, and cache metrics; the calculator obtains the operation metrics data and determines whether to perform scaling up or scaling down operations on the Pods corresponding to the deployed inference service based on the operation metrics data.

[0098] In one embodiment, the processor 20 is further configured to determine whether the throughput indicator, load status indicator and cache indicator included in the operation indicator data meet the corresponding preset expansion conditions or preset shrinkage conditions, and obtain a confirmation result; based on the confirmation result, determine to perform expansion operation or shrinkage operation on the Pod corresponding to the deployment of the inference service.

[0099] In one embodiment, the processor 20 is further configured to determine to perform a scaling-up operation on the Pod corresponding to the deployment of the inference service if the confirmation result is that the throughput indicator, the load status indicator, and the cache indicator all meet the corresponding preset scaling-up conditions; or, if the confirmation result is that the throughput indicator, the load status indicator, and the cache indicator all meet the corresponding preset scaling-down conditions, determine to perform a scaling-down operation on the Pod corresponding to the deployment of the inference service.

[0100] In one embodiment, the processor 20 is further configured to: determine to perform a scaling-up operation on the Pod corresponding to the deployment of the inference service if the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset scaling-up condition; and determine to perform a scaling-down operation on the Pod corresponding to the deployment of the inference service if the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset scaling-down condition.

[0101] In one embodiment, there are multiple Pods, and each Pod corresponds to one attribute information.

[0102] The processor 20 described above is further configured to create multiple monitoring threads, the same number as the number of Pods; and at preset time intervals, based on the attribute information corresponding to each Pod, use the multiple monitoring threads to collect the operational metric data corresponding to each of the multiple Pods in parallel.

[0103] In one embodiment, the processor 20 is further configured to monitor the updated attribute information of the Pod corresponding to the deployed inference service when the lifecycle of the Pod changes, so as to collect the updated Pod's runtime metric data based on the updated Pod's attribute information.

[0104] In one embodiment, the processor 20 is further configured to determine the number of Pods deploying the inference service to be increased or decreased; and based on the number, to perform a corresponding number of scaling up or down operations on the Pods corresponding to the deployment of the inference service.

[0105] Based on the above embodiments, this application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a scaling device. The computer program implements the Pod scaling method as described above.

[0106] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors and applied in a scaling device. The computer program implements the Pod scaling method as described above.

[0107] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the embodiments of this application.

[0109] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for scaling up or down a Pod, characterized in that, Applied to scaling up and down clusters, wherein a custom controller with control functions, a calculator with calculation functions, and a data collector with data acquisition functions are deployed in the scaling up and down cluster, the method includes: The controller monitors the attribute information of the Pod corresponding to the deployed inference service. The collector acquires the attribute information and, based on the attribute information, collects the Pod's operational metrics data, wherein the operational metrics data includes one or more of the following: throughput metrics, load status metrics, and cache metrics. The calculator is used to obtain the operational metrics data, and based on the operational metrics data, it is determined whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployment of the inference service.

2. The method according to claim 1, characterized in that, The step of determining whether to perform scaling up or scaling down operations on the Pod corresponding to the deployment of the inference service based on the operational metric data includes: Determine whether the throughput, load status, and cache metrics included in the operational metrics data meet the corresponding preset expansion or reduction conditions, and obtain a confirmation result. Based on the confirmation result, it is determined whether to perform a scaling up or scaling down operation on the Pod corresponding to the deployment of the inference service.

3. The method according to claim 2, characterized in that, Based on the confirmation result, determining whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployment of the inference service includes: If the confirmation result shows that the throughput metric, the load status metric, and the cache metric all meet the corresponding preset expansion conditions, then it is determined to perform an expansion operation on the Pod corresponding to the deployment of the inference service; or... If the confirmation result shows that the throughput metric, the load status metric, and the cache metric all meet the corresponding preset scaling-down conditions, it is determined to perform a scaling-down operation on the Pod corresponding to the deployment of the inference service.

4. The method according to claim 2, characterized in that, Based on the confirmation result, determining whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployment of the inference service includes: If the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset expansion condition, it is determined to perform an expansion operation on the Pod corresponding to the deployment of the inference service. If the confirmation result is that any one of the throughput metric, the load status metric, and the cache metric meets the corresponding preset scaling-down condition, it is determined that a scaling-down operation will be performed on the Pod corresponding to the deployment of the inference service.

5. The method according to claim 1, characterized in that, The number of Pods is multiple, and each Pod corresponds to a set of attribute information. Based on this attribute information, the operational metric data of the Pods is collected, including: Create multiple monitoring threads based on the number of Pods, the same number as the number of Pods; At preset time intervals, based on the attribute information corresponding to each Pod, the multiple monitoring threads collect the operation indicator data corresponding to each of the multiple Pods in parallel.

6. The method according to claim 1, characterized in that, After monitoring the attribute information of the Pod corresponding to the deployed inference service, the method further includes: When the lifecycle of the Pod corresponding to the deployed inference service changes, the attribute information of the updated Pod corresponding to the deployed inference service is monitored, so that the collector can collect the updated Pod's running metric data based on the updated Pod's attribute information.

7. The method according to claim 1, characterized in that, After determining whether to perform a scaling-up or scaling-down operation on the Pod corresponding to the deployment of the inference service based on the operational metric data, the method further includes: Determine whether to increase or decrease the number of Pods deploying the inference service; Based on the stated quantity, perform corresponding scaling up or scaling down on the Pods that deploy the inference service.

8. A capacity expansion / contraction device, characterized in that, The device includes a processor and a memory; the processor, when executing a running program stored in the memory, implements the method as described in any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images