AI model service elastic scaling system and method based on cloud native

Abstract

The invention discloses an AI model service elastic scaling system and method based on cloud native. The AI model service elastic scaling system comprises a model feature analysis module and an elastic strategy execution module. Wherein the model feature analysis module collects GPU operation data through a computing resource monitor, constructs a dual-threshold constraint mechanism by utilizing a batch processing efficiency analyzer, and generates a GPU stability curve and a memory efficiency curve by an efficiency curve analysis system; and the elastic strategy execution module converts the target batch processing parameters into an execution plan and realizes atomicity adjustment through a Kubernetes extension mechanism. According to the method, a double-layer analysis mechanism is innovatively provided, the GPU stability and the memory efficiency are analyzed as independent dimensions, and the problem of service oscillation caused by only paying attention to a single index in a traditional scheme is effectively solved. Through a progressive adjustment strategy and a perfect state tracking mechanism, accurate capacity expansion and shrinkage of AI model service are realized, the resource utilization efficiency is remarkably improved, and meanwhile, the stability of service quality is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a cloud-native-based AI model service elastic scaling system and method. Background Art

[0002] With the rapid development of artificial intelligence technology, the deployment scale of AI model services based on deep learning in cloud computing environments continues to expand. Especially in the fields of image processing and natural language processing, AI model services need to handle business loads with significant fluctuations, which puts higher requirements on the dynamic scheduling of computing resources. In order to improve resource utilization efficiency and ensure service quality, AI model services need to dynamically adjust resources according to load changes. Although the current mainstream cloud native platforms (such as Kubernetes) provide basic automatic expansion and contraction mechanisms, these general mechanisms still have many limitations in computing-intensive scenarios such as AI model services.

[0003] The existing scaling technologies have the following main shortcomings: First, traditional solutions often only focus on a single indicator (such as GPU utilization or video memory occupancy), and lack comprehensive consideration of multi-dimensional resource characteristics such as GPU computing efficiency, memory bandwidth, and batch processing efficiency in AI model services, which leads to the lack of comprehensiveness and accuracy in scaling decisions. Second, the existing automatic scaling solutions rely too much on simple threshold trigger mechanisms, which can easily cause frequent scaling operations when the load fluctuates greatly, causing service shocks and affecting business stability. Third, most solutions lack a fine-grained parameter optimization mechanism and cannot dynamically adjust batch processing parameters according to the computing characteristics of the AI ​​model, which not only causes inefficient resource utilization, but also may cause service response delays beyond expectations. Fourth, during the scaling process, due to the lack of a complete state tracking and rollback mechanism, once an abnormal situation occurs, it is easy to affect the availability of online services and even cause service interruptions.

[0004] At the same time, AI model services in cloud-native environments also face other challenges: first, there is the problem of resource fragmentation. There may be resource competition between service instances of different sizes, affecting overall performance. Second, there is the problem of configuration consistency. During the process of scaling up and down service instances, how to ensure the consistency of batch processing parameters and resource configuration is also an important challenge.

[0005] Therefore, how to design a system that can comprehensively consider multiple dimensions such as GPU computing efficiency, memory resource utilization, and service stability, and can achieve precise scaling and adaptive parameter optimization, is a technical problem that needs to be solved in the current cloud-native AI model service field. In particular, maximizing resource utilization efficiency while ensuring service stability requires innovative solutions to this technical problem. Summary of the invention

[0006] The problem to be solved by the present invention is how to provide a cloud-native AI model service elastic scaling system and method that can simultaneously ensure service stability and resource utilization efficiency. In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A cloud-native-based AI model service elastic scaling system, characterized by comprising: a model feature analysis module and an elastic strategy execution module;

[0008] Among them, the model feature analysis module includes a computing resource monitor, a batch efficiency analyzer and an efficiency curve analysis system. The computing resource monitor is integrated with the data center GPU manager and transmits the collected GPU utilization, video memory occupancy and memory bandwidth data during the AI ​​model runtime to the batch efficiency analyzer. The batch efficiency analyzer analyzes the AI ​​model reasoning batch efficiency according to the GPU utilization fluctuation characteristics and memory efficiency indicators and transmits the evaluation results to the efficiency curve analysis system. The efficiency curve analysis system constructs a GPU stability curve and a memory efficiency curve according to the evaluation results transmitted by the batch efficiency analyzer and outputs the target batch parameters adapted to the AI ​​model reasoning service to the elastic policy execution module;

[0009] Among them, the elastic policy execution module includes a scaling decision maker and a resource scheduler. The scaling decision maker receives the target batch processing parameters transmitted by the efficiency curve analysis system and converts them into an execution plan for the AI ​​model service instance. The resource scheduler receives the execution plan output by the scaling decision maker and integrates the Kubernetes extension mechanism to execute the change operation of the AI ​​model service instance scale and batch processing parameters.

[0010] A further technical solution is that: the computing resource monitor includes a basic monitoring layer and an indicator preprocessing layer; wherein the basic monitoring layer collects GPU utilization, video memory occupancy and memory bandwidth data in real time through a data center GPU manager interface;

[0011] The index preprocessing layer cleans the collected data, calculates the moving average, and marks abnormal data points before transmitting them to the batch processing efficiency analyzer.

[0012] A further technical solution is that: the batch efficiency analyzer sets an analysis mechanism with dual threshold constraints, receives GPU utilization, video memory occupancy and memory bandwidth data collected by the computing resource monitor, calculates the short-term variance value and change trend curve of the GPU utilization, marks the current batch processing parameters as unstable when the GPU utilization variance exceeds a preset threshold or shows a frequent fluctuation trend, further calculates the target interval deviation of the video memory occupancy and memory bandwidth when the GPU utilization variance value is within a preset threshold range, the batch efficiency analyzer generates an analysis data packet, the analysis data packet includes the current batch processing parameters, GPU utilization variance value, GPU stable state mark, video memory occupancy deviation and memory bandwidth deviation, and periodically outputs the analysis data packet to the efficiency curve analysis system for constructing a GPU stability curve and a memory efficiency curve;

[0013] Among them, the GPU utilization variance threshold is set according to the computing density of the AI ​​model, and the target interval deviation is obtained by calculating the difference between the current video memory occupancy and the median of the preset target interval.

[0014] A further technical solution is that: the efficiency curve analysis system receives the analysis data packet output by the batch efficiency analyzer, and uses the sliding window averaging technology to construct a GPU stability curve and a memory efficiency curve, wherein the GPU stability curve uses the batch processing parameter as the horizontal coordinate and the GPU utilization variance value as the vertical coordinate, and the memory efficiency curve uses the batch processing parameter as the horizontal coordinate and the deviation of the video memory occupancy rate and the memory bandwidth as the vertical coordinate. The efficiency curve analysis system screens out a batch processing parameter range whose GPU utilization variance value is lower than a preset threshold through the GPU stability curve, selects the batch processing parameter with the smallest deviation as the target batch processing parameter within the range based on the memory efficiency curve, and when there are multiple candidate parameters with similar deviations, gives priority to the batch processing parameter with a smaller value, and outputs the target batch processing parameter to the scaling decision maker. A further technical solution is that: the scaling decision maker receives the target batch processing parameter output by the efficiency curve analysis system, calculates the target value of the instance scale change, decomposes the change of the target number of instances into multiple progressive adjustment steps, dynamically adjusts the batch processing size through parameter mapping rules in each adjustment step, and combines container lifecycle management to ensure smooth service transition, and outputs the decomposed execution step sequence and corresponding batch processing parameter configuration to the resource scheduler;

[0015] Among them, the progressive adjustment step is to record the GPU utilization and inference delay indicators of the current service instance before executing each adjustment step, determine the adjustment step size according to the difference between the GPU utilization and the preset utilization target, and when the inference delay exceeds the preset threshold, suspend the execution of the current adjustment step.

[0016] A further technical solution is that the resource scheduler runs the Kubernetes extension mechanism to achieve docking with the execution step sequence output by the scaling decision maker, realize atomic adjustment of batch parameters and the number of instances, track the status of the scheduling process of each execution step, and trigger a configuration rollback operation to restore to the previous stable state when an execution anomaly is detected.

[0017] The present invention also discloses a cloud-native-based AI model service elastic scaling method, which is characterized by comprising the following steps:

[0018] Collect GPU utilization, video memory occupancy, and memory bandwidth data in real time when the AI ​​model is running, clean the collected data, calculate the moving average, and mark abnormal data points;

[0019] Calculate the short-term variance value and trend curve of GPU utilization during AI model inference, determine the GPU stable state, calculate the target interval deviation of video memory occupancy and memory bandwidth, and generate an analysis data package containing batch processing parameters, GPU utilization variance value, GPU stable state mark, video memory occupancy deviation, and memory bandwidth deviation;

[0020] The sliding window averaging technology is used to construct the GPU stability curve and the memory efficiency curve. Based on the GPU stability curve, the batch parameter range in which the GPU utilization variance value required for the AI ​​model service is lower than the preset threshold is screened out. Within this range, the batch parameter with the smallest deviation is selected as the target batch parameter according to the memory efficiency curve.

[0021] Convert the target batch parameter into the target value of the AI ​​model service instance scale change, decompose the change of the target instance number into multiple incremental adjustment steps, and dynamically adjust the batch size of the AI ​​model in each adjustment step;

[0022] Based on the Kubernetes extension mechanism, it performs atomic adjustments to the batch parameters and number of instances of AI model services, tracks the status of the scheduling process, and triggers configuration rollback operations when execution anomalies are detected.

[0023] The beneficial effects of the present invention are as follows:

[0024] (1) This invention innovatively proposes a two-layer analysis mechanism, which analyzes GPU stability and memory efficiency as independent dimensions, effectively solving the problem of one-sided decision-making caused by focusing on only a single indicator in traditional solutions. Through the collaborative analysis of the GPU stability curve and the memory efficiency curve, more accurate resource scheduling decisions are achieved.

[0025] (2) The present invention adopts a progressive adjustment strategy and an atomic adjustment mechanism, combined with perfect state tracking and rollback protection, which significantly improves the stability of the service expansion and contraction process. Progressive adjustment effectively avoids service shocks, while atomic adjustment ensures configuration consistency.

[0026] (3) The present invention maximizes resource utilization efficiency while ensuring service quality through refined batch processing parameter optimization and dual threshold constraint mechanism. The system can adaptively adjust batch processing parameters according to load characteristics, achieving efficient utilization of computing resources.

[0027] (4) The present invention is deeply integrated with the cloud-native architecture and provides complete lifecycle management capabilities through extended support for Kubernetes, significantly reducing operation and maintenance costs and improving system reliability and maintainability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An architecture diagram of the elastic scaling system for cloud-native AI model services.

[0029] Figure 2 Figure 2 is a workflow diagram of the two-layer analysis mechanism. DETAILED DESCRIPTION

[0030] In order to explain the working principle of the present invention, the technical solution of the present invention is described in detail below in conjunction with specific implementation methods.

[0031] The operating environment of this embodiment is an AI model service platform based on cloud native technology. The core of the platform includes two major infrastructures: the data center GPU manager and the Kubernetes container orchestration system. Among them, the data center GPU manager is responsible for the unified management and monitoring of GPU resources, and provides standardized interfaces for the collection of key indicators such as GPU utilization, video memory occupancy, and memory bandwidth; the Kubernetes container orchestration system provides an extension mechanism to support the deployment and lifecycle management of AI model service instances.

[0032] Preferably, the platform integrates GPU device plug-ins, provides GPU virtualization capabilities through NVIDIA Container Runtime, and realizes fine-grained allocation of GPU resources. Preferably, the platform also deploys the Prometheus monitoring system to provide highly reliable indicator collection support for computing resource monitors. Preferably, the system adopts a custom controller mode to implement the core logic of the model feature analysis module and the elastic policy execution module.

[0033] Preferably, this embodiment uses Kubernetes v1.20 as the basic version of the container orchestration platform and configures CUDA 11.0 as the GPU computing framework. Preferably, the platform also provides a standardized model service runtime, supports dynamic adjustment of batch processing parameters, and ensures the smooth operation of the AI ​​model service. The system adopts a layered architecture design to achieve a complete control process from monitoring data collection, feature analysis to elastic policy execution.

[0034] Each component communicates through a standardized interface. Preferably, a gRPC-based microservice architecture is used to achieve efficient data transmission between components. Preferably, the system provides a complete monitoring and alarm mechanism to promptly notify operation and maintenance personnel when key indicators are abnormal.

[0035] like Figure 1 As shown, the system implementation of this embodiment includes two core parts: the model feature analysis module and the elastic strategy execution module. In combination with the characteristics of AI model services, the system adopts a layered architecture design to achieve a complete closed-loop control process from data collection, feature analysis to strategy execution.

[0036] In the model feature analysis module, the computing resource monitor is deeply integrated with the data center GPU manager through the basic monitoring layer to collect key indicator data during the operation of the AI ​​model in real time. Specifically, the basic monitoring layer continuously collects three core data types: GPU utilization, video memory occupancy, and memory bandwidth through the standardized GPU manager interface. The collected raw data then enters the indicator preprocessing layer, which implements a complete data processing pipeline. Preferably, the preprocessing layer calculates the moving average with a time granularity of 15 seconds, and at the same time identifies and marks abnormal data points in combination with the distribution characteristics of historical data to ensure that the data transmitted to the batch efficiency analyzer has a high degree of credibility.

[0037] The batch efficiency analyzer performs in-depth analysis of the preprocessed monitoring data based on the set dual-threshold constraint analysis mechanism. The system first performs dynamic feature extraction on the GPU utilization data, calculates its variance value within the sliding time window, and combines trend analysis to identify the fluctuation pattern of utilization. When the variance value is detected to exceed the threshold pre-set according to the AI ​​model calculation density, or a frequent fluctuation trend is identified, the system will mark the current batch parameter as unstable. For parameters that are judged to be stable, the analyzer further calculates the degree of deviation of the video memory occupancy and memory bandwidth indicators relative to their respective target intervals. The system encapsulates these analysis results into a structured analysis data packet, which contains the current batch parameter value, GPU utilization variance value, GPU stable state mark, and deviation data of two memory-related indicators. Preferably, the generation cycle of the analysis data packet is 30 seconds to balance the analysis accuracy and system overhead.

[0038] The efficiency curve analysis system receives and processes the analysis data packets output by the batch efficiency analyzer, and constructs two characteristic curves through the sliding window averaging technology. The GPU stability curve uses the batch processing parameters as the horizontal axis and the GPU utilization variance value as the vertical axis, which intuitively shows the operating stability of the system under different parameter configurations; the memory efficiency curve also uses the batch processing parameters as the horizontal axis, and the vertical axis reflects the comprehensive deviation of the video memory occupancy rate and memory bandwidth. The system first uses the GPU stability curve to determine the batch processing parameter range for stable operation based on the preset variance threshold. On this basis, the parameter point with the smallest deviation is selected as the target batch processing parameter by analyzing the memory efficiency curve. When there are multiple candidate parameters with similar deviations, the system follows the minimization principle and gives priority to parameter configurations with smaller values.

[0039] After receiving the target batch parameters output by the efficiency curve analysis system, the scaling decision maker enters the execution plan generation phase. First, the required instance scale change target value is calculated based on the current system load and the target parameters. Taking into account the service stability requirements, the decision maker decomposes the scale change operation into a series of progressive adjustment steps, each of which contains a clear batch parameter configuration. Preferably, before executing each adjustment step, the system will record the GPU utilization and inference latency indicators of the current service instance in real time, and dynamically adjust the change step size based on the gap between the GPU utilization and the expected target value. When it is detected that the inference latency exceeds the preset threshold, the system will automatically suspend the execution of the current adjustment step to ensure service quality.

[0040] As the execution layer of the system, the resource scheduler achieves precise control over the execution plan output by the scaling decision maker through deep integration with the Kubernetes extension mechanism. The scheduler uses atomic operations to ensure that changes in batch parameters and the number of instances have transactional characteristics to avoid configuration inconsistencies. At the same time, the scheduler implements a complete state tracking mechanism to monitor the scheduling process of each execution step in real time. When an execution anomaly is detected, the system can quickly trigger a configuration rollback operation to restore the service status to the last stable point, thereby ensuring the reliability of the service.

[0041] like Figure 2 As shown, the two-layer analysis mechanism of the present invention first determines the available range of batch processing parameters through GPU stability analysis, and then selects the optimal parameters based on memory efficiency analysis. In the GPU stability analysis stage, the system uses a variance calculation method to determine the stable state of its GPU utilization for each batch processing parameter value. Specifically, for the collected GPU utilization sequence U = {u, u, ..., u}, its short-term variance value is calculated, and the formula is:

[0042] Var(U)=E[(U-μ) 2 ]

[0043] Where μ is the mean of the sequence U, Var(U) is the variance value of GPU utilization, which is used to measure the fluctuation of GPU utilization, μ is the mean of the GPU utilization sequence, and n is the number of sampling points, which is determined by the monitoring sampling frequency of the system. At the same time, the system identifies frequent fluctuations by analyzing the continuous change trend of GPU utilization. When the variance value exceeds the preset threshold for the calculation density of the AI ​​model or a frequent fluctuation trend is detected, the system will automatically mark the current batch parameters as unstable.

[0044] After determining the stable batch parameter range, the system enters the memory efficiency analysis phase. In this phase, the target interval deviations of two key indicators, memory occupancy and memory bandwidth, are calculated. For memory occupancy, the system calculates the difference between the current memory occupancy and the median of the interval based on the preset target interval to obtain a specific deviation value. Similarly, the system also calculates the degree of deviation of the actual usage of memory bandwidth relative to its target interval. This deviation-based evaluation method can effectively reflect the system's memory resource utilization efficiency under different batch parameters.

[0045] In the final parameter selection stage, the system first uses the GPU stability curve to screen out the batch parameter range with variance values ​​below the preset threshold. Then, within the stable parameter range, the batch parameter with the smallest deviation is identified based on the memory efficiency curve. Preferably, when there are multiple candidate parameters with similar deviations, the system will adopt a progressive search strategy to give priority to batch parameters with smaller values ​​while ensuring stability, so as to reduce system resource overhead. This two-layer analysis mechanism ensures that the selected batch parameters can not only ensure GPU operation stability, but also achieve high memory resource utilization efficiency.

[0046] The above embodiments are only used to illustrate the technical solution of the present invention and should not be regarded as limiting the present invention. Those skilled in the art can make various modifications and improvements without departing from the technical solution of the present invention, and these modifications and improvements should also be regarded as the protection scope of the present invention.

Claims

1. A cloud-native AI model service elastic scaling system, characterized in that: Includes model feature analysis module and elastic strategy execution module; Among them, the model feature analysis module includes a computing resource monitor, a batch efficiency analyzer and an efficiency curve analysis system. The computing resource monitor is integrated with the data center GPU manager and transmits the collected GPU utilization, video memory occupancy and memory bandwidth data during the AI ​​model runtime to the batch efficiency analyzer. The batch efficiency analyzer analyzes the AI ​​model reasoning batch efficiency according to the GPU utilization fluctuation characteristics and memory efficiency indicators and transmits the evaluation results to the efficiency curve analysis system. The efficiency curve analysis system constructs a GPU stability curve and a memory efficiency curve according to the evaluation results transmitted by the batch efficiency analyzer and outputs the target batch parameters adapted to the AI ​​model reasoning service to the elastic policy execution module; Among them, the elastic policy execution module includes a scaling decision maker and a resource scheduler. The scaling decision maker receives the target batch processing parameters transmitted by the efficiency curve analysis system and converts them into an execution plan for the AI ​​model service instance. The resource scheduler receives the execution plan output by the scaling decision maker and integrates the Kubernetes extension mechanism to execute the change operation of the AI ​​model service instance scale and batch processing parameters.

2. According to the cloud-native AI model service elastic scaling system according to claim 1, it is characterized in that: The computing resource monitor includes a basic monitoring layer and an indicator preprocessing layer; The basic monitoring layer collects GPU utilization, video memory occupancy and memory bandwidth data in real time through the data center GPU manager interface; The index preprocessing layer cleans the collected data, calculates the moving average, and marks abnormal data points before transmitting them to the batch processing efficiency analyzer.

3. According to the cloud-native AI model service elastic scaling system according to claim 1, it is characterized in that: The batch processing efficiency analyzer includes a batch processing efficiency analyzer, which sets an analysis mechanism with dual threshold constraints, receives GPU utilization, video memory occupancy and memory bandwidth data collected by the computing resource monitor, calculates the short-term variance value and change trend curve of the GPU utilization, marks the current batch processing parameter as unstable when the GPU utilization variance exceeds a preset threshold or shows a frequent fluctuation trend, further calculates the target interval deviation of the video memory occupancy and memory bandwidth when the GPU utilization variance value is within a preset threshold range, generates an analysis data packet, and the analysis data packet includes the current batch processing parameter, GPU utilization variance value, GPU stable state mark, video memory occupancy deviation and memory bandwidth deviation, and periodically outputs the analysis data packet to the efficiency curve analysis system for constructing a GPU stability curve and a memory efficiency curve; Among them, the GPU utilization variance threshold is set according to the computing density of the AI ​​model, and the target interval deviation is obtained by calculating the difference between the current video memory occupancy and the median of the preset target interval.

4. According to the cloud-native AI model service elastic scaling system according to claim 1, it is characterized in that: The invention comprises an efficiency curve analysis system, wherein the efficiency curve analysis system receives an analysis data packet output by a batch processing efficiency analyzer, and constructs a GPU stability curve and a memory efficiency curve by using a sliding window averaging technique, wherein the GPU stability curve takes a batch processing parameter as a horizontal coordinate and a GPU utilization variance value as a vertical coordinate, and the memory efficiency curve takes a batch processing parameter as a horizontal coordinate and a deviation of a video memory occupancy rate and a memory bandwidth as a vertical coordinate. The efficiency curve analysis system screens out a batch processing parameter range whose GPU utilization variance value is lower than a preset threshold value through the GPU stability curve, selects a batch processing parameter with the smallest deviation within the range as a target batch processing parameter based on the memory efficiency curve, and preferentially selects a batch processing parameter with a smaller value when there are multiple candidate parameters with similar deviations, and outputs the target batch processing parameter to the scaling decision maker.

5. According to the cloud-native AI model service elastic scaling system according to claim 1, it is characterized in that: The method comprises a scaling decision maker, wherein the scaling decision maker receives the target batch processing parameter output by the efficiency curve analysis system, calculates the target value of the instance scale change, decomposes the change of the target number of instances into multiple progressive adjustment steps, dynamically adjusts the batch processing size by parameter mapping rules in each adjustment step, ensures a smooth transition of services in combination with container lifecycle management, and outputs the decomposed execution step sequence and the corresponding batch processing parameter configuration to the resource scheduler; Among them, the progressive adjustment step is to record the GPU utilization and inference delay indicators of the current service instance before executing each adjustment step, determine the adjustment step size according to the difference between the GPU utilization and the preset utilization target, and when the inference delay exceeds the preset threshold, suspend the execution of the current adjustment step.

6. The cloud-native AI model service elastic scaling system according to claim 1, characterized in that: It includes a resource scheduler, which runs a Kubernetes extension mechanism to achieve docking with the execution step sequence output by the scaling decision maker, realize atomic adjustment of batch parameters and the number of instances, track the status of the scheduling process of each execution step, and trigger a configuration rollback operation to restore to the previous stable state when an execution anomaly is detected.

7. A cloud-native AI model service elastic scaling method, characterized in that include: Collect GPU utilization, video memory occupancy, and memory bandwidth data in real time when the AI ​​model is running, clean the collected data, calculate the moving average, and mark abnormal data points; Calculate the short-term variance value and trend curve of GPU utilization during AI model inference, determine the GPU stable state, calculate the target interval deviation of video memory occupancy and memory bandwidth, and generate an analysis data package containing batch processing parameters, GPU utilization variance value, GPU stable state mark, video memory occupancy deviation, and memory bandwidth deviation; The sliding window averaging technology is used to construct the GPU stability curve and the memory efficiency curve. Based on the GPU stability curve, the batch parameter range in which the GPU utilization variance value required for the AI ​​model service is lower than the preset threshold is screened out. Within this range, the batch parameter with the smallest deviation is selected as the target batch parameter according to the memory efficiency curve. Convert the target batch parameter into the target value of the AI ​​model service instance scale change, decompose the change of the target instance number into multiple incremental adjustment steps, and dynamically adjust the batch size of the AI ​​model in each adjustment step; Based on the Kubernetes extension mechanism, it performs atomic adjustments to the batch parameters and number of instances of AI model services, tracks the status of the scheduling process, and triggers configuration rollback operations when execution anomalies are detected.

Images