A cloud-native AI model service elasticity scaling system and method

Abstract

The application discloses a kind of AI model service elasticity scaling system and method based on cloud native, including model feature analysis module and elasticity strategy execution module. Among them, model feature analysis module passes through computing resource monitor and acquires GPU running data, utilizes batch processing efficiency analyzer to construct double threshold constraint mechanism, and is generated GPU stability curve and memory efficiency curve by efficiency curve analysis system;Elasticity strategy execution module converts target batch processing parameter into execution plan and realizes atomicity adjustment by Kubernetes extension mechanism.The application innovatively proposes double-layer analysis mechanism, analyzes GPU stability and memory efficiency as independent dimension, effectively solves the service shock problem caused by only paying attention to single index in traditional scheme. Through gradual adjustment strategy and perfect state tracking mechanism, the precise expansion of AI model service is realized, the resource utilization efficiency is significantly improved, and the stability of service quality is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to an AI model service elastic scaling system and method based on cloud native. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the deployment scale of AI model services based on deep learning in the cloud computing environment is continuously expanding. Especially in the fields of image processing and natural language processing, AI model services need to handle significantly fluctuating business loads, which puts higher requirements on the dynamic scheduling of computing resources. In order to improve resource utilization efficiency and guarantee service quality, AI model services need to adjust resources dynamically according to load changes. Although the mainstream cloud native platform (such as Kubernetes) provides basic automatic scaling mechanisms, these general mechanisms still have many limitations in the computing-intensive scenario of AI model services.

[0003] The existing scaling technology mainly has the following deficiencies: first, the traditional scheme often only focuses on a single indicator (such as GPU utilization or video memory occupancy), lacks comprehensive consideration of multi-dimensional resource characteristics such as GPU computing efficiency, memory bandwidth, and batch processing efficiency in AI model services, which leads to the lack of comprehensiveness and accuracy of scaling decisions. Second, the existing automatic scaling scheme excessively relies on a simple threshold triggering mechanism, which can easily cause frequent scaling operations when the load fluctuates greatly, causing service shocks and affecting business stability. Third, most schemes lack a fine-grained parameter optimization mechanism and cannot dynamically adjust batch processing parameters according to the computing characteristics of AI models, which not only causes low resource utilization efficiency, but also may cause service response delay to exceed expectations. Fourth, during the scaling process, due to the lack of a perfect state tracking and rollback mechanism, once an abnormal situation occurs, it is easy to affect the availability of online services, and even cause service interruption.

[0004] At the same time, AI model services in the cloud native environment also face other challenges: first, the problem of resource fragmentation, there may be resource competition between different sizes of service instances, affecting overall performance; second, the problem of configuration consistency, how to ensure the consistency of batch processing parameters and resource configuration during the scaling of service instances is also an important challenge.

[0005] Therefore, how to design a system that can comprehensively consider GPU computing efficiency, memory resource utilization, service stability and other multiple dimensions, and can realize accurate scaling and parameter adaptive optimization, is a technical problem that needs to be solved in the current cloud native AI model service field. Especially under the premise of ensuring service stability, the maximization of resource utilization efficiency, this technical problem needs an innovative solution. SUMMARY

[0006] The problem to be solved by the present application is how to provide a cloud-native AI model service elastic scaling system and method that can simultaneously ensure service stability and resource utilization efficiency. To solve the above technical problems, the technical solutions adopted by the present application are:

[0007] An AI model service elastic scaling system based on cloud native, characterized by comprising a model feature analysis module and an elastic policy execution module.

[0008] The model feature analysis module comprises a computing resource monitor, a batch processing efficiency analyzer, and an efficiency curve analysis system. The computing resource monitor is integrated with a data center GPU manager and transmits collected GPU utilization rate, video memory occupancy rate, and memory bandwidth data of the AI model runtime to the batch processing efficiency analyzer. The batch processing efficiency analyzer analyzes the AI model inference batch processing efficiency according to the GPU utilization rate fluctuation characteristics and memory efficiency indicators and transmits the evaluation results to the efficiency curve analysis system. The efficiency curve analysis system constructs the GPU stability curve and memory efficiency curve according to the evaluation results transmitted by the batch processing efficiency analyzer and outputs the target batch processing parameters for adapting the AI model inference service to the elastic policy execution module.

[0009] The elastic policy execution module comprises a scaling decision maker and a resource scheduler. The scaling decision maker receives the target batch processing parameters transmitted by the efficiency curve analysis system and converts them into an execution plan for the AI model service instance. The resource scheduler receives the execution plan output by the scaling decision maker and integrates the Kubernetes extension mechanism to perform the change operation of the AI model service instance scale and batch processing parameters.

[0010] Further technical solutions are that the computing resource monitor comprises a basic monitoring layer and an index preprocessing layer. The basic monitoring layer collects GPU utilization rate, video memory occupancy rate, and memory bandwidth data in real time through the data center GPU manager interface.

[0011] The index preprocessing layer performs data cleaning, calculates the moving average, and marks abnormal data points on the collected data before delivering them to the batch processing efficiency analyzer.

[0012] Further technical solutions are as follows: the batch processing efficiency analyzer sets a double-threshold constraint analysis mechanism, receives GPU utilization, video memory occupancy and memory bandwidth data collected by the computing resource monitor, calculates a short-term variance value and a change trend curve of the GPU utilization, marks the current batch processing parameter as unstable when the GPU utilization variance exceeds a preset threshold or presents a frequent fluctuation trend, further calculates a target interval deviation of the video memory occupancy and memory bandwidth when the GPU utilization variance value is within a preset threshold range, the batch processing efficiency analyzer generates an analysis data packet, the analysis data packet contains the current batch processing parameter, the GPU utilization variance value, the GPU stability state mark, the video memory occupancy deviation and the memory bandwidth deviation, and periodically outputs the analysis data packet to the efficiency curve analysis system for constructing a GPU stability curve and a memory efficiency curve;

[0013] The GPU utilization variance threshold is set according to the calculation density of the AI model, and the target interval deviation is obtained by calculating the difference between the current video memory occupancy and a preset target interval median.

[0014] Further technical solutions are as follows: the efficiency curve analysis system receives the analysis data packet output by the batch processing efficiency analyzer, and constructs a GPU stability curve and a memory efficiency curve using a sliding window average technique, wherein the GPU stability curve takes the batch processing parameter as the horizontal coordinate and the GPU utilization variance value as the vertical coordinate, and the memory efficiency curve takes the batch processing parameter as the horizontal coordinate and the deviation of the video memory occupancy and memory bandwidth as the vertical coordinate, the efficiency curve analysis system selects a batch processing parameter range with a GPU utilization variance value below a preset threshold through the GPU stability curve, selects a batch processing parameter with the smallest deviation as a target batch processing parameter based on the memory efficiency curve in the range, and when there are multiple candidate parameters with similar deviations, preferentially selects a batch processing parameter with a smaller value, and outputs the target batch processing parameter to the scaling decision maker. Further technical solutions are as follows: the scaling decision maker receives the target batch processing parameter output by the efficiency curve analysis system, calculates a target value of instance scale change, decomposes the change of the target instance number into multiple gradual adjustment steps, dynamically adjusts the batch size in each adjustment step through a parameter mapping rule, and ensures smooth service transition by combining container lifecycle management, outputs the decomposed execution step sequence and corresponding batch processing parameter configuration to the resource scheduler;

[0015] In each adjustment step, the GPU utilization and inference delay indicators of the current service instance are recorded before the execution of each adjustment step, the adjustment step size is determined according to the difference between the GPU utilization and a preset utilization target, and when the inference delay exceeds a preset threshold, the execution of the current adjustment step is paused.

[0016] Further, the technical solutions are as follows: the resource scheduler runs a Kubernetes extension mechanism to realize docking with the execution step sequence output by the scaling decision maker, realize atomic adjustment of the batch processing parameters and the instance number, track the state of the scheduling process of each execution step, and trigger a configuration rollback operation to restore to the last stable state when an execution exception is detected.

[0017] The application further discloses a cloud-native AI model service elastic scaling method, characterized by comprising the following steps:

[0018] Real-time collection of GPU utilization rate, video memory occupancy rate and memory bandwidth data during AI model running, cleaning of the collected data, calculation of moving average and marking of abnormal data points;

[0019] Calculation of GPU utilization rate short-term variance value and change trend curve in the AI model inference process to determine the GPU stable state, calculation of the target interval deviation of the video memory occupancy rate and the memory bandwidth, and generation of an analysis data packet containing the batch processing parameters, the GPU utilization rate variance value, the GPU stable state mark, the video memory occupancy rate deviation and the memory bandwidth deviation;

[0020] Construction of a GPU stability curve and a memory efficiency curve using a sliding window average technique, selection of a batch processing parameter range in which the GPU utilization rate variance value required by the AI model service is lower than a preset threshold based on the GPU stability curve, and selection of a batch processing parameter with the smallest deviation as a target batch processing parameter according to the memory efficiency curve in the range;

[0021] Conversion of the target batch processing parameter into a target value of AI model service instance scale change, and decomposition of the target instance number change into multiple gradual adjustment steps, dynamic adjustment of the batch processing size of the AI model in each adjustment step;

[0022] Atomic adjustment of the batch processing parameters and the instance number of the AI model service based on a Kubernetes extension mechanism, tracking of the scheduling process, and triggering of a configuration rollback operation when an execution exception is detected.

[0023] The application has the following advantages:

[0024] (1) The application innovatively proposes a double-layer analysis mechanism, analyzes GPU stability and memory efficiency as independent dimensions, effectively solves the problem of one-sidedness in decision-making caused by only focusing on a single indicator in traditional schemes, and realizes more accurate resource scheduling decisions through collaborative analysis of the GPU stability curve and the memory efficiency curve.

[0025] (2) The application adopts a gradual adjustment strategy and an atomic adjustment mechanism, combined with perfect state tracking and rollback protection, to significantly improve the stability of the service expansion and contraction process. The gradual adjustment effectively avoids service shock, and the atomic adjustment ensures the consistency of the configuration.

[0026] (3) The application maximizes resource utilization efficiency while ensuring service quality through fine batch processing parameter optimization and a double threshold constraint mechanism. The system can adaptively adjust batch processing parameters according to load characteristics, achieving efficient utilization of computing resources.

[0027] (4) The application is deeply integrated with the cloud native architecture, provides complete life cycle management capabilities through Kubernetes extension support, significantly reduces operation and maintenance costs, and improves system reliability and maintainability. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The figure is an AI model service elastic scaling system architecture diagram based on cloud native.

[0029] Figure 2 The figure is a workflow diagram of the double-layer analysis mechanism. DETAILED DESCRIPTION

[0030] In order to facilitate the description of the working principle of the application, the technical solutions of the application will be described in detail below in combination with specific embodiments.

[0031] The running environment of the embodiment is an AI model service platform based on cloud native technology. The core of the platform includes two major infrastructures: a data center GPU manager and a Kubernetes container orchestration system. The data center GPU manager is responsible for the unified management and monitoring of GPU resources, and provides standardized interfaces for the collection of key indicators such as GPU utilization, video memory occupancy, and memory bandwidth; the Kubernetes container orchestration system provides an extension mechanism to support the deployment and life cycle management of AI model service instances.

[0032] Preferably, the platform integrates a GPU device plug-in, provides GPU virtualization capabilities through NVIDIA Container Runtime, and realizes fine-grained allocation of GPU resources. Preferably, the platform also deploys a Prometheus monitoring system to provide high-reliability indicator collection support for the computing resource monitor. Preferably, the system uses a custom controller mode to realize the core logic of the model feature analysis module and the elastic policy execution module.

[0033] Preferably, the present embodiment adopts Kubernetes v1.20 as the base version of the container orchestration platform, and configures CUDA 11.0 as the GPU computing framework. Preferably, the platform also provides a standardized model service runtime, supports dynamic adjustment of batch parameters, and ensures smooth operation of AI model services. The system adopts a hierarchical architecture design, realizing a complete control process from monitoring data collection, feature analysis to elastic policy execution.

[0034] The components communicate through standardized interfaces, and preferably, a microservice architecture based on gRPC is used to realize efficient data transmission between components. Preferably, the system provides a complete monitoring and alarm mechanism, which can notify the operation and maintenance personnel in time when the key indicators are abnormal.

[0035] As shown in Figure 1 The system of the present embodiment includes a model feature analysis module and an elastic policy execution module. The system adopts a hierarchical architecture design, realizing a complete closed-loop control process from data collection, feature analysis to policy execution.

[0036] In the model feature analysis module, the computing resource monitor is deeply integrated with the data center GPU manager through the basic monitoring layer, and real-time key indicator data of the AI model runtime is collected. Specifically, the basic monitoring layer continuously collects three types of core data, GPU utilization, memory occupancy, and memory bandwidth, through the standardized GPU manager interface. The raw data collected is then sent to the indicator preprocessing layer, which realizes a complete data processing pipeline. Preferably, the preprocessing layer calculates the moving average with a time granularity of 15 seconds, and identifies and labels abnormal data points based on historical data distribution characteristics, ensuring that the data transmitted to the batch efficiency analyzer has high credibility.

[0037] The batch efficiency analyzer analyzes the preprocessed monitoring data based on a set of double-threshold constraint analysis mechanisms. The system first extracts the dynamic features of the GPU utilization data, calculates the variance value within the sliding time window, and identifies the fluctuation pattern of the utilization rate based on trend analysis. When the variance value exceeds the threshold set according to the AI model calculation density, or a frequent fluctuation trend is identified, the system will mark the current batch parameter as unstable. For parameters determined to be in a stable state, the analyzer further calculates the deviation of the memory occupancy and memory bandwidth indicators from their respective target intervals. The system encapsulates these analysis results into a structured analysis data packet, which contains the current batch parameter value, GPU utilization variance value, GPU stability state label, and deviation data of the two memory-related indicators. Preferably, the generation period of the analysis data packet is 30 seconds, to balance the analysis accuracy and system overhead.

[0038] The efficiency curve analysis system receives and processes the analysis data packet output by the batch efficiency analyzer, and constructs two characteristic curves through a sliding window average technique. The GPU stability curve takes the batch parameter as the abscissa and the GPU utilization variance value as the ordinate, directly showing the running stability of the system under different parameter configurations; the memory efficiency curve also takes the batch parameter as the abscissa, and the ordinate reflects the comprehensive deviation of the video memory occupancy rate and the memory bandwidth. The system first determines the batch parameter range for stable operation based on the preset variance threshold value by using the GPU stability curve. On this basis, the parameter point with the minimum deviation degree is selected as the target batch parameter by analyzing the memory efficiency curve. When there are multiple candidate parameters with similar deviation degrees, the system preferentially selects the parameter configuration with a smaller value according to the minimization principle.

[0039] After receiving the target batch parameter output by the efficiency curve analysis system, the scaling decision maker enters the generation stage of the execution plan. First, the required instance scale change target value is calculated based on the current system load and the target parameter. Considering the service stability requirement, the decision maker decomposes the scale change operation into a series of progressive adjustment steps, each of which contains an explicit batch parameter configuration. Preferably, the system records the GPU utilization rate and inference delay indicators of the current service instance in real time before executing each adjustment step, and dynamically adjusts the change step size according to the difference between the GPU utilization rate and the expected target value. When the inference delay is detected to exceed the preset threshold, the system will automatically pause the execution of the current adjustment step to ensure service quality.

[0040] As the execution layer of the system, the resource scheduler realizes accurate control of the execution plan output by the scaling decision maker through deep integration with the Kubernetes extension mechanism. The scheduler uses atomic operation mode to ensure that the changes of batch parameters and instance number have transaction characteristics, avoiding inconsistent configurations. At the same time, the scheduler realizes a complete state tracking mechanism, which monitors the scheduling process of each execution step in real time. When an execution exception is detected, the system can quickly trigger a configuration rollback operation to restore the service state to the last stable point, thereby ensuring the reliability of the service.

[0041] As shown in Figure 2 The double-layer analysis mechanism of the present application first determines the available range of batch parameters through GPU stability analysis, and then selects the optimal parameter based on memory efficiency analysis. In the GPU stability analysis stage, the system calculates the stability state of the GPU utilization rate for each batch parameter value using the variance calculation method. Specifically, for the collected GPU utilization rate sequence U = {u, u,..., u}, the short-term variance value is calculated, and the formula is:

[0042] Var(U) = E[(U - μ) 2 ]

[0043] wherein μ is the mean value of sequence U, Var(U) is the variance value of GPU utilization, used to measure the fluctuation degree of GPU usage, μ is the mean value of GPU utilization sequence, and n is the number of sampling points, determined by the monitoring sampling frequency of the system. At the same time, the system identifies the frequent fluctuation situation by analyzing the continuous change trend of GPU utilization. When the variance value exceeds the threshold preset for AI model calculation density or frequent fluctuation trend is detected, the system will automatically mark the current batch processing parameter as unstable state.

[0044] After determining the stable batch processing parameter range, the system enters the memory efficiency analysis stage. This stage calculates the target interval deviation degree of two key indicators of memory occupancy rate and memory bandwidth respectively. For memory occupancy rate, the system calculates the difference between the current memory occupancy rate and the median of the preset target interval, thereby obtaining the specific deviation degree value. Similarly, the system also calculates the deviation degree of the actual use of memory bandwidth relative to its target interval. This deviation degree-based evaluation method can effectively reflect the memory resource utilization efficiency of the system under different batch processing parameters.

[0045] In the final parameter selection stage, the system first uses the GPU stability curve to filter out the batch processing parameter range with a variance value below the preset threshold. Then, within this stable parameter range, the system identifies the batch processing parameter with the minimum deviation degree based on the memory efficiency curve. Preferably, when there are multiple candidate parameters with close deviation degrees, the system will adopt a gradual search strategy to preferentially select the batch processing parameter with smaller numerical value under the premise of ensuring stability, so as to reduce the system resource overhead. This double-layer analysis mechanism ensures that the selected batch processing parameter can not only guarantee the stability of GPU operation, but also achieve high memory resource utilization efficiency.

[0046] The above embodiments are only used to illustrate the technical solutions of the present application, and should not be regarded as limitations of the present application. Those skilled in the art can make various modifications and improvements to the technical solutions of the present application without departing from the scope of the present application, and these modifications and improvements should also be regarded as the protection scope of the present application.

Claims

1.A cloud-native AI model service elasticity scaling system, characterized in that, The model feature analysis module and the elastic policy execution module are included. The model feature analysis module includes a computing resource monitor, a batch processing efficiency analyzer, and an efficiency curve analysis system. The elastic policy execution module includes a scaling decision maker and a resource scheduler. The batch processing efficiency analyzer sets a double-threshold constraint analysis mechanism, receives GPU utilization rate, video memory occupancy rate, and memory bandwidth data collected by the computing resource monitor, calculates the short-term variance value and the trend curve of the GPU utilization rate, and marks the current batch processing parameter as unstable when the GPU utilization rate variance exceeds the preset threshold or presents a frequent fluctuation trend. The batch processing efficiency analyzer further calculates the target interval deviation of the video memory occupancy rate and the memory bandwidth when the GPU utilization rate variance is within the preset threshold range. The batch processing efficiency analyzer generates an analysis data packet containing the current batch processing parameter, the GPU utilization rate variance, the GPU stability state marker, the video memory occupancy rate deviation, and the memory bandwidth deviation, and periodically outputs the analysis data packet to the efficiency curve analysis system for constructing the GPU stability curve and the memory efficiency curve. The threshold of the GPU utilization rate variance is set according to the computing density of the AI model, and the target interval deviation is obtained by calculating the difference between the current video memory occupancy rate and the preset target interval median. The efficiency curve analysis system receives the analysis data packet output by the batch processing efficiency analyzer, uses the sliding window average technique to construct the GPU stability curve and the memory efficiency curve, and selects the batch processing parameter range with a GPU utilization rate variance below the preset threshold through the GPU stability curve. The efficiency curve analysis system selects the batch processing parameter with the smallest deviation as the target batch processing parameter based on the memory efficiency curve within the batch processing parameter range with a GPU utilization rate variance below the preset threshold. When there are multiple candidate parameters with similar deviation values, the batch processing parameter with a smaller numerical value is preferred. The target batch processing parameter is output to the scaling decision maker. The scaling decision maker receives the target batch processing parameter output by the efficiency curve analysis system and converts it into an execution plan of the AI model service instance, calculates the target value of instance scale change, decomposes the change of the target instance number into multiple gradual adjustment steps, dynamically adjusts the batch size in each adjustment step through parameter mapping rules, and combines container life cycle management to ensure smooth transition of the service, and outputs the decomposed execution step sequence and corresponding batch processing parameter configuration to the resource scheduler; wherein, before each adjustment step is executed, the GPU utilization rate and inference delay indicators of the current service instance are recorded, the adjustment step is determined according to the difference between the GPU utilization rate and the preset utilization rate target, and when the inference delay exceeds the preset threshold, the execution of the current adjustment step is paused; The resource scheduler receives the execution plan output by the scaling decision maker and integrates the Kubernetes extension mechanism to perform the change operation of the AI model service instance scale and batch processing parameter, runs the Kubernetes extension mechanism, realizes the docking with the execution step sequence output by the scaling decision maker, realizes the atomic adjustment of the batch processing parameter and the instance number, and tracks the state of the scheduling process of each execution step. When an execution exception is detected, a configuration rollback operation is triggered to restore to the last stable state. 2.The cloud-native AI model service elasticity system of claim 1, wherein, The computing resource monitor includes a basic monitoring layer and an indicator preprocessing layer; The basic monitoring layer collects GPU utilization rate, video memory occupancy rate and memory bandwidth data in real time through a data center GPU manager interface; The indicator preprocessing layer cleans the collected data, calculates the moving average and marks the abnormal data points, and then delivers them to the batch processing efficiency analyzer. 3.A method for elastic scaling of a cloud-native AI model service, characterized in that, The method is applied to the cloud-native AI model service elastic scaling system of claim 1 or 2, and the method comprises: Real-time collection of GPU utilization rate, video memory occupancy rate and memory bandwidth data during AI model runtime, and cleaning, calculation of moving average and marking of abnormal data points of the collected data; Calculation of GPU utilization rate short-term variance value and change trend curve in the AI model inference process, judgment of GPU stable state, calculation of target interval deviation of video memory occupancy rate and memory bandwidth, and generation of an analysis data packet containing batch processing parameters, GPU utilization rate variance value, GPU stable state marker, video memory occupancy rate deviation and memory bandwidth deviation; Using the sliding window average technique to build a GPU stability curve and a memory efficiency curve, selecting a batch processing parameter range with a GPU utilization rate variance value lower than a preset threshold based on the GPU stability curve, and selecting a batch processing parameter with the smallest deviation as the target batch processing parameter in the range according to the memory efficiency curve; Convert the target batch processing parameter into a target value of AI model service instance scale change, and decompose the change of the target instance number into multiple gradual adjustment steps, and dynamically adjust the batch size of the AI model in each adjustment step; The atomic adjustment of the batch processing parameters and the instance number of the AI model service is performed based on a Kubernetes extension mechanism, state tracking is performed on the scheduling process, and a configuration rollback operation is triggered when an execution exception is detected.

Images