Method for implementing inference service instance scaling, electronic device, and storage medium

Abstract

This invention provides a method, electronic device, and storage medium for scaling up and down inference service instances, belonging to the field of artificial intelligence technology. The method includes: predicting future business load based on historical operational data of the target inference service to generate proactive scaling up / down instance decisions; evaluating the current operational status based on real-time operational data of the target inference service to generate passive scaling up / down instance decisions; collaboratively deciding on the proactive and passive scaling up / down instance decisions to determine the final scaling up / down instance instruction; and adjusting the number of instances of the target inference service according to the final scaling up / down instance instruction. This invention establishes a dual-engine collaborative mechanism combining proactive prediction and passive response. While utilizing historical data for forward-looking scaling up / down to reduce latency, it also utilizes real-time data for fallback correction to cope with sudden load increases. This effectively solves the problem of delayed response or resource waste associated with single scaling up / down methods, significantly improving the resource utilization and service quality stability of inference services.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for scaling up and down inference service instances, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of 6th Generation Wireless Systems (6G), the network environment is characterized by high dynamism, ultra-low latency, high bandwidth, and wide connectivity. Against this backdrop, various Artificial Intelligence (AI) inference services, such as autonomous driving, Augmented Reality (AR) and Virtual Reality (VR) navigation, and industrial visual inspection, are experiencing increasingly demanding and volatile computing resource requirements. To meet the stringent Service Level Agreement (SLA) requirements of these services, especially under sub-millisecond latency constraints and high throughput demands, flexible and efficient resource management of instances hosting AI inference services within cloud-edge-device computing environments has become a key technical requirement for ensuring 6G network service quality and optimizing operating costs.

[0003] To adapt to changes in business load, existing technologies typically employ resource monitoring metrics-based adjustment strategies for scaling up and down inference service instances. This approach primarily involves real-time collection of resource load data (such as CPU utilization and memory usage) from running nodes, comparing the collected data with preset fixed thresholds. When resource load data reaches or exceeds these thresholds, the system determines whether resources are insufficient or excessive, and then executes corresponding instance addition or reduction operations to attempt to maintain the resource environment required for service operation.

[0004] However, when faced with the extremely high dynamism and quality-of-service (QoS) sensitivity of AI inference services in a 6G network environment, the aforementioned existing technical solutions struggle to simultaneously balance service performance and resource efficiency. In actual operation, these solutions often result in unstable Quality of Service (QoS), manifesting as significant fluctuations or even timeouts in end-to-end latency of the inference service during periods of fluctuating service load, failing to meet the continuity requirements of real-time services. Furthermore, to mitigate these QoS degradation issues, existing solutions often necessitate maintaining high resource redundancy, which directly leads to low average utilization of computing resources and excessive energy consumption. In short, existing technologies cannot achieve refined and efficient utilization of computing resources while ensuring high-standard Service Level Agreements (SLAs). Summary of the Invention

[0005] This invention provides a method, electronic device, and storage medium for scaling up and down inference service instances, which solves the technical problem that existing technologies struggle to ensure low latency performance while maintaining resource utilization efficiency when facing dynamically changing inference workloads, leading to unstable service quality or resource waste.

[0006] This invention provides a method for scaling up and down inference service instances, comprising:

[0007] Obtain historical and real-time operational data of the target inference service;

[0008] Based on the historical operating data, predict the future business load of the target inference service, and generate proactive scaling up and down instance decisions based on the future business load;

[0009] The current operating status of the target inference service is evaluated based on the real-time operating data, and a passive scaling instance decision is generated based on the current operating status.

[0010] Based on a preset collaborative strategy, the active scaling instance decision and the passive scaling instance decision are collaboratively decided to determine the final scaling instance instruction.

[0011] The number of instances of the target inference service is adjusted according to the final scaling instance instruction.

[0012] According to a method for scaling up and down inference service instances provided by the present invention, the historical running data and the real-time running data include at least one of the following or a combination thereof:

[0013] Application layer metrics data related to the service quality of the target inference service;

[0014] Resource layer index data related to the computing resource status of the target node carrying the target inference service;

[0015] Network layer metrics data related to the data transmission performance of the data interaction process with the target inference service.

[0016] According to a method for scaling up and down inference service instances provided by the present invention, the application layer indicator data includes at least one of inference latency, inference request length, inference accuracy, and service throughput.

[0017] The resource layer metrics data include at least one of the following: CPU utilization, GPU utilization, memory usage, video memory usage, disk I / O, and power consumption.

[0018] The network layer metrics data include at least one of the following: network latency, bandwidth utilization, packet loss rate, network jitter, and network slicing performance metrics.

[0019] According to a method for scaling up and down inference service instances provided by the present invention, the step of predicting the future business load of the target inference service based on the historical operating data, and generating an active scaling up and down instance decision based on the future business load, includes:

[0020] The historical operating data is input into the load prediction model to obtain the prediction results representing the future business load output by the load prediction model;

[0021] The proactive scaling instance decision is generated based on the future business load.

[0022] The load prediction model is obtained by training a time series prediction model based on multiple historical time series data samples and future time series data labels corresponding to each historical time series data sample.

[0023] According to a method for scaling up and down inference service instances provided by the present invention, the load prediction model includes an encoder, a decoder, and an attention mechanism module. The step of inputting the historical runtime data into the load prediction model and obtaining the prediction result representing the future business load output by the load prediction model includes:

[0024] The historical running data is input into the encoder so that the encoder can be used to extract features from the historical running data and generate a context vector containing time-dependent features.

[0025] The attention mechanism module performs a weighted combination of the features output by the encoder at each time step to determine the key historical state corresponding to the current prediction step.

[0026] The decoder is used to decode the context vector and the key historical states to gradually generate the prediction results that characterize the future business load.

[0027] According to a method for scaling up and down inference service instances provided by the present invention, the load prediction model further includes an input feature fusion module and an output processing module;

[0028] Before inputting the historical running data into the encoder, the method further includes: using the input feature fusion module to normalize and uniformly map the historical running data;

[0029] After using the decoder to decode the context vector and the key historical state to gradually generate the prediction result representing the future business load, the method further includes: using the output processing module to perform inverse normalization processing and / or error correction processing on the prediction result.

[0030] According to a method for scaling up and down inference service instances provided by the present invention, the step of evaluating the current operating state of the target inference service based on the real-time operating data, and generating a passive scaling up and down instance decision based on the current operating state, includes:

[0031] Determine the dynamic weights corresponding to each indicator data in the real-time operating data;

[0032] The dynamic weights are used to perform weighted calculations on the various indicator data in the real-time running data to obtain a comprehensive score that characterizes the current health status of the target inference service;

[0033] Compare the overall score with the preset expansion / contraction trigger threshold;

[0034] If the comprehensive score meets the triggering condition corresponding to the scaling trigger threshold, then the passive scaling instance decision is generated.

[0035] According to a method for scaling up and down inference service instances provided by the present invention, the step of using the dynamic weights to perform weighted calculations on various indicator data in the real-time running data to obtain a comprehensive score characterizing the current health status of the target inference service includes:

[0036] Obtain the total resource capacity information, allocated resource information, and used resource information of the target node carrying the target inference service;

[0037] Based on the total resource capacity information, the resource usage information, and the dynamic weight, a first sub-score is calculated, which is used to characterize the resource remaining degree of the target node.

[0038] Based on the total resource capacity information and the resource allocation information, determine the dispersion of the allocation ratio of various types of resources in the target node;

[0039] A second sub-score is calculated based on the degree of dispersion, and the second sub-score is used to characterize the resource balance of the target node;

[0040] The first sub-score and the second sub-score are fused together to obtain the comprehensive score.

[0041] According to a method for scaling up and down inference service instances provided by the present invention, the dynamic weight is determined through the following steps:

[0042] Obtain the business type characteristics of the target inference service;

[0043] If the service type characteristic indicates a latency-sensitive service, then increase the dynamic weight corresponding to the latency indicator;

[0044] If the business type characteristic indicates that it is a throughput-sensitive business, then the dynamic weight corresponding to the throughput indicator or energy consumption indicator is increased.

[0045] According to a method for scaling up and down inference service instances provided by the present invention, the step of coordinating the active scaling up instance decision and the passive scaling up instance decision based on a preset collaborative strategy to determine the final scaling up instance instruction includes:

[0046] Calculate the rate of change of the real-time request volume of the target inference service;

[0047] Obtain a status evaluation value that characterizes the current running state of the target inference service;

[0048] If the rate of change is greater than a preset rate threshold, and / or the state evaluation value indicates that the target inference service is in an abnormal operating state, the passive scaling instance decision is selected as the final scaling instance instruction.

[0049] If the rate of change is not greater than the rate threshold, and the state evaluation value indicates that the target inference service is operating normally, the active scaling instance decision is selected as the final scaling instance instruction.

[0050] According to a method for scaling up and down inference service instances provided by the present invention, obtaining a state evaluation value characterizing the current running state of the target inference service includes:

[0051] Obtain a comprehensive score, and use the comprehensive score as the status assessment value;

[0052] The comprehensive score is obtained by weighting each indicator in the real-time running data using the dynamic weights corresponding to each indicator in the real-time running data.

[0053] The status assessment value indicates that the target inference service is in an abnormal operating state, corresponding to the comprehensive score being lower than a preset health threshold.

[0054] According to a method for scaling up and down inference service instances provided by the present invention, before determining the passive scaling up instance decision as the final scaling up instance instruction, the method further includes:

[0055] Determine whether the active scaling up / down instance decision indicates a reduction in the number of instances and whether the passive scaling up / down instance decision indicates an increase in the number of instances;

[0056] If so, the selection of the active scaling instance decision is cancelled, and the passive scaling instance decision is selected as the final scaling instance instruction.

[0057] The present invention also provides a device for scaling up and down inference service instances, comprising the following modules:

[0058] The data acquisition module is used to acquire historical and real-time operational data of the target inference service;

[0059] The proactive engine module predicts the future business load of the target inference service based on the historical operating data, and generates proactive scaling decisions based on the future business load.

[0060] The passive engine module evaluates the current running status of the target inference service based on the real-time running data, and generates a passive scaling instance decision based on the current running status.

[0061] The decision fusion module performs collaborative decision-making on the active scaling instance decision and the passive scaling instance decision based on a preset collaborative strategy to determine the final scaling instance instruction.

[0062] The instance deployment module adjusts the number of instances of the target inference service according to the final scaling instance instruction.

[0063] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the inference service instance scaling method as described above.

[0064] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the inference service instance scaling method as described above.

[0065] The invention provides a method, electronic device, and storage medium for scaling up and down inference service instances. By establishing a dual-engine collaborative mechanism that combines proactive prediction and passive response, it utilizes historical data to achieve forward scaling up and down to reduce latency, while using real-time data for fallback correction to cope with sudden loads. This effectively solves the problem of delayed response or resource waste in a single scaling up and down method, and significantly improves the resource utilization and service quality stability of inference services. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating the method for scaling up and down inference service instances provided by the present invention.

[0068] Figure 2 This is a schematic diagram of the process for determining future business load based on historical operating data, provided by the present invention.

[0069] Figure 3 This is a schematic diagram of the process for generating passive scaling instance decisions based on the current running state, provided by the present invention.

[0070] Figure 4 This is a flowchart illustrating the comprehensive score for determining the current health status of a target inference service provided by the present invention.

[0071] Figure 5 This is a flowchart illustrating the process of generating the final scaling instance instruction provided by the present invention.

[0072] Figure 6 This is a system architecture diagram for implementing dynamically scaling instances of inference services, provided by the present invention.

[0073] Figure 7 This is a schematic diagram of the structure of the device for scaling up and down inference service instances provided by the present invention.

[0074] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0076] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0077] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0078] Traditional reactive scaling strategies typically trigger expansion only after system load has exceeded a threshold, a delay that is unacceptable in 6G ultra-low latency scenarios. To overcome this deficiency, this invention focuses on proactive scaling strategies based on service load prediction. The core idea of ​​the proposed proactive scaling strategy is to utilize machine learning models, particularly time series prediction models, to accurately predict the future load of AI inference services and prepare resources and expand services in advance. This allows for advance adjustment of the number of instances, ensuring stable system operation even during traffic surges. This is crucial for handling sudden, large-scale service requests that may occur in 6G networks (such as emergency response to sudden public events).

[0079] More importantly, this invention recognizes that while proactive scaling strategies are forward-looking, their predictive accuracy is limited and they struggle to handle completely random and unpredictable load fluctuations. Therefore, this invention creatively incorporates a highly efficient passive scaling strategy as a supplement and fallback mechanism to the proactive scaling strategy. Specifically, this invention designs a passive scaling strategy based on multi-dimensional resource monitoring metrics. This passive scaling strategy can perceive the current operating status of the system in real time and accurately, and respond quickly to trigger elastic scaling of the service when performance bottlenecks or resource pressure are detected. Unlike traditional passive scaling that relies solely on CPU and memory utilization, this invention constructs a more comprehensive and refined monitoring metric system that deeply integrates the characteristics of 6G networks and the performance features of the AI ​​service itself.

[0080] The following is combined Figures 1-8 This invention describes the method for scaling up and down inference service instances, the electronic device, and the storage medium provided by the present invention.

[0081] The inference service instance scaling method provided in this invention can be applied to various computing environments, such as cloud computing centers, edge computing nodes, or hybrid cloud environments. The executing entity can be an electronic device in the aforementioned environment, specifically a server, controller, or scheduling module in a cloud management platform responsible for resource scheduling and management. For ease of description, the following embodiments use an electronic device as the executing entity.

[0082] Figure 1 This is a flowchart illustrating the method for scaling up and down inference service instances provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps:

[0083] Step 11: Obtain historical and real-time running data of the target inference service.

[0084] In one specific implementation, a target inference service refers to an application or microservice deployed on a computing node to perform specific artificial intelligence inference tasks, such as image classification, speech recognition, and natural language processing. This target inference service is typically hosted and run by one or more instances, such as containers, virtual machines, or processes.

[0085] Historical operational data refers to various types of data generated and recorded during the operation of the target inference service within a preset time period prior to the current sampling period. Electronic devices can obtain this data by querying historical log databases, historical storage of monitoring systems, or data warehouses. Historical operational data reflects the load fluctuation patterns and resource consumption patterns of the service over a past period. Generally, historical operational data is the record of data generated by the target inference service during operation within a sampling period prior to the current sampling period.

[0086] Real-time operational data refers to the data generated by the target inference service within the current sampling period or a very short recent time window (e.g., within the last few seconds or hundreds of milliseconds). Electronic devices can acquire this data through real-time monitoring probes, sidecar agents, or telemetry interfaces. Real-time operational data reflects the immediate workload and health status of the target inference service.

[0087] Step 12: Predict the future business load of the target inference service based on the historical operating data, and generate an active scaling up / down instance decision based on the future business load.

[0088] Electronic devices utilize the time-series characteristics, periodic patterns, or trend information contained in historical operating data to estimate future business load, i.e., the amount of requests, concurrency, or computing tasks expected to arrive at a certain time or time period in the future, through preset prediction algorithms or statistical analysis methods.

[0089] After obtaining a forecast of future business load, electronic devices calculate the number of instances needed to meet that load, generating a proactive instance scaling decision. Specifically, this decision includes a recommended scaling plan based on the predicted future business load, such as "increasing the number of instances to 50 within the next 5 minutes" or "gradually reducing the number of instances to 10 within the next 10 minutes." Proactive instance scaling decisions are primarily used to address foreseeable periodic peaks or troughs, aiming to prepare resources in advance and avoid response delays.

[0090] Step 13: Evaluate the current running status of the target inference service based on the real-time running data, and generate a passive scaling instance decision based on the current running status.

[0091] The electronic device analyzes the acquired real-time operational data to assess the current operational status of the target inference service. The current operational status reflects the actual pressure and performance of the target inference service, such as whether it is in an overloaded, idle, congested, or healthy state.

[0092] When assessment results indicate that the current operating status does not meet expectations—for example, excessive load leading to processing backlog, or excessively low load leading to idle resources—electronic devices will generate passive scaling decisions based on current actual needs. Passive scaling decisions can include recommended scaling actions to be executed immediately based on the current real-time status, such as "overload detected, immediately add 5 instances." Passive scaling decisions are primarily used to address sudden, unpredictable load fluctuations as a safety guarantee for system stability.

[0093] Step 14: Based on a preset collaborative strategy, perform collaborative decision-making on the active scaling instance decision and the passive scaling instance decision to determine the final scaling instance instruction.

[0094] In actual operation, prediction-based proactive scaling decisions and real-time-based passive scaling decisions may coexist or even conflict. For example, proactive scaling decisions may recommend scaling down, while passive scaling decisions may recommend scaling up. Therefore, electronic devices will arbitrate, merge, or select between these two approaches based on a preset coordination strategy to make a collaborative decision.

[0095] Optionally, the preset collaborative strategy is a predefined set of rules or algorithmic logic used to process decision recommendations for both proactive and reactive scaling instance decisions. Collaborative decision-making methods may include, but are not limited to: setting priorities (e.g., prioritizing reactive scaling instance decisions in emergency situations), weighted fusion (combining the number of decisions from both scaling instances), or conditional triggering (switching decision sources when specific conditions are met).

[0096] Through collaborative decision-making, the electronic device determines a final scaling instruction. The final scaling instruction is the operation instruction that the system is actually prepared to execute after comprehensive consideration, and it clearly indicates the number of instances to be adjusted or the specific number of instances to be added / reduced.

[0097] Step 15: Adjust the number of instances of the target inference service according to the final scaling instance instruction.

[0098] Finally, electronic devices can issue the final scaling instruction by calling underlying resource orchestration interfaces, such as Kubernetes API and cloud platform control interfaces, to execute specific scaling (i.e., starting new instance nodes) or scaling (i.e. stopping and releasing existing instance nodes) operations, thereby adjusting the actual number of instances of the target inference service running in the cluster.

[0099] The inference service instance scaling method provided by this invention combines the foresight of proactive scaling decisions with the real-time performance of passive scaling decisions by introducing a dual-engine collaborative mechanism of proactive and passive engines. Specifically, proactive scaling decisions generated using historical runtime data can proactively address foreseeable load changes, eliminating scaling lag and reducing service latency. Passive scaling decisions generated using real-time runtime data can perceive system status in real time, serving as a fallback mechanism to quickly respond to sudden traffic surges and ensuring system robustness.

[0100] In summary, this invention establishes a dual-engine collaborative mechanism that combines proactive prediction and passive response. By utilizing historical data to achieve forward scaling up and down to reduce latency, and by using real-time data for fallback correction to cope with sudden loads, it effectively solves the problem of delayed response or resource waste in a single scaling up and down method, and significantly improves the resource utilization and service quality stability of inference services.

[0101] Based on the above embodiments, this embodiment will provide a further detailed description of the specific content of the historical and real-time operating data involved.

[0102] To gain a more comprehensive and accurate understanding of the operational status of the target inference service, historical and real-time operational data not only include traditional single-dimensional data but also incorporate a multi-dimensional monitoring indicator system. Specifically, the historical and real-time operational data include at least one of the following or a combination thereof:

[0103] Application layer metrics data related to the service quality of a single or multiple inference tasks of the target inference service;

[0104] Resource layer index data related to the computing resource status of the target node carrying the target inference service;

[0105] Network layer metrics data related to the data transmission performance of the data interaction process with the target inference service.

[0106] Application-layer metrics data refers to key data that directly reflects user experience and business performance quality. This layer of data focuses on the execution performance of the AI ​​model itself. Electronic devices can obtain this application-layer metrics data by embedding tracking points in the code of the target inference service, intercepting requests using service mesh, or analyzing application logs.

[0107] Resource layer metrics reflect the health status of the physical or virtual infrastructure supporting the service's operation. Target nodes can be physical servers, virtual machines, or container groups (Pods). Electronic devices can obtain this data by reading operating system kernel statistics through monitoring agents on the target nodes, such as cAdvisor and Node Exporter. Unlike traditional solutions that only focus on CPU, this embodiment emphasizes a comprehensive understanding of computing resource status, including not only general computing and storage resources but also, in particular, heterogeneous computing resources crucial for AI inference, such as the status of accelerator cards like GPUs and NPUs. These resource layer metrics constitute the material foundation for the stable operation of the inference service.

[0108] Network layer metrics reflect the smoothness of data flow between clients, servers, and different service components. In a 6G network environment, the performance of the data interaction process directly determines the end-to-end service experience. Electronic devices can obtain these network layer metrics through network probes or telemetry data from network devices. In particular, in 6G network slicing scenarios, these network layer metrics can reflect whether the current network environment meets the stringent transmission requirements of AI inference services for high bandwidth and low latency.

[0109] This invention, by introducing a multi-dimensional data system that includes application layer metrics, resource layer metrics, and network layer metrics, breaks through the limitations of traditional scaling methods that rely solely on a single resource metric (such as CPU utilization). This cross-layer, comprehensive monitoring enables electronic devices to perceive the actual operational status of the target inference service from three dimensions: service quality, physical infrastructure, and transmission channels. This provides rich and high-quality data support for generating more accurate and 6G-compliant proactive and reactive scaling decisions.

[0110] Based on the above embodiments, this embodiment will further elaborate on the specific composition of application layer indicator data, resource layer indicator data, and network layer indicator data.

[0111] Specifically, the application layer metrics data include at least one of the following: inference latency, inference request length, inference accuracy, and service throughput.

[0112] Inference latency refers to the time required to complete a single inference task. This metric specifically includes the sum of preprocessing latency, AI model computation latency, and post-processing latency. For latency-sensitive services (such as autonomous driving and AR navigation), this metric is a primary standard for measuring service quality. Electronic devices can calculate this metric by recording the timestamps of the request arriving at the service instance and the timestamps of the response returning.

[0113] Inference request length refers to the number of requests currently backlogged in the service instance waiting queue that have not yet been processed. This metric directly reflects the current concurrency pressure. When the length of the service instance waiting queue continues to grow, it indicates that the system is about to become overloaded.

[0114] Inference accuracy refers to the accuracy of the output results of an AI model, such as the accuracy rate of classification tasks or the mAP value of object detection. In some scenarios, model compression or quantization techniques may be used to pursue speed, which may lead to a decrease in accuracy. Monitoring inference accuracy can prevent sacrificing business availability due to over-optimization of performance.

[0115] Service throughput refers to the number of inference requests successfully processed by a service instance per unit of time, such as Queries Per Second (QPS) or Transactions Per Second (TPS).

[0116] In addition, in this embodiment, the resource layer indicator data includes at least one of the following: CPU utilization, GPU utilization, memory utilization, video memory usage, disk I / O, and power consumption.

[0117] CPU utilization and memory usage are fundamental metrics for measuring the load on general-purpose computing resources. For deep learning inference tasks, GPUs are often the computational bottleneck. Electronic devices can use relevant tools to accurately collect the utilization of GPU computing units and the amount of video memory used, which is crucial for determining whether an AI task is saturated.

[0118] Meanwhile, this embodiment incorporates energy consumption into the monitoring system. Electronic devices can obtain the real-time power consumption of nodes by reading the server power management interface or by using an estimation model based on voltage and frequency. This allows subsequent scaling decisions to meet performance requirements while also considering energy efficiency.

[0119] By statistically analyzing disk I / O, we can intuitively reflect the storage pressure on a node when reading model files or writing logs.

[0120] In this embodiment, the network layer metrics data include at least one of the following: network latency, bandwidth utilization, packet loss rate, network jitter, and network slicing performance metrics.

[0121] Network latency refers to the round-trip time (RTT) of data packets during network transmission. Bandwidth utilization refers to the proportion of the current transmission rate to the maximum bandwidth of the network link. For scenarios that require the transmission of high-definition video streams or large model parameters, bandwidth utilization is often a limiting factor.

[0122] Given that packet loss rate and network jitter are crucial for ensuring Ultra Reliable and Low Latency Communications (uRLLC) services in 6G networks, excessive packet loss or jitter can cause inference services to lag or fail. In a 6G context, if the inference service runs on a specific network slice, electronic devices also need to monitor network slice performance metrics, such as slice isolation guarantees or congestion levels within the slice, to ensure that network resource SLAs are met.

[0123] These multi-dimensional metrics can be collected and aggregated in real time through a unified monitoring and alerting system, such as a solution based on Prometheus and Grafana. This provides fine-tuning training data and realistic environmental data for the generation of proactive instance scaling decisions, improving predictive accuracy. It also provides quantitative metrics for the intelligent decision engine of passive instance scaling decisions. For example, when multiple key metrics simultaneously exceed preset thresholds, or when a performance degradation trend is identified through more complex algorithms (such as anomaly detection), the proactive and passive engines will work together to trigger corresponding scaling operations.

[0124] This invention constructs a comprehensive, detailed monitoring system that aligns with the characteristics of AI applications under 6G networks by limiting application-layer and resource-layer metrics, such as GPU utilization, energy consumption, and network-layer metrics. This fine-grained metric design enables refined resource management that ensures high-precision, low-latency service experience while achieving green and energy-saving performance when generating proactive and passive scaling decisions for instances.

[0125] In actual operation, in order to overcome the lag in response of passive scaling up and down, this invention predicts the future business load of the target inference service based on historical operating data, and generates proactive scaling up and down instance decisions based on the future business load. The specific implementation includes, but is not limited to, the following steps:

[0126] The historical operating data is input into the load prediction model to obtain the prediction results representing the future business load output by the load prediction model.

[0127] This embodiment can employ a variety of advanced load prediction models, including but not limited to Long Short Term Memory (LSTM) networks and attention-based Transformer models. These models excel at capturing long-term dependencies and complex patterns in time-series data, making them highly suitable for predicting dynamic changes such as network traffic and user request volume.

[0128] This embodiment constructs a multi-dimensional historical operational data set as input feature set by combining different elements of different business scenarios at different time periods. When the load prediction model predicts a significant increase in load within a certain future time period based on the input feature set, it can initiate the expansion process in advance to increase the number of service instances. Conversely, when the load prediction model predicts a sustained decrease in load, it will trigger scaling down to save resources and costs.

[0129] First, the electronic device extracts a continuous historical operating data from the monitoring system (for example, time-series data containing application layer indicators, resource layer indicators, and network layer indicators collected every minute over the past 24 hours), and uses this historical operating data as input to the load prediction model.

[0130] After receiving input, the load forecasting model performs a series of complex calculations to obtain a forecast result. This forecast result is also a time series, representing the future business load as a measure of how the business load changes over a future period. For example, if the input is historical operational data from the past 24 hours, the output could be the expected number of requests per minute (Queries Per Second, QPS) in the next hour, the expected end-to-end latency, or the expected GPU utilization, etc.

[0131] After obtaining the prediction results, proactive scaling decisions for instance operations can be generated based on the future business load, which may include, but are not limited to:

[0132] Based on the predicted future business load and the service capacity baseline of each instance (e.g., a single instance can stably support 200 QPS), the number of instances required to meet the future business load at various points in time is calculated.

[0133] The calculated number of instances needed at each future time point is compared with the current actual number of instances to generate specific proactive scaling decisions for instance expansion. This proactive scaling decision can be a timetable, for example: "In 10 minutes, when the predicted load reaches its peak, increase the number of instances from 20 to 50; in 30 minutes, when the predicted load decreases, reduce the number of instances from 50 to 30."

[0134] The pre-training process of the above load forecasting model is explained in detail below. Specifically, the load forecasting model is obtained by training a time series forecasting model based on multiple historical time series data samples and the corresponding future time series data labels for each historical time series data sample. The training process can be carried out using the following steps:

[0135] First, electronic devices generate a large number of training samples from massive amounts of historical monitoring data. Each historical time series data sample is a segment of past time series data (e.g., data from 8:00 AM to 9:00 AM on Monday), while the corresponding future time series data label is the real data that follows (e.g., real load data from 9:00 AM to 10:00 AM on Monday).

[0136] These paired historical time series data samples and future time series data labels are input into an initial time series prediction model, such as an untrained LSTM network model. Through repeated iterations and optimization algorithms (such as gradient descent), the internal parameters of the LSTM network model are continuously adjusted so that for any given input sample, its output prediction result is as close as possible to the true future time series data label.

[0137] Training is complete when the prediction error of the initial time series forecasting model converges to an acceptable range. At this point, the time series forecasting model is finalized and becomes the load forecasting model for subsequent use.

[0138] This invention constructs and trains a high-precision load prediction model, which uses the temporal patterns contained in historical operating data to accurately predict future business load. This deep learning-based prediction mechanism enables the system to overcome the time lag limitations of traditional passive response, generate proactive scaling decisions for instance expansion and reduce capacity in advance before business peaks arrive, and complete resource expansion. This effectively avoids service quality degradation caused by cold start delays and ensures the continuity and stability of AI inference services in a 6G network environment.

[0139] As an optional embodiment, the load prediction model is specifically an encoder-decoder model, designed for sequence-to-sequence conversion tasks, and is well-suited for scenarios where future time series are predicted based on a historical time series. This encoder-decoder model mainly includes an encoder, a decoder, and an attention mechanism module.

[0140] Figure 2 This is a schematic diagram of the process for determining future business load based on historical operating data provided by the present invention, such as... Figure 2 As shown, the step of inputting the historical operating data into the load forecasting model and obtaining the forecast result representing the future business load output by the load forecasting model includes the following specific processing steps:

[0141] Step 21: Input the historical running data into the encoder to extract features from the historical running data and generate a context vector containing time-dependent features.

[0142] In a specific implementation, the encoder may consist of one or more stacked recurrent neural network (RNN) units, such as long short-term memory (LSTM) units or gated recurrent units (GRUs). In an optional implementation, the encoder may also employ a bidirectional long short-term memory (BiLSTM) structure.

[0143] When the preprocessed historical running data sequence is input into the encoder, the LSTM or GRU units within the encoder read the time series of multi-dimensional feature vectors step by step. During this process, through its internal gating mechanisms (such as forget gates, input gates, and output gates), the encoder can effectively extract and learn complex, non-linear time-dependent features in the time series, such as the periodicity and trend of the load, and the correlation between multi-dimensional indicators. For the BiLSTM structure, it can capture the dependencies of the sequence simultaneously from both forward and backward directions, thereby gaining a more comprehensive understanding of the historical running data.

[0144] After processing the time series corresponding to the entire input historical running data, the encoder generates a fixed-length context vector from its final hidden state or a combination of the hidden states of all time steps. This context vector can be regarded as a condensation and summary of the entire historical running data.

[0145] Step 22: The attention mechanism module performs a weighted combination of the features of each time step output by the encoder to determine the key historical state corresponding to the current prediction step.

[0146] Traditional encoder-decoder architectures rely solely on the last context vector generated by the encoder, which can easily lead to information bottlenecks, especially when the input time series is long. To address this issue, this embodiment introduces an attention mechanism module. This module automatically calculates and assigns a set of weight coefficients at each decoding step of the decoder. These weight coefficients correspond to the hidden state features output by the encoder at each time step when processing the input time series, and the magnitude of the weight coefficients reflects the importance of each historical time step in generating the current prediction step.

[0147] In this embodiment, by weighting these coefficients and combining them with the output features of the encoder at each time step, the attention mechanism module can determine a dynamically changing key historical state for the current prediction time step. This allows the load forecasting model to focus on reviewing and utilizing the most relevant parts of historical operating data when predicting the load at a future point in time.

[0148] Step 23: Use the decoder to decode the context vector and the key historical state to gradually generate the prediction result that represents the future business load.

[0149] In this embodiment, the decoder also consists of one or more stacked LSTM or GRU units. At the beginning of its decoding process, the decoder receives the context vector generated by the encoder as its initial hidden state.

[0150] Subsequently, at each decoding time step, the decoder combines its output from the previous time step with the key historical states calculated by the attention mechanism module to calculate and output the prediction value for the current time step. This process is repeated until a complete prediction sequence containing multiple future time steps is gradually generated. This prediction sequence is the final prediction result representing the future service load.

[0151] This invention employs an encoder-decoder model and introduces an attention mechanism module to construct an advanced load prediction model that can effectively handle sequence-to-sequence prediction tasks. The addition of the attention mechanism solves the problem of information loss in long sequence prediction, enabling the load prediction model to dynamically focus on the most critical historical information during prediction. Compared with ordinary time-series prediction methods, it can more accurately capture the complex patterns of business load, thereby significantly improving the accuracy of prediction results and facilitating the generation of reliable proactive scaling instance decisions.

[0152] In an optional implementation, to further improve the model's predictive performance and robustness, the load prediction model, in addition to the encoder, decoder, and attention mechanism modules, also includes an input feature fusion module and an output processing module. The input feature fusion module may consist of one or more data processing units and a fully connected layer (FC).

[0153] Accordingly, before inputting the historical operation data into the encoder, the method further includes: using the input feature fusion module to normalize and uniformly map the historical operation data.

[0154] Historical operational data contains various metrics from the application layer, resource layer, and network layer, such as inference latency in milliseconds (ms), GPU utilization in percentage (%), and bandwidth utilization in Mbps. Directly inputting these metrics into the load prediction model can negatively impact its accuracy and training performance. Therefore, this invention introduces an input feature fusion module to normalize the metrics data for each dimension. For example, max-min normalization is used to scale all data values ​​to the [0, 1] range to eliminate metric differences.

[0155] After normalization, this module linearly transforms and maps the multi-dimensional input feature vectors to a unified, predefined internal dimension through one or more fully connected layers. This allows subsequent encoders (such as LSTM units) to learn in a dimensionally regular feature space, which helps improve the model's generalization ability and feature extraction efficiency.

[0156] After using the decoder to decode the context vector and the key historical state to gradually generate the prediction result representing the future business load, the method further includes: using the output processing module to perform inverse normalization processing and / or error correction processing on the prediction result.

[0157] The output processing module can be composed of a corresponding data inverse transformation unit and an error correction algorithm unit. Since the historical running data input to the load prediction model has been normalized, the output predicted values ​​are also on the normalized scale. Therefore, this embodiment uses the output processing module to perform inverse normalization processing on the predicted value sequence in the prediction results generated by the decoder. That is, using the normalization parameters (such as maximum and minimum values) saved during training, each predicted value is restored to its original dimension and numerical range with actual physical meaning. For example, the predicted CPU utilization rate is restored from 0.8 to 80%.

[0158] To further improve prediction accuracy, the output processing module analyzes the residual sequence between recent predicted values ​​and actual values ​​through an integrated error correction algorithm unit, and uses this residual sequence to compensate and correct the output of the current load prediction model, thereby reducing systematic prediction bias.

[0159] This invention, through the introduction of an input feature fusion module and an output processing module, constructs an end-to-end load prediction process. The input feature fusion module solves the problem of multi-source heterogeneous data input through normalization and dimensionality mapping, providing high-quality, regular input features for the load prediction model and improving the model's training efficiency and stability. The output processing module, through inverse normalization and error correction, transforms the internal output of the load prediction model into a final prediction result with practical business significance and higher accuracy. The addition of these two modules significantly enhances the engineering usability and prediction accuracy of the entire load prediction model in actual deployment.

[0160] Figure 3 This is a schematic diagram of the process for generating passive scaling instance decisions based on the current running state, as provided by the present invention. Figure 3 As shown, step 13 in the above embodiments, which involves evaluating the current running status of the target inference service based on the real-time running data, and generating a passive scaling instance decision based on the current running status, specifically includes, but is not limited to, the following implementation steps:

[0161] Step 31: Determine the dynamic weights corresponding to each indicator data in the real-time running data.

[0162] In this embodiment, to enable the passive response mechanism to adapt to different business needs and operational scenarios, the system assigns dynamically adjustable weights to each indicator in the real-time operational data. These dynamic weights are a set of numerical values ​​that reflect the importance of each indicator in assessing the current health status of the system.

[0163] For example, in an evaluation system that includes three metrics—CPU utilization, memory utilization, and inference latency—their weights might be set to [0.3, 0.2, 0.5]. It's important to note that these weights are not fixed but can be adjusted based on the specific characteristics of the target inference service. For instance, for latency-sensitive services, the weight of inference latency would be increased. When an electronic device begins making a passive scaling instance decision based on its current operating state, it first determines this set of dynamic weights that are effective in the current scenario.

[0164] Step 32: Use the dynamic weights to perform weighted calculations on the various indicator data in the real-time running data to obtain a comprehensive score that characterizes the current health status of the target inference service.

[0165] After acquiring real-time operational data (e.g., current CPU utilization is 60%, memory utilization is 50%, and inference latency is 80ms), the electronic device uses determined dynamic weights to perform weighted calculations on these metrics.

[0166] There are several ways to perform weighted calculations. In one optional implementation, the electronic device first converts the real-time value of each indicator data into a standardized individual score. For example, it calculates the individual score separately using methods such as (1 - utilization rate) or (preset latency limit - current latency) / preset latency limit. Then, it multiplies these individual scores by their respective weights and sums them to obtain a comprehensive score. The comprehensive score integrates real-time information from multiple dimensions and can quantify and characterize the overall health of the target inference service. A higher comprehensive score generally indicates a healthier system and more abundant resources; while a lower comprehensive score indicates greater system pressure and higher risk.

[0167] Step 33: Compare the comprehensive score with the preset expansion / shrinkage trigger threshold.

[0168] One or more scaling thresholds can be pre-set within the electronic device. These thresholds serve as critical values ​​to determine whether system intervention is necessary. For example, a scaling threshold, Threshold, can be set. UP A score of 40 and a reduction threshold (Threshold) DOWN It scores 90 points.

[0169] The electronic device will compare the calculated real-time comprehensive score with these preset expansion and contraction trigger thresholds.

[0170] Step 34: If the comprehensive score meets the triggering condition corresponding to the scaling trigger threshold, then the passive scaling instance decision is generated.

[0171] Based on the comparison results, the electronic device determines whether the triggering condition is met. For example:

[0172] If the overall score is lower than the expansion threshold (Threshold) UP If so, the expansion trigger condition is met.

[0173] If the overall score is higher than the reduction threshold (Threshold) DOWN If so, the shrinkage trigger condition is met.

[0174] Once the triggering conditions are met, the electronic device will immediately generate a passive scaling decision. This passive scaling decision is usually a clear and immediate instruction, such as "immediately add 2 instances" or "immediately reduce 1 instance".

[0175] The specific amount to be increased or decreased can be dynamically calculated based on the degree to which the comprehensive score deviates from the expansion / contraction trigger threshold.

[0176] This invention significantly improves the intelligence level of passive response by introducing a passive decision-making mechanism based on dynamic weights and comprehensive scoring. Compared to traditional methods that rely on a single static threshold, this invention can comprehensively and holistically assess the current true health status of the system. By dynamically adjusting the weights corresponding to different indicator data, the passive decision-making mechanism can flexibly adapt to the service quality requirements of different AI services. When the load suddenly increases or the system malfunctions, the comprehensive scoring can react quickly and sensitively, thereby triggering timely and accurate passive scaling decisions, providing a robust and reliable safety net for the entire inference service system.

[0177] Figure 4 This is a flowchart illustrating the comprehensive score for determining the current health status of a target inference service provided by the present invention, as shown below. Figure 4 As shown, regarding step 32 mentioned in the above embodiment, the dynamic weights are used to perform weighted calculations on the various indicator data in the real-time running data to obtain a comprehensive score characterizing the current health status of the target inference service. In a specific embodiment of the present invention, the comprehensive score is calculated by fusing two sub-scores of different dimensions: a first sub-score for measuring resource reserves and a second sub-score for measuring the balance of resource allocation. The specific calculation process is as follows:

[0178] Step 321: Obtain the total resource capacity information, allocated resource information, and used resource information of the target node carrying the target inference service.

[0179] Among them, the total resource capacity information (denoted as capacity) refers to the upper limit of the total amount of various resources owned by the target node, such as the total number of CPU cores, the total memory size, the total GPU memory size, etc.

[0180] The allocated resource information refers to the amount of resources that have been declared and allocated to various running instances by the container orchestration system (such as Kubernetes) on the target node. These resources cannot be allocated to other new instances even if they are not currently in use.

[0181] Resource usage information (used): refers to the amount of various resources actually consumed or occupied by the instance at the current moment.

[0182] Step 322: Calculate a first sub-score based on the total resource capacity information, the resource usage information, and the dynamic weight. The first sub-score is used to characterize the resource remaining level of the target node.

[0183] The first sub-score aims to measure the number of resources available for use on the current target node. The electronic device can calculate this sub-score based on the resource information obtained above and the dynamic weights determined in the previous embodiments.

[0184] As a concrete computational example, we first calculate the remaining proportions of various resources (such as CPU, memory, GPU, etc.) on the target node. Then, we multiply the remaining proportions of each resource by their corresponding dynamic weights and sum them to obtain the first sub-score, whose calculation formula can be expressed as:

[0185] score1=[(capacity-used) / capacity] 100 weight;

[0186] Here, score1 represents the first sub-score, capacity represents the total resource capacity information, used represents the resource usage information, and weight represents the dynamic weight of each type of resource.

[0187] Step 323: Determine the dispersion of the allocation ratio of various resources in the target node based on the total resource capacity information and the resource allocation information.

[0188] Step 324: Calculate a second sub-score based on the degree of dispersion. The second sub-score is used to characterize the resource balance of the target node.

[0189] The second sub-score (denoted as score2) is used to measure whether the various resources on the target node are evenly allocated and used. If a target node's CPU resources are exhausted while its memory resources are largely idle, this state is unhealthy because it can no longer schedule instances that require CPU.

[0190] To quantify this imbalance, the electronic device performs the following calculation:

[0191] First, calculate the unallocated fraction for each resource, i.e.:

[0192] fraction=(capacity-allocated) / capacity.

[0193] Then, the dispersion of these unassigned proportion values ​​is calculated.

[0194] One alternative implementation is to calculate the standard deviation of these proportions. The smaller the standard deviation, the closer the unallocated proportions of various resources are, and the more balanced the resource allocation; the larger the standard deviation, the more unbalanced the resource allocation.

[0195] Finally, based on the calculated dispersion (such as standard deviation), a second sub-score is calculated through an inverse mapping function to intuitively characterize resource balance. The higher the score, the more balanced the resource allocation and the better the node can adapt to the scheduling requirements of various new instances.

[0196] score2= ;

[0197] Where n is the total number of resource types being monitored; i is the index of the resource type, with values ​​ranging from 1 to n; This represents the unallocated proportion of the i-th type of resource.

[0198] Step 325: Perform a fusion calculation on the first sub-score and the second sub-score to obtain the comprehensive score.

[0199] After obtaining the first sub-score representing resource surplus and the second sub-score representing resource balance, the electronic device performs a fusion calculation on the two sub-scores to obtain the final comprehensive score.

[0200] A simple way to calculate the fusion is to take the arithmetic mean of the two, that is:

[0201] score = (score1 + score2) / 2.

[0202] Of course, other fusion methods such as weighted averaging can also be adopted according to actual needs. This final comprehensive score takes into account both the absolute surplus of resources and the structural health of resource allocation, and can more comprehensively assess the carrying capacity of nodes.

[0203] This invention significantly improves the scientific rigor and accuracy of the comprehensive score by designing a first sub-score representing resource availability and a second sub-score representing resource balance. The resulting comprehensive score not only focuses on how many resources the target node has left, but also innovatively introduces consideration of whether resource allocation is balanced. This enables the passive response mechanism to avoid scheduling failures caused by resource bottlenecks, thereby more accurately determining whether a node is truly healthy and providing a more reliable and comprehensive decision-making basis for passive scaling up and down instances.

[0204] The following examples will provide a more detailed explanation of the specific logic for determining the dynamic weights.

[0205] In this embodiment, the dynamic weights corresponding to each indicator data in the real-time running data are not static, but can be adaptively adjusted according to the business characteristics of the target inference service to meet the differentiated requirements for Quality of Service (QoAIS) in different scenarios. The specific steps include:

[0206] First, obtain the business type characteristics of the target inference service.

[0207] During deployment or runtime, each target inference service can be tagged with one or more business type characteristics, which describe the core business or technical requirements of the target inference service. These business type characteristics can be obtained by reading the target inference service's metadata, configuration files (such as annotations in Deployment YAML), or information from the service registry.

[0208] Common business type characteristics include, but are not limited to: latency-sensitive, throughput-sensitive, accuracy-priority, and cost-saving. Among them, latency-sensitive inference services mainly involve obstacle recognition in autonomous driving, real-time AR rendering, and online game AI, scenarios with extremely high requirements for response time.

[0209] Throughput-sensitive inference services mainly involve offline batch image processing, large-scale data analysis, and batch model training, aiming to maximize processing efficiency per unit time.

[0210] Accuracy-priority inference services mainly involve medical image diagnosis, financial fraud detection, etc., which have extremely high requirements for the accuracy of inference results and cannot tolerate scenarios where accuracy degrades.

[0211] Cost-saving inference services mainly involve some non-core internal data analysis tasks that can tolerate a certain delay, prioritizing scenarios that reduce resource consumption and operating costs.

[0212] Furthermore, key performance indicators of the target inference service can be identified based on the characteristics of the business type.

[0213] After obtaining the characteristics of the business type, the most important key performance indicators (KPIs) for the current business can be identified from the multi-dimensional monitoring indicator system based on these characteristics. For example, if the business type is latency-sensitive, inference latency, network latency and other indicators will be identified as key performance indicators; if the business type is throughput-sensitive, service throughput, GPU utilization and other indicators will be identified as key performance indicators.

[0214] Once key performance indicators (KPIs) are identified, their weights can be adjusted. Specifically, this involves increasing the dynamic weights of those KPIs. For example, for latency-sensitive services, the weight of inference latency can be adjusted from the default 0.3 to 0.6.

[0215] Correspondingly, the dynamic weights of other non-critical performance metrics will be reduced to ensure that the sum of all weights is a constant (e.g., 1). For example, when a latency-sensitive inference service is running, because the weight of the inference latency metric is increased, even if other resource metrics (such as CPU utilization) are still at a low level, a slight deterioration in inference latency will significantly lower the overall score, thereby triggering a scaling decision earlier and ensuring low-latency performance.

[0216] When a throughput-sensitive batch inference task is running, the weight of service throughput and energy consumption metrics is increased. In this case, the system is more tolerant of a certain increase in latency and focuses more on maximizing the utilization of core resources such as GPUs. As long as throughput does not reach its bottleneck or energy consumption is too high, the system will not easily expand its capacity, thereby achieving higher resource utilization and energy efficiency.

[0217] This invention introduces a dynamic weight adaptive adjustment mechanism based on business type characteristics, which enables the passive engine to have high flexibility and scene awareness. It can intelligently adjust the evaluation criteria according to the different core requirements of AI inference services, ensuring that the system's scaling behavior is consistent with business objectives. This allows the entire system to provide differentiated and optimal service quality assurance for different types of AI applications.

[0218] Figure 5 This is a flowchart illustrating the process of generating the final scaling instance instruction provided by the present invention, as shown below. Figure 5 As shown in the above embodiments, the collaborative decision-making process for the active scaling instance decision and the passive scaling instance decision based on a preset collaborative strategy to determine the final scaling instance instruction includes, but is not limited to, the following implementation steps:

[0219] Step 41: Calculate the rate of change of the real-time request volume of the target inference service.

[0220] This embodiment continuously monitors the number of requests flowing into the target inference service. By comparing the current request volume with the previous request volume and dividing by the time interval, the rate of change of the real-time request volume can be calculated. For example, the slope of the query rate per second over the last 10 seconds can be calculated. To avoid glitches, in an optional implementation, the request volume sequence can be smoothed (e.g., using a moving average) before calculating its rate of change. The resulting rate of change can quantify the severity of the current business load's growth or decline trend.

[0221] Step 42: Obtain a status evaluation value that characterizes the current running state of the target inference service.

[0222] Furthermore, this embodiment needs to obtain a status assessment value that can comprehensively reflect the current health status of the system, that is, to determine the result of quantitative evaluation of real-time operating data.

[0223] In an optional implementation, the status assessment value can be a comprehensive score calculated in the foregoing embodiments. Generally, a higher comprehensive score indicates a healthier system status; a lower comprehensive score indicates greater system stress or that the system is in an abnormal state.

[0224] Step 43: If the rate of change is greater than a preset rate threshold, and / or the state evaluation value indicates that the target inference service is in an abnormal operating state, determine that the passive scaling instance decision is selected as the final scaling instance instruction.

[0225] By comparing the rate of change of real-time request volume and the status evaluation value of the current running state obtained in steps 41 and 42 with the preset corresponding threshold, it is possible to determine what kind of collaborative conditions are currently met.

[0226] Generally, a rate of change in real-time request volume exceeding a threshold indicates a rapid increase in business load, representing a sudden event; while an abnormal operational status indicated by the status assessment value suggests that the system's current health is already poor. In both cases, the first coordination condition can be determined, which can be understood as an emergency response mode. At this point, proactive predictions based on historical data may be inaccurate or insufficient. Therefore, the system prioritizes ensuring the real-time stability and availability of the service, selecting a passive scaling-up / down decision that requires immediate execution as the final instruction.

[0227] Step 44: If the rate of change is not greater than the rate threshold, and the state evaluation value indicates that the target inference service is running normally, the active scaling instance decision is selected as the final scaling instance instruction.

[0228] When the rate of change is not greater than the rate threshold, and the state evaluation value indicates that the target inference service is operating normally, it indicates that the current business load is stable or the change is moderate, and the system is in good health. At this point, it can be determined that the second coordination condition is met, which can be understood as a conventional optimization mode. The system has sufficient confidence in the accuracy of proactive instance scaling decisions, and therefore selects proactive instance scaling decisions as the final instruction. The purpose of this is to utilize the foresight of prediction to adjust resources in advance and smoothly, thereby achieving refined resource management and optimization of operating costs.

[0229] This invention, based on collaborative decision-making using real-time request rate of change and state assessment values, cleverly integrates the advantages of both proactive prediction and reactive response modes, intelligently identifying stable and sudden system states. When the system is stable, proactive scaling decisions are prioritized to pursue resource efficiency and foresight; when the system faces sudden loads or state deterioration, a decisive switch to reactive scaling decisions is made to ensure real-time service stability and robustness. This dynamic decision switching effectively avoids the limitations of a single decision-making mode, enabling the entire scaling system to possess comprehensive performance characterized by high efficiency, stability, and reliability.

[0230] Based on the above embodiments, as an optional embodiment, a pre-emptive security check and correction mechanism is added before selecting the proactive scaling instance decision as the final scaling instance instruction. This mechanism further enhances the robustness of collaborative decision-making and prevents service interruption risks caused by prediction bias. Specifically, it includes:

[0231] Determine whether the active scaling up / down instance decision indicates a reduction in the number of instances and whether the passive scaling up / down instance decision indicates an increase in the number of instances.

[0232] If so, the selection of the active scaling instance decision is cancelled, and the passive scaling instance decision is selected as the final scaling instance instruction.

[0233] In actual operation, electronic devices can simultaneously examine the decision suggestions generated by both the active and passive engines to determine if specific conflict patterns exist. For example, at any given moment, the active scaling decision might be to reduce the number of instances, while the passive scaling decision might be to increase the number of instances. This situation likely indicates a deviation in the active prediction; that is, the load trough predicted by the load forecasting model has not arrived, while the actual peak is still ongoing or has just begun.

[0234] If the above judgment is true, then the collaborative decision-making mechanism will trigger a security veto, including revoking the original decision to actively scale up instances and selecting the passive scaling up instance decision that needs to be executed immediately as the final scaling up instance instruction.

[0235] Suppose that at 6:00 PM, the proactive engine, based on historical operational data, believes the evening peak has passed and generates a proactive scaling decision to "reduce the number of instances from 20 to 10". However, due to a breaking news event causing user traffic to increase instead of decrease, the passive engine, based on the real-time surge in inference latency and the decrease in the overall score, generates a passive scaling decision to "immediately add 5 instances".

[0236] At this point, according to the preset logic, the system might initially choose to execute an active scaling-up / scaling decision because the rate of change has not yet exceeded the threshold. However, before execution, the conflict checking mechanism in this embodiment is triggered. The system finds that the active scaling-up / scaling decision is to reduce the number of instances, while the passive scaling-up / scaling decision is to increase the number of instances, thus satisfying the conflict condition. Therefore, the system will immediately cancel the scaling-down plan and instead select and execute the instruction to expand by 5 instances, thereby effectively avoiding a serious service failure that could have been caused by a prediction error.

[0237] Figure 6 This is a system architecture diagram of the inference service dynamically scaling instances provided by the present invention, as shown below. Figure 6 As shown, the inference service dynamic scaling instance system (hereinafter referred to as the system) provided in this embodiment mainly includes an indicator source, an elastic configuration module, an intelligent decision engine, and an underlying workload execution layer.

[0238] The indicator source is the data input source of the system, responsible for providing various monitoring indicator data, which corresponds to the multi-dimensional data system mentioned in the previous embodiments. For example, it can provide basic CPU data and memory data, or it can provide more complex performance indicators, such as performance indicators (PM), efficiency indicators (EM), and reliability indicators (RM).

[0239] The elastic configuration module provides a user-configurable policy entry point for the system. Users can define the scaling range of workloads, select algorithm configurations such as the type of prediction model and weight allocation strategy, and make other configurations such as cooldown time and trigger sensitivity.

[0240] As the core of achieving dynamic scaling, the intelligent decision engine mainly consists of corresponding sub-modules that implement functions such as indicator collection, predictive analysis, prediction algorithms, boundary protection, and elastic scaling.

[0241] The indicator acquisition module is primarily responsible for retrieving historical operational data from indicator sources. The indicator acquisition unit continuously acquires real-time operational data. The indicator processing unit cleans and formats the collected historical and real-time operational data, and performs operations such as minute-level aggregation, for example, aggregating second-level data into smoother minute-level data. The compensation mechanism unit handles data loss or anomalies, for example, by using interpolation or nearest-neighbor value padding to ensure the continuity of the data stream. The processed data is stored in a data cache for use by subsequent modules.

[0242] The predictive analysis module of the intelligent decision engine serves as the decision-making center of the system. It includes an indicator query unit, which is responsible for retrieving historical and real-time running data from the data cache.

[0243] Real-time operational data (which can be called real-time metrics) is fed into the active engine to trigger decisions with high real-time requirements, while historical operational data (which can be called historical metrics) is fed into the passive engine for time-series-based analysis.

[0244] The system also includes a prediction algorithm module to provide algorithmic support for predictive analysis, particularly for predictions based on historical operational data. Its processing flow may include: performing data noise reduction on the input historical operational data to remove abnormal noise points; identifying periodic patterns in the historical operational data, such as the daily or weekly cycle of business load; performing robust processing to enhance the algorithm's resistance to abnormal data and avoid serious deviations in prediction results due to individual bad points; and instance prediction to obtain the predicted value of the future instance quantity requirement output by the load prediction model.

[0245] Figure 6 The system also includes an elastic scaling module and a boundary protection unit. The elastic scaling module receives and executes decision instructions from the predictive analytics module. The boundary protection unit performs security checks before executing instructions, such as ensuring that the number of instances after scaling up or down does not exceed the maximum or minimum instance limit set by the user in the elastic configuration, preventing unlimited scaling up or shrinking to zero.

[0246] The workload execution layer of the above system is the object that ultimately performs scaling operations. It can be various workload resources in the cloud-native environment, such as Deployment, horizontal Pod Autoscaler (HPA), Lightweight Workload Service (LWS), etc.

[0247] Through the collaborative work of the above modules, the dynamic scaling instance system for inference services provided by this invention can acquire data from multi-dimensional indicator sources, conduct intelligent predictive analysis and decision-making, and finally achieve efficient and refined management of AI inference service resources by safely and reliably scaling the underlying workload through a boundary protection mechanism.

[0248] Figure 7 This is a schematic diagram of the structure of the inference service instance scaling device provided by the present invention, as shown below. Figure 7 As shown, the inference service instance scaling device provided by the present invention mainly includes:

[0249] The data acquisition module 71 is used to acquire historical and real-time operational data of the target inference service.

[0250] The active engine module 72 predicts the future business load of the target inference service based on the historical operating data, and generates an active scaling up / down instance decision based on the future business load.

[0251] The passive engine module 73 evaluates the current running status of the target inference service based on the real-time running data, and generates a passive scaling instance decision based on the current running status.

[0252] The decision fusion module 74 performs collaborative decision-making on the active scaling instance decision and the passive scaling instance decision based on a preset collaborative strategy to determine the final scaling instance instruction.

[0253] The instance deployment module 75 adjusts the number of instances of the target inference service according to the final scaling instance instruction.

[0254] It should be noted that the inference service instance scaling device provided by the present invention can execute the inference service instance scaling method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0255] The inference service instance scaling device provided by this invention establishes a dual-engine collaborative mechanism that combines proactive prediction and passive response. While using historical data to achieve forward scaling to reduce latency, it also uses real-time data for fallback correction to cope with sudden loads. This effectively solves the problem of delayed response or resource waste in a single scaling method, and significantly improves the resource utilization and service quality stability of the inference service.

[0256] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for scaling up and down inference service instances. This method includes: acquiring historical and real-time operational data of the target inference service; predicting the future business load of the target inference service based on the historical operational data, and generating an active scaling up instance decision based on the future business load; evaluating the current operational state of the target inference service based on the real-time operational data, and generating a passive scaling up instance decision based on the current operational state; performing a collaborative decision on the active and passive scaling up instance decisions based on a preset collaborative strategy to determine a final scaling up instance instruction; and adjusting the number of instances of the target inference service according to the final scaling up instance instruction.

[0257] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0258] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the inference service instance scaling method provided in the above embodiments, the method including: acquiring historical operating data and real-time operating data of a target inference service; predicting the future business load of the target inference service based on the historical operating data, and generating an active scaling instance decision based on the future business load; evaluating the current operating state of the target inference service based on the real-time operating data, and generating a passive scaling instance decision based on the current operating state; performing a collaborative decision on the active scaling instance decision and the passive scaling instance decision based on a preset collaborative strategy to determine a final scaling instance instruction; and adjusting the number of instances of the target inference service according to the final scaling instance instruction.

[0259] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the inference service instance scaling method provided in the above embodiments. The method includes: acquiring historical and real-time operating data of a target inference service; predicting the future business load of the target inference service based on the historical operating data, and generating an active scaling instance decision based on the future business load; evaluating the current operating state of the target inference service based on the real-time operating data, and generating a passive scaling instance decision based on the current operating state; performing a collaborative decision on the active scaling instance decision and the passive scaling instance decision based on a preset collaborative strategy to determine a final scaling instance instruction; and adjusting the number of instances of the target inference service according to the final scaling instance instruction.

[0260] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0261] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scaling up and down inference service instances, characterized in that, include: Obtain historical and real-time operational data of the target inference service; Based on the historical operating data, predict the future business load of the target inference service, and generate proactive scaling up and down instance decisions based on the future business load; Determine the dynamic weights corresponding to each indicator data in the real-time operating data; The dynamic weights are used to perform weighted calculations on the various indicator data in the real-time running data to obtain a comprehensive score that characterizes the current health status of the target inference service; Compare the overall score with the preset expansion / contraction trigger threshold; If the comprehensive score meets the triggering condition corresponding to the scaling trigger threshold, a passive scaling instance decision is generated. Based on a preset collaborative strategy, the active scaling instance decision and the passive scaling instance decision are collaboratively decided to determine the final scaling instance instruction. Adjust the number of instances of the target inference service according to the final scaling instance instruction; The step of using the dynamic weights to weight and calculate the various indicator data in the real-time operating data to obtain a comprehensive score representing the current health status of the target inference service includes: Obtain the total resource capacity information, allocated resource information, and used resource information of the target node carrying the target inference service; Based on the total resource capacity information, the resource usage information, and the dynamic weight, a first sub-score is calculated, which is used to characterize the resource remaining degree of the target node. Based on the total resource capacity information and the resource allocation information, determine the dispersion of the allocation ratio of various types of resources in the target node; A second sub-score is calculated based on the degree of dispersion, and the second sub-score is used to characterize the resource balance of the target node; The first sub-score and the second sub-score are fused together to obtain the comprehensive score.

2. The method for scaling up and down inference service instances according to claim 1, characterized in that, The historical operational data and the real-time operational data include at least one of the following or a combination thereof: Application layer metrics data related to the service quality of the target inference service; Resource layer index data related to the computing resource status of the target node carrying the target inference service; Network layer metrics data related to the data transmission performance of the data interaction process with the target inference service.

3. The method for scaling up and down inference service instances according to any one of claims 1-2, characterized in that, The step of predicting the future business load of the target inference service based on the historical operational data, and generating proactive scaling decisions for instances based on the future business load, includes: The historical operating data is input into the load prediction model to obtain the prediction results representing the future business load output by the load prediction model; The proactive scaling instance decision is generated based on the future business load. The load prediction model is obtained by training a time series prediction model based on multiple historical time series data samples and future time series data labels corresponding to each historical time series data sample.

4. The method for scaling up and down inference service instances according to claim 3, characterized in that, The load prediction model includes an encoder, a decoder, and an attention mechanism module. The step of inputting the historical operating data into the load prediction model and obtaining the prediction result representing the future business load output by the load prediction model includes: The historical running data is input into the encoder so that the encoder can be used to extract features from the historical running data and generate a context vector containing time-dependent features. The attention mechanism module performs a weighted combination of the features output by the encoder at each time step to determine the key historical state corresponding to the current prediction step. The decoder is used to decode the context vector and the key historical states to gradually generate the prediction results that characterize the future business load.

5. The method for scaling up and down inference service instances according to claim 4, characterized in that, The load prediction model also includes an input feature fusion module and an output processing module; Before inputting the historical running data into the encoder, the method further includes: using the input feature fusion module to normalize and uniformly map the historical running data; After using the decoder to decode the context vector and the key historical state to gradually generate the prediction result representing the future business load, the method further includes: using the output processing module to perform inverse normalization processing and / or error correction processing on the prediction result.

6. The method for scaling up and down inference service instances according to claim 1, characterized in that, The dynamic weights are determined through the following steps: Obtain the business type characteristics of the target inference service; If the service type characteristic indicates a latency-sensitive service, then increase the dynamic weight corresponding to the latency indicator; If the business type characteristic indicates that it is a throughput-sensitive business, then the dynamic weight corresponding to the throughput indicator or energy consumption indicator is increased.

7. The method for scaling up and down inference service instances according to claim 1, characterized in that, The method of coordinating the active scaling and passive scaling decisions based on a preset collaborative strategy to determine the final scaling instruction includes: Calculate the rate of change of the real-time request volume of the target inference service; Obtain a status evaluation value that characterizes the current running state of the target inference service; If the rate of change is greater than a preset rate threshold, and / or the state evaluation value indicates that the target inference service is in an abnormal operating state, the passive scaling instance decision is selected as the final scaling instance instruction. If the rate of change is not greater than the rate threshold, and the state evaluation value indicates that the target inference service is operating normally, the active scaling instance decision is selected as the final scaling instance instruction.

8. The method for scaling up and down inference service instances according to claim 7, characterized in that, The step of obtaining a state evaluation value to characterize the current operating state of the target inference service includes: Obtain a comprehensive score, and use the comprehensive score as the status assessment value; The comprehensive score is obtained by weighting each indicator in the real-time running data using the dynamic weights corresponding to each indicator in the real-time running data. The status assessment value indicates that the target inference service is in an abnormal operating state, corresponding to the comprehensive score being lower than a preset health threshold.

9. The method for scaling up and down inference service instances according to claim 8, characterized in that, Before determining the passive scaling instance decision as the final scaling instance instruction, the method further includes: Determine whether the active scaling up / down instance decision indicates a reduction in the number of instances and whether the passive scaling up / down instance decision indicates an increase in the number of instances; If so, the selection of the active scaling instance decision is cancelled, and the passive scaling instance decision is selected as the final scaling instance instruction.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the inference service instance scaling method as described in any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the inference service instance scaling method as described in any one of claims 1 to 9.

Images