A cloud-native elasticity scaling method and system based on queue depth

Abstract

The application discloses a cloud native elasticity scaling method and system based on queue depth, relates to the technical field of cloud native elasticity scaling, and comprises the following steps: collecting message queue topic partition queue depth, aggregating global summaries according to topics, obtaining global queue message total quantity as a scaling invariant aggregation index, inputting the time series data into an extrapolation prediction model to obtain predicted global queue message total quantity, tracking container cold start delay by using an online exponential weighted moving average estimator to obtain an adaptive decision window, inputting the predicted global message total quantity and the adaptive decision window into a model predictive controller to solve a forward-looking scaling decision, determining a load type, issuing a scaling task to a steady-state or transient resource pool and horizontally scaling, and making a cooperative processing node cooperatively scale according to a replica number change amount and a throughput ratio. The application eliminates feedback interference of monitoring signals caused by capacity expansion and contraction, realizes anti-oscillation scaling, and takes into account baseline low cost and peak uninterrupted service.

Description

Technical Field

[0001] This invention relates to the field of cloud-native elastic scaling technology, specifically to a cloud-native elastic scaling method and system based on queue depth. Background Technology

[0002] In cloud-native data pipeline scenarios, stages such as document parsing, multi-agent annotation, and quality verification are linked through message queues to form an end-to-end processing chain. Existing technologies typically employ Kubernetes Horizontal Pod Autoscaler (HPA) to trigger scaling based on processor and memory utilization, or Kubernetes Event-Driven Autoscaler (KEDA) to trigger scaling based on event metrics such as message queue backlog. However, scaling based on resource utilization cannot reflect the true business pressure of task backlog in the queue. Inference containers may have low utilization due to input / output waiting, while the queue may already have a large backlog of tasks waiting to be processed. While event-based scaling more closely reflects real-world load, it remains a reactive mechanism with the following drawbacks: First, it only initiates expansion after detecting backlog exceeding a threshold, resulting in a cold start delay between the decision and the new instance's readiness. During this period, the task backlog continues to grow, potentially disrupting the pipeline. Second, using single-instance metrics means that after expansion, each instance's metrics drop instantaneously due to load redistribution, causing premature scaling-down decisions and resulting in scaling-down oscillations. Third, applying a uniform strategy to all workloads fails to differentiate between baseline steady-state loads and sudden peak loads, leading to resource waste during low-load periods and insufficient elastic capacity to absorb traffic surges during peak periods. Therefore, a proactive, oscillating, cost-effective, and peak-absorbing elastic scaling solution is needed. Summary of the Invention

[0003] Existing elastic scaling solutions based on resource utilization or message queue event metrics are reactive mechanisms, which suffer from problems such as cold start delays leading to lag in expansion and pipeline interruptions, scaling fluctuations caused by distortion of single-instance metrics, and baseline resource waste and insufficient peak capacity under a unified strategy.

[0004] To address the aforementioned technical problems, this invention provides a cloud-native elastic scaling method based on queue depth, comprising:

[0005] Collect real-time queue depth data for each topic partition in the message queue, aggregate the real-time queue depth data by topic dimension, and then summarize it globally to obtain the total number of messages in the global queue.

[0006] The time-series historical data of the total global queue message volume is input into the extrapolation prediction model for prediction, and the predicted total global queue message volume is output.

[0007] An adaptive decision window is obtained by tracking the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator.

[0008] The predicted total number of global queue messages and the adaptive decision window are input into the model prediction controller for solution to obtain a forward-looking scalable decision.

[0009] Based on the forward-looking scaling decision, the current load state type is determined, and the scaling task is distributed to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0010] Based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the co-processing node, the target number of replicas of the co-processing node is determined to achieve coordinated scaling between the co-processing node and the inference container unit.

[0011] Preferably, the real-time queue depth data of each topic partition in the message queue is collected, and the real-time queue depth data is aggregated by topic dimension and then globally summarized to obtain the total number of global queue messages. This includes: collecting the number of messages to be consumed in each topic partition of the message queue to obtain the real-time queue depth data; aggregating the real-time queue depth data by topic dimension to obtain the total queue depth of each topic; and globally summarizing the total queue depth of each topic to obtain the total number of global queue messages. The total number of global queue messages represents the total number of messages that have been enqueued but not yet consumed at the moment the scaling operation occurs. The scaling operation changes the consumption rate but does not change the total number of messages that have been enqueued but not yet consumed.

[0012] Preferably, the step of inputting the time-series historical data of the total global queue messages into the extrapolation prediction model for prediction and outputting the predicted total global queue messages includes: obtaining the time-series historical data of the total global queue messages within a preset historical time window; inputting the time-series historical data into the extrapolation prediction model for extrapolation prediction and outputting the predicted total global queue messages, wherein the predicted total global queue messages is a predicted value of the total global queue messages within a future preset prediction time window.

[0013] Preferably, the step of tracking the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator to obtain an adaptive decision window includes: in each decision cycle, recording the time interval from issuing the expansion command to the inference container unit created by the expansion being ready, to obtain the measured cold start latency; based on a decay factor with a preset initial value, using the measured cold start latency as the latest observation value, and performing exponentially weighted fusion of the latest observation value and the adaptive decision window of the previous decision cycle to obtain the adaptive decision window.

[0014] Preferably, the step of inputting the predicted total number of global queue messages and the adaptive decision window into the model prediction controller for solving to obtain a forward-looking scaling decision includes: within each decision period, the model prediction controller uses the predicted total number of global queue messages as the input signal and the adaptive decision window as the prediction window to solve for the optimal replica number sequence in a rolling optimization manner under a preset replica number constraint, and uses the replica number in the current decision period in the optimal replica number sequence as the forward-looking scaling decision, wherein the replica number constraint includes: a maximum replica number upper limit, a minimum replica number lower limit, a maximum step size for a single expansion, and a maximum step size for a single reduction.

[0015] Preferably, the step of determining the current load state type based on the forward scaling decision, and distributing scaling tasks to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit, includes: determining the current load state type, which includes a stable type and an overflow type, based on the comparison result between the target number of replicas determined by the forward scaling decision and the preset upper limit of the number of replicas in the stable resource pool; when the current load state type is a stable type, distributing scaling tasks to the stable resource pool composed of reserved nodes to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit; when the current load state type is an overflow type, distributing the scaling tasks corresponding to the portion exceeding the upper limit of the number of replicas in the stable resource pool to a transient resource pool composed of Serverless instances to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0016] Preferably, when the current load state type is an overflow type, it further includes:

[0017] Messages exceeding the carrying capacity of the steady-state resource pool in the original topic of the message queue are delivered to a dedicated overflow topic in the message queue; Serverless instances in the transient resource pool obtain and process messages from the overflow topic through a dedicated consumer group, thereby completing the horizontal scaling of the transient resource pool. The dedicated consumer group is isolated from the consumer group in the steady-state resource pool that consumes the original topic.

[0018] Preferably, when the horizontal scaling is horizontal scaling down, it includes: first scaling down the Serverless instances in the transient resource pool until the number of instances in the transient resource pool is scaled down to zero, and then scaling down the inference container units in the steady-state resource pool.

[0019] Preferably, after performing horizontal scaling on the inference container unit based on the forward scaling decision, the method further includes: collecting the actual total number of global queue messages after the scaling is performed, calculating the difference between the actual total number of global queue messages and the predicted total number of global queue messages to obtain the scaling decision error; when the absolute value of the scaling decision error exceeds a preset error threshold within a preset number of consecutive decision periods, adjusting the decay factor of the online exponentially weighted moving average estimator, and correcting the parameters of the prediction model of the model prediction controller.

[0020] Preferably, determining the target number of replicas of the co-processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the co-processing node, so as to achieve coordinated scaling between the co-processing node and the inference container unit, includes: determining the throughput ratio by comparing the steady-state throughput of a single instance of the inference container unit with the steady-state throughput of a single instance of the co-processing node, wherein the steady-state throughput of a single instance is the throughput value measured when the corresponding node is running independently and the resource utilization is within a preset stable range; calculating the product of the change in the number of replicas of the inference container unit and the ratio of the throughput, and rounding the product up to obtain the change in the number of replicas of the co-processing node; and determining the target number of replicas of the co-processing node based on the current number of replicas of the co-processing node and the change in the number of replicas of the co-processing node, so as to achieve coordinated scaling between the co-processing node and the inference container unit.

[0021] This invention also provides a cloud-native elastic scaling system based on queue depth, comprising:

[0022] The queue depth aggregation module is used to collect real-time queue depth data of each topic partition in the message queue, aggregate the real-time queue depth data by topic dimension and then summarize it globally to obtain the total number of global queue messages.

[0023] The queue depth prediction module is used to input the time series historical data of the total global queue message volume into the extrapolation prediction model for prediction, and output the predicted total global queue message volume.

[0024] The decision window estimation module is used to track the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator to obtain an adaptive decision window;

[0025] The scaling decision-solving module is used to input the total predicted global queue message volume and the adaptive decision window into the model prediction controller for solving, so as to obtain a forward-looking scaling decision.

[0026] A two-layer elastic scaling module is used to determine the current load state type based on the forward scaling decision, and to distribute scaling tasks to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0027] The collaborative scaling decision module is used to determine the target number of replicas of the collaborative processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the collaborative processing node, so as to realize the collaborative scaling of the collaborative processing node and the inference container unit.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. This invention uses the total number of messages in the global queue as the scaling indicator. The scaling value remains approximately constant during the scaling process and is not affected by the distortion of single-instance indicators caused by load redistribution during scaling, thereby suppressing scaling oscillations.

[0030] 2. This invention tracks the cold start delay and adaptively determines the decision window through an exponentially weighted moving average estimator. The model prediction controller rolls the solution and issues it in advance, so that scaling changes from reactive to prospective, avoiding backlog and interruption caused by capacity expansion lag.

[0031] 3. This invention divides the resource pool into a reserved node stable pool and a serverless instance transient pool. The target number of replicas and the stable upper limit are used to distinguish between stable and overflow. Overflow traffic is diverted through a dedicated overflow topic, which takes into account both low baseline cost and uninterrupted absorption of peak traffic.

[0032] 4. This invention uses the ratio of the change in the number of replicas of the inference container unit to the throughput to round up the collaborative processing node for synchronous scaling, so that inference and verification are matched, and the verification process is avoided from becoming a bottleneck during peak periods. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the overall process of a cloud-native elastic scaling method based on queue depth according to the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the isolation between the steady-state and transient two-layer elastic architecture and the consumer group in this invention;

[0036] Figure 3 This is a decision-making flowchart for the collaborative scaling of the collaborative processing node and the inference container unit in this invention;

[0037] Figure 4 This is a schematic diagram of the structure of a cloud-native elastic scaling system based on queue depth according to the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0039] Example 1

[0040] like Figure 1 As shown, this invention provides a cloud-native elastic scaling method based on queue depth, comprising:

[0041] S1: Collect real-time queue depth data for each topic partition in the message queue, aggregate the real-time queue depth data by topic dimension, and then summarize it globally to obtain the total number of messages in the global queue.

[0042] In cloud-native data pipelines, stages such as literature parsing, multi-agent annotation, and quality verification are linked together using message queues (e.g., Kafka or Pulsar). Metric collection probes deployed on each broker node of the message queue periodically read the number of currently enqueued but not yet consumed messages in each topic partition, obtaining real-time queue depth data for each topic partition. Edge computing nodes sum the queue depths of all partitions under the same topic to obtain the total queue depth for that topic, and then globally summarize the total queue depths of all topics to obtain the total global queue message volume. The total global queue message volume represents the total number of messages enqueued but not yet consumed at the moment of scaling. It remains approximately constant at the moment of scaling—horizontal scaling changes the message consumption rate rather than the existing queued message inventory. Therefore, this metric is unaffected by the load redistribution effect caused by scaling, providing a stable input signal for subsequent predictions. The total global queue message volume is summarized by the following formula:

[0043]

[0044] in D topic,i Let D be the total queue depth of the i-th topic. g d represents the total number of messages in the global queue. i,j Let M be the real-time queue depth of the j-th partition of the i-th topic, and P be the total number of topics. i This represents the total number of partitions for the i-th topic. Furthermore, the metric collection probe directly reads the location metadata from the message queue broker node to obtain depth, completely decoupling it from the processing capacity of specific consumer instances. Therefore, even if scaling up or down causes changes in the set of consumer instances, the total global queue message count obtained by the probe remains stable, further solidifying scalability invariance. This scalability-invariant aggregate metric serves as the common input for extrapolation prediction, adaptive decision window estimation, and model predictive control, freeing the entire elastic link from dependence on easily distorted single-instance dimension metrics. For example, in a link containing 3 topics, each topic has 4, 4, and 2 partitions respectively. The cumulative real-time queue depth of each partition yields a total global queue message count of 22,000, which serves as a stable input for subsequent predictions.

[0045] It's important to note that traditional scaling solutions based on single-instance metrics (such as CPU utilization of a single Pod or Consumer Lag of a single partition) suffer from inherent signal distortion. When a cluster expands by adding consumer instances, the Kafka or Pulsar consumer group rebalancing redistributes partitions between the old and new instances. This results in a decrease in the number of partitions allocated to each instance and a sudden drop in the amount of data pending processing per instance. The reactive controller misinterprets this as a load decline and prematurely triggers scaling down, leading to scaling-down oscillations. gAs the total number of messages in the global queue, the scaling depends only on the total number of messages that have been enqueued but not yet consumed, regardless of how partitions are allocated between instances, thus eliminating the aforementioned sources of distortion at the signal construction level.

[0046] S2: Input the time series historical data of the total number of global queue messages into the extrapolation prediction model for prediction, and output the predicted total number of global queue messages.

[0047] The cloud server continuously collects historical time-series data of the total global queue messages over time, and inputs the sequence within a preset historical time window into an extrapolation prediction model. The extrapolation prediction model uses the total global queue messages as the input signal rather than a single-instance dimensional indicator, thus avoiding the problem of distorted prediction signals caused by the instantaneous drop in single-instance indicators due to load redistribution after scaling. The extrapolation prediction model fits and extrapolates the changing trends of the historical sequence, outputting a predicted value of the total global queue messages within the preset prediction time window. The extrapolation prediction model uses the total global queue messages in each sampling period within the preset historical time window as samples, performs least-squares trend fitting in chronological order to obtain the trend slope and intercept, and extends the fitted trend along the time axis to the prediction time window to obtain the predicted value. The trend slope and intercept are updated with each new sampling period to reflect the latest evolution direction of the load. The predicted total global queue messages characterize the evolution trend of the pending message inventory over a future period, providing input for forward-looking scaling decisions. By using aggregated metrics instead of single-instance metrics as prediction inputs, the extrapolation results are not affected by the instantaneous decrease in the number of instances due to load redistribution after expansion. The prediction window length is kept consistent with the adaptive decision window, thereby ensuring that the total amount of global queue messages predicted directly corresponds to the future load required for subsequent model predictive control.

[0048] S3: An adaptive decision window is obtained by tracking the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator.

[0049] There is a cold start latency between the issuance of a scaling instruction and the new instance becoming ready to consume messages. This latency includes stages such as instance scheduling, image fetching, container initialization, and application readiness, and fluctuates with the cluster environment and time period. The cloud server records the measured cold start latency once per decision cycle and smooths it using an online exponentially weighted moving average (EWMA) estimator to obtain an adaptive decision window. The decision cycle is a fixed time interval between two consecutive executions of model predictive control solution and scaling decision issuance by the scaling controller. Its length is an integer multiple of the sampling interval for metric collection; for example, when the sampling interval is 4 seconds, the decision cycle can be 12 seconds (i.e., solving once every 3 sampling cycles). The number of decision cycles N contained in the adaptive decision window is... H It equals the adaptive decision window length divided by the duration of a single decision cycle and rounded down. Specifically, the estimator uses the measured cold start delay τ of the inference container unit in the k-th decision cycle. k Weighted by a decay factor α (ranging from 0.3 to 0.5), and compared with the adaptive decision window H of the previous decision cycle. k-1 By fusing the complementary weights 1-α, the adaptive decision window H for this decision cycle is obtained. k Because the adaptive decision window automatically adjusts according to changes in the measured cold start delay, the prediction window length for subsequent rolling optimization matches the actual cold start delay. For example, with a decay factor of 0.4, an adaptive decision window of 80 seconds in the previous cycle, and a measured cold start delay of 120 seconds in the current cycle, the updated adaptive decision window is 0.4 × 120 + 0.6 × 80 = 96 seconds, corresponding to N... H (For example, 8 decision cycles, obtained by dividing 96 seconds by 12 seconds) is consistent with the example of a prediction window containing 8 decision cycles.

[0050] S4: Input the predicted total number of global queue messages and the adaptive decision window into the model prediction controller for solving to obtain a forward-looking scalable decision.

[0051] The cloud server uses the predicted total number of messages in the global queue as input and an adaptive decision window as the prediction window, which are then input to the Model Predictive Control (MPC) for rolling optimization. Within the prediction window, the MPC simultaneously minimizes the deviation of the predicted queue depth from the target level and the variation in the number of replicas in adjacent decision periods. Its objective function is:

[0052]

[0053] Where, r kLet be the number of replicas in the k-th decision cycle. D is the predicted total number of global queue messages in the k-th decision period. tgt Let N be the target queue depth level (e.g., 8000 queues), λ1 be the weight of the deviation term, λ2 be the weight of the oscillation suppression term, and N be the depth level of the queue. H This represents the number of decision cycles contained in the adaptive decision window. The solution must satisfy the replica number constraint, including the maximum replica number upper limit R. max (e.g., 32), minimum replica count lower bound R min (Example 3) Maximum step size ∆R for a single expansion up (For example, 8) and the maximum step size ∆R of a single volume reduction down (Example 4). Model predictive control is essentially a rolling closed-loop optimal control: each decision cycle adopts only the first item of the complete optimal sequence, rather than issuing the entire plan at once, thus limiting the time-varying characteristics of prediction errors and cold start delays to a single cycle and absorbing them through re-optimization in the next cycle. Setting the prediction window length as an adaptive decision window, so that the optimization perspective exactly covers the delay from the expansion command to the new instance being ready, is the key to achieving look-ahead scaling in this invention; if the prediction window is shorter than the cold start delay, the expansion command will have no backlog buffer before the instance is ready, losing its look-ahead significance. This solution is executed periodically by the cloud server in each decision cycle, with computational overhead far lower than the requirements of real-time control loops, and the obtained first item replica count is issued to the corresponding resource pool for execution via the orchestration system. The model predictive controller uses the replica count of the current decision cycle in the obtained optimal replica count sequence as the look-ahead scaling decision, thereby completing the expansion deployment in advance before the cold start delay arrives.

[0054] S5: Based on the forward scaling decision, determine the current load state type, and according to the current load state type, distribute the scaling task to the corresponding resource pool to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0055] The elastic resource pool is divided into a stable resource pool composed of reserved nodes (i.e., pre-allocated and permanently running dedicated compute nodes, offering advantages such as low unit cost and image preheating) and a transient resource pool composed of serverless instances. The former carries the baseline inference load, featuring high resource utilization and low unit compute cost, while the latter is used to absorb sudden peak traffic. The scaling controller compares the target replica count determined by the forward scaling decision with the preset replica count limit of the stable resource pool to determine the current load state type: if the target replica count does not exceed the replica count limit of the stable resource pool, it is determined to be a stable type; otherwise, it is determined to be an overflow type. The scaling controller distributes scaling tasks to the corresponding resource pool based on the current load state type, performs horizontal scaling on the inference container units, and obtains the change in the replica count of the inference container units. When performing horizontal scaling, the branch processing for stable and overflow types unfolds along with the load state type determination and the two-layer elastic execution. The overflow topic and dedicated consumer group isolation mechanism under the overflow type are established in the consumer group isolation stage; when performing horizontal scaling down, the processing is carried out in the horizontal scaling down execution order. The change in the number of replicas of the resulting inference container unit serves as the basis for collaborative scaling.

[0056] S6: Based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the collaborative processing node, determine the target number of replicas of the collaborative processing node to achieve collaborative scaling between the collaborative processing node and the inference container unit.

[0057] The quality verification stage of the data ingestion and labeling pipeline is handled by collaborative processing nodes (e.g., data quality assurance nodes). If their replica count does not adjust synchronously with the inference container units, they will become a throughput bottleneck during peak periods. The cloud server determines the replica count change of the collaborative processing nodes based on the change in the replica count of the inference container units and the collaborative scaling factor, thereby determining the target replica count of the collaborative processing nodes.

[0058]

[0059] Where γ is the cooperative scaling factor, T r T represents the steady-state throughput of a single instance of the inference container unit. c Let Δn be the steady-state throughput of a single instance of the collaborative processing node, Δm be the change in the number of replicas of the inference container unit, and Δm be the change in the number of replicas of the collaborative processing node. This is for rounding up. The target number of replicas for a co-processing node is obtained by adding its current number of replicas to the change in the number of replicas. Since the co-processing nodes always scale in tandem with the inference container units according to their throughput ratio, the processing capabilities of the inference and verification stages remain balanced. For example, if the steady-state throughput of a single instance of the inference container unit is 50 messages per second, and the co-processing node's throughput is 40 messages per second, with a scaling factor of 1.25, when the inference container unit expands by 8 units, the co-processing nodes need to expand by 8 × 1.25 = 10 units.

[0060] Example 2

[0061] Based on Example 1, the real-time queue depth data of each topic partition in the collected message queue is aggregated by topic dimension and then globally summarized to obtain the total number of global queue messages, including:

[0062] S1.1: Collect the number of messages to be consumed in each topic partition of the message queue to obtain the real-time queue depth data.

[0063] Metric collection probes deployed on each broker node of the message queue read the consumption and production positions of each topic partition at a fixed sampling period. The difference between the two is the number of messages currently enqueued but not yet consumed in that topic partition. The calculation relationship is as follows:

[0064]

[0065] Where, d i,j L represents the real-time queue depth of the i-th topic and the j-th partition. prod,i,j L represents the production location (i.e., the cumulative offset of produced messages) of the j-th partition of the i-th topic. cons,i,j The difference between the corresponding consumption point (i.e., the cumulative offset of consumed messages) and the value of the consumption point is the number of unconsumed messages in that partition. The metric collection probe reads all topic partitions in parallel, aggregating the number of unconsumed messages in each topic partition into real-time queue depth data. To reduce the pressure on the message queue service, the metric collection probe uses a combination of batch fetching and local caching, and removes and interpolates abnormal reads. The real-time queue depth data is organized using topic identifiers and partition numbers as indexes, forming a raw data structure that can be used for subsequent topic-based aggregation. Production points can be obtained from the committed offset or high-water mark of the message queue, while consumption points can be obtained from the committed offset of the consumption group; the difference between the two is the number of unconsumed messages in the partition. The metric collection probe uses a combination of batch fetching and local caching, reading multiple partition points in one network round trip, significantly reducing the request pressure on the message queue broker; readings of points that fail to read or are obviously abnormal are removed and repaired by linear interpolation within adjacent sampling period intervals, avoiding single-point omissions from polluting the aggregation results.

[0066] S1.2: Aggregate the real-time queue depth data by topic dimension to obtain the total queue depth of each topic.

[0067] Edge computing nodes aggregate real-time queue depth data by topic dimension at the near end. This involves summing the number of messages to be consumed across all partitions under the same topic to obtain the total queue depth for that topic, or summing the real-time queue depths of all partitions under the same topic. This summation is then performed on all topics to obtain the total queue depth for each topic. Aggregating by topic first can shield against short-term fluctuations caused by partition-level load jitter and rebalancing, resulting in more stable topic-level metrics, as the instantaneous backlog fluctuations of a single partition are smoothed out after topic-level summation. The total queue depth of each topic reflects the backlog level at each processing stage, with topics having larger total queue depths corresponding to the current bottleneck stage. Since aggregation is completed near the edge computing nodes, only a small amount of aggregated values ​​at the topic level needs to be uploaded instead of the raw readings of all partitions, significantly reducing the amount of data transmitted to the cloud and network overhead. Aggregating by topic first effectively shields against instantaneous jitter at the partition level. For example, during partition rebalancing, the migration of partitions between consumer group instances can cause a brief and sudden change in the depth of a single partition, while topic-level summation cancels out the fluctuations of multiple partitions, resulting in a more stable total queue depth for each topic. After the edge computing nodes complete aggregation at the near end, they only need to upload a few topic-level aggregated values ​​to the cloud instead of the full original partition readings. In scenarios with hundreds of partitions, this can reduce the amount of data uploaded by one to two orders of magnitude, significantly saving cross-network bandwidth. The total depth of the topic queue corresponds one-to-one with each processing stage, and its size directly identifies the current bottleneck stage, providing a basis for subsequent load localization.

[0068] S1.3: Globally summarize the total queue depth of each topic to obtain the total global queue message volume, wherein the total global queue message volume represents the total number of messages that have been enqueued but not yet consumed at the moment the scaling operation occurs, and the scaling operation changes the consumption rate without changing the total number of messages that have been enqueued but not yet consumed.

[0069] Edge computing nodes further sum the total queue depth of each topic to complete the global aggregation, obtaining the total global queue message volume. This total volume is the sum of the total queue depth of each topic. The total global queue message volume represents the total number of messages enqueued but not yet consumed at the moment the scaling operation occurs. Horizontal scaling changes the message consumption rate by adding or removing consumption instances, but the stock of enqueued but unconsumed messages does not change at the moment the scaling action occurs. Therefore, this indicator is approximately constant at the moment of scaling, possessing scaling invariance. Since this indicator is not distorted by load redistribution caused by scaling, predictions and decisions built based on it are not affected by feedback. This scaling invariance is a key feature that distinguishes it from single-instance dimension indicators and is also the physical basis for suppressing scaling oscillations. Global aggregation is completed at the edge computing nodes, involving only a lightweight summation of the aggregated values ​​of each topic, with negligible computational overhead. Scaling invariance has a clear physical meaning: horizontal scaling changes the consumption capacity per unit time (i.e., consumption rate), and the stock of enqueued but unconsumed messages does not change at the moment the scaling command takes effect. Therefore, the total global queue message volume remains approximately constant at this moment. In contrast, looking at single-instance metrics, the reduction in partitions allocated to each instance after scaling up and the instantaneous decrease in the amount of data pending processing per instance can cause reactive controllers to misjudge the load as a drop and trigger premature scaling down. This solution avoids this distortion at its source by using global inventory metrics.

[0070] In one implementation, the scaling invariance of the total number of messages in the global queue can be further illustrated by the following comparative scenario. Suppose that at a certain moment, the message queue contains D messages. g There are 20,000 messages to be consumed. Currently, four inference container units are deployed, with each instance handling an average of 5,000 consumption tasks (measured by ConsumerLag, i.e., a single-instance metric of 5,000). When a scaling decision increases the number of instances to eight, the consumer group rebalancing redistributes the partitions among the eight instances. As a result, the single-instance metric instantly drops to approximately 2,500, a 50% decrease. The reactive controller misinterprets this as a significant load drop; however, the total number of messages in the global queue, D, remains unchanged. g The number of records remains unchanged at the moment the expansion command takes effect (the new instances have not yet started consuming), still at 20,000. The model prediction controller is based on D. g The decisions made are unaffected by this instantaneous disturbance, thus avoiding the problem of prematurely triggering shrinkage oscillations.

[0071] Example 3

[0072] Based on Example 1, the step of inputting the time-series historical data of the total global queue message volume into the extrapolation prediction model for prediction, and outputting the predicted total global queue message volume, includes:

[0073] S2.1: Obtain the time-series historical data of the total number of messages in the global queue within a preset historical time window.

[0074] The cloud server stores the total global queue messages obtained from each sampling period into a time-series storage system in chronological order, forming a historical sequence that evolves over time. In each decision period, samples within a preset historical time window are extracted from the time-series storage to obtain the time-series historical data of the total global queue messages. Each sample of the time-series historical data is a binary tuple (sampling time, total global queue messages), where the total global queue messages is the scalable invariant aggregate indicator (i.e., the total global queue messages, which is the global summary value of the messages to be consumed in each topic partition), rather than an arbitrary single-instance dimension indicator. The sequence consists of several of these binary tuples arranged chronologically, characterizing the evolution trajectory of the scalable invariant aggregate indicator over time. The preset historical time window can be configured from 60 seconds to 300 seconds, preferably 120 seconds. This range balances trend capture and noise suppression: windows shorter than 60 seconds are susceptible to transient jitter and may underfit the trend, while windows longer than 300 seconds are slow to respond to sudden trend changes. Therefore, the historical sequence undergoes uniform resampling and moving median denoising before prediction to align sampling times and suppress outliers. The historical time series data consists of the total global queue messages at equally spaced time intervals. Its time span is determined by a preset historical time window, and the sampling density is determined by the sampling interval. Both factors jointly determine the series' ability to characterize load trends and serve as input for extrapolation prediction. For example, with a sampling interval of 4 seconds and a preset historical time window of 120 seconds as an optimal value, the total global queue messages at the most recent 30 equally spaced time intervals are obtained as samples (i.e., 30 pairs of (sampling time, total global queue messages)), constituting the historical time series data.

[0075] S2.2: Input the time series historical data into the extrapolation prediction model for extrapolation prediction, and output the predicted total amount of global queue messages, wherein the predicted total amount of global queue messages is the predicted value of the total amount of global queue messages within a future preset prediction time window.

[0076] Extrapolation prediction models can use well-known trend fitting methods such as linear regression or exponential smoothing to fit and extrapolate the trend of historical time series data. Taking linear regression as an example, a linear trend term is fitted to the historical series with respect to time, obtaining the trend slope 'a' and the intercept term 'b'. Then, the current time 't' is added to the preset prediction time window length Δt (its value range can be configured from 60 seconds to 300 seconds, preferably 120 seconds) as the extrapolation time. The trend slope is multiplied by this extrapolation time and then added to the intercept term to obtain the total number of global queue messages predicted in the k-th decision period.

[0077]

[0078] Where t is the current time, a is the slope of the fitted trend, and b is the intercept term of the fitted trend. The predicted total amount of global queue messages is the predicted value of the total amount of global queue messages within a future preset prediction time window. Since the scaling-invariant aggregate index is used as input, the extrapolation result is not affected by the load redistribution during scaling up and down. Extrapolation using the scaling-invariant aggregate index as input ensures that the predicted value is not affected by the instantaneous decrease of the single instance index after scaling up; the trend fitting is based on the assumption that the load evolves approximately linearly within the prediction window, and satisfactory accuracy can be obtained by fitting a term to the historical sequence once. The prediction window length (60 seconds to 300 seconds, preferably 120 seconds) is connected to the adaptive decision window, so that the prediction covers the cold start delay period.

[0079] When using the exponential smoothing method, the extrapolation prediction model calculates a quadratic exponential smoothing value on the time-series historical data of the total global queue messages. Based on the smoothed value and trend components, an extrapolation formula is constructed to obtain the predicted total global queue messages at the end of the preset prediction time window. The exponential smoothing method assigns higher weights to recent samples, enabling faster response to sudden changes in load trends, and is suitable for scenarios with relatively regular cyclical changes in production. Linear regression, on the other hand, processes all historical samples with equal weights, making it suitable for scenarios with relatively stable load evolution. Both methods use the total global queue messages as input, and the resulting predicted values... As a unified input signal for the model predictive controller, it does not affect the subsequent control logic due to the choice of prediction method.

[0080] Example 4

[0081] Based on Example 1, the step of tracking the cold start latency of the inference container unit in the current environment using an online exponentially weighted moving average estimator to obtain an adaptive decision window includes:

[0082] S3.1: Within each decision cycle, record the time interval between issuing the expansion command and the inference container unit created by the expansion being ready, and obtain the measured cold start delay.

[0083] The scaling controller tracks the scaling action within each decision cycle: starting from the moment the scaling command is issued, and ending at the moment when the inference container unit created by the scaling completes instance scheduling, image retrieval, container initialization, and application readiness and begins consuming messages. The time interval between these two moments is the measured cold start latency for this decision cycle, and its calculation relationship is as follows:

[0084] τ k =t ready,k -t issue,k

[0085] Where, τ k t represents the cold start delay of the inference container unit in the k-th decision cycle. issue,kThe time when the expansion command is issued in the k-th decision cycle is t. ready,k This refers to the moment when a new inference container unit is ready and begins consumption in the k-th decision cycle. To avoid bias caused by a single abnormal pull, the scaling controller takes a representative value of the ready time of multiple instances created in parallel within the same decision cycle as the measured cold start latency. This measured cold start latency objectively reflects the deployment time of a new instance under the current cluster environment (including node specifications, image repository bandwidth, and network latency). Its value varies with cluster load, image cache hit rate, and scheduling queue length, therefore it needs to be measured in real-time in each decision cycle rather than using a fixed empirical value. Timing is implemented by a timer embedded in the scaling controller: the time of issuing the scaling command is recorded, and the ready time is recorded when the new instance's readiness probe returns to a healthy state. The difference between the two is the measured cold start latency.

[0086] S3.2: Based on the decay factor of the preset initial value, the measured cold start delay is taken as the latest observation value, and the latest observation value and the adaptive decision window of the previous decision period are subjected to exponential weighted fusion to obtain the adaptive decision window.

[0087] The online exponentially weighted moving average estimator uses a preset initial decay factor α as the weight. It takes the measured cold start delay of the current decision period as the latest observation and performs exponential weighted fusion with the adaptive decision window of the previous decision period. The fusion method is given as an exponentially weighted fusion. The decay factor determines the relative weight of the latest observation and historical estimates: the larger the decay factor, the faster the estimator responds to recent changes in the cold start delay, but the more sensitive it is to noise; the smaller the decay factor, the smoother the response. Due to the exponentially weighted fusion, the influence of historical observations decays geometrically over the decision period, allowing the adaptive decision window to both follow the trend of cold start delay changes and suppress occasional fluctuations.

[0088] The online exponentially weighted moving average estimator uses a preset initial decay factor α as the weight, takes the measured cold start delay of the current decision period as the latest observation, and performs exponential weighted fusion with the adaptive decision window of the previous decision period. The fusion method is given by the following formula:

[0089]

[0090] in, Hk Let τ be the adaptive decision window for the k-th decision period. k H is the measured cold start delay for the k-th decision cycle. k-1Let α be the adaptive decision window for the previous decision period, and let α be the decay factor, ranging from 0.3 to 0.5. The decay factor determines the relative weight of the latest observation and the historical estimate: the larger the decay factor, the faster the estimator responds to recent changes in the cold start delay, but the more sensitive it is to noise; the smaller the decay factor, the smoother the response. Due to the exponentially weighted fusion, the influence of historical observations decays geometrically with the decision period, allowing the adaptive decision window to both follow the trend of the cold start delay and suppress occasional fluctuations. Specifically, taking the k-th decision period (i.e., the current decision period) as the fusion time, the cold start delay observation from the m-th historical decision period (i.e., the km-th decision period, m=1,2,…) is recorded as the historical observation; the weight of this historical observation in the fusion result at time k is α(1-α). m As the backtracking interval m increases, the weight decays geometrically with a complementary weight (1-α) as the ratio increases. That is, the older the observation, the smaller its weight, and the more recent the observation, the more dominant it becomes. In the edge-cloud collaborative deployment, the measured cold start latency is reported to the cloud by the ready callback of the new instance registered by the edge gateway. The estimator performs an exponential weighted fusion in each decision cycle accordingly. When the cold start latency jumps due to cluster upgrades or image changes, the larger decay factor causes the adaptive decision window to rise rapidly over several cycles to cover the longer ready latency, avoiding the expansion lag caused by an excessively short prediction window.

[0091] It is worth noting that the adaptive decision window maintains the same length as the prediction window of the model prediction controller, ensuring that the optimization perspective precisely covers the entire latency from issuing the scaling command to the actual readiness of the new instance. In cluster environments with short cold start latency (e.g., images are preheated and scheduling queues are idle), H k Automatically shrinking to a smaller value, the predictive controller solves by scrolling through shorter fields of view, reducing computational overhead; in environments with long cold start latency (e.g., initial fetching of large visual language model images, or limited node resources), H k Automatic scaling to cover longer readiness latency ensures that scaling commands take effect before the backlog of messages reaches the processing capacity limit. This adaptive feature eliminates the need to configure fixed prediction window parameters for various cluster environments during the deployment phase, providing excellent cross-environment generalization capabilities.

[0092] Example 5

[0093] Based on Example 1, the step of inputting the predicted total number of global queue messages and the adaptive decision window into the model prediction controller for solving to obtain a forward-looking scalable decision includes:

[0094] S4.1: In each decision cycle, the model prediction controller takes the total number of predicted global queue messages as the input signal and the adaptive decision window as the prediction window, and solves the optimal replica number sequence in a rolling optimization manner under the preset replica number constraint. The replica number in the current decision cycle in the optimal replica number sequence is used as the forward scaling decision. The replica number constraint includes: the maximum replica number upper limit, the minimum replica number lower limit, the maximum step size for a single expansion, and the maximum step size for a single shrinkage.

[0095] Within each decision cycle, the model prediction controller uses the total predicted global queue message volume as input and an adaptive decision window as the prediction window. Within the prediction window, it solves for the optimal replica number sequence that minimizes the objective function using rolling optimization (i.e., rolling time-domain optimization, RecedingHorizon). The objective function and replica number constraints are specifically constructed as follows. Rolling optimization means that in each decision cycle k, the controller does not directly distribute the entire sequence at once, but only takes the first item of the sequence as the forward scaling decision for the current cycle. Then, in the next decision cycle k+1, it re-solves with the updated predicted global queue message volume and adaptive decision window, thus forming a rolling closed loop of "prediction—solution—execution of the first item—re-prediction," continuously absorbing the time-varying characteristics of prediction errors and cold start delays. The objective function includes two terms: a bias term forces the predicted global queue message volume in each cycle within the prediction window to approach the target queue depth level, thereby avoiding task backlog or resource idleness; a smoothing term penalizes the replica number changes in adjacent decision cycles, suppressing scaling oscillations caused by drastic changes in the replica number. In the replica count constraints, the maximum replica count upper limit (e.g., 32) and the minimum replica count lower limit (e.g., 3) define the feasible boundaries of resource size, while the maximum step size for a single expansion (e.g., 8) and the maximum step size for a single reduction (e.g., 4) limit the amplitude of replica count jumps between adjacent periods to further suppress oscillations. Because the length of the adaptive decision window adjusts adaptively with the measured cold start delay, the prediction window precisely covers the latency from issuing the expansion command to the new instance being ready, thus ensuring that the expansion determined by the first replica count is deployed ahead of time before the backlog actually arrives.

[0096] Specifically, the ratio of the deviation term weight λ1 to the smoothing term weight λ2 in the objective function determines the trade-off between the controller's two objectives: "rapidly eliminating backlog" and "suppressing replica count oscillations." When λ1 / λ2 is large, the controller tends to eliminate queue backlog quickly with a more aggressive adjustment in the replica count, suitable for pipeline scenarios that are extremely sensitive to latency. When λ1 / λ2 is small, the controller tends to trade off a more gradual change in the replica count for a lower scaling frequency, suitable for scenarios where cold start costs are high or Serverless billing granularity is coarse. These two weights are pre-configured based on actual business needs during the system deployment phase and can be automatically fine-tuned through error feedback during the closed-loop self-correction process.

[0097] Example 6

[0098] like Figure 2 As shown, based on Embodiment 1, the step of determining the current load state type based on the forward scaling decision, and distributing scaling tasks to the corresponding resource pools according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit, includes the following when the horizontal scaling is a horizontal expansion:

[0099] S5.1: Based on the comparison result between the target number of replicas determined by the forward scaling decision and the preset upper limit of the number of replicas in the steady-state resource pool, determine the current load state type, which includes steady-state type and overflow type.

[0100] The scaling controller reads the target replica count determined by the forward scaling decision and compares it with the preset replica count limit of the steady-state resource pool. The steady-state resource pool consists of reserved nodes that carry the baseline inference load, and its replica count limit is preset based on the capacity planning of the reserved nodes. When the target replica count does not exceed the replica count limit of the steady-state resource pool, the current load state type is determined to be steady-state; when the target replica count exceeds the replica count limit of the steady-state resource pool, the current load state type is determined to be overflow. Since the comparison result between the target replica count and the steady-state capacity limit is used as the basis for judgment, the baseline load and the sudden peak load are clearly distinguished, so that low-cost reserved resources and instant elastic resources are matched respectively. The replica count limit of the steady-state resource pool is determined by the number of reserved nodes, which is a constant that can be pre-configured during the deployment phase. Therefore, the comparison in this step is an O(1) constant time operation, which hardly increases the decision overhead. The target replica count is used directly instead of the real-time queue depth because the target replica count has integrated the optimization results of the predicted total number of global queue messages, the adaptive decision window, and the model predictive control, directly reflecting the future required capacity, and the judgment result is more robust. The binary mutual exclusion decision clearly divides the load into a baseline area and an overflow area: the baseline area is handled by low-cost reserved nodes, while the overflow area is handled by dedicated overflow topics and transient resource pools. The two are isolated from each other at the consumer group level and are processed in parallel without interference.

[0101] S5.2: When the current load state type is a steady state, the scaling task is sent to the steady state resource pool composed of reserved nodes to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0102] When the current load state is steady state, the scaling controller distributes scaling tasks to the steady state resource pool composed of reserved nodes. Inference container units are created on these reserved nodes to perform horizontal scaling. The steady state resource pool uses reserved nodes to carry the baseline inference load, featuring high resource utilization and low unit computation cost, making it suitable for handling continuous and stable processing tasks. After horizontal scaling is complete, the change in the number of inference container unit replicas is calculated as the difference between the number of replicas after scaling and the number before scaling. Since all scaling under steady state occurs on reserved nodes, and these nodes have already completed image warm-up and resource reservation, their cold start latency is lower than that of on-demand elastic instances. This allows the baseline load to be quickly absorbed at the lowest unit cost while maintaining the stability of the resource pool. Horizontal scaling of the steady state resource pool is achieved by the orchestration system creating inference container unit replicas on its reserved nodes. Because the reserved nodes have already completed image warm-up and resource reservation, the cold start latency of new replicas from creation to readiness is significantly lower than that of on-demand elastic instances, allowing for rapid absorption of baseline scaling needs. The change in the number of replicas after expansion is calculated as the difference between the number of replicas before and after expansion. This change is written back to the scaling controller for state tracking and serves as input to the collaborative scaling process of the collaborative processing nodes. Since steady-state loads are preferentially expanded on low-cost reserved nodes and only transferred to the transient resource pool when the limit is exceeded, the baseline load is carried at the lowest unit computational cost, while preserving the stability and predictability of the reserved resource pool.

[0103] S5.3: When the current load state type is overflow type, the scaling task corresponding to the portion exceeding the upper limit of the number of replicas in the steady-state resource pool is sent to the transient resource pool composed of Serverless instances to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

[0104] When the current load status is overflow, the scaling controller first distributes the portion not exceeding the upper limit of the stable resource pool replica count to the stable resource pool. Then, it distributes the scaling tasks corresponding to the portion exceeding the stable resource pool replica count upper limit to the transient resource pool, which consists of Serverless instances. Inference container units are created on the transient resource pool to perform horizontal scaling. The transient resource pool provides elastic computing power instantly using Serverless instances: Serverless instances are instantly launched by the serverless computing platform on request and billed based on actual calls, without the need for pre-reserving dedicated nodes. This allows for rapid scaling down to zero after peak traffic subsides, absorbing sudden surges exceeding the baseline capacity. After horizontal scaling is complete, the change in the number of inference container unit replicas is calculated using the number of replicas added jointly by the stable and transient resource pools. Because the transient resource pool immediately absorbs the surges, peak traffic is fully absorbed without the need for long-term high resource reservations. The horizontal scaling of the transient resource pool is achieved by serverless computing platforms (such as Knative or Function Compute) instantly launching Serverless instances on demand, without the need for pre-reserving dedicated nodes. Therefore, instances can automatically shrink to zero when peak loads subside and there are no messages to consume, achieving on-demand disposal of computing power. This step distributes the portion exceeding the steady-state resource pool's limit to the transient resource pool, allowing sudden peak loads to be immediately absorbed by elastic computing power, while the baseline portion remains stably supported by reserved nodes. The change in the number of replicas of the inference container unit is the sum of the newly added replicas in the steady-state and transient pools, serving as the joint input for collaborative scaling.

[0105] Example 7

[0106] Based on Embodiments 1 and 6, when the current load state type is an overflow type, it further includes:

[0107] S5.3.1: Deliver messages in the original topic of the message queue that exceed the carrying capacity of the steady-state resource pool to a dedicated overflow topic in the message queue.

[0108] When the current load status is overflow, the overflow forwarding unit in the steady-state resource pool monitors the original topic of the message queue (i.e., the original topic carrying normal pipeline messages, distinguished from newly created dedicated overflow topics). Messages exceeding the steady-state resource pool's capacity in the original topic are delivered to a dedicated overflow topic in the message queue. Overflow forwarding is based on the real-time consumption capacity of the steady-state resource pool: messages not exceeding the capacity are retained by the original topic for consumption by the steady-state resource pool; messages exceeding the capacity are forwarded to the overflow topic. The dedicated overflow topic and the original topic are independent of each other in the message queue, and their partition numbers and retention policies can be configured separately based on peak throughput. Because overflow traffic is diverted from the original topic to the dedicated overflow topic, the steady-state resource pool's consumption always proceeds smoothly within its rated capacity without being impacted by peak traffic, and it also avoids consumption point rollback triggered by excessive backlog in the original topic. The overflow forwarding unit is deployed on the steady-state resource pool side. It monitors the consumption progress of the original topic in a bypass manner and performs forwarding based on the real-time consumption capacity of the steady-state resource pool: when the message arrival rate of the original topic does not exceed the steady-state consumption capacity, the message remains in the original topic and is processed by the steady-state consumption group; when it exceeds the limit, the excess part is delivered to a newly created dedicated overflow topic.

[0109] S5.3.2: The Serverless instances in the transient resource pool obtain and process messages from the overflow topic through a dedicated consumer group, thereby completing the horizontal scaling of the transient resource pool. The dedicated consumer group is isolated from the consumer group in the steady-state resource pool that consumes the original topic.

[0110] Serverless instances in the transient resource pool subscribe to and consume overflow topics in dedicated consumer groups, retrieve messages from the overflow topics, and perform inference processing, thereby completing the horizontal scaling of the transient resource pool. These dedicated consumer groups are isolated from the consumer groups in the steady-state resource pool that consume the original topics, each with its own independent consumption points and rebalancing domains. Since the dedicated consumer groups do not need to participate in the rebalancing of the original topic consumer groups, the impact of paused consumption during rebalancing on consumption continuity is avoided, ensuring that the absorption of peak traffic does not interfere with baseline consumption. When the queue depth falls back to a safe level, overflow forwarding stops, no new messages enter the overflow topics, and the Serverless instances in the transient resource pool automatically shrink to zero due to the lack of messages to consume, thus achieving the "use and discard" of peak computing power. Because the dedicated consumer groups have independent consumption points and rebalancing domains in the message queue, completely isolated from the consumer groups in the steady-state resource pool that consume the original topics, the rebalancing that occurs when the transient resource pool consumes overflow topics only takes place within the dedicated consumer group and does not pause the consumption of the original topics, thus ensuring the continuity of baseline consumption. Serverless instances pull messages from overflow topics and perform inference, with their consumption throughput scaling linearly with the number of instances. When the queue depth falls back to a safe level, overflow forwarding stops, the overflow topic no longer receives new messages, and the Serverless instances automatically shrink to zero due to the lack of messages to consume, thus releasing peak computing power.

[0111] It's important to note that the core value of the consumer group isolation mechanism lies in avoiding the disruption to consumption continuity caused by Kafka / Pulsar consumer group rebalancing. In traditional Kafka consumption models, when consumer group members change (e.g., adding or removing consumer instances), the Group Coordinator triggers a full group rebalancing. All members pause consumption (i.e., a Stop-The-World behavior) until the rebalancing is complete, with pause durations ranging from hundreds of milliseconds to several seconds. In high-throughput pipelines, this can cause considerable backlog growth. This invention isolates the Serverless instances of the transient resource pool into an independent rebalancing domain by establishing dedicated overflow topics and dedicated consumer groups: when the transient resource pool elastically scales up or down, rebalancing only occurs within the dedicated consumer group, without affecting the continuous consumption of the original topic consumer group; similarly, when the inference container units of the stable resource pool scale up or down, rebalancing is also triggered only within the original topic consumer group, without interfering with the consumption progress of the overflow topic. The rebalancing domains of the two consumer groups are independent of each other, which ensures that the pipeline processing during the two-layer elastic expansion process remains continuous and avoids double backlog caused by rebalancing pauses and cold start delays during peak periods.

[0112] Example 8

[0113] In one embodiment, when the horizontal scaling is changed to horizontal shrinking, it includes:

[0114] A1: Prioritize scaling down the Serverless instances in the transient resource pool until the number of instances in the transient resource pool is reduced to zero, and then scale down the inference container units in the steady-state resource pool.

[0115] When horizontal scaling is converted to horizontal scaling down, the scaling controller executes scaling down according to resource pool priority: first, it scales down Serverless instances in the transient resource pool, gradually releasing elastic computing power used to absorb peak loads; only after the number of instances in the transient resource pool is scaled down to zero does it scale down inference container units in the stable resource pool. Since the transient resource pool is billed per call and has high marginal costs, prioritizing its release directly reduces operating costs. The stable resource pool, however, is supported by reserved nodes and needs to maintain baseline capacity to ensure continuous processing capability; therefore, a scaling down order of transient first, then stable is adopted. The scaling down process is also constrained by the maximum step size for a single scaling down operation to avoid drastic fluctuations in processing capacity caused by a sudden drop in the number of replicas. By following the scaling down order of transient first, then stable, the peak load decline phase can quickly recover high-cost elastic resources while maintaining the stable operation of the baseline load. The scaling-down order is determined based on a dual consideration of resource cost and baseline protection: Serverless instances in the transient resource pool are billed based on actual usage and have higher marginal costs; releasing them first directly reduces operational overhead. The stable resource pool, supported by reserved nodes, needs to retain baseline capacity to maintain continuous processing power, and is therefore placed later. Scaling down is also constrained by the maximum step size for a single scaling-down operation to avoid drastic fluctuations in processing power or even task reordering caused by a sudden drop in the number of replicas. For example, before scaling down, the transient resource pool contains 6 Serverless instances, and the stable resource pool contains 16 inference container units. During scaling down, the 6 Serverless instances are gradually released to zero first, and then the inference container units in the stable resource pool are scaled down according to the load.

[0116] Example 9

[0117] Based on Example 1, after performing horizontal scaling on the inference container unit based on the forward scaling decision, the method further includes:

[0118] S5.1': Collect the actual total number of global queue messages after scaling is executed, calculate the difference between the actual total number of global queue messages and the predicted total number of global queue messages, and obtain the scaling decision error.

[0119] After horizontal scaling of the inference container unit based on forward scaling decisions, the cloud server collects the actual total number of global queue messages after the scaling execution and subtracts it from the predicted total number of global queue messages for the corresponding decision period to obtain the scaling decision error:

[0120]

[0121] Among them, e k Let the scaling decision error be the value for the k-th decision period. The total number of messages in the global queue after the scaling operation in the k-th decision cycle. Let be the total predicted global queue message volume for the k-th decision period. The scaling decision error characterizes the deviation between the prediction and the actual value; a larger absolute value indicates a more severe deviation between the current prediction model or cold start delay estimate and the real environment. This scaling decision error serves as a criterion for triggering policy self-correction. The scaling decision error reflects both the extrapolated prediction residual and the implicit cold start delay estimation bias; therefore, its absolute value simultaneously carries mismatch information from both the prediction model and the adaptive decision window, making it a core observation for closed-loop self-correction. By subtracting the actual total global queue message volume from the predicted value rather than comparing it to the historical baseline, the error directly corresponds to the decision quality of the current decision period, facilitating the identification of whether continuous threshold exceedances indicate a deviation from the prediction trend or a failure of the cold start window.

[0122] S5.2': When the absolute value of the scaling decision error exceeds a preset error threshold within a preset number of consecutive decision periods, adjust the decay factor of the online exponentially weighted moving average estimator and correct the parameters of the prediction model of the model prediction controller.

[0123] The cloud server continuously monitors the scaling decision error for each decision cycle. When the absolute value of the scaling decision error exceeds a preset error threshold for a consecutive preset number of decision cycles, it determines that the current strategy is mismatched with the environment and triggers self-correction. Self-correction adjusts the decay factor of the online exponentially weighted moving average estimator to increase its weighting to respond to new observations delayed by cold start, allowing the adaptive decision window to follow environmental changes more quickly. It also fine-tunes the prediction model parameters of the model predictive controller, gradually approximating the extrapolation trend to the actual queue depth evolution. Since correction is only triggered when the error exceeds the threshold for multiple consecutive decision cycles, over-adjustment caused by occasional jitter is avoided. The determination condition for a consecutive preset number of decision cycles (e.g., three consecutive cycles) requires that the error exceed the threshold for multiple adjacent cycles before correction is triggered. This distinguishes single-point measurement jitter or brief load spikes from real-world drift, avoiding strategy oscillations caused by frequent retraining. The self-calibration process increases the weight of the adaptive decision window on new observations delayed by cold start, resulting in faster convergence. The fine-tuning of the prediction model parameters makes the slope of the extrapolation trend approximate the actual queue evolution. Together, they pull the prediction error back to a steady-state level of about 3.4%, thus forming a complete closed loop of prediction-execution-error-calibration.

[0124] Specifically, the fine-tuning of the prediction model parameters is performed using gradient descent: scaling the decision error e. kTake the partial derivatives of the prediction model parameters (taking linear regression as an example, i.e., the trend slope a and the intercept term b), and perform a one-step gradient correction on the parameters according to the following update rule:

[0125]

[0126] in, Let η be the loss function defined by the square of the scaling decision error, and let η be the gradient correction step size (ranging from 0.001 to 0.01 to ensure the stability of parameter updates). Let a be the first partial derivative of the loss function with respect to the trend slope a. Let be the first-order partial derivative of the loss function with respect to the intercept term b. Each time self-correction is triggered, a gradient update is performed, rather than batch retraining on historical data, thus gradually approaching the optimal parameters while maintaining low computational overhead. When cluster node specifications change or network latency experiences persistent drift, the aforementioned gradient descent fine-tuning allows the prediction model to automatically adapt to new load evolution patterns over several decision cycles, maintaining the accuracy of scalability decisions.

[0127] Example 10

[0128] like Figure 3 As shown, based on Embodiment 1, the step of determining the target number of replicas of the co-processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the co-processing node, in order to achieve coordinated scaling of the co-processing node and the inference container unit, includes:

[0129] S6.1: The ratio of the throughput is determined by the steady-state throughput of a single instance of the inference container unit and the steady-state throughput of a single instance of the collaborative processing node, wherein the steady-state throughput of a single instance is the throughput value measured when the corresponding node is running independently and the resource utilization rate is within a preset stable range.

[0130] The cloud server determines the ratio of the steady-state throughput of a single instance of the inference container unit to the steady-state throughput of a single instance of the collaborative processing node, i.e., the collaborative scaling factor, which is determined by the ratio of steady-state to transient throughput. The steady-state throughput of a single instance is the throughput value measured when the corresponding node is running independently and its resource utilization is within a preset stable range: within this range, the node is neither overloaded nor idle, and the measured throughput represents the node's stable processing capacity. To obtain reliable steady-state throughput, measurements are taken during the stable operation phase after warm-up, and representative values ​​are obtained from multiple measurements to eliminate instantaneous fluctuations, avoiding deviations in the collaborative scaling factor caused by using instantaneous peak throughput or cold start throughput as a benchmark. Because the throughput in the steady-state range is used as the benchmark, the collaborative scaling factor is stable and reliable, unaffected by instantaneous load fluctuations. The steady-state throughput of a single instance is measured during the stable operation phase after the node has warmed up, excluding abnormal readings during cold starts and resource contention phases. Representative values ​​(such as the median) are taken from multiple measurements to eliminate instantaneous fluctuations. The stable range of resource utilization is defined by the node's processor and memory usage falling within a preset bandwidth (such as 40% to 70%), and only the throughput measured within this range is used as the benchmark. Using the steady-state throughput as the ratio benchmark ensures that the collaborative scaling factor is stable and reliable, and does not drift with instantaneous load fluctuations, thereby guaranteeing the accuracy of collaborative scaling of collaborative processing nodes.

[0131] It should be understood that the collaborative processing node (i.e., the data quality assurance node) undertakes a comprehensive quality verification function in the data lake entry and labeling pipeline, internally comprising a quality verification module, a consistency detection module, and an anomaly labeling review module. The quality verification module is responsible for verifying each labeling result output by the inference container unit according to preset quality rules; the consistency detection module cross-compares different labeling results for the same data item in a multi-agent labeling scenario, identifying and marking discrepancies; the anomaly labeling review module captures and manually reviews labeling results that exceed the normal distribution range. These three modules process each labeling output sequentially; therefore, the steady-state throughput T of a single instance of the collaborative processing node is... c The throughput of the module with the lowest processing speed among the three modules is determined. In actual deployment, the throughput of each of the three modules can be benchmarked separately, and the minimum value is taken as T. c The benchmark is used to ensure that the calculation of the collaborative scaling factor γ is conservative and reliable, and to avoid any link in the quality verification link becoming a bottleneck during peak periods.

[0132] S6.2: Calculate the product of the change in the number of replicas of the inference container unit and the ratio of the throughput, and round the product up to obtain the change in the number of replicas of the collaborative processing node.

[0133] The cloud server multiplies the change in the number of replicas of the inference container unit by the ratio of its throughput, and then rounds the product up to obtain the change in the number of replicas of the collaborative processing node. This change is calculated as the difference between the number of replicas after expansion and before expansion. Rounding up ensures that the processing capacity of the collaborative processing node is not less than the required verification throughput after the inference container unit expansion, avoiding insufficient verification capacity due to rounding down, which could become a bottleneck. The change in the number of replicas of the collaborative processing node is consistent with the scaling direction of the inference container unit: when the inference container unit expands, it expands synchronously according to this ratio; when it shrinks, it shrinks synchronously according to this ratio. Because the collaborative processing node always matches the throughput requirements of the inference container unit with rounding up, the processing capacity of the quality verification stage is always not less than the inference output. Rounding up ensures that the processing capacity of the collaborative processing node is not less than the required verification throughput after the inference container unit expansion: if rounding down were used, the number of replicas in the verification stage would be too low when the scaling factor is not an integer, becoming a pipeline bottleneck during peak periods. The change in the number of replicas is strictly consistent with the scaling direction of the inference container unit. Inference expansion is synchronized proportionally, and inference shrinkage is synchronized proportionally, ensuring that the two stages are always in sync. This ratio also applies to the coordinated scaling of transient overflow components, ensuring that the verification capability of overflow consumption keeps pace. For example, when the inference container unit is expanded by 8 units with a scaling factor of 1.25, the product of 10 is rounded up to 10, ensuring that the verification nodes are sufficient to match the expanded inference output.

[0134] S6.3: Based on the current number of replicas of the collaborative processing node and the change in the number of replicas of the collaborative processing node, determine the target number of replicas of the collaborative processing node to achieve collaborative scaling between the collaborative processing node and the inference container unit.

[0135] The cloud server adds the current replica count of the collaborative processing nodes to the change in the replica count of the collaborative processing nodes to obtain the target replica count of the collaborative processing nodes, and then issues scaling tasks for the collaborative processing nodes accordingly. The collaborative processing nodes also employ a steady-state-transient two-tier deployment: the baseline number of collaborative processing nodes runs in the steady-state resource pool, while the overflow portion runs in the transient resource pool, keeping pace with the overflow consumption of the inference container units, thus elastically scaling along with the inference container units during peak periods. Since the target replica count of the collaborative processing nodes is determined jointly based on the throughput ratio and the change in the replica count of the inference container units, the processing capabilities of the inference and verification stages are always matched during scaling, preventing quality verification from becoming a pipeline bottleneck during peak periods. The collaborative processing nodes also employ a steady-state-transient two-tier deployment: the baseline number of collaborative processing nodes runs in the steady-state resource pool, while the overflow portion is deployed in the transient resource pool along with the transient expansion of the inference container units, keeping pace with the overflow consumption on the inference side, allowing verification capabilities to elastically scale along with inference during peak periods. The target number of replicas is obtained by adding the current number of replicas to the change in the number of replicas, and the collaborative scaling task is issued to the orchestration system accordingly. Since the target number of replicas for collaborative processing nodes is determined by both the throughput ratio and the change in the number of replicas for inference container units, the processing capabilities of the inference and quality verification stages are always matched throughout the scaling process, avoiding the verification stage becoming a bottleneck during peak periods and ensuring the continuous output of high-quality datasets.

[0136] Example 11

[0137] like Figure 4 As shown, the present invention discloses a cloud-native elastic scaling system based on queue depth, comprising:

[0138] The queue depth aggregation module 401 is used to collect real-time queue depth data of each topic partition in the message queue. It aggregates the real-time queue depth data by topic dimension and then globally summarizes it to obtain the total global queue message count. The queue depth aggregation module 401 periodically reads the number of messages to be consumed in each topic partition using indicator collection probes deployed on each proxy node of the message queue. It sums the data by topic dimension at the edge computing node and further globally summarizes it, outputting a scalable and invariant total global queue message count to the queue depth prediction module. This protects subsequent predictions and decisions from signal distortion caused by load redistribution during scaling.

[0139] The queue depth prediction module 402 is used to input the time-series historical data of the total global queue messages into the extrapolation prediction model for prediction, and output the predicted total global queue messages. The queue depth prediction module 402 maintains the time-series storage of the total global queue messages on a cloud server, extracts samples within a preset historical time window, resamples and denoises them, and inputs them into the extrapolation prediction model. It extrapolates the total global queue messages within a future preset prediction time window using linear regression or exponential smoothing, and outputs the predicted total global queue messages to the scaling decision-making module; thus enabling the scaling decision-making module to have a forward-looking capability for future load trends.

[0140] The decision window estimation module 403 is used to track the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator to obtain an adaptive decision window. In each decision cycle, the decision window estimation module 403 records the measured cold start latency from issuing the expansion command to the inference container unit becoming ready. It then uses the online exponentially weighted moving average estimator to exponentially weight and fuse the latest observation with the previous cycle's estimate, outputting an adaptive decision window to the scaling decision solver module. This allows the prediction window length to automatically adapt to cold start latency fluctuations in different cluster environments and time periods.

[0141] The scaling decision-solving module 404 is used to input the predicted total number of global queue messages and the adaptive decision window into the model prediction controller for solving, thereby obtaining a forward-looking scaling decision. On the cloud server, the scaling decision-solving module 404 uses the predicted total number of global queue messages as the input signal and the adaptive decision window as the prediction window. Under constraints of the maximum replica count upper limit, the minimum replica count lower limit, and the single scaling step size, it performs rolling optimization to solve for the optimal replica count sequence. It then takes the replica count of the current decision period and outputs the forward-looking scaling decision to the two-layer elastic scaling module, ensuring that scaling is completed before the cold start delay and suppressing replica count oscillations.

[0142] The dual-layer elastic scaling module 405 is used to determine the current load state type based on the forward-looking scaling decision, and to distribute scaling tasks to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit. The dual-layer elastic scaling module 405 compares the target number of replicas with the upper limit of the number of replicas in the steady-state resource pool to distinguish between steady-state and overflow types. Under the steady-state type, the scaling tasks are distributed to the steady-state resource pool composed of reserved nodes; under the overflow type, the excess portion is distributed to the transient resource pool composed of Serverless instances, and the peak traffic is isolated by using overflow topics and dedicated consumer groups. The change in the number of replicas of the inference container unit is then output to the collaborative scaling decision module, thus unifying the low-cost bearing of baseline load and the immediate absorption of peak traffic.

[0143] The collaborative scaling decision module 406 is used to determine the target number of replicas for the collaborative processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the collaborative processing node, so as to achieve collaborative scaling between the collaborative processing node and the inference container unit. The collaborative scaling decision module 406 uses the ratio of the steady-state throughput of a single instance of the inference container unit to that of the collaborative processing node as a collaborative scaling ratio factor on the cloud server. It multiplies this ratio by the change in the number of replicas of the inference container unit and rounds up to obtain the change in the number of replicas of the collaborative processing node. This is then added to the current number of replicas to determine the target number of replicas, and the collaborative scaling task is issued. This ensures that the processing capabilities of the inference and quality verification stages are always matched during the scaling process.

[0144] The aforementioned six modules are collaboratively supported by metric collection probes deployed on the message queue agent node side, edge computing nodes, and cloud servers, forming a complete closed loop from deep queue collection, aggregation, prediction, decision-making to two-layer execution and collaborative scaling. In summary, this invention replaces single-instance metrics with the total global queue message volume, which possesses scalability invariance. It achieves forward-looking scaling by combining adaptive cold-start awareness model predictive control, and through a steady-state-transient two-layer elastic architecture and collaborative scaling with processing nodes, it ensures uninterrupted data pipeline operation under traffic peaks, balancing baseline low cost with peak absorption capacity.

[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions should be covered within the scope of protection of the present invention.

Claims

1. A cloud-native elastic scaling method based on queue depth, characterized in that, include: Collect real-time queue depth data for each topic partition in the message queue, aggregate the real-time queue depth data by topic dimension, and then summarize it globally to obtain the total number of messages in the global queue. The time-series historical data of the total global queue message volume is input into the extrapolation prediction model for prediction, and the predicted total global queue message volume is output. An adaptive decision window is obtained by tracking the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator. The predicted total number of global queue messages and the adaptive decision window are input into the model prediction controller for solution to obtain a forward-looking scalable decision. Based on the forward-looking scaling decision, the current load state type is determined, and the scaling task is distributed to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit. Based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the co-processing node, the target number of replicas of the co-processing node is determined to achieve coordinated scaling between the co-processing node and the inference container unit.

2. The method according to claim 1, characterized in that, The real-time queue depth data of each topic partition in the collected message queue is aggregated by topic dimension and then globally summarized to obtain the total number of global queue messages, including: The number of messages to be consumed in each topic partition of the message queue is collected to obtain the real-time queue depth data; The real-time queue depth data is aggregated by topic dimension to obtain the total queue depth for each topic; The total queue depth of each topic is globally summarized to obtain the total global queue message volume. The total global queue message volume represents the total number of messages that have been enqueued but not yet consumed at the moment the scaling operation occurs. The scaling operation changes the consumption rate but does not change the total number of messages that have been enqueued but not yet consumed.

3. The method according to claim 1, characterized in that, The step of inputting the time-series historical data of the total global queue message volume into the extrapolation prediction model for prediction, and outputting the predicted total global queue message volume, includes: Obtain the time-series historical data of the total number of messages in the global queue within a preset historical time window; The time series historical data is input into the extrapolation prediction model for extrapolation prediction, and the predicted total number of global queue messages is output, wherein the predicted total number of global queue messages is the predicted value of the total number of global queue messages within a future preset prediction time window.

4. The method according to claim 1, characterized in that, The process of tracking the cold start latency of the inference container unit in the current environment using an online exponentially weighted moving average estimator to obtain an adaptive decision window includes: Within each decision cycle, the time interval between issuing the expansion command and the readiness of the inference container unit created by the expansion is recorded to obtain the measured cold start latency. Based on the decay factor of the preset initial value, the measured cold start delay is used as the latest observation value. The latest observation value and the adaptive decision window of the previous decision period are subjected to exponential weighted fusion to obtain the adaptive decision window.

5. The method according to claim 1, characterized in that, The step of inputting the predicted total number of global queue messages and the adaptive decision window into the model prediction controller for solving to obtain a forward-looking scalable decision includes: Within each decision cycle, the model prediction controller uses the total number of predicted global queue messages as the input signal and the adaptive decision window as the prediction window to solve for the optimal replica number sequence in a rolling optimization manner under the preset replica number constraint. The replica number in the current decision cycle in the optimal replica number sequence is used as the forward scaling decision. The replica number constraint includes: the maximum replica number upper limit, the minimum replica number lower limit, the maximum step size for a single expansion, and the maximum step size for a single reduction.

6. The method according to claim 1, characterized in that, The process involves determining the current load state type based on the forward-looking scaling decision, and then distributing scaling tasks to the corresponding resource pools to perform horizontal scaling on the inference container units according to the current load state type, thereby obtaining the change in the number of replicas of the inference container units. When the horizontal scaling is a horizontal expansion, the process includes: Based on the comparison between the target number of replicas determined by the forward scaling decision and the preset upper limit of the number of replicas in the steady-state resource pool, the current load state type, which includes steady-state type and overflow type, is determined. When the current load state type is a steady state, the scaling task is sent to the steady state resource pool composed of reserved nodes to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit; When the current load state type is overflow type, the scaling task corresponding to the portion exceeding the upper limit of the number of replicas in the steady-state resource pool is sent to the transient resource pool composed of Serverless instances to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit.

7. The method according to claim 6, characterized in that, When the current load state type is overflow type, it also includes: Messages in the original topic of the message queue that exceed the carrying capacity of the steady-state resource pool are delivered to a dedicated overflow topic in the message queue; The Serverless instances in the transient resource pool obtain and process messages from the overflow topic through dedicated consumer groups, thereby completing the horizontal scaling of the transient resource pool. The dedicated consumer groups are isolated from the consumer groups in the steady-state resource pool that consume the original topic.

8. The method according to claim 1, characterized in that, When the horizontal expansion is converted to horizontal contraction, it includes: First, scale down the Serverless instances in the transient resource pool until the number of instances in the transient resource pool is reduced to zero, and then scale down the inference container units in the steady-state resource pool.

9. The method according to claim 1, characterized in that, After performing horizontal scaling on the inference container unit based on the forward scaling decision, the method further includes: Collect the actual total number of global queue messages after scaling is executed, calculate the difference between the actual total number of global queue messages and the predicted total number of global queue messages, and obtain the scaling decision error; When the absolute value of the scaling decision error exceeds a preset error threshold within a preset number of consecutive decision periods, the decay factor of the online exponentially weighted moving average estimator is adjusted, and the parameters of the prediction model of the model prediction controller are corrected.

10. The method according to claim 1, characterized in that, The step of determining the target number of replicas for the collaborative processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the collaborative processing node, to achieve collaborative scaling between the collaborative processing node and the inference container unit, includes: The ratio of the throughput is determined by the steady-state throughput of a single instance of the inference container unit and the steady-state throughput of a single instance of the collaborative processing node, wherein the steady-state throughput of a single instance is the throughput value measured when the corresponding node is running independently and the resource utilization rate is within a preset stable range. The product of the change in the number of replicas of the inference container unit and the ratio of the throughput is calculated, and the product is rounded up to obtain the change in the number of replicas of the collaborative processing node. Based on the current number of replicas of the collaborative processing node and the change in the number of replicas of the collaborative processing node, the target number of replicas of the collaborative processing node is determined to enable the collaborative scaling of the collaborative processing node and the inference container unit.

11. A cloud-native elastic scaling system based on queue depth, characterized in that, include: The queue depth aggregation module is used to collect real-time queue depth data of each topic partition in the message queue, aggregate the real-time queue depth data by topic dimension and then summarize it globally to obtain the total number of global queue messages. The queue depth prediction module is used to input the time series historical data of the total global queue message volume into the extrapolation prediction model for prediction, and output the predicted total global queue message volume. The decision window estimation module is used to track the cold start latency of the inference container unit in the current environment through an online exponentially weighted moving average estimator to obtain an adaptive decision window; The scaling decision-solving module is used to input the total predicted global queue message volume and the adaptive decision window into the model prediction controller for solving, so as to obtain a forward-looking scaling decision. A two-layer elastic scaling module is used to determine the current load state type based on the forward scaling decision, and to distribute scaling tasks to the corresponding resource pool according to the current load state type to perform horizontal scaling on the inference container unit, thereby obtaining the change in the number of replicas of the inference container unit. The collaborative scaling decision module is used to determine the target number of replicas of the collaborative processing node based on the change in the number of replicas of the inference container unit and the ratio of the throughput of the inference container unit to the throughput of the collaborative processing node, so as to realize the collaborative scaling of the collaborative processing node and the inference container unit.

Images