System for carrying out self-adaptive current limiting on token number and QPS called by large model

Abstract

The invention relates to the technical field of large model service management, and discloses a system for carrying out self-adaptive current limiting on a token number and a QPS called by a large model. A dynamic monitoring module of the system is responsible for collecting a request flow data set comprising a token consumption sequence, a request frequency sequence and a response delay sequence; a traffic feature extraction module performs multi-dimensional feature analysis on the data set to generate a traffic feature matrix containing token consumption rate, request frequency fluctuation coefficient and delay sensitivity index; the adaptive current-limiting decision module dynamically matches a load balancing strategy based on the matrix, and generates a current-limiting control parameter set containing a token quota threshold and a QPS upper limit threshold; the real-time regulation and control module dynamically adjusts the request queue according to the request queue to generate a new request scheduling sequence; and the feedback optimization module monitors the execution effect and iteratively updates the load balancing strategy. According to the invention, through real-time perception and dynamic adjustment, refined and adaptive management of large model service resources is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of large model service management, in particular to a system for adaptively limiting the number of tokens and QPS of large model calling. BACKGROUND

[0002] With the deepening of commercial application of large-scale pre-training language models in natural language processing, intelligent content generation, human-computer interaction dialogue and other fields, when they are opened to the outside world as service interfaces, they face unprecedented challenges of high concurrency access and resource management. The service capacity of large models is not unlimited, and the computing overhead is positively correlated with the number of tokens processed by each inference request, and the request processing capacity of a single service instance in a unit of time is strictly restricted by the underlying hardware resources. In the actual production environment, the user request mode has high uncertainty, suddenness and heterogeneity. For example, some application scenarios may produce a large number of short text generation requests in an instant, causing a sharp increase in QPS; while some other scenarios may submit a single but extremely long document processing task, consuming a large number of tokens. This dynamically changing and unpredictable traffic mode is extremely easy to make the backend large model service instance exceed its carrying limit, causing service overload.

[0003] Service overload will bring a series of serious consequences. The most direct manifestation is the significant increase in request response delay or even timeout, and the user experience deteriorates sharply. Further, the emergency resource expansion to cope with peak load not only lags behind, but also produces high cloud computing costs. More seriously, if there is no effective resource isolation and flow limiting mechanism, the abnormal flow of a few users may "knock down" the entire service instance, produce a "noise neighbor" effect, deprive other normal users of the right to use the service fairly, and ultimately affect the overall availability and stability of the service.

[0004] The industry mostly uses the traditional static threshold method for the flow limiting technology of API services. For example, a fixed request number per second or a daily call quota is set for each API key or user. This method is simple to implement, but its limitations are very prominent. First, the static threshold cannot perceive the real-time load state of the service instance. When the instance load is light, the fixed low threshold will cause valuable computing resources to be idle, reducing resource utilization; when the load is already heavy, the static threshold may not be able to effectively prevent the further deterioration of the service instance, because the system cannot dynamically tighten the limit according to the real-time pressure. Second, the traditional flow limiting method usually only focuses on the request frequency, ignoring the resource consumption difference of individual requests. A simple question and answer request that only needs to process tens of tokens, and a long document summary request that needs to process tens of thousands of tokens, have completely different computing pressures on the backend model. Using the number of requests as a measurement standard is obviously unfair and cannot accurately prevent resource depletion caused by a small number of high-consumption requests. In addition, the static strategy lacks adaptive adjustment ability after configuration, and the threshold needs to be manually adjusted by the operation and maintenance personnel according to experience, which is not only laborious, but also difficult to keep up with the rapid changes of business traffic.

[0005] The existing static flow limiting scheme cannot meet the demand of large model services for fine and intelligent resource management. The industry urgently needs an adaptive flow limiting system that can deeply integrate real-time monitoring, multi-dimensional feature analysis and dynamic decision-making. The system should be able to continuously track the actual resource consumption of each request and the overall pressure indicators of the service instance, and dynamically calculate and implement the most appropriate flow limiting strategy based on these multi-dimensional data. The ideal adaptive flow limiting system should maximize resource utilization while ensuring the overall stability of the service, and automatically adapt to fluctuations in business traffic, thereby providing stable and reliable underlying support for large-scale and high-concurrency model services. The present invention is a solution to this technical pain point. SUMMARY

[0006] The purpose of the present invention is to provide a system for adaptive flow limiting of large model call token number and QPS to solve the problems raised in the background art.

[0007] To achieve the above purpose, the present invention provides a system for adaptive flow limiting of large model call token number and QPS, which comprises:

[0008] A dynamic monitoring module that collects a request traffic data set of the target large model service instance in real time, the request traffic data set containing a token consumption sequence of a single call, a request frequency sequence per unit time and a response delay sequence;

[0009] The flow feature extraction module performs multi-dimensional feature analysis processing on the request flow data set to generate a flow feature matrix, the flow feature matrix including a token consumption rate, a request frequency fluctuation coefficient, and a delay sensitivity index;

[0010] The adaptive flow limiting decision module performs dynamic matching processing based on the flow feature matrix and a preset load balancing strategy to generate a flow limiting control parameter set, the flow limiting control parameter set including a token quota threshold and a QPS upper threshold;

[0011] The real-time regulation module performs dynamic adjustment processing on the request queue of the target large model service instance according to the flow limiting control parameter set to generate an adjusted request scheduling sequence;

[0012] The feedback optimization module continuously monitors actual execution effect data of the adjusted request scheduling sequence and iteratively updates the load balancing strategy based on the actual execution effect data.

[0013] Preferably, the dynamic monitoring module collects the request flow data set of the target large model service instance in real time, including:

[0014] The input token quantity and the output token quantity of each request are collected through a buried probe to generate a token consumption sequence;

[0015] The number of requests arriving within a preset time window is counted to generate a request frequency sequence;

[0016] The time difference from initiation to completion of each request is recorded to generate a response delay sequence;

[0017] The token consumption sequence, the request frequency sequence, and the response delay sequence are aligned by timestamp and combined into the request flow data set.

[0018] Preferably, the flow feature extraction module performs multi-dimensional feature analysis processing on the request flow data set to generate a flow feature matrix, including:

[0019] The change rate of the total token consumption amount per unit time is calculated to generate a token consumption rate;

[0020] The standard deviation and mean ratio operation is performed on the request frequency sequence to generate a request frequency fluctuation coefficient;

[0021] The delay sensitivity index is generated according to the correlation coefficient of the response delay sequence and the token consumption sequence;

[0022] The token consumption rate, the request frequency fluctuation coefficient, and the delay sensitivity index are combined into a three-dimensional vector form of the flow feature matrix.

[0023] Preferably, the adaptive throttling decision module dynamically matches the traffic feature matrix with a preset load balancing strategy to generate a throttling control parameter set, including:

[0024] inputs the traffic feature matrix into a pre-trained throttling strategy selection model to output a current optimal throttling mode identifier;

[0025] queries a strategy database according to the optimal throttling mode identifier to obtain corresponding reference token quota and reference QPS upper limit;

[0026] dynamically scales the reference token quota based on a delay sensitivity index in the traffic feature matrix to generate a token quota threshold;

[0027] dynamically adjusts the reference QPS upper limit based on a request frequency fluctuation coefficient in the traffic feature matrix to generate a QPS upper limit threshold.

[0028] Preferably, the real-time regulation module dynamically adjusts a request queue of a target large model service instance according to the throttling control parameter set to generate an adjusted request scheduling sequence, including:

[0029] performs priority sorting processing on the current request queue to be processed to generate an initial scheduling sequence;

[0030] performs token total amount verification processing on the initial scheduling sequence according to the token quota threshold to eliminate requests exceeding the quota threshold;

[0031] performs time window allocation processing on the remaining requests based on the QPS upper limit threshold to generate the adjusted request scheduling sequence that meets the QPS constraint.

[0032] Preferably, the priority sorting processing on the current request queue to be processed to generate an initial scheduling sequence includes:

[0033] extracts historical response delay data and token consumption data of each request;

[0034] calculates a priority score of each request, the priority score being inversely proportional to historical response delay and proportional to token consumption;

[0035] reorders the request queue according to the priority score from high to low to generate the initial scheduling sequence.

[0036] Preferably, the feedback optimization module continuously monitors actual execution effect data of the adjusted request scheduling sequence, and iteratively updates the load balancing strategy based on the actual execution effect data, including:

[0037] collecting actual token consumption rate, actual QPS and actual delay data in the execution of the adjusted request scheduling sequence;

[0038] calculating deviation index of actual execution effect data and expected effect;

[0039] when the deviation index exceeds a preset threshold, triggering an online update process of the load balancing strategy;

[0040] synchronizing the updated load balancing strategy to a strategy database of the adaptive throttling decision module.

[0041] Preferably, the triggering of the online update process of the load balancing strategy comprises:

[0042] analyzing main influencing factors of the deviation index to identify strategy parameters that need to be adjusted;

[0043] generating a set of strategy parameter adjustment suggestions based on historical optimization records;

[0044] verifying actual effects of different adjustment suggestions through A / B testing;

[0045] selecting an adjustment suggestion with the optimal effect to update the load balancing strategy.

[0046] Preferably, the system further comprises:

[0047] an abnormal fuse module that forcibly enables a backup throttling strategy when it is detected that QPS of consecutive multiple time windows exceeds a safety threshold;

[0048] the backup throttling strategy adopts a combination control mode of fixed token quota and fixed upper limit of QPS.

[0049] Preferably, after the abnormal fuse module forcibly enables the backup throttling strategy, the system further comprises:

[0050] continuously monitoring a recovery state of system load indicators;

[0051] when the load indicators fall back to a normal range, gradually relaxing throttling intensity;

[0052] finally switching back to the adaptive throttling control parameter set generated by the adaptive throttling decision module.

[0053] Compared with the prior art, the present application has the following beneficial effects:

[0054] The core value brought by the present application lies in its dynamic adaptability. The system can surpass the static threshold limit, perceive the load pressure and flow characteristics of the large model service instance in real time, and make accurate regulation decisions. By continuously collecting multi-dimensional data such as token consumption, request frequency and response delay, the system can comprehensively depict the running state of the service instance, thereby identifying potential overload risks. Based on in-depth analysis of the flow characteristics, the system can intelligently set flow limiting parameters that match the current actual load. These parameters are no longer fixed values, but dynamically float with the changes in flow patterns. This dynamic nature makes resource allocation more reasonable, protecting service stability during peak periods by appropriate restrictions, and relaxing restrictions during flow valleys to improve resource utilization, avoiding resource idling.

[0055] Another outstanding advantage of the system lies in its fine-grained management capability. Traditional flow limiting methods often treat different types of requests equally, while the present application can implement more fine-grained control strategies by distinguishing token consumption rate and request frequency fluctuations. For example, the system can identify and limit complex requests that consume a large amount of computing resources, thereby prioritizing faster response to more simple requests when overall traffic is high, maintaining the experience fairness of the majority of users. This evaluation criterion based on resource consumption rather than the number of simple requests makes the flow limiting strategy more scientific and fair.

[0056] The present application introduces a closed-loop feedback optimization mechanism, enabling the system to have the ability to continuously evolve. The actual effect after each flow limiting strategy adjustment is continuously monitored and evaluated, and these effect data are used to optimize the internal decision model. This means that the system can learn from historical regulation experience and continuously correct its parameter matching logic, making subsequent flow limiting decisions more accurate and effective. This self-optimizing feature allows the system to gradually adapt to changing business patterns and user behavior, maintaining efficient and stable service state in the long term.

[0057] The comprehensive application of the system makes the large model service exhibit stronger robustness and higher resource utilization efficiency when facing complex and variable invocation scenarios. Service providers can support larger business throughput while guaranteeing service level agreements; and end users can obtain more stable and predictable service responses, effectively maintaining user experience. The efficiency and economy of the entire service system are thus improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A timing diagram for the system for adaptive flow limiting of large model invocation token number and QPS described in the present application;

[0059] Figure 2 A flowchart for the dynamic monitoring module request flow data collection;

[0060] Figure 3 Flow chart for generating a set of flow control parameters for the adaptive flow limiting decision module;

[0061] Figure 4 Analysis chart for processing and scheduling a request queue for the real-time regulation module. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0063] Please refer to Figure 1 The present application provides a system for adaptive flow limiting of token number and QPS of a large model, which comprises a dynamic monitoring module, a traffic feature extraction module, an adaptive flow limiting decision module, a real-time regulation module and a feedback optimization module. The specific implementation is as follows:

[0064] The dynamic monitoring module is responsible for continuously collecting a request traffic data set of a target large model service instance, which contains a token consumption sequence of a single call, a request frequency sequence in a unit of time and a response delay sequence. The traffic feature extraction module performs multi-dimensional feature analysis on the collected data to generate a traffic feature matrix containing token consumption rate, request frequency fluctuation coefficient and delay sensitivity index. The adaptive flow limiting decision module matches the traffic feature matrix with a preset load balancing strategy to output a set of flow limiting control parameters, including token quota threshold and QPS upper threshold. The real-time regulation module dynamically adjusts the request queue according to the flow limiting control parameters to generate a request scheduling sequence that meets the constraints. The feedback optimization module monitors the actual execution effect and iteratively updates the load balancing strategy based on the deviation to form an adaptive optimization cycle. The modules are connected through a data bus to realize real-time data exchange and collaborative work, ensuring that the system can automatically adjust the flow limiting strategy according to the actual load conditions.

[0065] Embodiment 1: Please refer to Figure 2The design goal of the data awareness and feature abstraction stage is to accurately capture the runtime state of service instances and transform it into quantifiable decision-making criteria. The dynamic monitoring module, as the system's data source, relies on a highly collaborative data acquisition mechanism. This module deploys lightweight instrumentation probes at the request entry point of the target large-scale model service instance. These probes are embedded in the service runtime environment as library files, performing non-intrusive listening and tracking of each model call request. When a new request arrives, the probe immediately intercepts the complete content packet of the request and starts an associated tracking context. For processing the request content, the probe calls a tokenizer matching the service model to parse the input text in real time to accurately count the number of tokens it contains. Simultaneously, for the response stream returned by the model, the probe asynchronously collects the complete output text and also counts the tokens, thus accurately recording the total token consumption of a single call and forming a token consumption sequence arranged in chronological order.

[0066] While collecting request content, this module also maintains a high-precision time series manager to capture the distribution characteristics of traffic over time. The manager uses a configurable fixed duration as a basic time window, continuously counting the total number of requests arriving at the service instance within that window. This counting process takes into account the out-of-order request issues that may be caused by network latency, correcting for these issues by using the request's creation timestamp, thus generating a request frequency sequence that accurately reflects the client's request pressure. For comprehensive service performance evaluation, response latency measurement is also crucial. The probe records a high-precision timestamp when a request begins processing by the service instance and another timestamp after the response is fully sent back to the client. The difference between the two timestamps is the response latency for that request. All request latency values ​​are arranged sequentially to form a response latency sequence. The collection of these three sequences—token consumption sequence, request frequency sequence, and response latency sequence—is synchronized, but they originate from the same request stream, therefore strict alignment is necessary to ensure data consistency over time. The dynamic monitoring module internally employs a time synchronization service that assigns a unified time tag to all collected events based on Coordinated Universal Time (UTC). A data aligner uses the request ID as the key to associate the number of tokens, frequency contribution (i.e., which time window it belongs to), and latency data of the same request, and integrates them into a structured request traffic data set according to the time label. This set is temporarily stored in a circular buffer with an overflow protection mechanism for downstream modules to consume.

[0067] The traffic feature extraction module pulls the latest request traffic data set from the buffer and performs a series of feature analysis and dimension reduction operations, aiming to extract a small number of indicators from the original data that best represent the essential characteristics of the current traffic. The module performs time domain analysis on the token consumption sequence. It uses a sliding time window to observe the dynamic changes in the total amount of token consumption, calculates the total consumption in the window and observes its growth or decline trend relative to the previous window, and generates the token consumption rate by calculating the change amplitude per unit time. This rate value not only reflects the current computing load intensity, but also implies the future short-term resource demand trend. For the request frequency sequence, the module focuses on its stability rather than the absolute value. It uses descriptive statistics to calculate the standard deviation and arithmetic mean of the sequence over a long observation period, and then divides the standard deviation by the mean to obtain the request frequency fluctuation coefficient. This coefficient is a dimensionless value that effectively eliminates the influence of the absolute order of magnitude, purely depicting the volatility or burstiness of the request arrival process. The higher the coefficient value, the more unstable the traffic, and the more it needs to be smoothed. Specifically, the module continuously collects data points of the request frequency sequence over a long observation period, and these data points represent the number of requests arriving per unit time. The module uses descriptive statistics to calculate the arithmetic mean of the sequence, which is the sum of the values of all data points divided by the total number of data points, to reflect the central tendency of the sequence. The module calculates the standard deviation of the sequence: first, calculate the deviation of each data point from the arithmetic mean, square the deviation, sum it up, divide by the total number of data points to get the variance, and finally take the square root of the variance to get the standard deviation, which quantifies the dispersion of the sequence. The module divides the standard deviation by the arithmetic mean to obtain the request frequency fluctuation coefficient. This coefficient is a dimensionless index that effectively eliminates the influence of the absolute order of magnitude, purely depicting the volatility or burstiness of the request arrival process. The higher the coefficient value, the more unstable the traffic, providing key feature input for subsequent adaptive throttling decisions. The entire process is closely related to the multi-dimensional feature analysis logic of the traffic feature extraction module in the document, ensuring alignment with the sequence data collected by the dynamic monitoring module.

[0068] The generation of the delay sensitivity indicator is a multivariate correlation analysis process. The module takes the response delay sequence and the corresponding token consumption sequence as two sets of observation values, and calculates the Pearson correlation coefficient between them. This calculation process evaluates the ratio of the covariance of the two sequences to the product of their respective standard deviations, resulting in a value between -1 and 1. The absolute value reflects the sensitivity of the model response delay to token consumption. A high absolute value indicates that the delay is largely influenced by the token amount, while a low absolute value indicates that the delay may be dominated by other factors. The three calculated and normalized indicators - token consumption rate, request frequency fluctuation coefficient, and delay sensitivity indicator - are combined into a fixed-format three-dimensional vector, i.e., the traffic feature matrix. This matrix structure is very compact, discarding redundant raw data and retaining only the most core feature information, greatly reducing the computational burden of the subsequent decision module, while providing clear and quantitative input for the adaptive decision-making process.

[0069] Embodiment 2: see Figure 3 The adaptive throttling decision module is the intelligent center of the entire system, and its operation is based on the mechanism of continuous learning and dynamic matching. The module maintains a pre-trained throttling strategy selection model inside. This model is not static, and its training data comes from the vast amount of historical working condition data accumulated by the system during long-term operation. These data record the actual control effects of various throttling strategies under different traffic feature matrices. The model itself adopts a lightweight neural network structure that guarantees inference speed while having sufficient non-linear mapping ability to understand the complex interaction between the three dimensions of the traffic feature matrix. When the traffic feature extraction module generates a new three-dimensional feature vector and sends it to the decision module, the vector is standardized for preprocessing to eliminate dimensional differences, and then input into the strategy selection model for forward propagation calculation. The model output layer usually uses the Softmax function to produce a probability distribution for all pre-set throttling modes, and the mode with the highest probability is determined as the current optimal throttling mode identifier. This identifier can be understood as a qualitative judgment of the system's current load status, such as "low load - high elasticity", "high load - strong constraint", or "burst traffic - conservative defense", etc.

[0070] After obtaining the optimal throttling mode identifier, the decision-making process enters the strategy parameter mapping phase. The system internally constructs a strategy database, which is organized in the form of key-value pairs. The key is a limited, pre-defined throttling mode identifier, and the value is a set of benchmark control parameters that match each mode, mainly including the benchmark token quota and the benchmark QPS upper limit. These benchmark parameters are not arbitrarily set. They are initial values determined by the system designer based on the hardware resource configuration of the large model service instance, the service level agreement (SLA) target, and historical experience, constituting the benchmark line of system control. The decision module uses the obtained optimal throttling mode identifier as the index key to quickly retrieve the corresponding benchmark token quota and benchmark QPS upper limit values from the strategy database through a query instruction.

[0071] Directly applying benchmark parameters lacks precision, so the dynamic adjustment link is crucial. The core idea of this link is to fine-tune the benchmark parameters using specific indicator values in the traffic feature matrix, enabling control parameters to more accurately respond to the instantaneous state of the system. For the determination of the token quota threshold, the delay sensitivity index plays a key role. This index quantifies the sensitivity of the system delay to the computational load (token consumption). If the delay sensitivity index is high, it means that the response delay of the current system is easily deteriorated due to the increase in token consumption, presenting a fragile state. To avoid this situation, the decision logic introduces a scaling factor that is negatively related to the delay sensitivity index. The benchmark token quota is multiplied by this factor less than 1, resulting in a tighter and safer token quota threshold, actively limiting the total computational amount allowed to be processed within a single time window, and reserving buffer space for system stability. The specific functional form of the scaling factor needs to be carefully designed to make it change smoothly and not cause dramatic oscillation of control parameters.

[0072] For the setting of the QPS upper limit threshold, the request frequency fluctuation coefficient is the main adjustment basis. This coefficient characterizes the stationarity of the request arrival process. A higher fluctuation coefficient means that there may be sudden or intermittent peaks in traffic, causing greater instantaneous impact on the system. To smooth this impact and prevent excessive requests from accumulating in a short period of time, causing queue overflow and delay soaring, the decision logic will refer to this fluctuation coefficient to adjust the benchmark QPS upper limit downward. The adjustment mechanism also uses a scaling function based on the fluctuation coefficient to multiply the benchmark QPS upper limit by an appropriate reduction factor, so that the actual QPS upper limit threshold generated can effectively suppress traffic peaks and control the request rate within a more stable range that the system can easily handle. This adjustment is essentially a preventive measure, sacrificing a small amount of instantaneous throughput to stabilize overall service quality.

[0073] After the above two steps of dynamic matching and parameter adjustment, the adaptive flow limiting decision module finally outputs a set of specific and contextual flow limiting control parameters, which explicitly includes the token quota threshold and QPS upper limit threshold that should be executed at the current time. This set of parameters is then packaged into a standard format control message and sent in real time to the real-time regulation module through a high-priority internal communication channel. The entire decision-making process from feature vector input to control parameter output needs to be completed in a very short time to meet the stringent requirements of online systems for real-time response, and its efficiency directly determines the intelligence level and final effectiveness of the entire adaptive flow limiting system. The module also has a decision log recording function, and each decision input, output and strategy identification will be recorded completely.

[0074] Embodiment 3: How does the real-time regulation module convert the control parameters generated by the adaptive flow limiting decision module into executable request scheduling logic? This module acts as the system action execution layer and directly acts on the request queue to be processed. Its processing flow starts with intelligent sorting of the queue and gradually imposing resource constraints. The real-time regulation module continuously monitors the flow limiting control parameter set from the upstream decision module, and once it receives a new token quota threshold and QPS upper limit threshold, it immediately triggers a dynamic adjustment cycle for the current request queue, which includes three core processing stages: priority sorting, token total amount verification, and time window allocation.

[0075] The priority sorting process is the initial step of the adjustment process, and its purpose is to determine the order of processing according to the potential resource demand and historical service performance of each request. The module extracts historical data associated with each request from the persistent storage, which is indexed by request source identifiers such as user ID or application ID. Historical data mainly includes the average response delay record and average token consumption record of the request source in the past period of time. After extraction, the module calculates a priority score for each request in the queue, which quantifies the relative importance and urgency of the request. To accurately calculate the score, we use the following mathematical relationship:

[0076]

[0077] Where: represents the priority score of request , the larger the value, the higher the priority; represents the estimated token consumption of request , which can be estimated based on the length of the request content or the historical average value, represents the arithmetic mean of the estimated token consumption of all requests in the current request queue, represents the estimated token consumption of request the historical average response latency of the request source, which is queried from the historical record, the arithmetic mean of the historical average response latencies corresponding to all requests in the current queue. This relationship embodies the basic principle that the priority score is positively correlated with token consumption and negatively correlated with historical latency, i.e., the request source whose current request obtains a higher priority is the one that consumes more computing resources and has a better historical service experience, which helps to optimize overall resource utilization efficiency and user experience in the context of traffic limiting. After calculating the scores of all requests, the module rearranges the request queue in order of score from high to low to generate an ordered initial scheduling sequence.

[0078] The module enters the token total check processing stage, and the goal of this stage is to ensure that the initial scheduling sequence does not exceed the token quota threshold set by the decision module in terms of overall token consumption. The checking process starts from the head of the initial scheduling sequence and scans sequentially, and maintains a running cumulative token consumption variable. For each request in the sequence, the module adds its estimated token consumption to the current cumulative value, and judges whether the cumulative value has exceeded the token quota threshold. If the cumulative value has not exceeded the threshold, the request is retained in the sequence; once the cumulative value reaches or exceeds the threshold, the request being processed and all subsequent requests in the sequence are immediately marked as exceeding the quota and are removed from the current scheduling period. These removed requests receive a system busy prompt and can be arranged for retry or discard according to the strategy. This sequential accumulation and truncation method ensures that the total token consumption of the final retained request set is necessarily within the quota allowed range, thereby effectively preventing the computing load of the model instance in a single time window from being overloaded.

[0079] After the total token amount check is completed, the module needs to allocate the remaining request set to a time window to ensure that the sending rate of the request meets the upper limit threshold of QPS. The module uses a smoothing mechanism based on the leaky bucket algorithm to handle this problem. The algorithm imagines the sending of requests as water flowing out of a leaky bucket, and the outlet rate of the bucket is strictly controlled by the upper limit threshold of QPS. The module divides the unit time into several more fine-grained time slices, and the number of requests that each slice can accommodate is determined by dividing the upper limit threshold of QPS by the number of slices. The module allocates the remaining requests to each time slice in order according to their order in the initial scheduling sequence (i.e. priority order). The allocation process strives to be uniform and avoids accumulating too many requests in any one slice. If the number of requests is large, the module will calculate the ideal sending time point of each request, and control the actual delivery time of the request through a high-precision timer, so that the request stream arrives at the large model service instance at a smooth rate. This process ensures that the number of requests processed per unit time does not exceed the upper limit of QPS in actual execution, thereby avoiding system congestion and delay surge caused by request flood.

[0080] Through the continuous processing of the above three stages, the real-time control module finally outputs an adjusted request scheduling sequence. This sequence not only meets the global constraints of the total token amount, but also meets the rate constraints of QPS, and also prioritizes high-priority requests when possible. The sequence is delivered to the underlying request scheduler for execution, and the scheduler sends requests to the target service instance in an asynchronous and non-blocking manner. The entire real-time control process is periodic and synchronized with the update period of the decision module, ensuring that the flow limiting strategy can respond to changes in system state in a timely manner. The module also has logic to handle abnormal situations, such as when there are ultra-high priority requests in the queue, a special fast lane mechanism may be enabled, but all of this strictly follows the established resource constraint rules to ensure system stability and fairness.

[0081] Referring to Figure 4This chart illustrates the core mechanism of request scheduling in an adaptive rate limiting system. The chart uses a bi-sub-chart structure: the upper part uses a bar chart to compare the changes in the number of requests before and after processing, while the lower part uses a time series graph to show the real-time relationship between request scheduling and QPS limits. In the upper part's comparison of request queue processing, the chart clearly presents the processing status of four different types of requests. High-priority and medium-priority requests are fully processed before and after the adjustment, demonstrating the system's guarantee mechanism for important requests and intuitively reflecting the strict verification performed by the system based on token quota thresholds, effectively preventing resource overload. The lower part's scheduling sequence time distribution graph shows the system's fine-grained traffic control at the time window level. Within the continuous time windows T1 to T4, the actual number of scheduled requests consistently remains below the QPS limit, demonstrating the system's strict adherence to request rate constraints. This time window allocation ensures the stability of the request flow and avoids the impact of sudden traffic surges on backend services. This chart professionally presents the complete processing flow of the real-time control module, from priority sorting and token total verification to time window allocation, providing system operation and maintenance personnel with intuitive monitoring data and fully demonstrating the technical advantages of the adaptive rate limiting system in ensuring service stability and resource utilization.

[0082] Example 4: The core of the system's self-optimization and evolution capability lies in establishing a closed loop that feeds back the control strategy from the execution effect. The feedback optimization module, like a neural reflex arc of the system, continuously observes the performance of the request scheduling sequence processed by the real-time control module in a real environment, compares the observed results with the expected goals, and thus drives the iterative update of the underlying control strategy. The operation of this module begins with the meticulous collection of actual execution effect data, which is the direct basis for evaluating the effectiveness of the current rate limiting strategy. The module collects the real performance indicators of the scheduled request sequences by reusing the instrumented probes of the dynamic monitoring module. The actual token consumption rate is calculated by statistically analyzing the total number of tokens consumed by all successfully processed requests within a complete evaluation period, and then dividing by the period duration. This value reflects the actual computational load of the model instance under the rate limiting strategy. The actual QPS is obtained by counting the number of requests that successfully return a response within the same evaluation period. Actual latency data is usually taken as the median or a specific quantile of the response latency of all requests within that period to eliminate the influence of extreme values ​​and more accurately represent the experience of most users.

[0083] After the actual data is collected, the module starts the deviation analysis process. It compares the actual execution effect data with the expected effect target set by the adaptive flow limiting decision module when generating the current flow limiting control parameters. The expected effect target is stored in the strategy database and is associated with each flow limiting mode identifier. For example, for the "smooth mode", the expected effect may be set to an average delay of less than 400 milliseconds and a QPS fluctuation range within ±5% of the target value. The deviation index is calculated using a comprehensive evaluation method, and the relative deviation rate is calculated for each key performance indicator, that is, the absolute value of (actual value - expected value) divided by the expected value. A comprehensive deviation index is obtained by weighted average of these relative deviation rates, and the weight configuration reflects the importance of the system to different indicators, for example, the weight of the delay indicator is usually the highest because it is directly related to the user experience.

[0084] When the calculated comprehensive deviation index exceeds the preset activation threshold (for example, 15%), the feedback optimization module determines that the current strategy effect is not expected, and then triggers the online update process of the load balancing strategy. The first step of the update process is root cause analysis, which aims to identify the main influencing factors that cause the deviation. The module will deeply review the historical records of the traffic feature matrix, the set values of the flow limiting control parameters, and the time series trend of the actual execution data. For example, through correlation analysis, it may be found that the actual delay surge has weak correlation with the token consumption rate, but has high correlation with the centralized arrival of a certain type of user request, which indicates that the current strategy may have deficiencies in the processing of the request frequency fluctuation coefficient, and the dynamic adjustment logic of the QPS upper limit needs to be adjusted instead of the token quota part.

[0085] Based on the direction identified by root cause analysis, the module enters the strategy parameter adjustment suggestion generation stage. It queries a special historical optimization record knowledge base, which is indexed by timestamp and scene characteristics, records the context of previous strategy adjustment, adjustment content and its resulting effect changes. The system uses these historical data to generate a set of targeted strategy parameter adjustment suggestions through pattern matching and simple recommendation algorithms. Specifically, after the deviation threshold triggers online update, the feedback optimization module first gives the parameter direction that needs to be adjusted and the affected strategy site according to the root cause analysis, then accesses the historical optimization record knowledge base indexed by timestamp and scene characteristics, which saves the context of previous adjustment, specific changes and the resulting effect changes. The system first does pattern matching: using the current optimal throttling mode identifier, the relative relationship of each indicator in the three-dimensional flow feature matrix and the main influencing factors of this deviation, retrieves historical entries that are consistent with the current mode and have the same influencing factors and produce good effect changes under similar feature conditions to get the candidate set, and then uses the simple recommendation algorithm of "rule-based scoring and sorting": for each candidate entry, accumulate priority points item by item according to three rules - rule one is consistency priority of throttling mode, rule two is adjustment direction consistent with root cause conclusion (for example, current adjustment pointing to QPS upper limit is preferred to records of adjusting QPS related parameters), and rule three is that the effect changes in historical records are more positive, and finally a few entries with high ranking are generated as "strategy parameter adjustment suggestions", for example, if there are records in the knowledge base that increasing the QPS upper limit to the volatility coefficient response strength in the high volatility scenario and producing better effect changes, then the suggestion of "adjusting the QPS volatility sensitivity coefficient" is preferred, these suggestions enter the A / B verification link to compare the performance of key indicators under the current target, and the better one is written back to the strategy library and notified to the decision module to refresh, at the same time, the context and effect of this selection are deposited back to the knowledge base for subsequent matching. Referring to Table 1, for the QPS volatility adaptability problem found above, the knowledge base may retrieve historical records that have achieved good results by increasing the sensitivity of QPS upper limit to volatility coefficient (i.e. amplifying its scaling factor) in similar high volatility scenarios, and thus generate a suggestion of "adjusting the QPS volatility sensitivity coefficient".

[0086] Table 1: A / B test effect comparison of strategy parameter adjustment suggestions

[0087] Test group identification Policy adjustment suggestion description Average delay during test (ms) QPS compliance rate during test (%) Comprehensive deviation index during test (%) Group A (baseline) Keep the existing policy parameters unchanged 450 85.5 18.2 Group B Increase the sensitivity of the upper limit of QPS to the fluctuation coefficient, and the scaling factor adjustment is 1.2 times the original value 380 92.1 8.5 Group C Reduce the weight of the delay sensitivity of the token quota, and the weight coefficient adjustment is 0.8 times the original value 520 88.3 25.1

[0088] The system verifies the actual effect of different adjustment suggestions through an A / B test framework. It randomly and uniformly divides the current incoming request traffic into several test groups according to dimensions such as source IP or user ID, for example, group A, group B, and group C as set in Table 1. Each group applies different strategy adjustment suggestions during the test, and group A maintains the existing strategy unchanged as the baseline group. During the test, the module continuously monitors and records the core performance indicators of each group, such as the average delay, QPS compliance rate, and the finally calculated comprehensive deviation index as shown in Table 1. After the test, the module compares and analyzes the data of each group, and selects the adjustment suggestion corresponding to the group with the lowest comprehensive deviation index and the key indicators most consistent with the expected target as the optimal solution. In this example, group B performs best in all indicators, and the adjustment suggestion corresponding to it is determined as the optimal solution.

[0089] The module performs a strategy update operation. It converts the optimal adjustment suggestion verified by group B, i.e., "increase the sensitivity of QPS upper limit to fluctuation coefficient, and scale factor adjustment to 1.2 times the original value", into specific policy parameter modification instructions. This update operation is written into the load balancing policy library in an atomic transaction manner, updating the corresponding throttling mode policy parameters. After the update is completed, the feedback optimization module sends a notification to the adaptive throttling decision module that the policy has been refreshed, and the decision module will call the updated policy parameters to generate a new set of throttling control parameters in the subsequent decision period, thus completing a complete closed loop from effect evaluation to policy optimization. The entire process ensures that the system's throttling strategy can evolve continuously with changes in service load characteristics and external environment, and has a certain self-optimization ability. The module also stores the complete context of this optimization, including the trigger reason, test data, and final selection, as a new record in the historical optimization record knowledge base, providing more abundant reference for future optimization decisions.

[0090] Embodiment 5: The system's self-protection and recovery mechanism in extreme abnormal situations aims to quickly and stably control the system through a more direct and strict forced means when the adaptive throttling strategy cannot effectively control the situation, preventing the service from collapsing completely. The abnormal fuse module acts as a safety fuse of the system, and its activation does not depend on complex feature analysis and policy matching, but is based on a series of simple and hard threshold judgments. The module continuously monitors the request traffic data set published by the dynamic monitoring module, especially the request frequency sequence. It calculates the real-time QPS value within a fixed time window. The module has a pre-set safety threshold, which is usually determined by multiplying the absolute maximum QPS that the large model service instance can withstand under stress testing by a safety factor. It is a clear red line value.

[0091] When the module detects that the real-time QPS value calculated by the current time window exceeds the safety threshold for the first time, it does not immediately act, but starts a counting process to prevent false triggering caused by a short-term traffic pulse. The module continues to monitor the subsequent consecutive time windows, and if the QPS of three consecutive windows is detected to exceed the safety threshold, the module determines that the system is in an abnormal state of continuous overload, and the adaptive throttling strategy may have failed or adjusted at a speed that cannot keep up with the deterioration. At the moment of determination, the abnormal fuse module broadcasts a highest-priority fuse event signal to the entire system, and forces the current adaptive throttling mode to switch to the backup throttling strategy.

[0092] The design concept of the backup throttling strategy is simplicity and conservatism, which completely bypasses the complex calculation process of the adaptive throttling decision module. This strategy uses a set of pre-configured, fixed control parameters, mainly including fixed token quota and fixed QPS upper limit. These fixed values are usually set near the lower limit of the range in which the system can stably run, for example, the fixed token quota may be set to 50% of the lowest quota in the normal adaptive mode, and the fixed QPS upper limit is set to 60% of the safety threshold. This strong contraction control method aims to quickly reduce the load of the system at the expense of most of the throughput, in exchange for the reduction of response delay and the recovery of service availability. Once enabled, the real-time regulation module will strictly follow this set of fixed, non-dynamic parameters to screen and schedule the request queue, and all requests exceeding the fixed quota and upper limit will be immediately rejected.

[0093] After the backup throttling strategy is forcibly enabled, the focus of the abnormal fuse module shifts from abnormal detection to recovery state monitoring. It is not satisfied with maintaining this strict conservative mode all the time, but continues to track a series of key system load indicators to evaluate the effect of forced throttling and the recovery of the system. The monitored indicators include the central processor usage rate of the service instance, the memory occupancy rate, and the median of the response delay of all requests. The module sets a "normal interval" for these indicators, for example, the central processor usage rate is less than 70%, the memory occupancy rate is less than 80%, and the delay median is less than 1000 milliseconds. The module checks whether these indicators have all fallen within the normal interval at a one-minute cycle.

[0094] When the system load indicators are first monitored to enter the normal interval, the fuse module starts a recovery confirmation timer, which needs to observe the load indicators to be stable in the normal interval for a period of time to confirm that the recovery is not a temporary fluctuation but a trend of improvement. After the confirmation period ends, the module starts to execute the process of gradually relaxing the strength of flow control. This is a cautious and gradual trial operation. For example, the module will increase the fixed QPS upper limit value by 10%, continue to observe the change of system load indicators in the next monitoring period. If the indicators remain in the normal interval, the module will again increase the upper limit value by 10%; if any load indicator rebounds after relaxation and exceeds the normal interval again, the module will immediately undo the last relaxation operation, restore the flow control strength to the previous stable level, and wait for a longer time before trying to relax. This "two steps forward and one step backward" trial strategy ensures the smoothness and safety of the system recovery process, avoiding the avalanche caused by premature and rapid removal of restrictions.

[0095] When the flow control strength gradually recovers to the parameter level close to that generated by the original adaptive flow control strategy through multiple gradual relaxations, and the system load indicators remain stable under more relaxed conditions, the abnormal fuse module determines that the system has completely left the abnormal state. At this time, the module will send a switching instruction to inform the system modules to remove the fuse state and return the flow control to the adaptive flow control decision module. The real-time control module will no longer use the fixed backup strategy parameters, but will start to receive and execute the flow control parameter set generated by the adaptive flow control decision module according to the real-time flow characteristics. The whole switching process is realized through the semaphore mechanism, ensuring that there is no vacuum or conflict of control rules at the switching moment, so as to complete the seamless transition from abnormal fuse to normal adaptive flow control. The implementation mechanism of the whole embodiment provides the last solid defense line for the system when facing sudden huge flow impact, and enhances the resilience and survivability of the system.

[0096] It should be noted that in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0097] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A system for adaptive throttling of token number and QPS of large model invocation, characterized in that, The method comprises the following steps: a dynamic monitoring module, which collects a request traffic data set of a target large model service instance in real time, wherein the request traffic data set comprises a token consumption sequence of a single call, a request frequency sequence within a unit time, and a response delay sequence; a traffic feature extraction module, which performs multi-dimensional feature analysis processing on the request traffic data set to generate a traffic feature matrix, wherein the traffic feature matrix comprises a token consumption rate, a request frequency fluctuation coefficient, and a delay sensitivity index; an adaptive flow limiting decision module, which performs dynamic matching processing based on the traffic feature matrix and a preset load balancing strategy to generate a flow limiting control parameter set, wherein the flow limiting control parameter set comprises a token quota threshold and a QPS upper limit threshold; a real-time regulation module, which performs dynamic adjustment processing on a request queue of the target large model service instance according to the flow limiting control parameter set to generate an adjusted request scheduling sequence; a feedback optimization module, which continuously monitors actual execution effect data of the adjusted request scheduling sequence and iteratively updates the load balancing strategy based on the actual execution effect data.

2. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, wherein, The dynamic monitoring module collects the request traffic data set of the target large model service instance in real time, comprising: collecting the number of input tokens and the number of output tokens of each request through a buried probe to generate a token consumption sequence; counting the number of requests arriving within a preset time window to generate a request frequency sequence; recording the time difference from the initiation to the completion of each request to generate a response delay sequence; aligning and merging the token consumption sequence, the request frequency sequence, and the response delay sequence according to timestamps to obtain the request traffic data set.

3. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, characterized in that, The traffic feature extraction module performs multi-dimensional feature analysis processing on the request traffic data set to generate a traffic feature matrix, comprising: calculating the change rate of the total amount of token consumption within a unit time to generate a token consumption rate; performing standard deviation and mean ratio operations on the request frequency sequence to generate a request frequency fluctuation coefficient; generating a delay sensitivity index according to the correlation coefficient of the response delay sequence and the token consumption sequence; combining the token consumption rate, the request frequency fluctuation coefficient, and the delay sensitivity index into a three-dimensional vector form of the traffic feature matrix.

4. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, wherein, The adaptive flow limiting decision module performs dynamic matching processing based on the traffic feature matrix and a preset load balancing strategy to generate a flow limiting control parameter set, comprising: inputting the traffic feature matrix into a pre-trained flow limiting strategy selection model to output a current optimal flow limiting mode identifier; querying a strategy database according to the optimal flow limiting mode identifier to obtain a corresponding reference token quota and a reference QPS upper limit; performing dynamic scaling processing on the reference token quota based on the delay sensitivity index in the traffic feature matrix to generate a token quota threshold; performing dynamic adjustment processing on the reference QPS upper limit based on the request frequency fluctuation coefficient in the traffic feature matrix to generate a QPS upper limit threshold.

5. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, characterized in that, The real-time regulation module dynamically adjusts a request queue of a target large model service instance according to the set of flow control parameters, and generates an adjusted request scheduling sequence, including: performing priority sorting processing on the current request queue to be processed to generate an initial scheduling sequence; performing token total amount checking processing on the initial scheduling sequence according to the token quota threshold, and eliminating requests exceeding the quota threshold; based on the QPS upper threshold, performing time window allocation processing on the remaining requests to generate the adjusted request scheduling sequence that meets the QPS constraint.

6. The system for adaptive throttling of token number and QPS for large model invocation according to claim 5, characterized in that, The priority sorting processing on the current request queue to be processed to generate an initial scheduling sequence includes: extracting historical response delay data and token consumption data of each request; calculating the priority score of each request, which is inversely proportional to the historical response delay and proportional to the token consumption; reordering the request queue according to the priority score from high to low to generate the initial scheduling sequence.

7. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, characterized in that, The feedback optimization module continuously monitors the actual execution effect data of the adjusted request scheduling sequence, and iteratively updates the load balancing strategy based on the actual execution effect data, including: collecting actual token consumption rate, actual QPS and actual delay data in the execution process of the adjusted request scheduling sequence; calculating the deviation index of the actual execution effect data and the expected effect; when the deviation index exceeds a preset threshold, triggering an online update process of the load balancing strategy; synchronizing the updated load balancing strategy to the strategy database of the adaptive flow limiting decision module.

8. The system for adaptive throttling of token number and QPS for large model invocation according to claim 7, wherein, The online update process of the load balancing strategy includes: analyze the main influencing factors of the deviation index and identify the strategy parameters that need to be adjusted; generate a set of strategy parameter adjustment suggestions based on historical optimization records; verify the actual effect of different adjustment suggestions through A / B testing; select the optimal adjustment suggestion and update the load balancing strategy.

9. The system for adaptive flow control of token number and QPS for large model invocation according to claim 1, wherein, The system further includes: an abnormal fuse module that forcibly enables a backup flow limiting strategy when it detects that the QPS of multiple consecutive time windows exceeds a safety threshold; the backup flow limiting strategy adopts a combined control method of fixed token quota and fixed QPS upper limit.

10. The system for adaptive throttling of token number and QPS for large model invocation according to claim 9, wherein, After the abnormal fuse module forcibly enables the backup flow limiting strategy, it further includes: continuously monitor the recovery state of the system load index; when the load index falls back to the normal range, gradually relax the flow limiting intensity; finally switch back to the adaptive flow limiting control parameter set generated by the adaptive flow limiting decision module.

Images