Multi-tenant maas platform token quota control and overflow flow limiting processing method

Abstract

The application discloses a multi-tenant MaaS platform token quota control and over-limit flow limiting processing method, comprising the following steps: S1, assigning a data object identifier to each tenant's quota strategy; S2, an API gateway node receives a model invocation request initiated by a tenant; a control data set is formed by combination; S3, reference information of the data object corresponding to the Handle identifier is obtained; S4, the object version number obtained by analysis is compared with the local cached strategy version number; S5, the number of modified consumed tokens of this request is calculated, the gateway node queries the current token number of the local dynamic token bucket, and the modified consumed token number obtained by calculation is compared with the number; S6, a remote quota segment application process is started; S7, the quota segment application request is double-checked in the remote storage node. Through dynamic calculation of the number of modified consumed tokens, the application can distinguish between reasonable burst traffic and malicious traffic occupation, and realizes real-time adaptive flow limiting of the tenant usage mode.

Description

Technical Field

[0001] This invention relates to the field of network traffic technology, specifically to a method for token quota management and over-limit rate limiting in a multi-tenant MaaS platform. Background Technology

[0002] With the rapid development and widespread application of large-scale model technology, MaaS (Model as a Service) platforms, as a cloud computing service model that encapsulates AI models as services and provides them to multiple tenants, have become an important way for enterprises and developers to acquire AI capabilities. In MaaS platforms, tenants call model inference interfaces via APIs and consume token resources. The platform needs to set token quotas for each tenant and implement rate limiting for requests exceeding these quotas. Therefore, token quota management and rate limiting methods are required to achieve fair resource allocation among multiple tenants and ensure the overall stability of the platform's services.

[0003] Currently, most quota control and over-limit rate limiting methods use the fixed-rate token bucket algorithm. For example, Chinese publication CN 116170381 A discloses a method and related apparatus for rate limiting access requests based on multi-tenancy, which discloses a private token bucket to perform differentiated rate limiting on access requests initiated by tenants of different levels, avoiding affecting access to the multi-tenant platform by higher-level tenants; at the same time, it prioritizes trying to obtain tokens from the public token bucket to make full use of token resources.

[0004] However, the above methods cannot distinguish between intermittent, reasonable bursts of traffic and continuous, malicious traffic consumption. If an administrator wants to temporarily increase concurrency or adjust the speed for advanced tenants, they need to manually modify the configuration and restart or reload the service, resulting in delayed responses and an inability to dynamically and adaptively adjust based on the tenant's actual usage patterns. Furthermore, in a distributed gateway cluster, using a centralized database to synchronize global quotas can easily lead to single-point performance bottlenecks and failure risks, and policy updates suffer from synchronization delays between cluster nodes, resulting in inaccurate quota management. Summary of the Invention

[0005] To address this issue, the present invention provides a method for token quota management and over-limit rate limiting in multi-tenant MaaS platforms, in order to solve the problems in the prior art where dynamic adjustments cannot be made based on tenant usage patterns, and where policy updates are delayed and inconsistent, leading to inaccurate quota management.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Methods for token quota management and rate limiting over-limit handling in multi-tenant MaaS platforms include the following steps:

[0008] S1: Assign a data object identifier to each tenant's quota policy and register the quota policy object set to the Handle registration system; the quota policy object set includes the basic fill rate, burst factor, decay coefficient, object version number and object storage node locator;

[0009] S2: The API gateway node receives model call requests initiated by tenants, extracts the tenant identifier and the number of tokens to be consumed in this request from the request, and combines them to form a control dataset;

[0010] S3: Based on the tenant identifier in the control dataset, generate a Handle identifier corresponding to the tenant quota policy according to the preset identifier construction rules; and use the local Handle parser to send a parsing request to the Handle parsing system to obtain the reference information of the data object corresponding to the Handle identifier;

[0011] S4: Compare the parsed object version number with the local cache policy version number; if the local version number is lower than the parsed version number, determine the complete quota policy object set from the corresponding remote storage node based on the object storage node locator.

[0012] S5: Based on the number of tokens to be consumed in this request as stated in S2, calculate the revised token consumption number for this request. The gateway node queries the current number of tokens in the local dynamic token bucket and compares it with the calculated corrected token consumption number to determine whether the current number of tokens in the local dynamic token bucket is greater than or equal to the corrected token consumption number.

[0013] S6: Initiate the remote quota segment application process, and send a quota segment application request to the remote storage node based on the object storage node locator;

[0014] S7: Perform double verification on the quota segment application request at the remote storage node, determine the number of allocation tokens returned by the remote storage node, calculate the dynamic window threshold, and determine whether to approve the quota segment application.

[0015] Furthermore, the tenant identifier in S2 is identity authentication information, located in a custom field in the request message header; the number of tokens to be consumed in this request is obtained by adding the number of input tokens and the number of expected output tokens.

[0016] Furthermore, the specific steps of S3 are as follows:

[0017] S3.1: Receive the control dataset and generate a Handle identifier corresponding to the tenant quota policy according to the preset identifier construction rules;

[0018] S3.2: Invoke the locally deployed Handle resolver, initiate a resolution request to the Handle resolution system, locate the resolution server that stores the reference information corresponding to the Handle identifier, and have the resolution server return the reference information of the data object bound to the Handle identifier.

[0019] Furthermore, the modified token consumption number The calculation formula is as follows:

[0020]

[0021] in, To request the consumption number; The preset attenuation coefficient has a value range of [value range missing]. ; The preset burst factor has a value range of [value range missing]. ; For the request time interval; It is a natural constant and the base of the exponential function.

[0022] Furthermore, the local dynamic token bucket is automatically filled with tokens over time at a base filling rate, and the upper limit of the bucket capacity is the product of the base filling rate and the burst factor.

[0023] In S5, if the current number of tokens in the local dynamic token bucket is greater than or equal to the number of tokens consumed for correction, it means that the tenant's local token quota is sufficient. Then, the number of tokens consumed for correction is deducted from the local dynamic token bucket, and the model call request is forwarded to complete the normal request processing flow.

[0024] If the current number of tokens is less than the number of tokens to be consumed, it means that the token quota cached locally by the gateway node for the corresponding tenant is insufficient to cover the consumption requirements of this request, and the remote quota segment application process of S6 is triggered.

[0025] Furthermore, the specific content of S6 is as follows:

[0026] S6.1: The gateway node locates the remote storage node that manages the global quota status of the current tenant using the object storage node locator that has been cached and obtained in S3;

[0027] S6.2: Construct a quota segment application request message, which includes the number of application tokens and the context timestamp of this request;

[0028] S6.3: Send the request message to the remote storage node pointed to by the object storage node locator. After sending the request, the gateway node enters a timed waiting state. If it receives a response from the remote storage node within the specified timeout period, it will enter the token replenishment and forwarding process or the rate limiting execution process in S7 according to the response type.

[0029] Furthermore, the dual verification in S7 includes a first verification and a second verification;

[0030] The first verification is a global quota sufficiency judgment: the remote storage node reads and maintains the global remaining quota amount, and determines whether the global remaining quota amount is greater than or equal to the number of application tokens included in this application request;

[0031] The second verification is a dynamic window threshold determination: The dynamic window threshold is calculated by combining the context timestamps included in this application request. And determine whether the sum of the number of requested tokens and the total number of tokens consumed by the tenant within the sliding time window does not exceed the threshold of the dynamic window;

[0032] Only when both checks pass will the remote storage node deduct the number of request tokens from the global remaining quota, update the current tenant's sliding time window consumption record, and return a success response including the number of allocated tokens to the gateway node; if either check fails, the remote storage node will reject the request and return an error response of insufficient quota to the gateway node.

[0033] Furthermore, the dynamic window threshold The calculation formula is as follows:

[0034]

[0035] in, The base token filling rate for tenants per unit of time; This is the size of the sliding time window; The preset attenuation coefficient; Pre-set sudden factors; This represents the average request interval.

[0036] Furthermore, after the model call request is processed, the gateway node receives the response returned by the backend inference service, extracts the actual number of output tokens from the response, and uses the actual number of output tokens to reclaim and replenish the local dynamic token bucket; when the gateway node goes offline normally or scales down, the unused tokens in the local dynamic token bucket are recycled to the global remaining quota of the remote storage node.

[0037] This invention has the following advantages: By calculating and adjusting the number of consumed tokens, it can distinguish between reasonable burst traffic and malicious traffic occupation. When a tenant continuously sends requests, causing the time interval to decrease, the adjusted number of consumed tokens automatically increases based on the attenuation coefficient and the burst penalty factor, imposing additional consumption penalties on high-frequency requests. When the request interval increases, the adjusted number of consumed tokens approaches the original intended number of consumed tokens, and normal use is not affected. This process is automatically controlled by parameters, requiring no manual adjustment by the administrator, and achieves real-time adaptive rate limiting based on tenant usage patterns.

[0038] Meanwhile, by separating the quota policy reference information from the storage location through the Handle resolution system, the policy synchronization delay and single point of failure issues of centralized solutions are resolved. After updating the object version number in the Handle registration record, each gateway node can detect the change and pull the latest policy through resolution, keeping the synchronization delay within the local cache validity period (TTL). The effective time of policy changes is determined by the TTL setting. When a remote storage node fails, only the object storage node locator needs to be modified to point to the backup node. The gateway node can then achieve seamless switching through resolution or forced resolution, avoiding single point of failure bottlenecks.

[0039] Furthermore, through dual verification by remote storage nodes and two-tiered management of local dynamic token buckets, the linkage between global quotas and tenant usage patterns is achieved. When calculating the dynamic window threshold, the remote node introduces the average request interval, so that the quota limit within the sliding time window automatically adjusts with the request frequency. This works in conjunction with the local correction of the consumed token number penalty. The local correction of the consumed token number A applies a single consumption penalty to high-frequency requests, while the remote dynamic window threshold simultaneously tightens the total window amount for high-frequency requests. Both are non-decreasing functions of the request frequency, forming a local-remote two-tiered unidirectional rate limiting effect.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0041] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0042] Figure 1 This is a flowchart illustrating the implementation of the multi-tenant MaaS platform token quota control and over-limit rate limiting method of the present invention. Detailed Implementation

[0043] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Technical engineers in the field can make some non-essential improvements and adjustments to the present invention based on the above-described content. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see Figure 1 The method for token quota management and rate limiting handling in multi-tenant MaaS platforms includes the following steps:

[0045] S1: Assign a data object identifier to each tenant's quota policy. The data object identifier uses the Handle identifier. At the same time, register the quota policy object set to the Handle registration system. This separates the policy data storage location from the reference information, providing a foundation for seamless switching when remote storage nodes fail by modifying the locator.

[0046] The quota policy object set includes a base fill rate, burst factor, decay factor, object version number, and object storage node locator. The base fill rate represents the baseline rate at which tokens are allocated to tenants under normal conditions; the burst factor is the allowed burst traffic multiple; the decay factor is used to adjust the rate at which burst penalties decay over time; the object version number is a monotonically increasing integer value used to identify the update status of this quota policy object set; and the object storage node locator is a network-addressable identifier pointing to the remote storage node that actually stores the complete quota policy object set.

[0047] After the quota policy object set is created, a unique Handle identifier is generated according to the preset identifier naming rules. This Handle identifier is the unique index of the quota policy object set in the Handle resolution system. The identifier naming rules include information such as the platform prefix and tenant identifier to ensure that quota policy object sets of different tenants have mutually exclusive data object identifiers.

[0048] Register the Handle identifier along with the reference information of the quota policy object set to the Handle registration system: Initiate a record creation request to the Handle registration system, in which the Handle identifier is bound to a set of key-value pairs, including the VERSION key and the LOC key. The VERSION key value is the object version number; the LOC key value is the object storage node locator.

[0049] All attribute fields of the quota policy object set (including basic fill rate, burst factor, decay coefficient, etc.) are persistently stored on the remote storage node pointed to by the object storage node locator, and are not stored in the Handle registration system, thereby achieving the separation of policy data and addressing metadata.

[0050] S2: The API gateway node receives model call requests initiated by tenants, extracts the tenant identifier and the number of tokens to be consumed in this request from the request, and combines them to form a control dataset. This provides accurate input parameters for policy addressing based on Handle identifiers and token consumption calculation based on time decay, enabling subsequent adaptive rate limiting decisions to be based on real request data, thereby supporting the fine-grained and fair management of dynamic token buckets.

[0051] The API gateway node serves as the data plane entry point, capable of receiving model call requests initiated by tenants through predefined application programming interfaces. Model call requests are transmitted using standard application layer protocols (such as HTTP / 1.1 or gRPC), and their message structure includes metadata fields to identify the tenant and the scale of the call.

[0052] The tenant identifier (e.g., "tenant-pro-001") is the authentication information and is located in a custom field in the request header. The number of tokens to be consumed in this request is the sum of the input token count and the expected output token count. The input token count is a positive integer, calculated by the tenant client after segmenting the submitted prompt text. The expected output token count is the maximum number of tokens allowed to be generated by the model in this call, as set by the tenant client, and is provided by the tenant client as an output limit parameter in the request.

[0053] When a model call request reaches the API gateway node, the gateway node's built-in request parsing module performs protocol parsing on the request message. The specific content of the request parsing module is as follows:

[0054] 1) Locate the specific field containing the tenant identifier according to the interface specification, and extract the tenant identifier in string form. The tenant identifier is used to obtain reference information about the quota policy object set corresponding to the tenant from the Handle parsing system;

[0055] 2) Locate the field containing the number of tokens to be consumed, convert the value of this field to an integer type, and thus obtain the number of tokens to be consumed in this request. The entire parsing process is limited to reading control metadata and does not involve in-depth processing of specific business data (such as user-input text content) in the request message body. It is also performed in a stateless synchronous manner to ensure that this step does not affect request processing latency.

[0056] 3) After extraction, combine the tenant identifier and the number of tokens to be consumed in this request into a control dataset and pass it in the current request processing context of the API gateway node.

[0057] S3: Based on the tenant identifier in the control dataset, generate a Handle identifier corresponding to the tenant quota policy through preset identifier construction rules; and through the local Handle resolver, send a resolution request to the Handle resolution system to obtain the reference information of the data object corresponding to the Handle identifier; the reference information includes the object version number and the object storage node locator.

[0058] The specific steps for S3 are as follows:

[0059] S3.1: Receive the control dataset and generate a Handle identifier corresponding to the tenant quota policy according to the preset identifier construction rules. The preset identifier construction rules are as follows:

[0060] The platform pre-applies for and registers the naming authority prefix with the Handle registration system, the fixed separator string used to identify the policy object type, and the tenant identifier extracted from the request, and concatenates the strings in a predetermined order to generate a complete Handle identifier.

[0061] For example, given the naming authority prefix "88.1000", policy type identifier "maas-quota", and tenant identifier "tenant-pro-001", the resulting Handle identifier after concatenation is "88.1000 / maas-quota-tenant-pro-001". This generation process is a pure function computation, does not rely on network interaction, and always generates the same Handle identifier for the same tenant identifier, thus ensuring the determinism and consistency of subsequent resolution operations.

[0062] S3.2: Invoke the locally deployed Handle resolver to initiate a resolution request to the Handle resolution system. The local gateway node maintains a local cache containing Handle identifiers and reference information, with each cached record associated with a time-to-live (TTL). During resolution, it first checks if a valid reference information record corresponding to the Handle identifier exists in the local cache: if it exists and the TTL has not been exceeded, the reference information in the cache is used directly without initiating a network resolution request; if it does not exist or the TTL has been exceeded, a standard Handle resolution request message is sent to the Handle resolution system.

[0063] The local Handle resolver is a software component that implements the Handle protocol client functionality. The resolution request message it sends to the Handle resolution system includes the Handle identifier to be resolved. Upon receiving this request, the Handle resolution system recursively queries the global Handle registry service using the Handle protocol's hierarchical resolution mechanism, ultimately locating the resolution server storing the reference information corresponding to the Handle identifier. This server then returns the reference information of the data object bound to the Handle identifier. After successful resolution, the returned reference information is updated in the local cache, and the Time-To-Live (TTL) is reset.

[0064] The above reference information is a set of key attributes extracted from the parsing results returned by the Handle parsing system, including the object version number and the object storage node locator.

[0065] S4: Compare the parsed object version number with the local cache policy version number; if the local version number is lower than the parsed version number, determine the complete quota policy object set from the corresponding remote storage node according to the object storage node locator, set the capacity limit of the local dynamic token bucket (capacity limit = base fill rate × burst factor) using the determined base fill rate and burst factor, and reset the filling parameters of the local dynamic token bucket using the base fill rate and decay coefficient.

[0066] The local dynamic token bucket is maintained according to the following rules: the capacity limit of the local dynamic token bucket is set to the product of the basic filling rate and the burst factor; the local dynamic token bucket is automatically filled with tokens over time at the basic filling rate, and the increment of each filling is equal to the time interval from the last filling time to the current time; if the number of tokens exceeds the bucket capacity limit after filling, it is truncated to the bucket capacity limit; the S4 reset operation only updates the configuration of parameters such as the bucket capacity limit and filling rate, and retains the number of tokens currently accumulated in the bucket (if the bucket capacity limit after reset is less than the current number of tokens, the current number of tokens is truncated to the new bucket capacity limit), and does not clear the existing tokens, so as to avoid accidentally clearing the quota already obtained by the tenant during the policy version upgrade.

[0067] S5: Based on the number of tokens to be consumed in this request as stated in S2, calculate the revised token consumption number for this request. The gateway node queries the current number of tokens in the local dynamic token bucket and compares it with the calculated corrected token consumption number to determine whether the current number of tokens in the local dynamic token bucket is greater than or equal to the corrected token consumption number. The local dynamic token bucket is automatically filled with tokens over time at a basic filling rate, and the upper limit of the bucket capacity is the product of the basic filling rate and the burst factor. In step S5, if the current number of tokens in the local dynamic token bucket is greater than or equal to the corrected token consumption number, it means that the tenant has sufficient local token quota. The corrected token consumption number is then deducted from the local dynamic token bucket, and the model call request is forwarded to complete the normal request processing flow.

[0068] If the current token count is less than the corrected consumption token count, it means that the token limit cached locally by the gateway node for the corresponding tenant is insufficient to cover the consumption requirements of this request, thus triggering the S6 remote quota allocation process. The calculation formula is as follows:

[0069]

[0070] in, To request the consumption number; The preset attenuation coefficient has a value range of [value range missing]. ; The preset burst factor has a value range of [value range missing]. ; For the request time interval; It is a natural constant and the base of the exponential function.

[0071] S6: Initiate the remote quota segment application process. Based on the object storage node locator, send a quota segment application request to the remote storage node. The quota segment application request includes the number of application tokens and the context timestamp of this request. The specific content of S6 is as follows:

[0072] S6.1: The gateway node locates the remote storage node that manages the global quota status of the tenant using the object storage node locator that has been cached and obtained in S3, without having to initiate Handle resolution again.

[0073] S6.2: Construct a quota segment application request message, which includes the number of application tokens and the context timestamp of this request. The number of application tokens is a positive integer, and its value is set to the predefined maximum segment size parameter in the quota policy object; the context timestamp is the precise time when the original model call request that triggered this application arrived at the gateway node, which is used to determine the time base when the remote storage node performs dynamic window threshold calculation.

[0074] S6.3: Send the request message to the remote storage node pointed to by the object storage node locator. After sending the request, the gateway node enters a timed waiting state. If it receives a response from the remote storage node within the specified timeout period, it will proceed to the token replenishment and forwarding process or the rate limiting execution process in S7, depending on the response type.

[0075] If no response is received within a timeout period or a connection error indication is received, the gateway node determines that the current remote storage node is unreachable, and proactively triggers a forced Handle resolution for the same Handle identifier to obtain the updated object storage node locator, and then re-initiates a quota segment request to the standby storage node pointed to by the new locator.

[0076] S7: The quota segment application request undergoes dual verification at the remote storage node to determine the number of allocation tokens returned by the remote storage node. A dynamic window threshold is calculated and used to determine whether to approve the quota segment application. This ensures that global quota allocation is linked in real-time with the tenant's actual usage pattern, preventing malicious resource consumption by high-frequency requests while providing reasonable burst resilience for normal usage patterns. Simultaneously, this step replenishes the allocation tokens to the local token bucket and completes request forwarding, achieving a closed-loop connection between global scheduling and local execution in two-level quota management.

[0077] The first verification is a global quota sufficiency check: the remote storage node reads the global remaining quota it maintains and determines whether the global remaining quota is greater than or equal to the number of application tokens included in this application request.

[0078] The second verification involves determining the dynamic window threshold: the remote storage node calculates the dynamic window threshold based on its recorded historical request logs for the tenant within the sliding time window, combined with the context timestamp included in the current request. It then determines whether the sum of the number of requested tokens and the total number of tokens consumed by the tenant within the sliding time window does not exceed the dynamic window threshold. The calculation formula is as follows:

[0079]

[0080] in, The base token filling rate for tenants per unit of time; This is the size of the sliding time window; The preset attenuation coefficient; Pre-set sudden factors; For the average request interval, , This represents the actual request frequency (times / second) of the tenant within the sliding time window. When the tenant's request frequency increases, it leads to... When decreasing, Follow The global quota limit decreases monotonically as the request frequency decreases, meaning the global quota limit tightens within the sliding time window; when the request frequency decreases, it leads to... When it increases, Follow The increase is monotonically due to the increase in the window quota, that is, the upper limit of the window quota is relaxed.

[0081] Only when both checks pass will the remote storage node deduct the requested tokens from the global remaining quota, update the tenant's sliding window consumption record, and return a success response to the gateway node including the allocated tokens. If either check fails, the remote storage node rejects the request and returns an insufficient quota error response to the gateway node.

[0082] After receiving a successful response from the remote storage node, the gateway node extracts the allocation token count from the response message. This allocation token count represents the token quota allocated to the gateway node from the global quota by the remote storage node. The gateway node then adds this allocation token count to its local dynamic token bucket, accumulating the value to the current token count in the bucket, thereby expanding the available token quantity in the local dynamic token bucket.

[0083] After replenishing tokens, the gateway node re-compares the current token count in its local dynamic token bucket with the corrected token consumption count calculated in S5. If the current token count is greater than or equal to the corrected token consumption count, the corrected token consumption count is deducted from the replenished local dynamic token bucket, and the original model call request is forwarded to the backend large language model inference service to complete the normal processing of the request. If the current token count after replenishment is still less than the corrected token consumption count, the gateway node re-executes the remote quota segment application process of S6 according to the current gap (i.e., repeating S6.1 to S6.3) until the local tokens are sufficient or the remote storage node determines that the quota is insufficient and returns an error response. If the remote returns an error response or the preset maximum number of retries is reached, the gateway node returns an over-limit rate limiting response to the tenant and does not forward this model call request.

[0084] After the model call request is processed, the gateway node receives the response from the backend inference service, extracts the actual output token count from the response, replaces the expected output token count in S2 with the actual output token count, and recalculates the actual token consumption for this request. The gateway node then performs a write-off and replenishment in its local dynamic token bucket for the difference between the actual token consumption and the corrected token consumption count already deducted in S5: if the actual consumption is less than the deducted corrected token consumption, the difference is added back to the local dynamic token bucket (but not exceeding the bucket capacity limit); if the actual consumption is greater than the deducted corrected token consumption (because the model's actual output exceeds the expected output), the difference is deducted from the local dynamic token bucket. If the local tokens are insufficient, a new remote quota segment application is triggered. The penalty premium portion in the aforementioned corrected token consumption count (is only used as a virtual metering weight for rate limiting decisions and is not included in the tenant's billing consumption; the tenant's actual billing consumption is based on the actual output token count after write-off).

[0085] When a gateway node goes offline normally or scales down, it aggregates the unused tokens in its local dynamic token bucket before terminating service, constructs a token reclamation request, and sends it to the remote storage node. Upon receiving the reclamation request, the remote storage node adds the reclaimed tokens back to the tenant's global remaining quota and synchronously updates the sliding time window consumption record (deducting the refunded portion), completing the reclamation of the unconsumed quota segment. If a gateway node crashes abnormally, the remote storage node, upon detecting a heartbeat timeout, automatically marks the most recently allocated unconfirmed consumed quota segment as reclaimable, and the remote storage node reclaims it uniformly during subsequent window sliding.

[0086] If the remote storage node determines that any condition is not met and rejects the application, the gateway node will not perform the token replenishment operation and will return an over-limit rate throttling response to the tenant.

[0087] A numerical example is also given below:

[0088] The tenant quota strategy parameters are set as follows: baseline fill rate B=100, burst factor=1.0, decay coefficient=0.1, sliding time window=10s, and number of tokens to be consumed in the request=10.

[0089] When the request interval The corrected token consumption A is calculated for 1s, 10s, and 60s respectively:

[0090] =1s: A=19.05;

[0091] =10s; A=13.68;

[0092] =60s: A≈10.02;

[0093] It is evident that the shorter the interval, the greater the number of tokens required for correction, and the stronger the penalty; as the interval increases, the correction cost approaches the original cost of 10.

[0094] Set the average request interval T between tenants within the window to 1s, 10s, and 60s, and set the dynamic window threshold. The calculation is as follows:

[0095] =1s: =1904.8;

[0096] =10s: =1367.9;

[0097] =60s: ≈1002.5;

[0098] The results show that the more frequent the requests ( The smaller the threshold, the larger the dynamic window threshold, and the greater the total amount that can be consumed within the window.

[0099] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform, characterized in that, Includes the following steps: S1: Assign a data object identifier to each tenant's quota policy and register the quota policy object set to the Handle registration system; The quota policy object set includes the basic fill rate, burst factor, decay coefficient, object version number, and object storage node locator; S2: The API gateway node receives model call requests initiated by tenants, extracts the tenant identifier and the number of tokens to be consumed in this request from the request, and combines them to form a control dataset; S3: Based on the tenant identifier in the control dataset, generate a Handle identifier corresponding to the tenant quota policy according to the preset identifier construction rules; and use the local Handle parser to send a parsing request to the Handle parsing system to obtain the reference information of the data object corresponding to the Handle identifier; S4: Compare the parsed object version number with the local cache policy version number; If the local version number is lower than the version number obtained by parsing, the complete quota policy object set is determined from the corresponding remote storage node based on the object storage node locator; S5: Based on the number of tokens to be consumed in this request in S2, calculate the corrected token consumption number for this request. The gateway node queries the current token count in the local dynamic token bucket and compares it with the calculated corrected token consumption number to determine whether the current token count in the local dynamic token bucket is greater than or equal to the corrected token consumption number. S6: Initiate the remote quota segment application process, and send a quota segment application request to the remote storage node based on the object storage node locator; S7: Perform double verification on the quota segment application request at the remote storage node, determine the number of allocation tokens returned by the remote storage node, calculate the dynamic window threshold, and determine whether to approve the quota segment application.

2. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The tenant identifier in S2 is identity authentication information, located in a custom field in the request message header; the number of tokens to be consumed in this request is the sum of the number of input tokens and the number of expected output tokens.

3. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The specific steps of S3 are as follows: S3.1: Receive the control dataset and generate a Handle identifier corresponding to the tenant quota policy according to the preset identifier construction rules; S3.2: Invoke the locally deployed Handle resolver, initiate a resolution request to the Handle resolution system, locate the resolution server that stores the reference information corresponding to the Handle identifier, and have the resolution server return the reference information of the data object bound to the Handle identifier.

4. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The number of tokens consumed for the correction The calculation formula is as follows: in, To request the consumption number; The preset attenuation coefficient has a value range of [value range missing]. ; The preset burst factor has a value range of [value range missing]. ; For the request time interval; It is a natural constant and the base of the exponential function.

5. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The local dynamic token bucket is automatically filled with tokens over time at a base filling rate, and the upper limit of the bucket capacity is the product of the base filling rate and the burst factor. In S5, if the current number of tokens in the local dynamic token bucket is greater than or equal to the number of tokens consumed for correction, it means that the tenant's local token quota is sufficient. Then, the number of tokens consumed for correction is deducted from the local dynamic token bucket, and the model call request is forwarded to complete the normal request processing flow. If the current number of tokens is less than the number of tokens to be consumed, it means that the token quota cached locally by the gateway node for the corresponding tenant is insufficient to cover the consumption requirements of this request, and the remote quota segment application process of S6 is triggered.

6. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The specific content of S6 is as follows: S6.1: The gateway node locates the remote storage node that manages the global quota status of the current tenant using the object storage node locator that has been cached and obtained in S3; S6.2: Construct a quota segment application request message, which includes the number of application tokens and the context timestamp of this request; S6.3: Send the request message to the remote storage node pointed to by the object storage node locator. After sending the request, the gateway node enters a timed waiting state. If it receives a response from the remote storage node within the specified timeout period, it will enter the token replenishment and forwarding process or the rate limiting execution process in S7 according to the response type.

7. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, The dual verification in S7 includes a first verification and a second verification. The first verification is a global quota sufficiency judgment: the remote storage node reads and maintains the global remaining quota amount, and determines whether the global remaining quota amount is greater than or equal to the number of application tokens included in this application request; The second verification is a dynamic window threshold determination: The dynamic window threshold is calculated by combining the context timestamps included in this application request. And determine whether the sum of the number of requested tokens and the total number of tokens consumed by the tenant within the sliding time window does not exceed the threshold of the dynamic window; Only when both checks pass will the remote storage node deduct the number of request tokens from the global remaining quota, update the current tenant's sliding time window consumption record, and return a success response including the number of allocated tokens to the gateway node; if either check fails, the remote storage node will reject the request and return an error response of insufficient quota to the gateway node.

8. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 7, characterized in that, The dynamic window threshold The calculation formula is as follows: in, The base token filling rate for tenants per unit of time; This is the size of the sliding time window; The preset attenuation coefficient; Pre-set sudden factors; This represents the average request interval.

9. The method for token quota control and over-limit rate limiting in a multi-tenant MaaS platform according to claim 1, characterized in that, After the model call request is processed, the gateway node receives the response returned by the backend inference service in step S7, extracts the actual output token number from the response, replaces the expected output token number with the actual output token number, recalculates the actual token consumption number of this request, and performs verification and replenishment on the local dynamic token bucket according to the difference between the actual token consumption number and the corrected token consumption number deducted in step S5. When a gateway node goes offline or scales down normally, the unused tokens in the local dynamic token bucket are recycled to the global remaining quota of the remote storage node.

Images