Flow control methods, devices, electronic equipment and storage media

Abstract

This disclosure provides a flow control method, apparatus, electronic device, and storage medium, relating to artificial intelligence fields such as distributed storage, large model services, and the Internet. The method may include: acquiring a user's access request; determining a first sliding window of predetermined length with the current time as its end time; and determining the priority of the current request as its current priority; determining the allowed number of tokens based on the number of successful requests of the current priority within the first sliding window, the number of requests of each higher priority within the first sliding window, and the remaining number of tokens in the target token bucket corresponding to the current request, where higher priority refers to priorities higher than the current priority; adding one token to the target token bucket every predetermined time interval, and the total number of tokens stored must be less than or equal to a predetermined upper limit; and responding to the request in response to determining that the number of tokens required for the current request is less than or equal to the allowed number of tokens.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, apparatuses, electronic devices and storage media for traffic control in the fields of distributed storage, large model services and the Internet. Background Technology

[0002] For large-scale services, traffic control (or traffic management) is a critical step in ensuring service stability and user experience. As business expands and user numbers increase dramatically, the access pressure also rises. To effectively manage and allocate this access traffic and prevent system overload or service crashes, implementing rate limiting strategies is essential. Summary of the Invention

[0003] This disclosure provides flow control methods, apparatus, electronic devices, and storage media.

[0004] A flow control method, comprising:

[0005] Obtain the user's access request, determine the first sliding window of a predetermined length with the current time as the end time, and determine the priority of this request as the current priority;

[0006] Based on the number of successful requests for the current priority in the first sliding window, the number of requests for each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request, the number of tokens allowed to be used is determined. The higher priority is the priority higher than the current priority. One token is added to the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to the predetermined upper limit.

[0007] The request is responded to if it is determined that the number of tokens required for this request is less than or equal to the number of tokens allowed.

[0008] A flow control device includes: a first processing module, a second processing module, and a third processing module;

[0009] The first processing module is used to acquire the user's access request, determine a first sliding window of a predetermined length with the current time as the end time, and determine the priority of the current request as the current priority;

[0010] The second processing module is used to determine the number of tokens that can be used based on the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request. The higher priority is a priority higher than the current priority. The target token bucket is put into the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to a predetermined upper limit.

[0011] The third processing module is used to respond to the request in response to determining that the number of tokens required for this request is less than or equal to the number of tokens allowed to be used.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of the first embodiment of the flow control method described in this disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the implementation process of the traditional token bucket algorithm.

[0022] Figure 3 This is a flowchart of the second embodiment of the flow control method described in this disclosure;

[0023] Figure 4 This is a schematic diagram illustrating the meaning of λ and limit as described in this disclosure;

[0024] Figure 5 This is a flowchart of an embodiment of the dynamic current limiting threshold adjustment method described in this disclosure;

[0025] Figure 6 This is a schematic diagram of the composition structure of Embodiment 600 of the flow control device described in this disclosure;

[0026] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] Figure 1 This is a flowchart of the first embodiment of the flow control method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0030] In step 101, the user's access request is obtained, a first sliding window of a predetermined length with the current time as the end time is determined, and the priority of this request is determined as the current priority.

[0031] In step 102, the number of tokens allowed to be used is determined based on the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request. Higher priority is a priority higher than the current priority. One token is added to the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to the predetermined upper limit.

[0032] In step 103, the request is responded to in response to the determination that the number of tokens required for this request is less than or equal to the number of tokens allowed.

[0033] In practical applications, different access requests may have different priorities. High-priority requests should be prioritized, and low-priority requests should not affect high-priority requests. Traditionally, fixed quotas are allocated proportionally to different priorities. However, to ensure high-priority requests are not affected by low-priority requests, and considering the dynamic nature of requests, as much quota as possible is allocated to high-priority requests. This excess quota cannot be used by low-priority requests, often leading to resource waste. Furthermore, situations may arise where a priority's quota is exhausted, but lower-priority requests have surplus quotas that cannot be used, resulting in incorrect rate limiting.

[0034] The above-described method embodiment proposes an adaptive priority control method. For the obtained access request, the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket can be used to determine the number of tokens allowed to be used. The request can be responded to when it is determined that the number of tokens required for this request is less than or equal to the number of tokens allowed to be used. This avoids affecting high-priority requests, minimizes resource waste, and reduces erroneous rate limiting.

[0035] In practical applications, Figure 1 The execution entity in the illustrated embodiment can be a rate limiter. The token bucket algorithm is a basic algorithm for rate limiters. Figure 2 This is a schematic diagram illustrating the implementation process of the traditional token bucket algorithm. (For example...) Figure 2 As shown, assuming the user-configured (set) average sending rate is r, where r is typically a positive integer greater than 1 and its specific value can be determined according to actual needs, a token can be added to the token bucket every 1 / r seconds (corresponding to the predetermined duration mentioned above). Thus, r tokens will be added to the token bucket every second. Furthermore, the token bucket can hold a maximum of b tokens, where b is a positive integer greater than 1 and its specific value can also be determined according to actual needs. Once the token bucket is full when a token arrives, i.e., the predetermined limit has been reached, the token can be discarded. When a request requiring n tokens arrives, n tokens can be deducted from the token bucket. If the deduction is successful, the subsequent logic of the request can continue to execute. If the deduction fails, i.e., the remaining number of tokens in the token bucket is less than n, no tokens will be deducted, and the request will be considered rate-limited, thus rejecting / discarding the request and preventing the execution of subsequent logic. Additionally, if necessary, the token bucket can be allowed to over-issue a certain number of tokens, but from long-term operating results, the average sending rate is limited to a constant r.

[0036] The scheme described in this disclosure improves upon the traditional token bucket algorithm and proposes an adaptive priority control method. The main control objectives are as follows:

[0037] 1) Low-priority requests must not affect high-priority requests. At all times, it must be ensured that high-priority requests have sufficient quota, and low-priority requests can only be allocated from the remaining quota.

[0038] 2) Maximize resource utilization: Low priority should fill the remaining quota of high priority as much as possible to reduce resource waste.

[0039] For example, assuming that the common rate limiting threshold for different priorities is 100, and assuming that there are three priorities: P0, P1, and P2, then requests for P0 can be allocated first, followed by requests for P1. When the number of requests for P0 and P1 exceeds 100, the allocation of requests for P1 should be reduced. In addition, when the number of requests for P0 and P1 is less than 100, requests for P2 should fill the remaining quota as much as possible, while ensuring that the total number of requests does not exceed 100.

[0040] To achieve adaptive priority control, the scheme described in this disclosure combines a sliding window with a token bucket. The sliding window can be a millisecond-level sliding window, and its sliding step size can be accurate to 1 millisecond, thereby improving the accuracy of the relevant processing results.

[0041] Accordingly, upon receiving a user's access request, a first sliding window of predetermined length, with the current time as its end time, can be determined. The specific value of the predetermined length can be determined according to actual needs. For example, if the current time is "now", then "now" can be used as the end time of the first sliding window, and "now" minus the predetermined length can be used as the start time of the first sliding window. Additionally, the priority of this request can be determined as the current priority.

[0042] Then, the number of tokens allowed to be used can be determined based on the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of tokens remaining in the target token bucket corresponding to this request. Higher priority is the priority that is higher than the current priority.

[0043] In some embodiments of this disclosure, the sum of the number of requests for each high priority within the first sliding window can be obtained as a first value result. The first value result and the number of successful requests for the current priority within the first sliding window can be subtracted sequentially from the target rate limiting threshold to obtain a second value result. The target rate limiting threshold is the rate limiting threshold corresponding to this request. Then, the smaller value between the second value result and the remaining number of tokens in the target token bucket can be used as the number of tokens allowed to be used.

[0044] For example, assuming the current priority is P2, then the higher priorities are P0 and P1. We can obtain the number of requests corresponding to priorities P0 and P1 within the first sliding window, respectively, and add the two request counts together. The sum is the first value, representing the number of tokens that the higher priority should consume within the first sliding window. Additionally, we can obtain the number of successful requests corresponding to priority P2 within the first sliding window, representing the number of tokens already consumed for the current priority. Furthermore, the remaining tokens in the target token bucket corresponding to this request can be determined using existing methods.

[0045] Accordingly, the number of tokens allowed can be: min(target rate limiting threshold - high_consume - self_consume, store_permits), where high_consume represents the first value result, self_consume represents the number of successful requests of the current priority within the first sliding window, and store_permits represents the number of remaining tokens in the target token bucket.

[0046] The target rate limiting threshold -high_consume represents the number of tokens that can be used at the current priority level. The number of tokens consumed by higher priority levels must be deducted. On this basis, the target rate limiting threshold -high_consume-self_consume represents the number of tokens that have already been consumed, which is further deducted from the number of tokens that can be used at the current priority level. Taking the minimum value (min) takes into account that tokens may be consumed by lower priority levels that arrive earlier.

[0047] In the above processing method, the most recent past can be used to predict the most recent future, and the number of high-priority requests is used as the number of tokens consumed by high-priority requests in the first sliding window, even if some of them are not successfully processed. This can suppress the number of tokens that low-priority requests can obtain, and thus reserve a sufficient number of tokens for high-priority requests.

[0048] Furthermore, the number of tokens required for this request can be compared with the calculated number of allowed tokens. If the number of tokens required for this request is less than or equal to the number of allowed tokens, then a response can be given to this request, meaning the subsequent logic of the request can be executed normally. Thus, when the next sliding window is used for statistics, this request will be counted in both the current priority request count (access_slide_window) and the current priority successful request count (success_slide_window).

[0049] In addition, in some embodiments of this disclosure, after responding to the request, the number of tokens consumed by the request can be deducted from the target token bucket.

[0050] Furthermore, in some embodiments of this disclosure, in response to determining that the number of tokens required for this request is greater than the number of tokens allowed, the request can be rejected to achieve the purpose of rate limiting. However, when the statistics are performed in the next sliding window, this request will still be counted in the number of requests of the current priority.

[0051] The target token bucket mentioned above can be: a token bucket that is shared by different priorities. In other words, only one token bucket can be set, which is shared by all priorities. Correspondingly, the target rate limiting threshold can be: a rate limiting threshold that is shared by different priorities, such as 100.

[0052] Based on the above introduction, Figure 3 This is a flowchart of a second embodiment of the flow control method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0053] In step 301, the user's access request is obtained, a first sliding window of a predetermined length with the current time as the end time is determined, and the priority of this request is determined as the current priority.

[0054] In step 302, the sum of the number of requests for each high-priority request within the first sliding window is obtained as the first value result, where high priority is the priority higher than the current priority.

[0055] In step 303, the number of successful requests with the current priority within the first sliding window is obtained.

[0056] In step 304, the number of remaining tokens in the target token bucket corresponding to this request is obtained.

[0057] Assuming that different priorities in this embodiment all correspond to the same token bucket, then that token bucket is the target token bucket corresponding to this request.

[0058] In step 305, the first value result and the number of successful requests of the current priority in the first sliding window are subtracted sequentially from the target rate limiting threshold to obtain the second value result. The target rate limiting threshold is the rate limiting threshold corresponding to this request. The smaller value between the second value result and the remaining number of tokens in the target token bucket is taken as the number of tokens allowed to be used.

[0059] In step 306, it is determined whether the number of tokens required for this request is less than or equal to the number of tokens allowed. If so, step 307 is executed; otherwise, step 308 is executed.

[0060] In step 307, the request is responded to, the number of tokens consumed in this request is deducted from the target token bucket, and then the process ends.

[0061] When the statistics are performed in the next sliding window, this request will be counted in both the current priority request count and the current priority successful request count.

[0062] In step 308, the request is rejected, and the process ends.

[0063] When the statistics are performed in the next sliding window, this request will also be counted in the number of requests with the current priority.

[0064] exist Figure 3 Based on the embodiments shown, this disclosure further proposes a multi-level threshold control method.

[0065] This means that different rate limiting thresholds can be configured for different priorities. For example, assuming there are three priorities, P0, P1, and P2, the rate limiting threshold for P0 is 100, and the rate limiting thresholds for P1 and P2 are 80. In other words, the quota that P1 and P2 can use is: max(80-P0's used tokens, 0). When the quota used by P0 exceeds 80, all P1 and P2 requests will be rejected.

[0066] Accordingly, the aforementioned single token bucket needs to be changed to multiple serial token buckets. The multiple token buckets have the same window but different quotas, that is, different rate limiting thresholds. The multiple token buckets can be sorted in descending order of quota size.

[0067] Thus, in some embodiments of this disclosure, the target rate limiting threshold corresponding to the current request is the rate limiting threshold corresponding to the current priority. There are at least two rate limiting thresholds, and each rate limiting threshold corresponds to at least one priority. The target token bucket corresponding to the current request is the token bucket corresponding to the target rate limiting threshold. Different rate limiting thresholds correspond to different token buckets.

[0068] The specific implementation of multi-level threshold control method and Figure 3 The embodiments shown (corresponding to the adaptive priority control method) are similar, with the main differences being:

[0069] 1) In step 304, Figure 3 In the illustrated embodiment, the number of remaining tokens in the single token bucket that is set is obtained, while in the multi-level threshold control method, the number of remaining tokens in the token bucket corresponding to the current priority's rate limiting threshold is obtained.

[0070] 2) In step 305, Figure 3 In the illustrated embodiment, the target rate limiting threshold refers to the rate limiting threshold that is common to all priorities, while in the multi-level threshold control method, the target rate limiting threshold refers to the rate limiting threshold corresponding to the current priority.

[0071] 3) In step 307, Figure 3 In the illustrated embodiment, only the number of tokens consumed in this request is deducted from the target token bucket (because there is only one token bucket). In the multi-level threshold control method, after responding to this request, the number of tokens consumed in this request is deducted from each token bucket respectively. This allows all token buckets to be traversed, ensuring that the number of tokens consumed in this request is successfully deducted from each token bucket. In particular, if the number of tokens in a certain token bucket is insufficient for deduction, the method of pre-drawing tokens from the next sliding window can be used to achieve this.

[0072] By adopting a multi-level threshold control method, more accurate flow control can be achieved, thereby further reducing resource waste and erroneous flow limiting.

[0073] The scheme described in this disclosure also proposes an adaptive distributed traffic fluctuation control method, which mainly includes the following three parts.

[0074] 1) Rate limiting threshold algorithm

[0075] In some embodiments of this disclosure, in response to determining that the number of token buckets is greater than 1, the corresponding rate limiting threshold for any token bucket can be determined in the following ways: the token bucket is taken as the token bucket to be processed, and a pre-set threshold is taken as the rate limiting threshold corresponding to the token bucket to be processed; or, a third value is determined based on the pre-set expected query per second (QPS) and error rate limiting ratio, and the third value is taken as the rate limiting threshold corresponding to the token bucket to be processed.

[0076] Specifically, in some embodiments of this disclosure, a first difference between 1 and the error current limiting ratio can be obtained, and a third value can be calculated using the inverse Poisson distribution cumulative function based on the first difference and the expected QPS.

[0077] You can directly use the set threshold as the rate limiting threshold corresponding to the token bucket to be processed, such as 80. Alternatively, you can determine the rate limiting threshold corresponding to the token bucket to be processed based on the set expected QPS and error rate limiting ratio, which is very flexible and convenient.

[0078] If the service's maximum QPS is λ, and the rate limiting threshold is set to λ, the actual QPS may be less than λ. This is because the number of requests fluctuates significantly over a short period. At certain moments, the number of requests may exceed λ, and the excess requests will be rejected. Conversely, at other moments, the number of requests may fall below λ, resulting in a final actual QPS less than λ. Therefore, if the desired average QPS is λ, the rate limiting threshold should not be fixed at λ but should be dynamically adjusted based on the number of requests.

[0079] Generally speaking, the number of requests received per unit time follows a Poisson distribution, and the probability density function of the Poisson distribution is:

[0080]

[0081] Where k represents the number of times a random event occurs, λ represents the expected value of k, and specifically in the scheme described in this disclosure, k represents the number of requests actually occurring within a unit rate limiting period (e.g., 1 second), while λ represents the average value over several periods.

[0082] Accordingly, assuming the expected QPS is λ, the values ​​of λ and the error rate limiting ratio x can be preset, such as λ = 100 and x = 0.01%, and the third value result can be calculated as follows:

[0083] limit=poisson.ppf(1-x,λ); (2)

[0084] Where `poisson.ppf()` represents the inverse cumulative function of the Poisson distribution, and `limit` represents the third value, which is the cumulative distribution value of the Poisson distribution, or the quantile value. Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the meaning of λ and limit as described in this disclosure.

[0085] By using the methods described above, the accuracy of the obtained flow limiting threshold can be improved, thereby further improving the accuracy of the flow control results.

[0086] 2) Dynamic current limiting threshold adjustment

[0087] QPS can also be called the number of requests per second that can be processed. The rate limiting threshold obtained by the method in 1) is mainly applicable to the case where the average number of requests per second is λ. However, if the average number of requests per second exceeds λ and the traffic continues to grow, the actual number of requests per second will reach the limit. In this case, the underlying service will be under pressure beyond its capacity, and the rate limiter will actually fail. Therefore, preferably, the rate limiting threshold can also be dynamically adjusted.

[0088] In some embodiments of this disclosure, after responding to the current request, a second sliding window of a predetermined length with the current time as the end time can be determined, and the sum of the number of requests with different priorities within the second sliding window can be obtained as a fourth value result. Further, in response to determining that the adjustment conditions are met based on the fourth value result, the rate limiting threshold corresponding to the token bucket to be processed can be adjusted.

[0089] After processing the current request, a second sliding window can be determined. The length of the second sliding window is the same as that of the first sliding window. The total number of requests for each priority within the second sliding window can be counted as the fourth value result, denoted as s, where s ~ Poisson(s). If the adjustment conditions are met according to the fourth value result, the rate limiting threshold corresponding to the token bucket to be processed can be adjusted, thereby achieving the purpose of dynamically adjusting the rate limiting threshold.

[0090] In some embodiments of this disclosure, the sum of 1 and the error rate limiting ratio can be obtained, and the product of the sum and the expected QPS can be obtained. The product is used as the fifth value result. In response to determining that the fourth value result is greater than the fifth value result, the adjusted rate limiting threshold can be determined based on the fourth value result and the expected QPS. The rate limiting threshold corresponding to the token bucket to be processed can be adjusted to the adjusted rate limiting threshold. In response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is less than the third value result, the rate limiting threshold corresponding to the token bucket to be processed can be adjusted to the third value result.

[0091] In some embodiments of this disclosure, the method for determining the adjusted current limiting threshold based on the fourth value result and the expected QPS may include: obtaining a second difference between the fourth value result and the expected QPS, obtaining the ratio of the second difference to the fourth value result, obtaining a third difference between 1 and the ratio, calculating a sixth value result based on the third difference and the fourth value result using the inverse Poisson distribution cumulative function, and using the sixth value result as the adjusted current limiting threshold in response to determining that the sixth value result is greater than or equal to the expected QPS, and using the expected QPS as the adjusted current limiting threshold in response to determining that the sixth value result is less than the expected QPS.

[0092] In addition, in some embodiments of this disclosure, in response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is the third value result, the rate limiting threshold corresponding to the token bucket to be processed can be kept unchanged.

[0093] Through the above processing, the rate limiting threshold corresponding to the token bucket to be processed can be limited to fluctuate between [λ, limit]. Specifically, when the average number of requests per second does not exceed λ, the rate limiting threshold is adjusted to limit; conversely, the rate limiting threshold is adjusted downwards, with the lower limit being λ. The number of requests for online services fluctuates greatly in a short period, with instantaneous request peaks sometimes exceeding troughs by more than double, while remaining smooth over a relatively long time. Dynamically adjusting the rate limiting threshold can effectively address this traffic spike phenomenon.

[0094] Based on the above introduction, Figure 5This is a flowchart illustrating an embodiment of the dynamic current limiting threshold adjustment method described in this disclosure. Figure 5 As shown, the specific implementation methods are as follows.

[0095] In step 501, after responding to the current request, a second sliding window of a predetermined length with the current time as the end time is determined, and the total number of requests s of different priorities within the second sliding window is obtained.

[0096] In step 502, the conditions for s are determined. If s <= λ*(1+x) and the rate limiting threshold corresponding to the token bucket to be processed is determined to be limit, then step 503 is executed. If s <= λ*(1+x) and the rate limiting threshold corresponding to the token bucket to be processed is determined to be less than limit, then step 504 is executed. If s > λ*(1+x), then step 505 is executed.

[0097] In step 503, the rate limiting threshold corresponding to the token bucket to be processed is kept unchanged, and then the process ends.

[0098] If s <= λ*(1+x) and the rate limiting threshold corresponding to the token bucket to be processed is limit, then it means that the current state is as expected, so no adjustment is needed.

[0099] In step 504, the rate limiting threshold corresponding to the token bucket to be processed is adjusted to limit, and then the process ends.

[0100] If s <= λ*(1+x) and the rate limiting threshold corresponding to the token bucket to be processed is less than limit, then the rate limiting threshold can be adjusted to limit.

[0101] In step 505, the adjusted rate limiting threshold is determined based on s and λ, and the rate limiting threshold corresponding to the token bucket to be processed is adjusted to the adjusted rate limiting threshold, and then the process ends.

[0102] If s > λ*(1+x), it means that the previously calculated limit is invalid, and the current limiting threshold needs to be reduced. The adjustment method is as follows:

[0103] e = (s - λ) / s; (3)

[0104] limit'=poisson.ppf(1-e,s); (4)

[0105] Here, limit' represents the recalculated rate limiting threshold, i.e., the adjusted rate limiting threshold.

[0106] 3) Distributed threshold adjustment

[0107] In some embodiments of this disclosure, the execution entity of the method described herein may be a rate limiter. For the token bucket to be processed, in response to the common rate limiter corresponding to different data centers, the expected QPS may be the overall QPS set for all data centers. Different data centers correspond to different regions (such as North China, Northeast China, etc.). In response to the different data centers corresponding to their own rate limiters, the expected QPS may be the product of the overall QPS and a predetermined percentage, where the predetermined percentage is the proportion of the traffic of the data center corresponding to the token bucket to be processed to the total traffic of all data centers.

[0108] Internet services are typically deployed geographically. One approach is centralized rate limiting, which involves setting up a common rate limiter for different regions. However, this can lead to increased network latency due to cross-data center access, affecting rate limiting performance. Alternatively, each data center can be equipped with its own rate limiter to reduce network latency.

[0109] For example, suppose there are four data centers: A, B, C, and D, with a traffic ratio of 2:1:1:1. According to the Poisson distribution, the sum of two independent random variables that follow a Poisson distribution still follows a Poisson distribution. That is, if A ~ Poisson(λ1), B ~ Poisson(λ2), C ~ Poisson(λ3), and D ~ Poisson(λ4), then A + B + C + D ~ Poisson(λ1 + λ2 + λ3 + λ4). Therefore, in a distributed scenario, the rate limiting threshold can be adjusted according to the expected QPS of a single data center. This way, each data center can be rate-limited independently, avoiding cross-data center access and improving performance.

[0110] Assuming the token bucket to be processed is the token bucket corresponding to data center A, and assuming the total QPS is 100, then when determining the rate limiting threshold corresponding to the token bucket to be processed, the corresponding expected QPS (i.e., λ) can be 100 * 40% = 40, where 40% is the proportion of the traffic of the data center corresponding to the token bucket to be processed to the total traffic of all data centers.

[0111] All flow limiters operate in the same way, supporting the aforementioned adaptive priority control, multi-level threshold control, and adaptive distributed flow fluctuation control methods.

[0112] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0114] Figure 6 This is a schematic diagram of the structural composition of embodiment 600 of the flow control device described in this disclosure. Figure 6 As shown, it includes: a first processing module 601, a second processing module 602 and a third processing module 603.

[0115] The first processing module 601 is used to obtain the user's access request, determine a first sliding window of a predetermined length with the current time as the end time, and determine the priority of the current request as the current priority.

[0116] The second processing module 602 is used to determine the number of tokens that can be used based on the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request. Higher priority refers to priorities higher than the current priority. One token is added to the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to the predetermined upper limit.

[0117] The third processing module 603 is used to respond to the request in response to determining that the number of tokens required for this request is less than or equal to the number of tokens allowed.

[0118] After the first processing module 601 obtains the user's access request, it can determine a first sliding window of a predetermined length with the current time as the end time. For example, if the current time is now, then now can be used as the end time of the first sliding window, and now minus the predetermined length can be used as the start time of the first sliding window. In addition, the priority of this request can be determined as the current priority.

[0119] In some embodiments of this disclosure, the second processing module 602 can obtain the sum of the number of requests for each high priority within the first sliding window as a first value result, and can sequentially subtract the first value result and the number of successful requests for the current priority within the first sliding window from the target rate limiting threshold to obtain a second value result. The target rate limiting threshold is the rate limiting threshold corresponding to this request. Then, the smaller value between the second value result and the remaining number of tokens in the target token bucket can be used as the number of tokens allowed to be used.

[0120] In some embodiments of this disclosure, the target rate limiting threshold includes: a rate limiting threshold commonly corresponding to different priorities; the target token bucket includes: a token bucket commonly corresponding to different priorities; or, the target rate limiting threshold includes: a rate limiting threshold corresponding to the current priority; the number of rate limiting thresholds is at least two, each rate limiting threshold corresponds to at least one priority; the target token bucket includes: a token bucket corresponding to the target rate limiting threshold; and different rate limiting thresholds correspond to different token buckets.

[0121] The third processing module 603 may respond to the request if it determines that the number of tokens required for this request is less than or equal to the allowed number of tokens. Additionally, in some embodiments of this disclosure, after responding to the request based on a common token bucket corresponding to different priorities, the third processing module 603 may deduct the number of tokens consumed by this request from the target token bucket; and after responding to the request based on a determination that the number of token buckets is greater than one, it may deduct the number of tokens consumed by this request from each token bucket separately.

[0122] Furthermore, in some embodiments of this disclosure, the third processing module 603 may reject the request in response to determining that the number of tokens required for the current request is greater than the number of tokens allowed to be used.

[0123] In some embodiments of this disclosure, in response to determining that the number of token buckets is greater than 1, the second processing module 602 may determine the corresponding rate limiting threshold for any token bucket in the following ways: taking the token bucket as the token bucket to be processed and taking the preset threshold as the rate limiting threshold corresponding to the token bucket to be processed; or, determining a third value result based on the preset expected QPS and error rate limiting ratio, and taking the third value result as the rate limiting threshold corresponding to the token bucket to be processed.

[0124] Specifically, in some embodiments of this disclosure, the second processing module 602 can obtain a first difference between 1 and the error current limiting ratio, and can calculate a third value result based on the first difference and the expected QPS using the inverse Poisson distribution cumulative function.

[0125] In addition, in some embodiments of this disclosure, after determining to respond to the current request, the second processing module 602 can also determine a second sliding window of a predetermined length with the current time as the end time, and can obtain the sum of the number of requests of different priorities within the second sliding window as a fourth value result. Furthermore, in response to determining that the adjustment conditions are met based on the fourth value result, the rate limiting threshold corresponding to the token bucket to be processed can be adjusted.

[0126] In some embodiments of this disclosure, the second processing module 602 can obtain the sum of 1 and the error rate limiting ratio, and can obtain the product of the sum and the expected QPS, and use the product as the fifth value result. In response to determining that the fourth value result is greater than the fifth value result, the adjusted rate limiting threshold can be determined according to the fourth value result and the expected QPS, and the rate limiting threshold corresponding to the token bucket to be processed can be adjusted to the adjusted rate limiting threshold. In response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is less than the third value result, the rate limiting threshold corresponding to the token bucket to be processed can be adjusted to the third value result.

[0127] In some embodiments of this disclosure, the second processing module 602 determines the adjusted rate limiting threshold based on the fourth value result and the expected QPS in the following ways: obtaining a second difference between the fourth value result and the expected QPS, obtaining the ratio of the second difference to the fourth value result, obtaining a third difference between 1 and the ratio, calculating a sixth value result based on the third difference and the fourth value result using the inverse Poisson distribution cumulative function, and using the sixth value result as the adjusted rate limiting threshold in response to determining that the sixth value result is greater than or equal to the expected QPS, and using the expected QPS as the adjusted rate limiting threshold in response to determining that the sixth value result is less than the expected QPS.

[0128] In addition, in some embodiments of this disclosure, the second processing module 602 can maintain the rate limiting threshold corresponding to the token bucket to be processed unchanged in response to determining that the fourth value result is less than or equal to the fifth value result and determining that the rate limiting threshold corresponding to the token bucket to be processed is the third value result.

[0129] Furthermore, in some embodiments of this disclosure, Figure 6 The device shown can be applied to a rate limiter. For a token bucket to be processed, in response to the common rate limiter for different data centers, the expected QPS can be the total QPS set for all data centers. Different data centers correspond to different regions, and in response to the different rate limiters for different data centers, the expected QPS can be the product of the total QPS and a predetermined percentage, where the predetermined percentage is the proportion of the traffic of the data center corresponding to the token bucket to be processed to the total traffic of all data centers.

[0130] Figure 6The specific workflow of the device embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0131] In summary, the scheme described in this disclosure can achieve high-performance distributed multi-priority traffic control and can be applied to various scenarios that require traffic control, with wide applicability. For example, it can be applied to an online service of a large language model, assuming that it includes 5 priorities and 3 token buckets can be set.

[0132] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as distributed storage, large model services, and the Internet. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0133] The requests and other information described in the embodiments of this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0135] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0137] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0145] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A flow control method, comprising: Obtain the user's access request, determine the first sliding window of a predetermined length with the current time as the end time, and determine the priority of this request as the current priority; Based on the number of successful requests for the current priority in the first sliding window, the number of requests for each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request, the number of tokens allowed to be used is determined. The higher priority is the priority higher than the current priority. One token is added to the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to the predetermined upper limit. The request is responded to if it is determined that the number of tokens required for this request is less than or equal to the number of tokens allowed to be used. The determination of the number of tokens allowed to be used includes: obtaining the sum of the number of requests of each high priority within the first sliding window as a first value result; subtracting the first value result and the number of successful requests of the current priority within the first sliding window from the target rate limiting threshold in turn to obtain a second value result, wherein the target rate limiting threshold is the rate limiting threshold corresponding to this request; The smaller of the second value and the remaining number of tokens in the target token bucket is taken as the number of tokens allowed to be used.

2. The method according to claim 1, wherein, The target rate limiting threshold includes: rate limiting thresholds that are common to different priorities; The target token bucket includes: token buckets that correspond to different priorities; or, The target rate limiting threshold includes: the rate limiting threshold corresponding to the current priority, and the number of the rate limiting thresholds is at least two, with each rate limiting threshold corresponding to at least one priority; The target token bucket includes the token bucket corresponding to the target rate limiting threshold, with different rate limiting thresholds corresponding to different token buckets.

3. The method according to claim 2, further comprising: In response to determining that different priorities correspond to a common token bucket, after responding to this request, the number of tokens consumed by this request is deducted from the target token bucket; In response to determining that the number of token buckets is greater than 1, after responding to this request, the number of tokens consumed in this request is deducted from each token bucket.

4. The method according to claim 2, wherein, In response to determining that the number of token buckets is greater than 1, the corresponding rate limiting threshold is determined for each token bucket in the following manner: The token bucket is used as the token bucket to be processed, and the pre-set threshold is used as the rate limiting threshold corresponding to the token bucket to be processed. Alternatively, a third value can be determined based on the pre-set expected query rate per second and the error rate limiting ratio, and the third value can be used as the rate limiting threshold corresponding to the token bucket to be processed.

5. The method according to claim 4, wherein, The process of determining the third value based on the pre-set expected query rate per second and error rate limiting ratio includes: Obtain the first difference between 1 and the error rate limiting ratio, and calculate the third value result based on the first difference and the expected query rate per second using the inverse Poisson cumulative function.

6. The method according to claim 4, further comprising: After responding to this request, a second sliding window of the predetermined length with the current time as the end time is determined; The sum of the number of requests with different priorities within the second sliding window is obtained as the fourth value result; In response to determining that the adjustment conditions are met based on the fourth value result, the rate limiting threshold corresponding to the token bucket to be processed is adjusted.

7. The method according to claim 6, wherein, The step of adjusting the rate limiting threshold corresponding to the token bucket to be processed based on the fourth value result to determine that the adjustment conditions are met includes: Get the sum of 1 and the error rate limiting ratio, and get the product of the sum and the expected query rate per second, and use the product as the fifth value result; In response to determining that the fourth value result is greater than the fifth value result, an adjusted rate limiting threshold is determined based on the fourth value result and the expected query rate per second, and the rate limiting threshold corresponding to the token bucket to be processed is adjusted to the adjusted rate limiting threshold; In response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is less than the third value result, the rate limiting threshold corresponding to the token bucket to be processed is adjusted to the third value result.

8. The method according to claim 7, wherein, The step of determining the adjusted rate limiting threshold based on the fourth value result and the expected query rate per second includes: Obtain the second difference between the fourth value result and the expected query rate per second, and obtain the ratio of the second difference to the fourth value result; Obtain the third difference between 1 and the ratio, and calculate the sixth value based on the third difference and the fourth value result using the inverse Poisson distribution cumulative function; In response to determining that the sixth value result is greater than or equal to the expected query rate per second, the sixth value result is used as the adjusted rate limiting threshold; in response to determining that the sixth value result is less than the expected query rate per second, the expected query rate per second is used as the adjusted rate limiting threshold.

9. The method according to claim 7, further comprising: In response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is the third value result, the rate limiting threshold corresponding to the token bucket to be processed remains unchanged.

10. The method according to claim 4, wherein, The method is executed by a current limiter; In response to the common rate limiter for different data centers, the expected query rate per second is the overall query rate per second set for all data centers, with each data center corresponding to a different region; In response to the fact that different data centers correspond to their respective rate limiters, the expected query rate per second is the product of the overall query rate per second and the predetermined percentage, where the predetermined percentage is the proportion of the traffic of the data center corresponding to the pending token bucket to the total traffic of all data centers.

11. The method according to any one of claims 1 to 10, further comprising: The request is rejected if it is determined that the number of tokens required for this request is greater than the number of tokens allowed.

12. A flow control device, comprising: The first processing module, the second processing module, and the third processing module; The first processing module is used to acquire the user's access request, determine a first sliding window of a predetermined length with the current time as the end time, and determine the priority of the current request as the current priority; The second processing module is used to determine the number of tokens that can be used based on the number of successful requests of the current priority in the first sliding window, the number of requests of each higher priority in the first sliding window, and the number of remaining tokens in the target token bucket corresponding to this request. The higher priority is a priority higher than the current priority. The target token bucket is put into the target token bucket every predetermined time interval, and the number of tokens stored must be less than or equal to a predetermined upper limit. The third processing module is used to respond to the request in response to determining that the number of tokens required for this request is less than or equal to the number of tokens allowed to be used; The second processing module obtains the sum of the number of requests for each high priority within the first sliding window as a first value result, and subtracts the first value result and the number of successful requests for the current priority within the first sliding window from the target rate limiting threshold to obtain a second value result. The target rate limiting threshold is the rate limiting threshold corresponding to this request. The smaller value between the second value result and the remaining number of tokens in the target token bucket is taken as the number of tokens allowed to be used.

13. The apparatus according to claim 12, wherein, The target rate limiting threshold includes: rate limiting thresholds that are common to different priorities; The target token bucket includes: token buckets that correspond to different priorities; or, The target rate limiting threshold includes: the rate limiting threshold corresponding to the current priority, and the number of the rate limiting thresholds is at least two, with each rate limiting threshold corresponding to at least one priority; The target token bucket includes the token bucket corresponding to the target rate limiting threshold, with different rate limiting thresholds corresponding to different token buckets.

14. The apparatus according to claim 13, wherein, The third processing module is further configured to, in response to determining that different priorities correspond to a common token bucket, deduct the number of tokens consumed by this request from the target token bucket after responding to this request; and in response to determining that the number of token buckets is greater than 1, deduct the number of tokens consumed by this request from each token bucket after responding to this request.

15. The apparatus according to claim 13, wherein, In response to determining that the number of token buckets is greater than 1, the second processing module determines the corresponding rate limiting threshold for any token bucket in the following ways: taking the token bucket as the token bucket to be processed and taking the preset threshold as the rate limiting threshold corresponding to the token bucket to be processed; or, determining a third value result based on the preset expected query rate per second and the error rate limiting ratio, and taking the third value result as the rate limiting threshold corresponding to the token bucket to be processed.

16. The apparatus according to claim 15, wherein, The second processing module obtains the first difference between 1 and the error rate limiting ratio, and calculates the third value result based on the first difference and the expected query rate per second using the inverse Poisson cumulative function.

17. The apparatus according to claim 15, wherein, The second processing module is further configured to, after responding to the current request, determine a second sliding window of a predetermined length with the current time as the end time, obtain the sum of the number of requests of different priorities within the second sliding window as a fourth value result, and adjust the rate limiting threshold corresponding to the token bucket to be processed in response to determining that the adjustment conditions are met based on the fourth value result.

18. The apparatus according to claim 17, wherein, The second processing module obtains the sum of 1 and the error rate limiting ratio, and obtains the product of the sum and the expected query rate per second. The product is used as the fifth value result. In response to determining that the fourth value result is greater than the fifth value result, the module determines the adjusted rate limiting threshold based on the fourth value result and the expected query rate per second, and adjusts the rate limiting threshold corresponding to the token bucket to be processed to the adjusted rate limiting threshold. In response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is less than the third value result, the module adjusts the rate limiting threshold corresponding to the token bucket to be processed to the third value result.

19. The apparatus according to claim 18, wherein, The second processing module obtains the second difference between the fourth value result and the expected query rate per second, and obtains the ratio of the second difference to the fourth value result, obtains the third difference between 1 and the ratio, calculates the sixth value result using the inverse Poisson cumulative function based on the third difference and the fourth value result, and in response to determining that the sixth value result is greater than or equal to the expected query rate per second, uses the sixth value result as the adjusted rate limiting threshold, and in response to determining that the sixth value result is less than the expected query rate per second, uses the expected query rate per second as the adjusted rate limiting threshold.

20. The apparatus according to claim 18, wherein, The second processing module is further configured to, in response to determining that the fourth value result is less than or equal to the fifth value result, and determining that the rate limiting threshold corresponding to the token bucket to be processed is the third value result, maintain the rate limiting threshold corresponding to the token bucket to be processed unchanged.

21. The apparatus according to claim 15, wherein, The device is used in a current limiter; In response to the common rate limiter for different data centers, the expected query rate per second is the overall query rate per second set for all data centers, with each data center corresponding to a different region; In response to the fact that different data centers correspond to their respective rate limiters, the expected query rate per second is the product of the overall query rate per second and the predetermined percentage, where the predetermined percentage is the proportion of the traffic of the data center corresponding to the pending token bucket to the total traffic of all data centers.

22. The apparatus according to any one of claims 12 to 21, wherein, The third processing module is further configured to reject the request in response to determining that the number of tokens required for this request is greater than the number of tokens allowed.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-11.

Images