User-level concurrent request throttling method, system, device and medium for ai model service
Patent Information
- Application Number
- CN202610977458.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-02
- Publication Date
- 2026-10-30
AI Technical Summary
[0004]1、粒度不精准：limit_req_module仅能控制整体请求速率，无法区分用户维度；limit_conn_module统计的是TCP连接数，而HTTP/1.1及以上版本支持连接复用，单个连接可承载多个请求，导致无法准确反映用户的实际并发请求数
[0045]限流粒度精准：基于用户唯一标识实现维度化限流，可精准控制单个用户的并发请求数，避免因个别用户过载影响整体服务；
Smart Images
Figure CN123020656A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method, system, device, and medium for rate limiting of user-level concurrent requests for AI model services. Background Technology
[0002] With the rapid development of artificial intelligence technology, the application scenarios of AI model services (such as large language models and computer vision models) are becoming increasingly widespread, and the scale of user requests continues to grow. In actual service deployments, high concurrency requests from a single user may exhaust AI model service resources, leading to problems such as response latency and service crashes, which in turn affect the service experience of other users. Therefore, rate limiting for AI model services is a key means to ensure service stability. Rate limiting refers to a protective measure taken to restrict user access traffic in order to ensure service stability and prevent service crashes caused by excessive load.
[0003] Common rate limiting solutions are mostly implemented based on native Nginx modules. For example, `limit_req_module` is used to limit rate based on request rate (e.g., a fixed number of requests per second), and `limit_conn_module` is used to limit rate based on concurrent connections. However, existing technologies have at least the following drawbacks:
[0004] 1. Inaccurate granularity: limit_req_module can only control the overall request rate and cannot distinguish user dimensions; limit_conn_module counts the number of TCP connections, while HTTP / 1.1 and above support connection reuse, and a single connection can carry multiple requests, which makes it impossible to accurately reflect the actual number of concurrent requests of users.
[0005] 2. Counting logic defects: The existing solution cannot accurately count the number of requests being processed. It can only intercept the request entry point and cannot associate it with the entire life cycle of the request, such as when the request is completed, the client is interrupted, or the service times out. This can easily lead to incorrect or missed rate limiting due to residual counts.
[0006] 3. Poor distributed adaptability: In multi-node deployment scenarios, the native Nginx local storage cannot achieve global count synchronization, which makes it impossible to uniformly limit the rate of requests from the same user on different nodes, and the rate limiting strategy fails.
[0007] 4. Insufficient flexibility: It is difficult to set differentiated rate limiting thresholds for different users (such as ordinary users and paid users), and it is impossible to dynamically adjust the thresholds to adapt to changes in service load. Summary of the Invention
[0008] In view of this, on the one hand, some embodiments disclose a method for rate limiting of user-level concurrent requests for AI model services, including the following steps:
[0009] S1. Environment initialization: Initialize the global default rate limiting threshold.
[0010] S2. Request access to the AI model service to verify the legitimacy of the user identifier;
[0011] S3. Perform rate limiting threshold matching to determine the rate limiting threshold for each user;
[0012] S4. Count the user's current processing requests and determine whether the rate limiting threshold is exceeded;
[0013] S5. For user requests that pass the rate limiting judgment, the AI model service executes the business logic;
[0014] S6. Request lifecycle count maintenance.
[0015] Furthermore, some embodiments of the user-level concurrent request rate limiting method for AI model services disclosed also include the following steps:
[0016] S7, Threshold dynamic adjustment.
[0017] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S1 specifically includes:
[0018] Deploy the OpenResty service and configure the Lua shared dictionary my_shared_dict using the lua_sgared_dict command to store concurrent request counts and user rate limiting thresholds for authorized users; initialize the global default rate limiting threshold.
[0019] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S2 specifically includes:
[0020] OpenResty receives AI model service requests initiated by users;
[0021] Extract a unique user identifier from the AI model service request; verify the validity of the user identifier, and return a 400 error response if it is missing or invalid.
[0022] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S3 specifically includes:
[0023] Query the current user's custom rate limiting threshold from the shared dictionary my_shared_dict, and use it as the user's current rate limiting threshold;
[0024] If no custom rate limiting threshold exists, the global default threshold will be used as the user's current rate limiting threshold.
[0025] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S4 specifically includes:
[0026] Retrieve the number of requests currently being processed by the user from the count store;
[0027] If the number of requests currently being processed is greater than or equal to the rate limiting threshold, a 429 error response will be returned, rejecting the request;
[0028] If the number of requests currently being processed is less than the rate limiting threshold, then increment the user's request count by 1 in my_shared_dict and register a request interruption callback function.
[0029] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S5 specifically includes:
[0030] User requests that pass the rate limiting test are reverse proxied to the corresponding AI model service node based on the address, where the AI model service executes the specific business logic.
[0031] Some embodiments disclose a user-level concurrent request rate limiting method for AI model services, wherein step S6 specifically includes:
[0032] When a request is completed, times out, or the client actively interrupts the request, the user's request count is decremented by 1 using an atomic operation; if the count is less than or equal to 0 after decrementing, the user's count record is deleted to release storage resources.
[0033] On the other hand, some embodiments disclose a user-level concurrent request rate limiting system for AI model services, used to implement a user-level concurrent request rate limiting method for AI model services, including:
[0034] The initialization module is configured to perform environment initialization and initialize the global default rate limiting threshold.
[0035] The verification module is configured to request access to the AI model service and verify the legitimacy of the user identifier.
[0036] The rate limiting matching module is configured to perform rate limiting threshold matching to determine the rate limiting threshold for users.
[0037] The counting module is configured to count the current processing requests and determine whether the rate limiting threshold is exceeded.
[0038] The request execution module is configured to have the AI model service execute business logic for user requests that pass the rate limiting judgment.
[0039] The counting maintenance module is configured to request lifecycle counting maintenance.
[0040] Furthermore, some embodiments disclose an electronic device comprising:
[0041] processor;
[0042] Memory; memory stores computer programs, which, when executed by a processor, implement user-level concurrent request rate limiting methods for AI model services.
[0043] On the other hand, some embodiments disclose a computer-readable storage medium storing computer instructions that, when executed by a processor, implement a user-level concurrent request rate limiting method for AI model services.
[0044] The user-level concurrent request rate limiting method for AI model services disclosed in this invention has at least the following beneficial technical effects:
[0045] Precise rate limiting granularity: Based on the user's unique identifier, dimensional rate limiting can be implemented, which can accurately control the number of concurrent requests of a single user and avoid the overall service being affected by the overload of a few users;
[0046] Reliable counting logic: It covers the entire lifecycle of a request for counting management, and ensures that count increments and decrements are executed in pairs through mechanisms such as interrupt callbacks and timeout handling, completely solving the problem of residual counts; atomic operations ensure the accuracy of counting in multi-process / multi-node scenarios.
[0047] Distributed compatibility: Global counting synchronization is achieved through distributed caching such as Redis, supporting multi-node deployment and meeting the rate limiting requirements of large-scale AI service clusters;
[0048] Flexible and controllable strategy: Supports user-differentiated threshold configuration and dynamic adjustment, and can dynamically optimize rate limiting strategy according to user level such as free / paid and service load to improve service resource utilization;
[0049] Excellent performance: Based on the event-driven architecture of OpenResty, Lua scripts have high execution efficiency, the response time of shared dictionary / Redis operations is ≤1ms, and it supports the processing of 100,000 concurrent requests without affecting the response performance of AI services. Attached Figure Description
[0050] Figure 1 , one Some embodiments disclose a flowchart of a user-level concurrent request rate limiting method for AI model services;
[0051] Figure 2 Example 1 discloses a flowchart of a user-level concurrent request rate limiting method for AI model services. Detailed Implementation
[0052] The term "embodiment" used herein, as an example, is not necessarily to be construed as superior to or better than other embodiments. Performance testing in these embodiments of the invention, unless otherwise specified, employs conventional testing methods in the art. It should be understood that the terminology used in these embodiments is merely for describing particular implementations and is not intended to limit the scope of the disclosure of these embodiments.
[0053] Unless otherwise stated, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this invention pertain; other experimental methods and technical means not specifically noted in the embodiments of this invention refer to experimental methods and technical means commonly used by one of ordinary skill in the art.
[0054] The terms “basic” and “approximately” as used herein are used to describe small fluctuations. For example, they can mean less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Numerical data presented or expressed in range format herein are used for convenience and brevity only, and should therefore be interpreted flexibly to include not only the explicitly listed values that define the range, but also all independent values or subranges contained within that range. For example, a numerical range of “1–5%” should be interpreted to include not only the explicitly listed values from 1% to 5%, but also the independent values and subranges within the indicated range. Thus, this numerical range includes independent values such as 2%, 3.5%, and 4%, and subranges such as 1%–3%, 2%–4%, and 3%–5%, etc. This principle also applies to ranges that list only one value. Furthermore, this interpretation applies regardless of the width of the range or the characteristics described.
[0055] In this document, including in the claims, conjunctions such as "comprising," "including," "with," "having," "containing," "involving," and "accommodating" are understood to be open-ended, meaning "including but not limited to." Only the conjunctions "consisting of" and "composed of" are closed conjunctions.
[0056] To better illustrate the content of this invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the invention can be practiced even without certain specific details. In the embodiments, some methods, means, instruments, and devices well-known to those skilled in the art are not described in detail, in order to highlight the main points of the invention.
[0057] Without conflict, the technical features disclosed in the embodiments of the present invention can be combined arbitrarily, and the resulting technical solution belongs to the content disclosed in the embodiments of the present invention.
[0058] In some implementations, such as Figure 1 As shown, the method for rate limiting user-level concurrent requests for AI model services includes the following steps:
[0059] S1. Environment initialization: Initialize the global default rate limiting threshold. In some embodiments, the OpenResty service is deployed, and the Lua shared dictionary `my_shared_dict` is configured using the `lua_sgared_dict` command to store the concurrent request counts of authorized users and the user rate limiting threshold configuration; the global default rate limiting threshold is initialized. Typically, this is implemented based on OpenResty (Nginx + LuaJIT), combining the flexibility of Lua scripts with the high concurrency capabilities of Nginx, providing an efficient foundation for user-level rate limiting; the rate limiting threshold usually refers to the upper limit of the number of concurrent requests.
[0060] S2. Request access to the AI model service and verify the legitimacy of the user identifier; In some embodiments, OpenResty receives the AI model service request initiated by the user; Extracts a unique user identifier from the AI model service request; Verifies the legitimacy of the user identifier, and returns a 400 error response if it is missing or invalid; Typically, the user identifier includes identifiers such as the X-User-ID request header and the user ID parsed from the authentication token; Extracts unique identifiers such as X-User-ID and authentication token from the request, and verifies their legitimacy to ensure the credibility of the identity, avoid forgery, achieve accurate user identification, and provide a premise for dimensional rate limiting;
[0061] S3. Perform rate limiting threshold matching to determine the rate limiting threshold for the user; in some embodiments, the user's custom rate limiting threshold is queried from the shared dictionary my_shared_dict and used as the user's current rate limiting threshold; if no custom rate limiting threshold exists, the global default threshold is matched and used as the user's current rate limiting threshold.
[0062] Typically, it supports both user-defined thresholds and global default thresholds, which can be dynamically adjusted through the operation and maintenance interface to adapt to different user levels such as free / paid and service load changes, thus achieving differentiated threshold management.
[0063] S4. Count the user's current processing requests and determine whether the rate limiting threshold is exceeded; In some embodiments, the number of requests currently being processed by the user is obtained from the count storage; If the number of requests currently being processed is greater than or equal to the rate limiting threshold, a 429 error response is returned to reject the request; If the number of requests currently being processed is less than the rate limiting threshold, the user's request count is incremented by 1 in my_shared_dict and a request interruption callback function is registered.
[0064] Typically, a single node uses a shared dictionary in Lua to implement lightweight counting, while a distributed scenario uses Redis to achieve global counting synchronization. By combining connection pools and key expiration mechanisms, performance and reliability are optimized, thus achieving scenario-specific storage adaptation.
[0065] S5. For user requests that pass the rate limiting judgment, the AI model service executes the business logic; in some embodiments, the user requests that pass the rate limiting judgment are reverse proxied to the corresponding AI model service node according to the corresponding address, and the AI model service executes the specific business logic.
[0066] S6. Request lifecycle count maintenance; In some embodiments, when a request is completed, times out, or the client actively interrupts the request, the user's request count is decremented by 1 through atomic operations; If the count is ≤0 after decrementing, the user's count record is deleted to release storage resources; Typically, request counts are managed through atomic operations, such as shared dictionary INCR / DECR or Redis INCR / DECR, covering the entire process of request access, processing, completion, interruption, and timeout, ensuring that count increments and decrements are paired, resolving residual issues, and achieving full lifecycle counting;
[0067] Typically, rate limiting is performed during the access phase, the forwarding phase is proxied to the AI service, and the completion / interruption phase triggers count rollback, forming a closed-loop control and achieving full-process control.
[0068] S7. Dynamic Threshold Adjustment. Typically, an operations and maintenance interface can be provided, allowing administrators to modify the rate limiting threshold for specific users in real time, with the new threshold taking effect immediately.
[0069] In some implementations, a user-level concurrent request rate limiting system for AI model services includes:
[0070] The initialization module is configured to perform environment initialization and initialize the global default rate limiting threshold.
[0071] The verification module is configured to request access to the AI model service and verify the legitimacy of the user identifier.
[0072] The rate limiting matching module is configured to perform rate limiting threshold matching to determine the rate limiting threshold for users.
[0073] The counting module is configured to count the current processing requests and determine whether the rate limiting threshold is exceeded.
[0074] The request execution module is configured to have the AI model service execute business logic for user requests that pass the rate limiting judgment.
[0075] The counting maintenance module is configured to request lifecycle counting maintenance.
[0076] Some embodiments disclose an electronic device including: a processor; a memory; the memory storing a computer program that, when executed by the processor, implements a user-level concurrent request rate limiting method for AI model services.
[0077] Some embodiments disclose a computer-readable storage medium storing computer instructions that, when executed by a processor, implement a user-level concurrent request rate limiting method for AI model services.
[0078] The technical details are further illustrated below with reference to the embodiments.
[0079] Example 1
[0080] In Example 1, as Figure 2 As shown, the methods for rate limiting user-level concurrent requests for AI model services include:
[0081] A user initiates an AI service request; OpenResty accesses the AI service request.
[0082] Extract and verify the X-User-ID. If the user's unique identifier is missing or invalid, return a 400 error.
[0083] If the user's unique identifier is valid, then query the user's rate limiting threshold;
[0084] If no custom threshold is specified, the global default threshold will be used as the user's current rate limiting threshold.
[0085] If a custom threshold is specified, the user-defined threshold will be used as the current rate limiting threshold.
[0086] Get the current request count from the shared dictionary; determine whether the current request count is greater than or equal to the current rate limiting threshold;
[0087] If so, return a 429 error to remind the user that too many requests have been made;
[0088] If not, increment the count by 1, collect the user input context information, and if it is within the AI model input context, extract the complete user input as the model input; otherwise, truncate it according to the maximum input context of the model and use it as the final input of the model; then register the interrupt callback and reverse proxy to the AI model service.
[0089] The request processing result is evaluated; if the service times out, the count is decremented by 1 and invalid records are cleared.
[0090] If the service succeeds or fails, the count is decremented by 1, and invalid records are cleared.
[0091] If the client connection is interrupted, a callback is triggered, and the counter is decremented by 1.
[0092] Finally, the response result is returned to the user.
[0093] The technical solutions and technical details disclosed in the embodiments of this invention are merely illustrative of the inventive concept of this invention and do not constitute a limitation on the technical solutions of the embodiments of this invention. Any conventional changes, substitutions, or combinations made to the technical details disclosed in the embodiments of this invention have the same inventive concept as this invention and are within the protection scope of the claims of this invention.
Claims
1. A method for rate limiting user-level concurrent requests for AI model services, characterized in that, Including the following steps: S1. Environment initialization: Initialize the global default rate limiting threshold. S2. Request access to the AI model service to verify the legitimacy of the user identifier; S3. Perform rate limiting threshold matching to determine the rate limiting threshold for each user; S4. Count the user's current processing requests and determine whether the rate limiting threshold is exceeded; S5. For user requests that pass the rate limiting judgment, the AI model service executes the business logic; S6. Request lifecycle count maintenance; S7, Threshold dynamic adjustment.
2. The user-level concurrent request rate limiting method for AI model services according to claim 1, characterized in that, Step S1 specifically includes: Deploy the OpenResty service and configure the Lua shared dictionary my_shared_dict using the lua_sgared_dict command to store concurrent request counts and user rate limiting thresholds for authorized users; initialize the global default rate limiting threshold.
3. The user-level concurrent request rate limiting method for AI model services according to claim 2, characterized in that, Step S2 specifically includes: OpenResty receives AI model service requests initiated by users; Extract a unique user identifier from the AI model service request; verify the validity of the user identifier, and return a 400 error response if it is missing or invalid.
4. The user-level concurrent request rate limiting method for AI model services according to claim 3, characterized in that, Step S3 specifically includes: Query the current user's custom rate limiting threshold from the shared dictionary my_shared_dict, and use it as the user's current rate limiting threshold; If no custom rate limiting threshold exists, the global default threshold will be used as the user's current rate limiting threshold.
5. The user-level concurrent request rate limiting method for AI model services according to claim 4, characterized in that, Step S4 specifically includes: Retrieve the number of requests currently being processed by the user from the count store; If the number of requests currently being processed is greater than or equal to the rate limiting threshold, a 429 error response will be returned, rejecting the request; If the number of requests currently being processed is less than the rate limiting threshold, then increment the user's request count by 1 in my_shared_dict and register a request interruption callback function.
6. The user-level concurrent request rate limiting method for AI model services according to claim 5, characterized in that, Step S5 specifically includes: User requests that pass the rate limiting test are reverse proxied to the corresponding AI model service node based on the address, where the AI model service executes the specific business logic.
7. The user-level concurrent request rate limiting method for AI model services according to claim 6, characterized in that, Step S6 specifically includes: When a request is completed, times out, or the client actively interrupts the request, the user's request count is decremented by 1 using an atomic operation; if the count is less than or equal to 0 after decrementing, the user's count record is deleted to release storage resources.
8. A user-level concurrent request rate limiting system for AI model services, used to implement the user-level concurrent request rate limiting method for AI model services as described in any one of claims 1 to 7, characterized in that, include: The initialization module is configured to perform environment initialization and initialize the global default rate limiting threshold. The verification module is configured to request access to the AI model service and verify the legitimacy of the user identifier. The rate limiting matching module is configured to perform rate limiting threshold matching to determine the rate limiting threshold for users. The counting module is configured to count the current processing requests and determine whether the rate limiting threshold is exceeded. The request execution module is configured to have the AI model service execute business logic for user requests that pass the rate limiting judgment. The counting maintenance module is configured to request lifecycle counting maintenance.
9. An electronic device, characterized in that, include: processor; Memory; The memory stores a computer program, which, when executed by the processor, implements the user-level concurrent request rate limiting method for AI model services as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The device stores computer instructions, which, when executed by a processor, implement the user-level concurrent request rate limiting method for AI model services as described in any one of claims 1 to 7.