A method and system for managing concurrent requests of target applications

Abstract

The present application provides a method and system for managing concurrent requests of target applications, wherein the method maintains the concurrent request status of each target application in a distributed cache, including the maximum concurrent limit value max_active_requests and the active request set active_requests, performs concurrency checks and access control when new requests are received, and automatically releases resources when the request processing is completed or an exception occurs. The system also includes a periodic cleanup module for scanning and removing timed-out requests to prevent zombie requests from occupying resources for a long time. The present invention further supports dynamic adjustment of concurrency limits at runtime, and adopts a streaming task encapsulation mechanism to simplify the request processing logic. This technical solution is suitable for artificial intelligence model service deployment platforms and can effectively improve the stability of the system.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method and system for managing concurrent requests of target applications. Background Art

[0002] Retrieval-augmented generation (RAG) technology, as an important component of artificial intelligence application service management, is widely used in scenarios such as intelligent question answering, content generation, and knowledge reasoning. In related technologies, a complete RAG service system is constructed through the collaborative operation of information retrieval, language model generation, and request scheduling mechanisms. Specifically, the system covers the entire process from user request access, knowledge retrieval, content generation to response return, including key links such as request queue management, resource allocation, and streaming output control. With the development of generative AI technology, RAG applications are gradually evolving towards distributed deployment to support high-concurrency and high-availability service requirements. In this context, the request management mechanism has become one of the core technologies to ensure system stability and resource utilization. Its design needs to take into account complex characteristics such as long connections, streaming output, and abnormal interruptions.

[0003] However, existing request management methods directly adopt flow limiting strategies based on request frequency, such as token bucket or leaky bucket algorithms, without adapting to the long duration, streaming output, and easy interruption characteristics of RAG tasks. This may lead to uneven resource usage or service response delays. In addition, the connection limit mechanism of traditional Web services is usually configured globally and cannot be fine-grained by application (app_id). After the request is abnormally interrupted, there is a lack of effective automatic cleanup methods, resulting in "zombie requests" occupying concurrent resources for a long time. In a distributed deployment environment, state synchronization and consistency assurance between multiple service nodes are particularly critical. However, existing technologies often rely on local state management or manual intervention, making it difficult to achieve efficient cross-node collaboration and resource recovery, thereby affecting the overall performance and availability of the system.

[0004] Therefore, when dealing with the streaming, long-time, and easily interrupted request characteristics unique to RAG applications, especially in a distributed environment, the existing technology lacks a comprehensive solution that can effectively limit concurrency, automatically clean up abnormal residual requests, and ensure efficient resource utilization. Summary of the Invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] The present invention proposes a method for managing concurrent requests of a target application, which is suitable for dynamically controlling concurrent requests of RAG applications and automatically recycling resources in a distributed environment, so as to improve system stability, resource utilization and user experience.

[0007] Another object of the present invention is to provide a system for managing concurrent requests of a target application.

[0008] A third object of the present invention is to provide a computer device.

[0009] To achieve the above objectives, the present invention provides a method for managing concurrent requests of a target application, comprising:

[0010] In response to a user's access request to a target application, obtaining an application identifier app_id of the target application;

[0011] In the distributed cache, a maximum concurrency limit value max_active_requests and a collection active_requests for recording active requests are maintained for app_id. The active_requests collection stores multiple request identifiers request_id and corresponding timestamps.

[0012] When a new request is received, the current number of requests in the active_requests collection is queried based on the app_id and compared with max_active_requests;

[0013] If the current number of requests is less than max_active_requests, a request_id is generated for the new request, and the request_id and the corresponding timestamp are added to the active_requests collection to allow the current request to be processed;

[0014] If the current number of requests reaches or exceeds max_active_requests, new requests will be rejected.

[0015] Configure a resource release mechanism for the request processing process. When the request processing is completed or an exception occurs, the resource release mechanism automatically removes the corresponding request_id from the active_requests collection.

[0016] Periodically execute the cleanup task, which scans the timestamps of all request_ids in the active_requests collection and removes the request_id when the difference between the current time and the timestamp exceeds the preset maximum survival time to prevent zombie requests from occupying concurrent resources for a long time.

[0017] The method for managing concurrent requests of a target application according to an embodiment of the present invention may also have the following additional technical features:

[0018] In one embodiment of the present invention, the operations of querying the current number of requests in the active_requests collection based on app_id and comparing the number with max_active_requests are completed by executing an atomic operation or a script on the distributed cache.

[0019] In one embodiment of the present invention, the resource release mechanism includes a streaming task encapsulator for automatically triggering the removal of request_id when the generation process ends normally or encounters any exception when the request processing process involves streaming generation output.

[0020] In one embodiment of the present invention, the distributed cache is Redis, and the active_requests collection is a hash data structure in Redis, where the key is request_id and the value is the corresponding timestamp.

[0021] In one embodiment of the present invention, the maximum concurrent number limit value max_active_requests is modified while the system is running, and subsequent request checks use the updated value.

[0022] To achieve the above objectives, the present invention further provides a system for managing concurrent requests of a target application, comprising:

[0023] Configure the storage module to store the application identifier app_id and the corresponding maximum concurrent request limit max_active_requests for each target application;

[0024] The distributed cache module is connected to the configuration storage module and is used to store the max_active_requests value and the active request set active_requests of each target application. The active_requests set contains multiple request identifiers request_id and corresponding timestamps;

[0025] The request admission module is connected to the distributed cache module. When responding to user requests, it queries the current number of requests in the active_requests collection based on the app_id and decides whether to allow the request to enter based on the comparison result between the current number of requests and max_active_requests.

[0026] The request processing module is connected to the request admission module to process the requests that are allowed in. When the request processing is completed or an exception occurs, the corresponding request_id is removed from the active_requests collection through the resource release module;

[0027] The periodic cleaning module is connected to the distributed cache module and is used to scan the active_requests collection according to a predetermined period. When it is detected that the difference between the timestamp of a request_id and the current time exceeds the preset maximum survival time, the request_id is removed to prevent zombie requests from occupying concurrent resources for a long time.

[0028] Furthermore, the request processing module includes a streaming task encapsulator for automatically executing resource release operations when the request processing process involves streaming generated output.

[0029] Furthermore, the distributed cache module is Redis, and the active_requests collection is a hash data structure in Redis, where the key is request_id and the value is the corresponding timestamp.

[0030] Furthermore, the configuration storage module supports dynamic updating of the max_active_requests value while the system is running, and synchronizes the updated value to the distributed cache module.

[0031] This embodiment of the present invention provides a system for managing concurrent requests for target applications, specifically targeting the application scenario of Retrieval Enhanced Generation (RAG). It provides a technology for limiting and automatically reclaiming the number of concurrent requests across distributed environments for publicly accessible URLs generated after application creation. By maintaining the active request status of each application in a distributed cache (Redis), the system dynamically limits the number of concurrent requests based on configuration. This adapts to the typical streaming output, long processing time, and proneness to abnormal interruption of RAG generation tasks, thereby ensuring service stability and efficient resource utilization.

[0032] To achieve the above-mentioned object, the present invention provides a computer device according to a third aspect, comprising:

[0033] processor;

[0034] a memory storing a computer program;

[0035] When the processor executes the computer program, it implements the method for managing concurrent requests of the target application as described in the embodiment of the first aspect.

[0036] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 is a flowchart of a method for managing concurrent requests of a target application according to an embodiment of the present invention;

[0039] Figure 2 is a specific data logic diagram of a method for managing concurrent requests of a target application according to an embodiment of the present invention;

[0040] Figure 3 is a diagram of a system structure for managing concurrent requests of a target application according to an embodiment of the present invention;

[0041] Figure 4 is a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0044] The following describes a method and system for managing concurrent requests of a target application according to an embodiment of the present invention with reference to the accompanying drawings.

[0045] The technical terms mentioned in the present invention are introduced below:

[0046] 1. RAG (Retrieval-Augmented Generation): An AI approach that combines information retrieval and text generation techniques.

[0047] 2. Redis (Remote Dictionary Server): An open-source, memory-based key-value data structure storage system, commonly used in scenarios such as caching and message queues.

[0048] 3. URL (Uniform Resource Locator): A Uniform Resource Locator (URL). It is used to identify the address of a resource on the Internet.

[0049] 4. app_id: Client Identifier. In this invention, it specifically refers to the unique identifier of each RAG application.

[0050] 5. request_id: Request Identifier. A unique identifier generated for each user request entering the system.

[0051] 6. Distributed Cache: refers to a technology or system in which cached data is distributed across multiple servers and can be shared and accessed by multiple service nodes.

[0052] 7. Streaming Generation: refers to the process of computing or generating data in a step-by-step manner, which is then sent to the receiver in blocks and continuously in the form of a data stream.

[0053] 8. Concurrency Count: The number of requests being processed by the system at the same time.

[0054] 9. Zombie Request: A request that has actually ended or was abnormally interrupted, but its status record in the system has not been cleared in time and still occupies resources.

[0055] 10. API (Application Programming Interface): A set of rules and protocols that define the interactions between software components.

[0056] Figure 1 FIG. 1 is a flow chart of a method for managing concurrent requests of a target application according to an embodiment of the present invention. Figure 1 Shown, including:

[0057] S1, in response to a user's access request to a target application, obtaining an application identifier app_id of the target application;

[0058] S2 maintains a maximum concurrency limit value max_active_requests and a collection active_requests for recording active requests for app_id in the distributed cache. The active_requests collection stores multiple request identifiers request_id and corresponding timestamps.

[0059] S3, when a new request is received, the current number of requests in the active_requests set is queried based on the app_id, and compared with max_active_requests; if the current number of requests is less than max_active_requests, a request_id is generated for the new request, and the request_id and the corresponding timestamp are added to the active_requests set to allow the current request to be processed; if the current number of requests reaches or exceeds max_active_requests, the new request is rejected;

[0060] S4, configure a resource release mechanism for the request processing process, and the resource release mechanism automatically removes the corresponding request_id from the active_requests collection when the request processing is completed or an exception occurs;

[0061] S5 periodically executes the cleanup task, which scans the timestamps of all request_ids in the active_requests collection and removes the request_id when the difference between the current time and the timestamp exceeds the preset maximum survival time to prevent zombie requests from occupying concurrent resources for a long time.

[0062] In this embodiment of the present invention, to precisely control the number of concurrent requests per application, the system employs an atomic request admission mechanism based on a distributed cache (e.g., Redis). Specifically, when a new request arrives, the system first constructs a corresponding rate limiting key based on its associated application's unique identifier, app_id. For example, rag:rate_limit:{app_id}:active_requests stores the currently active request set, while rag:rate_limit:{app_id}:max_active_requests records the application's maximum concurrent request limit.

[0063] To ensure accurate and consistent concurrency control, the system queries and compares the current number of requests by executing Lua scripts or atomic operations on Redis. This script retrieves the number of members in the active_requests hash table (i.e., the current number of active requests) and the value of max_active_requests, and performs a comparison. If the current number of requests does not reach the maximum concurrency limit, the newly generated request_id and the corresponding entry timestamp are inserted into the hash table as a field-value pair, and the current request is allowed to proceed. Otherwise, the current request is rejected and an appropriate error message (such as HTTP 429 Too Many Requests) is returned. This atomic execution avoids data races and state inconsistencies that may arise from concurrent access from multiple nodes, thereby ensuring the correctness and stability of the rate limiting policy.

[0064] To release resources, the system introduces a streaming task wrapper specifically designed for handling task scenarios involving streaming output (such as word-by-word generation of large language models and conversational interactions). This wrapper can be a Python decorator or a context manager, and its core function is to automatically monitor task status changes during the streaming generation process. Regardless of whether the streaming task completes normally or is interrupted by an exception (such as a network disconnect or internal error), the wrapper captures the corresponding end event and immediately triggers a cleanup operation—deleting the request_id record of the current request from the active_requests hash table in Redis, thereby releasing the concurrency quota it occupied. This mechanism not only simplifies the caller's logical burden but also effectively prevents "zombie" requests from occupying resources for a long time, improving system robustness and resource utilization.

[0065] Furthermore, the distributed cache is implemented using a Redis database, with the active_requests collection designed as a Redis hash data structure. Each request's unique identifier, request_id, serves as the hash table key, and the corresponding timestamp serves as the value, facilitating subsequent expiration detection and cleanup. This structure offers excellent scalability and efficient operational performance, making it suitable for use in high-concurrency, distributed environments.

[0066] Furthermore, to enhance system flexibility and adaptability, the system supports dynamic modification of the maximum concurrent request limit (max_active_requests) at runtime. Administrators can directly update the value of the corresponding key in Redis through the management interface or the configuration center without restarting the service. Subsequent request admission checks will automatically read the latest max_active_requests value, enabling real-time adjustment of the current limiting policy. This feature enables the system to flexibly configure resource quotas based on actual load conditions, adapting to the elastic scaling requirements of different business scenarios and improving the maintainability and responsiveness of the overall system.

[0067] In summary, the present invention uses Redis atomic scripts to achieve precise concurrency control, automatic resource release by streaming encapsulators, efficient management of active requests by hash structures, and dynamic adjustment of current limiting thresholds at runtime, among other key technical means, to build an efficient, stable, and scalable request current limiting and resource management mechanism, which is particularly suitable for high-concurrency application scenarios under large-scale distributed service architectures.

[0068] Specifically, if Figure 2 As shown, the data interaction process of the present invention is as follows:

[0069] The concurrent limit module interacts as follows:

[0070] Each RAG application has a unique application identifier (app_id) in the system, such as app_model.id in the database, which is used to build an independent rate limiting control unit in the distributed cache environment. Based on this identifier, the system generates two key rate limiting keys in Redis: a maximum concurrency key, rag:rate_limit:{app_id}:max_active_requests, which stores the maximum concurrency limit of the current application; and an active request hash key, rag:rate_limit:{app_id}:active_requests, whose data format is request_id:timestamp and records the set of requests currently being processed. This design enables the system to implement refined and isolated concurrency control for access requests from different applications.

[0071] When creating an application, users can set the maximum number of concurrent requests (max_active_requests). A positive value enables concurrency limits; a zero or negative value activates unlimited concurrency mode. To support multi-node consistency in dynamic deployment environments, the system uses regular synchronization or on-demand reads between the local cache and Redis to ensure that the concurrency limits obtained by each node are always consistent, avoiding current limit failures or incorrect limits due to inconsistent configurations.

[0072] Whenever a new request arrives, the system first queries Redis for the corresponding active request count based on the app_id it belongs to and compares it with the preset maximum concurrency. If the current number of active requests has reached the upper limit, the new request is rejected and an error response (such as HTTP 429 Too Many Requests) is returned. If the upper limit has not been reached, the current request's unique identifier (request_id) and the corresponding entry timestamp are written to the active_requests hash table to complete admission control. This process effectively prevents system overload and ensures service stability and availability.

[0073] The interaction between the active request tracking and automatic release modules is as follows:

[0074] When the system receives each new request, it generates a unique request identifier (request_id), for example, using a UUID algorithm. It also records the timestamp of the current request and binds this identifier to the corresponding RAG application ID (app_id) to accurately track the request's status throughout its lifecycle. This mechanism ensures that when processing multiple requests concurrently, the system can effectively distinguish and manage the execution path and resource allocation of each request.

[0075] For tasks requiring streaming output, such as generating responses for large language models, the system employs a "streaming generator wrapper" mechanism. This mechanism automatically monitors the execution status of requests through a wrapper (such as a decorator or context manager in Python). Whether the task completes normally or is interrupted by an exception, the wrapper automatically triggers a "request exit" operation, removing the record with the request_id from the Redis active_requests hash table. This prevents zombie requests from occupying system resources for a long time.

[0076] For non-streaming tasks, the caller can explicitly call the release_request method after task completion, passing in the app_id and request_id to release associated resources. For streaming tasks, the wrapper automatically captures completion or abnormal termination events, allowing the system to automatically execute the requested exit process without manual intervention from the caller. This design greatly simplifies the logical processing flow, improving the system's automation and resource management efficiency.

[0077] The automatic recovery mechanism for abnormal interruption has the following interactive process:

[0078] To prevent long-term resource usage due to abnormal interruptions in request processing, the system sets a maximum lifetime (e.g., 10 minutes) for each request registered in the active_requests hash table. If a request is not released within this time (either manually by calling exit or automatically cleaned up by the streaming encapsulation), it is considered a potential "hanging" or "zombie" request. The system uses a periodic background task (such as a cron job or daemon thread) to perform status scanning and cleanup operations to ensure timely resource recovery.

[0079] In a multi-node distributed deployment environment, node failures, network partitions, or abnormal process exits can prevent timely synchronization or clearing of request status. To address this, the system relies on timestamp information recorded in Redis and a periodic background refresh mechanism, combined with time-based expiration logic, to effectively identify and clear abnormal requests, thereby avoiding "false fullness" in concurrency control. This design ensures the eventual consistency of global throttling status and improves the system's robustness and availability in complex environments.

[0080] After being triggered at a predetermined interval (for example, every minute), the background task begins scanning the Redis hash table for all requests matching the pattern rag:rate_limit:*:active_requests. For each active request record, the task reads its corresponding timestamp and compares it with the current time. If the time difference exceeds the preset maximum lifetime, the request is deemed expired and removed from the hash table using the HDEL command, releasing the corresponding concurrency quota. This mechanism automatically detects and reclaims resources for abnormally interrupted requests without manual intervention, effectively maintaining the integrity of the system's concurrency control logic.

[0081] The distributed deployment adaptability has the following interactive process:

[0082] In this system, Redis is used as a unified, centralized state storage component. All distributed nodes access the same Redis instance to obtain the maximum concurrency configuration, query the number of currently active requests, and perform request registration and release operations. This centralized design ensures consistent and synchronized state information across nodes. To prevent data races during concurrent access, the system utilizes Redis's atomic operation mechanisms (such as Lua scripts) to encapsulate concurrency checks, request registration, and release operations into uninterruptible execution units, ensuring the security and accuracy of data operations in a multi-node environment.

[0083] The system features dynamic rate limiting capabilities. Administrators can modify the max_active_requests value stored in Redis without restarting the service to adapt to varying business loads and deployment requirements. Each service node automatically reads the latest concurrency limit when processing new requests or performing periodic cleanup tasks, enabling real-time updates to the rate limiting policy. This mechanism enhances system flexibility and maintainability, enabling rate limiting control to more quickly respond to changes in the actual operating environment.

[0084] To address challenges common in distributed systems, such as network latency, node failures, and abnormal request interruptions, this system utilizes a three-in-one mechanism: centralized state storage, automatic release and encapsulation, and periodic expiration and recycling. This effectively ensures eventual consistency of throttling state. Redis's centralized management ensures a unified global view, while the streaming encapsulation mechanism automatically handles both normal and abnormal exits. A periodic background cleanup task reclaims unreleased zombie requests. These three elements work together to ensure the system maintains effective throttling and high fault tolerance, even in complex network environments and with unstable nodes.

[0085] Furthermore, the following is a typical application process in an embodiment of the present invention:

[0086] In this embodiment of the present invention, the flow of the current limiting and management mechanism involved in the RAG system when a user accesses an application URL is as follows:

[0087] Step 1: User access request and application identification:

[0088] When a user accesses an application URL supported by the RAG system through a browser, API client, or other means, the system backend receives the HTTP request and parses the corresponding app_id (e.g., from the URL path, header, or token). This app_id is the application's unique identifier in the system and is used for subsequent rate limiting and resource scheduling.

[0089] Step 2: Request unique identifier generation and access registration attempt:

[0090] To ensure that each request is traceable and manageable, the system generates a unique request_id (e.g., a UUID) for the current request and prepares to register it in the distributed cache. During this phase, the system uses Redis as a centralized state store, constructs a throttling key based on the app_id, and prepares to perform a concurrency check.

[0091] Step 3: Concurrency check and atomic admission control:

[0092] The system queries two key values ​​through Redis:

[0093] rag:rate_limit:{app_id}:max_active_requests: Get the maximum number of concurrent requests for the application;

[0094] rag:rate_limit:{app_id}:active_requests: Gets the number of members in the current active request hash table, that is, the current concurrency number.

[0095] The system then compares the current number of concurrent requests with the maximum concurrency limit. If the limit is within the limit, a predefined Redis Lua script is called to atomically write the request_id and timestamp to the active_requests hash table, completing the admission registration. If the limit is exceeded, the request is directly rejected and an error message such as HTTP 429 Too Many Requests is returned to ensure system stability.

[0096] Step 4: Request processing and streaming output encapsulation:

[0097] For requests that are allowed in, the system begins processing specific RAG tasks (such as knowledge retrieval and text generation). If the task type is streaming output (such as conversational interaction and content generation), the system uses a "streaming generator encapsulation" mechanism to encapsulate the actual generation logic. This encapsulation mechanism not only outputs results step by step but also automatically monitors state changes during the generation process, ensuring that subsequent cleanup actions are triggered upon successful completion or abnormal interruption.

[0098] Step 5: Request completion and resource release:

[0099] Depending on whether the task is streaming output, the system adopts different resource release strategies:

[0100] For streaming tasks, after the generator iteration is completed or an exception occurs, the encapsulation module automatically captures the event and calls the cleanup interface to delete the corresponding request_id record from Redis.

[0101] For non-streaming tasks, the business logic needs to explicitly call the release_request(app_id, request_id) function after processing is completed to manually complete the request exit operation.

[0102] This design takes into account the flexibility and automation level in different scenarios, and improves the robustness and ease of use of the system.

[0103] Step 6: Backend expired request recovery and fault tolerance processing:

[0104] To address "zombie requests" that may arise from node downtime, network anomalies, or program crashes, the system configures a standalone background task to periodically scan all active_requests hash tables in Redis. For each record, the system checks whether its timestamp exceeds a preset maximum lifetime (e.g., 10 minutes). If a timed-out request is found, it is immediately removed from the hash table using the HDEL command, freeing up the concurrency slot it occupied. This mechanism ensures eventual consistency of the throttling state and enhances the system's fault tolerance in complex environments.

[0105] Furthermore, in order to clearly illustrate the implementation path of the present invention, the key implementation steps of the present invention are described below in combination with the actual deployment architecture and core function design.

[0106] Step S11: Application creation and concurrency threshold configuration.

[0107] When you create a new RAG application through the admin panel, you'll be asked to set the maximum concurrent requests for the application (max_active_requests). This value controls the number of concurrent requests. The system persists this configuration item in the database and immediately writes it to the Redis cache after successful creation. The corresponding key is: rag:rate_limit:{app_id}:max_active_requests.

[0108] The system supports subsequent dynamic updates of this configuration and maintains synchronization between the cache and the database through a background mechanism.

[0109] Step S12: Request reception and current limiting determination.

[0110] When an application is accessed (i.e., the corresponding URL is triggered), the service node receives the request and extracts the app_id from the request parameters. The system then generates a unique identifier for the current request, request_id (e.g., using a UUID).

[0111] To ensure consistency in multi-node processing states in high-concurrency scenarios, the system uses Lua script atomic operations provided by Redis to complete the following operations:

[0112] Get the current active request set of the application in Redis: rag:rate_limit:{app_id}:active_requests;

[0113] Get the current concurrency limit value max_active_requests;

[0114] Determine whether the current number of active requests has reached the limit;

[0115] If the upper limit is not reached, the request_id is written into the hash table together with the current timestamp;

[0116] If the limit has been reached, atomically return a rejection.

[0117] This logic is all completed at the Redis layer to avoid state competition or inconsistency issues under concurrent operations.

[0118] Step S13: Streaming task encapsulation and automatic release.

[0119] RAG applications typically use streaming generation, gradually delivering output content to front-end users in the form of data streams. To automatically release request status, this paper uses a wrapper mechanism written in Python to encapsulate the streaming generation function. The main logic includes:

[0120] The wrapper object is an SSE generator, which is a Python generator that supports Server-Sent Events;

[0121] The wrapper registers the request status before the task starts;

[0122] When the generator completes normally or throws an exception, the state is automatically released and the request_id is removed from Redis.

[0123] This mechanism does not require the caller to explicitly handle the release logic, which improves development convenience and reduces the risk of misuse.

[0124] Step S14: Request status release mechanism.

[0125] The present invention supports two types of request release methods:

[0126] 1. Automatic release: As mentioned above, for streaming tasks, the wrapper automatically handles the logic of releasing the request_id after the task ends;

[0127] 2. Manual release: For non-streaming requests or requests that cannot be automatically packaged, the caller can complete the release operation by calling a standard function (such as release_request(app_id, request_id)).

[0128] The release operation is essentially a Redis HDEL command that removes the specified request ID from the active_requests hash table.

[0129] Step S15: Timed clearing mechanism for expired requests.

[0130] To prevent "zombie occupation" caused by abnormal request exit or network interruption, the system has designed a background scheduled cleanup task that runs once every minute. The task functions include:

[0131] 1. Iterate over all keys in Redis that match the pattern rag:rate_limit:*:active_requests;

[0132] 2. For each requested entry in the hash table, read its storage timestamp;

[0133] 3. If the timestamp is greater than the set maximum lifetime threshold (10 minutes) from the current time, the current request is considered expired;

[0134] 4. The system removes it through the HDEL instruction, freeing up the concurrent quota.

[0135] This mechanism is implemented at the Redis layer to ensure the consistency of state and the timeliness of resource recovery in the distributed system.

[0136] Step S16: Support for consistency and dynamic regulation in a distributed environment.

[0137] This invention uses Redis as the central state storage node, and all service instances share this state. With Redis's atomic instructions and consistency guarantees, the system supports:

[0138] Any node registers and releases the request;

[0139] Any node performs cleanup tasks;

[0140] Administrators adjust the concurrency threshold at runtime and it takes effect in real time.

[0141] The above mechanism realizes a highly consistent, highly available, and highly flexible current limiting and resource management system under distributed deployment.

[0142] The proposed RAG application-oriented streaming generation request flow limiting and automatic recycling mechanism has the following significant benefits compared to the existing technology:

[0143] Highly adaptable and precise control: This mechanism directly manages the number of active concurrent requests, rather than simply limiting request frequency. This makes it more suitable for the streaming and long-running tasks common in RAG applications. By setting independent concurrency limits for each application (app_id), refined resource allocation and isolation are achieved.

[0144] Automated resource release reduces the operational burden: For streaming tasks, generator encapsulation enables automatic tracking of the request lifecycle and resource release. Whether a task completes normally or terminates abnormally for any reason (such as network outages or internal errors), occupied concurrency slots are promptly returned, eliminating the need for complex exception handling and manual intervention by application developers or operations personnel.

[0145] Effectively prevent resource leaks: Combining active release (either upon normal termination or exception) and passive recycling (periodic timeout cleanup) mechanisms, this system reliably handles the "zombie request" problem. Even in extreme situations like node disconnection or process crashes in a distributed environment, expired request records are eventually removed by background cleanup tasks, preventing permanent concurrent resource leaks and ensuring long-term stable system operation.

[0146] Strong consistency in distributed environments: Using a centralized distributed cache (such as Redis) as state storage, all service nodes share consistent concurrent state information and configuration. Leveraging Redis's atomic operations or Lua scripts ensures the accuracy and consistency of state updates in high-concurrency scenarios, making it ideal for large-scale distributed deployments.

[0147] Dynamic Adjustment and Flexibility: The maximum concurrent request limit (max_active_requests) is stored in Redis, allowing administrators to dynamically adjust it while the system is running, without restarting the service. This provides great flexibility for responding to sudden traffic spikes, managing service capacity, or conducting A / B testing.

[0148] Improved system stability and user experience: By effectively controlling concurrency, we avoid server crashes or performance degradation caused by request overload, ensuring overall service stability and availability. Furthermore, we provide clear prompts for requests exceeding the limit, guiding users to retry, providing a better user experience than unresponsiveness or service anomalies.

[0149] Furthermore, the key points of the present invention are:

[0150] Combining concurrency control and automatic recycling for long-running streaming tasks: The core is to combine the concurrency limit with the automatic resource release mechanism (encapsulator) and periodic expiration cleanup mechanism for RAG streaming task characteristics to form a complete closed-loop management solution.

[0151] State management based on distributed cache: Use the hash table structure (active_requests) of distributed cache (such as Redis) to store active request IDs and corresponding timestamps to achieve concurrent state sharing, tracking, and atomic updates across nodes.

[0152] Dual-request cleanup mechanism: This includes both active and immediate release based on task lifecycle events (completion / exception) and periodic passive recycling based on timestamp timeouts, ensuring that resources can be effectively recycled in various situations (including exceptions).

[0153] Dynamic configuration and distributed consistency: The maximum concurrency limit can be dynamically adjusted at runtime by modifying the value in the cache, and the adjustment can be perceived and applied by all distributed nodes in near real time, while ensuring state consistency.

[0154] According to an embodiment of the present invention, a method for managing concurrent requests of a target application maintains the active request status of each application in a distributed cache (Redis), dynamically limits the number of simultaneous requests based on configuration, and adapts to the common characteristics of RAG generation tasks, such as streaming output, long processing time, and prone to abnormal interruptions, thereby ensuring service stability and efficient resource utilization.

[0155] In order to implement the above embodiment, Figure 3 As shown, this embodiment also provides a system 10 for managing concurrent requests of a target application, including:

[0156] The configuration storage module 100 is used to store the unique application identifier app_id of each target application and the corresponding maximum concurrent request limit value max_active_requests;

[0157] The distributed cache module 200 is connected to the configuration storage module and is used to store the max_active_requests value and the active request set active_requests of each target application, where the active_requests set includes multiple request identifiers request_id and corresponding timestamps;

[0158] The request admission module 300 is connected to the distributed cache module and is used to query the current number of requests in the active_requests collection based on the app_id when responding to a user request, and decide whether to allow the request to enter based on the comparison result between the current number of requests and max_active_requests;

[0159] The request processing module 400 is connected to the request admission module and is used to process the requests that are allowed to enter. When the request processing is completed or an exception occurs, the resource release module removes the corresponding request_id from the active_requests collection;

[0160] The periodic cleaning module 500 is connected to the distributed cache module and is used to scan the active_requests collection according to a predetermined period, and remove the request_id when it is detected that the difference between the timestamp of a request_id and the current time exceeds the preset maximum survival time to prevent zombie requests from occupying concurrent resources for a long time.

[0161] Specifically, the system for managing concurrent requests of a target application includes the following steps:

[0162] a. Associate a unique application identifier (app_id) for the target application;

[0163] b. Store a maximum concurrent request limit (max_active_requests) and a set of active requests (active_requests) for the app_id in a distributed cache accessible to all service nodes.

[0164] c. When a new request for the target application is received: i. Check the current number of requests in the active_requests set; ii. If the current number of requests is less than the max_active_requests value (or if the value is unlimited), generate a unique request identifier (request_id) for the new request, add the request_id and the current timestamp to the active_requests set, and then process the current request; iii. If the current number of requests reaches or exceeds the max_active_requests value, reject the new request;

[0165] d. Associate a resource release mechanism for the request processing, which is configured to remove the corresponding request_id from the active_requests collection when the request processing is completed or an exception occurs;

[0166] e. Periodically execute a cleanup task that scans the active_requests collection and removes request_ids whose timestamps exceed the predetermined maximum survival time.

[0167] The operations of checking the current number of requests and adding request_id in step c are accomplished by executing an atomic operation or script on the distributed cache.

[0168] When the processing of the request involves streaming generation of output, the resource release mechanism is implemented by a wrapper that wraps the streaming generation process and automatically triggers the removal of the request_id when the generation process ends normally or encounters any exception.

[0169] The distributed cache is Redis, and the active_requests collection is a hash data structure in Redis, where the key is request_id and the value is the corresponding timestamp.

[0170] The maximum concurrent limit value (max_active_requests) can be modified while the system is running, and subsequent request checks will use the updated value.

[0171] According to an embodiment of the present invention, a system for managing concurrent requests of target applications maintains the active request status of each application in a distributed cache (Redis), dynamically limits the number of simultaneous requests based on configuration, and adapts to the common characteristics of RAG generation tasks, such as streaming output, long processing time, and prone to abnormal interruptions, thereby ensuring service stability and efficient resource utilization.

[0172] Further, if Figure 4 As shown, the present invention also provides an electronic device, comprising:

[0173] processor;

[0174] a memory storing a computer program;

[0175] When the processor executes the computer program, it implements the method for managing concurrent requests of the target application as described in the embodiment of the first aspect.

[0176] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0177] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Claims

1. A method for managing concurrent requests of a target application, characterized in that: The following steps are involved: In response to a user's access request to a target application, obtaining an application identifier app_id of the target application; In the distributed cache, a maximum concurrency limit value max_active_requests and a collection active_requests for recording active requests are maintained for app_id. The active_requests collection stores multiple request identifiers request_id and corresponding timestamps. When a new request is received, the current number of requests in the active_requests set is queried based on the app_id and compared with max_active_requests; if the current number of requests is less than max_active_requests, a request_id is generated for the new request, and the request_id and corresponding timestamp are added to the active_requests set to allow the current request to be processed; if the current number of requests reaches or exceeds max_active_requests, the new request is rejected; Configure a resource release mechanism for the request processing process. When the request processing is completed or an exception occurs, the resource release mechanism automatically removes the corresponding request_id from the active_requests collection; Periodically execute the cleanup task, which scans the timestamps of all request_ids in the active_requests collection and removes the request_id when the difference between the current time and the timestamp exceeds the preset maximum survival time to prevent zombie requests from occupying concurrent resources for a long time.

2. The method according to claim 1, characterized in that The operation of querying the current number of requests in the active_requests collection based on app_id and comparing it with max_active_requests is completed by executing atomic operations or scripts on the distributed cache.

3. The method according to claim 1, characterized in that The resource release mechanism includes a streaming task wrapper, which is used to automatically trigger the removal of request_id when the generation process ends normally or encounters any exception when the request processing involves streaming generation output.

4. The method according to claim 1, wherein The distributed cache is Redis, and the active_requests collection is a hash data structure in Redis, where the key is request_id and the value is the corresponding timestamp.

5. The method according to claim 1, wherein The maximum concurrent limit value max_active_requests is modified while the system is running, and subsequent request checks use the updated value.

6. A system for managing concurrent requests of a target application, characterized in that: include: Configure the storage module to store the application identifier app_id and the corresponding maximum concurrent request limit max_active_requests for each target application; The distributed cache module is connected to the configuration storage module and is used to store the max_active_requests value and the active request set active_requests of each target application. The active_requests set contains multiple request identifiers request_id and corresponding timestamps; The request admission module is connected to the distributed cache module. When responding to user requests, it queries the current number of requests in the active_requests collection based on the app_id and decides whether to allow the request to enter based on the comparison result between the current number of requests and max_active_requests. The request processing module is connected to the request admission module to process the requests that are allowed in. When the request processing is completed or an exception occurs, the corresponding request_id is removed from the active_requests collection through the resource release module; The periodic cleaning module is connected to the distributed cache module to scan the active_requests collection at a predetermined period and remove the request_id when it is detected that the difference between the timestamp of a request_id and the current time exceeds the preset maximum survival time to prevent zombie requests from occupying concurrent resources for a long time.

7. The system according to claim 6, characterized in that The request processing module includes a streaming task wrapper that automates resource release when request processing involves streaming generated output.

8. The system according to claim 6, wherein: The distributed cache module is Redis, and the active_requests collection is a hash data structure in Redis, where the key is request_id and the value is the corresponding timestamp.

9. The system according to claim 6, wherein: The configuration storage module supports dynamic updating of the max_active_requests value while the system is running, and synchronizes the updated value to the distributed cache module.

10. An electronic device, characterized in that: include: processor; a memory storing a computer program; When the processor executes the computer program, the method for managing concurrent requests of a target application as claimed in any one of claims 1 to 5 is implemented.

Images