Multi-tenant agent-oriented gateway dynamic routing and load balancing method and system

By adopting a tenant-independent caching architecture and dynamic routing strategies, the load skew problem of multi-tenant agent clusters is solved, achieving efficient load balancing and stability, adapting to large-scale multi-tenant concurrent access, and improving the scalability and maintainability of the system.

CN123123253APending Publication Date: 2026-11-17BEIJING QINGSONG YIKANG INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611297403.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies cannot balance the instantaneous and long-term load of multi-tenant agent clusters, resulting in cluster load skew and resource waste, and failing to meet the requirements for large-scale stable operation.

Method used

It adopts a tenant-independent dedicated caching architecture, replaces database queries with memory cache retrieval, and combines routing strategies and health status verification to achieve stable reuse of tenant sessions and dynamic node selection, reduce the loss of repeated loading, and adapt to automatic switching of node failures.

Benefits of technology

It improves the security and stability of gateway routing data in multi-tenant scenarios, reduces system IO pressure and computational overhead, shortens single-request routing latency, improves overall throughput and concurrent processing capabilities, and achieves efficient traffic scheduling and stability of multi-tenant agent clusters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123253A_ABST
    Figure CN123123253A_ABST
Patent Text Reader

Abstract

This disclosure presents a method and system for dynamic gateway routing and load balancing for multi-tenant agents. The method includes: obtaining a user identifier corresponding to a received user request; querying the system cache to obtain query information; if a match is found, obtaining the node identifier of the matched gateway node and its corresponding health status based on the node identifier; if the health status is healthy, invoking a first strategy to obtain the target node corresponding to the user identifier; if the health status is unhealthy or the query information is not found, invoking a second strategy to obtain the target node corresponding to the user identifier; and forwarding the received user request to the Agent instance on the target node for processing. This method, when used with multi-tenant agents, enables dynamic gateway routing and dynamic load balancing, balancing both instantaneous and long-term load balancing, and meeting the requirements for large-scale stable operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a gateway dynamic routing and load balancing method and system for multi-tenant agents. Background Technology

[0002] With the widespread adoption of AI Agent services driven by large language models, multi-tenant isolated Agent workspaces for massive numbers of tenants have become a mainstream business requirement. The industry generally adopts a clustered deployment architecture, consisting of multiple gateway nodes forming an Agent execution node cluster. Each gateway node runs an Agent runtime process, and a single process can simultaneously host Agent instances for multiple tenants. External user requests are distributed through the routing layer to support large-scale concurrent Agent services. Each Agent instance maintains a complete set of state data, including session context, tenant identity credentials, installed skills, and personalized configurations. Instance creation involves numerous serialization and deserialization operations, resulting in high creation overhead, and creation latency increases with the number of Agent instances already hosted on the node. This imposes a special constraint on the state locality of the routing distribution mechanism, rendering traditional stateless load balancing solutions unsuitable for the operational characteristics of AI Agent clusters.

[0003] Most existing routing load balancing solutions are designed for stateless services. Directly applying them to stateful multi-tenant agent clusters exposes multiple drawbacks. Specifically, the round-robin and least-connection distribution strategies of ordinary reverse proxies are completely unaware of the actual deployment location of tenant agent instances. Requests are randomly distributed to different gateway nodes, causing the same tenant to repeatedly create agent instances on multiple nodes, significantly increasing system latency and resource overhead. Consistent hashing can bind user identifiers to nodes and ensure state locality, but this mechanism can only achieve uniform hash distribution at the user ID level. It cannot perceive the actual instantaneous on-transit request load of a node, nor can it calculate the long-term accumulated number of tenants bound to a node. Dynamic scheduling strategies based solely on the instantaneous number of on-transit requests are prone to multi-node load balancing during periods of low traffic. The selection results in such scenarios are solidified over time through tenant and node binding relationships, accumulating as business operations continue. Ultimately, this leads to a large number of tenants being concentrated on at least a few nodes, causing cluster load skew or even single-point overload. Summary of the Invention

[0004] In view of this, the present disclosure provides a gateway dynamic routing and load balancing method and system for multi-tenant agents, which can solve the problems of existing technologies being unable to balance instantaneous and long-term load balancing, resulting in cluster load tilt and failure to meet the requirements of large-scale stable operation.

[0005] In a first aspect, embodiments of this disclosure provide a gateway dynamic routing and load balancing method for multi-tenant agents, employing the following technical solution: Obtain the user identifier corresponding to the received user request; The system cache is queried based on the user identifier to obtain query information; the system cache stores several dedicated caches for storing information associated with authorized users, and each dedicated cache corresponds to an authorized user; If the query information is a match, obtain the node identifier of the matched gateway node and obtain the health status corresponding to the matched gateway node based on the node identifier; If the health status is healthy, the first strategy is invoked to obtain the target node corresponding to the user identifier; If the health status is unhealthy or the query information is not found, the second strategy is invoked to obtain the target node corresponding to the user identifier; The received user request is forwarded to the Agent instance on the target node for processing.

[0006] Secondly, this disclosure also provides a gateway dynamic routing and load balancing system for multi-tenant agents, employing the following technical solution: The client is used to initiate user requests; A caching layer is used to store system cache, which contains several dedicated caches for storing information associated with authorized users, with each dedicated cache corresponding to an authorized user. The routing layer is used to obtain the user identifier corresponding to the received user request, query the system cache according to the user identifier to obtain query information, and when the query information is a hit, obtain the node identifier of the hit gateway node and obtain the health status corresponding to the hit gateway node according to the node identifier. The load balancing layer is used to obtain the target node corresponding to the user identifier by invoking a first strategy when the health status is healthy, and to obtain the target node corresponding to the user identifier by invoking a second strategy when the health status is unhealthy or the query information is a miss, and to forward the received user request to the Agent instance on the target node for processing.

[0007] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution: The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor; the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-described gateway dynamic routing and load balancing methods for multi-tenant agents.

[0008] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions; the computer instructions are used to cause a computer to execute any of the above-described gateway dynamic routing and load balancing methods for multi-tenant agents.

[0009] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0010] The gateway dynamic routing and load balancing method for multi-tenant agents provided in this disclosure uses a tenant-independent dedicated caching architecture to achieve partitioned storage and independent operation and maintenance of tenant routing mapping data, eliminating mutual interference between multi-tenant data and improving the security and stability of gateway routing data in multi-tenant scenarios, thus adapting to business scenarios with large-scale multi-tenant concurrent access. It replaces traditional database queries with memory cache retrieval, reducing system IO pressure and computational overhead during the routing decision-making stage, shortening single-request routing latency, and improving the overall throughput and concurrent processing capabilities of the gateway, adapting to the scheduling requirements of high-concurrency, low-latency agent services. It achieves stable reuse of tenant sessions through routing strategies, reducing the waste caused by repeated loading of agent instance resources. It enables automatic node failure switching through dynamic point selection strategies, overcoming the shortcomings of static routing strategies in adapting to dynamic node failures, achieving a balance between stability and fault tolerance, automatically implementing multi-tenant traffic scheduling logic, reducing agent business coupling and development and maintenance costs, achieving standardized and centralized management of multi-tenant agent cluster traffic, and improving the overall system's scalability and maintainability.

[0011] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the gateway dynamic routing and load balancing method for multi-tenant agents provided in this embodiment of the disclosure.

[0014] Figure 2 This is a flowchart illustrating a method for obtaining a target node corresponding to a user identifier by invoking a first strategy, as provided in an embodiment of this disclosure.

[0015] Figure 3 This is a flowchart illustrating a method for obtaining a target node corresponding to a user identifier by invoking a second strategy, as provided in an embodiment of this disclosure.

[0016] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0017] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0018] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0020] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0022] Reference Figure 1 This application discloses a gateway dynamic routing and load balancing method for multi-tenant agents, including: S100, obtain the user identifier corresponding to the received user request.

[0023] Specifically, after receiving a user's session request, the routing layer parses out the user identifier corresponding to the current user request, which enables accurate identification of different tenant agent requests, eliminates cross-tenant routing crosstalk from the request entry point, and realizes standardized and refined access control of multi-tenant traffic.

[0024] S200 queries the system cache based on the user identifier to obtain query information.

[0025] The system cache contains several dedicated caches for storing information associated with authorized users, with each dedicated cache corresponding to one authorized user.

[0026] S300, if the query information is a hit, obtain the node identifier of the hit gateway node and obtain the health status corresponding to the hit gateway node based on the node identifier; S400, if the health status is healthy, invoke the first strategy to obtain the target node corresponding to the user identifier; S500: If the health status is unhealthy or the query information is not found, the second strategy is invoked to obtain the target node corresponding to the user identifier.

[0027] S600 forwards the received user request to the Agent instance on the target node for processing.

[0028] The gateway dynamic routing and load balancing method for multi-tenant agents disclosed in this application is used to stably route requests from the same user to the node where their Agent instance resides. By completely decoupling the routing hit logic and the concurrency limiting logic, it breaks the coupled architecture of traditional caching that simultaneously undertakes routing addressing and traffic control. By setting up independent routing cache slots with time-to-live, it is specifically designed to achieve fast hits of tenant routing information. Relying on short-term caching characteristics to maintain tenant routing, it stably reuses the Agent's local session state, reduces the performance loss caused by repeated instance creation, and effectively compresses request forwarding latency. At the same time, it deploys independent node in-transit counters to verify the real-time concurrency limit of nodes, accurately manages the instantaneous load on nodes, and completely avoids problems such as traffic jitter and request overload caused by cache expiration, routing policies and rate limiting rules conflicts in traditional architectures. It achieves independent operation of efficient routing addressing and secure traffic control, eliminates the risk of node overload while ensuring service state locality and routing efficiency, and significantly improves the stability and controllability of cluster traffic scheduling.

[0029] For S200, the process includes: constructing a system cache based on authorized user association information. This system cache contains several dedicated caches for storing authorized user association information, each corresponding to an authorized user. Then, the system cache is queried based on the user identifier to obtain the query information. The user association information includes the user identifier and the cache validity period.

[0030] Specifically, the dedicated cache is a route cache slot, and each route cache slot corresponds to a time-to-live (TTL). This slot is only used to accelerate route hits and does not assume any responsibility for concurrency limiting. That is, the system cache allocates a dedicated cache record for each online user to store the gateway identifier corresponding to the user. The cache validity period is synchronized with the user login session. Its sole purpose is to reduce route allocation calculations and speed up traffic forwarding. This cache is purely for query acceleration and does not carry any control logic such as user concurrency statistics, rate limiting, or resource exclusivity.

[0031] In this step, a tenant-specific independent caching architecture is adopted, configuring an independent dedicated cache partition for each authorized tenant. Each tenant's cached data is physically isolated and maintained independently. Compared to a globally shared cache architecture, this completely avoids cross-contamination and entry overwriting issues in multi-tenant caches. Expiration cleanup, data updates, and abnormal reset operations of a single-tenant cache will not affect the cache services of other tenants. Simultaneously, relying on in-memory cache retrieval to replace database disk queries significantly reduces the IO overhead of the routing decision link.

[0032] Furthermore, the application also includes maintaining an ordered set of routing statistics for each node in the caching layer, with members being user identifiers and scores being expiration timestamps, for statistical purposes and expiration cleanup.

[0033] Furthermore, the application also includes: maintaining an in-transit counter for each node in the caching layer, including an in-transit counter at the gateway node level and an in-transit marker at the user level. The user-level in-transit marker is configured with a safety fallback time-to-live (TTL) to prevent count leakage caused by abnormal node exit. That is, it records which gateway node a user's current request belongs to and sets a timeout for this record as a fallback protection. Even if a gateway node crashes abnormally or the program exits abnormally and the system does not have time to actively delete this record, the user's in-transit status marker will automatically expire and be cleared when the timeout expires, preventing the request count from being abnormally occupied and unable to be released, and preventing leakage of concurrency quota.

[0034] All system caches are used as temporary runtime states, and their associated information comes from the metadata database. The metadata database maintains user tables, tenant tables, and gateway node registry. The tenant table records the permanent binding relationship between tenants and gateway nodes through the gateway node identifier field. A value of 0 represents an unbound state. The gateway node registry records the node access address, token, health status, and concurrent capacity limit.

[0035] When the system cache expires, the routing logic reads the metadata database from the source to complete the route selection and writes the route selection result back to the corresponding special cache. After the request is processed, the in-transit status cache is updated. When the session expires, the cache entries are cleared by the TTL and the background mechanism. In the case of node abnormality, the fallback TTL of the in-transit mark automatically releases the concurrent resources occupied by the abnormality.

[0036] Reference Figure 2 For S300, the methods for obtaining the health status of the matched gateway node include: S310, if the user identifier is found in the system cache, obtain the node identifier of the found gateway node.

[0037] Specifically, it receives the service request sent by the user, reads the route cache slot with the user ID as the key, and if it exists, it means that the request has been hit. Then it obtains the node identifier of the hit gateway node.

[0038] S320 identifies the corresponding status field from a pre-built metadata database based on the node identifier.

[0039] The metadata database includes the URLs, access tokens, health status, running status, and concurrency limits of each gateway node.

[0040] S330, retrieve the availability status flag and the running status flag from the status field.

[0041] The availability status flag and the running status flag are periodically probed by the background maintenance process and then uniformly written into the metadata database. This method does not initiate real-time field probes to the gateway nodes during the execution process.

[0042] S340 obtains the health status of the hit gateway node based on the availability status flag and the running status flag.

[0043] Specifically, when the availability status is marked as true and the running status is marked as enabled, the health status of the matched gateway node is determined to be healthy. When the availability status is not marked as true and the running status is marked as enabled, the health status of the matched gateway node is determined to be unhealthy.

[0044] In this system, gateway node health data is pre-stored in a metadata database. The main business process no longer probes each gateway node individually at the user request stage, eliminating time-consuming operations such as network probing, timeout waiting, and connection handshakes. This significantly reduces the time spent on a single request in scenarios with high concurrency gateway queries, avoiding interface timeouts and congestion caused by numerous real-time probes. The health probe logic is decentralized to an offline backend process, with the main business process only responsible for looking up the status in a table; the two are completely isolated. Subsequent adjustments to the probe cycle, health judgment rules, or the addition of new node status dimensions only require modifications to the backend maintenance process, without altering the online business query logic, resulting in lower iteration risk and lower operational costs.

[0045] In this embodiment, after a cache retrieval hit, the health indicators reported in real time by the gateway cluster are retrieved based on the node identifier to complete the node availability verification. This can solve the technical defect of inconsistency between static cache mapping data and the dynamic operating status of gateway nodes, effectively filter out faulty, overloaded, and offline node failure mapping entries left in the cache, and avoid request forwarding failure and Agent instance connection abnormalities caused by dirty cache data.

[0046] Furthermore, the dedicated backend maintenance process M1 periodically probes all gateways and writes them to the metadata database in batches, instead of probing each gateway independently for each business request. The gateway device only needs to withstand one set of periodic probe traffic, and in scenarios with massive business concurrency, it will not cause high-frequency repeated inspection impact on the gateway node, thus protecting the gateway's computing power and network bandwidth.

[0047] Reference Figure 3 For S400, the method for obtaining the target node corresponding to the user identifier by invoking the first strategy includes: S410, when the availability status is marked as true and the running status is marked as enabled, determine that the health status of the hit gateway node is healthy.

[0048] In this step, two conditions must be met simultaneously for the gateway to be considered healthy. An availability status of true indicates that the node itself is not faulty; an operational status of enabled indicates that the service is allowed to receive traffic. Both conditions are indispensable; if either is not met, the gateway will be discarded and no requests will be allocated to it.

[0049] S420: Get the number of requests in transit corresponding to the hit gateway node.

[0050] This step is used to read the number of pending requests that the candidate gateway node currently has, which is the total number of requests that the gateway is currently processing but has not yet finished.

[0051] S430 increments the number of requests in transit to obtain the expected number of requests.

[0052] Specifically, before the request actually begins execution, the number of in-transit requests corresponding to the hit gateway node is incremented by 1 to obtain the expected number of requests. That is, before the request is sent to the gateway for execution, the count is incremented by 1 to calculate the number of requests that will be reached after accepting this request, which is called the expected number of requests. The purpose of this step is to reserve the quota in advance, to reserve the position of this request in advance, and to prevent multiple requests from being judged at the same time in a concurrent scenario, which would result in exceeding the limit.

[0053] S440: If the expected number of requests does not exceed the concurrency limit, the hit gateway node will be used as the target node.

[0054] Furthermore, if the expected number of requests exceeds the limit, it means that the gateway's concurrency has been fully utilized and it will no longer forward requests directly. You can choose to reject the request directly or put the request into a queue and process it when the gateway has free time.

[0055] This embodiment completely decouples the number of routing cache slots from the concurrency limit. The slots only reflect which machine the most recent route was to, and the hard concurrency limit is only enforced by the number of routes in transit.

[0056] Furthermore, synchronously performing cache renewal means extending the expiration time of this cache, refreshing and renewing the effective lifecycle of the currently hit cache slot, retaining valid cache data, and ensuring a fast response capability for subsequent requests of the same type. This renewal action and the return result action are completed together in this request process, without starting a separate asynchronous background task to delay the renewal, and without waiting for the next request to arrive before updating the expiration time.

[0057] The processing logic disclosed in this embodiment is a system-preset low-overhead priority execution hot path, which is the mainstream processing branch for business requests. This branch only completes data query, status verification, and cache renewal operations based on local memory cache, which can effectively avoid high-time-consuming and high-resource-consuming processes such as network interaction, remote data retrieval, and full node verification, significantly reducing the computing power consumption and response latency of request processing, and effectively improving the overall throughput performance of service discovery and business response efficiency.

[0058] This embodiment enables targeted distribution of requests from the same tenant, ensuring the reuse of node information such as tenant agent session context, permission configuration, and running status. It eliminates the need to repeatedly load tenant configurations and rebuild session links across nodes, significantly reducing the overhead of repeated computation and resource consumption of agent instances, and effectively ensuring the session continuity and operational stability of multi-tenant agent services.

[0059] Reference Figure 4 The method for S500 to call the second strategy to obtain the target node corresponding to the user identifier includes: S510: If the health status is unhealthy or the query information is not found, obtain the binding information corresponding to the user identifier.

[0060] Specifically, the binding information corresponding to the user identifier is obtained from the tenant table of the pre-built metadata database. The binding information includes the bound nodes and the health status of the corresponding nodes. In the tenant table, the node identifier records the node to which the corresponding user is permanently bound. A value of 0 indicates that the user is not bound, meaning that the system has not yet assigned a node to them.

[0061] S520 analyzes the binding information. If the binding information meets the preset conditions, the bound node is recorded as the target node and stored in the system cache.

[0062] If the binding information meets the preset conditions, it means that the user who previously made the request has bound a certain node and the bound node is healthy; recording the bound node as the target node and storing it in the system cache means that it is backfilled into the routing cache slot and used as the target node, thus accelerating subsequent execution.

[0063] S530: If the binding information does not meet the preset conditions, the most idle healthy node is identified as the target node according to the load analysis strategy.

[0064] If the binding information does not meet the preset conditions, it means that the user who initiated the request has not bound a node before, or has bound a node before but the bound node is unhealthy.

[0065] Furthermore, the method for identifying the most idle healthy node as the target node based on the load analysis strategy specifically includes: S531, selects all healthy nodes as candidate nodes from the metadata database.

[0066] S532, construct the load triplet corresponding to each candidate node.

[0067] The triple contains, in order, the number of requests in transit for the node, the number of bound tenants, and the node identifier.

[0068] S533 performs a lexicographical comparison on the load triples and selects the node corresponding to the triple with the smallest lexicographical order as the target node.

[0069] The lexicographical comparison follows a rule of judging from left to right, prioritizing the comparison of the first field of the triplet, namely the number of requests in transit. This indicator reflects the instantaneous load status of the node and is the most timely signal for identifying instantaneous overload. Nodes with a smaller number of requests in transit have a better overall load triplet. As long as there is a difference in the value of this field, the comparison result can be determined directly without reading subsequent fields. This allows new tenants or new requests to be prioritized and scheduled to the currently more idle nodes.

[0070] Only when the number of requests in transit on multiple nodes is exactly equal will the comparison of the second field, the number of bound tenants, be entered. This field is used to address the cumulative load imbalance caused by long-term binding in low-traffic, flat-out scenarios. Even if the instantaneous request pressure on nodes is similar, the number of tenants bound to different nodes in the past may vary greatly. By prioritizing the selection of nodes with a smaller number of bound tenants, the accumulation of tenants on some nodes can be suppressed from the source of allocation, and the load imbalance caused by long-term operation can be gradually corrected.

[0071] If both the number of requests in transit and the number of bound tenants are the same, the third field, node identifier, is used to make the final decision. The smaller value is selected according to the lexicographical order of the node identifier. This field is only used as a fallback condition and will not interfere with the regular load scheduling logic. Its purpose is to ensure that the scheduling result is certain and reproducible when the indicators of multiple nodes are completely overlapping, and to avoid scheduling jitter caused by random selection.

[0072] In this embodiment, the entire process does not use the weighted summation of indicators to calculate the comprehensive score, strictly maintains the priority order of keywords, with instantaneous load having the highest priority, followed by tenant number correction, and node identifiers are only used to eliminate uncertainty in the results, taking into account the three aspects of instantaneous overload protection, long-term cumulative tilt management, and reproducibility of scheduling behavior.

[0073] In this embodiment, after selection, the node corresponding to the triple with the smallest lexicographical order is updated to the target node; if the user has never established an Agent instance on the target node before, a call to create the user's Agent instance is sent to the target node; and the target node is backfilled into the routing cache slot.

[0074] In this embodiment, the second strategy dynamically selects the optimal target node based on the available node resource pool of the gateway cluster. It can be compatible with two types of core business scenarios. The first is the disaster recovery scenario when the bound node of the existing tenant fails, which breaks the routing constraints of the invalid cache and realizes the automatic removal of the faulty node and traffic migration. The second is the cold start scenario when the new tenant connects for the first time. When there is no historical cached data, the node can be automatically allocated, which completely avoids the problems of request blocking and service unavailability caused by cache failure and no cached data, and improves the fault tolerance capability of the system access.

[0075] Furthermore, the application also includes: background tasks can perform health checks on each node at fixed intervals, such as 30 seconds; to avoid frequent changes in binding caused by detection jitter, hysteresis judgment is preferred, for example, it is marked as unhealthy only after N consecutive failures, such as 3, and it is restored to health only after M consecutive successes.

[0076] In this embodiment, upon receiving a user request, the routing cache slot is first read. If a match is found and the corresponding node is healthy, the node is directly returned and the slot is renewed. This is a hot path with minimal overhead. If the cache is not found, the user's tenant record is queried in the database. If a bound node exists and is healthy, the binding is used, the routing cache slot is filled, and the request is returned. If there is no binding, or the bound node is unhealthy / unavailable, the process of selecting the least busy node is initiated. After obtaining the target node, the binding is written to / updated in the database (if it is a fault migration, the old binding is overwritten), the routing cache slot is filled, and the request is returned. During the routing phase, only routing is performed, and the concurrency limit is not checked.

[0077] In this application, the metadata of each Agent instance on each node is persisted in an incremental and independent recording manner. This means that independent file records can be generated for each Agent, or stored in an embedded database or key-value database on the local node, rather than writing all Agents into a single large configuration file.

[0078] During the Agent lifecycle operations, only one corresponding metadata record is appended to the storage medium when an Agent is created. When listing Agent instances, each record is read sequentially using the storage index, eliminating the need for full serialization and deserialization of the complete configuration. During runtime configuration hot updates, the system only applies incremental changes to the actual changed records. Noop changes that do not produce substantial changes are skipped, preventing a full reload of the configuration in unchanged scenarios. This addresses the issue of Agent creation and listing operations increasing linearly with the number of instances, avoids related IO and parsing operations blocking the runtime event loop, and improves system stability and response performance in multi-Agent scenarios.

[0079] To adapt to request scheduling requirements across all scenarios, this solution constructs a three-tiered progressive routing system: route cache hit, permanent database binding, and selection of the least busy node. This achieves a comprehensive balance between service performance, operational stability, and global cluster balance. Specifically, the first-level route cache prioritizes in-memory millisecond-level routing, continuously maintaining the session binding relationship between tenants and nodes, maximizing the preservation of Agent instance state locality, and avoiding resource consumption and increased latency caused by frequent instance initialization.

[0080] When the cache expires, becomes invalid, or is abnormal, the tenant's routing affiliation is restored by relying on the persistent data permanently bound to the second-level database, preventing session interruptions caused by random route jumps and ensuring long-term stable operation of the service.

[0081] For special scenarios such as node failure, cluster load imbalance, and cold start of new tenants, the third-level most idle node selection mechanism is automatically triggered. It breaks away from the constraints of fixed route binding and dynamically selects the best available node in the cluster. The three-level routing mechanism is layered and complementary, which not only ensures high-speed forwarding and session continuity of normal business, but also realizes adaptive dynamic scheduling capabilities in abnormal scenarios.

[0082] For S600, concurrent admission is performed only after the target node is determined. Specifically, the routing layer forwards the received user requests to the Agent instance on the target node to perform business processing. The workspace file data used by the Agent is stored in the shared OSS object storage. Regardless of which compute node in the cluster can access and read this workspace data, the workspace is not bound to a specific node.

[0083] Once the Agent completes processing the request, regardless of whether the request is executed normally, encounters an error, or times out, after the instance is reclaimed through the pre-defined safety fallback TTL timeout mechanism, the node currently processing the request needs to decrement its local in-transit request count. The in-transit request count is used to track the number of active Agent requests currently being processed on the node. The count increments with each incoming request and decrements with each completed request, whether it ends normally or times out and is reclaimed. This count helps determine the node's current load for subsequent routing, overload assessment, and instance resource management. Even if a request crashes due to an error, the TTL fallback mechanism ensures the count will decrease, preventing the counter from only increasing and causing scheduling distortion.

[0084] This solution targets multi-node shared AI Agent execution clusters. Under the core premise of strictly ensuring the locality of user and Agent instance states, avoiding duplicate instance creation, and reducing request processing latency, it effectively solves core technical pain points such as high-speed routing requests, dynamic and static load balancing, automatic fault migration, and low-cost cluster rebalancing through decoupling of routing and concurrency management, a three-level tiered routing mechanism, triplet multi-dimensional load scoring, and a shared object storage architecture. It can support the long-term autonomous and balanced operation of the cluster without any manual operation and maintenance intervention.

[0085] Secondly, this application discloses a gateway dynamic routing and load balancing system for multi-tenant agents, used to execute the gateway dynamic routing and load balancing method for multi-tenant agents disclosed in the first aspect of this application, including: The client is used to initiate user requests; The cache layer is used to store system cache. The system cache contains several dedicated caches for storing information associated with authorized users. Each dedicated cache corresponds to one authorized user. The routing layer is used to obtain the user identifier corresponding to the received user request, query the system cache based on the user identifier to obtain query information, and when the query information is a match, obtain the node identifier of the matched gateway node and obtain the health status corresponding to the matched gateway node based on the node identifier. The load balancing layer is used to obtain the target node corresponding to the user identifier by invoking the first strategy when the health status is healthy, and to obtain the target node corresponding to the user identifier by invoking the second strategy when the health status is unhealthy or the query information is not found, and forward the received user request to the Agent instance on the target node for processing.

[0086] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0087] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the gateway dynamic routing and load balancing method for multi-tenant agents described in the foregoing embodiments of this disclosure.

[0088] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0089] like Figure 4 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 4 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0090] like Figure 4 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0091] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 4 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0092] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from ROM. When the computer program is executed by a processor, all or part of the steps of the gateway dynamic routing and load balancing method for multi-tenant agents according to embodiments of this disclosure are performed.

[0093] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0094] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the gateway dynamic routing and load balancing methods for multi-tenant agents described in the foregoing embodiments of the present disclosure are performed.

[0095] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0096] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0097] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0098] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0099] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0100] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0101] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0102] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0103] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A gateway dynamic routing and load balancing method for multi-tenant agents, characterized in that, include: Obtain the user identifier corresponding to the received user request; The system cache is queried based on the user identifier to obtain query information; The system cache contains several dedicated caches for storing information associated with authorized users, and each dedicated cache corresponds to one authorized user. If the query information is a match, obtain the node identifier of the matched gateway node and obtain the health status corresponding to the matched gateway node based on the node identifier; If the health status is healthy, the first strategy is invoked to obtain the target node corresponding to the user identifier; If the health status is unhealthy or the query information is not found, the second strategy is invoked to obtain the target node corresponding to the user identifier; The received user request is forwarded to the Agent instance on the target node for processing.

2. The gateway dynamic routing and load balancing method for multi-tenant agents according to claim 1, characterized in that, If the query information is a match, the node identifier of the matched gateway node is obtained, and the health status corresponding to the matched gateway node is obtained based on the node identifier, including: If the user identifier is found in the system cache, obtain the node identifier of the found gateway node; Identify the corresponding status field from a pre-built metadata database based on the node identifier; Obtain the availability status flag and the running status flag from the status field; The health status of the matched gateway node is obtained based on the availability status flag and the running status flag.

3. The gateway dynamic routing and load balancing method for multi-tenant agents according to claim 2, characterized in that, The availability status flag and the running status flag are periodically probed by the background maintenance process and then uniformly written into the metadata database.

4. The gateway dynamic routing and load balancing method for multi-tenant agents according to claim 2, characterized in that, If the health status is healthy, the first strategy is invoked to obtain the target node corresponding to the user identifier, including: When the availability status is marked as true and the running status is marked as enabled, the health status of the matched gateway node is determined to be healthy. Get the number of in-transit requests corresponding to the hit gateway node; The number of requests in transit is incremented to obtain the expected number of requests; If the expected number of requests is not greater than the concurrency limit, the hit gateway node will be used as the target node.

5. The gateway dynamic routing and load balancing method for multi-tenant agents according to claim 2, characterized in that, If the health status is unhealthy or the query information is not found, the second strategy is invoked to obtain the target node corresponding to the user identifier, including: If the health status is unhealthy or the query information is not found, obtain the binding information corresponding to the user identifier. The binding information includes the bound nodes and the corresponding node health status. The binding information is analyzed. If the binding information meets the preset conditions, the bound node is recorded as the target node and stored in the system cache. If the binding information does not meet the preset conditions, the most idle healthy node is identified as the target node according to the load analysis strategy.

6. The gateway dynamic routing and load balancing method for multi-tenant agents according to claim 5, characterized in that, If the binding information does not meet the preset conditions, the least idle healthy node is identified as the target node according to the load analysis strategy, including: All healthy nodes are selected as candidate nodes from the metadata database; Construct a load triplet for each of the candidate nodes; Perform a lexicographical comparison on the load triples and select the node corresponding to the triple with the smallest lexicographical order as the target node.

7. A gateway dynamic routing and load balancing system for multi-tenant agents, characterized in that, include: The client is used to initiate user requests; A caching layer is used to store system cache, which contains several dedicated caches for storing information associated with authorized users, with each dedicated cache corresponding to an authorized user. The routing layer is used to obtain the user identifier corresponding to the received user request, query the system cache according to the user identifier to obtain query information, and when the query information is a hit, obtain the node identifier of the hit gateway node and obtain the health status corresponding to the hit gateway node according to the node identifier. The load balancing layer is used to obtain the target node corresponding to the user identifier by invoking a first strategy when the health status is healthy, and to obtain the target node corresponding to the user identifier by invoking a second strategy when the health status is unhealthy or the query information is a miss, and to forward the received user request to the Agent instance on the target node for processing.

8. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor; the instructions are executed by the at least one processor to enable the at least one processor to perform the gateway dynamic routing and load balancing method for multi-tenant agents as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions; the computer instructions are used to cause the computer to perform the gateway dynamic routing and load balancing method for multi-tenant agents as described in any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-6.