A Routing Group Access Authorization and Scheduling Control Method Based on AI Gateway

Abstract

This invention relates to the field of communication technology and discloses a routing group access authorization and scheduling control method based on an AI gateway, applied to a gateway server. The method includes: establishing a unified entry routing group and a single URL address; creating sub-routes associated with model service instances; and establishing a binding relationship between API keys and authorized groups; receiving access requests pointing to the single URL; extracting the API key to be verified and request feature information; identifying the target authorized group and obtaining the interface access range based on the binding relationship; matching the target sub-routes and verifying whether they are within the authorized range; if the verification passes, executing a three-layer scheduling strategy including sub-route scheduling, priority scheduling, and weighted scheduling to determine the final target model service instance and forward the request. This invention achieves unified convergence and fine-grained permission isolation for heterogeneous AI model interfaces, effectively improving resource utilization and system security in large-scale model access scenarios.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a routing group access authorization and scheduling control method based on an AI gateway. Background Technology

[0002] With the explosive growth of artificial intelligence technology, large language models, multimodal models, and various specialized models are being deployed on a large scale within enterprises. To unify the management of heterogeneous model services and reduce business access costs, Model-as-a-Service (MaaS) gateways have become a key component of enterprise-level AI infrastructure. Currently, existing technologies are MaaS gateway systems built on cloud-native gateways (such as Higress), typically implementing functions such as API Key-based authentication, weighted load balancing, and basic monitoring and statistics.

[0003] However, in large-scale multi-model access scenarios, existing technologies face significant system coupling and management challenges: First, they lack interface convergence and protocol adaptation capabilities. Different model interface addresses are scattered and protocol standards are inconsistent, resulting in extremely high adaptation costs for business systems and making it difficult to quickly access large models or agents with non-standard interfaces. Second, they lack access control and lifecycle management, supporting only coarse key and interface binding, failing to achieve fine-grained access control based on groups, and making it difficult to dynamically migrate permissions when user roles change, posing serious security risks.

[0004] In addition, existing gateways also face severe challenges in terms of resource scheduling and data security: their scheduling strategies are simplistic, supporting only static weighted load balancing, and they cannot perceive the differences in resource consumption caused by long and short contexts in AI business, resulting in low GPU resource utilization and unstable system response latency; at the same time, data security protection is weak, call logs are generally stored in plaintext and the collection strategy is inflexible, which can neither effectively protect user privacy nor meet the stringent data security compliance requirements. Summary of the Invention

[0005] This invention provides a routing group access authorization and scheduling control method based on an AI gateway. By converging heterogeneous model interfaces through a single URL, combined with a fine-grained authorization grouping mechanism and a three-layer intelligent scheduling engine, it significantly reduces the access and maintenance costs of business systems, and significantly improves the utilization rate of model computing resources and system response speed while ensuring data security and compliance.

[0006] This invention provides a routing group access authorization and scheduling control method based on an AI gateway, applied to a gateway server. The method includes: S1. Establish a unified entry route group and configure it with a single external URL address. Under the unified entry route group, create multiple sub-routes that are associated with corresponding model service instances. Create API keys and authorization groups. Configure an interface access range consisting of at least one sub-route for each authorization group. Establish a binding relationship between the API key and the authorization group. Preset a scheduling strategy. S2. Receive an access request sent by the client pointing to the single URL address; wherein the access request carries an API key to be verified and request feature information; S3. Extract the API key to be verified, identify the target authorization group to which the API key to be verified belongs according to the binding relationship, and obtain the interface access range corresponding to the target authorization group. At the same time, match the target sub-route corresponding to the access request based on the request feature information, and verify whether the target sub-route is included within the interface access range. S4. If the target sub-route is included within the access range of the interface, the permission verification is confirmed to be successful. According to the preset scheduling policy, the target model service instance to be used to process the access request is determined from the model service instances associated with the target sub-route, and the access request is forwarded to the target model service instance.

[0007] Furthermore, S1 specifically includes: S11. Deploy a Kubernetes cluster, along with a single Redis server for storing route cache, rate limiting counters, and high-frequency access configurations; an Elasticsearch cluster for storing encrypted logs and monitoring statistics; and a Kafka cluster for asynchronously transmitting encrypted logs and statistical metrics. Create an independent URL route for the existing model service as a single route entry point for compatibility. S12. Establish a unified entry route group in the gateway control plane and configure a unified entry address for external access. Create multiple sub-routes corresponding to different AI model services under the unified entry route group, set the target service address and request protocol of the sub-routes, and configure matching rules for each sub-routes, including URL path matching, Header key-value matching, Body JSON field parsing matching, and request context length numerical matching, as well as set the corresponding traffic weight and priority. S13. Create a new authorization group and define the interface access range consisting of at least one sub-route according to the category or tag. Create an API key and select an authentication mode including Bearer mode, query parameter mode or custom Header mode. Assign the API key to the corresponding authorization group to establish the binding relationship between the API key and the authorization group and the interface access range. S14. Preset a three-layer scheduling strategy and clearly define the order in which the strategy for determining the target model service instance takes effect: sub-route scheduling takes precedence over priority scheduling, and priority scheduling takes precedence over weight scheduling.

[0008] Furthermore, when creating multiple sub-routes corresponding to different AI model services in step S12, a model category identifier field is added to identify large models and intelligent agents. In step S4, when forwarding the access request to the target model service instance, the OpenAI API specification protocol is uniformly adopted externally. For non-standard interfaces that do not support the specification, a dynamically loaded protocol conversion plugin is used to convert the data format of the non-standard interface to the OpenAI API specification protocol.

[0009] Furthermore, after establishing the binding relationship between the API key and the authorized group and interface access scope in step S13, the method also includes performing lifecycle management operations on the API key. The lifecycle management operations include editing operations to modify the group to which the API key belongs, deletion operations when it is confirmed that it is not associated with any authorized group, temporary disabling and re-enabling operations, and unbinding operations. It also supports automatically and dynamically assigning the API key to the corresponding authorized group based on the user's role change to achieve dynamic grouping.

[0010] Furthermore, S2 specifically includes: S21. The gateway receives an access request sent by the client pointing to a preset unified entry URL address; S22. Extract the API key to be verified from the specified location of the access request according to the preset authentication mode; wherein, the preset authentication mode includes the Bearer mode of storing the API key to be verified in the Authorization Header of the request, the query parameter mode of storing it in the URL query parameters, or the custom Header mode of storing it in the preset custom Header. S23. Parse the URL, Header, and Body information of the access request to extract the request feature information, including the URL path, Header key value, Body JSON field, and context length value. S24. Generate a globally unique link ID for the access request using the UUID algorithm, and add the link ID to the request header to pass it to the downstream service.

[0011] Furthermore, S3 specifically includes: S31. After extracting the API key to be verified, the gateway verifies the validity of the API key to be verified. Specifically, it checks whether the key exists in the system and whether the key is enabled. If the authentication fails due to not carrying the key or the key not having permission, it returns a 403 Unauthorized error. S32. If the API key to be verified is valid, the target authorization group to which the API key to be verified belongs is identified according to the established binding relationship, and the interface access range corresponding to the target authorization group is obtained. At the same time, the preset plugin restriction policy information associated with the API key to be verified is queried to check whether the target authorization group meets the corresponding plugin restrictions. If the target authorization group does not meet the corresponding plugin restrictions, resulting in the policy verification failing, a 403 Forbidden error is returned. S33. If the strategy verification passes, then based on the extracted request feature information, the sub-route matching rules are used to attempt to match the sub-route, thereby matching the target sub-route corresponding to the access request; wherein, the matching order of the sub-route matching rules is as follows: URL path matching, Body JSON field parsing matching, Header key-value matching, and request context length numerical matching. S34. Verify whether the target sub-route obtained by matching is included within the interface access range corresponding to the target authorized group.

[0012] Furthermore, in step S33, when attempting to match a sub-route based on the extracted request feature information according to the configured sub-route matching rules, the request feature information also includes the source IP address and User-Agent header of the request. This allows for IP address matching via the source IP address or user agent matching via the User-Agent header. Additionally, it supports a feature information conversion plugin that replaces the authentication key in the user header with model authentication information for specific sub-routes, and a plugin rule that includes IP blacklists and whitelists for the unified entry route group.

[0013] Furthermore, S4 specifically includes: S41. If the matched target sub-route is within the interface access range of the authorized group, the permission verification is confirmed to be successful. First, try to match the sub-route. If the sub-route is matched, the access request is directly forwarded to the target model service instance and no further scheduling strategy is executed. S42. If a sub-route is not hit, the priority scheduling policy is executed. The corresponding priority scheduling policy is matched based on the Header information, Body content information or context length value carried in the request. The request is forwarded to an available instance according to the route availability feedback from the monitoring status. If the priority scheduling policy values ​​configured in multiple sub-routes are the same, the weight scheduling phase is entered. S43. When a request fails to hit a sub-route and priority scheduling policy or encounters multiple priority scheduling policy values ​​with the same value, the weighted scheduling policy is executed. The traffic distribution ratio between each instance is calculated based on the weight value configured for each service instance, and the access request is distributed and forwarded to the corresponding target model service instance according to the calculated traffic distribution ratio. S44. When the availability of the routing state causes a conflict between the weighted scheduling strategy and the system priority scheduling strategy, the corresponding abnormal routes are avoided and blocked through business logic to ensure the correct sub-route scheduling strategy is implemented. S45. During the process of forwarding the request to the downstream service, the generated and globally unique link ID is added to the request header and passed to the downstream service. Each target model service instance generates the corresponding span ID and records the parent span ID to form a link tree structure when processing the access request. The link span information is recorded in the log for encrypted storage.

[0014] Furthermore, the method also includes a log encryption processing step, specifically: S5. Under the premise that sensitive information is not collected by default, when the detailed call data retention switch is enabled, the basic information of the access request is collected and the request body and response body corresponding to the access request are encrypted using the AES-256 symmetric encryption algorithm. The unencrypted basic information and the encrypted request body and response body are written to the Elasticsearch cluster, and the encrypted log data is sent to the Kafka message queue for downstream systems to call the decryption API interface for consumption.

[0015] The present invention also provides a routing group access authorization and scheduling control system based on an AI gateway, which is deployed on the gateway server. Based on the routing group access authorization and scheduling control method based on the AI ​​gateway described above, the system includes a configuration initialization unit, a request receiving unit, a permission verification unit, and a scheduling and forwarding unit. The configuration initialization unit is used to establish a unified entry route group and configure a single URL address for it to be exposed to the outside world. Under the unified entry route group, multiple sub-routes are created, each associated with a corresponding model service instance. API keys and authorization groups are created. For each authorization group, an interface access range consisting of at least one sub-route is configured. The binding relationship between the API key and the authorization group is established. A scheduling strategy is preset. The request receiving unit is used to receive an access request sent by the client pointing to the single URL address, wherein the access request carries an API key to be verified and request feature information; The permission verification unit is used to extract the API key to be verified, identify the target authorization group to which the API key to be verified belongs according to the binding relationship, obtain the interface access range corresponding to the target authorization group, and at the same time, match the target sub-route corresponding to the access request based on the request feature information, and verify whether the target sub-route is included within the interface access range. The scheduling and forwarding unit is used to confirm that the permission verification is passed when the target sub-route is included within the access range of the interface, determine the target model service instance to be used to process the access request from the model service instances associated with the target sub-route according to the preset scheduling policy, and forward the access request to the target model service instance.

[0016] The beneficial effects of this invention are as follows: 1. Unified entry point, significantly reducing access and maintenance costs. This invention converges dispersed, heterogeneous AI model interfaces into a single URL address. Business systems only need to maintain one interface address, without needing to concern themselves with the specific deployment of the underlying model services. When the model service changes (such as version upgrades or address migrations), only the configuration on the gateway needs to be modified, without synchronously modifying the code and configuration of all callers, significantly reducing the access cost and maintenance complexity of business systems. Simultaneously, it is compatible with existing multi-URL binding schemes, enabling a smooth transition for business systems.

[0017] 2. Fine-grained authorization for enhanced system security. This invention employs an authorization grouping mechanism to achieve fine-grained permission isolation, supporting the principle of least privilege. Administrators can create different authorization groups for different business lines and users, with each group only able to access a specified range of interfaces. Even if an API key is leaked, attackers can only access the interfaces of the group to which that key belongs, and cannot access other interfaces, effectively reducing security risks. Furthermore, it supports API key lifecycle management and dynamic visibility scope, further enhancing the flexibility and security of access control.

[0018] 3. Intelligent scheduling significantly improves resource utilization and response speed. This invention constructs a three-layer intelligent scheduling engine consisting of independent sub-routes, priorities, and weights. It can handle requests on demand, dynamically scheduling traffic based on individual requests, system priorities, and service weights. It also adds route status judgment, shielding abnormal routes with poor status. By allocating requests to different routes, it avoids significant resource waste, significantly improving model resource utilization and overall system response speed. Furthermore, it clearly defines the priority order of scheduling strategies, avoiding multi-strategy conflicts.

[0019] 4. Encrypted storage meets compliance requirements and protects user privacy. This invention uses the AES-256 algorithm to encrypt and store the request and response bodies in user call logs. Sensitive information is not collected by default; encryption is implemented when collection is enabled. The encryption key is stored locally, and access permissions are strictly controlled. Only authorized administrators can decrypt and view the logs. The log storage period is 180 days, and logs are automatically deleted after expiration, meeting compliance requirements and effectively protecting user privacy and data security.

[0020] 5. Scalable architecture for rapid business evolution. This invention adopts a plug-in architecture and a unified protocol standard, supporting unified management and access for multiple model types. By expanding the model category identifier field and developing a general protocol conversion plugin, new model types and model providers can be quickly integrated without modifying the gateway's core code, significantly reducing the cost of integrating new models. Simultaneously, it supports custom plugin development and integration, flexibly expanding gateway functionality to adapt to the continuous evolution of enterprise MaaS platforms.

[0021] 6. High concurrency and high availability, supporting large-scale business scenarios. This invention is built on a cloud-native architecture and deployed using Kubernetes containers, supporting automatic service scaling and failover. A single node supports 100,000 QPS, and a cluster deployment supports tens of millions of QPS, meeting the needs of high-concurrency AI call scenarios. The overall system availability is ≥99.95%, and the annual downtime is ≤4.38 hours, providing stable and reliable support for enterprise-level large-scale AI applications. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the routing group access authorization and scheduling control method based on an AI gateway according to the present invention.

[0023] Figure 2 This is a flowchart illustrating the overall process of the present invention.

[0024] Figure 3 This is a schematic diagram of the routing group access authorization and scheduling control system based on the AI ​​gateway of the present invention.

[0025] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] In the description of the embodiments of the present invention, in order to enable those skilled in the art to more clearly understand the technical solution, the core terms involved in the text are defined as follows: AI Gateway: Refers to the unified access layer deployed between business clients and backend AI model services, which is used to undertake functions such as protocol conversion, identity authentication, traffic scheduling and security auditing.

[0028] Unified entry routing group: refers to a logical container built at the gateway level. It exposes a single URL access address to the outside world and internally achieves the aggregation of multiple heterogeneous model services through a sub-routing mechanism.

[0029] API Key: A credential string used to identify and authenticate the caller, supporting multiple authentication modes such as Bearer, query parameters, or custom headers.

[0030] Authorization Groups: These are logical sets defined within the gateway, used to bind specific API keys to preset sub-route access ranges, enabling fine-grained role-based access control.

[0031] Sub-route: refers to a secondary routing unit under a unified entry route group. Each sub-route corresponds to one or more specific model service instances and supports independent matching rules and scheduling configurations.

[0032] Model service instance: refers to the listening address and port of the actual running large language model, intelligent agent or various algorithm model on the server.

[0033] Scheduling strategy: refers to the logical rules that the gateway follows when forwarding requests, including but not limited to scheduling based on sub-route matching, scheduling based on service priority, and load balancing scheduling based on traffic weight.

[0034] In view of the above-mentioned problems existing in the prior art, the present invention aims to solve the following technical problems: 1. Solve the problem of scattered multi-model interfaces and difficulty in unified access: Converge heterogeneous AI model interfaces into a single URL address to achieve unified access and reduce the access and maintenance costs of business systems.

[0035] 2. Address the issues of loose access control and high security risks: Implement fine-grained key management to achieve the goals of group-based authorization, group-based control, and key-based restrictions. Based on the principle of minimizing permissions, it supports precise control down to the route and key level, thereby improving system security.

[0036] 3. Address the issues of simplistic scheduling strategies and low resource utilization: Construct a multi-dimensional intelligent scheduling engine that supports combined scheduling based on context length, system priority, and weights, as well as intelligent scheduling based on multi-dimensional information such as request body, header feature information, and model health status. This improves model resource utilization, reduces system response speed, and lowers model usage costs.

[0037] 4. Address the issues of plaintext log storage and insufficient data security: Implement encrypted storage and access control for user access logs, data channel encryption, record-by-record authentication, unified management, and other capabilities to meet compliance requirements and protect user privacy and data security.

[0038] 5. Solve the problems of difficult model expansion and high cost of adapting to non-standard interfaces: Build an extensible model class management architecture, add a compatibility layer, unify external protocol standards, and reduce the cost of accessing new models and statistical requirements.

[0039] 6. Meet the access and management requirements of a unified interface for intelligent agents and solve the dynamic adaptation of non-standard interfaces: Build intelligent agent access management capabilities, add a conversion layer, unify external protocol standards, support dynamic parameter response and on-demand customized collection strategies, reduce the access cost of intelligent agents and reduce business adaptation.

[0040] like Figure 1 , 2 As shown, this invention provides a routing group access authorization and scheduling control method based on an AI gateway, applied to a gateway server. The method includes: S1. Establish a unified entry route group and configure it with a single external URL address. Under the unified entry route group, create multiple sub-routes that are associated with corresponding model service instances. Create API keys and authorization groups. Configure an interface access range consisting of at least one sub-route for each authorization group and establish a binding relationship between the API key and the authorization group.

[0041] In a preferred embodiment of the present invention, step S1 specifically includes the following sub-steps: S11. Deploy the gateway's basic environment and configure compatibility.

[0042] Deploy a Kubernetes cluster that supports dynamic gateway routing and sub-routing, along with a single-machine Redis cache server for routing, an Elasticsearch cluster for storing encrypted logs, and a Kafka message queue for asynchronous data transmission. Meanwhile, to ensure a smooth system transition, create independent URL routes for existing model services as a single-route entry point for compatibility.

[0043] S12. Create a unified entry route group and associate it with the configured sub-routes.

[0044] Establish a unified entry route group in the gateway control plane, configure a unified public URL address (such as / v1 / chat / completions) and enable the sub-route function; under the unified entry route group, create a corresponding sub-route for each model service, set the target service address and request protocol to associate with the underlying model service instance; further, configure matching rules for each sub-route, including URL path matching, Header key-value matching, Body JSON field parsing matching and request context length numerical matching, and set the corresponding traffic weight and priority.

[0045] In the S12 process described above (creating sub-routes and associating model service instances), to support unified management of multiple model types, the system adds model category identifier fields (such as type and model_type) to identify large models and intelligent agents when registering interfaces for model services. Supported preset types include, but are not limited to: large language models, code generation models, vector embedding, reordering, graph-to-text, text-to-graph, and other large models, as well as various AI intelligent agent models.

[0046] S13. Configure the authorization group and API key and establish a binding relationship.

[0047] Create new authorization groups in the gateway's authorization management module, and define the interface access scope of each group by category or label, consisting of at least one sub-route; at the same time, create API keys that support Bearer mode, query parameter mode or custom Header mode, and assign the API keys to the corresponding authorization groups, so that the API keys are bound to the authorization groups and the interface access scope.

[0048] Following S13 (establishing the binding relationship between the API key and the authorization group), the system further provides complete lifecycle management operations for the API key. Specifically, these management operations include: Editing operations: Supports modifying the key's name, description, tags, plugin information, and group (but the key itself and authentication mode cannot be modified).

[0049] Deletion operation: Before execution, the system must confirm that the key is not associated with any authorized group. After deletion, the key becomes invalid immediately and cannot be recovered.

[0050] Disable and enable operations: Supports temporarily disabling the key so that it cannot call any interface, and re-enabling it when needed.

[0051] Dynamic group control: The system supports automatically assigning the API key to the corresponding authorization group based on the user's role change to achieve dynamic grouping (for example, when the user role changes from "tester" to "developer", the system automatically completes the migration of the permission scope), and supports remote unauthorization binding.

[0052] S14. Preset multi-layer scheduling control strategy. Build and preset a three-layer scheduling strategy in the gateway, and clearly define the order in which the strategy for determining the target model service instance takes effect: sub-route scheduling takes precedence over priority scheduling, and priority scheduling takes precedence over weighted scheduling, thereby establishing the underlying rules for subsequent dynamic traffic allocation.

[0053] As an optional implementation of scheduling policy priority, the priority order of the preset three-layer scheduling policy can be dynamically adjusted according to business needs in practical applications. For example, in scenarios with strict routing rule requirements, the policy for determining the target model service instance can be set to take effect in the following order: sub-route scheduling takes precedence over weighted scheduling, and weighted scheduling takes precedence over priority scheduling. This adjustment ensures that sub-route rules are executed first, weighted allocation second, and system priority scheduling is executed last, thus flexibly adapting to the complex routing requirements of different businesses.

[0054] S2. Receive an access request sent by the client pointing to the single URL address; wherein the access request carries an API key to be verified and request feature information.

[0055] In a preferred embodiment of the present invention, step S2 specifically includes the following sub-steps: S21. The gateway receives API access requests sent by users through clients. These requests point to a pre-defined unified entry URL address in the gateway control plane, such as / v1 / chat / completions.

[0056] S22. The gateway retrieves the API key to be verified from the specified location in the access request according to the preset authentication mode. The authentication modes support the following three: Bearer mode: The API key to be verified is stored in the Authorization Header of the request.

[0057] Query parameter mode: The API key to be verified is stored in the URL query parameters, in a format such as "?api_key=".

[0058] Custom Header Mode: The API key to be verified is stored in a preset custom header, such as "X-API-Key:".

[0059] S23. Parse the URL, Header, and Body of the access request to extract request feature information for subsequent route matching and scheduling. The request feature information includes, but is not limited to: URL path: Used to match a specific service path.

[0060] Header key-value pairs: Match the target service by specifying key-value pairs in the request header.

[0061] Body JSON fields: Match the target service by parsing the specified JSON fields in the request body (such as the model_id field parsed using jsonparse.model_id).

[0062] Context length value: The context length value of the request is obtained by parsing the length of the request body content.

[0063] S24. Generate a globally unique Trace ID for this access request to enable full-link tracing and problem localization. The Trace ID is generated using the UUID algorithm and added to the request header to be passed to downstream services. The Trace ID header must use English names and values. Additionally, during subsequent processing, the gateway will record information such as the start time of the span, service name, and interface name.

[0064] S3. Extract the API key to be verified, identify the target authorization group to which the API key to be verified belongs according to the binding relationship, and obtain the interface access range corresponding to the target authorization group. At the same time, match the target sub-route corresponding to the access request based on the request feature information, and verify whether the target sub-route is included within the interface access range.

[0065] In a preferred embodiment of the present invention, step S3 specifically includes the following sub-steps: S31. Verify the validity of the API key.

[0066] After retrieving the API key to be verified, the gateway verifies the validity of the API key, specifically by checking whether the key exists in the system and whether the key is enabled. If the key is not carried or the key lacks permissions, resulting in authentication failure, a 403 Unauthorized error is returned.

[0067] S32, Obtain grouping and policy restrictions.

[0068] If the API key to be verified is valid, the target authorization group to which it belongs is identified according to the pre-established binding relationship, and the interface access scope corresponding to the target authorization group is obtained; at the same time, the plugin restriction policy information to which the key belongs is queried, and the target authorization group is checked to see if it meets the corresponding plugin restrictions. If the policy verification fails, a 403 Forbidden error is returned.

[0069] S33, Match the target sub-route.

[0070] If the strategy verification passes, then based on the extracted request feature information, the sub-routes are matched sequentially according to the configured sub-route matching rules to find the target sub-route corresponding to the access request; wherein, the sub-route matching order is as follows: URL path matching > Body JSON field parsing matching > Header key-value matching > Request context length numerical matching.

[0071] In the above S33 (matching target sub-routes based on request feature information) process, in addition to the regular URL, Header, Body and context length matching, the extracted request feature information can further include the source IP address of the request and the User-Agent header.

[0072] Matching expansion: The system can then perform IP address matching by source IP address (e.g., distinguishing between office network IP and public network IP for different environments), or perform user agent matching by User-Agent header (e.g., forwarding mobile requests to the lightweight model and PC requests to the high-performance model).

[0073] Plugin rule application: During the matching phase, the system supports applying feature information transformation plugins to specific sub-routes (for example, dynamically replacing the authentication key in the user header with the actual authentication information of the underlying model); at the same time, it supports applying general plugin rules, including IP blacklists and whitelists, to the entire unified entry route group.

[0074] S34. Execution scope and permission verification.

[0075] Verify whether the target sub-route obtained by the matching is included within the interface access range corresponding to the obtained target authorization group.

[0076] S4. If the target sub-route is included within the access range of the interface, the permission verification is confirmed to be successful. According to the preset scheduling policy, the target model service instance to be used to process the access request is determined from the model service instances associated with the target sub-route, and the access request is forwarded to the target model service instance.

[0077] In a preferred embodiment of the present invention, step S4 specifically includes the following sub-steps: S41, Sub-route scheduling execution.

[0078] If the matched target sub-route is within the interface access scope of the authorized group, the permission verification is confirmed to be successful. According to the preset three-layer scheduling strategy including sub-route, priority, and weight, the system first attempts to match the sub-route. If a sub-route is matched, the access request is directly forwarded to the target model service instance, and no further scheduling strategy is executed.

[0079] S42, Priority scheduling execution.

[0080] If no sub-route is matched, a priority scheduling strategy is executed. The system matches the corresponding priority scheduling strategy based on the header information, body content information, or context length value carried in the request, prioritizing high-priority business requests (such as production environment requests) or forwarding them to routes that support response feature information. During this process, requests are forwarded to available instances based on route availability feedback from monitoring status; if a route is unavailable, scheduling fails, and a normal route call is performed. If multiple sub-routes have the same priority scheduling strategy value, the weighted scheduling phase is entered.

[0081] S43, Weighted scheduling execution.

[0082] When a request fails to hit a sub-route or priority scheduling policy, or when multiple priority scheduling policy values ​​are identical, a weighted scheduling policy is executed. The system calculates the traffic distribution ratio among instances based on a weight value (range 1-100) configured for each service instance. According to the calculated traffic distribution ratio, access requests are proportionally distributed and forwarded to the corresponding target model service instance.

[0083] S44, Strategy Conflict Handling.

[0084] During scheduling execution, when the availability of route status causes a conflict between the weighted scheduling strategy and the system priority scheduling strategy, the system avoids this conflict through business logic and blocks the corresponding abnormal routes to ensure that the correct sub-route scheduling strategy is implemented.

[0085] S45, End-to-end tracking and recording.

[0086] During the request forwarding to downstream services, a generated and globally unique trace ID is added to the request header and passed to the downstream service. Each target model service instance, when processing the access request, generates a corresponding span ID and records its parent span ID, thus forming a complete trace tree structure. The system records the start time, end time, service name, interface name, request information, response information, and error information of each span, and logs this trace information. Finally, it is encrypted and stored in an Elasticsearch cluster to support querying the entire request processing flow by trace ID.

[0087] In the aforementioned S4 process (forwarding access requests to the target model service instance), to reduce the access cost of heterogeneous models, the gateway uniformly adopts the OpenAI API specification protocol when forwarding. For non-standard interfaces that do not support this specification at the underlying level (such as some privately deployed large models or agents), the system uses a dynamically loaded protocol conversion plugin at the forwarding node to convert the data format of the non-standard interface into the OpenAI API specification protocol in real time before communication.

[0088] After completing the request forwarding action in S4, the system proceeds to the log processing step S5. This is to meet data security compliance requirements. By default, the gateway does not collect sensitive information.

[0089] When the detailed call data retention switch is enabled, the system collects basic information about the access request (request time, status code, token consumption, etc.) and uses a locally generated 256-bit key, combined with the AES-256 symmetric encryption algorithm, to perform high-strength encryption on the request body and response body of the request.

[0090] Subsequently, the system writes the unencrypted basic information along with the encrypted request / response body to the Elasticsearch cluster, while simultaneously sending the encrypted complete log data to the Kafka message queue.

[0091] Downstream systems (such as security auditing and data warehouses) need to call the decryption API interface provided by the gateway through identity authentication in order to consume and decrypt encrypted data in Kafka.

[0092] Furthermore, to achieve lifecycle management and long-term compliant storage of logs, the system configures the storage period of the log data (e.g., 180 days) and automatically cleans up expired log data. At the same time, if long-term storage is required, the system supports archiving the encrypted logs to external object storage (e.g., Ceph cluster). The archived log data still needs to be decrypted through the decryption API interface when viewed.

[0093] As an optional implementation of the aforementioned log encryption storage module, in addition to the default AES-256 algorithm, this embodiment also supports other encryption algorithms: for example, for scenarios with national cryptographic requirements, the SM4 national cryptographic algorithm can be used instead of the AES-256 algorithm to encrypt the request body and response body; for scenarios with extremely high security requirements, a combination of RSA asymmetric encryption algorithm and AES symmetric encryption algorithm can be used, that is, the AES key is encrypted with the RSA algorithm in advance, and then the log data is encrypted with the encrypted AES key. Thus, even if the AES key file is leaked, the log data cannot be decrypted without the RSA private key, thereby greatly improving the data security of the system.

[0094] like Figure 3 As shown, this embodiment of the invention also provides a routing group access authorization and scheduling control system based on an AI gateway. The system is deployed on the gateway server and serves as the unified access layer for AI model services, undertaking core functions such as traffic forwarding and access control. This system corresponds one-to-one with the aforementioned method embodiments, achieving unified access and secure scheduling of multiple models through the collaborative operation of various functional units. The system specifically includes: The configuration initialization unit is responsible for presetting and building rules for the gateway's underlying resources. Specifically, this unit is responsible for creating a unified entry route group on the Higress gateway control plane and configuring the publicly exposed URL address of this route group to a single entry point, such as / v1 / chat / completions. This unit enables the sub-routing function, creating corresponding sub-routes for each model service. Simultaneously, this unit performs the creation and binding of authorization groups and API keys, supporting the automatic generation of secure random string keys or custom keys, and linking the keys to authorization groups bound to specific model interface ranges. Furthermore, this unit is also responsible for building and presetting a multi-layered intelligent scheduling engine consisting of sub-route scheduling, priority scheduling, and weighted scheduling.

[0095] The request receiving unit acts as the unified communication gateway for the system, receiving API requests sent by users. This unit receives access requests pointing to a single URL address of the unified entry point and extracts the API key to be verified from a specified location in the access request. Simultaneously, this unit is responsible for the preliminary parsing of the information carried in the request, extracting request feature information including the URL path, key-value pairs in the request header, specific JSON fields in the request body, and the request context length.

[0096] The permission verification unit ensures the security of the system's group-based authorization mechanism. This unit extracts the API key to be verified and checks its enabled status. If the key is valid, the unit identifies the target authorization group to which the key belongs based on pre-established binding relationships and obtains the interface resource access scope of that group. Simultaneously, the unit uses the request feature information to attempt to match a target sub-route according to the configured sub-route matching rules. Finally, the unit verifies whether the matched target sub-route is indeed within the interface access scope of the target authorization group, thereby achieving fine-grained verification based on the principle of least privilege.

[0097] The scheduling and forwarding unit is used for intelligent traffic scheduling based on multi-dimensional system status and request characteristics. When the permission verification unit confirms that the target sub-route is within the authorized scope, the scheduling and forwarding unit officially takes over the request. This unit strictly executes the priority scheduling policy according to preset rules, where sub-route scheduling has the highest priority; if the request is matched, it is forwarded directly. If the request is not matched, the unit executes system priority scheduling, such as allocating traffic according to business priority to prioritize production environment requests. In the case of not matching the priority policy or having the same priority, the unit randomly distributes traffic according to a weight value of 1-100 configured for each service instance. This unit is also responsible for generating and transmitting a unique link ID during the scheduling process to support subsequent full-link tracing.

[0098] The routing group access authorization and scheduling control method and system based on AI gateway provided in this invention have broad industrial application prospects and potential technical application value. Specific application areas and methods include: 1. Enterprise-level large model integration platform: This invention can be applied to the MaaS (Model as a Service) platform within an enterprise. It can converge scattered heterogeneous model interfaces (such as Qwen, Llama and various private models) through a unified entry point, providing a unified and secure entry point for calling AI capabilities for various business departments of the enterprise, and reducing the complexity of system integration.

[0099] 2. Intelligent Agent Swarm Scheduling System: In multi-agent collaborative scenarios, the sub-routing and three-layer scheduling mechanism of this invention are used to achieve accurate allocation of agents for different task types and ensure that high-frequency and high-priority tasks receive priority response when computing power is limited.

[0100] 3. Sensitive data processing and compliance audit scenarios: Applied to financial, medical or government fields with extremely high data security requirements, the log encryption storage and fine-grained authorization mechanism provided by this invention ensures that privacy data is not leaked during model invocation and meets legal compliance audit requirements.

[0101] 4. Enhanced cross-platform cloud-native gateway: As an AI plugin module for cloud-native gateways (such as Higress, Kong, etc.), it enhances the technical capabilities of existing gateways in handling long and short contexts of large models, non-standard protocol conversion, and dynamic permission migration, thereby improving the gateway's versatility in the AI ​​2.0 era.

[0102] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0103] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A routing group access authorization and scheduling control method based on an AI gateway, characterized in that, Applied to the gateway server, the method includes: S1. Establish a unified entry route group and configure it with a single external URL address. Under the unified entry route group, create multiple sub-routes that are associated with corresponding model service instances. Create API keys and authorization groups. Configure an interface access range consisting of at least one sub-route for each authorization group. Establish a binding relationship between the API key and the authorization group. Preset a scheduling strategy. S2. Receive an access request sent by the client pointing to the single URL address; wherein the access request carries an API key to be verified and request feature information; S3. Extract the API key to be verified, identify the target authorization group to which the API key to be verified belongs according to the binding relationship, and obtain the interface access range corresponding to the target authorization group. At the same time, match the target sub-route corresponding to the access request based on the request feature information, and verify whether the target sub-route is included within the interface access range. S4. If the target sub-route is included within the access range of the interface, the permission verification is confirmed to be successful. According to the preset scheduling policy, the target model service instance to be used to process the access request is determined from the model service instances associated with the target sub-route, and the access request is forwarded to the target model service instance.

2. The routing group access authorization and scheduling control method based on an AI gateway according to claim 1, characterized in that, S1 specifically includes: S11. Deploy a Kubernetes cluster, along with a single Redis server for storing route cache, rate limiting counters, and high-frequency access configurations; an Elasticsearch cluster for storing encrypted logs and monitoring statistics; and a Kafka cluster for asynchronously transmitting encrypted logs and statistical metrics. Create an independent URL route for the existing model service as a single route entry point for compatibility. S12. Establish a unified entry route group in the gateway control plane and configure a unified entry address for external access. Create multiple sub-routes corresponding to different AI model services under the unified entry route group, set the target service address and request protocol of the sub-routes, and configure matching rules for each sub-routes, including URL path matching, Header key-value matching, Body JSON field parsing matching, and request context length numerical matching, as well as set the corresponding traffic weight and priority. S13. Create a new authorization group and define the interface access range consisting of at least one sub-route according to the category or tag. Create an API key and select an authentication mode including Bearer mode, query parameter mode or custom Header mode. Assign the API key to the corresponding authorization group to establish the binding relationship between the API key and the authorization group and the interface access range. S14. Preset a three-layer scheduling strategy and clearly define the order in which the strategy for determining the target model service instance takes effect: sub-route scheduling takes precedence over priority scheduling, and priority scheduling takes precedence over weight scheduling.

3. The routing group access authorization and scheduling control method based on an AI gateway according to claim 2, characterized in that, When creating multiple sub-routes corresponding to different AI model services in step S12, a model category identifier field is added to identify large models and intelligent agents. In step S4, when forwarding the access request to the target model service instance, the OpenAI API specification protocol is uniformly adopted externally. For non-standard interfaces that do not support the specification, a dynamically loaded protocol conversion plugin is used to convert the data format of the non-standard interface to the OpenAI API specification protocol.

4. The routing group access authorization and scheduling control method based on an AI gateway according to claim 2, characterized in that, After establishing the binding relationship between the API key and the authorized group and interface access scope in step S13, the method further includes performing lifecycle management operations on the API key. The lifecycle management operations include editing operations to modify the group to which the API key belongs, deletion operations when it is confirmed that it is not associated with any authorized group, temporary disabling and re-enabling operations, and unbinding operations. Furthermore, it supports automatically and dynamically assigning the API key to the corresponding authorized group based on changes in the user's role to achieve dynamic grouping.

5. The routing group access authorization and scheduling control method based on an AI gateway according to claim 1, characterized in that, S2 specifically includes: S21. The gateway receives an access request sent by the client pointing to a preset unified entry URL address; S22. Extract the API key to be verified from the specified location of the access request according to the preset authentication mode; wherein, the preset authentication mode includes the Bearer mode of storing the API key to be verified in the Authorization Header of the request, the query parameter mode of storing it in the URL query parameters, or the custom Header mode of storing it in the preset custom Header. S23. Parse the URL, Header, and Body information of the access request to extract the request feature information, including the URL path, Header key value, Body JSON field, and context length value. S24. Generate a globally unique link ID for the access request using the UUID algorithm, and add the link ID to the request header to pass it to the downstream service.

6. The routing group access authorization and scheduling control method based on an AI gateway according to claim 1, characterized in that, S3 specifically includes: S31. After extracting the API key to be verified, the gateway verifies the validity of the API key to be verified. Specifically, it checks whether the key exists in the system and whether the key is enabled. If the authentication fails due to not carrying the key or the key not having permission, it returns a 403 Unauthorized error. S32. If the API key to be verified is valid, the target authorization group to which the API key to be verified belongs is identified according to the established binding relationship, and the interface access range corresponding to the target authorization group is obtained. At the same time, the preset plugin restriction policy information associated with the API key to be verified is queried to check whether the target authorization group meets the corresponding plugin restrictions. If the target authorization group does not meet the corresponding plugin restrictions, resulting in the policy verification failing, a 403 Forbidden error is returned. S33. If the strategy verification passes, then based on the extracted request feature information, the sub-route matching rules are used to attempt to match the sub-route, thereby matching the target sub-route corresponding to the access request; wherein, the matching order of the sub-route matching rules is as follows: URL path matching, Body JSON field parsing matching, Header key-value matching, and request context length numerical matching. S34. Verify whether the target sub-route obtained by matching is included within the interface access range corresponding to the target authorized group.

7. The routing group access authorization and scheduling control method based on an AI gateway according to claim 6, characterized in that, In step S33, when attempting to match a sub-route based on the extracted request feature information according to the configured sub-route matching rules, the request feature information also includes the source IP address and User-Agent header of the request. Then, IP address matching is performed through the source IP address or user agent matching is performed through the User-Agent header. Furthermore, it supports feature information conversion plugins that replace the authentication key in the user header with model authentication information for specific sub-routes, and plugin rules including IP blacklists and whitelists are uniformly applied to the unified entry route group.

8. The routing group access authorization and scheduling control method based on an AI gateway according to claim 2, characterized in that, S4 specifically includes: S41. If the matched target sub-route is within the interface access range of the authorized group, the permission verification is confirmed to be successful. First, try to match the sub-route. If the sub-route is matched, the access request is directly forwarded to the target model service instance and no further scheduling strategy is executed. S42. If a sub-route is not hit, the priority scheduling policy is executed. The corresponding priority scheduling policy is matched based on the Header information, Body content information or context length value carried in the request. The request is forwarded to an available instance according to the route availability feedback from the monitoring status. If the priority scheduling policy values ​​configured in multiple sub-routes are the same, the weight scheduling phase is entered. S43. When a request fails to hit a sub-route and priority scheduling policy or encounters multiple priority scheduling policy values ​​with the same value, the weighted scheduling policy is executed. The traffic distribution ratio between each instance is calculated based on the weight value configured for each service instance, and the access request is distributed and forwarded to the corresponding target model service instance according to the calculated traffic distribution ratio. S44. When the availability of the routing state causes a conflict between the weighted scheduling strategy and the system priority scheduling strategy, the corresponding abnormal routes are avoided and blocked through business logic to ensure the correct sub-route scheduling strategy is implemented. S45. During the process of forwarding the request to the downstream service, the generated and globally unique link ID is added to the request header and passed to the downstream service. Each target model service instance generates the corresponding span ID and records the parent span ID to form a link tree structure when processing the access request. The link span information is recorded in the log for encrypted storage.

9. The routing group access authorization and scheduling control method based on an AI gateway according to claim 1, characterized in that, The method also includes a log encryption processing step, specifically: S5. Under the premise that sensitive information is not collected by default, when the detailed call data retention switch is enabled, the basic information of the access request is collected and the request body and response body corresponding to the access request are encrypted using the AES-256 symmetric encryption algorithm. The unencrypted basic information and the encrypted request body and response body are written to the Elasticsearch cluster, and the encrypted log data is sent to the Kafka message queue for downstream systems to call the decryption API interface for consumption.

10. A routing group access authorization and scheduling control system based on an AI gateway, deployed on a gateway server, based on the routing group access authorization and scheduling control method based on an AI gateway as described in any one of claims 1 to 7, characterized in that, The system includes a configuration initialization unit, a request receiving unit, a permission verification unit, and a scheduling and forwarding unit; The configuration initialization unit is used to establish a unified entry route group and configure a single URL address for it to be exposed to the outside world. Under the unified entry route group, multiple sub-routes are created, each associated with a corresponding model service instance. API keys and authorization groups are created. For each authorization group, an interface access range consisting of at least one sub-route is configured. The binding relationship between the API key and the authorization group is established. A scheduling strategy is preset. The request receiving unit is used to receive an access request sent by the client pointing to the single URL address, wherein the access request carries an API key to be verified and request feature information; The permission verification unit is used to extract the API key to be verified, identify the target authorization group to which the API key to be verified belongs according to the binding relationship, obtain the interface access range corresponding to the target authorization group, and at the same time, match the target sub-route corresponding to the access request based on the request feature information, and verify whether the target sub-route is included within the interface access range. The scheduling and forwarding unit is used to confirm that the permission verification is passed when the target sub-route is included within the access range of the interface, determine the target model service instance to be used to process the access request from the model service instances associated with the target sub-route according to the preset scheduling policy, and forward the access request to the target model service instance.

Images