AI model scheduling method and system

Abstract

The invention discloses an AI model scheduling method and system, and aims to solve the problems that in the prior art, a heterogeneous AI model is complex in integration, low in scheduling efficiency, insufficient in safety and cost management and control, and the AI model scheduling method comprises the following steps: receiving a call request from a client through a unified gateway; performing multi-layer security review on the calling request; according to a preset intelligent routing strategy, selecting a target AI model from a model pool composed of a plurality of heterogeneous AI models, and carrying out load balancing on a request sent to the target AI model; sending the calling request to the target AI model, and receiving a response returned by the target AI model; and adapting the response to a uniform format and then returning the response to the client. According to the invention, by constructing the unified scheduling center, unified management of the multi-source AI model is realized, the scheduling efficiency and the service stability are improved, the security compliance capability is enhanced, and a transparent cost management and control system is provided.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence service management technology, specifically to AI model scheduling methods and systems. Background Technology

[0002] With the popularization of large-scale artificial intelligence modeling technology, enterprises and developers are increasingly demanding the integration of AI capabilities into their applications.

[0003] However, existing technologies face the following technical challenges when accessing and using multi-source, heterogeneous AI models: First, at the integration level, the APIs provided by different AI model vendors differ significantly in interface format, parameter definition, and authentication methods. This necessitates developers to perform independent adaptation development for each model, resulting in high integration complexity and significant costs. Furthermore, models may be deployed in different environments such as public, private, or hybrid clouds, lacking a unified management and invocation mechanism. Second, at the scheduling level, traditional model invocation methods rely heavily on manual configuration or simple polling, lacking intelligent routing decision-making capabilities. This prevents the system from dynamically selecting the optimal model based on real-time task requirements (such as cost sensitivity or performance priority), and makes it difficult to achieve effective load balancing in high-concurrency scenarios, leading to low service stability and resource utilization. Additionally, simple caching strategies cannot handle semantically similar duplicate requests, resulting in unnecessary cost waste. Third, at the security and compliance level, existing solutions have relatively simple security review mechanisms, typically only performing basic filtering of input content and lacking end-to-end review of model output content and abnormal user invocation behavior, posing risks of data leakage and misuse. Especially in government and enterprise application scenarios, existing solutions struggle to meet stringent compliance requirements such as cybersecurity level protection and data sovereignty protection. Finally, at the operational management level, the inconsistent billing methods across different models make it difficult for enterprises to establish a unified and transparent cost control and auditing system. Furthermore, monitoring dimensions are limited, lacking comprehensive monitoring of key performance indicators such as response time and token consumption in the call chain, making troubleshooting and performance optimization challenging.

[0004] Therefore, there is an urgent need for an efficient, secure, and scalable unified management solution for AI models that can solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies in terms of multi-model integration, scheduling efficiency, security compliance and cost control, and to provide an AI model scheduling system and method that aims to provide users with a unified, intelligent, secure and measurable AI model invocation service.

[0006] To achieve the above objectives, a specific embodiment of the present invention provides an AI model scheduling method, the method comprising the following steps: Receive call requests from clients through a unified gateway; The call request undergoes multi-layered security review; According to the preset intelligent routing strategy, a target AI model is selected from a model pool consisting of multiple heterogeneous AI models, and load balancing is performed on the requests sent to the target AI model. The invocation request is sent to the target AI model, and the response returned by the model is received. The response is then adapted to a uniform format and returned to the client.

[0007] In one or more embodiments of the present invention, the step of performing multi-layer security review on the call request includes at least: The input content of the call request is reviewed; The output content of the response returned by the AI ​​model is reviewed; And to review the client's calling behavior patterns.

[0008] In one or more embodiments of the present invention, prior to selecting a target AI model, the method further includes: Perform multi-level cache queries, wherein the multi-level cache includes at least a memory cache, a distributed key-value cache, and a vector cache; If any cache level is hit, the cached response data is returned directly, and subsequent steps are aborted.

[0009] In one or more embodiments of the present invention, the step of selecting a target AI model according to a preset intelligent routing strategy includes: Based on the strategy type specified in the call request, select a dominant strategy from cost-first strategy, performance-first strategy, quality-first strategy, and load balancing strategy; The target AI model is determined by combining the real-time load status of each AI model in the model pool.

[0010] In one or more embodiments of the present invention, the method further includes: Billing is based on the amount of tokens consumed by the AI ​​model, according to a unified billing rule. The system also performs performance monitoring on the entire process from receiving the call request to returning the response.

[0011] In one or more embodiments of the present invention, the step of adapting to a unified format includes: The responses returned by different AI models, which have different data structures and parameter definitions, are mapped to a pre-defined, vendor-independent, standardized interface format.

[0012] In one or more embodiments of the present invention, the heterogeneous AI models in the model pool include models deployed in public cloud, private cloud or hybrid cloud environments.

[0013] In another aspect of the present invention, an AI model scheduling system is provided, the system comprising: The gateway layer serves as a unified entry point to receive call requests from clients and performs authentication and traffic control. A security review layer is used to perform multi-layered security reviews on the call requests passing through the gateway layer and the responses generated by the AI ​​model. The scheduling decision layer is used to select a target AI model from the model pool according to the preset intelligent routing strategy, and manage the load balancing of the target AI model. The model access layer is used to communicate with multiple heterogeneous AI models in the model pool and to standardize and adapt the interfaces of different models.

[0014] In one or more embodiments of the present invention, the scheduling decision layer further includes a multi-layer caching module, which is configured to perform a cache query before executing the routing policy and to provide a response directly when the query is successful.

[0015] In one or more embodiments of the present invention, the system further includes a monitoring and billing layer, which is linked with the gateway layer, the scheduling decision layer and the model access layer to realize unified billing and full-link monitoring of model calls.

[0016] Beneficial effects: Compared with the prior art, the AI ​​model scheduling method of the present invention has the following significant improvements: This invention enables one-time access and ubiquitous invocation of multi-source, heterogeneous, and multi-deployment models through unified API and standardized interface adaptation, significantly reducing development, integration, and vendor switching costs.

[0017] The intelligent routing strategy in this invention can dynamically select the optimal model based on dimensions such as cost, performance, and quality. Combined with multi-layer caching and load balancing mechanisms, it significantly improves response speed and service stability in high-concurrency scenarios.

[0018] The end-to-end, multi-layered security review system in this invention effectively intercepts illegal content, ensures data transmission and storage security, and meets the stringent compliance requirements of government and enterprise applications.

[0019] This invention provides enterprises with transparent and controllable cost management and refined performance monitoring capabilities through a unified billing system and a full-chain monitoring system, thereby improving overall operational efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of an AI model scheduling method according to one embodiment of the present invention; Figure 2 This is a schematic diagram of the basic architecture of the AI ​​model scheduling method in one embodiment of the present invention; Figure 3 This is a schematic diagram of model invocation and invocation process in one embodiment of the present invention; Figure 4 This is a schematic diagram of an AI model scheduling system according to one embodiment of the present invention; Figure 5 This is a schematic diagram of the UI interface of the system at startup in one embodiment of the present invention; Figure 6 This is a schematic diagram of the UI interface of the model in one embodiment of the present invention; Figure 7 This is a schematic diagram of the UI interface used in one embodiment of the present invention; Figure 8 This is a schematic diagram of the UI interface for real-name authentication in one embodiment of the present invention; Figure 9 This is a schematic diagram of the UI interface for API key management in one embodiment of the present invention; Figure 10 This is a schematic diagram of the UI interface for recharge and consumption bills according to one embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0023] like Figure 1 As shown, the present invention provides an AI model scheduling method, which includes: Step 1: Receive call requests from clients through the unified gateway; Step 2: Perform multi-layered security review on the call request; Step 3: According to the preset intelligent routing strategy, select a target AI model from a model pool consisting of multiple heterogeneous AI models, and perform load balancing on the requests sent to the target AI model. Step 4: Send the call request to the target AI model and receive its response; Step 5: After adapting the response to a unified format, return it to the client.

[0024] In a specific embodiment, such as Figures 2-3 As shown, the method of this invention uses an AI gateway as a unified entry point for all client requests. Clients can be external applications or developers, who manage users (such as registration and authentication) and recharge their accounts through the model marketplace portal provided by the platform. All requests to call AI models carry the user's API key and are accessed through this AI gateway. The gateway layer first performs security verification on the requests, including identity authentication, i.e., verifying the validity of the API key, and controls the user's access permissions to specific models based on preset role and permission policies. Simultaneously, the gateway also implements traffic control, such as limiting the request rate (QPS) and the number of concurrent connections, to achieve rate limiting and circuit breaking, preventing DDoS attacks and system overload.

[0025] like Figure 3 As shown, the workflow of the entire AI model scheduling method includes: the user / application initiates an API call request with a security certificate to the gateway layer. The gateway layer first verifies the validity of the certificate and then distributes the request to the scheduling decision layer. The scheduling decision layer first checks the user's points balance and then submits the request content to the security review layer for content security review. If the review passes, the scheduling decision layer forwards the request to the scheduling layer. The scheduling layer selects the target model based on the user profile and forwards the request to the model service layer. After the model service layer returns the result and token statistics, the scheduling decision layer initiates another content security review. If the review passes, it enters the response stage. At the same time, it monitors the billing layer to record token consumption and deduct user points. The scheduling decision layer updates the points balance and finally returns the response result to the user / application. If the content security review fails, the model service layer returns a review failure message. The scheduling decision layer provides feedback to the user / application based on the corresponding reason in the request prompt. If the check finds that the user's points are insufficient, the scheduling decision layer directly returns a points shortage prompt and guides the user to recharge, and then provides feedback to the user / application.

[0026] Furthermore, the step of performing multi-layer security review on the call request includes at least: reviewing the input content of the call request; reviewing the output content of the response returned by the AI ​​model; and reviewing the call behavior pattern of the client.

[0027] This multi-layered security review is performed by a dedicated security review layer, constituting the end-to-end security guarantee of this invention.

[0028] Input vetting: Before a request is routed to any AI model provider, the system scans the input in real time. This vetting is based on a rule base that can be customized by the administrator to block politically sensitive, illegal, or company-defined trade secrets.

[0029] Output review: After receiving a response from the AI ​​model provider, the system also performs a compliance check on the AI-generated output to filter out inappropriate or illegal results and can mark or block related content.

[0030] Behavioral review: The system continuously analyzes client call behavior patterns, such as detecting abnormal call patterns like high-frequency calls occurring within a short period or access from unusual IP addresses. Once an anomaly is detected, account risk control mechanisms can be triggered, such as temporarily restricting call permissions.

[0031] In addition, the security review layer also includes data review to ensure that encryption protocols (such as HTTPS) are used during data transmission and that sensitive information (such as mobile phone numbers and ID card numbers) is de-identified and encrypted when data is persistently stored, in order to meet the compliance requirements of government and enterprises such as network security level protection.

[0032] Furthermore, before selecting a target AI model, the process includes: performing a multi-level cache query, wherein the multi-level cache includes at least a memory cache, a distributed key-value cache, and a vector cache; if any cache level is hit, the cached response data is returned directly, and subsequent steps are terminated.

[0033] To optimize response and reduce costs, this method introduces a multi-layered caching mechanism before performing model scheduling.

[0034] L1 memory cache: Requests first query the cache located in the local memory of the gateway node. This cache is used to store frequently accessed hot issues, and its time-to-live (TTL) can be set to a short time period, such as A minutes.

[0035] L2Redis Cache: If the L1 cache is not found, the second-level cache deployed in a distributed key-value store such as Redis is queried. This cache is shared by all gateway nodes and is used to store frequently asked questions. Its lifecycle can be set to a relatively long time period, such as B hours.

[0036] L3 Vector Cache: If the first two cache layers fail to find the request, the system proceeds to the third-level vector cache. The system vectorizes the core semantic content of the current request and searches the vector database for historical requests with semantic similarity exceeding a preset threshold S. If a similar request is found, the corresponding cached response is returned.

[0037] A request will only proceed to the next model scheduling stage if all cache levels fail to retrieve the result. Each successful model call result will be synchronously written to the corresponding cache level based on its characteristics.

[0038] Furthermore, the step of selecting a target AI model according to a preset intelligent routing strategy includes: selecting a dominant strategy from cost-first strategy, performance-first strategy, quality-first strategy, and load balancing strategy according to the strategy type specified in the call request; and deciding on the target AI model by combining the real-time load status of each AI model in the model pool.

[0039] This step is executed by the scheduling decision layer. When the client initiates a request, it can specify the desired routing strategy, or the system can configure it by default according to the scenario.

[0040] Cost-first strategy: Suitable for cost-sensitive scenarios such as batch processing. The system will traverse all available models in the model pool and select the model with the lowest unit token billing price.

[0041] Performance-first strategy: suitable for real-time interactive scenarios, the system will select the model with the lowest current network latency and fastest response time based on real-time monitoring data.

[0042] Quality-first strategy: Applicable to critical business decision-making scenarios, the system will select the model with the highest user rating or model performance evaluation score based on historical call records.

[0043] Load balancing strategy: In high-concurrency scenarios, the system can adopt this strategy to distribute requests evenly to multiple model nodes in a round-robin or weighted manner.

[0044] When executing any strategy, the system monitors load metrics such as CPU utilization, memory usage, and concurrent connections of each model node in real time through a dynamic load awareness module. When the load of a node exceeds a preset threshold T, the routing algorithm will automatically reduce the probability of distributing requests to that node or temporarily remove it from the availability pool.

[0045] Furthermore, the method also includes: billing according to a unified billing rule based on the token consumption of the AI ​​model; and performing performance monitoring on the entire process from receiving the call request to returning the response.

[0046] This method utilizes a separate monitoring and billing layer to perform this function, establishing a unified points-based billing system. Users recharge their accounts through the model marketplace portal to earn points. The system configures differentiated point conversion ratios for the input and output tokens of different AI models in the model pool (such as deepseek-chat and siliconflow-llama). After each call, the system calculates token consumption and settles accounts in real time. Simultaneously, this layer monitors the entire call chain, covering business metrics including but not limited to API call volume and points consumption, and technical metrics including but not limited to response time, error rate, and concurrency capabilities. It can also send alerts through pre-defined channels when a user's points balance is insufficient or service anomalies occur. All billing and monitoring data are structured and stored in the data storage layer for generating bills and providing audit traceability.

[0047] The step of adapting to a unified format includes mapping responses returned by different AI models, which have different data structures and parameter definitions, to a preset, vendor-independent, standardized interface format.

[0048] Furthermore, to address the issue of inconsistent interfaces among heterogeneous models, this method implements standardized interface adaptation through a model access layer. This layer maintains an adapter for each model in the model pool (including third-party models and enterprise-developed models). When a model needs to be called, the adapter is responsible for converting the internal standard request into the specific format of that model. When the original response from the model is received, the adapter is responsible for parsing it and encapsulating it into a unified, standardized response format compatible with mainstream industry APIs (such as the OpenAI API). This allows client applications to call all types of models (such as dialogue, image, and vector models) simply by interfaceing with this set of standard APIs.

[0049] Furthermore, the heterogeneous AI models in the model pool include models deployed in public cloud, private cloud, or hybrid cloud environments.

[0050] The model access layer of this method has good deployment compatibility, enabling unified management and scheduling of AI models deployed in different infrastructure environments, meeting the complex IT architecture needs and data sovereignty requirements of enterprises. It also supports deployment and operation in domestically produced CPUs and operating systems within the context of domestic IT innovation environments.

[0051] like Figure 4 As shown, the present invention also provides an AI model scheduling system, which includes: The functions of the gateway layer, security review layer, scheduling decision layer, and model access layer have been described in detail in the method description above.

[0052] In one specific embodiment, the scheduling decision layer further includes a multi-layer cache module, which implements the aforementioned L1, L2 and L3 multi-layer cache logic.

[0053] In another specific embodiment, the system further includes a monitoring and billing layer that works closely with the other layers of the system to achieve the aforementioned unified billing and end-to-end monitoring functions.

[0054] The system of this invention can be deployed using containerization technology (such as Docker) and cluster management using container orchestration tools (such as Kubernetes) to support high concurrency and dynamic scaling. The operation of the entire system relies on components such as persistent storage (such as PostgreSQL) and cache storage (such as Redis), which together constitute a complete data storage layer.

[0055] In specific embodiments, such as Figures 5-10 The image shows the UI interface when using this invention. Its application scenarios include, but are not limited to: government service scenarios, financial industry scenarios, and developer / SME scenarios. Different core configurations can be selected based on different application scenarios to reduce usage costs and improve work efficiency.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0061] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An AI model scheduling method, characterized in that, include: Receive call requests from clients through a unified gateway; The call request undergoes multi-layered security review; According to the preset intelligent routing strategy, a target AI model is selected from a model pool consisting of multiple heterogeneous AI models, and load balancing is performed on the requests sent to the target AI model. The invocation request is sent to the target AI model, and the response returned by the model is received. The response is then adapted to a uniform format and returned to the client.

2. The AI ​​model scheduling method according to claim 1, characterized in that, The steps of performing multi-layered security reviews on the call request include at least: The input content of the call request is reviewed; The output content of the response returned by the AI ​​model is reviewed; And to review the client's calling behavior patterns.

3. The AI ​​model scheduling method according to claim 1, characterized in that, Before selecting a target AI model, the following is also included: Perform multi-level cache queries, wherein the multi-level cache includes at least a memory cache, a distributed key-value cache, and a vector cache; If any cache level is hit, the cached response data is returned directly, and subsequent steps are aborted.

4. The AI ​​model scheduling method according to claim 1, characterized in that, The step of selecting a target AI model according to a preset intelligent routing strategy includes: Based on the strategy type specified in the call request, select a dominant strategy from cost-first strategy, performance-first strategy, quality-first strategy, and load balancing strategy; The target AI model is determined by combining the real-time load status of each AI model in the model pool.

5. The AI ​​model scheduling method according to claim 1, characterized in that, The method further includes: Billing is based on the amount of tokens consumed by the AI ​​model, according to a unified billing rule. The system also performs performance monitoring on the entire process from receiving the call request to returning the response.

6. The AI ​​model scheduling method according to claim 1, characterized in that, The step of adapting to a unified format includes: The responses returned by different AI models, which have different data structures and parameter definitions, are mapped to a pre-defined, vendor-independent, standardized interface format.

7. The AI ​​model scheduling method according to claim 1, characterized in that, The heterogeneous AI models in the model pool include models deployed in public cloud, private cloud, or hybrid cloud environments.

8. An AI model scheduling system, characterized in that, include: The gateway layer serves as a unified entry point to receive call requests from clients and performs authentication and traffic control. A security review layer is used to perform multi-layered security reviews on the call requests passing through the gateway layer and the responses generated by the AI ​​model. The scheduling decision layer is used to select a target AI model from the model pool according to the preset intelligent routing strategy, and manage the load balancing of the target AI model. The model access layer is used to communicate with multiple heterogeneous AI models in the model pool and to standardize and adapt the interfaces of different models.

9. The AI ​​model scheduling system according to claim 8, characterized in that, The scheduling decision layer also includes a multi-layer caching module, which is configured to perform cache lookups before executing routing policies and provide a response directly when a lookup is successful.

10. The AI ​​model scheduling system according to claim 8, characterized in that, The system also includes a monitoring and billing layer, which works in conjunction with the gateway layer, scheduling decision layer and model access layer to achieve unified billing and end-to-end monitoring of model calls.

Images