Model routing method, system, electronic device, storage medium and program product

Abstract

The application discloses a model routing method and system, electronic equipment, storage medium and program product, relates to the technical field of artificial intelligence, and is used for guaranteeing uninterrupted user service and automatically switching models in the routing middleware layer, driving the automatic switching of the model by counting token consumption, and better realizing the control of the model calling cost. The method comprises the following steps: receiving a request message initiated by a user corresponding to a node of a tree structure, wherein the tree structure comprises n levels, each level has m nodes, and n is an integer greater than or equal to 2; determining a model gear according to token consumption parameters of each level on a path where the node initiating the request message is located; routing the request message to a large model corresponding to the model gear for processing in the routing middleware layer; the switching of the model gear occurs in the routing middleware layer; and when the model gear is switched, a client API interface protocol remains unchanged and an established connection remains valid and continues.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model routing method, system, electronic device, storage medium, and program product. Background Technology

[0002] Large language model APIs (Application Programming Interfaces) are generally priced using tokens as the basic unit of measurement, with the output token cost of a flagship model typically being 3 to 5 times that of its input tokens. In enterprise-scale deployments and routine implementations, cost control has become a core consideration in architecture design and selection decisions. A single flagship model handling all types of user requests would result in the ineffective consumption of high-value computing power and significant token overhead. Therefore, how to reduce model call costs while ensuring uninterrupted user service has become a pressing technical problem to be solved. Summary of the Invention

[0003] This invention provides a model routing method, system, electronic device, storage medium, and program product, which ensures uninterrupted user service while automatically switching models at the routing middleware layer. By driving automatic model switching through statistical token consumption, it better manages the cost of model calls.

[0004] In a first aspect, embodiments of the present invention provide a model routing method, the method comprising: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0005] As an optional implementation, determining the model level based on the token consumption parameters at each level of the path where the node initiating the request message is located includes: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0006] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0007] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0008] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0009] As an optional implementation, the method further includes: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0010] As an optional implementation, the method further includes: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0011] As an optional implementation, the method further includes: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0012] As an optional implementation, the step of routing the request message to the large model corresponding to the model tier for processing at the routing middleware layer includes: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0013] Secondly, embodiments of the present invention provide a model routing system, which includes multiple large models and a control unit, wherein the control unit is configured to execute: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0014] As an optional implementation, the control unit is specifically configured to perform: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0015] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0016] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0017] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0018] As an optional implementation, the control unit is further configured to perform: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0019] As an optional implementation, the control unit is further configured to perform: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0020] As an optional implementation, the control unit is further configured to perform: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0021] As an optional implementation, the control unit is specifically configured to perform: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0022] Thirdly, embodiments of the present invention also provide an electronic device, which includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0023] As an optional implementation, the processor is specifically configured to execute: The model level is determined based on the token consumption parameters at each level along the path of the node that initiated the request message.

[0024] As an optional implementation, the processor is specifically configured to execute: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0025] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0026] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0027] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0028] As an optional implementation, the processor is further configured to execute: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0029] As an optional implementation, the processor is further configured to execute: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0030] As an optional implementation, the processor is further configured to execute: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0031] As an optional implementation, the processor is specifically configured to execute: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0032] Fourthly, embodiments of the present invention also provide a model routing device, the device comprising: The request initiation module is used to receive request messages initiated by users corresponding to nodes in a tree structure. The tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1. The tier selection module is used to determine the model tier based on the token consumption parameters of each level on the path where the node that initiated the request message is located. The routing module is used to route the request message to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0033] As an optional implementation, the gear selection module is specifically used for: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0034] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0035] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0036] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0037] As an optional implementation, the device also includes a prompting module specifically used for: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0038] As an optional implementation, the device also includes an approval module specifically used for: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0039] As an optional implementation, the routing module is further used for: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0040] As an optional implementation, the routing module is specifically used for: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0041] Fifthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in any of the first aspects above.

[0042] In a sixth aspect, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0043] This application provides a model routing method that classifies and intelligently distributes user requests through a reasonable model routing approach. Simple tasks (with low token consumption parameters) are routed to large models at the corresponding lightweight and low-cost model tiers, while complex reasoning and professional creation tasks (with high token consumption parameters) are routed to large models at the corresponding high-cost model tiers. The method can automatically switch to a more suitable large model based on the token consumption parameters, effectively reducing the overall call cost while ensuring the quality of the large models, and achieving fine-grained scheduling and efficient utilization of large model cluster resources.

[0044] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a specific implementation of a model routing method provided in this embodiment of the invention; Figure 2 A schematic diagram of a tree structure provided in an embodiment of the present invention; Figure 3 A flowchart for triggering a degraded route is provided as an embodiment of the present invention; Figure 4 A schematic diagram of a model routing device provided in an embodiment of the present invention; Figure 5 A schematic diagram of a model routing system provided in an embodiment of the present invention; Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a model routing device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0049] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0050] Before introducing the model routing method provided in the embodiments of this application, for ease of understanding, the technical background of the embodiments of this application will be described in detail below.

[0051] Large language model API pricing typically uses tokens as the basic unit of measurement, with the output token cost of a flagship model usually being 3 to 5 times that of its input tokens. In enterprise-scale deployment and routine implementation scenarios, cost control has become a core consideration in architecture design and selection decisions. A single flagship model handling all user requests results in the ineffective consumption of high-value computing power and high token costs. Existing solutions only support single-level or two-level budget control, failing to reflect organizational hierarchies and leading to unclear cross-departmental cost attribution, with sub-department overspending affecting the entire enterprise's quota. Existing solutions typically address service unavailability (availability assurance) rather than quality degradation driven by budget consumption (cost governance). The triggering conditions, switching logic, and user experience design for these two are completely different; existing solutions use both interchangeably, which is unsuitable for cost governance scenarios and introduces unnecessary quality fluctuations. Existing solutions typically handle budget exhaustion by rejecting requests and returning error codes, resulting in service interruption for the user. Existing solutions lack sub-level budget isolation mechanisms. If a project in the technology department overspends, only that project should be restricted, without affecting other normal projects within the department; existing solutions cannot achieve this level of granular isolation at the global budget level. Therefore, reducing model invocation costs while ensuring uninterrupted user service has become a pressing technical problem. It should be noted that the model fallback mechanisms in existing solutions (such as LiteLLM, Portkey, and Bifrost) are triggered by model service unavailability (availability events). This differs fundamentally from the model degradation routing (cost governance event) driven by a real-time token consumption rate threshold in this application in terms of triggering logic, switching semantics, and business objectives. They are not equivalent replacements for the same technical means, and the two cannot be simply combined to obtain the technical solution of this application. Reducing model invocation costs has become a pressing technical problem.

[0052] To address the aforementioned technical issues, this application provides a model routing method for establishing a tree-like hierarchical structure. Each level uses statistical token consumption parameters to trigger the selection of model tiers, thereby enabling the switching of large models. Furthermore, the large model switching is completed at the routing middleware layer, ensuring that the user-side interface protocol remains unchanged, ongoing requests are unaffected, business operations are not interrupted, and established connections do not need to be reset.

[0053] like Figure 1 As shown, the specific implementation process of the model routing method provided in this embodiment is as follows: Step 100: Receive the request message initiated by the user corresponding to the node of the tree structure. The tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1. Optionally, the first level has one node, which is the root node in the tree structure. Each level other than the first level has one or more nodes. Optionally, the tree structure's levels, from top to bottom, correspond to root node → parent node → child node → leaf node. That is, the first level of the tree structure includes one root node, the second level includes one or more parent nodes, the third level includes one or more child nodes, and the fourth level includes one or more leaf nodes. The root node has one or more parent nodes, each parent node has one or more child nodes, and each child node has one or more leaf nodes. In implementation, the tree structure can be built based on the enterprise's organizational structure. The levels can be defined according to the enterprise's organizational hierarchy, and the nodes in each level can be defined according to the levels under each level of the enterprise's organizational hierarchy. For example, the organizational hierarchy of an enterprise can be divided into the following levels from top to bottom: enterprise level, department level, project level, and user level. The enterprise level is the root node. Each enterprise level has one or more department levels, each department level has one or more project levels, and each project level has one or more user levels.

[0054] In one example, the enterprise organizational structure is mapped as a four-level budget tree. Enterprise Level: The global monthly total budget limit for tokens (global monthly token quota limit), configured by the CFO or IT administrator, is the absolute limit for all sub-level quotas. Department Level: Token quotas independently configured for each business department, applied for by the department head and approved and allocated by the IT administrator. The sum of the token quotas configured by departments cannot exceed the total token quota configured by the enterprise. Project Level: Sub-budgets for each project or product line within a department, supporting fine-grained configuration of token quotas according to actual business needs. User Level: Daily / monthly token quotas configured for individual users or service accounts, suitable for controlling abnormal overuse by a single user.

[0055] Optionally, the token consumption budget constraint propagation rule (tree constraint) is as follows: B_child ≤ B_parent - C_parent_others; where B_child is the token limit configured for the child node, B_parent is the token limit configured for the parent node, and C_parent_others is the total token limit consumed by other child nodes at the same level. This constraint ensures that the token limit configured for any child node cannot exceed the currently remaining available token limit of the parent node, and is enforced by the system during configuration. Optionally, when a child node reaches its limit, only the routing policy of that child node is affected, without affecting the normal operation of other child nodes at the same level.

[0056] Optionally, each node is configured with an independent token quota. In one possible example, the token quota configured at the (i+1)th level of the tree structure in this embodiment is less than or equal to the currently available token quota at the ith level; i is an integer greater than or equal to 1 and less than or equal to n. In implementation, the ith level acts as the parent of the (i+1)th level, and the (i+1)th level acts as the child of the ith level. The token quota of the child cannot exceed the currently available token quota of the parent, that is, it cannot exceed the remaining token quota of the parent.

[0057] In one possible implementation, each level of the tree structure in this embodiment is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters for that level. Optionally, the token consumption device includes, but is not limited to, a token consumption counter and / or a token consumption rate calculation device, which can calculate the token consumption parameters for that level. Optionally, the token consumption parameters include, but are not limited to, the number of tokens consumed and / or the token consumption rate.

[0058] Optionally, each node is configured with an independent token consumption device, which is used to calculate the token consumption parameters for that node. The token consumption device at each level is used to calculate the token consumption parameters for that level from different fine-grained scales, which include at least one of hour, day, week, month, and year.

[0059] In implementation, each node at each level is configured with three independent token consumption mechanisms. For example, each node has an independent hourly token counter (H), daily token counter (D), and monthly token counter (M), corresponding to different granularities of budget control periods. The token consumption mechanism for each node can be implemented using a counter cluster, which can be a Redis cluster and supports atomic increment operations to ensure counting accuracy in high-concurrency scenarios. The token consumption count of the parent node is equal to the real-time sum of the token consumption counts of all child nodes (consistency is ensured through an aggregation mechanism rather than dual writes).

[0060] In practice, for each request message Q entering the system (which may also carry the organizational hierarchy label org_path), the system performs the following calculations: The token consumption rate is calculated as follows: R_level = C_level / B_level; where C_level is the number of tokens consumed in the current period for this level (from the real-time counter cluster), and B_level is the configured token limit for this level (from the budget registry).

[0061] In one possible implementation, each level of the tree structure in this embodiment is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large model corresponding to different model tiers are different. For example, the quality of the large model in this embodiment includes, but is not limited to, any one or more of the following: inference response latency (first character output time, whole sentence response time), throughput (how many tokens / requests can be processed per second), resource consumption (video memory, GPU computing power consumption), stability (no crashes, no timeouts, no freezes under high concurrency), and fine-tuning capability (whether it supports industry data fine-tuning, adaptation to private businesses), etc. The cost of the large model includes, but is not limited to, input / output token unit price, call cost, etc.

[0062] In one possible example, four token consumption threshold ranges are defined for each level. These ranges can be identical; for example, each level could define four token consumption rate threshold ranges: [0~50%), [50~80%), [80~100%), and 100%. When a token consumption threshold is met, the request message is automatically routed to the corresponding model tier for processing. Optionally, there can be multiple model tiers. For example, four tiers could be T1, T2, T3, and T4. The enterprise administrator can customize the configuration of the corresponding model for each tier.

[0063] In one example, a tree structure can be built based on the enterprise's organizational structure. Taking n=4 as an example, a 4-level structure can be built. See [link to relevant documentation]. Figure 2 As shown in the diagram, this embodiment provides a schematic of a tree structure. The first level of the tree structure includes a root node, which corresponds to the enterprise level. This first level is the enterprise level, where token limits can be configured (e.g., a monthly total token budget limit of 5,000,000,000 tokens / month). The second level includes one or more nodes, each corresponding to a department within the enterprise. This second level is the department level, where each department has an independent token limit. For example, the technology department has an 800M token limit, and the marketing department has a 200M token limit. The third level includes one or more nodes, each corresponding to one or more projects within a department. This third level is the project level, where each project has an independent token limit. For example, the AI ​​(Artificial Intelligence) assistant project has a 300M token limit. The fourth level includes one or more nodes, each corresponding to one or more users within a project. This fourth level is the user level, where each user has an independent token limit. For example, an individual user has a daily token limit of 5,000 tokens / day.

[0064] Step 101: Determine the model level based on the token consumption parameters at each level of the path where the node that initiated the request message is located; Optionally, there may be multiple model tiers. Taking four model tiers as an example, they are Tier 1 (T1), Tier 2 (T2), Tier 3 (T3), and Tier 4 (T4). The enterprise administrator can customize the large model corresponding to each tier. See Table 1, which provides an example table of models for each tier. The large model corresponding to Tier 1 can be a Premium model, Tier 2 a Standard model, Tier 3 a Lightweight model, and Tier 4 an Emergency model. The token consumption rate threshold range for Tier 1 is [0~50%], for Tier 2 it is [50~80%], for Tier 3 it is [80~100%], and for Tier 4 it is 100%.

[0065] Table 1. Model Examples for Each Model Tier

[0066] In one possible implementation, the model gear is determined as follows: The model level is determined based on the token consumption threshold range that the token consumption parameter of the node that initiated the request message falls into.

[0067] In implementation, the model tier is determined based on the token consumption threshold range that the token consumption parameter of the node initiating the request message falls into. Each model tier corresponds to a token consumption threshold range.

[0068] In some embodiments, the present application can also combine the joint judgment model of each level to determine the gear level. The specific implementation process is as follows: The model level is determined based on the token consumption parameters at each level along the path of the node that initiated the request message.

[0069] In implementation, the path of the node initiating the request message can be determined first. This path refers to the complete hierarchical link extending from the request initiating node upwards to the root node of the tree structure. After determining this path, the token consumption parameters of each level along the path are calculated, and the final model level is determined by combining the token consumption parameters of each level.

[0070] In one possible example, the model tier is determined based on the token consumption threshold range into which the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls.

[0071] In one example, the system simultaneously queries the token consumption rate of all levels (user / project / department / enterprise) along the path of the request message. It takes the maximum token consumption rate R_max among all levels along the path as the routing decision basis, determines the model tier corresponding to the token consumption rate threshold range into which this maximum value falls, and routes the request message to the large model corresponding to that model tier. If a department's token consumption rate R_dept = 85% (already in the third tier T3 range), even if the user's individual token consumption rate R_user = 30%, the request message will still be routed to the third tier large model (Lightweight model) to ensure departmental budget constraint priority.

[0072] Step 102: In the routing middleware layer, the request message is routed to the large model corresponding to the model level for processing. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0073] It should be noted that if the model profile corresponding to the node that initiated the current request message is different from the model profile corresponding to the node that initiated the request message at the previous moment, a model profile switch will occur, switching the model profile to the model profile corresponding to the node that initiated the current request message. However, during the model switching process, since the request message is routed at the routing middleware layer, that is, the online switching of the model profile in this application occurs at the routing middleware layer, it can ensure that the client API interface protocol remains unchanged and the established connection remains valid and continues to exist, ensuring that the established connection is not interrupted and does not need to be rebuilt.

[0074] In this embodiment, the switching of the large model is implemented at the routing middleware layer. The user-side interface protocol remains unchanged, and established connections are not reset. This ensures that the degradation switching is transparent to the user and that services are not interrupted. Routing middleware is an "interception / processing function" that is bound to a route or route group and only applies to a specific path. The routing middleware is positioned between the client request and the client response. Client request → routing middleware → routing service Handler → routing middleware (post-processing) → response to the client.

[0075] In one possible implementation, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing within the large model corresponding to the model level.

[0076] This embodiment provides a routing algorithm. After determining the model tier Tier(R), a large model is selected from the candidate model set for that tier according to the following rules: Route(Q) = argmin_m cost(m), satisfying Tier(m) = Tier(R) and quality(m, Q) ≥ theta_quality. That is, the model with the lowest cost is selected under the premise of satisfying the quality lower limit constraint. Route(Q) represents the optimal large model assigned / routed to request message Q, argmin_m cost(m) represents finding the m with the lowest cost among all available large models m; Tier(m) = Tier(R) represents the model tier of large model m, which must match the tier of request message R, and quality(m, Q) ≥ theta_quality represents the large model quality of large model m in processing request message Q, which must be greater than or equal to the predefined quality lower limit. The quality threshold theta_quality is set by the enterprise administrator through a configuration file (for example, the overall response quality score should not be lower than 0.65 points, and the scoring dimensions include but are not limited to: inference response latency, output content accuracy, and task completion). The candidate model set and the quality threshold for each level can be customized according to business needs. The cost cost(m) is calculated as follows: the unit price of the input token of the model multiplied by the estimated number of input tokens, plus the unit price of the output token multiplied by the estimated number of output tokens, measured in monetary units.

[0077] In one embodiment, when the token consumption parameter reaches the target threshold, it indicates that the tokens have been exhausted. At this point, routing to the degraded large model can be triggered, and the specific process is as follows: In response to the model tier being the target tier, the request message is added to the pending queue, and the budget increase request message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0078] Optionally, the target tier includes a fourth tier. In implementation, when the token consumption rate is 100%, the model tier is the target tier. The queue can be suspended for approval without directly interrupting the connection, ensuring uninterrupted service. If approval is granted, the token consumption rate can be updated based on the temporarily configured token limit. The model tier corresponding to the token consumption rate threshold range that the updated token consumption rate falls into is determined, and the request message is automatically routed to the larger model in that tier at the routing middleware layer.

[0079] In one possible embodiment, in response to the token consumption parameter reaching a target threshold, where the model level is the target level, the request message is routed to a local large model for processing. Optionally, when the token consumption parameter is the token consumption rate, the target threshold is 100%, and the target level is the fourth level. In practice, when the token consumption rate is 100%, the request message can also be routed to a local large model for processing, where the local large model incurs almost zero cost, enabling emergency use for model switching.

[0080] In one possible implementation, this embodiment can also prompt the user when a model gear shift occurs, with the specific implementation steps as follows: In response to the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0081] Optionally, token consumption information includes, but is not limited to, any one or more of the following: token consumption count, token consumption rate; model level switching information includes, but is not limited to, any one or more of the following: hierarchical path, the actual large model called, the reason for model degradation, and the basis for routing decisions.

[0082] For example, taking the token consumption rate as the token consumption parameter, when the token consumption rate exceeds the first tier's threshold range (0-50%), the request message is automatically routed to the second tier's large model, while simultaneously prompting the user "Currently using standard mode," without interrupting business operations. When the token consumption rate exceeds the second tier's threshold range (50-80%), the request message is automatically routed to the third tier's large model, while simultaneously prompting the user "Reason for model switching," and can also provide the administrator's contact information. When the token consumption rate exceeds the third tier's threshold range (80-100%) and falls within the fourth tier's threshold range, the user is prompted "Service limited," the request message is added to the pending queue, and a budget supplement request message is pushed to the target approval user, without directly interrupting the business connection.

[0083] This application provides a four-level budget hierarchy (enterprise level → department level → project level → user level). Through a counter cluster, each level maintains an independent token consumption counter, maintained on a three-cycle basis (hourly, daily, and monthly). The parent level's token consumption counter equals the sum of all child level token consumption. When a child level's token consumption exceeds its configured token limit, it does not affect other child nodes at the same level. Isolation is provided, meaning that if a single department's token limit is exhausted, it does not freeze the token limits of other departments. Optionally, this embodiment provides a budget inheritance rule, where the child level's token limit is less than or equal to the parent level's remaining available token limit (tree constraint). The configuration strategy for the child level's token limit can be tightened but cannot loosen the parent level's token limit strategy. End-of-period reset is performed level by level, starting with the parent level and then moving to the child level. It provides independent counting on three cycles, a parent-child aggregation relationship, child level isolation guarantees, and a budget inheritance rule.

[0084] This embodiment also provides a threshold-triggered degradation routing engine, displaying four stages of routing strategies, trigger actions, and user experience segmented by token consumption rate range (0~50% green / 50~80% yellow / 80~100% red / 100% dark). In the first tier, with a token consumption rate of 0~50%, which falls within the normal operating range, the routing strategy selects the optimal large model based on the task semantic intent of the request message, prioritizing quality. The trigger action remains unchanged, following normal routing logic. Users can also be notified of updated consumption progress; the user experience is completely transparent and imperceptible. In the second tier, with a token consumption rate of 50~80%, which falls within the budget warning range, the routing strategy downgrades the large model from the first tier to the second tier, saving approximately 60% of costs. The trigger action can be sending a budget warning email to the department head, or notifying the user of a token consumption warning with a lightweight notification of "currently using standard mode," without interrupting business operations. The third tier, with a token consumption rate of 80-100%, falls into the critical alert range. The routing strategy is to force all request messages to be downgraded to the third-tier large model (the lowest-cost compliant model). The triggering action is to send an emergency budget warning to management or trigger a budget application approval workflow. A clear message to the user is displayed: "Lightweight mode has been switched. For full functionality, please contact the administrator." Example: Individual 5,000 Tokens / day. The fourth tier, with a token consumption rate of 100%, indicating budget exhaustion, triggers an approval workflow and suspends request messages (maximum TTL (Time To Live) timeout). The triggering action is to send an emergency application to management. Service is restored after approval, or a zero-cost local large model can be configured as an emergency model. The user is clearly informed of the reason for service limitation, provided with contact channels, and the established connection is not directly interrupted. Audit logs and user prompts can also be provided; for example, each request record should include: hierarchical path / total token consumption / actual large model called / reason for model degradation / one or more of the following: The user's right to know is protected through a differentiated prompting strategy (Tier 1 T1 No prompt / Tier 2 T2 Light prompt / Tier 3 T3 Noticeable prompt / Tier 4 T4 Service restriction notice).

[0085] It also supports drill-down analysis by department / project / user dimension and automatic monthly AI expenditure report generation. Optionally, if the token consumption parameter of the first level, such as the enterprise level, reaches the threshold, the system will be downgraded globally. That is, the model level corresponding to the request message of all nodes in the tree structure will be downgraded by one level, such as from the first level to the second level, and from the second level to the third level.

[0086] Optionally, this embodiment provides an approval workflow. When the token quota budget is exhausted, new request messages are placed in a TTL queue and suspended. Service is restored immediately after approval without restarting. The TTL queue is a message queue with an expiration time. Messages / queues are automatically cleaned up when the time expires to avoid unlimited message backlog and to delay business operations (such as closing orders due to timeout). Optionally, this embodiment can also generate complete cost attribution audit logs and support BI analysis dashboards that drill down to the organizational level, meeting the visibility and manageability requirements of AI expenditures.

[0087] like Figure 3 As shown in the figure, this embodiment also provides a flowchart for triggering a degraded route, and the specific implementation process is as follows: Step 300: Receive the request message initiated by the user corresponding to the node in the tree structure; Optionally, the request message may also carry a hierarchy label to indicate the hierarchy to which the node that initiated the request message belongs.

[0088] Step 301: Query the token consumption rate R of all levels on the path of the node containing the request message; In practice, by using the hierarchical tags carried in the request message, all levels on the path where the node of the request message is located are determined, and then the token consumption parameters of all levels, such as the token consumption rate R, are queried.

[0089] Step 302: Determine if the token consumption rate R is less than 50%. If yes, proceed to step 303; otherwise, proceed to step 304. Step 303: Route the request message to the large model corresponding to the first tier for processing; Optionally, the cost and quality of the large models corresponding to each model tier can be progressively reduced, from the first to the fourth tier. The large model corresponding to the first tier can be the optimal model.

[0090] Step 304: Send a budget alert notification to the user of this node and push it to the department head; Optionally, it can be sent to users via email, WeChat Work, DingTalk, or other communication software.

[0091] Step 305: Determine if the token consumption rate R is less than 80%. If yes, proceed to step 306; otherwise, proceed to step 307. Step 306: Route the request message to the large model corresponding to the second tier for processing; Optionally, the larger model corresponding to the second tier can be the standard model.

[0092] Step 307: Send an emergency alarm notification to the user of this node to trigger the approval workflow; Optionally, the approval workflow can be executed asynchronously in the background without blocking the current request message. The approval workflow is used to push budget supplement request messages to the target approval user.

[0093] Step 308: Determine if the token consumption rate R is less than 100%. If yes, proceed to step 309; otherwise, proceed to step 310. Step 309: Route the request message to the large model corresponding to the third tier for processing; Optionally, the large model corresponding to the third tier can be a lightweight model.

[0094] Step 310: Add the request message to the pending queue, trigger the approval process, and wait for the approval result; Optionally, the suspended queue includes a TTL queue, which is a message queue with an expiration time.

[0095] Step 311: Determine whether the approval has been granted or whether there is an emergency quota. If yes, proceed to step 312; otherwise, proceed to step 313. At this point, the token quota budget is exhausted. First, check if there are any temporarily allocated token quotas or emergency token quotas. Temporary quotas refer to token quotas allocated temporarily, while emergency quotas refer to token quotas that can be used in emergencies.

[0096] Step 312: Route the request message to the large model corresponding to the fourth tier for processing; Optionally, the large model corresponding to the fourth tier can be an emergency zero-cost model, such as a local large model. After approval, the model tier can be updated again, and the system will route to the large model corresponding to the updated model tier. For example, if the model tier is restored to the third tier after approval, the system will route to the large model in the third tier for processing.

[0097] Step 313: Return to the service restriction prompt, provide contact channels, and do not interrupt the connection.

[0098] The model routing method provided in this application ensures business continuity and uninterrupted degradation. Model tier switching is entirely completed at the routing middleware layer; the user-side API (Application Programming Interface) protocol (no reconnection required), WebSocket connection (maintaining connection), and streaming response format remain unchanged. Users only perceive a change in response quality, not service interruption. Request messages are isolated; degradation strategy changes only affect newly arriving request messages, while pending requests remain unaffected and continue to use the original model to complete the response. Request messages in progress before model degradation (switching) are not interrupted; request messages in the queue are routed according to the new model tier, and the degradation result is labeled with the model tier origin for BI dashboard analysis. A transparent warning strategy is provided; the system adds a transparent degradation description to the response request message (including metadata) based on the current model tier (T1: no warning; T2: light warning; T3: explicit warning; T4: service restricted description). The warning information can be customized by the enterprise administrator.

[0099] Optionally, this embodiment may also provide an automatic recovery mechanism: ① Month-end counter reset, automatically restoring the full T1 quota; ② Approval of additional budget takes effect immediately (no service restart required); ③ Historical refunds or false alarms related to consumption rate can trigger a score recovery. In this embodiment, when the token consumption rate reaches 100%, an approval workflow is triggered. When the token quota is exhausted, newly arriving request messages are placed in a queue with TTL (e.g., default timeout of 30 minutes), and an application notification is simultaneously pushed to the approver. After approval, suspended requests in the queue automatically resume execution without requiring the user to resend.

[0100] In one possible embodiment, this embodiment can also record the following cost attribution audit fields for each LLM call, supporting multi-dimensional drill-down analysis. See Table 2 for the cost attribution audit details.

[0101] Table 2 Cost Attribution Audit

[0102] In one possible embodiment, this embodiment can also provide a BI dashboard, which transforms business data into charts, numbers, trend graphs, etc., and displays them on a visualization interface / screen to allow users to view business conditions. For example, this embodiment can display any one or more of the following through a visualization interface: monthly AI expenditure rankings for departments; project-level budget consumption trends; large model usage heatmaps distributed by model tier; model degradation event timelines; and budget warning early trigger predictions (based on linear extrapolation of token consumption rates). It can also automatically generate monthly AI expenditure reports and send them to the company's CFO (Chief Financial Officer) and department heads.

[0103] Referring to Table 3, this embodiment provides an example of a model tier definition matrix, which can be customized by enterprise administrators. It displays the complete attribute definitions of the four model tiers T1 to T4, using typical model / cost index / quality index / applicable tasks / user tips / automatic recovery as references. In Table 3, R represents the token consumption rate.

[0104] Table 3 Model Tier Definition Matrix

[0105] This embodiment provides a tree-structured budget hierarchy. Model degradation is triggered in real-time by token consumption, introducing cost governance semantics into routing decisions. Model tier switching is completed at the routing middleware layer, with the user-side API interface and connection protocol remaining unchanged, and ongoing requests unaffected. Token consumption parameters are queried simultaneously at all levels along the request path, and the maximum value is used as the basis for routing decisions, with enterprise-level constraints always taking precedence. When the budget is exhausted, new requests enter a suspended queue, automatically resuming execution after approval, without requiring users to resend requests. Different model tiers correspond to different levels of user prompts (T1 no prompt / T2 slight prompt / T3 obvious prompt / T4 service restricted), with enterprise administrators customizing the prompt text. Enterprise-level AI usage costs can be reduced by 40%~70% (by automatically routing a large number of low-complexity tasks to low-cost models); the cost of a T3 lightweight model is only about 8% of that of a T1 flagship model, and the departmental budget alert mechanism can reduce the probability of monthly overspending by 80%. The aforementioned cost savings data are based on theoretical calculations for typical enterprise scenarios: Assuming approximately 60% of the enterprise's monthly total token requests fall into simple FAQ / translation / summarization tasks (suitable for the T3 lightweight model), and considering the cost difference of approximately 12.5 times between the T1 flagship model and the T3 lightweight model, the overall weighted cost savings are approximately 48%. Triggering T2 routing when the budget consumption rate exceeds the 50% threshold can further extend the savings range to 40%~70%. The 80% reduction in monthly overspending probability is based on: budget alerts are triggered when the consumption rate reaches 50%, allowing departments approximately 50% of the remaining period for budget adjustments, theoretically eliminating most passive overspending situations. Four-level hierarchical cost attribution allows the finance department to accurately trace AI expenditures to specific teams and projects, supporting cost accounting and cost allocation (Chargeback); the BI dashboard integrates AI expenditures into the enterprise's digital governance system. The uninterrupted model degradation design ensures a continuous user experience, eliminating service interruptions caused by budget overruns; users are essentially unaware of model switching within normal usage limits. Systematic budget tiering and threshold alerts enable enterprises to identify AI cost risks in advance, rather than reacting passively after invoices arrive; the approval workflow integrates AI budget management into the enterprise's standard procurement approval process. Model tier configuration is fully customizable by enterprise administrators, and the candidate model list for each tier can be flexibly adjusted with the release of new models; the four-level hierarchical structure can be expanded to more levels through configuration (such as adding a "sub-project" level).

[0106] In one possible implementation, this embodiment provides a model routing process for a month-end budget control scenario involving an AI code assistant in the technology department. The specific scenario is as follows: An internet company's technology department has a monthly AI budget of 500 million tokens, comprising three projects (AI assistant 200 million, code review 150 million, and data analysis 150 million). On the 18th day of the month, the AI ​​assistant project's token consumption rate reaches 82%, the code review project is normal (35%), and the overall token consumption rate for the technology department is 64%.

[0107] Step a: Real-time system query; Specifically, AI assistant developer Xiao Zhang sends a request message "Help me optimize this Python code", which carries the hierarchical tag org_path = Enterprise / Technology Department / AI Assistant Project / Xiao Zhang.

[0108] Step b: Multi-level joint judgment; Specifically, the token consumption rate for the enterprise level is R=41% (corresponding to the first tier T1); the token consumption rate for the technology department is R=64% (corresponding to the second tier T2); the token consumption rate for the AI ​​assistant project is R=82% (corresponding to the third tier T3); and the token consumption rate for user Xiao Zhang is R=28% (corresponding to the first tier T1). The maximum token consumption rate R_max = 82% is taken as the threshold for model tier determination.

[0109] Step c, Model gear selection; The specific judgment is: if R_max = 82% ≥ 80% and < 100%, route to the third tier T3 (Lightweight) large model. For example, select Claude Haiku 4.5 (cost is only about 8% of Opus).

[0110] Step d: Xiao Zhang received a response with metadata indicating that "the current AI assistant project has entered lightweight mode (82% of the monthly budget has been used). If you need the full functionality, please contact Manager Zhang (manager@tech.co) to increase the budget."

[0111] The response quality was acceptable, and business operations were uninterrupted. Optionally, the system simultaneously sent an alert email to the AI ​​assistant project manager stating, "Monthly budget remaining is 18%, expected to be exhausted in 2-3 days; please apply for additional funding promptly." Code review and data analysis projects remained unaffected and continued to be routed to the large model in Tier 1.

[0112] In one possible implementation, this embodiment provides a budget exhaustion approval recovery - operations department end-of-month sprint scenario. In this scenario, during the Double 11 sprint period (the 30th day of the month), the monthly AI budget of an e-commerce company is completely exhausted (R=100%), and the operations staff still need to use AI to generate campaign copy.

[0113] Step a: Operations specialist Xiao Li sends a request message "Help me write a Double 11 promotional email, 200 words". The system queries the maximum value of the token consumption rate at each level of the path where the request message is located, R_max=100% (the operations department's budget is exhausted).

[0114] Step b: Determine the model's gear position; Specifically, check if there are any approved temporary quotas. If there are currently no temporary quotas, the request message is placed in the TTL queue (timeout 30 minutes), and an emergency approval application is sent to the Operations Director and CFO: "The Operations Department's monthly AI budget has been exhausted. Request an additional 50 million Tokens temporary quota, estimated cost ¥2,500".

[0115] Step c: Xiao Li's interface displays "Your request has been submitted, budget approval in progress (expected to be completed within 15 minutes). Service is temporarily limited, thank you for your patience." The connection remains open, and no further action is required.

[0116] Step d: The Operations Director receives the approval notification on WeChat Work and clicks "Agree" 5 minutes later. The system immediately updates the Operations Department's quota (adding 50 million Tokens) and switches the routing engine to the third tier, T3.

[0117] Step e: Xiao Li's request in the TTL queue is automatically resumed. The lightweight model at the T3 level (due to limited additional quotas, the T3 level costs approximately 2,000 Tokens) is used to route the request message to the large model at the T3 level for processing and return a draft promotional email. Xiao Li receives the draft promotional email, and the business returns to normal.

[0118] In one possible implementation, this embodiment provides a multi-tenant enterprise SaaS (Software as a Service) scenario with tiered billing based on customers. In this scenario, an AI SaaS platform provides AI Agent services to enterprise customers, which need to provide differentiated AI capabilities according to different subscription packages (basic / professional / flagship), and smoothly downgrade rather than directly reject services when the package quota is exhausted.

[0119] Step a, Platform Configuration: Flagship Edition customers are in the first tier T1 (Claude Opus 4.6), with a monthly quota of 1 billion tokens; Professional Edition customers are in the second tier T2 (Claude Sonnet), with a monthly quota of 200 million tokens; Basic Edition customers are in the third tier T3 (Claude Haiku), with a monthly quota of 20 million tokens.

[0120] Step b: When Flagship Customer A's Token consumption rate reaches 90% at the end of the month, the system automatically routes Customer A's request message to the third tier T3 and prompts "Your Flagship Package has 10% of your monthly quota remaining. We have switched you to Standard Mode. If you need to upgrade or add more, please contact the Customer Success Team."

[0121] Step c: Professional version customer B uses it normally. Within the 50%~80% range, it will automatically downgrade to the second tier T2 (which is still the "standard mode" for the customer, since its package does not include the flagship capability model of the T1 tier).

[0122] Regardless of the customer's plan or credit limit status, the API interface format remains consistent (OpenAI compatible format), requiring no modification to the integration code on the customer's side. The SaaS platform can calculate the profit margin based on actual call costs, and cost savings during the downgrade period for flagship customers are directly included in the platform's profit.

[0123] In one possible example, the specific implementation steps of the model routing method provided in this embodiment are as follows: (a) Hierarchical construction steps: Establish a four-level tree-shaped budget hierarchy that includes at least enterprise level, department level, project level and user level. Each level maintains an independent token consumption counter. The token amount of a lower-level node cannot exceed the current remaining available token amount of its upper-level node. Optionally, the four-level tree structure satisfies the budget constraint propagation rule: B_child ≤ B_parent - C_parent_others, where B_child is the token limit of the child node, B_parent is the total token limit of the parent node, and C_parent_others is the sum of the tokens consumed by other child nodes at the same level. When a child node reaches its token limit, only the degradation routing strategy corresponding to that child node is triggered, without affecting the routing behavior of other child nodes at the same level.

[0124] (b) Consumption rate query step: For each request message entering the system, based on the organizational level tag carried by the request message, query the token consumption rate of all level nodes on the request path at the same time, where the consumption rate is defined as the ratio of the number of tokens consumed to the budget quota of that level. (c) Joint decision-making step: Take the maximum token consumption rate among all the queried levels as the basis for routing decision; (d) Degradation routing step: Based on the token consumption threshold range where the maximum token consumption rate is located, the request message is routed to the large model (such as LLM) of the model level corresponding to the token consumption threshold range for execution; Optionally, the token consumption threshold range includes at least the normal operation range, the budget alarm range, and the critical alarm range. Optionally, the token consumption threshold range includes four ranges and corresponding model tiers: the first range has a token consumption rate of less than 50%, corresponding to the best quality tier (Tier 1 T1 Premium); the second range has a token consumption rate of 50% to 80%, corresponding to the standard quality tier (Tier 2 T2 Standard); the third range has a token consumption rate of 80% to 100%, corresponding to the lightweight tier (Tier 3 T3 Lightweight); and the fourth range has a token consumption rate of 100%, triggering budget exhaustion emergency handling, corresponding to the zero-cost emergency tier (Tier 4 T4 Emergency) or request suspension.

[0125] Optionally, when the token consumption rate reaches 100%, newly arriving request messages are placed in a suspended queue with a preset timeout (TTL), and a budget increase application notification is pushed to a predefined approver. Once the approval is granted and the budget quota is increased, the request messages in the suspended queue automatically resume route execution without requiring the user to resend the request.

[0126] Optionally, in the degradation routing step, after determining the model tier (Tier(R)), the target model is selected from the candidate model set of that model tier according to the following rules: under the premise of meeting the predefined quality lower limit constraint, the largest model with the lowest cost is selected; the candidate model set and the quality lower limit threshold of each model tier are customized by the enterprise administrator through the configuration file.

[0127] (e) Business continuity steps: The model level switching is completed in the routing middleware layer, the user-side interface protocol remains unchanged, and the requests being processed are not affected by the level switching; (f) Recording steps: Record a cost attribution audit log for each request message, including the hierarchical path, token consumption, actual model level called, and the reason for model degradation.

[0128] Optionally, the system can return differentiated transparent downgrade prompts to the user based on the current gear level: no prompts are added during normal operation; a light prompt explaining the current operating mode is added during budget alarm intervals; a prominent prompt is added during critical alarm intervals, and a channel to contact the administrator to add budget is provided; and a clear explanation of service limitations and the estimated recovery time is provided when the budget is exhausted.

[0129] like Figure 4 As shown, this embodiment also provides a model routing device, which includes: The budget hierarchy registration module 400 stores and manages the four-level tree-structured budget hierarchy and the budget quotas of each node; the real-time counter cluster 401 maintains a token consumption counter that supports atomic operations for each level node; the consumption rate calculation module 402 queries the token consumption parameters of all levels on the request path in real time and calculates the maximum value; the degradation routing engine 403 determines the model level based on the token consumption threshold range and outputs the routing decision based on the candidate model selection algorithm; the business continuity controller 404 performs model switching at the routing layer to ensure that the user-side interface protocol remains unchanged; and the cost attribution audit module 405 records complete request cost tracking information.

[0130] In one possible example, the device also includes an approval workflow module that puts new requests into a TTL pending queue when the budget is exhausted, pushes approval applications via enterprise communication tools (including email, WeChat Work, or DingTalk), and automatically resumes the routed execution of pending requests in the queue after approval.

[0131] In one possible example, the device also includes a BI cost attribution dashboard module that supports cost drill-down analysis at four levels: enterprise / department / project / user, as well as multi-dimensional visualization based on model tier distribution, budget consumption trends, and degrade event timelines; and supports linear extrapolation prediction of budget exhaustion time based on the current consumption rate.

[0132] In one possible example, this embodiment also provides an enterprise AIAgent platform that includes the above-mentioned model routing device. The model routing device is integrated as a transparent middleware between the user request layer and the large model API call layer, maintaining interface compatibility with both the user side and the large model provider side. It also provides a multi-tenant mode, supporting the mapping of the above-mentioned hierarchical structure to the subscription package system of SaaS customers.

[0133] The routing method provided in this application establishes a hierarchical token budget quota system according to the enterprise's organizational hierarchy (e.g., enterprise > department > project > user). When the token consumption rate at each level falls within the token consumption threshold range, it automatically and smoothly downgrades model call requests from high-quality, high-cost models to low-cost, compliant models. While ensuring business continuity (the downgrade transition is seamless for users), it achieves refined governance of enterprise AI usage costs and a multi-level security network.

[0134] Based on the same inventive concept, this embodiment of the invention also provides a model routing system. Since this system is the same as the system in the method of this embodiment of the invention, and the principle of solving the problem by this system is similar to that of this method, the implementation of this system can refer to the implementation of the method, and the repeated parts will not be described again.

[0135] like Figure 5As shown, the system includes multiple large models 500 and a control unit 501, which is configured to perform: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0136] As an optional implementation, the control unit 501 is specifically configured to perform: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0137] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0138] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0139] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0140] As an optional implementation, the control unit 501 is further configured to perform: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0141] As an optional implementation, the control unit 501 is further configured to perform: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0142] As an optional implementation, the control unit 501 is further configured to perform: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0143] As an optional implementation, the control unit 501 is specifically configured to perform: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0144] Based on the same inventive concept, this embodiment of the invention also provides an electronic device. Since this electronic device is the same as the electronic device in the method of this embodiment of the invention, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0145] like Figure 6 As shown, the electronic device includes a processor 600 and a memory 601. The memory 601 stores programs executable by the processor 600. The processor 600 reads the programs from the memory 601 and performs the following steps: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0146] As an optional implementation, the processor 600 is specifically configured to perform: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0147] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0148] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0149] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0150] As an optional implementation, the processor 600 is further configured to perform: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0151] As an optional implementation, the processor 600 is further configured to perform: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0152] As an optional implementation, the processor 600 is further configured to perform: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0153] As an optional implementation, the processor 600 is specifically configured to perform: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0154] Based on the same inventive concept, this embodiment of the invention also provides a model routing device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0155] like Figure 7 As shown, the device includes: The request initiation module 700 is used to receive request messages initiated by users corresponding to nodes in a tree structure. The tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1. The gear selection module 701 is used to determine the model gear based on the token consumption parameters of each level on the path where the node that initiated the request message is located. The routing module 702 is used to route the request message to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

[0156] As an optional implementation, the gear selection module 701 is specifically used for: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

[0157] As an optional implementation, the token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

[0158] As an optional implementation, each level of the tree structure is configured with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

[0159] As an optional implementation, each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

[0160] As an optional implementation, the device also includes a prompting module specifically used for: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

[0161] As an optional implementation, the device also includes an approval module specifically used for: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

[0162] As an optional implementation, the routing module 702 is further configured to: In response to the model level being the target level, the request message is routed to the local large model for processing.

[0163] As an optional implementation, the routing module 702 is specifically used for: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

[0164] Based on the same inventive concept, this disclosure provides a computer storage medium comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the model routing methods discussed above. Since the principle by which the computer storage medium solves the problem is similar to that of the model routing method, the implementation of the computer storage medium can be found in the implementation of the method, and repeated details will not be elaborated further.

[0165] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0166] Based on the same inventive concept, this disclosure also provides a computer program product, which includes computer program code that, when executed on a computer, causes the computer to perform any of the model routing methods discussed above. Since the principle by which the above-described computer program product solves the problem is similar to that of the model routing method, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0167] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0169] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A model routing method, characterized in that, The method includes: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

2. The method according to claim 1, characterized in that, The step of determining the model level based on the token consumption parameters at each level of the path where the node initiating the request message is located includes: The model level is determined based on the token consumption threshold range that the maximum value of the token consumption parameters at each level along the path of the node that initiated the request message falls into.

3. The method according to claim 1, characterized in that, The token quota configured at the (i+1)th level of the tree structure is less than or equal to the token quota currently available at the ith level; i is an integer greater than or equal to 1 and less than or equal to n.

4. The method according to claim 1, characterized in that, Each level of the tree structure is equipped with an independent token consumption device, and the token consumption device at each level is used to calculate the token consumption parameters of that level.

5. The method according to claim 1, characterized in that, Each level of the tree structure is configured with multiple token consumption threshold ranges corresponding to different model tiers; the quality and / or cost of the large models corresponding to different model tiers are different.

6. The method according to claim 2, characterized in that, The method further includes: In response to the maximum value of the token consumption parameter exceeding the token consumption threshold range, a prompt message is sent to the user, the prompt message including token consumption information and / or model level switching information.

7. The method according to claim 1, characterized in that, The method further includes: In response to the model level being the target level, the request message is added to the pending queue, and a budget increase application message is pushed to the target approval user; In response to approval, the model profile is updated, and the request message is routed to the larger model corresponding to the updated model profile for processing in the routing middleware layer.

8. The method according to claim 1, characterized in that, The method further includes: In response to the model level being the target level, the request message is routed to the local large model for processing.

9. The method according to claim 1, characterized in that, The step of routing the request message to the large model corresponding to the model tier for processing at the routing middleware layer includes: In the large model corresponding to the model level, the request message is routed at the routing middleware layer to the large model that meets the predefined quality lower limit and has the lowest cost for processing.

10. A model routing system, characterized in that, The system comprises multiple large models and control units, which are configured to perform: Receive request messages initiated by users corresponding to nodes in a tree structure, wherein the tree structure includes n levels, each level has m nodes, where n is an integer greater than or equal to 2, and m is an integer greater than or equal to 1; The model level is determined based on the token consumption parameters at each level of the path where the node that initiated the request message is located. The request message is routed to the large model corresponding to the model level for processing in the routing middleware layer. The switching of the model level occurs in the routing middleware layer. When the model level is switched, the client API interface protocol remains unchanged and the established connection remains valid.

11. An electronic device, characterized in that, The device includes a processor and a memory for storing a program executable by the processor, and the processor for reading the program in the memory and executing the steps of the method according to any one of claims 1 to 9.

12. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 9.

Images