A resource-constrained-oriented large language model online routing method and system

Abstract

This invention provides an online routing method and system for large language models under resource constraints. The method includes: acquiring the context features of the task to be processed and the current remaining resource status; based on the context features, estimating the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed; determining dynamic scheduling parameters according to the remaining resource status, and calculating the routing score of each candidate model by combining the dynamic scheduling parameters, success benefit representation value, and long-tail resource consumption evaluation value; and selecting a target model from among the candidate models to execute the task to be processed based on the routing score. This invention can quantify and avoid the budget overrun risk that may be caused by a single extremely long output before invocation, enabling the model selection strategy to adaptively respond to changes in remaining resources, thereby more efficiently transforming the limited budget into more verified successful tasks under the hard constraint of path-level budget.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an online routing method and system for resource-constrained large language models. Background Technology

[0002] With the rapid development of large language models, multimodal models, and autonomous intelligent agent technologies, large language model inference resources have become an industrialized productive force that can be allocated on demand. In practical applications, in order to achieve a balance between model capabilities and operating costs, developers typically deploy routing systems that include multiple candidate models. They use resource metrics such as API calls, token consumption limits, or GPU computing power quotas to schedule and allocate resources online for tasks of varying complexity.

[0003] Current online routing technologies primarily employ static selection based on human preferences, cascading calls of multiple models, or contextual bandit methods based on historical win rates. These approaches attempt to maximize the overall response quality or task success rate of the system by training routers offline or learning the expected return and average cost of each model online, within predefined resource constraints, through screening candidate models or adjusting retry strategies.

[0004] However, existing technologies often overlook the tail cost risk arising from the high degree of randomness in the length of generative outputs when handling large language model inference tasks. In real-world engineering systems, there are usually hard constraints on path-level budgets, where service is stopped upon the first overrun. However, traditional methods cannot accurately detect and avoid budget overruns that may occur from a single extremely long output before the call, resulting in the limited total budget being unexpectedly exhausted by a single high-risk call. Summary of the Invention

[0005] This invention provides an online routing method and system for large language models with resource constraints, which solves the problems in the prior art that cause path-level budget out-of-bounds errors and the finite total budget to be unexpectedly exhausted by a single high-risk call due to ignoring the tail cost risk of output length, and maximizes the number of verified successful tasks under a limited budget.

[0006] This invention provides an online routing method for resource-constrained large language models, comprising: Obtain the contextual characteristics of the task to be processed and the current status of remaining resources; Based on the aforementioned contextual features, the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed are estimated respectively. Based on the remaining resource status, dynamic scheduling parameters are determined, and combined with the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value, the routing score of each candidate model is calculated. Based on the routing score, a target model is selected from the candidate models to execute the task to be processed.

[0007] According to the resource-constrained online routing method for large language models provided by the present invention, the step of determining dynamic scheduling parameters based on the remaining resource status, and calculating the routing score of each candidate model by combining the dynamic scheduling parameters, the success reward representation value, and the long-tail resource consumption evaluation value, includes: The basic task share is calculated based on the remaining available resources before the call and the number of remaining tasks, including the current task. Obtain a rhythm coefficient for adjusting the rate of resource consumption, and combine the rhythm coefficient with the basic task share to obtain the local resource share of the current task; Obtain the resource shadow price, which represents the current overall resource scarcity level, as the dynamic scheduling parameter; Divide the long-tail resource consumption assessment value by the local resource share, and then multiply it by the resource shadow price as a risk penalty item; The unified routing score for each candidate model is obtained by subtracting the risk penalty term from the success reward representation value.

[0008] According to the resource-constrained online routing method for large language models provided by the present invention, the step of estimating the long-tail resource consumption evaluation value of each candidate model for the task to be processed based on the context features includes: The threshold grid is determined based on the effective output length quantiles of the historical call logs, and an explicit overflow tail boundary is set. The threshold grid contains multiple incremental length thresholds. Using a multi-threshold survival assessment branch, combined with the aforementioned contextual features, the probability of exceeding each candidate model's output length by more than each length threshold is estimated, and the confidence value of the probability of exceeding each length threshold is calculated. Along the increasing direction of the length threshold, monotonic projection processing is performed on the over-confidence value corresponding to each length threshold, so that the over-confidence value of the over-confidence value is monotonically non-increasing with the increase of the length threshold. In the threshold grid, the minimum length threshold that satisfies the condition that the confidence value on the probability of exceeding the threshold is less than or equal to the preset risk limit is selected as the safe output length; A corresponding security resource proxy value is generated based on the security output length, which serves as the long-tail resource consumption evaluation value.

[0009] According to the resource-constrained online routing method for large language models provided by the present invention, the step of estimating the success reward representation value of each candidate model for the task to be processed based on the context features includes: The context features are projected to generate universal features shared across models. These universal features are used to characterize the current task difficulty and context complexity. From the context features, model-specific features for each candidate model are extracted. These model-specific features are used to characterize the ability attributes of the corresponding candidate model to process the task to be processed. By integrating the general features and the model-specific features, the success rate estimate and the corresponding uncertainty metric value of each candidate model when performing the task to be processed are calculated. Based on the success rate estimate and the uncertainty metric, the confidence value of the success return for each candidate model is calculated and determined, and the confidence value of the success return is used as the success return characterization value.

[0010] According to the resource-constrained online routing method for large language models provided by the present invention, the step of obtaining the contextual features of the task to be processed includes: Obtain the original task text, task retry history, tool output feedback, and preset prompt templates, and combine them to form the current prompt information; Feature extraction is performed on the current prompt information to obtain context representation features, and the retry depth of the current task, the historical model used in the previous round, the reward feedback of the previous round, and the resource consumption value of the previous round are obtained. The contextual features are obtained by combining the contextual representation features, the retry depth, the historical model, the reward feedback, and the resource consumption value.

[0011] According to the resource-constrained online routing method for large language models provided by the present invention, the step of selecting a target model from each candidate model to execute the task to be processed based on the routing score includes: before calling the target model, if the available resource value in the current remaining resource state is greater than zero, triggering the target model to execute the task to be processed; After executing the target model, the actual resource consumption of this call is obtained and added to the historical total consumption to obtain the updated cumulative resource consumption. When the cumulative resource consumption is detected to exceed the preset total resource limit for the first time, subsequent model call operations for the current task flow are blocked.

[0012] According to the resource-constrained online routing method for large language models provided by the present invention, after the step of selecting the target model from each candidate model based on the routing score to execute the task to be processed, the method further includes: The system receives the task response result and actual resource consumption returned by the target model for the task to be processed; verifies the task response result through a preset verification mechanism and generates a corresponding verification reward result; when the verification reward result indicates that the task is successful, the system ends the task to be processed and triggers the processing flow of the next task; when the verification reward result indicates that the task is unsuccessful, the system updates the cumulative resource consumption status and call log according to the actual resource consumption, and triggers the task retry decision process.

[0013] According to the resource-constrained online routing method for large language models provided by the present invention, the step of triggering the task retry decision process includes: Determine whether the current retry depth of the task to be processed is less than the preset maximum retry depth; When the retry depth is less than the maximum retry depth, a dynamic retry threshold is calculated based on the retry depth, the accumulated resource consumption of the task to be processed, and the local resource share. The dynamic retry threshold is positively correlated with the retry depth and the degree of task resource overrun. Determine whether the highest unified routing score among the current candidate models is greater than or equal to the dynamic retry threshold. If yes, construct the prompt information for the next round based on the failure history and re-execute the steps of calculating the routing score and selecting the target model. If no, stop the retry action of the pending task and reserve the unused resource share for subsequent tasks.

[0014] According to the resource-constrained online routing method for large language models provided by the present invention, after the step of selecting a target model from each candidate model to execute the task to be processed based on the routing score, the method further includes: obtaining the actual resource consumption of the target model in executing the task to be processed; calculating the deviation of the actual resource consumption from the current task step budget share, and adjusting the resource shadow price based on the deviation using an exponential gradient update rule; calculating the exponential moving average of recent historical actual resource consumption, comparing the ratio of the exponential moving average to the basic task share, and updating the rhythm coefficient according to the ratio.

[0015] The resource-constrained online routing method for large language models provided by the present invention further includes, before the step of obtaining the context features of the task to be processed and the current remaining resource status: Obtain a task flow containing multiple task sequences, and parse out the task that needs to be executed from the task flow according to a preset execution order, as the task to be processed; Obtain the preset total resource limit, and directly set the initial available resource value to the total resource limit to generate the current remaining resource status; Obtain a set of available candidate models, and obtain warm-up data based on historical call logs or warm-up execution results for the set of candidate models; The preheating data is used to initialize the underlying parameters so that, based on the context features, the success benefit representation value and the long-tail resource consumption evaluation value of each candidate model in the candidate model set for the task to be processed can be estimated respectively.

[0016] This invention also provides an online routing system for large language models oriented towards resource constraints, comprising: The initialization module is used to obtain the context characteristics of the task to be processed and the current status of remaining resources; The risk prediction module is used to predict the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed based on the context features. The routing scoring module is used to determine dynamic scheduling parameters based on the remaining resource status, and to calculate the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value. The task execution module is used to select the target model from each candidate model based on the routing score and execute the task to be processed.

[0017] According to the resource-constrained online routing system for large language models provided by the present invention, the routing scoring module is configured to: calculate the basic task share based on the remaining available resources before the call and the number of remaining tasks including the current task; obtain a rhythm coefficient for adjusting the resource consumption rate, and combine the rhythm coefficient with the basic task share to obtain the local resource share of the current task; obtain the resource shadow price for characterizing the current overall resource tension, as the dynamic scheduling parameter; divide the long-tail resource consumption assessment value by the local resource share, and then multiply it by the resource shadow price as a risk penalty term; subtract the risk penalty term from the success reward characterization value to obtain a unified routing score for each candidate model.

[0018] According to the resource-constrained large language model online routing system provided by the present invention, the risk prediction module is used for: The threshold grid is determined based on the effective output length quantiles of the historical call logs, and an explicit overflow tail boundary is set. The threshold grid contains multiple incremental length thresholds. Using a multi-threshold survival assessment branch, combined with the aforementioned contextual features, the probability of exceeding each candidate model's output length by more than each length threshold is estimated, and the confidence value of the probability of exceeding each length threshold is calculated. Along the increasing direction of the length threshold, monotonic projection processing is performed on the over-confidence value corresponding to each length threshold, so that the over-confidence value of the over-confidence probability is monotonically non-increasing with the increase of the length threshold. In the threshold grid, the minimum length threshold that satisfies the condition that the confidence value on the probability of exceeding the threshold is less than or equal to the preset risk limit is selected as the safe output length; A corresponding security resource proxy value is generated based on the security output length, which serves as the long-tail resource consumption evaluation value.

[0019] According to the resource-constrained large language model online routing system provided by the present invention, the risk prediction module is used for: The context features are projected to generate universal features shared across models. These universal features are used to characterize the current task difficulty and context complexity. From the context features, model-specific features for each candidate model are extracted. These model-specific features are used to characterize the ability attributes of the corresponding candidate model to process the task to be processed. By integrating the general features and the model-specific features, the success rate estimate and the corresponding uncertainty metric value of each candidate model when performing the task to be processed are calculated. Based on the success rate estimate and the uncertainty metric, the confidence value of the success return for each candidate model is calculated and determined, and the confidence value of the success return is used as the success return characterization value.

[0020] According to the resource-constrained large language model online routing system provided by the present invention, the initialization module is used for: Obtain the original task text, task retry history, tool output feedback, and preset prompt templates, and combine them to form the current prompt information; Feature extraction is performed on the current prompt information to obtain context representation features, and the retry depth of the current task, the historical model used in the previous round, the reward feedback of the previous round, and the resource consumption value of the previous round are obtained. The contextual features are obtained by combining the contextual representation features, the retry depth, the historical model, the reward feedback, and the resource consumption value.

[0021] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the resource-constrained large language model online routing method as described above.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the resource-constrained large language model online routing method as described above.

[0023] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the resource-constrained large language model online routing method as described above.

[0024] This invention provides an online routing method and system for large language models with resource constraints. By acquiring the contextual features of the task to be processed and the current state of remaining resources, and based on the contextual features, it estimates the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the current task. The long-tail resource consumption evaluation value captures the tail risk of the large language model's output length, thus identifying the possibility of a single extremely high cost before invocation. Dynamic scheduling parameters are determined based on the remaining resource state, and combined with the dynamic scheduling parameters, success benefit representation value, and long-tail resource consumption evaluation value, a routing score for each candidate model is calculated. This allows the routing score to dynamically adjust the trade-off weight between success benefit and long-tail resource consumption according to the scarcity of remaining resources. Finally, the target model is selected from the candidate models to execute the task based on the routing score. Through this method, this invention can quantify and avoid the budget overrun risk that may be caused by a single extremely long output before invocation, overcoming the limitations of traditional methods that rely solely on average cost or expected consumption and cannot perceive tail risk. Simultaneously, it enables the model selection strategy to adaptively respond to changes in remaining resources, thereby more efficiently transforming a limited budget into more validated successful tasks under the hard constraint of path-level budget. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the resource-constrained online routing method for large language models provided by this invention.

[0027] Figure 2This is a schematic diagram of the budget shadow price and rhythm closed loop provided by the present invention.

[0028] Figure 3 This is a schematic diagram of the threshold survival cost head provided by the present invention.

[0029] Figure 4 This is a schematic diagram of the retry stop gating provided by the present invention.

[0030] Figure 5 This is a single-task multi-round routing flowchart provided by the present invention.

[0031] Figure 6 This is one of the structural diagrams of the resource-constrained large language model online routing system provided by the present invention.

[0032] Figure 7 This is the second schematic diagram of the structure of the resource-constrained large language model online routing system provided by the present invention.

[0033] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] In existing large language model service orchestration scenarios, to strike a balance between model capabilities and operational costs, a routing system containing multiple candidate models is typically deployed. These candidate models may include small models with fast response times but low costs, and large models with strong inference capabilities but high costs. In practical applications, the system needs to make an online decision on which model to select to execute the current task based on task difficulty, context complexity, and remaining resources, and decide whether to retry if the task fails.

[0036] Currently available routing methods can be mainly categorized as follows: The first type is fixed or static routing. The system pre-defines a fixed model selection strategy, such as using the same strongest model for all tasks, or following simple rules, such as using a smaller model if the input length is less than a threshold, and a larger language model otherwise. This type of method is simple to implement, but it cannot be dynamically adjusted according to the actual difficulty of the task and real-time feedback. It is easy to waste expensive model resources on simple tasks, or to cause repeated failures on complex tasks due to the selection of a weak model.

[0037] The second type is offline-trained routers. By collecting historical task data, such as task features, model response quality, and cost, a classifier or ranking model is trained to predict the most suitable model for each task. The performance of this type of method heavily relies on the quality and coverage of the offline data and cannot adapt to changes in the distribution of online tasks or dynamic adjustments to budget constraints. When a new model is added or the model price changes, data needs to be collected and retrained.

[0038] The third category is cascading or serial invocation methods. The system first invokes a low-cost model; if its answer confidence falls below a certain threshold, it then invokes a more expensive model for secondary inference. This type of method can balance cost and quality to some extent, but its decision-making logic is local and lacks awareness of global remaining budget and tail cost risks. In addition, cascading invocation has high latency and is not suitable for scenarios sensitive to response speed.

[0039] The fourth category is online routing based on contextual bandit. The system maintains a linear or non-linear payoff prediction function for each model and updates it in real time based on contextual features. Some works also introduce knapsack constraints, using average cost or expected consumption as a limiting condition. However, these methods typically assume that the cost of each call can be precisely known before the decision is made, or that historical average costs are used as an alternative. They cannot handle the risk posed by the high randomness of the output length of large language models—that is, a single call may produce a far greater number of output tokens than expected, resulting in actual costs several times higher than the average. In scenarios with hard constraints on path-level budgets, this tail risk may cause the system to exhaust its entire budget early in the task flow due to a single abnormal output, causing all subsequent tasks to be interrupted.

[0040] The fifth type is the fixed retry strategy. When a model's response is deemed a failure by the validator, the system simply retryes the same model or switches to a stronger model. The number of retries is usually a preset fixed value (e.g., 3 times). This type of method does not consider the cost already consumed by the current task, the remaining budget, or the marginal benefit of continuing to retries. When the budget is tight, multiple invalid retries will crowd out resources for subsequent tasks, thus reducing the overall number of successful tasks.

[0041] In summary, existing technologies generally lack the ability to anticipate tail cost risks before invocation, the ability to control the overall budget tightness in a closed loop, and the ability to adaptively retry and stop based on marginal benefits. These deficiencies are particularly prominent in real-world deployment scenarios with limited budgets, continuous task flows, and path-level stopping rules.

[0042] To address the aforementioned technical problems, this invention provides an online routing method and system for large language models with resource constraints. Before each model invocation, the method estimates both the success benefit representation value and the long-tail resource consumption assessment value for each candidate model based on the contextual features of the current task. Simultaneously, a dynamic scheduling parameter is determined based on the current remaining resource status, representing the current resource scarcity. Subsequently, the success benefit representation value, the long-tail resource consumption assessment value, and the dynamic scheduling parameter are combined to calculate a unified routing score, and the model with the highest score is selected to execute the task. This method enables routing decisions to balance the high benefits of success with avoiding tail resource overruns in real time, automatically favoring low-cost, low-risk models when resources are scarce. Furthermore, this invention introduces an adaptive retry and stopping mechanism. After a task failure, based on a comparison between the current score and a dynamic threshold, it decides whether to continue retrying or abandon the current task, thereby avoiding ineffective invocations with low marginal benefits. The entire method continuously updates the success benefit prediction model, the long-tail cost prediction model, and the dynamic scheduling parameter through an online feedback loop, achieving adaptive control of the task flow and budget, significantly increasing the number of verified successful task completions under limited budgets.

[0043] Before describing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.

[0044] A pending task refers to a unit of work that needs to be processed by the large language model. It can be a question-and-answer question, a piece of code that needs to be fixed, a document that needs to be reviewed, a tool call instruction that needs to be executed, or any information processing request that can be completed by the large language model or multimodal model. Each pending task can be tried once or multiple times (i.e., retried) until the validator determines that the task is successful or the stopping condition is met.

[0045] Contextual features: Feature vectors describing the current call state. These can be extracted from information such as the original task text, task retry history, tool output feedback, preset prompt templates, current retry depth, the model identifier used in the previous round, the reward feedback obtained in the previous round, and the actual resource consumption in the previous round. Contextual features can include numerical features, categorical features, and embedded features, and are used as inputs for success reward prediction and long-tail resource consumption prediction.

[0046] Remaining resource status: This refers to the amount of resources that the system has not yet consumed and can be used for subsequent tasks at the current invocation time. Resources can be monetary costs, token quantity, inference latency, GPU computing power quota, carbon emission estimates, or a weighted combination of the above resources. Remaining resource status typically includes the remaining budget value and the number of remaining pending tasks.

[0047] Success Reward Representation: This value quantifies the potential reward of successfully executing a task using a candidate model given the current context features and being validated by the validator. This value can be calculated based on an estimate of the success rate and its uncertainty, for example, using an upper confidence bound. This reflects the model's historical success rate and encourages exploration of models with higher uncertainty.

[0048] Long-tail resource consumption assessment: Used to estimate the extreme resource consumption that a candidate model might generate when executing the current task before invocation, i.e., tail risk. Unlike traditional average cost estimation, long-tail resource consumption assessment focuses on the right-tail distribution of output length or cost, such as the expected excess cost exceeding a certain safety boundary or the upper bound of the safety cost. Its purpose is to identify and avoid high-risk behaviors that may unexpectedly exhaust the remaining budget in a single invocation.

[0049] Dynamic scheduling parameters: Adjustable parameters determined based on the current remaining resource status, used to adjust the weighting of the trade-off between pursuing high success gains and controlling resource consumption risks in routing scoring. When resources are abundant, this parameter takes a lower value, and routing decisions are more biased towards success gains; when resources are scarce or there is a recent overspending, this parameter takes a higher value, and routing decisions are more biased towards low-cost, low-risk models.

[0050] Routing Score: A comprehensive score calculated for each candidate model, used to measure the overall utility of invoking the model under the current resource conditions. This score typically combines the success benefit representation value and the long-tail resource consumption evaluation value through dynamic scheduling parameters. For example, the success benefit representation value minus the long-tail resource consumption evaluation value weighted by the dynamic scheduling parameters. The higher the score, the more suitable the model is for executing the current task.

[0051] Base task share: The average resource allocation calculated based on the current total remaining resources and the number of remaining tasks, usually expressed as remaining resources divided by the number of remaining tasks. This value serves as the basis for allocating the current task budget.

[0052] Rhythm Coefficient: A multiplier factor used to dynamically adjust the base task share. Its value is updated online based on the deviation of the recent exponential moving average of actual resource consumption from the base task share. A rhythm coefficient greater than 1 indicates that the current task receives a larger share of resources, while a coefficient less than 1 indicates that the current task receives a smaller share of resources.

[0053] Local resource share: The upper limit of resources actually allowed to be used in the current task, which is obtained by multiplying the base task share by the rhythm coefficient. If necessary, a lower limit threshold can be set to avoid a share of zero or negative.

[0054] Resource shadow price: A specific implementation of a dynamic scheduling parameter used to characterize the marginal cost of current resource scarcity. The shadow price is low when remaining resources are abundant, and rises when remaining resources are scarce or there has been a recent overspending. The shadow price can be adjusted in a closed-loop manner through online learning based on the deviation of actual resource consumption from the step budget.

[0055] Risk penalty item: An item in routing score that penalizes the risk of long-tail resource consumption. The typical form is the resource shadow price multiplied by the long-tail resource consumption assessment value and divided by the local resource share. It is used to suppress high-risk calls when the budget is tight.

[0056] Unified Route Score: A comprehensive score obtained by combining the success reward representation value with the risk penalty term, for example, the success reward representation value minus the risk penalty term. This score can be directly used for model ranking and selection.

[0057] Threshold grid: A set of incremental numerical thresholds used to discretize output length or resource consumption. The threshold grid can be pre-determined based on the effective output length quantiles in historical logs, and an explicit overflow tail boundary is set to cover cases exceeding the grid range.

[0058] Overflow tail boundary: When the actual output length exceeds the maximum threshold in the threshold grid, a preset boundary value is used to conservatively estimate resource consumption, ensuring that the long tail risk is not underestimated.

[0059] Over-probability upper confidence value: For a certain threshold, the upper confidence bound of the probability that the system's estimated output length is greater than that threshold. It is an optimistic estimate of the over-probability and is used to provide a safety margin under uncertainty.

[0060] Safe output length: Within the threshold grid, the minimum length threshold that satisfies the condition that the probability of exceeding the confidence value does not exceed a preset risk limit. The safe output length is used to calculate a conservative resource consumption proxy value.

[0061] Success rate estimate: The probability that a candidate model will successfully complete the task under the current context features, as predicted by the system. It is usually obtained by modeling based on historical data and context features.

[0062] Uncertainty measures: Indicators that quantify the uncertainty of success rate estimates, such as the variance of the estimate or the radius of the confidence interval. Combined with success rate estimates, they can form upper confidence bounds, used to balance exploitation and exploration.

[0063] Confidence level of success payoff: The sum of the success rate estimate and the uncertainty measure, serving as a preferred implementation of the success payoff representation, used to encourage high-probability success models while maintaining the exploration of uncertain models.

[0064] Original task text: The core text content of the task to be processed, such as questions raised by users, code segments that need to be fixed, sentences that need to be translated, etc.

[0065] Task retry history: Records of the current task in the previous few attempts, including the model used in each attempt, the model response obtained, the reward feedback given by the validator, and the actual amount of resources consumed.

[0066] Tool output feedback: When the large language model calls external tools, such as search engines, code interpreters, database queries, etc., the execution results or error information returned by the tools will be appended to the context for subsequent model calls.

[0067] Preset prompt templates: These templates are used to organize information such as original tasks, historical messages, and tool outputs into a format that the model can understand. They can include system commands, example dialogues, task descriptions, etc.

[0068] Cumulative resource consumption: The total resources consumed by all completed model calls from the start of the task flow to the current moment. This value is compared with the total resource limit to determine whether to trigger path-level budget stop.

[0069] Task response result: The generated content returned by the large language model for the current prompt can be in the form of text, code, structured data, or tool call parameters.

[0070] Actual resource consumption: The actual resource consumption of a single model call, including the number of input tokens, the number of output tokens, API call costs, inference time, etc. This value is usually only known after the model returns a response.

[0071] Validation reward results: The validator outputs a feedback signal after judging the task response results. This signal can be binary, continuous score, or classification label. This signal is used to update the success reward prediction model and retry decisions.

[0072] Retry depth: The number of attempts already made or the remaining number of attempts for the current task. The retry depth for the first call is usually recorded as 0 or 1, and it increases with each subsequent retry after a failure.

[0073] Dynamic retry threshold: A dynamic threshold used to determine whether it is worthwhile to retry a failed task. This threshold can be calculated based on factors such as the current retry depth, the amount of resources consumed by the current task, and the local resource share. As the number of retries increases or the degree of resource overrun increases, this threshold usually increases, making the system more likely to abandon tasks with low marginal returns.

[0074] Warm-up data: During system startup or the initial stage of model pool updates, a small amount of historical logs or warm-up task execution results are used to initialize the successful revenue model and the long-tail resource consumption model. Warm-up data can come from historical call logs, public benchmark tests, simulation traffic, gray-scale traffic, or manually labeled samples, helping the system avoid blind exploration during the cold start phase.

[0075] The resource-constrained online routing method for large language models provided in this invention can be implemented by a model gateway, API gateway, or large language model orchestration server deployed on a cloud service platform, or by a model scheduling middleware, intelligent agent platform, or edge computing node running within an enterprise. For ease of description, this implementation entity will be collectively referred to as the routing system. The routing system typically includes at least one processor and memory, and communicates with the candidate large language model server and validator through a network interface.

[0076] Figure 1 This is a flowchart illustrating the resource-constrained online routing method for large language models provided by the present invention. The method includes the following: Step 101: Obtain the contextual features of the task to be processed and the current status of remaining resources.

[0077] The routing system first retrieves a task from the task queue. This task can be a natural language problem, a piece of code to be fixed, a multi-step instruction to be executed, or any information request suitable for processing by a large language model. Simultaneously, the routing system reads the currently stored remaining resource status, which records the amount of resources that have not yet been consumed up to the current moment, such as remaining monetary budget, remaining callable tokens, remaining time quota, or computing power quota. In practice, the remaining resource status can be stored as a numerical variable in the routing system's memory or database, and this variable is read by the routing system before each call. Furthermore, the routing system extracts a set of contextual features based on the current task itself and its historical records, such as whether the task has been tried several times before, and the input and output content of each attempt.

[0078] Contextual features may include, but are not limited to: the vectorized representation of the task text, the length of the task input, the domain category to which the task belongs, the number of retries for the current attempt, the model identifier used in the previous call, and the validation result of the previous call. These features form the basis for subsequent decision-making.

[0079] Step 102: Based on the context features, estimate the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed.

[0080] The routing system maintains a set of candidate models, each with its own price parameters and service capabilities. For the current task, the routing system performs two predictions for each candidate model, utilizing extracted contextual features.

[0081] The first type of prediction is a success reward representation, which quantifies the potential reward for the model's response to pass the validator and be judged as successful after calling the current model. One implementation is for the routing system to use an online learning model to predict the success rate, and add a reward term related to uncertainty, thereby encouraging high-success-rate models while maintaining attempts at less explored models.

[0082] The second type of prediction is the long-tail resource consumption assessment, which focuses on the extreme resource consumption risks that model calls may generate, rather than the average consumption. Unlike traditional average cost prediction, the long-tail resource consumption assessment aims to capture the right-tail distribution characteristics of output length or resource consumption, enabling the routing system to identify model call scenarios with low average cost but occasional extremely high cost before the call. The routing system can provide a conservative upper bound estimate of resource consumption based on the output length distribution characteristics of different models in historical calls, combined with the current context characteristics.

[0083] Step 103: Determine the dynamic scheduling parameters based on the remaining resource status, and calculate the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value.

[0084] After obtaining the success benefit representation value and long-tail resource consumption evaluation value for each candidate model, the routing system needs to decide how to balance pursuing success with controlling costs based on the current resource scarcity.

[0085] Therefore, the routing system first determines a dynamic scheduling parameter based on the acquired remaining resource status. An intuitive implication of this parameter is: when remaining resources are plentiful, the parameter value is lower, and the routing system can focus more on success benefits; when remaining resources are scarce or there is a recent trend of overspending, the parameter value increases, and the routing system should focus more on avoiding high-risk calls.

[0086] In practical implementation, this parameter can be calculated through a preset functional relationship or adjusted online using a feedback loop. Subsequently, the routing system calculates the routing score for each candidate model based on a preset combination of the dynamic scheduling parameter, the success benefit representation value, and the long-tail resource consumption evaluation value. A typical combination is: Routing Score = Success Benefit Representation Value - Dynamic Scheduling Parameter × Long-Tail Resource Consumption Evaluation Value. This score reflects the overall utility of calling the model under the current resource constraints.

[0087] Step 104: Select the target model from the candidate models based on the routing score and execute the task to be processed.

[0088] After calculating the routing scores of all candidate models, the routing system typically selects the model with the highest score as the target model for the current call. In some implementations, if the highest score is below a certain preset threshold, the routing system may choose to abandon the call, for example, when the risk of all models is too high. However, in general, the routing system will select the model with the optimal score.

[0089] Subsequently, the routing system sends a request containing the current prompt to the API endpoint or local inference engine of the selected target model, and waits for the model to return a response and actual resource usage information, such as the number of input tokens, the number of output tokens, and the response time. After the call is completed, the routing system records the actual amount of resources consumed and adds it to the cumulative consumption to update the remaining resource status for subsequent tasks.

[0090] Simultaneously, the routing system passes the model's response to the validator, for example, by comparing it with the standard answer, performing unit tests, or conducting rule checks, to obtain the validation result. This result is used for subsequent successful reward model updates and possible task retry decisions. If the validation result is successful, the current task ends, and the routing system continues to process the next task; if it fails, the routing system can decide whether to retry based on the set strategy. During the retry, steps 101 to 104 above will be re-executed based on the updated context features after the failure.

[0091] The online routing method for large language models with resource constraints provided in this invention provides real-time basis for subsequent decisions by acquiring the contextual features of the task to be processed and the current state of remaining resources. Based on the contextual features, it estimates the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the current task. The long-tail resource consumption evaluation value can capture the tail risk of the output length of the large language model, thereby identifying the possibility of a single extremely high cost before invocation. Dynamic scheduling parameters are determined according to the state of remaining resources, and the routing score of each candidate model is calculated by combining the dynamic scheduling parameters, success benefit representation value, and long-tail resource consumption evaluation value. This allows the routing score to dynamically adjust the trade-off weight between success benefit and long-tail resource consumption according to the tension of remaining resources. Finally, the target model is selected from the candidate models to execute the task based on the routing score. In this way, this invention can quantify and avoid the budget overrun risk that may be caused by a single extremely long output before invocation, overcoming the limitation of traditional methods that only rely on average cost or expected consumption and cannot perceive tail risk. At the same time, it enables the model selection strategy to adaptively respond to changes in remaining resources, thereby more efficiently transforming the limited budget into more verified successful tasks under the hard constraint of path-level budget.

[0092] Furthermore, before the routing system begins processing the task flow, initialization configuration is required to prepare for subsequent online routing decisions.

[0093] The routing system first obtains a task flow containing multiple task sequences. This task flow can originate from user-submitted Q&A requests in batches, a list of defects to be fixed in a code repository, a queue of documents to be processed in an enterprise knowledge base, or any continuous set of tasks that need to be processed sequentially by a large language model. The routing system parses the tasks currently to be executed from the task flow according to a preset execution order, ensuring that each task in the task flow can be scheduled and processed sequentially.

[0094] The routing system simultaneously obtains a preset total resource limit, which can be a monetary cost limit, a token quantity limit, an inference latency limit, a GPU computing power quota limit, or a weighted combination of the above resources. The routing system directly sets the initial available resource value to this total resource limit, generating the current remaining resource status. This means that before the task flow begins processing, all budgeted resources are in an available state. With each subsequent model call's actual consumption, the remaining resource status will be gradually deducted and updated in real time.

[0095] The routing system also acquires a set of available candidate models. This set can include multiple large language models of varying sizes and pricing, such as small models with fast response times but low costs, and large models with strong inference capabilities but high costs. Each candidate model is accompanied by information such as its API address, input / output price, context window limit, maximum output limit, and service status. The routing system maintains this set of candidate models as a selection space for subsequent routing decisions.

[0096] The routing system obtains warm-up data based on historical call logs or warm-up execution results for the candidate model set. Historical call logs can come from real call records accumulated during previous system operation, containing the contextual characteristics of each call, the model used, the actual resource consumption, and the success or failure label given by the validator. Warm-up execution results can come from a small number of test tasks executed before the system's official launch, obtaining initial call feedback by actively calling each candidate model to process several representative tasks. If the system lacks historical logs, publicly available benchmark test data, simulated traffic data, grayscale traffic data, or manually labeled samples can also be used as warm-up data.

[0097] The routing system utilizes preheating data for underlying parameter initialization. Specifically, it uses preheating data to initialize model parameters in the success reward learning module, such as the shared feature projection matrix and the personalized weight vectors of each candidate model, enabling the system to have basic success rate prediction capabilities during the cold start phase. Simultaneously, it uses preheating data to initialize the threshold grid and exceedance probability estimates for each threshold in the tail cost learning module, allowing the system to perform a preliminary assessment of the output length tail risk even before online call records are available. By introducing preheating data, the routing system can obtain reasonable initial parameter values ​​before the task flow officially begins processing, avoiding blind exploration due to a lack of historical information during the cold start phase, thereby improving online learning efficiency and the quality of early routing decisions.

[0098] By acquiring the task flow and total resource limit, the routing system clearly defines the scope of tasks to be processed and the upper limit of available resources at the beginning of startup, providing a global constraint framework for each subsequent routing decision. By acquiring the candidate model set and preheating data and using the preheating data to initialize the underlying parameters, the routing system gains the ability to make preliminary estimates of the success probability and tail cost of each candidate model during the cold start phase, avoiding blind exploration and low quality of early decisions due to lack of historical information.

[0099] In this embodiment, obtaining the context features of the task to be processed in step 101 above is specifically achieved in the following way.

[0100] The routing system first acquires the original task text, task retry history, tool output feedback, and preset prompt templates. The original task text is the core content of the task to be processed, such as a user-submitted question, a code segment that needs fixing, or a sentence that needs translation. The task retry history records information generated in the previous attempts of the current task, including the model used in each attempt, the model's returned response, and the success or failure label given by the validator. Tool output feedback refers to the execution results or error messages returned by external tools when the large language model invoked them in previous attempts. The preset prompt template is a pre-designed prompt structure used to organize the original task text, failure responses from the retry history, tool output feedback, and other information in a format understandable to the target model. The routing system concatenates the above information according to the preset prompt template to form the current prompt information. This current prompt information contains all relevant context of the task and serves as the input basis for the model's inference.

[0101] The routing system then extracts features from the current prompt information to obtain contextual representation features. Feature extraction uses an embedding model to convert the textual prompt information into a dense vector representation, which captures the semantic information and structural features of the prompt. The routing system also obtains the retry depth of the current task, the historical model used in the previous round, the reward feedback for the previous execution, and the resource consumption value for the previous round. Specifically, the retry depth records the number of times the current task has been attempted; the historical model used in the previous round identifies which candidate model was used in the last call; the reward feedback for the previous execution records the success or failure result given by the validator after the last call; and the resource consumption value for the previous round records the actual number of tokens or the amount of fees consumed in the last call.

[0102] The routing system combines contextual representation features, retry depth, historical models, reward feedback, and resource consumption values ​​to obtain contextual features. These contextual features can be multi-dimensional vectors, containing both semantic information from the prompt text and structured information from the call state. This fusion of multi-source information allows the contextual features to comprehensively reflect the current call state, enabling the routing system to understand not only the difficulty and type of the task itself, but also the historical progress and resource consumption of the current attempt.

[0103] Through the above methods, the routing system obtains comprehensive features that include both task semantic information and call status information, making the prediction of success benefits and the assessment of long-tail resource consumption based on these features more accurate and comprehensive.

[0104] In this embodiment, the estimation of the long-tail resource consumption evaluation value of each candidate model for the task to be processed based on contextual features in step 102 above is specifically implemented in the following way: The routing system first determines the threshold grid based on the valid output length quantiles of the historical call logs and sets an explicit overflow tail boundary. The threshold grid contains multiple incremental length thresholds, covering a range from normal output length to extreme output lengths. The overflow tail boundary is used to handle situations where the actual output length exceeds the maximum threshold of the threshold grid.

[0105] like Figure 2As shown, the tail cost learning module in this embodiment of the invention uses a threshold survival cost head to predict the long-tail resource consumption assessment value. This module takes cost features as input, processes them through multiple threshold branches, and outputs a safe cost proxy value. The routing system utilizes multi-threshold survival assessment branches, combined with contextual features, to predict the probability of the output length exceeding each length threshold for each candidate model, and calculates the confidence value on the probability of exceeding each length threshold. Specifically, the routing system maintains an online assessment branch for each candidate model and each length threshold, transforming the output length prediction into multiple binary classification problems to reduce the prior assumptions required for the output length distribution, and superimposing a confidence radius on the probability estimation to provide a conservative safety margin.

[0106] The routing system performs monotonic projection processing on the over-probability confidence values ​​corresponding to each length threshold along the increasing length threshold direction. Since each threshold branch learns independently, the original over-probability confidence values ​​may fluctuate in a way that violates the physical properties of the survival function. The routing system corrects these fluctuations through monotonic projection to values ​​that are monotonically non-increasing with increasing thresholds, ensuring the physical consistency of cost assessment.

[0107] The routing system selects the smallest length threshold from the threshold grid that satisfies a probability confidence value less than or equal to a preset risk limit as the safe output length. If none of the thresholds are met, an explicit overflow tail boundary is used as the safe output length. Finally, a corresponding safe resource proxy value is generated based on the safe output length as the long-tail resource consumption assessment value. This preset risk limit is pre-set by operations personnel according to business needs, for example, it can be set to 5% or 10%, which means the upper limit of the system's allowed budget overrun risk.

[0108] Through the above implementation method, the routing system can obtain the long-tail resource consumption evaluation value of each candidate model before the call, without relying on a complete modeling of the output length distribution. It can form the pre-call tail cost safety certificate by learning a few binary problems of threshold exceedance probabilities, which provides a reliable basis for risk penalty calculation in subsequent routing scoring.

[0109] Furthermore, in step 102 above, the prediction of the success reward representation value of each candidate model for the task to be processed based on contextual features is specifically achieved in the following way: The routing system first projects the context features to generate universal features shared across models. These universal features characterize the difficulty and context complexity of the current task. In practice, the routing system maintains a projection matrix that projects the original high-dimensional context features into a low-dimensional shared feature space. These shared features capture the inherent properties of the task itself, such as the difficulty of the problem, the type of domain it belongs to, and the complexity of the context length. These properties are common to all candidate models; that is, if a task is inherently difficult, it will be relatively harder to solve successfully regardless of which model is called; conversely, if a task is relatively simple, most models are likely to complete it successfully. Through feature projection, the routing system extracts common information at the task level for all candidate models to share.

[0110] The routing system extracts model-specific features for each candidate model from the contextual features. These model-specific features characterize the corresponding candidate model's ability to handle the current task. Unlike shared features, model-specific features focus on the unique characteristics of each candidate model, such as its historical performance on specific task types, its price range, and its inference ability level. The routing system maintains a set of personalized parameters for each candidate model, which, combined with the contextual features, characterize the model's specific capabilities for the current task.

[0111] The routing system integrates general features and model-specific features to calculate the success rate estimate and corresponding uncertainty metric for each candidate model performing the current task. Specifically, the routing system fuses general features and model-specific features through addition or concatenation to obtain a comprehensive feature representation. This representation includes both common task difficulty information and model-specific processing capability information. Based on this comprehensive feature, the routing system predicts the success rate and obtains the success rate estimate. Simultaneously, the routing system also calculates the corresponding uncertainty metric, such as prediction variance or confidence interval radius. This metric reflects the reliability of the current estimate: a larger uncertainty metric occurs when a candidate model has limited historical data on similar tasks, and a smaller uncertainty metric occurs when historical data is abundant.

[0112] The routing system calculates and determines the upper confidence value of the success payoff for each candidate model based on the success rate estimate and the uncertainty metric, and uses this upper confidence value as the representation of the success payoff. In practice, the routing system adds the success rate estimate to the uncertainty metric to obtain the upper confidence value of the success payoff. This upper confidence value format makes the routing system tend to select models with less historical data but potentially higher potential when facing uncertainty, thus achieving a balance between utilizing known good models and exploring unknown potential models.

[0113] Through the above implementation method, the routing system can obtain the success reward representation value of each candidate model before invocation. This representation value reflects the probability of the model succeeding on the current task, and also preserves the exploration space through the form of upper confidence bound, providing a basis for the calculation of the reward item in subsequent routing scoring.

[0114] In this embodiment, step 103 includes: The routing system first calculates the basic task share based on the remaining available resources before the call and the number of remaining tasks, including the current task. Specifically, it divides the remaining available resources by the number of remaining tasks to obtain the average resource quota for each task, ensuring that the routing decision is always within the framework of global resource constraints.

[0115] The routing system obtains a tempo coefficient to adjust the rate of resource consumption and multiplies this tempo coefficient by the base task share to obtain the local resource share for the current task. The tempo coefficient is adjusted online based on the recent sliding trend of actual resource consumption. When recent consumption is lower than expected, the budget is released appropriately; when recent consumption is higher than expected, the budget is tightened, thereby achieving smooth control of the budget consumption rate.

[0116] The routing system obtains a resource shadow price, which represents the current overall resource scarcity level, as a dynamic scheduling parameter. The shadow price is lower when resources are abundant and higher when resources are scarce, enabling the model's selection strategy to respond in real time to changes in resource status.

[0117] The routing system divides the long-tail resource consumption assessment value by the local resource share and then multiplies it by the resource shadow price to obtain a risk penalty term. This penalty term reflects the marginal risk cost that calling a candidate model may bring under the current resource state. The higher the long-tail resource consumption assessment value, the smaller the local resource share, or the higher the shadow price, the larger the risk penalty term.

[0118] Finally, the routing system subtracts the risk penalty from the success reward representation value to obtain a unified routing score for each candidate model, and selects the model with the highest score to execute the task. Through this method, when resources are abundant, the routing score is dominated by success rewards, and the system favors high-success-rate models; when resources are scarce, the routing score is dominated by risk penalties, and the system favors low-tail-risk models, achieving an adaptive balance between pursuing success and controlling risk.

[0119] In this embodiment, after step 104 above, in which the target model is selected based on the routing score to perform the task, the routing system also performs the following budget closed-loop update operation.

[0120] like Figure 3 As shown, the budget control module updates the shadow price and rhythm coefficient through the following closed-loop mechanism: Actual call cost c and step budget b. stepThe shadow price λ is updated after comparison; the rhythm coefficient is updated after comparing the recent cost index moving average EMA with the basic task share. Updated shadow price λ and rhythm coefficient The data flows back to the local budget calculation and routing score, forming a complete budget feedback control loop.

[0121] The routing system first obtains the actual resource consumption *c* of the target model executing the task to be processed. This actual resource consumption *c* is returned by the target model along with the task response result, and may include information such as the number of input tokens, the number of output tokens, API call costs, or inference time. The routing system records this actual resource consumption *c* as the basis for subsequent budget updates and shadow price adjustments.

[0122] The routing system calculates the actual resource consumption c relative to the budget share of the current task step b. step The degree of deviation, i.e., calculating c and b. step The ratio reflects the direction and magnitude of the deviation between the actual consumption of this call and the expected step budget. When the actual consumption c is significantly higher than the step budget b... step When the deviation is positive and large, it indicates that recent consumption is too rapid; when the actual consumption c is lower than the step budget b... step When the deviation is negative, it indicates that recent resource consumption is relatively low. The routing system trims this deviation to a reasonable range to avoid drastic disturbances caused by extreme values. Then, it multiplies the trimmed deviation by a preset learning rate and adjusts the shadow price λ exponentially. When the deviation is positive, the shadow price λ increases exponentially (λ is multiplied by a factor greater than 1), increasing the penalty for resource consumption in subsequent routing decisions, thus making the system more inclined to choose models with low tail risk. When the deviation is negative, the shadow price λ decreases exponentially (λ is multiplied by a factor less than 1), reducing the penalty for resource consumption in subsequent routing decisions, allowing the system to moderately relax its focus on cost. The shadow price λ is updated after each call. Through this exponential gradient update method, λ always maintains an accurate representation of the current resource scarcity level.

[0123] The routing system calculates the exponential moving average (EMA) of recent historical actual resource consumption and compares the ratio of this EMA to the base task share, updating the cadence coefficient based on this ratio. The Exponential Moving Average (EMA) is a weighted average of the actual consumption of a number of recent calls. More recent consumption data has a higher weight, while earlier data has a lower weight, allowing the average to sensitively reflect recent consumption trends. The routing system calculates the ratio of the EMA to the base task share. When this ratio is greater than 1, it indicates that recent consumption is higher than the global average, meaning the budget consumption rate is too fast. The routing system will adjust the pace coefficient accordingly. Adjust in the direction less than 1, that is Multiplying by a factor less than 1 reduces the local resource share of subsequent tasks, giving them tighter budget constraints to compensate for previous overspending. When this ratio is less than 1, it indicates that recent consumption is below the global average, and the routing system will adjust the tempo coefficient. Adjust in the direction greater than 1, that is Multiplying by a factor greater than 1 increases the local resource share of subsequent tasks, allowing them to have more relaxed budget constraints and fully utilize unused resources. Similar to the update of the shadow price λ, the tempo coefficient... The updates also employ an exponential approach to maintain the smoothness and stability of parameter changes.

[0124] like Figure 3 As shown, the updated shadow price λ and rhythm coefficient The return path is used to influence subsequent route score calculations and local resource share calculations. Specifically, the updated shadow price λ participates in route score calculations as a dynamic scheduling parameter in the next round of route decision-making, and the updated tempo coefficient... In the next round of routing decisions, the new local resource share is obtained by multiplying it by the share of the basic task. Thus, starting from the actual call cost, through deviation calculation, exponential gradient update, shadow price adjustment, exponential moving average calculation, rhythm coefficient adjustment, and then parameter feedback influencing subsequent decisions, a complete budget feedback control closed loop is formed.

[0125] In this embodiment, before invoking the target model, the routing system first checks whether the available resource value in the current remaining resource status is greater than zero. Only when the remaining resources are greater than zero will the routing system trigger the target model to execute the pending task. This check ensures that the system will not initiate any invalid calls when the budget is completely exhausted, avoiding the risk of running beyond the budget.

[0126] After executing the target model, the routing system obtains the actual resource consumption of this call and adds it to the historical total consumption to obtain the updated cumulative resource consumption. The routing system then checks whether this cumulative resource consumption exceeds the preset total resource limit for the first time. If so, it immediately blocks subsequent model calls for the current task flow. Since the actual resource consumption of each call cannot be accurately known before the call, and the output length of large language models is highly random, there is a possibility that the actual consumption of a single call may exceed the remaining budget. By checking whether the cumulative consumption exceeds the total limit after each call and immediately blocking it when it exceeds the limit for the first time, the routing system ensures that the hard constraint of path-level budget, which requires stopping service upon the first overspending in a real engineering system, is strictly adhered to.

[0127] After the model call is completed, the routing system receives the task response result and actual resource consumption returned by the target model. The routing system verifies the task response result through a preset verification mechanism and generates a corresponding verification reward result. The verification mechanism can take various forms: for question-and-answer tasks with standard answers, the verification mechanism can perform precise comparison or fuzzy matching between the model's answer and the standard answer; for code repair tasks, the verification mechanism can perform unit testing or static analysis; for enterprise question-and-answer or customer service tasks, the verification mechanism can perform retrieval consistency checks, rule validation, or manual sampling. The verification reward result can be a binary success or failure, or a continuous score, such as the test pass rate or task completion rate.

[0128] When the task of verifying the reward result indicates success, the routing system terminates the current pending task and triggers the processing flow of the next task. When the task of verifying the reward result indicates failure, the routing system updates the cumulative resource consumption status and call logs based on the actual resource consumption, and triggers the task retry decision process.

[0129] like Figure 4 As shown, the specific decision-making logic of the retry-stop gating is as follows: When the routing system receives a failure response, the retry-stop module is activated. The inputs to this module include the highest routing score Q corresponding to the current round of calls, the retry depth h of the current task, the accumulated resource consumption value spend of the current task, and the local resource share b of the current task. taskAfter the above input is fed into the gating function, the gating function calculates the dynamic retry threshold κ and determines whether Q is greater than or equal to κ. Based on the result, it outputs a decision to continue retrying or stop the current task. The dynamic retry threshold κ is positively correlated with the retry depth h and the degree of task resource overrun. As the number of retries h increases and the amount of resources consumed by the task increases, the gating threshold κ increases accordingly, making the conditions for subsequent attempts to be approved increasingly stringent. This design makes the system more likely to abandon retries in scenarios with low marginal returns, while allowing more exploration opportunities in the early stages of retrying when resources are sufficient.

[0130] The retry decision-making process follows the gating logic below. The routing system first determines whether the retry depth h of the current task is less than the preset maximum retry depth H. If the retry depth h has reached the maximum retry depth H, then no further retrying is performed; the current task is stopped directly, and any unused resource share is reserved for subsequent tasks. If the retry depth h is less than the maximum retry depth H, the routing system determines the retry depth h, the accumulated resource consumption (spend) of the current task, and the local resource share (b). task Calculate the dynamic retry threshold κ. This dynamic retry threshold κ is positively correlated with the retry depth h and the degree of task resource overrun; that is, the more retries there are and the more resources the task has consumed, the higher the retry threshold becomes.

[0131] The routing system then determines whether the highest unified routing score Q among the current candidate models is greater than or equal to the dynamic retry threshold κ. If Q is greater than or equal to κ, the routing system constructs the next round's prompt information based on the failure history and re-executes the steps of calculating the routing score and selecting the target model. If Q is less than κ, the routing system stops the retrying action of the current task and reserves the unused resource share for subsequent tasks. Figure 4 As shown, when the gating function determines that Q is greater than or equal to κ, the routing system returns and appends the failure history, constructs the next round of prompts, and re-enters the call flow; when the gating function determines that Q is less than κ, the routing system exits the processing of the current task and starts processing the next pending task in the task flow.

[0132] Through such Figure 4 The retry stop gating logic shown determines whether to continue retrying after a task failure by comparing the current highest routing score Q with a dynamic threshold value κ. The dynamic threshold value κ is related to the retry depth h and the resource overrun level spend / b. task The positive correlation design enables the system to automatically tighten the retry approval conditions in high-retry-count and high-consumption scenarios, and automatically abandon low-return tasks when the budget is tight, avoiding invalid calls from crowding out the resources of subsequent tasks, thereby making the limited budget more efficient for tasks with success potential.

[0133] like Figure 5 As shown, in a complete single-task multi-round routing process, the routing system, from the start of task processing, first checks if the remaining budget is sufficient, then constructs hints and context features, calculates the confidence value of the success reward and the long-tail resource consumption evaluation value for each candidate model, and calculates a unified routing score by combining shadow price and local resource share. Based on the score, a model is selected and executed. If the verification is successful after the call, the current task ends; if the verification fails, a retry gate is entered to determine whether to continue retrying. If the budget exceeds the limit, the entire task flow is stopped. The following is a comparison... Figure 5 The complete method flow of the embodiments of the present invention will be described in detail in the form of steps S1 to S10.

[0134] Step S1, Initialization.

[0135] Before executing the method flow of this embodiment of the invention, the routing system first performs an initialization operation. The routing system obtains the candidate model set, the task flow to be processed, the total resource limit, the maximum retry depth, and the resource price table of each model, and optionally uses a small amount of historical logs or preheating tasks to preheat the parameters, and initializes the model parameters in the success revenue learning module and the tail cost learning module.

[0136] Step S2: Construct the current call state.

[0137] After the task flow is officially started, for the h-th attempt of the t-th task, the routing system generates the current prompt information p based on the original task text, failure history, tool output feedback, and preset prompt template. s And calculate the context embedding representation x of the current prompt information. s Simultaneously, it acquires structured information such as the retry depth h of the current task, the model identifier used in the previous round, the validation reward result obtained in the previous round, and the actual resource consumption in the previous round. The routing system combines the context embedding representation with the above structured information to obtain the complete context feature x. s The routing system further applies the projection matrix P to the context features x. s Projecting yields the shared features z across models. s This shared feature is used to characterize the difficulty and contextual complexity of the current task.

[0138] Step S3: Estimate the success reward.

[0139] Routing systems are based on contextual features x s and shared features z s For each candidate model a, its success reward representation value is estimated. Specifically, the routing system uses a hybrid context Bandit model, where shared features z are used. s Multiply by the shared parameter β, the model's personalized feature x sMultiply by the model's individualized parameter θ a The two are added together to obtain the success rate estimate, and then the confidence radius is added to obtain the upper confidence value U of the success payoff. rs(a) This serves as a representation of the success of the candidate model in the current context.

[0140] Step S4: Estimate tail costs.

[0141] Routing systems are based on contextual features x s Given the input length of the current task, the system estimates the long-tail resource consumption value for each candidate model 'a'. Specifically, the routing system uses a multi-threshold survival assessment branch to estimate the probability that the output length of each candidate model exceeds each preset length threshold. It ensures the physical consistency of the probability estimates through monotonic projection, selects the minimum threshold that meets the risk limit as the safe output length, and then calculates the safe resource proxy value based on the model price, which serves as the long-tail resource consumption value for that candidate model in the current context.

[0142] Step S5: Calculate the partial budget and budget price.

[0143] The routing system determines dynamic scheduling parameters based on the current remaining resource status. Let R... s N represents the remaining resources before the current call. s The routing system calculates the basic task share as R, which represents the number of remaining tasks, including the current task. s Divide by N s The routing system obtains the currently maintained tempo coefficient. , rhythm coefficient Multiply by the base task share to obtain the current task's local resource share b. tasks Simultaneously, determine the step budget b based on the remaining number of attempts for the current task. steps The routing system obtains the currently maintained resource shadow price λ. s As a dynamic scheduling parameter, this parameter represents the current overall resource stress level.

[0144] Step S6, Model Selection.

[0145] The routing system calculates a unified routing score Qs(a) for each candidate model a. Specifically, the routing system divides the long-tail resource consumption assessment value by the local resource share b. tasks Then multiply by the resource shadow price λ s A risk penalty is obtained; the confidence value U for successful returns is obtained. rs(a) Subtracting the risk penalty from the result yields the unified routing score Q. s (a) After calculating a unified routing score for all candidate models, the routing system selects model a with the highest score. sAs the target model, send a message containing the current prompt information p to the API endpoint of that model. s The call request.

[0146] Step S7: Response verification and online update.

[0147] The routing system receives the task response results and actual resource consumption returned by the target model. The routing system verifies the task response results using a pre-defined verification mechanism and generates a verification reward result r. s Verify the reward result r s It can be a binary success or failure, or it can be a continuous score. If r s If the task is successful, the current task ends, and the routing system moves on to process the next task in the task flow; if r s If a task fails, the routing system updates the cumulative resource consumption status and call logs based on the actual resource consumption and triggers the task retry decision process.

[0148] Step S8: Update the budget loop.

[0149] The routing system updates the budget status based on actual resource consumption. Specifically, the routing system calculates the actual consumption relative to the step budget b. steps The degree of deviation is used to adjust the resource shadow price λ using an exponential gradient update rule. s Simultaneously, calculate the recent historical exponential moving average of actual consumption, compare the ratio of this moving average to the basic task share, and update the rhythm coefficient based on the ratio. The updated shadow price λ s and rhythm coefficient This will be used for routing decisions in subsequent tasks.

[0150] Step S9, retry and stop.

[0151] If this call fails and the maximum retry depth has not been reached, the routing system triggers a retry stop gating. The routing system determines the retry stop based on the highest routing score Q corresponding to this round of calls. s (a s ), current retry depth h, and the cumulative resource consumption of the current task (spend) s and local resource share b tasks Calculate the dynamic retry threshold κ s The dynamic retry threshold κ s It is positively correlated with the retry depth h and the degree of task resource overrun. The routing system determines the highest route score Q. s (a s Is it greater than or equal to the dynamic retry threshold κ? sIf so, construct the prompt message for the next round based on the failure history and re-execute steps S2 to S6; otherwise, stop the retry action of the current task and reserve the unused resource share for subsequent tasks.

[0152] Step S10: Path-level budgeting stops.

[0153] The routing system checks the remaining resources R before each call. s Whether it is positive or negative, record the actual cost c after each call. s And accumulate to the total consumption C s If the cumulative consumption is C s If the total resource limit B is exceeded for the first time, the current task flow will be stopped immediately. This is because each call consumes c... s Since the exact amount of data cannot be known before the call, and the output length of a large language model is highly random, the routing system ensures that the hard constraints of the path-level budget are strictly adhered to by detecting whether the cumulative consumption exceeds the total limit after each call and immediately blocking it when the first out-of-bounds error occurs.

[0154] The resource-constrained online routing system for large language models provided in the embodiments of the present invention is described below. The resource-constrained online routing system for large language models described below can be referred to in correspondence with the resource-constrained online routing method for large language models described above.

[0155] This invention provides an online routing system for resource-constrained large language models, see [link to relevant documentation]. Figure 6 ,include: The initialization module 610 is used to obtain the context characteristics of the task to be processed and the current status of the remaining resources; The risk prediction module 620 is used to predict the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed based on the context features. The routing scoring module 630 is used to determine dynamic scheduling parameters based on the remaining resource status, and to calculate the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value. The task execution module 640 is used to select a target model from each candidate model based on the routing score and execute the task to be processed.

[0156] like Figure 7 As shown in the illustration, this embodiment of the invention provides an online routing system for large language models oriented towards resource constraints. Wherein: The initialization module 610 is used to obtain the context features of the task to be processed and the current status of remaining resources. In this embodiment, the initialization module 610 specifically includes a task access module, a model pool management module, and a context feature module.

[0157] The task access module receives inputs such as task text, images, code, tool status, or historical dialogues, forming a task flow to be processed, and records the task number, remaining task count, and maximum retries. The model pool management module maintains the candidate model set and its input / output prices, call limits, availability, and priority. The context feature module generates normalized context features x based on the current prompt, retry depth, historical failure responses, and budget status, and obtains shared features z=Px across models through the projection matrix P.

[0158] The context feature module obtains context features in the following ways: it acquires the original task text, task retry history, tool output feedback, and preset prompt templates, and concatenates them to form the current prompt information; it extracts features from the current prompt information to obtain context representation features, and obtains the retry depth of the current task, the historical model used in the previous round, the reward feedback of the previous round, and the resource consumption value of the previous round; it combines the context representation features, retry depth, historical model, reward feedback, and resource consumption value to obtain the context features.

[0159] The risk prediction module 620 is used to predict the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed based on contextual features. In this embodiment, the risk prediction module 620 specifically includes a success benefit learning module and a tail cost learning module.

[0160] The success reward learning module takes contextual features x and shared features z, model number, and validator reward r as input, estimates the probability of each candidate model solving the task in the current context, and provides a confidence value U for the success reward. r As a representation of successful returns, the tail cost learning module receives input length, output length or cost, retry information, and model price as inputs, learns the probability that the cost exceeds each preset threshold, and generates a safety cost surrogate value. As an assessment value for long-tail resource consumption.

[0161] The routing scoring module 630 is used to determine dynamic scheduling parameters based on the remaining resource status, and to calculate the routing score of each candidate model by combining the dynamic scheduling parameters, success benefit representation values, and long-tail resource consumption evaluation values. In this embodiment, the routing scoring module 630 specifically includes a budget control module and a routing selection module.

[0162] The budget control module receives the remaining budget R, the number of remaining tasks N, the cost already spent, and the cost exponential moving average EMA as input, and updates the local task budget b. task Step budget b step Shadow price λ and rhythm coefficient The shadow price λ serves as a dynamic scheduling parameter, and the rhythm coefficient... Used to adjust the rate of resource consumption. The routing module 107 is used to receive the confidence value U for successful revenue. r Security cost agency value Shadow price λ and local task budget b task As input, calculate the unified routing score Q and select candidate models.

[0163] The task execution module 640 is used to select a target model from each candidate model based on the routing score and execute the task to be processed. In this embodiment, the task execution module 640 specifically includes a large language model pool, a verification and update module, and a retry and stop module.

[0164] The large language model pool contains multiple callable large language models. After the routing module selects a target model, the task execution module 640 sends a call request to the API endpoint of the selected model and receives the return result. The verification and update module receives the model's answer, standard answer, test result, rule verification, or manual verification result, as well as the actual usage, as input to generate a binary or continuous reward and update the success model parameters in the success reward learning module, the cost model parameters in the tail cost learning module, and the budget status in the budget control module. The retry and stop module receives the routing score Q, retry depth h, current task resource expenditure (spend), and local budget (b) for this round. task As input, a dynamic retry threshold κ is calculated after a task fails to determine whether it is worthwhile to continue retrying, thus avoiding low-reward, high-risk calls.

[0165] also, Figure 7 The system also includes a log storage module, which saves data such as task number, model candidate set, current prompt summary, feature version, confidence value of each model's successful revenue, security cost proxy value, shadow price, rhythm coefficient, selected model, actual input and output token count, cost, verification results, reason for stopping and retrying, and budget status, for use in preheating, audit trail, and subsequent strategy parameter tuning.

[0166] The inputs, outputs, and functions of each of the above modules are shown in Table 1 below.

[0167] Table 1

[0168] The specific functions of each module in the above system correspond to the corresponding steps in the method embodiments of the present invention. The same or similar parts can be referred to each other, and will not be repeated here.

[0169] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a resource-constrained online routing method for large language models. This method includes: acquiring the context features of the task to be processed and the current remaining resource status; based on the context features, estimating the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed; determining dynamic scheduling parameters according to the remaining resource status, and calculating the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value; and selecting a target model from among the candidate models to execute the task to be processed based on the routing score.

[0170] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the resource-constrained online routing method for large language models provided by the above methods. The method includes: obtaining the context features of the task to be processed and the current remaining resource status; based on the context features, estimating the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed; determining dynamic scheduling parameters according to the remaining resource status, and calculating the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value; and selecting a target model from the candidate models to execute the task to be processed according to the routing score.

[0172] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an online routing method for resource-constrained large language models provided by the methods described above. This method includes: acquiring contextual features of a task to be processed and the current remaining resource status; based on the contextual features, estimating the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed; determining dynamic scheduling parameters according to the remaining resource status, and calculating a routing score for each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value; and selecting a target model from among the candidate models to execute the task to be processed based on the routing score.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource-constrained online routing method for large language models, characterized in that, include: Obtain the contextual characteristics of the task to be processed and the current status of remaining resources; Based on the aforementioned contextual features, the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed are estimated respectively. Based on the remaining resource status, dynamic scheduling parameters are determined, and combined with the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value, the routing score of each candidate model is calculated. Based on the routing score, a target model is selected from the candidate models to execute the task to be processed.

2. The online routing method for resource-constrained large language models according to claim 1, characterized in that, The step of determining dynamic scheduling parameters based on the remaining resource status, and calculating the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value, includes: The basic task share is calculated based on the remaining available resources before the call and the number of remaining tasks, including the current task. Obtain a rhythm coefficient for adjusting the rate of resource consumption, and combine the rhythm coefficient with the basic task share to obtain the local resource share of the current task; Obtain the resource shadow price, which represents the current overall resource scarcity level, as the dynamic scheduling parameter; Divide the long-tail resource consumption assessment value by the local resource share, and then multiply it by the resource shadow price as a risk penalty item; The unified routing score for each candidate model is obtained by subtracting the risk penalty term from the success reward representation value.

3. The online routing method for resource-constrained large language models according to claim 1, characterized in that, The step of estimating the long-tail resource consumption evaluation value of each candidate model for the task to be processed based on the context features includes: The threshold grid is determined based on the effective output length quantiles of the historical call logs, and an explicit overflow tail boundary is set. The threshold grid contains multiple incremental length thresholds. Using a multi-threshold survival assessment branch, combined with the aforementioned contextual features, the probability of exceeding each candidate model's output length by more than each length threshold is estimated, and the confidence value of the probability of exceeding each length threshold is calculated. Along the increasing direction of the length threshold, monotonic projection processing is performed on the over-confidence value corresponding to each length threshold, so that the over-confidence value of the over-confidence probability is monotonically non-increasing with the increase of the length threshold. In the threshold grid, the minimum length threshold that satisfies the condition that the confidence value on the probability of exceeding the threshold is less than or equal to the preset risk limit is selected as the safe output length; A corresponding security resource proxy value is generated based on the security output length, which serves as the long-tail resource consumption evaluation value.

4. The online routing method for resource-constrained large language models according to claim 1, characterized in that, The step of estimating the success reward representation value of each candidate model for the task to be processed based on the context features includes: The context features are projected to generate universal features shared across models. These universal features are used to characterize the current task difficulty and context complexity. From the context features, model-specific features for each candidate model are extracted. These model-specific features are used to characterize the ability attributes of the corresponding candidate model to process the task to be processed. By integrating the general features and the model-specific features, the success rate estimate and the corresponding uncertainty metric value of each candidate model when performing the task to be processed are calculated. Based on the success rate estimate and the uncertainty metric, the confidence value of the success return for each candidate model is calculated and determined, and the confidence value of the success return is used as the success return characterization value.

5. The online routing method for resource-constrained large language models according to claim 1, characterized in that, The step of obtaining the contextual features of the task to be processed includes: Obtain the original task text, task retry history, tool output feedback, and preset prompt templates, and combine them to form the current prompt information; Feature extraction is performed on the current prompt information to obtain context representation features, and the retry depth of the current task, the historical model used in the previous round, the reward feedback of the previous round, and the resource consumption value of the previous round are obtained. The contextual features are obtained by combining the contextual representation features, the retry depth, the historical model, the reward feedback, and the resource consumption value.

6. The online routing method for resource-constrained large language models according to claim 1, characterized in that, The step of selecting the target model from each candidate model based on the routing score to execute the task to be processed includes: Before invoking the target model, if the available resource value in the current remaining resource status is greater than zero, the target model is triggered to execute the task to be processed. After executing the target model, the actual resource consumption of this call is obtained and added to the historical total consumption to obtain the updated cumulative resource consumption. When the cumulative resource consumption is detected to exceed the preset total resource limit for the first time, subsequent model call operations for the current task flow are blocked.

7. The online routing method for resource-constrained large language models according to claim 1, characterized in that, After the step of selecting the target model from each candidate model based on the routing score to execute the task to be processed, the method further includes: Receive the task response result and actual resource consumption returned by the target model for the task to be processed; The task response result is verified through a preset verification mechanism to generate a corresponding verification reward result; When the verification reward result indicates that the task is successful, the pending task ends and the processing flow of the next task is triggered; When the verification reward result indicates that the task has failed, the cumulative resource consumption status and call log are updated according to the actual resource consumption, and the task retry decision process is triggered.

8. The online routing method for resource-constrained large language models according to claim 7, characterized in that, The steps of triggering the task retry decision process include: Determine whether the current retry depth of the task to be processed is less than the preset maximum retry depth; When the retry depth is less than the maximum retry depth, a dynamic retry threshold is calculated based on the retry depth, the accumulated resource consumption of the task to be processed, and the local resource share. The dynamic retry threshold is positively correlated with the retry depth and the degree of task resource overrun. Determine whether the highest unified routing score among the current candidate models is greater than or equal to the dynamic retry threshold. If so, construct the next round of prompts based on the failure history, and re-execute the steps of calculating the route score and selecting the target model; If not, then stop the retry action of the pending task and reserve the unused resource share for subsequent tasks.

9. The online routing method for resource-constrained large language models according to claim 2, characterized in that, After the step of selecting the target model from the candidate models based on the routing score to execute the task to be processed, the method further includes: Obtain the actual resource consumption of the target model when executing the task to be processed; Calculate the deviation of the actual resource consumption from the current task step's budget share, and adjust the resource shadow price based on the deviation using an exponential gradient update rule; Calculate the exponential moving average of recent historical actual resource consumption, compare the ratio of the exponential moving average to the basic task share, and update the rhythm coefficient based on the ratio.

10. The online routing method for resource-constrained large language models according to claim 1, characterized in that, Before the step of obtaining the contextual characteristics of the task to be processed and the current remaining resource status, the method further includes: Obtain a task flow containing multiple task sequences, and parse out the task that needs to be executed from the task flow according to a preset execution order, as the task to be processed; Obtain the preset total resource limit, and directly set the initial available resource value to the total resource limit to generate the current remaining resource status; Obtain a set of available candidate models, and obtain warm-up data based on historical call logs or warm-up execution results for the set of candidate models; The preheating data is used to initialize the underlying parameters so that, based on the context features, the success benefit representation value and the long-tail resource consumption evaluation value of each candidate model in the candidate model set for the task to be processed can be estimated respectively.

11. An online routing system for resource-constrained large language models, characterized in that, include: The initialization module is used to obtain the context characteristics of the task to be processed and the current status of remaining resources; The risk prediction module is used to predict the success benefit representation value and long-tail resource consumption evaluation value of each candidate model for the task to be processed based on the context features. The routing scoring module is used to determine dynamic scheduling parameters based on the remaining resource status, and to calculate the routing score of each candidate model by combining the dynamic scheduling parameters, the success benefit representation value, and the long-tail resource consumption evaluation value. The task execution module is used to select the target model from each candidate model based on the routing score and execute the task to be processed.

12. The resource-constrained online routing system for large language models according to claim 11, characterized in that, The routing scoring module is used for: The basic task share is calculated based on the remaining available resources before the call and the number of remaining tasks, including the current task. Obtain a rhythm coefficient for adjusting the rate of resource consumption, and combine the rhythm coefficient with the basic task share to obtain the local resource share of the current task; Obtain the resource shadow price, which represents the current overall resource scarcity level, as the dynamic scheduling parameter; Divide the long-tail resource consumption assessment value by the local resource share, and then multiply it by the resource shadow price as a risk penalty item; The unified routing score for each candidate model is obtained by subtracting the risk penalty term from the success reward representation value.

13. The resource-constrained online routing system for large language models according to claim 11, characterized in that, The risk assessment module is used for: The threshold grid is determined based on the effective output length quantiles of the historical call logs, and an explicit overflow tail boundary is set. The threshold grid contains multiple incremental length thresholds. Using a multi-threshold survival assessment branch, combined with the aforementioned contextual features, the probability of exceeding each candidate model's output length by more than each length threshold is estimated, and the confidence value of the probability of exceeding each length threshold is calculated. Along the increasing direction of the length threshold, monotonic projection processing is performed on the over-confidence value corresponding to each length threshold, so that the over-confidence value of the over-confidence probability is monotonically non-increasing with the increase of the length threshold. In the threshold grid, the minimum length threshold that satisfies the condition that the confidence value on the probability of exceeding the threshold is less than or equal to the preset risk limit is selected as the safe output length; A corresponding security resource proxy value is generated based on the security output length, which serves as the long-tail resource consumption evaluation value.

14. The resource-constrained online routing system for large language models according to claim 11, characterized in that, The risk assessment module is used for: The context features are projected to generate universal features shared across models. These universal features are used to characterize the current task difficulty and context complexity. From the context features, model-specific features for each candidate model are extracted. These model-specific features are used to characterize the ability attributes of the corresponding candidate model to process the task to be processed. By integrating the general features and the model-specific features, the success rate estimate and the corresponding uncertainty metric value of each candidate model when performing the task to be processed are calculated. Based on the success rate estimate and the uncertainty metric, the confidence value of the success return for each candidate model is calculated and determined, and the confidence value of the success return is used as the success return characterization value.

15. The resource-constrained online routing system for large language models according to claim 11, characterized in that, The initialization module is used for: Obtain the original task text, task retry history, tool output feedback, and preset prompt templates, and combine them to form the current prompt information; Feature extraction is performed on the current prompt information to obtain context representation features, and the retry depth of the current task, the historical model used in the previous round, the reward feedback of the previous round, and the resource consumption value of the previous round are obtained. The contextual features are obtained by combining the contextual representation features, the retry depth, the historical model, the reward feedback, and the resource consumption value.

Images