Model routing method and device of LLM inference gateway, equipment and medium

Abstract

Embodiments of the present application disclose a model routing method and device of an LLM inference gateway, equipment and a medium. The method comprises: receiving an inference request, the inference request carrying a target business scenario label; collecting index data of a plurality of different preset indicators respectively corresponding to each candidate model in a candidate model set in real time; based on the plurality of index data of each candidate model in the candidate model set, running a non-dominated sorting algorithm on the candidate model set to calculate a Pareto optimal solution set; determining an index weight corresponding to each preset indicator according to the target business scenario label; based on the index data and the index weight corresponding to each preset indicator, selecting a target model within the Pareto optimal solution set using a weighted Chebyshev distance minimization criterion; and forwarding the inference request to the target model. The embodiments of the present application improve the accuracy of model selection by implementing the method.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to model routing methods, apparatus, devices and media for LLM inference gateways. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, enterprise business scenarios typically integrate multiple large language models from different service providers and with different specifications, such as general dialogue models, code models, and long text generation models. To achieve unified access, traffic scheduling, and security management for multiple models, an LLM inference gateway (such as a large model enterprise gateway) is usually deployed as an intermediate scheduling layer between the business end and the underlying models. The gateway selects the optimal model for inference response in real time based on the request.

[0003] Existing LLM model routing schemes mostly adopt single-index scheduling, such as selecting only the model with the lowest latency or only the model with the lowest cost; some multi-index scheduling methods use simple linear weighted summation, which does not consider the conflict relationship between multiple indices, and cannot solve the optimal solution set under multiple indices, resulting in low accuracy of the finally selected model. Summary of the Invention

[0004] This application provides a model routing method, apparatus, device, and medium for an LLM inference gateway, which can improve the accuracy of model selection.

[0005] In a first aspect, embodiments of this application provide a model routing method for an LLM inference gateway, comprising: Receive an inference request, the inference request carrying a target business scenario tag; Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set; Based on multiple index data of each candidate model in the candidate model set, a non-dominated sorting algorithm is run on the candidate model set to calculate the Pareto front optimal solution set; Determine the weight of each preset indicator based on the target business scenario label; Based on the indicator data and indicator weights corresponding to each preset indicator, the target model is selected within the optimal solution set of the Pareto front using the weighted Chebyshev distance minimization criterion. The inference request is forwarded to the target model.

[0006] In some embodiments, the step of running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set includes: For each candidate model in the candidate model set, if the candidate model is not dominated by other candidate models in the candidate model set, then the candidate model is determined as the preferred model. Wherein, if the indicator data of the first candidate model in the candidate model set are not inferior to the indicator data of the second candidate model, and at least one indicator data of the first candidate model is strictly superior to the indicator data of the second candidate model, then the second candidate model is determined to be dominated by the first candidate model. The Pareto front optimal solution set is constructed based on all the preferred models in the candidate model set.

[0007] In some embodiments, the step of selecting the target model within the Pareto front optimal solution set based on the indicator data and indicator weights corresponding to each preset indicator, using the weighted Chebyshev distance minimization criterion, includes: Based on the index data corresponding to each preset index, determine the optimal index data corresponding to each preset index in multiple candidate models within the Pareto front optimal solution set. For each candidate model in the Pareto front optimal solution set, a first index difference is determined for each preset index. The first index difference is the difference between the index data of the corresponding preset index and the optimal index data. For each candidate model in the Pareto front optimal solution set, a second index difference for each preset index is determined. The second index difference is determined based on the first index difference and the corresponding index weight. For each candidate model in the Pareto front optimal solution set, the second index difference with the largest median value among the preset indexes is determined as the target index difference of the corresponding candidate model. The candidate model with the smallest difference in the target index in the Pareto front optimal solution set is determined as the target model.

[0008] In some embodiments, the indicator data of the plurality of different preset indicators includes cost data, quality data, and latency data; the real-time acquisition of indicator data of the plurality of different preset indicators corresponding to each candidate model in the candidate model set includes: The initial cost data for each candidate model is obtained from at least one of the official pricing channels and billing receipts for each candidate model in the candidate model set. The initial quality data for each candidate model are determined based on the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set. Extract the first byte delay value and generation rate of each candidate model in the candidate model set from the gateway historical call log, and determine the initial delay data of each candidate model based on the first byte delay value and generation rate; The initial cost data, the initial quality data, and the initial delay data are normalized to obtain the cost data, the quality data, and the delay data.

[0009] In some embodiments, determining the initial quality data for each candidate model based on its automatic evaluation score, manual sampling score, downstream task pass rate, and explicit user feedback evaluation score in the candidate model set includes: Obtain the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score for each candidate model in the candidate model set; The initial quality data for each candidate model are determined based on the automatic evaluation score, the manual sampling score, the downstream task pass rate, the user explicit feedback evaluation score, and the weight of each score source.

[0010] In some embodiments, the inference request further carries hard constraints, and before the real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set, the method further includes: Candidate models in the candidate model set that do not meet the hard constraints are filtered out.

[0011] In some embodiments, before calculating the Pareto front optimal solution set by running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set, the method further includes: Determine whether the number of candidate models in the candidate model set is greater than a preset threshold; If the number exceeds the preset threshold, then the bucketing rule corresponding to the target business scenario label is determined, and the candidate models in the candidate model set are bucketed to obtain multiple candidate buckets. The candidate bucket corresponding to the target business scenario label among the multiple candidate buckets is taken as the target bucket; Filter out candidate models in the candidate model set that are not located in the target bucket.

[0012] Secondly, embodiments of this application also provide a model routing device for an LLM inference gateway, comprising: The transceiver unit is used to receive inference requests, which carry target business scenario tags; and to collect indices data of multiple different preset indicators corresponding to each candidate model in the candidate model set in real time. The processing unit is configured to: run a non-dominated ranking algorithm on the candidate model set based on multiple indicator data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set; determine the indicator weights corresponding to each preset indicator according to the target business scenario label; select the target model in the Pareto front optimal solution set by using the weighted Chebyshev distance minimization criterion based on the indicator data and indicator weights corresponding to each preset indicator; and forward the inference request to the target model.

[0013] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0015] This application provides a model routing method, apparatus, device, and medium for an LLM inference gateway. The method includes: receiving an inference request, the inference request carrying a target service scenario tag; The process involves real-time acquisition of indicator data for multiple preset indicators corresponding to each candidate model in the candidate model set; running a non-dominated sorting algorithm on the candidate model set based on the multiple indicator data of each candidate model to calculate the Pareto front optimal solution set; determining the indicator weights corresponding to each preset indicator according to the target business scenario label; selecting the target model within the Pareto front optimal solution set based on the indicator data and indicator weights corresponding to each preset indicator, using the weighted Chebyshev distance minimization criterion; and forwarding the inference request to the target model. This embodiment abandons the traditional single-indicator scheduling and simple linear weighted scheduling methods. By running a non-dominated sorting algorithm, it can effectively solve for the Pareto front optimal solution set under multiple indicators, properly taking into account the conflict relationships between various indicators. Combined with business scenario matching weights and relying on weighted Chebyshev distance, it completes the selection of the target model, effectively improving the accuracy of model selection and optimizing the overall gateway routing scheduling effect. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1A flowchart illustrating the model routing method for an LLM inference gateway provided in this embodiment of the application; Figure 2 A schematic diagram of a sub-process of the model routing method for an LLM inference gateway provided in an embodiment of this application; Figure 3 Another sub-process diagram of the model routing method for the LLM inference gateway provided in the embodiments of this application; Figure 4 Another sub-process diagram of the model routing method for the LLM inference gateway provided in the embodiments of this application; Figure 5 A schematic block diagram of a model routing device for an LLM inference gateway provided in an embodiment of this application; Figure 6 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been obtained by authorized entities (who have known and consented) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] This application provides a model routing method, apparatus, device, and medium for an LLM inference gateway.

[0024] The execution subject of the model routing method of the LLM inference gateway can be the model routing device of the LLM inference gateway provided in the embodiments of this application, or a computer device that integrates the model routing device of the LLM inference gateway. The model routing device of the LLM inference gateway can be implemented in hardware or software. The computer device is an LLM inference gateway (such as a large model enterprise gateway). The following embodiments use the LLM inference gateway as the execution subject to describe in detail the model routing method of the LLM inference gateway provided in this application.

[0025] Figure 1 This is a flowchart illustrating the model routing method for the LLM inference gateway provided in this embodiment of the application. Figure 1 As shown, the method includes the following steps S110-S160.

[0026] S110. Receive an inference request, wherein the inference request carries a target business scenario tag.

[0027] Specifically, the LLM inference gateway receives inference requests from the business side and then obtains the business scenario tags by parsing the request headers or request parameters.

[0028] In some embodiments, the target business scenario tags include tags corresponding to real-time customer service scenarios, asynchronous report generation scenarios, code review scenarios, or general chat scenarios.

[0029] S120: Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set.

[0030] In the engineering practice of enterprise-level LLM inference gateways, routing decisions face an inherent contradiction between three core metrics: cost, quality, and latency. Specifically, LLM call scenarios naturally exhibit a mutually constraining relationship among these three metrics: high-quality flagship models (such as GPT-4o and Claude Opus) typically have the highest token price (up to 50 times that of low-cost models) and the longest response latency (first-byte latency can reach 5-10 seconds); low-cost lightweight models often fail to meet quality standards for complex inference and long-context tasks; and low-latency models generally come at the cost of quality compromise. Any single-metric optimization in existing technologies will inevitably harm the other two metrics, while simple weighted summation methods cannot characterize the non-linear trade-offs among the three metrics.

[0031] To address the aforementioned issues, the method provided in this application performs joint route optimization based on three objectives: cost, quality, and delay.

[0032] Specifically, in some embodiments, the indicator data of multiple different preset indicators in step S120 include cost data, quality data, and delay data, wherein the quality data is a negative number of quality score.

[0033] In some embodiments, a three-objective function formal modeling is required. Specifically, the LLM routing decision is abstracted into the following multi-objective optimization problem: min F(m) = [f1(m), f2(m), f3(m)], where m is a decision variable (a model in the candidate model set), f1(m) = Cost(m) (call cost, unit: USD / 1K Token), f2(m) = -Quality(m) (negative quality score, which transforms maximization into minimization to keep the objective direction consistent), and f3(m) = Latency(m) (response latency, unit: seconds).

[0034] In some embodiments, the candidate model set in step S120 includes all candidate models that can be invoked by the LLM inference gateway.

[0035] In other embodiments, the inference request also carries hard constraints. Before the real-time acquisition of indicator data of multiple different preset indicators corresponding to each candidate model in the candidate model set, the method further includes: filtering candidate models in the candidate model set that do not meet the hard constraints.

[0036] At this point, the candidate model set in step S120 includes candidate models that can be called by the LLM inference gateway and meet the hard constraints.

[0037] In this application embodiment, the hard constraints include context length, compliance requirements, API availability, geographic availability, etc.

[0038] For example, for the hard constraint of context length: the inference request requires 128k context. If a candidate model only supports 8k, it means that the context length of the candidate model does not meet the requirement of the hard constraint, so the candidate model is eliminated. For the hard constraint of compliance requirements, the inference request requires that the model must support data that does not leave region A. If a candidate model is an API in region B, it means that the compliance requirement does not meet the hard constraint, and the candidate model will be eliminated. For the API availability hard constraint, the required model must be callable. If a candidate model is down, has a rate limit, or is under maintenance, it means that the candidate model cannot be called, indicating that the API availability does not meet the hard constraint requirement, and the candidate model is then removed. For the hard constraint of geographic availability, the model to be called must be a node in region A. If a candidate model only has nodes in region B, it means that the geographic availability of the candidate model does not meet the requirements of the hard constraint, so the candidate model is eliminated.

[0039] In this embodiment, before acquiring the model's metric data and performing Pareto multi-objective optimization (step S130), hard constraint verification is performed on candidate models. Models that do not meet mandatory conditions such as context length requirements, compliance requirements, API availability, and geographical availability can be directly eliminated, retaining only legally usable models that meet all hard constraints for subsequent optimization processes. This constraint-before-optimization strategy avoids invalid computation and ensures that routing decisions are compliant, reliable, and executable.

[0040] In some embodiments, after obtaining the model's metric data and before performing Pareto multi-objective optimization, the method further includes: Determine whether the number of candidate models in the candidate model set is greater than a preset threshold; If the number exceeds the preset threshold, then the bucketing rule corresponding to the target business scenario label is determined, and the candidate models in the candidate model set are bucketed to obtain multiple candidate buckets. The candidate bucket corresponding to the target business scenario label among the multiple candidate buckets is taken as the target bucket; Filter out candidate models in the candidate model set that are not located in the target bucket.

[0041] In this embodiment, when the candidate model set is large, the O(n) time complexity of the Pareto front non-dominated sorting is reduced. 2This complexity introduces additional computational overhead. To ensure the real-time performance of online routing, this embodiment further pre-filters the candidate model set by bucketing. Specifically, this embodiment pre-defines the correspondence between business scenario tags and bucketing rules. The scenario tag for real-time customer service corresponds to the delayed data bucketing rule, the scenario tag for asynchronous reporting corresponds to the cost data bucketing rule, and the scenario tag for code review corresponds to the quality data bucketing rule.

[0042] For example, the scenario of real-time customer service corresponds to the delay data binning rule. In this case, the candidate models in the candidate model set whose delay data is less than or equal to the delay data threshold in the binning rule need to be put into the target bucket, the candidate models in the candidate model set whose delay data is greater than the delay data threshold need to be put into the non-target bucket, and finally the models in the candidate model set that are located in the target bucket are retained, and the remaining models are removed.

[0043] As can be seen, this embodiment achieves pre-selection simplification when there are too many candidate models, significantly reducing the number of models participating in non-dominated ranking operations, effectively reducing the computational pressure on the LLM inference gateway, and improving the overall response speed of model routing.

[0044] In other embodiments, when it is determined that the number of candidate models in the candidate model set is greater than a preset threshold, a recent PF (Primary Power) caching strategy can be executed to reduce the computation of the non-dominated ranking algorithm. Specifically, The most recently calculated Pareto front optimal solution set is cached along with candidate model sets and constraints. When a newly requested candidate model set is identical or similar to a cached item, it is directly reused to avoid redundant calculations. Recalculation is only performed when the candidate model set changes, metric fluctuations exceed limits, or the cache expires. Through these optimizations, even with a large candidate set, millisecond-level decision latency can be maintained, meeting the high real-time requirements of online LLM inference gateways.

[0045] In some embodiments, please refer to Figure 2 Specifically, the following steps are used to collect indicator data in real time for each candidate model in the candidate model set, corresponding to multiple different preset indicators: S1201. Obtain the initial cost data of each candidate model from at least one of the official pricing channels and billing receipts of each candidate model in the candidate model set.

[0046] The official pricing channels include model supplier APIs and / or official pricing pages. When the official pricing channels obtained only include model supplier APIs or official pricing pages, the token price of the candidate model in the model supplier API or official pricing page will be used as the cost data of the official pricing channels. When the official pricing channels obtained include model vendor API and official pricing page, the token price of the model vendor API is used as the cost data for the official pricing channels; When the only acquisition method is the official pricing channel, the cost data of the official pricing channel will be used as the initial cost data for the corresponding candidate model. When the cost data acquisition methods include official pricing channels and billing receipts, the cost data in the official pricing channels and the cost data in the billing receipts are compared. If the cost data in the official pricing channels and the cost data in the billing receipts are the same, then the cost data in the official pricing channels or the cost data in the billing receipts are used as the initial cost data for the corresponding candidate model. If the cost data in the official pricing channels and the cost data in the billing receipts are different, then the cost data in the billing receipts are used as the initial cost data for the corresponding candidate model.

[0047] S1202. Determine the initial quality data of each candidate model based on the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set.

[0048] In some embodiments, step S1202 includes: obtaining the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set; and determining the initial quality data of each candidate model based on the automatic evaluation score, manual sampling score, downstream task pass rate, user explicit feedback evaluation score, and the weight of each score source.

[0049] In some embodiments, initial quality data of each candidate model under multiple different business scenario labels can also be obtained. In this case, it is necessary to obtain the automatic evaluation score, manual sampling score, downstream task pass rate and user explicit feedback evaluation score of each candidate model under different business scenario labels in the candidate model set. Then, based on the automatic evaluation score, manual sampling score, downstream task pass rate, user explicit feedback evaluation score and the weight of each score source under different business scenario labels, the initial quality data of each candidate model under different business scenario labels can be determined respectively.

[0050] In this embodiment of the application, different weights can be set for different score sources based on the credibility of the score source.

[0051] S1203. Extract the first byte delay value and generation rate of each candidate model in the candidate model set from the gateway historical call log, and determine the initial delay data of each candidate model based on the first byte delay value and generation rate.

[0052] The embodiments of this application can extract the first-byte latency and average generation rate of each candidate model from the gateway's historical call logs, and use a sliding time window (e.g., 1 hour) to maintain data freshness; Specifically, the first byte delay can be one or more of the first byte delays of P50, P90, and P99, such as the P99 first byte delay.

[0053] Specifically, the two types of data, the first-byte delay value and the generation rate, can be normalized to eliminate the difference in dimensions; then, they can be weighted and fused according to preset weights, and the fused value can be used as the initial delay data for the candidate model.

[0054] S1204. Normalize the initial cost data, the initial quality data, and the initial delay data to obtain the cost data, the quality data, and the delay data.

[0055] Specifically, the normalization process can be Min-Max normalization or Z-Score standardization to eliminate dimensional differences.

[0056] S130. Based on multiple index data of each candidate model in the candidate model set, run a non-dominated sorting algorithm on the candidate model set to calculate the Pareto front optimal solution set.

[0057] In some embodiments, please refer to Figure 3 The Pareto front optimal solution set is calculated through the following steps: S1301. For each candidate model in the candidate model set, if the candidate model is not dominated by other candidate models in the candidate model set, then the candidate model is determined as the preferred model. Wherein, if the index data of the first candidate model in the candidate model set is not inferior to the index data corresponding to the second candidate model, and at least one index data of the first candidate model is strictly superior to the index data corresponding to the second candidate model, then the second candidate model is determined to be dominated by the first candidate model. S1302. Construct the Pareto front optimal solution set based on all preferred models in the candidate model set.

[0058] Specifically, this embodiment involves Pareto dominance determination and frontier calculation. Specifically, for the candidate model set M = {m1 ..., m...} n Running the non-dominated sorting algorithm has a time complexity of O(n^2). 2 For any two models m i With m j If m iIt is not inferior to mj on all three objectives and is strictly superior to m on at least one objective. j Then m is called i Dominate m j (denoted as m) i ≺m j ), that is, m j m i Domination. Identify all models in M ​​that are not dominated by any other model, forming the Pareto front optimal solution set PF = { m∈M | ∄m'∈M, m'≺m}. PF represents the complete theoretically optimal candidate set for this round of decision-making, and its size is typically much smaller than the original candidate model set.

[0059] S140. Determine the weight of each preset indicator based on the target business scenario label.

[0060] Specifically, the gateway pre-stores a table of indicator weights for different business scenarios, matches the current business scenario label, and retrieves the indicator weights of different preset indicators.

[0061] For example, the preset metrics include cost metrics, quality metrics, and latency metrics. The gateway maintains a multi-scenario weight template library, pre-setting recommendation weight vectors for common business scenarios. The weight sequence corresponding to the cost metrics, quality metrics, and latency metrics is w = (cost metric weight w1, quality metric weight w2, latency metric weight w3), for example: Real-time customer service: w = (0.2, 0.3, 0.5), with priority given to delayed responses; Asynchronous report generation: w = (0.5, 0.4, 0.1), cost priority; Code review: w = (0.2, 0.7, 0.1), quality first; General chat: w = (0.33, 0.34, 0.33), the three indicators are balanced; Custom scenarios: Supports enterprises to define scenarios as needed. Each weight vector satisfies w1 + w2 + w3 = 1. When a request arrives, the gateway automatically matches the corresponding weight vector based on the scenario tag in the request metadata.

[0062] S150. Based on the indicator data and indicator weights corresponding to each preset indicator, the target model is selected within the optimal solution set of the Pareto front using the weighted Chebyshev distance minimization criterion.

[0063] In some embodiments, please refer to Figure 4 Specifically, the target model is selected through the following steps: S1501. Based on the index data corresponding to each preset index, determine the optimal index data corresponding to each preset index in multiple candidate models within the Pareto front optimal solution set. S1502. For each candidate model in the Pareto front optimal solution set, determine the first index difference for each preset index. The first index difference is the difference between the index data of the corresponding preset index and the optimal index data. S1503. For each candidate model in the Pareto front optimal solution set, determine the second index difference of each preset index. The second index difference is determined based on the first index difference and the corresponding index weight. S1504. For each candidate model in the Pareto front optimal solution set, the second index difference with the largest median value among the preset indexes is determined as the target index difference of the corresponding candidate model. S1505. The candidate model with the smallest difference in the target index in the Pareto front optimal solution set is determined as the target model.

[0064] Specifically, this embodiment applies the weighted Chebyshev distance minimization criterion to the Pareto front optimal solution set (PF). The formula corresponding to this weighted Chebyshev distance minimization criterion is as follows: ; in, For the final selected target model, PF is the Pareto front optimal solution set. The weight corresponding to the k-th preset indicator, For the index data of the k-th index of the candidate model m, This represents the optimal data for the k-th indicator.

[0065] The advantage of using the weighted Chebyshev distance minimization criterion to select the target model is that it can find the optimal solution at any position on the Pareto front (including non-convex front regions) and will not fall into the limitation of only being able to find the convex hull vertices.

[0066] In some embodiments, after determining the target model, the LLM inference gateway also provides decision visualization and interpretable output for selecting the target model. Specifically, the intermediate data of each routing decision process (candidate model set, Pareto front optimal solution set, coordinates of each candidate model in the three index space, weight vector, and final selection) are presented to the operation and maintenance and business parties in a visual form (such as 3D scatter plot and parallel coordinate plot) to facilitate the review and optimization of routing strategies.

[0067] S160. Forward the inference request to the target model.

[0068] In this embodiment, after selecting the target model, the LLM inference gateway forwards the inference request to the target model.

[0069] Specifically, the routing execution unit can forward inference requests to the target model and record a complete decision log (including the candidate model set, Pareto front optimal solution set, weights, and reasons for selection) for post-analysis and flywheel feedback.

[0070] In some embodiments, the model routing method of the LLM inference gateway includes a target data acquisition unit (execution step S120); a Pareto calculation unit (execution step S130); a weight configuration unit (execution step S140); and a routing execution unit (execution step S150). Each unit is decoupled and independently expandable, working in conjunction with the LLM inference gateway to complete the entire request-level routing process.

[0071] In summary, the embodiments of this application abandon the traditional single-index scheduling and simple linear weighted scheduling methods. By running the non-dominated sorting algorithm, the optimal solution set of Pareto front under multiple indicators can be effectively solved, properly taking into account the conflict relationship between various indicators. Then, the weights are matched in combination with the business scenario, and the target model is selected by relying on the weighted Chebyshev distance, which effectively improves the accuracy of model selection and optimizes the overall routing scheduling effect of the gateway.

[0072] Furthermore, the method implemented through the embodiments of this application also includes at least the following beneficial effects: Mathematically rigorous multi-objective (index) equilibrium: Compared with single-objective optimization or simple weighted summation schemes, the Pareto front algorithm in this application is a classic and rigorous method in the field of multi-objective optimization. It can find all theoretically optimal solutions (non-dominated solution sets), fundamentally eliminating the suboptimality caused by human preference selection; and providing mathematical optimality guarantee for enterprise LLM routing decisions.

[0073] Decision visualization and interpretability: The Pareto front can be presented intuitively to business users and operations teams in the form of 3D scatter plots, parallel coordinate plots, etc., making the trade-offs of routing strategies (cost vs. quality vs. latency) clearly visible. Business users can see which position on the Pareto front their current weight selection corresponds to, and the specific path that can reduce costs to the left or improve quality to the right, which is far superior to black-box decision-making.

[0074] Flexible adaptation to differentiated needs across multiple scenarios: Through the configurability of weight vectors (w1, w2, w3), a single system (the model routing system in the LLM inference gateway) can serve the differentiated needs of various business scenarios, including real-time customer service (0.2, 0.3, 0.5) with latency priority, asynchronous reporting (0.5, 0.4, 0.1) with cost priority, code review (0.2, 0.7, 0.1) with quality priority, and general chat (0.33, 0.34, 0.33) with a balance of three objectives; multiple business scenarios within an enterprise can share the same routing system but each can achieve its intended purpose.

[0075] Coverage capability of Chebyshev distance minimization: Compared with the weighted summation criterion, which can only find the convex hull vertices of the Pareto front, the weighted Chebyshev distance minimization criterion adopted in this application can find the optimal solution at any position on the Pareto front (including non-convex front regions), and the decision-making flexibility is significantly higher than that of traditional schemes.

[0076] Real-time and accurate multi-source data acquisition: cost data comes from real-time pricing API, quality data is a combination of RAGAS automatic assessment + manual sampling + user feedback, and latency data comes from sliding window P50 / P90 / P99 actual measurements; all three types of target data are normalized and integrated into the same decision framework as needed to avoid bias caused by differences in units.

[0077] Real-time performance meets online routing requirements: The non-dominated sorting algorithm for Pareto front computation has a computational cost of less than 2ms for candidate models with n ≤ 20, which meets the real-time requirements of online routing; if the candidate set is larger, optimization measures such as bucket pre-filtering and caching recent PF can be adopted, and the overall decision latency is kept in the millisecond range.

[0078] Multi-scenario weight template library: The LLM inference gateway has pre-built recommended weight vector templates for common business scenarios (real-time customer service, asynchronous reporting, code review, general chat, etc.). When a new business is launched, the template can be directly selected or fine-tuned based on the template, avoiding the dilemma of not knowing how to select values ​​when dealing with pure weight numbers. The template library supports enterprise-customized extensions.

[0079] Visual interface support for weight configuration: This application provides a visual configuration interface with weight slider control, constraint verification (ensuring w1+w2+w3=1), preview of routing results, API export, etc. Business users can complete the configuration through the intuitive interface without understanding the mathematical principles of multi-objective optimization, thus reducing the threshold for use.

[0080] Accuracy of decision-making driven by real-time data: The target data collected are all real-time measured values ​​(not historical averages or manual estimates), ensuring that decisions are always based on the latest status. The current price, current latency, and current quality score of a model are all collected in real time. This real-time nature enables the solution to respond immediately to changes in the status of the model supplier (rate limiting, capacity expansion, pricing adjustments, etc.).

[0081] Operational Reports and Continuous Optimization: The LLM inference gateway continuously records complete data for each decision, including the candidate set, Pareto front, the position of each model in the three-objective space, and the final selected model and weights. This data supports internal routing strategy review meetings, enabling the operations team to adjust weight templates, identify new capability dimension requirements, and optimize the candidate model set.

[0082] The combination of hard constraints and soft optimization: Before Pareto optimization, this application eliminates candidate models that do not meet hard constraints (context length, compliance requirements, API availability, geographical availability, etc.), and then performs multi-objective optimization on the remaining candidates; this two-stage architecture of first hard constraints and then soft optimization ensures that the routing decisions always comply with business rules in regulated industries (finance, healthcare).

[0083] Supports multiple quality assessment methods: Quality objectives support multiple quantification methods, including RAGAS automatic scoring (RAGAS (Retrieval Augmented Generation Assessment) is a fully automatic, no-reference (or weak-reference) evaluation framework for retrieval augmented generation (RAG) systems), Rouge-L scoring (ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is an automatic evaluation metric for text generation; Rouge-L specifically refers to the F1 score based on the longest common subsequence (LCS), which measures the global sequence similarity between the generated text and the reference text), HumanEval pass rate (HumanEval is a code generation capability benchmark proposed by OpenAI in 2021), user feedback scores, downstream business success rate, etc.; Enterprises can flexibly configure the specific definition of quality according to the scenario, so that the solution can adapt to the specific needs of various businesses.

[0084] Synergistic integration with other solutions: The request-level multi-target routing in this application first selects the optimal model, and then other solutions perform token-level or context-level optimization, which can form a multi-level cost-quality-latency comprehensive optimization.

[0085] Figure 5 This is a schematic block diagram of a model routing device for an LLM inference gateway provided in an embodiment of this application. Figure 5 As shown, corresponding to the above-described model routing method for an LLM inference gateway, this application also provides a model routing apparatus 500 for an LLM inference gateway. This model routing apparatus 500 includes a unit for executing the above-described model routing method for an LLM inference gateway, and can be configured within an LLM inference gateway. Specifically, please refer to... Figure 5 The model routing device 500 of the LLM inference gateway includes a transceiver unit 501 and a processing unit 502, wherein: The transceiver unit 501 is used to receive inference requests, the inference requests carrying target business scenario tags; and to collect in real time the indicator data of multiple different preset indicators corresponding to each candidate model in the candidate model set. Processing unit 502 is configured to: run a non-dominated sorting algorithm on the candidate model set based on multiple indicator data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set; determine the indicator weights corresponding to each preset indicator according to the target business scenario label; select the target model in the Pareto front optimal solution set by using the weighted Chebyshev distance minimization criterion based on the indicator data and indicator weights corresponding to each preset indicator; and forward the inference request to the target model.

[0086] In some embodiments, when the processing unit 502 executes the step of running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set to calculate the optimal solution set of the Pareto front, it is specifically used for: For each candidate model in the candidate model set, if the candidate model is not dominated by other candidate models in the candidate model set, then the candidate model is determined as the preferred model. Wherein, if the indicator data of the first candidate model in the candidate model set are not inferior to the indicator data of the second candidate model, and at least one indicator data of the first candidate model is strictly superior to the indicator data of the second candidate model, then the second candidate model is determined to be dominated by the first candidate model. The Pareto front optimal solution set is constructed based on all the preferred models in the candidate model set.

[0087] In some embodiments, when the processing unit 502 performs the step of selecting the target model within the Pareto front optimal solution set based on the indicator data and indicator weights corresponding to each preset indicator, using the weighted Chebyshev distance minimization criterion, it is specifically used for: Based on the index data corresponding to each preset index, determine the optimal index data corresponding to each preset index in multiple candidate models within the Pareto front optimal solution set. For each candidate model in the Pareto front optimal solution set, a first index difference is determined for each preset index. The first index difference is the difference between the index data of the corresponding preset index and the optimal index data. For each candidate model in the Pareto front optimal solution set, a second index difference for each preset index is determined. The second index difference is determined based on the first index difference and the corresponding index weight. For each candidate model in the Pareto front optimal solution set, the second index difference with the largest median value among the preset indexes is determined as the target index difference of the corresponding candidate model. The candidate model with the smallest difference in the target index in the Pareto front optimal solution set is determined as the target model.

[0088] In some embodiments, the indicator data of the plurality of different preset indicators includes cost data, quality data, and latency data; when the transceiver unit 501 performs the step of real-time acquisition of indicator data of the plurality of different preset indicators corresponding to each candidate model in the candidate model set, it is specifically used for: The initial cost data for each candidate model is obtained from at least one of the official pricing channels and billing receipts for each candidate model in the candidate model set. The initial quality data for each candidate model are determined based on the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set. Extract the first byte delay value and generation rate of each candidate model in the candidate model set from the gateway historical call log, and determine the initial delay data of each candidate model based on the first byte delay value and generation rate; The processing unit 502 normalizes the initial cost data, the initial quality data, and the initial delay data to obtain the cost data, the quality data, and the delay data.

[0089] In some embodiments, when the transceiver unit 501 performs the step of determining the initial quality data of each candidate model based on the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set, it is specifically used for: Obtain the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score for each candidate model in the candidate model set; The processing unit 502 determines the initial quality data of each candidate model based on the automatic evaluation score, the manual sampling score, the downstream task pass rate, the user explicit feedback evaluation score, and the weight of each score source.

[0090] In some embodiments, the inference request further carries hard constraints, and before executing the step of collecting indicator data of multiple different preset indicators corresponding to each candidate model in the real-time candidate model set, the processing unit 502 is further configured to: Candidate models in the candidate model set that do not meet the hard constraints are filtered out.

[0091] In some embodiments, before executing the step of running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set, the processing unit 502 is further configured to: Determine whether the number of candidate models in the candidate model set is greater than a preset threshold; If the number exceeds the preset threshold, then the bucketing rule corresponding to the target business scenario label is determined, and the candidate models in the candidate model set are bucketed to obtain multiple candidate buckets. The candidate bucket corresponding to the target business scenario label among the multiple candidate buckets is taken as the target bucket; Filter out candidate models in the candidate model set that are not located in the target bucket.

[0092] In summary, the embodiments of this application abandon the traditional single-index scheduling and simple linear weighted scheduling methods. By running the non-dominated sorting algorithm, the optimal solution set of Pareto front under multiple indicators can be effectively solved, properly taking into account the conflict relationship between various indicators. Then, the weights are matched in combination with the business scenario, and the target model is selected by relying on the weighted Chebyshev distance, which effectively improves the accuracy of model selection and optimizes the overall routing scheduling effect of the gateway.

[0093] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the model routing device and each unit of the above-mentioned LLM inference gateway can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0094] The model routing device of the aforementioned LLM inference gateway can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0095] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. Specifically, the computer device 600 can be an LLM inference gateway.

[0096] See Figure 6 The computer device 600 includes a processor 602, a memory, and a network interface 605 connected via a system bus 601. The memory may include a non-volatile storage medium 603 and internal memory 604.

[0097] The non-volatile storage medium 603 may store an operating system 6031 and a computer program 6032. The computer program 6032 includes program instructions that, when executed, cause the processor 602 to perform a model routing method for an LLM inference gateway.

[0098] The processor 602 provides computing and control capabilities to support the operation of the entire computer device 600.

[0099] The internal memory 604 provides an environment for the execution of the computer program 6032 in the non-volatile storage medium 603. When the computer program 6032 is executed by the processor 602, the processor 602 can execute a model routing method for an LLM inference gateway.

[0100] This network interface 605 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 600 to which the present application is applied. The specific computer device 600 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] The processor 602 is used to run a computer program 6032 stored in the memory to perform the following steps: Receive an inference request, the inference request carrying a target business scenario tag; Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set; Based on multiple index data of each candidate model in the candidate model set, a non-dominated sorting algorithm is run on the candidate model set to calculate the Pareto front optimal solution set; Determine the weight of each preset indicator based on the target business scenario label; Based on the indicator data and indicator weights corresponding to each preset indicator, the target model is selected within the optimal solution set of the Pareto front using the weighted Chebyshev distance minimization criterion. The inference request is forwarded to the target model.

[0102] It should be understood that, in the embodiments of this application, the processor 602 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0103] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0104] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: Receive an inference request, the inference request carrying a target business scenario tag; Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set; Based on multiple index data of each candidate model in the candidate model set, a non-dominated sorting algorithm is run on the candidate model set to calculate the Pareto front optimal solution set; Determine the weight of each preset indicator based on the target business scenario label; Based on the indicator data and indicator weights corresponding to each preset indicator, the target model is selected within the optimal solution set of the Pareto front using the weighted Chebyshev distance minimization criterion. The inference request is forwarded to the target model.

[0105] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0108] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model routing method for an LLM inference gateway, characterized in that, include: Receive an inference request, the inference request carrying a target business scenario tag; Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set; Based on multiple index data of each candidate model in the candidate model set, a non-dominated sorting algorithm is run on the candidate model set to calculate the Pareto front optimal solution set; Determine the weight of each preset indicator based on the target business scenario label; Based on the indicator data and indicator weights corresponding to each preset indicator, the target model is selected in the optimal solution set of the Pareto front using the weighted Chebyshev distance minimization criterion. Forward the inference request to the target model; Before calculating the Pareto front optimal solution set by running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set, the method further includes: Determine whether the number of candidate models in the candidate model set is greater than a preset threshold; If the number exceeds the preset threshold, then the bucketing rule corresponding to the target business scenario tag is determined, and the candidate models in the candidate model set are bucketed to obtain multiple candidate buckets. Among them, the real-time customer service scenario tag corresponds to the delayed data bucketing rule, the asynchronous report generation scenario tag corresponds to the cost data bucketing rule, and the code review scenario tag corresponds to the quality data bucketing rule. The candidate bucket corresponding to the target business scenario label among the multiple candidate buckets is taken as the target bucket; Filter out candidate models in the candidate model set that are not located in the target bucket.

2. The method according to claim 1, characterized in that, The method involves running a non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set, including: For each candidate model in the candidate model set, if the candidate model is not dominated by other candidate models in the candidate model set, then the candidate model is determined as the preferred model. Wherein, if the indicator data of the first candidate model in the candidate model set are not inferior to the indicator data of the second candidate model, and at least one indicator data of the first candidate model is strictly superior to the indicator data of the second candidate model, then the second candidate model is determined to be dominated by the first candidate model. The Pareto front optimal solution set is constructed based on all the preferred models in the candidate model set.

3. The method according to claim 1, characterized in that, The step of selecting the target model within the Pareto front optimal solution set based on the indicator data and indicator weights corresponding to each preset indicator, using the weighted Chebyshev distance minimization criterion, includes: Based on the index data corresponding to each preset index, determine the optimal index data corresponding to each preset index in multiple candidate models within the Pareto front optimal solution set. For each candidate model in the Pareto front optimal solution set, a first index difference is determined for each preset index. The first index difference is the difference between the index data of the corresponding preset index and the optimal index data. For each candidate model in the Pareto front optimal solution set, a second index difference is determined for each preset index. The second index difference is determined based on the first index difference and the corresponding index weight. For each candidate model in the Pareto front optimal solution set, the second index difference with the largest median value among the preset indexes is determined as the target index difference of the corresponding candidate model. The candidate model with the smallest difference in the target index in the Pareto front optimal solution set is determined as the target model.

4. The method according to claim 1, characterized in that, The indicator data for the multiple different preset indicators include cost data, quality data, and latency data; the indicator data for the multiple different preset indicators corresponding to each candidate model in the real-time acquisition candidate model set includes: The initial cost data for each candidate model is obtained from at least one of the official pricing channels and billing receipts for each candidate model in the candidate model set. The initial quality data for each candidate model are determined based on the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score of each candidate model in the candidate model set. Extract the first byte delay value and generation rate of each candidate model in the candidate model set from the gateway historical call log, and determine the initial delay data of each candidate model based on the first byte delay value and generation rate; The initial cost data, the initial quality data, and the initial delay data are normalized to obtain the cost data, the quality data, and the delay data.

5. The method according to claim 4, characterized in that, The initial quality data for each candidate model is determined based on its automatic evaluation score, manual sampling score, downstream task pass rate, and explicit user feedback evaluation score in the candidate model set. This includes: Obtain the automatic evaluation score, manual sampling score, downstream task pass rate, and user explicit feedback evaluation score for each candidate model in the candidate model set; The initial quality data for each candidate model are determined based on the automatic evaluation score, the manual sampling score, the downstream task pass rate, the user explicit feedback evaluation score, and the weight of each score source.

6. The method according to claim 1, characterized in that, The inference request also carries hard constraints. Before the real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set, the method further includes: Candidate models in the candidate model set that do not meet the hard constraints are filtered out.

7. A model routing device for an LLM inference gateway, characterized in that, include: The transceiver unit is used to receive inference requests, the inference requests carrying target business scenario tags; Real-time acquisition of indicator data for multiple different preset indicators corresponding to each candidate model in the candidate model set; The processing unit is configured to: run a non-dominated ranking algorithm on the candidate model set based on multiple indicator data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set; determine the indicator weights corresponding to each preset indicator according to the target business scenario label; select the target model in the Pareto front optimal solution set using the weighted Chebyshev distance minimization criterion based on the indicator data and indicator weights corresponding to each preset indicator; and forward the inference request to the target model. Before the step of executing the non-dominated ranking algorithm on the candidate model set based on multiple index data of each candidate model in the candidate model set to calculate the Pareto front optimal solution set, the processing unit is further configured to: Determine whether the number of candidate models in the candidate model set is greater than a preset threshold; If the number exceeds the preset threshold, then the bucketing rule corresponding to the target business scenario tag is determined, and the candidate models in the candidate model set are bucketed to obtain multiple candidate buckets. Among them, the real-time customer service scenario tag corresponds to the delayed data bucketing rule, the asynchronous report generation scenario tag corresponds to the cost data bucketing rule, and the code review scenario tag corresponds to the quality data bucketing rule. The candidate bucket corresponding to the target business scenario label among the multiple candidate buckets is taken as the target bucket; Filter out candidate models in the candidate model set that are not located in the target bucket.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model routing method of the LLM inference gateway as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the model routing method of the LLM inference gateway as described in any one of claims 1-6.

Images