A model calling request routing method, device, equipment, storage medium and product

Abstract

The application discloses a model calling request routing method and device, equipment, storage medium and product, and relates to the computer technical field. The method comprises the following steps: obtaining a request text of a model calling request; calculating the similarity between the request vector of the request text and the embedding vector of each task domain, and determining the task domain with the highest similarity as a target task domain; if the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold value, determining the fusion quality score of each candidate model in the target task domain, determining the utility value of each candidate model based on the fusion quality score, the cost score and the delay score of each candidate model, determining the candidate model with the highest utility value as a target model, and routing the model calling request to the target model. The method can improve the accuracy of model calling request routing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and product for routing model call requests. Background Technology

[0002] As generative artificial intelligence enters a new stage of large-scale industrial application, deploying or integrating multiple large language models (LLMs) on a unified platform can adapt to the diverse requirements of various business scenarios for model capabilities. These models differ in overall performance, inference cost, response speed, context length, and performance on specialized tasks (such as code generation, mathematical reasoning, and legal text parsing). Although general-purpose large models (such as GPT-4) perform exceptionally well, their high invocation cost and slow response speed limit their use in high-frequency or cost-sensitive scenarios.

[0003] Traditional solutions often rely on load balancing strategies or static routing mechanisms to bind a specific model to a particular type of task. While this approach was adequate in the early stages when model functionality was relatively simple and request patterns were predictable, it is ill-suited to today's highly dynamic, complex, and cost-sensitive business environments.

[0004] Therefore, traditional routing methods suffer from poor accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and product for routing model invocation requests, which can improve the accuracy of model invocation request routing.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides a method for routing model invocation requests, including:

[0008] Get the request text of the model call request;

[0009] Calculate the similarity between the request vector of the request text and the embedding vector of each task domain, and determine the task domain with the highest similarity as the target task domain;

[0010] If the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold, the fusion quality score of each candidate model in the target task domain is determined, and the utility value of each candidate model is determined based on the fusion quality score, cost score and latency score of each candidate model.

[0011] The candidate model with the highest utility value is selected as the target model;

[0012] The model call request is routed to the target model.

[0013] Optionally, the method further includes:

[0014] If the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold, the single quality score of each candidate model in the target task domain is determined, and the utility value of each candidate model is determined based on the single quality score, cost score and latency score of each candidate model.

[0015] The candidate model with the highest utility value is selected as the target model;

[0016] The model call request is routed to the target model.

[0017] Optionally, determining the candidate model with the highest utility value as the target model includes:

[0018] If the utility value is greater than or equal to the utility value threshold, the candidate model with the highest utility value is determined as the target model;

[0019] If the utility value is less than the utility value threshold, the fusion quality score of each candidate model in the target task domain is determined. Based on the fusion quality score, cost score and latency score of each candidate model, the utility value of each candidate model is determined, and the candidate model with the highest utility value is determined as the target model.

[0020] Optionally, the method further includes:

[0021] Obtain the latency deviation, quality deviation, remaining budget, and instantaneous endpoint evaluation load of the model call request;

[0022] Adaptive weights are determined based on delay skew, quality skew, remaining budget, and instantaneous load at endpoints.

[0023] The determination of the utility value of each candidate model based on its fusion quality score, cost score, and latency score includes:

[0024] Based on the fusion quality score, cost score, and latency score of each candidate model, and combined with adaptive weights, the utility value of each candidate model is determined.

[0025] Optionally, the method further includes:

[0026] Calculate the request difficulty corresponding to the request text;

[0027] Based on the difficulty of the request, the constraints of the candidate model are dynamically determined;

[0028] The step of determining the candidate model with the highest utility value as the target model includes:

[0029] The candidate model with the highest utility value and that satisfies the constraints is determined as the target model.

[0030] Secondly, this application provides a routing device for model invocation requests, comprising:

[0031] The acquisition module is used to obtain the request text of the model call request;

[0032] The calculation module is used to calculate the similarity between the request vector of the request text and the embedding vector of each task domain, and to determine the task domain with the highest similarity as the target task domain.

[0033] The determination module is used to determine the fusion quality score of each candidate model in the target task domain if the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold; and to determine the utility value of each candidate model based on the fusion quality score, cost score, and latency score of each candidate model; and to determine the candidate model with the highest utility value as the target model.

[0034] The processing module is used to route the model call request to the target model.

[0035] Thirdly, this application provides a computing device, including a memory and a processor;

[0036] The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0037] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0038] As can be seen from the above technical solution, this application has at least the following beneficial effects:

[0039] First, by calculating the similarity between the request text and the embedding vectors of each task domain, the target task domain is accurately located. Combined with the difficulty of the request, the constraints of the candidate model are dynamically adjusted, which greatly improves the matching accuracy between the task and the model capability. This avoids the waste of resources of general-purpose large models in high-frequency / low-cost scenarios, and also ensures the processing quality of professional tasks or high-difficulty requests, breaking through the adaptation limitations of the traditional fixed binding mode.

[0040] Secondly, a dual-dimensional evaluation system integrating quality score and single quality score is introduced. A utility value evaluation model is constructed by combining cost score and latency score. Adaptive weights are dynamically adjusted by latency deviation, quality deviation, remaining budget and instantaneous load. This achieves multi-objective optimization of model performance, call cost and response speed. Under the premise of meeting business quality requirements, it effectively reduces the operating cost of cost-sensitive scenarios and improves the response efficiency of high-frequency requests.

[0041] Third, by using both similarity threshold and utility threshold, as well as dynamic adaptation of constraints, the robustness of routing decisions is enhanced. It can quickly lock the optimal single model when the task domain matching is clear, and select the optimal solution through fusion quality evaluation when the matching is ambiguous or the utility of a single model is insufficient. This avoids the quality shortcomings or resource redundancy caused by the rigid decision-making of traditional solutions.

[0042] Fourth, the design of adaptive weights and dynamic constraints enables it to flexibly adapt to the dynamic changes of different business scenarios (such as load fluctuations, budget adjustments, quality requirement upgrades, etc.), and achieve optimal resource allocation without manual intervention. This enhances the scalable application capability of the multi-LLM integration platform in complex business environments and provides a routing solution that combines accuracy, economy and flexibility for diverse business scenarios.

[0043] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a model invocation request routing method provided in an embodiment of this application;

[0045] Figure 2 A schematic diagram of a routing device for a model invocation request provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0047] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0048] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0049] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0050] Large Language Models (LLMs) refer to artificial intelligence models with a large number of parameters and powerful natural language understanding and generation capabilities. They can be adapted to diverse professional tasks such as code generation, mathematical reasoning, and legal text parsing. Different models have significant differences in overall performance, inference cost, response speed, and context length.

[0051] As generative artificial intelligence enters the stage of large-scale industrial application, enterprises have an increasingly urgent need for multi-LLM integration platforms. Different business scenarios require both the comprehensive performance of general-purpose large models and the cost advantages and response efficiency of specialized models, which places higher demands on the routing decisions for model calls. However, traditional solutions generally adopt load balancing strategies or static routing mechanisms, which bind a certain type of task to a specific model. This model is no longer suitable for today's highly dynamic, complex, and cost-sensitive business environment. The main technical problems are low accuracy in matching tasks with models, rigid resource allocation, and difficulty in balancing cost and performance.

[0052] There are three main reasons for this problem: First, traditional solutions lack a precise matching mechanism between request text and task domain characteristics. They only roughly bind models based on task type, which cannot fully adapt to the differences in capabilities of different models in professional tasks. This results in serious resource waste of general-purpose large models in high-frequency / low-cost scenarios, while in professional scenarios, insufficient model adaptation affects processing quality. Second, traditional solutions do not establish a multi-dimensional comprehensive evaluation system. They only focus on load or a single performance indicator, ignoring the coordinated optimization of cost, latency, and quality. This makes it impossible to meet the operational needs of cost-sensitive scenarios and the response efficiency requirements of high-frequency requests. Third, the routing rules of traditional solutions are fixed and lack the ability to adapt to dynamic changes in business (such as load fluctuations, budget adjustments, and differences in request difficulty). This makes it difficult to cope with dynamic changes in requirements in complex scenarios, ultimately resulting in insufficient robustness and flexibility of routing decisions.

[0053] In view of this, embodiments of this application provide a method for routing model invocation requests, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0054] To address the issues of low matching accuracy, rigid resource allocation, and difficulty in balancing cost and performance inherent in traditional load balancing or static routing mechanisms in multi-LLM integration platforms, this application firstly addresses the problem of task type identification by calculating the similarity between request text and task domain embedding vectors, and dynamically defines model constraints based on request difficulty to resolve the mismatch between task and model capabilities. Secondly, it introduces utility value as a decision indicator, and dynamically adjusts adaptive weights based on real-time business status, including latency deviation, quality deviation, remaining budget, and instantaneous load, to achieve multi-objective collaborative optimization. Finally, through dual judgment of similarity threshold and utility value threshold, it flexibly switches between single model invocation and fusion model evaluation modes, ensuring the robustness of routing decisions in different scenarios and providing a routing solution for large-scale industrial applications of multi-LLM.

[0055] The routing decision process for this application is as follows:

[0056] Phase 1: Semantic Candidate Set Filtering. By calculating the similarity between the request vector of the request text and the embedding vectors of each task domain, the target task domain with the highest similarity is identified, and all pre-configured models under this target task domain are used as the initial candidate model set. In this way, only models that match the semantic domain of the request are selected, excluding cross-domain irrelevant models. For example, code generation requests are excluded from the candidate range of legal text parsing models, significantly reducing the number of models used in subsequent decision-making.

[0057] The second stage: multi-objective utility decision-making. Within the candidate model set selected in the first stage, the quality score, cost score, and latency score of each candidate model are further calculated. These scores are then combined with adaptive weights to determine the utility value, ultimately selecting the optimal model. This avoids the inefficient approach of scoring each model individually within the system, improving routing efficiency while ensuring decision-making accuracy.

[0058] Therefore, the solution proposed in this application ensures that the routing decision has both semantic-level domain adaptability and reduces computational overhead by narrowing the decision scope. This differs from the conventional approach of directly scoring all models with multiple objectives in traditional routing solutions, and achieves collaborative optimization of screening and decision-making.

[0059] To make the technical solution of this application clearer and easier to understand, the following describes a routing method for model call requests provided by an embodiment of this application, in conjunction with the accompanying drawings. Figure 1 As shown, this figure is a flowchart of a model invocation request routing method provided in an embodiment of this application. The method includes:

[0060] S201, The processing device obtains the request text of the model call request.

[0061] A model invocation request refers to a requirement instruction initiated by a user or business system to a multi-LLM integration platform, which needs to be processed through a specific model. Examples include code generation requests, legal text parsing requests, and mathematical reasoning requests. This model invocation request contains the information to be processed, i.e., the request text, as well as relevant business parameters, such as latency requirements and budget constraints.

[0062] The request text is the text content in the model call request that carries the business requirements. It is the basis for the model to process, such as writing a piece of Python data visualization code or parsing the breach of contract clauses in this contract. It directly reflects the user's specific task requirements.

[0063] First, the processing device receives model call requests from users or business systems through a preset communication interface. Then, the processing device extracts the request text carrying the business requirements from the request. This step is the basis for all subsequent routing decisions. Its purpose is to obtain key information that can characterize what the user needs the model to do, and to provide raw data support for subsequent steps such as calculating request vectors, matching target task domains, and judging request difficulty, so as to ensure that subsequent routing decisions can meet the user's actual task requirements.

[0064] In short, the purpose of this step is to obtain key information so that the processing equipment can clearly understand the user's specific needs, laying the foundation for selecting the optimal model in the future.

[0065] S202, The processing device calculates the similarity between the request vector of the request text and the embedding vector of each task domain, and determines the task domain with the highest similarity as the target task domain.

[0066] A request vector is a high-dimensional numerical vector obtained by extracting features from a request text and encoding it through a pre-trained language model. It can quantitatively represent the semantic features of the request text, such as task type, demand direction, and professional domain attributes, thus transforming the textual demand into a computable numerical form.

[0067] The embedding vector of a task domain refers to a high-dimensional numerical vector obtained by uniformly encoding the features of each preset professional task domain. These professional task domains include code generation domain, legal text parsing domain, mathematical reasoning domain, etc. Each vector corresponds to a feature set of a specific task domain and is a numerical identifier of that task domain. For example, the embedding vector of the legal text parsing domain will highlight features such as legal terminology and clause logic.

[0068] Similarity refers to the degree of semantic association between the request vector and the embedding vectors of each task domain. The higher the value, the higher the matching degree between the request text and the corresponding task domain. For example, the similarity between the request vector and the embedding vector of the legal text parsing domain is 0.92, which is much higher than the 0.35 with the code generation domain, indicating that the request is more suitable for the legal text parsing task.

[0069] The target task domain refers to the task domain with the highest similarity to the request vector of the request text. It is the result of positioning the professional field to which the user's needs belong. Subsequently, candidate models will only be selected from this task domain to avoid inefficiency or quality deviation caused by cross-domain model matching.

[0070] First, the processing device uses a pre-defined feature encoding algorithm, such as a Transformer-based pre-trained model, to convert the acquired request text into a request vector that can represent its semantic requirements. Second, the processing device calls the embedding vectors of all task domains pre-stored in the system and compares the degree of correlation between the request vector and each task domain embedding vector through a similarity calculation algorithm, such as cosine similarity. Finally, the processing device selects the task domain with the highest similarity to the request vector from all task domains and determines it as the target task domain.

[0071] The expression for calculating similarity is:

[0072]

[0073] in, Indicates similarity. This represents the request vector, which is the numerical vector obtained after the current user's request text has been transformed by the embedding model. The embedding vector represents the task domain. This represents the Euclidean norm.

[0074] This step breaks through the limitations of the traditional solution's extensive task classification. Through semantic-level matching, it enables the processing device to clearly identify which type of professional task the user's needs belong to.

[0075] S203. If the similarity between the request vector and the embedding vector of the target task domain is less than the similarity threshold, determine the fusion quality score of each candidate model in the target task domain, and determine the utility value of each candidate model based on the fusion quality score, cost score and latency score of each candidate model.

[0076] The similarity threshold is a pre-defined critical value used to judge the quality of semantic matching. If the cosine similarity between the request vector and the target task domain embedding vector is lower than this similarity threshold, it indicates that the semantics of the current request are relatively ambiguous, complex, or belong to the edge case. Fast semantic matching alone is insufficient to make a high-quality routing decision, so a more complex and accurate second-level decision mechanism needs to be activated.

[0077] The fusion quality score is a comprehensive prediction of the quality of a candidate model's response to a specific request. It's called fusion because it doesn't come from a single source but integrates the results of multiple quality assessment methods. These include the model's capability-request difficulty matching based on item response theory, the ELO dynamic score reflecting the model's recent overall performance, and a performance score predicted in real-time based on request characteristics. This multi-source fusion strategy significantly improves the accuracy and robustness of quality predictions.

[0078] The cost score refers to the economic cost of using a candidate model to complete a current request. It is calculated based on the model's pricing strategy and the estimated resource consumption of the request. The model's pricing strategy includes the cost per thousand tokens, while the estimated resource consumption covers the total number of tokens required for the request and the expected response. This cost score is a practical economic indicator, and optimizing it directly affects the system's operating costs.

[0079] The latency score is a predicted value for the time required for a candidate model to process the current request and return a result. It reflects the model's response speed and is determined by the model's inference performance, current network conditions, and server load.

[0080] The utility value is a comprehensive score that unifies three competing objectives—quality, cost, and delay—into a single dimension. It serves as the ultimate basis for routing decisions. The utility value is calculated through a utility function that integrates the three scores according to the weights determined by the system's real-time strategy. Its goal is to find the model that maximizes overall revenue under given constraints.

[0081] When a user request cannot be clearly categorized into any task domain (i.e., the similarity is below a threshold), a thorough evaluation of all candidate models within the target task domain is first performed. A fusion quality score is calculated for each model to more accurately predict its ability to handle this complex request. The expression for calculating the fusion quality score is:

[0082]

[0083] in, Representation Model Regarding the request The fusion quality score, This represents the quality score based on the project response theory. Representation Model ELO dynamic score, This represents the quality score predicted in real time based on a machine learning model. This represents the first weighting coefficient. This represents the second weighting coefficient. This represents the third weighting coefficient.

[0084] Subsequently, this ability prediction of each model is combined with its cost score and delay score, and a utility function is used to make a comprehensive trade-off to obtain the utility value of the candidate model.

[0085] Before calculating the utility value of the candidate model, adaptive weights must be determined. The specific steps are as follows:

[0086] The processing device obtains the latency deviation, quality deviation, remaining budget, and endpoint evaluation instantaneous load of the model call request.

[0087] Latency deviation refers to the difference between the actual response speed of a system during runtime and the pre-set performance target. It measures whether the system's current service timeliness meets the promised standards, such as whether it meets the millisecond-level response time promised to users.

[0088] Quality deviation refers to the degree to which the overall quality level of the responses or content produced by the system deviates from the minimum acceptable standard. It reflects whether the generated results meet the standards in terms of accuracy, relevance, and usefulness.

[0089] Remaining budget refers to the total amount of model call costs that have not yet been consumed within the current billing cycle. It is directly related to the operating costs of the business and is an indicator for controlling economic expenditures. The amount of remaining budget determines the aggressiveness of the system's cost control strategy.

[0090] Endpoint load assessment provides an overall evaluation of the current processing capacity and workload of all model service nodes in the system. It reflects the real-time pressure level of the entire computing resource pool. High load typically indicates strained system resources, requiring priority to be given to rapid response and traffic diversion to avoid service overload or crashes.

[0091] The processing device will actively pull four key global status indicators from the observation and monitoring module: obtain the latency deviation by aggregating the response time data of each model node in real time and comparing it with the preset target; obtain the quality deviation by sampling and scoring the model output through an automated evaluation process and comparing it with the quality baseline; obtain the remaining budget in the current billing cycle by querying the budget management service; and obtain the endpoint evaluation instantaneous load by summarizing the real-time resource utilization of all model endpoints through the health check service.

[0092] The processing equipment determines adaptive weights based on delay deviation, quality deviation, remaining budget, and instantaneous load at the endpoint evaluation.

[0093] Adaptive weights are a weight vector consisting of three components, representing the degree of importance that routing decisions should place on the three objectives of quality, cost, and delay under the current system state.

[0094] First, the system state indicators are mapped to the basic weights using a sigmoid function, and the calculation expression is as follows:

[0095]

[0096] in, The base weights representing the delay, The basic weights representing quality Indicates the basic weight of costs. This represents the Sigmoid function. Indicates delay deviation, Indicates quality deviation. Indicates the remaining budget. Represents the budget risk function. This represents a small constant to prevent the denominator from being zero. The sensitivity coefficient represents the delay. The sensitivity coefficient represents the quality. This represents the cost sensitivity coefficient.

[0097] The basic weights are then normalized, and the calculation expression is as follows:

[0098]

[0099] in, Represents the normalized result The basic weights, express The basic weights, Indicates delay ,quality ,cost A set of.

[0100] The processing equipment makes targeted adjustments to the normalized base weights based on the overall load conditions. The calculation expression is as follows:

[0101] If the endpoint evaluates instantaneous load High load threshold (Overload):

[0102]

[0103] in, This represents the corrected delay weight. The base weights represent the normalized delay. This represents the delay-weighted load correction factor. This indicates that the endpoint is evaluating the instantaneous load. This indicates a high load threshold.

[0104] If the endpoint evaluates instantaneous load Indicates low load threshold (Underloaded):

[0105]

[0106] in, This represents the corrected quality weight. The base weights representing the normalized quality. This represents the quality weighting load correction factor. This indicates a low load threshold.

[0107] The corrected weights are then normalized to form an adaptive weight vector used for decision-making in this request. :

[0108]

[0109] in, Indicates a request of Adaptive weights Indicates the revised version Weight, This represents the adjusted cost weights. Represents the adaptive weight vector. Indicates a request Quality adaptive weights, Indicates a request Cost adaptive weights, Indicates a request Delayed adaptive weights.

[0110] The processing device determines the utility value of each candidate model based on its fusion quality score, cost score, and latency score, combined with adaptive weights. The expression for calculating the utility value is:

[0111]

[0112] in, Indicates a request Routing to candidate models The utility value is the value of the model; the higher the value, the better the model is in this decision. Representation Model Processing requests Cost fraction, Representation Model Processing requests The delay score.

[0113] The method also includes a step of determining the target model when the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to a similarity threshold, as shown below:

[0114] If the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold, determine the individual quality score of each candidate model in the target task domain. Based on the individual quality score, cost score, and latency score of each candidate model, determine the utility value of each candidate model.

[0115] A single quality score is a predicted quality score of a candidate model's ability to handle the current request, obtained from a single data source or a simplified model. Compared to a fusion quality score, a single quality score originates from a pre-generated model profile, which is a data table recording the average performance of each model on different tasks. This type of score is faster to obtain and has lower computational overhead, making it suitable for scenarios requiring rapid decision-making.

[0116] When the processing device finds that a user request is highly semantically matched with a target task domain, that is, when the similarity is greater than or equal to the threshold, it determines that the request type is clear and the complexity is likely not high, and therefore initiates an efficient, low-latency decision-making process.

[0117] In this approach, the processing device directly retrieves the pre-defined individual quality scores for each candidate model within the target task domain from the pre-set model profiles, without initiating complex fusion quality calculations. Subsequently, the processing device combines the cost score and latency score of the model obtained from real-time monitoring to quickly calculate the utility value of each candidate model using a unified utility function. The expression for calculating the utility value is:

[0118]

[0119] in, Representative request Routing to candidate models The utility value, Representative model A single quality score when processing requests in this target task domain. This is a preliminary estimate obtained from a table lookup in a static model profile, reflecting the model's average capability in this domain.

[0120] S204. The processing device determines the candidate model with the highest utility value as the target model.

[0121] If the utility value is greater than or equal to the utility value threshold, the candidate model with the highest utility value is determined as the target model.

[0122] The utility value threshold is a pre-defined acceptable standard, representing the minimum overall performance standard that the system can accept. It ensures that the selected model is not only the best among all candidates, but its absolute performance must also reach a basic level.

[0123] The target model refers to the optimal model that is ultimately selected to execute the current model call request after the system's intelligent routing decision-making process.

[0124] After calculating the utility value of each candidate model, the processing device does not simply select the one with the highest score. Instead, it first establishes a quality threshold. The processing device compares the highest utility value with a preset utility value threshold. Only if the highest score reaches or exceeds this threshold will the candidate model with the highest utility value be officially designated as the target model.

[0125] If the utility value is less than the utility value threshold, the fusion quality score of each candidate model in the target task domain is determined. Based on the fusion quality score, cost score and latency score of each candidate model, the utility value of each candidate model is determined, and the candidate model with the highest utility value is determined as the target model.

[0126] When the processing device discovers that even the candidate model with the highest utility value cannot reach the basic passing grade (utility value threshold) after the initial screening, it will not force the selection of an unqualified model, but will initiate a higher-level decision loop.

[0127] In this upgrade process, the processing device first recalculates a more accurate fusion quality score for each candidate model in the target task domain, replacing the previous potentially overly coarse single quality score. Next, using this more reliable quality score, along with cost and latency scores, the processing device recalculates the utility value of each candidate model. Finally, based on the results of this re-evaluation, the processing device identifies the candidate model with the highest utility value as the target model.

[0128] This method also includes constraints on the candidate models, and the specific steps are as follows:

[0129] The processing device calculates the request difficulty corresponding to the request text; based on the request difficulty, it dynamically determines the constraints of the candidate models; and selects the candidate model with the highest utility value that meets the constraints as the target model.

[0130] Request difficulty is a quantitative assessment of the complexity of the request text itself and the cognitive load required to process it. It is a scalar value derived through a computational model based on the inherent characteristics of the request, such as length, semantic ambiguity, logical complexity, and domain specialization.

[0131] The constraint is a dynamically changing threshold that ensures the selected model has the minimum capability required to handle the current request, thus preventing an inadequate model from handling a high-difficulty task and causing a collapse in response quality.

[0132] The processing device first analyzes the characteristics of the request text to quantitatively assess the difficulty of the request. Then, based on this difficulty value, a quality constraint is dynamically set: the model's quality score must reach or exceed a minimum threshold determined by the difficulty. Finally, during routing decisions, instead of simply selecting the model with the highest utility value, the device chooses the target model from all candidate models that satisfy the constraint.

[0133] The quality constraint is embodied in a dynamic quality threshold function. The expression is:

[0134]

[0135] in, This represents the quality threshold function. This is a preset basic quality threshold, suitable for the simplest requests. It's a difficulty coefficient, controlling the slope of the quality threshold as the difficulty increases. This is the calculated request difficulty. This quality threshold function indicates that the higher the request difficulty, the higher the quality requirements for the model.

[0136] Target Model The following optimization problem was identified:

[0137]

[0138] in, Represent the target model, It is a model The quality score (single quality score or combined quality score). It is a set of feasible models that satisfy dynamic quality constraints.

[0139] In some embodiments, the constraint based on request difficulty is specifically implemented as follows:

[0140] First difficulty high-difficulty request constraint: When the calculated request difficulty is greater than or equal to the preset high difficulty threshold, a quality lower limit constraint is imposed on the candidate model. That is, the quality score (single quality score or fused quality score) of the candidate model must be greater than the preset quality threshold to ensure the processing quality of high-difficulty tasks and avoid sacrificing performance due to cost or delay optimization.

[0141] The second difficulty level is the low-difficulty request constraint: when the calculated request difficulty is less than the preset low-difficulty threshold, a cost upper limit constraint and / or a delay upper limit constraint are imposed on the candidate model. That is, the cost score of the candidate model must be less than the preset cost threshold, and / or the delay score must be less than the preset delay threshold. Priority is given to ensuring the economy and response efficiency of high-frequency, low-value scenarios. These scenarios include simple information queries, regular text generation, etc.

[0142] The third difficulty level includes a difficulty request constraint: when the request difficulty is between the high difficulty threshold and the low difficulty threshold, a balance constraint of quality, cost, and latency is adopted. That is, the candidate model must simultaneously meet the minimum quality requirement, the maximum cost limit, and the maximum latency limit to achieve a balanced optimization of multiple objectives.

[0143] The relationship between the above constraints and request difficulty can be preset through the configuration file, allowing for flexible adjustment of the constraint thresholds corresponding to each difficulty level according to business scenario requirements.

[0144] S205. The processing device routes the model call request to the target model.

[0145] Routing refers to the process of intelligent scheduling and forwarding. It is not merely a simple network layer packet forwarding, but an application layer decision-making and execution action that accurately distributes specific tasks to the optimal computing resources based on preset strategies and real-time decision results.

[0146] After determining the target model, the processing device will forward the original model call request, which contains information such as request text and business parameters, to the target model through the platform's preset communication link and scheduling mechanism.

[0147] The purpose of this step is to achieve a seamless connection between decision results and actual processing: on the one hand, accurate routing ensures that requests are received by the most suitable model, avoiding problems such as poor processing quality and resource waste caused by traditional coarse forwarding; on the other hand, relying on the scheduling capabilities of the integration platform, it ensures the stability and efficiency of request transmission, laying the foundation for the subsequent target model to respond quickly and process requests with high quality, and meeting the demands of diverse business scenarios.

[0148] In some embodiments, to achieve dynamic optimization and continuous iteration of routing decisions, this application also introduces a routing policy training and update process, enabling the system to have self-learning capabilities and avoiding the rigid limitations of static rules. Specifically, this includes:

[0149] First, collect historical data. Record the characteristics of each model call request, including request text, request vector, request difficulty, target task domain, latency bias, quality bias, remaining budget, endpoint evaluation instantaneous load, the target model identifier selected during routing, and the actual feedback results after the call, including the actual quality score of the model's returned results, the actual call cost, and the actual response latency.

[0150] Secondly, based on the collected historical data, the model quality parameters are dynamically updated using supervised learning, Item Response Theory (IRT), or ELO scoring systems. Specifically, the IRT model optimizes the quality fit coefficients of different models under various task domains and request difficulties, the ELO dynamic scoring adjusts the dynamic performance rating of the model in real time, and supervised learning trains the parameter weights of the routing scoring function, i.e., the utility value calculation function.

[0151] Finally, the update trigger mechanism is implemented. Two modes are adopted: scheduled updates and real-time updates. Scheduled updates can process historical data in batches according to preset periods to optimize global parameters, while real-time updates can trigger incremental learning after a single call feedback result is generated to quickly adapt to sudden business changes, such as a sudden drop in the performance of a certain model or the introduction of new business scenarios.

[0152] Through the aforementioned self-learning update mechanism, the system can continuously accumulate business experience, continuously optimize the accuracy of model quality assessment and the adaptability of routing decisions, and gradually improve the routing accuracy and resource utilization efficiency in complex scenarios.

[0153] Based on the above description, this application has the following beneficial effects:

[0154] First, by calculating the similarity between the request text and the embedding vectors of each task domain, the target task domain is accurately located. Combined with the difficulty of the request, the constraints of the candidate model are dynamically adjusted, which greatly improves the matching accuracy between the task and the model capability. This avoids the waste of resources of general-purpose large models in high-frequency / low-cost scenarios, and also ensures the processing quality of professional tasks or high-difficulty requests, breaking through the adaptation limitations of the traditional fixed binding mode.

[0155] Secondly, a dual-dimensional evaluation system integrating quality score and single quality score is introduced. A utility value evaluation model is constructed by combining cost score and latency score. Adaptive weights are dynamically adjusted by latency deviation, quality deviation, remaining budget and instantaneous load. This achieves multi-objective optimization of model performance, call cost and response speed. Under the premise of meeting business quality requirements, it effectively reduces the operating cost of cost-sensitive scenarios and improves the response efficiency of high-frequency requests.

[0156] Third, by using both similarity threshold and utility threshold, as well as dynamic adaptation of constraints, the robustness of routing decisions is enhanced. It can quickly lock the optimal single model when the task domain matching is clear, and select the optimal solution through fusion quality evaluation when the matching is ambiguous or the utility of a single model is insufficient. This avoids the quality shortcomings or resource redundancy caused by the rigid decision-making of traditional solutions.

[0157] Fourth, the design of adaptive weights and dynamic constraints enables it to flexibly adapt to the dynamic changes of different business scenarios (such as load fluctuations, budget adjustments, quality requirement upgrades, etc.), and achieve optimal resource allocation without manual intervention. This enhances the scalable application capability of the multi-LLM integration platform in complex business environments and provides a routing solution that combines accuracy, economy and flexibility for diverse business scenarios.

[0158] The above text combined Figure 1 The routing method for model call requests provided in the embodiments of this application has been described in detail. The apparatus and device provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0159] like Figure 2 As shown in the figure, this is a schematic diagram of a model invocation request routing device provided in an embodiment of this application. The device includes:

[0160] The acquisition module 301 is used to acquire the request text of the model call request;

[0161] The calculation module 302 is used to calculate the similarity between the request vector of the request text and the embedding vector of each task domain, and to determine the task domain with the highest similarity as the target task domain.

[0162] The determining module 303 is configured to determine the fusion quality score of each candidate model in the target task domain if the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold; determine the utility value of each candidate model based on the fusion quality score, cost score and latency score of each candidate model; and determine the candidate model with the highest utility value as the target model.

[0163] Processing module 304 is used to route the model call request to the target model.

[0164] Optionally, the determining module 303 is further configured to: determine the single quality score of each candidate model in the target task domain if the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold; determine the utility value of each candidate model based on the single quality score, cost score, and latency score of each candidate model; determine the candidate model with the highest utility value as the target model; and route the model invocation request to the target model.

[0165] Optionally, the determining module 303 is specifically used to determine the candidate model with the highest utility value as the target model when the utility value is greater than or equal to the utility value threshold;

[0166] If the utility value is less than the utility value threshold, the fusion quality score of each candidate model in the target task domain is determined. Based on the fusion quality score, cost score and latency score of each candidate model, the utility value of each candidate model is determined, and the candidate model with the highest utility value is determined as the target model.

[0167] Optionally, the acquisition module 301 is also used to acquire the latency deviation, quality deviation, remaining budget, and instantaneous load of the endpoint evaluation of the model call request;

[0168] The calculation module 302 is also used to determine adaptive weights based on delay deviation, quality deviation, remaining budget, and instantaneous load of endpoint evaluation;

[0169] The determination module 303 is specifically used to determine the utility value of each candidate model based on the fusion quality score, cost score, and latency score of each candidate model, combined with adaptive weights.

[0170] Optionally, the calculation module 302 is also used to calculate the request difficulty corresponding to the request text;

[0171] The determination module 303 is also used to dynamically determine the constraints of the candidate model based on the difficulty of the request;

[0172] The determination module 303 is specifically used to determine the candidate model with the highest utility value and that satisfies the constraints as the target model.

[0173] The routing device for model invocation requests according to the embodiments of this application can correspond to the execution of the methods described in the embodiments of this application, and the other operations and / or functions of each module / unit of the routing device for model invocation requests are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0174] This application also provides a computing device. For example... Figure 3As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.

[0175] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0176] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0177] The communication interface 703 is used for communication with external devices.

[0178] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0179] The memory 704 stores executable code, which the processor 702 executes to perform the routing method for the aforementioned model call request.

[0180] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the routing device for model invocation requests described in the embodiments are implemented in software, the execution... Figure 2The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 704. Processor 702 executes the program code corresponding to each unit stored in memory 704 and executes the routing method for the aforementioned model call request.

[0181] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the routing method for the aforementioned model invocation request.

[0182] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0183] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0184] When the computer program product is executed by a computer, the computer executes any of the routing methods for the aforementioned model invocation request. The computer program product can be a software installation package; when any of the routing methods for the aforementioned model invocation request needs to be used, the computer program product can be downloaded and executed on the computer.

[0185] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0186] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for routing model invocation requests, characterized in that, The method includes: Get the request text of the model call request; Calculate the similarity between the request vector of the request text and the embedding vector of each task domain, and determine the task domain with the highest similarity as the target task domain; If the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold, the fusion quality score of each candidate model in the target task domain is determined. The fusion quality score includes the model capability and request difficulty matching degree based on item response theory, the ELO dynamic score reflecting the recent comprehensive performance of the model, and the performance score predicted in real time based on request features. The expression for the fusion quality score is: in, Representation Model Regarding the request The fusion quality score, This represents the quality score based on the project response theory. Representation Model ELO dynamic score, This represents the quality score predicted in real time based on a machine learning model. This represents the first weighting coefficient. This represents the second weighting coefficient. Indicates the third weighting coefficient; Obtain the latency deviation, quality deviation, remaining budget, and instantaneous endpoint evaluation load of the model call request; Adaptive weights are determined based on delay skew, quality skew, remaining budget, and instantaneous load at endpoints. Based on the fusion quality score, cost score, and latency score of each candidate model, and combined with adaptive weights, the utility value of each candidate model is determined. The process of determining adaptive weights based on delay deviation, quality deviation, remaining budget, and endpoint evaluation instantaneous load includes: Based on delay deviation, quality deviation, and remaining budget, basic weights are obtained; these basic weights are then normalized; finally, the normalized basic weights are adjusted based on the instantaneous load of the endpoint evaluation. High load threshold: in, This represents the corrected delay weight. The base weights represent the normalized delay. This represents the delay-weighted load correction factor. This indicates that the endpoint is evaluating the instantaneous load. Indicates a high load threshold; If the endpoint evaluates instantaneous load Low load threshold: in, This represents the corrected quality weight. The base weights representing the normalized quality. This represents the quality weighting load correction factor. Indicates a low load threshold; The corrected weights are then normalized to form an adaptive weight vector. If the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold, the single quality score of each candidate model in the target task domain is determined, and the utility value of each candidate model is determined based on the single quality score, cost score and latency score of each candidate model. The candidate model with the highest utility value is selected as the target model; The model call request is routed to the target model.

2. The method according to claim 1, characterized in that, The step of determining the candidate model with the highest utility value as the target model includes: If the utility value is greater than or equal to the utility value threshold, the candidate model with the highest utility value is determined as the target model; If the utility value is less than the utility value threshold, the fusion quality score of each candidate model in the target task domain is determined. Based on the fusion quality score, cost score and latency score of each candidate model, the utility value of each candidate model is determined, and the candidate model with the highest utility value is determined as the target model.

3. The method according to claim 1, characterized in that, The method further includes: Calculate the request difficulty corresponding to the request text; Based on the difficulty of the request, the constraints of the candidate model are dynamically determined; The step of determining the candidate model with the highest utility value as the target model includes: The candidate model with the highest utility value and that satisfies the constraints is determined as the target model.

4. A routing device for model invocation requests, characterized in that, The device includes: The acquisition module is used to obtain the request text of the model call request; The calculation module is used to calculate the similarity between the request vector of the request text and the embedding vector of each task domain, and to determine the task domain with the highest similarity as the target task domain. The determination module is used to determine the fusion quality score of each candidate model in the target task domain if the similarity between the request vector and the embedding vector of the target task domain is less than a similarity threshold. The fusion quality score includes the model capability and request difficulty matching degree based on item response theory, the ELO dynamic score reflecting the recent comprehensive performance of the model, and the performance score predicted in real time based on request features. The expression for the fusion quality score is: in, Representation Model Regarding the request The fusion quality score, This represents the quality score based on the project response theory. Representation Model ELO dynamic score, This represents the quality score predicted in real time based on a machine learning model. This represents the first weighting coefficient. This represents the second weighting coefficient. Indicates the third weighting coefficient; Obtain the latency deviation, quality deviation, remaining budget, and instantaneous endpoint evaluation load of the model call request; Adaptive weights are determined based on delay skew, quality skew, remaining budget, and instantaneous load at endpoints. Based on the fusion quality score, cost score, and latency score of each candidate model, and combined with adaptive weights, the utility value of each candidate model is determined. The process of determining adaptive weights based on delay deviation, quality deviation, remaining budget, and endpoint evaluation instantaneous load includes: Based on delay deviation, quality deviation, and remaining budget, basic weights are obtained; these basic weights are then normalized; finally, the normalized basic weights are adjusted based on the instantaneous load of the endpoint evaluation. High load threshold: in, This represents the corrected delay weight. The base weights represent the normalized delay. This represents the delay-weighted load correction factor. This indicates that the endpoint is evaluating the instantaneous load. Indicates a high load threshold; If the endpoint evaluates instantaneous load Low load threshold: in, This represents the corrected quality weight. The base weights representing the normalized quality. This represents the quality weighting load correction factor. Indicates a low load threshold; The corrected weights are then normalized to form an adaptive weight vector. If the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold, the single quality score of each candidate model in the target task domain is determined, and the utility value of each candidate model is determined based on the single quality score, cost score and latency score of each candidate model. The candidate model with the highest utility value is selected as the target model; The processing module is used to route the model call request to the target model.

5. The apparatus according to claim 4, characterized in that, The determining module is further configured to determine the single quality score of each candidate model in the target task domain if the similarity between the request vector and the embedding vector of the target task domain is greater than or equal to the similarity threshold, and to determine the utility value of each candidate model based on the single quality score, cost score and latency score of each candidate model. The candidate model with the highest utility value is selected as the target model; The model call request is routed to the target model.

6. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 3.

8. A computer program product, characterized in that, The computer program product includes one or more computer instructions, which, when executed by a computer, perform the method as described in any one of claims 1 to 3.

Images