A model routing method, apparatus and electronic device

Abstract

The application discloses a model routing method and device and electronic equipment, and relates to the technical field of artificial intelligence. The model routing method comprises the following steps: obtaining a target task to be processed at present; determining performance uncertainty information of each candidate task model for the target task by a routing model based on semantic features of the target task and model features of each candidate task model, wherein the performance uncertainty information is used for quantifying fluctuation of inference results of the candidate task model; and determining model routing information based on the performance uncertainty information, wherein the model routing information is used for determining a task model to process the target task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model routing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of large-scale model technology, in practical applications, it is often necessary to route user tasks to the most suitable task model to achieve a balance between performance and cost. However, in known technologies, the routing strategies learned by the routing models are unstable and have poor generalization ability, which can easily lead to suboptimal or even incorrect routing decisions in actual deployments. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A model routing method, comprising:

[0005] Obtain the current pending target task;

[0006] Based on the semantic features of the target task and the model features of each candidate task model, the performance uncertainty information of each candidate task model for the target task is determined by the routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models.

[0007] Based on the performance uncertainty information, model routing information is determined, and the model routing information is used to determine the task model to process the target task.

[0008] Optionally, determining model routing information based on the performance uncertainty information includes:

[0009] The routing model generates routing decision results and corresponding confidence levels based on the performance uncertainty information corresponding to each alternative task model.

[0010] The model routing information includes the routing decision result and the confidence level.

[0011] Optionally, the performance uncertainty information characterizes the performance index distribution statistics when the alternative task model processes the target task.

[0012] Optionally, the performance metric distribution statistics include at least one of the following:

[0013] The mean and variance of the model accuracy;

[0014] The mean and variance of the lexical consumption of the model.

[0015] Optionally, the routing model is trained by learning the performance uncertainty information of each alternative task model for the task sample;

[0016] The performance uncertainty information of each candidate task model for the task sample is obtained by performing multiple independent inference samplings on the task sample using the candidate task model, and by performing posterior distribution statistics of the model performance based on the performance indicators obtained from each independent inference sampling.

[0017] Optionally, the routing model is obtained through the following training process:

[0018] Obtain a training sample set; wherein each training sample includes: semantic features of the task samples included in the corresponding task and model combination and model features of the candidate task models included, as well as corresponding labels; the labels represent the ground truth value of the performance uncertainty of the candidate task model for the task samples;

[0019] The semantic features of the task samples in the training samples and the model features of the candidate task models are input into the model to be trained to obtain the performance uncertainty prediction value of the candidate task model for the task samples output by the model to be trained.

[0020] Based on the predicted performance uncertainty value and the corresponding label, the model loss of the model to be trained is determined;

[0021] Based on the model loss, the model parameters of the model to be trained are adjusted to obtain the routing model.

[0022] Optionally, obtain the training sample set, including:

[0023] Based on each task sample in the task sample set and each candidate task model, multiple task and model combinations are determined.

[0024] For each task and model combination, multiple independent inference samplings are performed to obtain sampled data, which includes model inference performance metrics from multiple samplings.

[0025] Based on the sampled data corresponding to each task and model combination, the posterior distribution of model performance is estimated to obtain the posterior distribution statistics of the performance indicators of the candidate task models in the task and model combination for the corresponding task samples.

[0026] For each task and model combination, training samples are generated, including semantic features of the task samples, model features of the candidate task models, and corresponding labels, wherein the labels include the posterior distribution statistics of the performance metrics.

[0027] Optionally, for each task and model combination, perform multiple independent inference samplings, including:

[0028] For each task and model combination, perform an initial independent inference sampling.

[0029] The method further includes:

[0030] In response to the corresponding performance metric posterior distribution statistic indicating that the performance uncertainty of the candidate task model in the task and model combination for the corresponding task sample is higher than a first threshold, the number of independent inference samplings of the task and model combination is added to a second number, and the posterior distribution of model performance is estimated based on the sampling data of the second number.

[0031] Wherein, the second number is greater than the first number;

[0032] Under the second sampling data, the posterior probability of the candidate task model in the task and model combination for the target performance feature of the corresponding task sample is higher than the second threshold. The target performance feature includes: the performance of the candidate task model for the corresponding task sample is better or worse than that of other candidate task models, and the degree of performance difference meets the significance condition.

[0033] A model routing device, comprising:

[0034] The acquisition module is used to obtain the target task currently pending processing;

[0035] The determination module is used to determine the performance uncertainty information of each candidate task model for the target task based on the semantic features of the target task and the model features of each candidate task model through a routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models.

[0036] The routing decision module is used to determine model routing information based on the performance uncertainty information, and the model routing information is used to determine the task model to process the target task.

[0037] An electronic device, comprising:

[0038] Memory, used to store at least one set of computer instructions;

[0039] A processor is configured to perform the following processing by executing a set of instructions stored in the memory:

[0040] Obtain the current pending target task;

[0041] Based on the semantic features of the target task and the model features of each candidate task model, the performance uncertainty information of each candidate task model for the target task is determined by the routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models.

[0042] Based on the performance uncertainty information, model routing information is determined, and the model routing information is used to determine the task model to process the target task.

[0043] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement any of the model routing methods provided above. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This is a flowchart of the model routing method provided in this application;

[0046] Figure 2 This is a diagram of the routing model training process provided in this application;

[0047] Figure 3 This is a diagram illustrating the generation process of the training sample set provided in this application;

[0048] Figure 4 This is a diagram illustrating the overall implementation process from the preparation of the training sample set provided in this application to model training;

[0049] Figure 5 This is a structural diagram of the model routing device provided in this application;

[0050] Figure 6 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a model routing method, apparatus, and electronic device, applicable to, but not limited to, model scheduling in high-concurrency and multi-task mixed scenarios, so as to stably and accurately route task requests to the most cost-effective model in the model resource pool.

[0053] See Figure 1The flowchart shown in this application illustrates that the model routing method provided in this embodiment may include the following steps 101 to 103, which are described in detail below.

[0054] Step 101: Obtain the target task to be processed.

[0055] Optionally, the user can input the target task through natural language interaction, or obtain the target task based on parsing or intent recognition of the user's input information.

[0056] This application does not limit the specific type of the target task, which can be divided based on any classification dimension. For example, from the perspective of the determinacy of the task result, the target task can be an open-ended task, long text generation, mathematical reasoning, or other uncertain tasks, or it can be a deterministic task such as multiple-choice questions or short text classification; from the perspective of user application, the target task can also be a question-answering task, information retrieval task, content generation and creation task, or logical reasoning and calculation task, etc.

[0057] Step 102: Based on the semantic features of the target task and the model features of each candidate task model, determine the performance uncertainty information of each candidate task model for the target task through the routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models.

[0058] After obtaining the target task to be processed, the semantic features of the target task can be obtained by vectorizing the task text.

[0059] Each alternative task model is a set of multiple optional models provided in the model resource pool for matching and reasoning about tasks. These models can be heterogeneous models, including but not limited to multiple large language models (LLMs) with different parameter scales (such as tens of billions or hundreds of billions), different capabilities and specialties (such as general logical reasoning, code writing and debugging, multimodal text and image understanding) and / or different calling costs.

[0060] The model features of the candidate task models may include, but are not limited to, features in various dimensions such as model identifier (e.g., model name), capabilities and expertise, parameter size and / or invocation cost.

[0061] The performance uncertainty information represents the distribution statistics of performance indicators when the alternative task model processes the target task.

[0062] The performance metric distribution statistics may include, but are not limited to, the mean and variance of the performance metric distribution. For each candidate task model, the larger the mean of its corresponding performance metric distribution, the more significant the model's performance on the target task for that performance metric; the larger the variance, the greater the performance volatility and the worse the stability of the model under multiple routing and inference scenarios for the target task.

[0063] In this step, the semantic features of the target task and the model features of each candidate task model can be input into the pre-trained routing model. Based on the input information, the routing model determines (predicts) the performance uncertainty information such as the mean and variance of the performance indicators of each candidate task model for the target task.

[0064] Step 103: Determine model routing information based on the performance uncertainty information. The model routing information is used to determine the task model to process the target task.

[0065] After obtaining the performance uncertainty information (such as the mean and variance of the performance index distribution) of each candidate task model for the target task, the routing model can further make routing decisions based on the performance uncertainty information, select the target candidate task model that meets the preset conditions from each candidate task model, and generate the corresponding model routing information.

[0066] The preset conditions may include: a relatively high average performance level for the target task and relatively low performance volatility. The model routing information is used to indicate which of the target candidate task models should be selected as the task model for task processing to handle the target task.

[0067] In summary, to address the technical problems of unstable routing strategies, poor generalization ability, and susceptibility to suboptimal or erroneous routing decisions in known technologies, the solution of this application introduces performance uncertainty information of each alternative task model for the target task into the routing decision. This provides a key stability dimension for the routing model, enabling it to proactively avoid alternative task models with unstable and highly volatile performance. Consequently, the model makes a better trade-off in routing decisions, significantly reducing the risk of suboptimal or erroneous routing caused by model performance fluctuations, and improving the overall reliability and robustness of routing decisions.

[0068] In one optional embodiment, the performance metric distribution statistics may include, but are not limited to, at least one of the following:

[0069] 1) The mean and variance of the model accuracy;

[0070] 2) The mean and variance of token consumption in the model.

[0071] For each candidate task model, the higher the average accuracy, the higher the average accuracy level of the model on the target task; the higher the average word consumption, the heavier the average inference cost burden of the model on the target task; the higher the variance, the greater the performance fluctuation and the worse the stability of the model under multiple routing and inference scenarios for the target task.

[0072] After obtaining performance uncertainty information such as the mean and variance of model accuracy and the mean and variance of model lexical consumption for each candidate task model for the target task, the routing model can further use this (performance uncertainty information) as a basis for reasoning to generate model routing information.

[0073] Optionally, model routing information includes routing decision results and confidence levels.

[0074] The routing model can score or estimate the performance level of each candidate task model based on performance uncertainty information such as the mean and variance of model accuracy and the mean and variance of model token consumption. The higher the performance score or performance level, the better the overall performance of the candidate task model for the target task (e.g., relatively high accuracy and relatively low token consumption). Then, the model that meets the preset conditions (e.g., the highest score or the highest level) is selected as the target candidate task model for task processing. Based on this, the routing model can further generate a routing decision result that indicates the use of the target candidate task model for task processing.

[0075] The confidence level can be determined based on the variance of the target candidate task model indicated in the routing decision results; the larger the variance, the lower the confidence level.

[0076] This embodiment comprehensively considers multi-dimensional performance uncertainty information such as the mean and variance corresponding to model accuracy and word consumption, respectively, to achieve accurate evaluation and trade-off between accuracy and cost of candidate task models. The mean is used to reflect the average performance level and inference cost burden of the model, and the variance is used to quantify the volatility / stability of the model under multiple routing and inference. Then, the optimal model that balances high accuracy, low cost and high stability is selected by combining scoring or ranking mechanisms. The confidence level reflecting the reliability of the decision is generated based on the variance. Thus, while ensuring the processing effect of the target task, resource consumption is effectively optimized, and the accuracy, robustness and intelligence of the model's routing decision are significantly improved.

[0077] In an optional embodiment, the method of this application may further include confidence-based fallback processing.

[0078] Specifically, when the confidence level of the routing decision result is lower than a preset threshold (indicating that the routing confidence level is too low or the performance uncertainty is too high), a rollback process based on the rollback strategy can be triggered.

[0079] The rollback process based on the rollback strategy may include, but is not limited to: routing the target task to the model with the smallest prediction variance of the corresponding performance index among the candidate task models (representing the model with the smallest fluctuation in the performance of the target task), or routing it to a pre-set fallback model (such as the model with the strongest overall performance among the candidate task models) for processing, so as to ensure the reliability of task processing.

[0080] In addition, low-confidence tasks that trigger the fallback strategy can be recorded and stored for subsequent manual analysis and evaluation, or used to expand the training sample set of the model.

[0081] This embodiment effectively addresses the reliability issue of routing decisions in low-confidence or high-uncertainty scenarios by introducing a confidence-based fallback mechanism. When the confidence level of a routing decision result is detected to be lower than a preset threshold, the system can automatically trigger a fallback strategy, intelligently switching the task to the model with the lowest performance volatility (lowest prediction variance) or the fallback model with the strongest overall performance. This avoids task processing failures caused by excessively low confidence in the routing decision result, significantly improving the system's robustness and service stability.

[0082] Meanwhile, this embodiment also constructs a data closed loop from business operation to model optimization by automatically recording and storing low-confidence tasks, providing high-quality long-tail data support for subsequent manual analysis and expansion of the training sample set, which helps to continuously iterate and improve the decision accuracy of the routing model.

[0083] In one alternative embodiment, the routing model is trained by learning the performance uncertainty information of each alternative task model for the task samples.

[0084] The performance uncertainty information of each candidate task model for the task sample is obtained by performing multiple independent inference samplings on the task sample using the candidate task model, and then performing posterior distribution statistics of the model performance based on the performance indicators obtained from each independent inference sampling.

[0085] See Figure 2 The diagram shown illustrates the model training process. The routing model can be obtained through the following training process:

[0086] Step 201: Obtain the training sample set.

[0087] This embodiment pre-combines task samples and candidate task models within a task sample set to obtain multiple task-model combinations. Each task-model combination includes one task sample and one candidate task model.

[0088] Building upon this, a training sample set for model training is further constructed based on a combination of tasks and models.

[0089] Each training sample includes: the semantic features of the task samples included in the corresponding task and model combination, the model features of the candidate task models included, and the corresponding labels.

[0090] The semantic features of the task samples may include the task sample vectors obtained by vectorizing the task samples. The model features of the candidate task models may include, but are not limited to, features in various dimensions such as model identifier (e.g., model name), capabilities and expertise, parameter size and / or invocation cost.

[0091] The label represents the true value of the performance uncertainty of the candidate task model for the task sample. The true value of the performance uncertainty may include the posterior distribution statistics of the actual performance indicators when the candidate task model processes the task sample, such as the mean and variance of the model accuracy when the candidate task model processes the task sample, and / or the mean and variance of the model word consumption, etc.

[0092] Step 202: Input the semantic features of the task samples in the training samples and the model features of the candidate task models into the model to be trained, and obtain the performance uncertainty prediction value of the candidate task models for the task samples output by the model to be trained.

[0093] The model to be trained is the routing model to be trained.

[0094] The performance uncertainty prediction values ​​of the alternative task model for the task sample may include, but are not limited to, the predicted values ​​of the mean and variance of the model accuracy when the alternative task model processes the task sample, and / or the predicted values ​​of the mean and variance of the model word consumption.

[0095] For example, the model to be trained may use a four-output head structure to predict both the mean and variance of the model accuracy when the candidate task model processes the task samples, and the mean and variance of the model word consumption when the candidate task model processes the task samples.

[0096] Step 203: Based on the predicted performance uncertainty value and the corresponding label, determine the model loss of the model to be trained.

[0097] Based on the aforementioned steps, this step can calculate the model loss of the model to be trained using a pre-set loss function, based on the predicted performance uncertainty value and the corresponding true value of performance uncertainty represented by the label.

[0098] The model loss can be NLL (Negative Log-Likelihood) loss to measure the difference between the performance uncertainty prediction and the true label.

[0099] The following provides an example loss function for NLL loss:

[0100] .

[0101] in, Indicates the model loss. This represents the model accuracy loss component. λ represents the loss component of the model's lexical consumption, and λ represents the balance coefficient. This represents the true mean of the model accuracy when the i-th candidate task model processes the j-th task sample. This represents the mean predicted accuracy of the model when the i-th candidate task model processes the j-th task sample. This represents the predicted model accuracy variance when the i-th alternative task model processes the j-th task sample; Let represent the true value of the mean word consumption of the model when the i-th candidate task model processes the j-th task sample. This represents the average predicted word consumption of the model when the i-th candidate task model processes the j-th task sample. This represents the predicted variance of the word consumption of the model when the i-th alternative task model processes the j-th task sample.

[0102] Step 204: Based on the model loss, adjust the model parameters of the model to be trained to obtain the routing model.

[0103] After determining the model loss, the model parameters of the model to be trained can be adjusted based on the current loss value and with minimizing the model loss as the optimization objective. During training, by adjusting the model parameters with the goal of minimizing the model loss, the model to be trained can effectively learn the inherent uncertainties of each candidate task model for the task sample data.

[0104] Subsequently, the above parameter adjustment process can be iterated until the preset training termination condition is met, at which point training stops, and the training-completed model is used as the routing model.

[0105] Optionally, the training termination condition may include, but is not limited to, at least one of the following: the number of iterations reaches the upper limit, the training time reaches the upper limit, or the model loss converges to below a preset threshold.

[0106] This embodiment explicitly models the performance fluctuations caused by the inherent randomness during the inference process of candidate task models, such as large language models, as performance uncertainty, and incorporates this uncertainty as supervisory information into the training process of the routing model. This enables the model to not only learn the average performance level of each candidate task model, but also accurately learn its variability characteristics during task processing. This dual learning mechanism of performance level and variability characteristics effectively overcomes the shortcomings of traditional methods that treat statistical noise as interference. It allows the routing model to simultaneously grasp the performance expectations and prediction reliability (variability) of each candidate task model, thereby significantly improving the stability, reliability, and generalization ability of the routing model's routing decisions.

[0107] In an alternative embodiment, see Figure 3 The training sample set for model training can be obtained through the following process.

[0108] Step 301: Based on each task sample in the task sample set and each candidate task model, determine multiple task and model combinations.

[0109] In this way, multiple task-model combinations can be formed by combining each task sample in the task sample set with each candidate task model.

[0110] For example, for each task sample in the task sample set, all candidate task models are traversed, and the task sample is combined with each traversed candidate task model to generate multiple task-model combinations. Each task-model combination includes one task sample and one candidate task model.

[0111] Step 302: For each task and model combination, perform multiple independent inference samplings to obtain sampled data, which includes model inference performance metrics from multiple samplings.

[0112] Model inference performance metrics may include, but are not limited to, model accuracy and / or lexical consumption.

[0113] Here, we assume that the task and model combination is represented as (model, query), where model represents the candidate task model and query represents the task sample. Multiple independent inference samples can be performed on each (model, query), and the model inference performance metrics such as accuracy (acc, e.g., 0 / 1, where 0 indicates inaccuracy and 1 indicates accuracy) and lexical consumption are recorded for each sample, thus obtaining the original sampled dataset.

[0114] .

[0115] In the above expression, This represents the i-th alternative task model. Let j represent the j-th task sample. Indicates the number of samplings, k represents the number of times the sample is taken. right The current inference sampling round during multiple inference sampling processes.

[0116] Step 303: Based on the sampled data corresponding to each task and model combination, perform posterior distribution estimation of model performance to obtain the posterior distribution statistics of the performance indicators of the candidate task models in the task and model combination for the corresponding task samples.

[0117] The posterior distribution statistics of the performance metrics may include, but are not limited to, at least one of the mean and variance of model accuracy and the mean and variance of model word consumption.

[0118] Specifically, this step can utilize multiple independent sampling data corresponding to each task and model combination to estimate the posterior distribution of model accuracy, word consumption, etc., and then calculate the corresponding statistical quantities such as mean and variance to quantify the performance uncertainty of the candidate task model when processing the task sample.

[0119] For example, suppose that in the alternative task model for the task sample In a series of independent inference samples, if p observations are correct and an uninformative prior Beta(1,1) is used, the posterior mean of the model accuracy can be calculated. and variance They are as follows:

[0120] ;

[0121] .

[0122] Meanwhile, the posterior distribution of the model's lexical consumption can be estimated using the following formula to obtain the mean and variance of the model's lexical consumption:

[0123] ;

[0124] .

[0125] Step 304: For each task and model combination, generate training samples including semantic features of the task samples, model features of the candidate task models, and corresponding labels, wherein the labels include the posterior distribution statistics of the performance index.

[0126] This step is used to transform the data obtained in the previous steps into standardized supervised learning samples.

[0127] For each task and model combination, the task samples are first vectorized to extract semantic features, and the model features of the candidate task models (such as model identifier, parameter size, capability expertise, etc.) are obtained. Then, the two are used together as input features of the training samples. Subsequently, the posterior distribution statistics of the performance indicators (such as the mean and variance of accuracy or word consumption) estimated based on multiple sampling data in the previous steps can be used as the supervision label of the training sample. This results in a training sample set including the semantic features of the task samples, the model features of the candidate task models, and the corresponding labels, forming a training sample set for model training.

[0128] This embodiment constructs a training sample set containing semantic features of task samples and model features of candidate task models, and uses posterior distribution statistics (such as mean and variance) of performance metrics obtained from multiple independent sampling estimates as supervision labels. This enables joint modeling of the expected performance and volatility of candidate task models on a specific task. This approach not only allows the training sample data to quantify the performance uncertainty of the model's inference but also provides supervision signals with rich statistical information for subsequent routing models, thereby significantly improving the accuracy and robustness of routing decisions.

[0129] In one alternative embodiment, performing multiple independent inference samplings for each task and model combination may include: performing an initial number of independent inference samplings for each task and model combination.

[0130] The first number is an integer greater than 1, such as 3.

[0131] The method provided in this application may further include: in response to the corresponding performance index posterior distribution statistic indicating that the performance uncertainty of the candidate task model in the task and model combination for the corresponding task sample is higher than a first threshold, adding the number of independent inference samplings of the task and model combination to a second number, and estimating the posterior distribution of model performance based on the sampling data of the second number.

[0132] The second number is greater than the first number.

[0133] In the second sampling data, the posterior probability of the candidate task model in the task-model combination for the target performance feature of the corresponding task sample is higher than the second threshold. The target performance feature includes: the performance of the candidate task model for the corresponding task sample is better or worse than that of other candidate task models, and the degree of performance difference meets the significance condition.

[0134] The performance uncertainty of the candidate task model for the corresponding task sample can be characterized by the variance of the accuracy and / or the variance of the word consumption of the candidate task model for the corresponding task sample. If the variance is higher than the first threshold, it indicates that the candidate task model has high performance uncertainty for the corresponding task sample. At this time, the task sample is a high uncertainty task sample for the candidate task model.

[0135] For high-uncertainty task samples, the number of samplings can be increased from the first number to the second number, and the posterior distribution estimation of model performance can be performed again based on the total sampled data after the increase.

[0136] The second number is the number of samples dynamically determined based on an adaptive sampling strategy. In practice, optionally, a sequential hypothesis testing method can be used to calculate the posterior probability that, given the current number of samples, the accuracy of the candidate task model is significantly better or worse than that of other candidate task models on this high-uncertainty task sample. If this posterior probability is higher than the second threshold (e.g., greater than 0.95), then additional sampling stops; otherwise, it continues.

[0137] The second count is the total number of samples taken when additional sampling stops. This approach maximizes information gain within the constraints of the evaluation budget.

[0138] The entire process from preparing the training sample set to training the model is as follows: Figure 4 As shown in the diagram, the evaluation set refers to the task sample set. The evaluation phase is the preparation phase for the training sample set. This phase is used to combine each task sample in the task sample set with each candidate task model to form multiple task-model combinations. For each task-model combination, multiple independent inference samplings are performed. Here, the performance fluctuations generated by multiple samplings are no longer regarded as noise that needs to be eliminated by averaging. Instead, they are modeled as performance uncertainty information of the candidate task model on the sampled task samples. This performance uncertainty information is used as a supervision signal to construct the labels corresponding to the sample input features, so as to further construct training samples containing the semantic features of the task samples, the model features of the candidate task models, and the corresponding labels, thereby obtaining the training sample set.

[0139] The training phase is used to iteratively train the routing model to be trained based on the training sample set. This involves propagating the performance uncertainty information of the candidate task models on the sampled task samples as a supervision signal into the training process of the routing model. This allows the routing model to simultaneously learn the performance level and prediction reliability of the candidate task models, thereby improving the stability and reliability of routing decisions. More detailed implementation processes can be found in the descriptions of the embodiments above, and will not be repeated here.

[0140] This embodiment introduces an adaptive sampling and decision enhancement mechanism based on performance uncertainty. By improving the fixed number of inference samplings to a dynamic stopping strategy based on sequential probability ratio testing, it achieves fine-grained control over computing resources. For task samples with high uncertainty, this embodiment can automatically increase the number of samplings until the posterior probability of model performance being good / bad meets the significance condition before stopping sampling. This not only ensures the statistical confidence of routing decisions in complex scenarios, but also effectively avoids oversampling of low-uncertainty samples while ensuring evaluation accuracy, thereby maximizing information gain under evaluation budget constraints.

[0141] In summary, the solutions of this application have at least the following technical advantages compared to known technologies:

[0142] 1) Significantly reduced the misleading effect of evaluation noise being directly incorporated into routing decision training as deterministic labels;

[0143] 2) Significantly improved the training stability and generalization ability of the routing model in high-fluctuation task scenarios;

[0144] 3) By enabling the routing model to output confidence / credibility information corresponding to the routing decision results, the interpretability and controllability of the system are improved.

[0145] 4) Automatically adapts to the variance characteristics of different task types. Without the need for manual setting of sampling times or filtering thresholds, the system can dynamically adjust the sampling times according to the actual performance of the task and the model. While ensuring the accuracy of the evaluation, it also effectively avoids oversampling of low-uncertainty samples and can maximize information gain under the constraints of the evaluation budget.

[0146] Corresponding to the above method, this application embodiment also provides a model routing device, the composition of which is as follows: Figure 5 As shown, it includes:

[0147] Module 501 is used to obtain the target task currently pending processing;

[0148] The determining module 502 is used to determine the performance uncertainty information of each candidate task model for the target task based on the semantic features of the target task and the model features of each candidate task model through a routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models.

[0149] The routing decision module 503 is used to determine model routing information based on the performance uncertainty information, and the model routing information is used to determine the task model to process the target task.

[0150] In an optional implementation, the routing decision module 503 is specifically used for:

[0151] The routing model generates routing decision results and corresponding confidence levels based on the performance uncertainty information corresponding to each alternative task model.

[0152] The model routing information includes the routing decision result and the confidence level.

[0153] In one alternative implementation, the performance uncertainty information characterizes the performance index distribution statistics when the alternative task model processes the target task.

[0154] In one optional implementation, the performance metric distribution statistics include at least one of the following:

[0155] The mean and variance of the model accuracy;

[0156] The mean and variance of the lexical consumption of the model.

[0157] In one alternative implementation, the routing model is trained by learning the performance uncertainty information of each alternative task model for the task sample.

[0158] The performance uncertainty information of each candidate task model for the task sample is obtained by performing multiple independent inference samplings on the task sample using the candidate task model, and by performing posterior distribution statistics of the model performance based on the performance indicators obtained from each independent inference sampling.

[0159] In an optional embodiment, the apparatus further includes a model training module for obtaining the routing model through the following training process:

[0160] Obtain a training sample set; wherein each training sample includes: semantic features of the task samples included in the corresponding task and model combination and model features of the candidate task models included, as well as corresponding labels; the labels represent the ground truth value of the performance uncertainty of the candidate task model for the task samples;

[0161] The semantic features of the task samples in the training samples and the model features of the candidate task models are input into the model to be trained to obtain the performance uncertainty prediction value of the candidate task model for the task samples output by the model to be trained.

[0162] Based on the predicted performance uncertainty value and the corresponding label, the model loss of the model to be trained is determined;

[0163] Based on the model loss, the model parameters of the model to be trained are adjusted to obtain the routing model.

[0164] In one optional implementation, the model training module, when obtaining the training sample set, is specifically used for:

[0165] Based on each task sample in the task sample set and each candidate task model, multiple task and model combinations are determined.

[0166] For each task and model combination, multiple independent inference samplings are performed to obtain sampled data, which includes model inference performance metrics from multiple samplings.

[0167] Based on the sampled data corresponding to each task and model combination, the posterior distribution of model performance is estimated to obtain the posterior distribution statistics of the performance indicators of the candidate task models in the task and model combination for the corresponding task samples.

[0168] For each task and model combination, training samples are generated, including semantic features of the task samples, model features of the candidate task models, and corresponding labels, wherein the labels include the posterior distribution statistics of the performance metrics.

[0169] In one alternative implementation, the model training module, when performing multiple independent inference samplings for each task and model combination, is specifically used to: perform the first number of independent inference samplings for each task and model combination.

[0170] The model training module is further configured to: in response to the corresponding performance metric posterior distribution statistic indicating that the performance uncertainty of the candidate task model in the task and model combination for the corresponding task sample is higher than a first threshold, append the number of independent inference samplings of the task and model combination to a second number, and estimate the posterior distribution of the model performance based on the sampling data of the second number.

[0171] Wherein, the second number is greater than the first number;

[0172] Under the second sampling data, the posterior probability of the candidate task model in the task and model combination for the target performance feature of the corresponding task sample is higher than the second threshold. The target performance feature includes: the performance of the candidate task model for the corresponding task sample is better or worse than that of other candidate task models, and the degree of performance difference meets the significance condition.

[0173] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 6 As shown, it may include:

[0174] Memory 10 is used to store the computer instruction set;

[0175] Computer instruction sets can be implemented in the form of computer programs.

[0176] Processor 20 is configured to implement the model routing method provided in any of the above method embodiments by executing a set of computer instructions in the memory.

[0177] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.

[0178] Optionally, the electronic device may also include a display device for displaying information.

[0179] Optionally, electronic devices may also include storage resources such as memory and cache.

[0180] Optionally, the electronic device may also include an image acquisition device.

[0181] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0182] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0183] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the model routing method provided in any of the above method embodiments.

[0184] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0185] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0186] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0187] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0188] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model routing method, comprising: Obtain the current pending target task; Based on the semantic features of the target task and the model features of each candidate task model, the performance uncertainty information of each candidate task model for the target task is determined by the routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models. Based on the performance uncertainty information, model routing information is determined, and the model routing information is used to determine the task model to process the target task.

2. The model routing method according to claim 1, wherein determining model routing information based on the performance uncertainty information includes: The routing model generates routing decision results and corresponding confidence levels based on the performance uncertainty information corresponding to each alternative task model. The model routing information includes the routing decision result and the confidence level.

3. The model routing method according to claim 1, wherein the performance uncertainty information characterizes the performance index distribution statistics when the alternative task model processes the target task.

4. The model routing method according to claim 3, wherein the performance metric distribution statistics include at least one of the following: The mean and variance of the model accuracy; The mean and variance of the lexical consumption of the model.

5. The model routing method according to claim 1, wherein the routing model is trained by learning the performance uncertainty information of each candidate task model for the task sample; The performance uncertainty information of each candidate task model for the task sample is obtained by performing multiple independent inference samplings on the task sample using the candidate task model, and by performing posterior distribution statistics of the model performance based on the performance indicators obtained from each independent inference sampling.

6. The model routing method according to claim 5, wherein the routing model is obtained through the following training process: Obtain the training sample set; among which, Each training sample includes: semantic features of the task samples included in the corresponding task and model combination, model features of the included alternative task models, and corresponding labels; the labels represent the true value of the performance uncertainty of the alternative task models for the task samples. The semantic features of the task samples in the training samples and the model features of the candidate task models are input into the model to be trained to obtain the performance uncertainty prediction value of the candidate task model for the task samples output by the model to be trained. Based on the predicted performance uncertainty value and the corresponding label, the model loss of the model to be trained is determined; Based on the model loss, the model parameters of the model to be trained are adjusted to obtain the routing model.

7. The model routing method according to claim 6, obtaining a training sample set, includes: Based on each task sample in the task sample set and each candidate task model, multiple task and model combinations are determined. For each task and model combination, multiple independent inference samplings are performed to obtain sampled data, which includes model inference performance metrics from multiple samplings. Based on the sampled data corresponding to each task and model combination, the posterior distribution of model performance is estimated to obtain the posterior distribution statistics of the performance indicators of the candidate task models in the task and model combination for the corresponding task samples. For each task and model combination, training samples are generated, including semantic features of the task samples, model features of the candidate task models, and corresponding labels, wherein the labels include the posterior distribution statistics of the performance metrics.

8. The model routing method according to claim 7, wherein for each task and model combination, multiple independent inference samplings are performed, including: For each task and model combination, perform an initial independent inference sampling. The method further includes: In response to the corresponding performance metric posterior distribution statistic indicating that the performance uncertainty of the candidate task model in the task and model combination for the corresponding task sample is higher than a first threshold, the number of independent inference samplings of the task and model combination is added to a second number, and the posterior distribution of model performance is estimated based on the sampling data of the second number. Wherein, the second number is greater than the first number; Under the second sampling data, the posterior probability of the candidate task model in the task and model combination for the target performance feature of the corresponding task sample is higher than the second threshold. The target performance feature includes: the performance of the candidate task model for the corresponding task sample is better or worse than that of other candidate task models, and the degree of performance difference meets the significance condition.

9. A model routing device, comprising: The acquisition module is used to obtain the target task currently pending processing; The determination module is used to determine the performance uncertainty information of each candidate task model for the target task based on the semantic features of the target task and the model features of each candidate task model through a routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models. The routing decision module is used to determine model routing information based on the performance uncertainty information, and the model routing information is used to determine the task model to process the target task.

10. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor is configured to perform the following processing by executing a set of instructions stored in the memory: Obtain the current pending target task; Based on the semantic features of the target task and the model features of each candidate task model, the performance uncertainty information of each candidate task model for the target task is determined by the routing model. The performance uncertainty information is used to quantify the volatility of the inference results of the candidate task models. Based on the performance uncertainty information, model routing information is determined, and the model routing information is used to determine the task model to process the target task.

Images