Large language model routing method, device and equipment based on zero sample difficulty perception

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a large language model routing method, device and equipment based on zero sample difficulty perception. The method comprises the steps that the relation between the performance and the cost of a candidate large language model and the difficulty is determined based on sample data, the sample data is generated by sampling a preset data set, data in the data set is a query prompt word with the difficulty pre-calibrated, and the difficulty is determined by the text attribute of the query prompt word; obtaining a query prompt word, and predicting the difficulty of the query prompt word; and estimating the cost and performance of each candidate large language model based on the cue word query difficulty, and determining a large language model for processing the cue word query from the candidate large language models based on the cost and performance.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence technology, and particularly relates to a large language model routing method, device, and equipment based on zero-shot difficulty perception. Background Art

[0002] Currently, cloud platforms such as Amazon Web Services host a large number of large language models to meet diverse user needs. Among them, larger models usually have higher performance, but the high computational cost of large-scale large language models makes them very inefficient in processing daily queries, while these queries can actually be effectively solved by smaller models.

[0003] However, current large language model routing frameworks are troubled by model locking and lack a dynamic adaptation mechanism. As a result, current large language model routing frameworks either limit deployment to a static model library, which may sacrifice performance improvement; or absorb high operating latency and computational costs to completely transform the routing logic, which disrupts the scalable and cost-aware alignment between the model and the task. Summary of the Invention

[0004] Embodiments of the present disclosure propose a large language model routing solution based on zero-shot difficulty perception to solve the problem that current large language model routing frameworks lack a dynamic adaptation mechanism due to model locking.

[0005] The first aspect of the embodiments of the present disclosure provides a large language model routing method based on zero-shot difficulty perception, including:

[0006] Determine the relationships between the performance, cost, and difficulty of candidate large language models based on sample data, where the sample data is generated by sampling a preset data set, the data in the data set is a query prompt word with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompt word;

[0007] Obtain a query prompt word and predict the difficulty of the query prompt word;

[0008] Estimate the cost and performance of each candidate large language model based on the difficulty of the query prompt word, and determine a large language model for processing the query prompt word from the candidate large language models based on the cost and performance.

[0009] In some embodiments of the present disclosure, the sample data being generated by sampling a preset data set includes:

[0010] Stratify the preset data set into K equally wide difficulty intervals, and sample N / K representative data from each interval, where N is the total number of sampled sample data, K is an integer, and N is not less than 10 times K.

[0011] In some embodiments of the present disclosure, N is not greater than 100.

[0012] In some embodiments of the present disclosure, determining the relationship between the performance and difficulty of a candidate large language model based on sample data includes:

[0013] Processing the query prompt represented by the sample data with the candidate large language model to obtain the performance of the candidate large language model corresponding to the difficulty of the query prompt;

[0014] Determining the corresponding relationship between the performance and difficulty of the candidate large language model within the difficulty range to which the sample data belongs based on spline interpolation;

[0015] Determining the corresponding relationship between the performance and difficulty of the candidate large language model within the entire difficulty range through cubic spline interpolation based on the corresponding relationship between the performance and difficulty of the candidate large language model within each difficulty range.

[0016] In some embodiments of the present disclosure, estimating the cost of a model for a candidate large language model on a preset sample data set includes:

[0017] Processing the query prompt represented by the sample data in the sample data set with the candidate large language model to obtain the output length of the candidate large language model corresponding to the difficulty of the query prompt;

[0018] Determining the corresponding relationship between the output length and difficulty of the candidate large language model within the difficulty range to which the sample data belongs based on spline interpolation;

[0019] Statistical averaging the output lengths of each difficulty range to obtain the corresponding relationship between the average output length and difficulty of the candidate large language model;

[0020] Determining the corresponding relationship between the cost and difficulty of the candidate large language model based on the unit prices of the input and output tokens of the candidate large language model and the corresponding relationship between the average output length and difficulty.

[0021] In some embodiments of the present disclosure, predicting the difficulty of the query prompt includes:

[0022] Combining the text embedding features and semantic features of the query prompt to predict the difficulty of the query prompt.

[0023] In some embodiments of the present disclosure, combining the text embedding features and semantic features of the query prompt to predict the difficulty of the query prompt includes:

[0024] Predict the difficulty of the query prompt based on a difficulty prediction model with DistilBERT as the backbone network. The difficulty prediction model includes at least one feature fusion layer and a regularized regression head. The feature fusion layer concatenates the embedding encoding representation characterizing semantic depth and the semantic features characterizing structural complexity, and then dynamically weights the contributions of the features through an attention mechanism. The regression head maps the fused features to a difficulty score aligned with the item response theory.

[0025] In some embodiments of the present disclosure, determining the large language model for processing the query prompt from the candidate large language models based on the cost and performance includes:

[0026] Determine the large language model for processing the query prompt based on the following constraint conditions in the principle of maximizing the total performance under the cost constraint;

[0027]

[0028] where M is the set of large language models, S is the set of samples, and the binary variable x ij represents whether the sample q j is assigned to the large language model l i and p ij is the performance score generated for each assignment, c ij is the cost generated by the assignment, and C max is the total cost budget.

[0029] The second aspect of the embodiments of the present disclosure provides a large language model routing device based on zero-shot difficulty awareness, including:

[0030] A determination module, configured to respectively determine the relationships between the performance and cost of the candidate large language models and the difficulty based on sample data, where the sample data is generated by sampling a preset data set, and the data in the data set is a query prompt with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompt;

[0031] A prediction module, configured to obtain a query prompt and predict the difficulty of the query prompt;

[0032] A routing module, configured to estimate the cost and performance of each of the candidate large language models based on the difficulty of the query prompt, and determine the large language model for processing the query prompt from the candidate large language models based on the cost and performance.

[0033] The third aspect of the embodiments of the present disclosure provides a large language model routing device based on zero-shot difficulty awareness, including a memory and a processor,

[0034] The memory is configured to store a computer program;

[0035] The processor is configured to implement the zero-shot difficulty-aware large language model routing method according to any one of claims 1-8 when executing the computer program.

[0036] In summary, the zero-shot difficulty-aware large language model routing methods, apparatuses, and devices provided by the embodiments of the present disclosure establish the relationship between model-agnostic difficulty values and large model performance / cost, and only need to select as few as 100 representative samples to accurately and efficiently evaluate the newly integrated large language model. It solves the problem that the current routing system cannot be integrated or requires high computational resource overhead for integration, provides a reliable basis for dynamic routing and cost optimization, and significantly improves the scalability and efficiency of the LLM ecosystem; and by combining the text embedding features and semantic features of the prompts, it predicts the difficulty score for unseen prompts in real time, solving the problem that existing prediction methods either lack cross-domain generalization ability or require comprehensive model-prompt testing. It realizes a zero-shot method for directly inferring complexity from the contextual attributes of prompts; finally, it uses a linear integer programming algorithm to balance cost and accuracy under different constraints. It solves the problem of balancing cost and performance in model selection and realizes the function of dynamically selecting models according to the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings. The drawings are schematic and should not be construed as imposing any limitation on the present disclosure. In the drawings:

[0038] Figure 1 is a schematic diagram of a zero-shot difficulty-aware large language model routing framework proposed by the present disclosure;

[0039] Figure 2 is a schematic diagram of a computer system applicable to the present disclosure;

[0040] Figure 3 is a flowchart of a zero-shot difficulty-aware large language model routing method according to some embodiments of the present disclosure;

[0041] Figure 4 is the performance comparison result of our method with each single model under different cost constraints;

[0042] Figure 5 is a schematic diagram of a zero-shot difficulty-aware large language model routing apparatus according to some embodiments of the present disclosure;

[0043] Figure 6 is a schematic diagram of a zero-shot difficulty-aware large language model routing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant disclosures. However, it will be apparent to those of ordinary skill in the art that the present disclosure may be practiced without these details. It should be understood that the terms "system", "apparatus", "unit" and / or "module" used in the present disclosure are a means of distinguishing between different components, elements, parts or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0045] It should be understood that when a device, unit or module is referred to as being "on", "connected to" or "coupled to" another device, unit or module, it may be directly on, connected or coupled to or in communication with the other device, unit or module, or there may be intermediate devices, units or modules, unless the context clearly dictates otherwise. For example, the term "and / or" as used in the present disclosure includes any and all combinations of one or more of the associated listed items.

[0046] The terms used in the present disclosure are for the purpose of describing particular embodiments only and are not intended to limit the scope of the present disclosure. As used in the specification and claims of the present disclosure, unless the context clearly dictates otherwise, the words "a", "an", "one" and / or "the" are not specifically intended to be singular and may also include the plural. Generally speaking, the terms "comprising" and "including" merely indicate the inclusion of the features, wholes, steps, operations, elements and / or components that have been expressly identified, and such expressions do not constitute an exclusive listing, and other features, wholes, steps, operations, elements and / or components may also be included.

[0047] Referring to the following description and the accompanying drawings, these or other features and characteristics of the present disclosure, the operating methods, the functions of the relevant elements of the structure, the combination of parts and the economy of manufacture can be better understood, wherein the description and the drawings form a part of the specification. However, it is to be clearly understood that the drawings are for the purpose of illustration and description only and are not intended to limit the scope of the present disclosure. It is understood that the drawings are not drawn to scale.

[0048] A variety of structure diagrams are used in the present disclosure to illustrate various variations in accordance with embodiments of the present disclosure. It should be understood that the foregoing or following structures are not intended to limit the present disclosure. The scope of protection of the present disclosure is defined by the claims.

[0049] The rapid development of large language models, from compact task-specific systems (such as Med-PaLM) to large-scale general models (such as DeepSeek-R1, GPT-o1, Qwen-2, Llama-3.1), has reshaped the field of natural language processing. While scaling up models generally improves performance, the prohibitively high computational costs of large-scale large language models make them inefficient for handling daily queries, which can actually be effectively solved by smaller models.

[0050] Cloud platforms like Amazon Web Services host a large number of large language models to meet diverse user needs. However, current systems require users to manually specify the model for each query invocation. As the number of deployed models surges, this approach becomes increasingly unsustainable, mainly having two problems: (1) For simpler tasks, it fails to fully utilize more cost-effective small models; (2) Over-reliance on ultra-large models leads to unnecessary operating costs.

[0051] With the accelerating release speed and expanding application scenarios of large language models, an automated model routing system becomes crucial, which can dynamically select the optimal model for each query, thereby minimizing costs while maintaining service quality. Current large language model routing frameworks adopt two paradigms, but both have inherent scalability limitations.

[0052] (1) Direct routing framework. Such methods use static model-task mappings or cascading workflows, which require a full evaluation of queries among predefined models. These methods cannot integrate new models without a complete architectural overhaul because their logic is hard-coded for a fixed set of models; while cascading workflows incur quadratic evaluation costs as the model pool grows.

[0053] (2) Two-stage routing framework. Such methods separate performance prediction from model selection, but current practices usually rely on predictors trained based on specific architectural features. When new models are introduced, these predictors cannot capture the task-query-large language model interactions that are crucial for generalization and need to be retrained to obtain new interaction data.

[0054] Both of these paradigms suffer from model locking, that is, the rigid dependence on static model representations or training data limits scalability. The core challenge of existing routing frameworks in integrating emerging large language models is that they lack a mechanism for dynamic adaptation and have to resort to costly manual intervention or resource-intensive updates. This limitation creates a dilemma: either restrict deployment to a static model library, potentially sacrificing performance improvement; or absorb high operating latency and computational costs to completely transform the routing logic, which disrupts the scalable and cost-aware alignment between models and tasks.

[0055] In view of this, the present disclosure proposes a large language model routing framework based on zero-shot difficulty perception. First, the model-agnostic difficulty values of the query dataset are obtained by modeling through the item response theory model. Then, based on these difficulty values decoupled from the model capabilities, lightweight performance and cost analysis are provided for newly integrated large language models, thereby establishing a large language model routing framework in which the routing logic is decoupled from model-specific data, as Figure 1 shown. The key advantages of choosing difficulty over other metrics are as follows:

[0056] Model agnosticism: Using benchmarks calibrated by the item response theory as a general measure of prompt complexity avoids the model-specific biases inherent in architecture-dependent metrics.

[0057] Efficient scalability: Only as few as 100 representative samples are required to quickly complete model analysis, while other metrics, such as accuracy metrics, require exhaustive testing, or model-centric features require expensive per-model analysis.

[0058] Stability and interpretability: Different from task classification or ad-hoc complexity scores that need to be frequently updated, the difficulty values remain stable between model releases.

[0059] Figure 2 is a schematic diagram of a computer system applicable to the present disclosure. Figure 2 The computer system shown includes a large language model routing server network-connected to a large language model hosting platform. The large language model hosting platform hosts multiple large language models with different performances and different scales. The large language model routing server receives a query request from a user and selects the most suitable large language model from the large language model hosting platform to process this query request.

[0060] The large language model routing server can be any one of a single machine, a cluster, or a distributed server. In particular, the large language model routing server can be deployed on the large language model hosting platform.

[0061] Figure 3 is a flowchart of a large language model routing method based on zero-shot difficulty perception according to some embodiments of the present disclosure. In some embodiments, the large language model routing method based on zero-shot difficulty perception is executed by the Figure 2 large language model routing server shown, and the method includes the following steps:

[0062] S310, respectively determine the relationships between the performance, cost, and difficulty of candidate large language models based on sample data, where the sample data is generated by sampling a preset dataset, the data in the dataset are query prompt words with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompt words.

[0063] The present disclosure first models the difficulty determined by the text attributes of the query prompt.

[0064] To model the hint difficulty b i , the model's latent ability θ u and the guessing probability c i The interaction relationship between them, the present disclosure calibrates the difficulty value offline on the data sets in each field based on the established probability framework (such as the item response theory). The probability framework is as follows:

[0065]

[0066] where X ui indicates whether the model u correctly responds to the hint i. Although the item response theory provides a method to quantify the difficulty, the present disclosure creatively separates the difficulty b i and the ability θ u , allowing b i to be frozen as a general anchor point while updating θ u for the new model.

[0067] To obtain the offline calibrated difficulty value, the present disclosure models the interaction relationship between the hint difficulty b i , the model's latent ability θ u and the guessing probability c i based on the probability framework of the above item response theory. The specific steps include:

[0068] 1. Set the normal distribution priors for θ u , b i , c i .

[0069] 2. Use stochastic variational inference for joint estimation. The core of stochastic variational inference is to optimize the evidence lower bound (ELBO), and its formula is:

[0070]

[0071] where q(θ u , b i , c i ) is the variational distribution, p(θ u , b i , c i ) is the prior distribution, is the expectation of the log-likelihood function, representing the expected value of the log-likelihood of the observed data X under the variational distribution. The KL divergence is used to measure the difference between the variational distribution and the prior distribution. By maximizing the ELBO, the accurate parameter estimate value of b i is finally obtained.

[0072] Then construct a sample data set for evaluating the performance and cost of the large language model:

[0073] Using the pre-calibrated difficulty value b i Stratify the domain or general dataset into K equally wide difficulty intervals. From each interval, sample N / K representative anchor prompts (N is the total number of sampled sample data, K is an integer, N is not less than 10 times K and usually N ≤ 100).

[0074] Determine the relationship between the performance and difficulty of candidate large language models on the sample dataset:

[0075] When introducing a new model u new Only analyze its capabilities on these anchor points. This sparse evaluation is sufficient because the difficulty hierarchy serves as a shared reference framework: the performance of the model on the anchor points is directly mapped to its expected accuracy across the entire difficulty range through spline interpolation. For the new model v new The ability parameters Are estimated by maximum likelihood estimation (MLE), using a regularized log-likelihood function:

[0076]

[0077] Where y i ∈ {0, 1} indicates the correctness of the model on anchor point i, ∈ = 10 -6 Is used to prevent numerical underflow, and λ controls the strength of L2 regularization.

[0078] Finally, construct a cubic spline interpolator that maps the difficulty value b covering the entire interval to the expected accuracy Thus, performance prediction for any unseen data point can be achieved through a simple lookup based on difficulty.

[0079] Determine the relationship between the cost and difficulty of candidate large language models on the sample dataset:

[0080] This disclosure efficiently estimates the cost using the relationship between difficulty and output length. The effectiveness of this method is mainly based on the following two observations:

[0081] 1. Correlation between difficulty and output length: More difficult prompts from various tasks (such as mathematical reasoning, code generation) usually produce longer outputs (detailed derivations, multi-step code), thus forming a cost structure related to the difficulty hierarchy.

[0082] 2. Reuse of representative samples: Since the representative samples have been stratified by difficulty, reusing them for cost estimation can avoid additional prompt testing.

[0083] Method flow:

[0084] 1. Obtain a partial relationship table from a representative sample: Evaluate the output length obtained by the new model on the sample data set, and associate the difficulty value of the sample with the output length.

[0085] 2. Interpolate to obtain a complete relationship table: Use the data obtained in the previous step for spline interpolation to directly infer the model output length for the entire difficulty range without additional testing.

[0086] 3. Statistically calculate the average output length for each difficulty range to obtain an average model output length mapping table.

[0087] 4. Finally, multiply the average model output length by the price of a single output token of the corresponding model, and add the known input length multiplied by the price of a single input token of the corresponding model to obtain the estimated model inference cost.

[0088] S320, Obtain a query prompt and predict the difficulty of the query prompt.

[0089] The present disclosure evaluates the difficulty of prompts from two perspectives:

[0090] 1. Semantic depth: Use the DistilBERT model to embed and encode the context semantics (such as technical terms, implicit inference chains), and these features are related to higher difficulty.

[0091] 2. Structural complexity: Quantitatively represented by 11 semantic features: including rhetorical structure ambiguity, average sentence length, parse tree depth and other dimensions.

[0092] Specifically, the difficulty prediction model of the present disclosure employs a DistilBERT backbone network with 66M parameters to process the input token sequence and output the context embedding representation. The feature fusion layer concatenates the embedding encoding representation characterizing semantic depth with the semantic features characterizing structural complexity, and then dynamically weights the contributions of these features through an attention mechanism (for example, emphasizing syntactic depth in programming tasks and focusing on lexical diversity in creative writing). A regularized regression head maps the fused representation to a difficulty score aligned with the item response theory.

[0093] The difficulty prediction model of the present disclosure aims to inherit the statistical robustness of psychometric theory while avoiding its computational overhead. In addition, the model can handle unseen task types in zero-shot mode because its features are extracted from the inherent properties of the text rather than relying on the model's post hoc interactions.

[0094] S330, Estimate the cost and performance of each of the candidate large language models based on the difficulty of the query prompt, and determine a large language model for processing the query prompt from the candidate large language models based on the cost and performance.

[0095] After predicting the difficulty of the query prompt word input by the user using the difficulty prediction model, the cost and performance of each candidate large language model for processing this prompt word can be determined based on the relationship between the large language model cost, performance, and difficulty obtained in S310.

[0096] The present disclosure formulates the large language model routing assignment problem as an integer linear programming problem to maximize the total accuracy under cost constraints.

[0097] Specifically, based on the following constraint conditions, the total performance is maximized while complying with the cost constraints:

[0098]

[0099] Where M is the set of large language models, S is the set of samples, and the binary variable x ij represents whether the sample q j is assigned to the large language model l i , p ij is the performance score generated for each assignment, c ij is the cost generated by the assignment, and C max is the total cost budget.

[0100] The second constraint condition ensures that each prompt is routed to and only routed to one model, while the third constraint condition ensures that the total cost does not exceed the budget limit.

[0101] An embodiment of the present disclosure conducts a comparative test on Figure 3 the method described in S310 - S330 (referred to as our method) and the state - of - the - art two - stage method FORC as a comparison baseline.

[0102] The test implemented the item response theory model based on the py-irt library of the Mendoza Business Analytics, Learning, and Language Lab, and trained for 2000 epochs on a single A100 40GB GPU. The test developed a DistilBERT model with 66M parameters as the difficulty prediction model, which was fine-tuned for 50 epochs on a single A100 GPU with a learning rate of 3e-5 and a batch size of 32. The datasets selected for the test included ARC Challenge for scientific question answering, GSM8K for math problem solving, HellaSwag for common sense reasoning, TruthfulQA for real-world question answering, and the WinoGrande dataset for coreference resolution, totaling 14,617 samples. When training the item response theory model, to avoid redundant work, the test utilized the evaluation results collected by TinyBenchmark from the OpenLLM leaderboard of HuggingFace. These results covered the above five tasks and involved 395 LLMs. In addition, to train the difficulty prediction model, the test divided each collected dataset into a training set and a test set at a ratio of 7:3. The list of 9 candidate large language models selected by the test in the routing is shown in Table 1.

[0103]

[0104] Table 1 List of candidate large language models selected in the test ($ / per million tokens)

[0105] The test set all candidate models as a new integrated model and adopted the cost savings ratio as the measurement metric, which reflects the cost savings achieved by our method when matching the performance of a single model. Table 2 shows the average accuracy and total cost of the top two most accurate models on each dataset. And it compares the cost savings ratio of our method and the baseline method when matching the corresponding performance. Our method achieved significant cost savings on all datasets while maintaining competitive accuracy. In particular, we achieved cost savings of 24.3% and 70.1% on HellaSwag without accuracy loss and with minimal accuracy loss respectively, which is significantly better than the baseline method. Secondly, we achieved a higher cost savings ratio than the baseline method in 7 out of 10 groups of data, and there were only gaps of 0.2%, 1.69%, and 4% respectively in the other 3 groups of data. Figure 4 It shows the performance comparison results of our method and each single model under different cost constraints, and these results highlight the cost efficiency and scalability of our method in the model routing task.

[0106]

[0107] Table 2 Comparison of the cost savings ratio between our method and the baseline method

[0108] Figure 5 It is a schematic diagram of a large language model routing device based on zero-shot difficulty perception shown in some embodiments of the present disclosure. As Figure 5 shown, the large language model routing device 500 based on zero-shot difficulty perception includes a determination module 510, a prediction module 520, and a routing module 530. In some embodiments of the present disclosure, the large language model routing function based on zero-shot difficulty perception is performed by Figure 2 the large language model routing server shown, where:

[0109] The determination module 510 is configured to respectively determine the relationships between the performance and cost and the difficulty of candidate large language models based on sample data, where the sample data is generated by sampling a preset data set, the data in the data set is a query prompt word with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompt word;

[0110] The prediction module 520 is configured to obtain a query prompt word and predict the difficulty of the query prompt word;

[0111] The routing module 530 is configured to estimate the cost and performance of each of the candidate large language models based on the difficulty of the query prompt word, and determine a large language model for processing the query prompt word from the candidate large language models based on the cost and performance.

[0112] Figure 6 It is a schematic diagram of a large language model routing device based on zero-shot difficulty perception shown in some embodiments of the present disclosure. As Figure 6 shown, the large language model routing device 600 based on zero-shot difficulty perception includes a memory 620 and a processor 610. The memory 620 is configured to store a computer program; the processor 610 is configured to implement the Figure 3 large language model routing method based on zero-shot difficulty perception described in S310-S330.

[0113] In summary, the large language model routing methods, devices, and equipment provided by the embodiments of the present disclosure based on zero-shot difficulty perception can accurately and efficiently evaluate newly integrated large language models by establishing the relationship between model-agnostic difficulty values and large model performance / cost, only requiring as few as 100 representative samples. This solves the problem that current routing systems cannot be integrated or require high computational resource overhead for integration, provides a reliable basis for dynamic routing and cost optimization, and significantly improves the scalability and efficiency of the LLM ecosystem; and by combining the text embedding features and semantic features of prompts, predicts the difficulty score for unseen prompts in real time, solving the problem that existing prediction methods either lack cross-domain generalization ability or require comprehensive model-prompt testing. It realizes a zero-shot method for directly inferring complexity from the context attributes of prompts; finally, uses a linear integer programming algorithm to balance cost and accuracy under different constraints. This solves the problem of balancing cost and performance in model selection and realizes the function of dynamic model selection according to the diverse needs of users.

[0114] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the foregoing apparatus embodiments and will not be elaborated herein.

[0115] Although the subject matter described herein is provided in the general context of execution in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations can also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art can understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and can also be used in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0116] Those of ordinary skill in the art can realize that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0117] It should be understood that the above specific embodiments of the present disclosure are only used for exemplary illustration or explanation of the principles of the present disclosure, and do not constitute a limitation to the present disclosure. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present disclosure shall be included within the protection scope of the present disclosure. In addition, the appended claims of the present disclosure are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A large language model routing method based on zero-shot difficulty perception, characterized in that including: respectively determining the relationships between the performance, cost, and difficulty of candidate large language models based on sample data, where the sample data is generated by sampling a preset data set, the data in the data set is query prompts with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompts; obtaining a query prompt and predicting the difficulty of the query prompt; estimating the cost and performance of each candidate large language model based on the difficulty of the query prompt, and determining a large language model for processing the query prompt from the candidate large language models based on the cost and performance.

2. The method according to claim 1, wherein The sample data being generated by sampling a preset data set includes: stratifying the preset data set into K equally wide difficulty intervals, and sampling N / K representative data from each interval, where N is the total number of sampled sample data, K is an integer, and N is not less than 10 times K.

3. The method according to claim 2, wherein: N is not greater than 100.

4. The method according to claim 2, wherein Determining the relationship between the performance and difficulty of candidate large language models based on sample data includes: processing the query prompts represented by the sample data with the candidate large language models to obtain the performance of the candidate large language models corresponding to the difficulty of the query prompts; determining the corresponding relationship between the performance and difficulty of the candidate large language models within the difficulty interval where the sample data belongs based on spline interpolation; determining the corresponding relationship between the performance and difficulty of the candidate large language models within the entire difficulty interval through cubic spline interpolation based on the corresponding relationships between the performance and difficulty of the candidate large language models within each difficulty interval.

5. The method according to claim 1, wherein Estimating the cost of a model for a candidate large language model on a preset sample data set includes: processing the query prompts represented by the sample data in the sample data set with the candidate large language models to obtain the output lengths of the candidate large language models corresponding to the difficulty of the query prompts; determining the corresponding relationship between the output length and difficulty of the candidate large language models within the difficulty interval where the sample data belongs based on spline interpolation; statistically calculating the average output length of each difficulty interval to obtain the corresponding relationship between the average output length and difficulty of the candidate large language models; determining the corresponding relationship between the cost and difficulty of the candidate large language models based on the unit prices of the input and output tokens of the candidate large language models and the corresponding relationship between the average output length and difficulty.

6. The method according to claim 1, wherein The predicting the difficulty of the query prompt includes: combining the text embedding features and semantic features of the query prompt to predict the difficulty of the query prompt.

7. The method according to claim 6, wherein The combining the text embedding features and semantic features of the query prompt to predict the difficulty of the query prompt includes: predicting the difficulty of the query prompt based on a difficulty prediction model with DistilBERT as the backbone network, where the difficulty prediction model includes at least one feature fusion layer and a regularized regression head. The feature fusion layer concatenates the embedding encoding representation characterizing semantic depth and the semantic features characterizing structural complexity, and then dynamically weights the contributions of the features through an attention mechanism. The regression head maps the fused features to a difficulty score aligned with the item response theory.

8. The method according to claim 1, wherein Determining the large language model for processing the query prompt from the candidate large language models based on the cost and performance includes: Determining the large language model for processing the query prompt based on the principle of maximizing the total performance under the cost constraint according to the following constraints; Among them, M is a set of large language models, S is a set of samples, and the binary variable X ij represents whether the sample q j is assigned to the large language model l i , p ij is the performance score generated for each assignment, c ij is the cost generated by the assignment, and C max is the total cost budget.

9. A large language model routing device based on zero-shot difficulty perception, characterized in that, Including: A determination module, configured to respectively determine the relationship between the performance, cost, and difficulty of the candidate large language models based on sample data, where the sample data is generated by sampling a preset data set, the data in the data set are query prompts with pre-calibrated difficulty, and the difficulty is determined by the text attributes of the query prompts; A prediction module, configured to obtain a query prompt and predict the difficulty of the query prompt; A routing module, configured to estimate the cost and performance of each of the candidate large language models based on the difficulty of the query prompt, and determine the large language model for processing the query prompt from the candidate large language models based on the cost and performance.

10. A large language model routing device based on zero-shot difficulty perception, characterized in that: Including a memory and a processor, The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the large language model routing method based on zero-shot difficulty awareness according to any one of claims 1-8.

Images