An Automatic Language Model Selection Method

Abstract

This invention relates to an automatic selection method for large language models, comprising the following steps: S1, structured requirement vector extraction based on LLM; S2, adaptive probing based on uncertainty entropy; S3, multi-level model filtering based on hard constraints; S4, capability ranking based on consensus benchmarks and four levels of evidence confidence; and S5, cost-closed-loop reordering and report assembly based on token estimation. This invention unifies natural language scene description and the capability space of large language models within the same framework, achieving efficient, comparable, and interpretable large language model selection, overcoming the shortcomings of existing technologies in areas such as missing requirement-capability mapping, incomparable benchmarks, failure of multi-constraint coordination, and black-box recommendation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and decision support systems, and to an automatic selection method for large language models (LLMs), particularly an automatic selection method for large language models oriented towards scenario-based requirements. Background Technology

[0002] With the rapid penetration of large language models in enterprise-level scenarios, there are now over four thousand open-source and commercial LLMs available on the market, along with thousands of public benchmarks that characterize their capabilities from different dimensions. However, when enterprises or developers select an LLM for specific business scenarios (such as e-commerce customer service, legal review, medical Q&A, content creation, etc.), they generally face the following technical problems: 1) Lack of structured mapping between requirements and capability dimensions. Current practices usually rely on users manually comparing model cards or benchmark lists. On the one hand, business scenario descriptions are natural language and open-ended; on the other hand, model capabilities are discrete values ​​scattered across multiple benchmarks. There is no unified intermediate representation between the two, leading to a selection process that is highly dependent on experience and difficult to reproduce. 2) Heterogeneity among benchmarks, making horizontal comparison difficult. Different benchmarks have different value ranges, evaluation methods, and covered model sets (e.g., the score range of BBH (Big-BenchHard), the ELO rating system, normalized accuracy, etc., are not comparable). Directly weighting and summing the raw scores will lead to a mismatch of dimensions; while relying solely on a single benchmark will lose the diversity of evaluation dimensions. 3) The selection decision-making process lacks engineered multi-constraint coordination and interpretable closed loops. This problem includes three intertwined sub-problems: First, hard constraints (budget, data residency, deployment method, compliance) and soft preferences (capability preferences, cost trade-offs) are mixed into the same weighted scoring function, which means that models that exceed budget limits or violate compliance may be recommended due to high capability scores, resulting in high risk for enterprise-level selection; Second, model pricing is based on "per million tokens", while the actual monthly cost depends on the token usage distribution in specific scenarios. Existing methods either ignore costs or only rank by pricing, lacking a scenario-oriented cost feedback closed loop; Third, the selection results usually only give a black-box conclusion that "model A is the most suitable", without informing the source of evidence, confidence level, uncovered capability dimensions, and how to verify before going live, making it difficult for decision-makers to further implement the decision. Summary of the Invention

[0003] To address the shortcomings of the prior art, the technical problem to be solved by the present invention is reflected in the following aspects: 1) This invention achieves automatic mapping from open-scene text to structured requirement vectors, and completes the input as needed when it is incomplete. The invention designs a structured requirement vector (RequirementVector), which maps natural language scene descriptions into multi-dimensional structured vectors through a dual-judgment process of LLM calls and code rules. Furthermore, it calculates weighted uncertainty scores for each dimension. When global uncertainty exceeds a threshold or key risk dimensions are not confirmed, it automatically selects the question with the highest information gain for follow-up questioning, converging in a maximum of N rounds, serving as a unified intermediate representation for subsequent filtering, ranking, and report generation.

[0004] 2) Achieve comparability assessment of multi-source heterogeneous benchmarks. This invention prioritizes the "consensus benchmark" that covers the most candidate models for each evaluation dimension, and performs min-max normalization within the consensus benchmark, so that most candidate models can be compared under the same "ruler"; at the same time, it corrects the benchmarks that LLM may select incorrectly through a keyword-driven benchmark correction mechanism, solving the problems of inconsistent dimensions and dimension mismatch across benchmarks.

[0005] 3) Achieving an interpretable decision-making closed loop under multi-constraint coordination. This invention completely decouples hard constraints (budget, data residency, deployment method, compliance) from soft preferences (capability preferences, cost trade-offs): hard constraints are executed independently through multi-level filtering, and capability scores are only calculated within the candidate set that passes the hard filter; further, monthly costs are inversely calculated by estimating the token usage of business scenarios through LLM, and a differentiated slot allocation strategy is adopted based on budget sensitivity (Slot 0 takes the optimal capability within the budget, and Slots 1-2 take the global mixed score) to achieve capability-cost balance; at the same time, the four-level evidence confidence level (verified / claimed / inferred / missing) makes the scores of each capability dimension verifiable and the missing data visible, and automatically generates a three-stage verification roadmap (rapid verification → deep PoC (Proof of Concept, feasibility verification) → pre-deployment confirmation), making the selection results interpretable and implementable.

[0006] This invention provides an automatic selection method for large language models, specifically a method for automatic selection of large language models oriented towards scenario-based needs. Through techniques such as structured requirement vectors, adaptive follow-up questions, two-stage screening (hard filtering-soft reordering), consensus benchmark normalization, four-level evidence confidence, and cost-closed-loop reordering driven by token estimation, it achieves end-to-end automated decision-making from open scenario text to a shortlist of candidates. The decision report serves only as a visual output of the selection results, facilitating user review and implementation verification.

[0007] This invention provides an automatic language model selection method, comprising the following steps: S1. Based on the Extracted Large Language Model (LLM), the user's natural language business scenario text is parsed to obtain an initial structured requirement vector and the internal logical consistency of the initial structured requirement vector is verified to obtain a structured requirement vector. S2. Calculate the dimension uncertainty score and global uncertainty of the structured demand vector. When the global uncertainty exceeds the preset threshold or the key risk dimension is missing, select the question with the greatest information gain and adaptively ask the user to iterate and update the structured demand vector until convergence. S3. Based on multi-level hard constraints, perform pre-filtering on the entire large model library to obtain a set of compliant candidate models; S4. For each business evaluation question, select the consensus evaluation benchmark that covers the most candidate models. Based on the four-level evidence confidence level, filter valid evaluation data, normalize the original scores, and then weight and aggregate them to obtain the model's basic capability score; and S5. Set a differentiated allocation strategy based on business budget sensitivity, estimate the model usage cost based on token usage, fine-tune the basic capability score of the model in combination with cost weight, and select and generate the optimal model shortlist to obtain the selection results.

[0008] Further, in step S1, the user's natural language business scenario text is parsed based on the extracted large language model to obtain the initial structured requirement vector. This initial structured requirement vector is obtained by mapping the natural language business scenario text to the initial structured requirement vector through a single LLM call. : in: For text in natural language business scenarios; : Represents the modal requirements of a business scenario; among them, text_only indicates a task requirement involving only text input processing, multimodal indicates a multimodal processing requirement involving both text and non-text input (images, charts, scanned documents, or videos, etc.), and text_to_image indicates a task requirement that takes text as input and generates an image output; The capability dimension vector includes seven dimensions: factual accuracy, reasoning depth, language quality, long text processing, domain expertise, creative flexibility, and visual understanding. Deployment constraints include data residency, deployment method (API / localization), whether open source is required, budget amount, and daily active user scale; Each capability dimension includes the accompanying business evaluation questions (decision_questions). Each Also includes a list of candidate Benchmark IDs selected by LLM. ( It is tied to a single business evaluation issue. (A set of benchmark numbers above); Risk tolerance, including the level of consequences of errors. Density of manual supervision Reversibility .

[0009] Further, in step S1, the internal logical consistency of the initial structured requirement vector is verified to obtain the structured requirement vector as follows: the initial structured requirement vector is post-processed according to code rules to ensure internal consistency of the requirement vector; if local private deployment is selected in the requirement, i.e. This will automatically set restrictions that business data can only be stored domestically and require the corresponding large model to be open-source and downloadable. and Modal requirements The LLM output and keyword matching are used for dual judgment to avoid single point of failure in LLM.

[0010] Further, in step S2, the dimensional uncertainty score and global uncertainty of the structured demand vector are calculated. When the global uncertainty exceeds a preset threshold or a key risk dimension is missing, the question with the greatest information gain is selected and adaptively followed up with the user. The structured demand vector is iteratively updated until convergence, including the following steps: For structured demand vectors Each dimension Calculate its uncertainty score : in The entropy of the normalized probability distribution output by the LLM in this dimension. As the importance weight of the dimension; Calculate global uncertainty : in, The set of indexes for the required dimensions for participating in global uncertainty calculations includes dimensions such as factual accuracy, reasoning depth, language quality, information processing volume, domain expertise, creative flexibility, visual understanding, and error consequences. This is the index for the aforementioned demand dimension; To extract large language models for the first The normalized probability distribution of candidate states or levels output by each demand dimension; The normalized entropy of the probability distribution is [0,1], and the larger the value, the higher the uncertainty of this dimension. For the first The importance weight of each demand dimension, satisfying And the sum of the weights of each dimension is 1; For the first The weighted contribution of each demand dimension to global uncertainty. This is a weighted summary of the uncertainties across various demand dimensions; A preset global uncertainty threshold θ is set based on the completeness of the text in the natural language business scenario; when U≥θ (θ is 0.15 when the text length is less than the preset length, otherwise θ is 0.40 or the default θ=0.40) or key risk dimensions (such as error_consequence) are not explicitly confirmed or the modal requirements do not meet the confirmation conditions, follow-up questions are triggered; From the candidate question pool Choose the problem that maximizes the expected information gain. : The expected information gain is estimated by the LLM in a small invocation based on the current demand vector and the problem list; and After each round of follow-up questions, the user's answer Update demand vector Recalculate ;when Or ask if the number of rounds has reached the limit. Stop when the time comes.

[0011] Further, in step S3, a pre-filter is performed on the entire large model library based on multi-level hard constraints to obtain a set of compliant candidate models, including the following steps: Input candidate model set and structured demand vector Pre-filtering is performed based on multi-level hard constraints, which include basic quality filtering, modality matching filtering, deployment method filtering, data residency compliance filtering, open source requirement filtering, and localized budget parameter filtering. The budget hard filter uses a 10% tolerance, that is... Models within the specified range are still considered candidates to avoid false rejection due to boundary values; and The candidate model set filtered through all 6 layers of hard constraints This serves as input for subsequent evidence matching and ability ranking.

[0012] Further, in step S4, for each business evaluation question, a consensus evaluation benchmark covering the most candidate models is selected. Valid evaluation data is filtered based on the four-level evidence confidence level. The original scores are normalized within the range and then weighted and aggregated to obtain the model's basic capability score. This includes the following steps: For each business evaluation question Statistical analysis of each candidate benchmark Number of coverage models The benchmark with the most coverage is selected as the consensus benchmark. : For each model With each business evaluation question ( (Representing any candidate model within the compliance candidate model set), evidence is matched according to the following priority: Verified evidence (in the consensus benchmark). The following are considered as evidence: official scores, official statements (Claimed evidence, such as "We achieved 90% on Task X"), logical inference evidence (scores on other benchmarks in the same field, which can be used as a reference), and missing data evidence (no usable evidence). Among these, the evaluation data corresponding to the confidence marker "Verified" is used as valid evaluation data in the score calculation, while the information corresponding to "Claimed" and "Inferred" is used as reference information and is not included in the score. "Missing" means there is no usable evaluation data. Each piece of evidence is recorded as: When only the evidence confidence level label corresponding to a single evaluation data point is "Verified", that is, only evidence points with status=verified participate in the ranking calculation; For each business evaluation question Collect the raw scores of all verified evidence points. According to consensus benchmark Grouping normalization: in, These are the candidate models currently to be evaluated; An index for business evaluation questions; The traversal model index used to find the maximum and minimum original scores in the same group; To establish consensus evaluation benchmarks The set of candidate models with verifiable evidence, i.e. ; For the model Regarding business evaluation issues The original evaluation score after unifying the score direction; the higher the score, the stronger the model's ability. For the model In the Normalization capability score for each business evaluation question; only those with evidence confidence status of verified and evaluation benchmark identified as [missing information]. The evaluation data were used in the above normalization calculation; If only one model has a verified evidence point for a certain business evaluation problem, or if the highest raw score of the models in the same group equals the lowest raw score, due to the lack of an effective basis for relative comparison, the corresponding model is assigned a fixed score of 0.5 for that dimension to avoid division by zero and over-rewarding of single data points; and For each model The basic capability score of the model is obtained by weighting and summing the results according to the weights of the business evaluation questions. : in, For business evaluation issues The corresponding demand dimension level, let's assume For business evaluation issues The corresponding demand dimension index, then , The required dimensions include capability dimensions and error consequence dimensions; when the level of the corresponding required dimension is missing or not greater than 0, a preset default weight is used.

[0013] Furthermore, in step S5, a differentiated allocation strategy is set based on business budget sensitivity to initially generate an optimal shortlist of models, including the following steps: If the business budget sensitivity is a defined budget, that is... Then: Filter to obtain a subset of budget constraints ; Recalculate the model's fundamental capability score within the budget constraint subset. Sort (renormalize within budget), and assign the strongest to Slot 0; Through weighted formula Calculate the global mixed score, and classify models with slight budget overruns into Slots 1 and 2. The score is the result after normalizing the monthly costs; This global blending As a score reflecting the overall capabilities of the model; If the business budget sensitivity is cost sensitive, that is... Then: A weighted formula is applied to all candidate models. Calculate the global mixture score; This global blending As a score reflecting the overall capabilities of the model; If the business budget sensitivity is that the budget is not specified, i.e. Then: Skip the budget-weighted ranking process and directly assign the model's basic capabilities output from step S4. As a score reflecting the overall capabilities of the model; and Sort the models by their overall performance scores in descending order, select the top 3 models, generate a shortlist of optimal models, and denote the set of these shortlisted models. .

[0014] Furthermore, after selecting and generating the shortlist of optimal models, the process also includes automatically assembling the basic report, supplementing business parameters through a second LLM call, and calculating the final score of the model by performing a cost closed-loop rearrangement based on the LLM output parameters from the second call. The automatic assembly of the basic report includes the following steps: Read the short list model set Assemble the following report fields in sequence: The shortlist field stores the model's strengths, evidence quality, and basic capability scores. The scoring matrix field is used to construct an evidence matrix corresponding to all candidate models and all business evaluation questions. The decision comparison field outputs the horizontal comparison content of the shortlist model under each business evaluation question; Verify the gap fields and summarize the key business dimensions where evidence is missing in the shortlist; The verification route field should be written to the three-stage process of quick verification, deep PoC, and pre-deployment confirmation; and The cost summary field estimates the monthly cost of the shortlist model based on model pricing and the default token amount.

[0015] Furthermore, the supplementary business parameters for the secondary LLM call are: taking the structured demand vector and the optimal model shortlist as input, a secondary large model call is executed to generate: The model adaptation recommendation conditions, selection decision branch tree, detailed description of the implementation and verification route, average input token prediction for business scenarios, average output token prediction for business scenarios, and cost_weight suggestion in the final ranking.

[0016] Furthermore, the final score of the cost closed-loop rearrangement calculation model based on the LLM output parameters called twice includes: The estimated average input token quantity and average output token quantity for the business scenario are calculated using the LLM with a second call. and model pricing Calculate monthly costs: in, This refers to the average number of daily users. This represents the average number of queries per user per day. and These represent the average number of input tokens and the average number of output tokens per query, respectively. and The models are respectively Input Token price and output Token price; when the price is denominated in USD / million Tokens, it is first converted according to the RMB to USD conversion factor. And the conversion factor for millions of tokens Convert it to RMB / Token and then substitute it into the above formula; If LLM returns Then, the shortlist of optimal models will be fine-tuned: in, and , which represents the weight of cost factors in the final score of the model; For the model Basic ability score; For the model The normalized cost advantage score indicates that the higher the score, the lower the estimated cost of the model and the more obvious the cost advantage.

[0017] in, ; A monthly budget given to the user; For the model Estimated monthly costs; and These are the minimum and maximum estimated monthly costs for models with valid cost data in the shortlist of optimal models, respectively; when Alternatively, if the model lacks effective cost data, set the model's cost advantage score to a preset neutral value of 0.5.

[0018] Furthermore, the selection results include a shortlist of optimal models, complete evaluation evidence, model usage costs, and deployment and validation paths. The selection results are displayed in the form of a visual report page. The visual report page includes shortlist information cards, a full-dimensional scoring table, a selection decision branch tree, a deployment and validation roadmap, and a multi-model cost comparison chart, and supports exporting a complete report in PDF format.

[0019] This invention has at least the following beneficial effects: 1) By combining structured demand vectors, consensus benchmark normalization, and an interpretable decision-making closed loop of multi-constraint coordination, this invention unifies natural language scene description and large language model capability space under the same framework, achieving efficient, comparable, and interpretable large language model selection, overcoming the shortcomings of existing technologies in terms of missing demand-capability mapping, incomparable benchmarks, failure of multi-constraint coordination, and black-box recommendation; 2) This invention adopts an adaptive inquiry mechanism driven by structured demand vectors and uncertainty entropy, mapping open natural language scene descriptions into multi-dimensional computable and comparable intermediate representations through a single LLM call and code chain reasoning, and supplementing key dimensions according to the principle of maximizing information gain, solving the problems of lack of unified mapping between demand and capability dimensions and high dependence on experience in existing methods. 3) This invention uses consensus benchmark selection, keyword correction, and intra-group min-max normalization to enable candidate models to be evaluated under the same "ruler," improving the mathematical rationality and ranking credibility of capability score aggregation under multi-source heterogeneous benchmarks. It solves problems such as dimensional mismatch caused by weighted summation of original scores and loss of dimensional diversity when relying on a single benchmark. 4) This invention uses a two-stage screening of hard constraints and soft preferences, cost closed-loop reordering based on LLMToken estimation, and four-level evidence confidence display to make the selection results verifiable in three dimensions: compliance and security, scenario-based cost perception, and evidence traceability. It also provides a three-stage verification roadmap to provide a pre-launch implementation path, solving problems such as mixed hard and soft constraint scoring, separation between model pricing and real scenario costs, and black box-like recommendation conclusions in existing methods. Attached Figure Description

[0020] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the embodiments of the invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0021] Figure 1 A flowchart illustrating the automatic selection method for large language models in some embodiments of the present invention is shown; Figure 2The following diagram illustrates the decision report display in some embodiments of the present invention. Detailed Implementation

[0022] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0023] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0024] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0025] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0026] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0027] It should also be noted that in the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0028] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0029] The following embodiments provide a method for automatic selection of large language models. Figure 1 The flowchart illustrates the automatic language model selection method for large languages, including the following steps: Step S1: Structured Demand Vector Extraction Based on LLM 1-1) Input: Natural language scene description text input by the user. .

[0030] 1-2) LLM Extraction: Extracting text using a single LLM call. Mapped to the initial structured requirement vector : in: : Represents the modal requirements of a business scenario; among them, text_only indicates a task requirement that only involves text input processing, multimodal indicates a multimodal processing requirement that involves both text and non-text input, and text_to_image indicates a task requirement that takes text as input and generates an image output; The capability dimension vector includes seven dimensions: factual accuracy, reasoning depth, language quality, long text processing, domain expertise, creative flexibility, and visual understanding. Deployment constraints include data residency, deployment method (API / local), whether open source is required, budget amount and daily active user scale; Each capability dimension comes with decision questions (business evaluation questions). Also includes a list of candidate benchmark IDs selected by LLM. ; Risk tolerance includes the level of consequences of errors, the density of human supervision, and reversibility.

[0031] 1-3) Chained Reasoning Constraints: This invention introduces several code rules to post-process the LLM output, ensuring consistency within the demand vector: like Then set automatically. and ; Modal requirements By using both LLM output and keyword matching for dual verification, single-point errors in LLM are avoided.

[0032] This step yields the complete structured requirement vector. This serves as the unified input for all subsequent steps.

[0033] Step S2: Adaptive probing based on uncertainty entropy The demand vector extracted in a single LLM query may have uncertainties in certain dimensions (e.g., the user did not mention the budget, or did not express their tolerance for errors). This invention designs an adaptive follow-up questioning mechanism based on uncertainty entropy to supplement key information as needed.

[0034] 2-1) Calculation of Dimensional Uncertainty: For the demand vector Each dimension Calculate its uncertainty score : in The entropy of the normalized probability distribution output by the LLM in this dimension. The importance weights for each dimension.

[0035] 2-2) Global uncertainty: in, The set of indexes for the required dimensions for participating in global uncertainty calculations includes dimensions such as factual accuracy, reasoning depth, language quality, information processing volume, domain expertise, creative flexibility, visual understanding, and error consequences. This is the index for the aforementioned demand dimension; To extract large language models for the first The normalized probability distribution of candidate states or levels output by each demand dimension; The normalized entropy of the probability distribution is [0,1], and the larger the value, the higher the uncertainty of this dimension. For the first The importance weight of each demand dimension, satisfying And the sum of the weights of each dimension is 1; For the first The weighted contribution of each demand dimension to global uncertainty. This is a weighted summary of the uncertainty of each demand dimension.

[0036] 2-3) Follow-up Questioning Trigger Conditions: Set a preset global uncertainty threshold θ based on the completeness of the text in the natural language business scenario; when (When the text length is less than the preset length, θ is 0.15; otherwise, θ is 0.40 or the default value.) If a key risk dimension (such as error_consequence) is not explicitly confirmed or the modal requirement does not meet the confirmation conditions, follow-up questions are triggered.

[0037] 2-4) Problem selection: From the candidate problem pool Choose the problem that maximizes the expected information gain. : The expected information gain is estimated by the LLM in a small call based on the current demand vector and the problem list.

[0038] 2-5) Convergence condition: After each round of follow-up questions, the user's answer... Update demand vector Recalculate ;when Or ask if the number of rounds has reached the limit. (default Stop when ).

[0039] Through this mechanism, the present invention achieves a balance between user experience (avoiding excessive questioning) and decision quality (key dimensions must be confirmed), ensuring complete input for subsequent steps.

[0040] Step S3: Multi-level model filtering based on hard constraints This invention completely decouples hard constraints from soft preferences: hard constraints (budget, compliance, deployment method, open source, etc.) are filtered layer by layer in this step using pure code, and capability preferences do not participate in this stage.

[0041] 3-1) Input: Set of candidate models With demand vector .

[0042] 3-2) Filtering levels: Layer 1 (Quality Threshold): Excluding embedded models, pedestal models, and low-quality fine-tuning models, we get... ; Layer 2 (Multimodal Requirements): If Only retain The model, ; Layer 3 (Deployment Method): If Only models with publicly available APIs are retained. The model that retains only open weights yields ; Layer 4 (Data Resident): If and Only Chinese manufacturers' models are retained. ; Layer 5 (Open Source Requirements): If Only retain the open-source / open weight model, and obtain ; Layer 6 (Parameters and Architecture): If ,according to Infer the upper limit of the number of parameters that can be carried and exclude models that exceed the limit; in low-budget local deployment scenarios, further exclude the MoE architecture, and obtain .

[0043] 3-3) Budget Hard Filter Tolerance: Budget hard filtering uses a 10% tolerance, that is, in Models within the specified range are still considered candidates to avoid incorrect rejection due to boundary values.

[0044] 3-4) Output: The set of candidate models that have passed all 6 layers of hard filtering. This serves as input for subsequent evidence matching and ability ranking.

[0045] Step S4: Ranking capabilities based on consensus benchmark and Level 4 evidence confidence This step matches Benchmark evidence for each candidate model on each decision_question (an evaluation question derived from business (e.g., "How well does the model understand Chinese customer service?"), and normalizes the heterogeneous benchmarks to aggregate them into a capability score.

[0046] 4-1) Consensus Benchmark Selection: For each question The present invention scans the candidate set For all models, statistical analysis is performed for each candidate benchmark. Number of coverage models The benchmark with the most coverage is selected as the consensus benchmark. : This step ensures that most candidate models are compared on the same scale, avoiding incomparability across benchmarks.

[0047] 4-2) Determining the confidence level of fourth-level evidence: For each model With each question Evidence is matched according to the following priority: Verified: In Consensus Benchmark Official scores are publicly available. Claimed: There is an official statement in the model card or paper (such as "We achieved 90% on Task X"); Inferred: Scores on other benchmarks in the same field can be used as a reference; Missing: No evidence is available.

[0048] Each piece of evidence is recorded as: Only evidence points with status=verified are included in the ranking calculation.

[0049] 4-3) Intra-group min-max normalization: For each question Collect the raw scores of all verified evidence points. According to consensus benchmark Grouping normalization: in, These are the candidate models currently to be evaluated; An index for business evaluation questions; The traversal model index used to find the maximum and minimum original scores in the same group; To establish consensus evaluation benchmarks The set of candidate models with verifiable evidence, i.e. ; For the model Needle business evaluation issues The original evaluation score after unifying the score direction; the higher the score, the stronger the model's ability. For the model In the Normalization capability score for each business evaluation question; only those with evidence confidence status of verified and evaluation benchmark identified as [missing information]. The evaluation data were used in the above normalization calculation.

[0050] 4-4) Isolated Benchmark and Tie-Breaking: If only one model has verified evidence for a certain question, or if the maximum raw score of the models in the same group is equal to the minimum raw score, normalization lacks an effective basis for relative comparison. This invention assigns a fixed score of 0.5 (instead of the full score of 1.0) to the corresponding model in this dimension to avoid division by zero and over-rewarding of single data points.

[0051] 4-5) Capability aggregation: For each model The ability score is obtained by weighting and summing the results according to the question weights. : in, For question The corresponding demand dimension level, let's assume For business evaluation issues The corresponding demand dimension index, then , The required dimensions include capability dimensions and error consequence dimensions; when the level of the corresponding required dimension is missing or not greater than 0, a preset default weight is used.

[0052] Step 5: Cost closed-loop rearrangement and report assembly based on token estimation 5-1) Preranking with Cost before slicing shortlist: This involves re-ranking the shortlist by cost. After sorting and before cutting the top 3, this invention considers budget sensitivity. Adopt a differentiation strategy: like (Define the budget): Slot0: In the budget constraint subset Internal recalculation (Renormalize within budget) Select the one with the strongest ability; Slots 1-2: Global Mixed Score Allow for slight budget overruns to retain high-capacity options; like (Cost sensitive): All use mixed components ; like Skip budget rearrangement, only sort by Sort.

[0053] 5-2) Report on pure code assembly: Shortlist (top 3) model Assemble the following structured report fields: shortlist: highlights, evidence_quality, and capability_score for each model; scoring_table: Evidence matrix of all candidate models × all questions; decision_dimensions: A detailed comparison of the shortlist model for each question; verification_gaps: an important dimension of missing evidence (status=missing) in the shortlist model; verification_roadmap: Three stages: fast verification → deep PoC → pre-deployment confirmation; cost_summary: Monthly cost estimate for each shortlist model (temporary model pricing × default token usage).

[0054] 5-3) LLM Polishing Call: Through the second LLM call (Input: Demand Vector) + shortlist), generate: conditions: Recommendation conditions text for each shortlist model (e.g., "If you value more..."); forks: decision branching tree; roadmap_descriptions: Verifies the detailed description of each item on the roadmap; avg_input_tokens, avg_output_tokens: Estimated token usage for business scenarios; cost_weight: Suggested weighting of cost in the final ranking.

[0055] 5-4) Cost Reranking: Using the token usage estimated by LLM in step 5-3. With model pricing Calculate monthly costs: in, This refers to the average number of daily users. This represents the average number of queries per user per day. and These represent the average number of input tokens and the average number of output tokens per query, respectively. and The models are respectively The input token price and output token price. When the price is denominated in USD / million tokens, it is first converted using the RMB to USD conversion factor. And the conversion factor for millions of tokens Convert it to RMB / Token and then substitute it into the above formula; If LLM returns Then, perform final fine-tuning on the shortlist: in, and , which represents the weight of cost factors in the final score of the model; For the model Basic ability score; For the model The normalized cost advantage score indicates that the higher the score, the lower the estimated cost of the model and the more obvious the cost advantage. in, ; A monthly budget given to the user; For the model Estimated monthly costs; and These represent the minimum and maximum estimated monthly costs for models with valid cost data in the shortlist of optimal models, respectively. Alternatively, if the model lacks effective cost data, set the model's cost advantage score to a preset neutral value of 0.5.

[0056] 5-5) Output: The final selection results (shortlist and its complete evidence, cost, and validation path) are the core output of this method; a visual report page (shortlist cards, scoring table, decision tree, validation roadmap, cost comparison chart, and exportable PDF) serves as the presentation format for the selection results. Figure 2 The following diagram illustrates the decision report display in some embodiments of the present invention.

[0057] Example 1: LLM Selection in E-commerce Customer Service Scenarios Scenario Input: A user enters on the system homepage "50 e-commerce customer service staff to handle returns and exchanges, with a monthly budget of 50,000 yuan, and requires good Chinese performance."

[0058] Step 1: Requirements Analysis: The system extracts a requirements vector from the text using an LLM call. The modality is determined to be plain text; in terms of capabilities, language quality is assigned the highest score, followed by factual accuracy, then domain expertise, reasoning depth and long text processing, and finally creative flexibility and visual understanding; deployment constraints are confirmed as public API, no data residency requirements, no need for open source, a monthly budget of 50,000 RMB, and a scale of approximately 50 customer service personnel; simultaneously, the LLM generates evaluation questions such as "Can the system understand Chinese customer context?", "Can the system accurately identify return and exchange policies?", and "Is the response polite and appropriate?", and selects candidate benchmarks for each question; in terms of risk, the consequences and reversibility of errors are initially estimated as moderate.

[0059] Step 2: Adaptive Follow-up Questioning: The system detects high uncertainty in the error consequence dimension (not explicitly stated by the user) and automatically triggers the follow-up question, "What would be the consequences if the model provides an incorrect return and exchange policy response?" The user answers, "Customer service will complain, but customer service will review it." Based on this, the system updates the error consequence and human supervision density metrics, reducing the overall uncertainty below the threshold, and stops following up.

[0060] Step 3 Hard Filtering: From the more than 4,800 models pre-stored in the system, the models are filtered through layers such as quality threshold, API availability, no data residency constraints, and no open source requirements, and finally about 47 candidate models are retained, mainly including API models from Chinese vendors and some general API models.

[0061] Step 4: Capability Ranking. For the question of "Can the model understand Chinese customer context?", candidate benchmarks include opencompass_language, livebench_language, etc. The system scanned the candidate set and found that opencompass_language covered 38 models, having the highest coverage, and was selected as the consensus benchmark. Subsequently, within this benchmark, intra-group min-max normalization was performed to obtain the normalized score of each model in this dimension. The remaining questions were processed in the same way, and then aggregated according to the question weights to obtain the capability score of each candidate model.

[0062] Step 5: Cost Reordering (Closed-Loop): Since the user explicitly provided a budget amount, the system entered a specific budget strategy—Slot 0 selected the strongest performer within a budget of 50,000 yuan (ultimately GLM-4-Air), and Slots 1 and 2 selected the top two performers in the overall mixed score (DeepSeek-V3 and Qwen3-Max). Subsequently, LLM estimated the average input in the customer service scenario to be approximately 800 tokens and the output approximately 200 tokens. Combining this with the pricing of each model, the monthly cost was calculated as follows: GLM-4-Air approximately 12,000 yuan, DeepSeek-V3 approximately 28,000 yuan, and Qwen3-Max approximately 45,000 yuan. LLM assigned a cost weight of 0.20, and the final cost adjustment kept the shortlist order unchanged.

[0063] Output: The report page displays highlights of the 3 shortlist models, complete evidence matrices of 47 candidate models, decision branch trees ("If you value Chinese experience more → GLM-4-Air; If you value inference depth more → DeepSeek-V3"), a three-stage validation roadmap, and a monthly cost comparison bar chart.

[0064] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. A method for automatic selection of large language models, characterized in that, Includes the following steps: S1. Based on the Extracted Large Language Model (LLM), the user's natural language business scenario text is parsed to obtain an initial structured requirement vector and the internal logical consistency of the initial structured requirement vector is verified to obtain a structured requirement vector. S2. Calculate the dimension uncertainty score and global uncertainty of the structured demand vector. When the global uncertainty exceeds the preset threshold or the key risk dimension is missing, select the question with the greatest information gain and adaptively ask the user to iterate and update the structured demand vector until convergence. S3. Based on multi-level hard constraints, perform pre-filtering on the entire large model library to obtain a set of compliant candidate models; S4. For each business evaluation question, select the consensus evaluation benchmark that covers the most candidate models, filter the effective evaluation data based on the four-level evidence confidence, normalize the original score array and then weight and aggregate it to obtain the basic capability score of the model. as well as S5. Set a differentiated allocation strategy based on business budget sensitivity, estimate the model usage cost based on token usage, fine-tune the basic capability score of the model in combination with cost weight, filter and generate the optimal shortlist of models, and obtain the selection results.

2. The automatic selection method for large language models according to claim 1, characterized in that, In step S1, the user's natural language business scenario text is parsed based on the extracted large language model to obtain the initial structured requirement vector. This is achieved by mapping the natural language business scenario text to the initial structured requirement vector through a single LLM call. : in: For text in natural language business scenarios; : Represents the modal requirements of a business scenario; among them, text_only indicates a task requirement that only involves text input processing, multimodal indicates a multimodal processing requirement that involves both text and non-text input, and text_to_image indicates a task requirement that takes text as input and generates an image output; The capability dimension vector includes seven dimensions: factual accuracy, reasoning depth, language quality, long text processing, domain expertise, creative flexibility, and visual understanding. Deployment constraints include data residency, deployment method, whether open source is required, budget amount, and daily active user scale; Business evaluation questions accompanying each capability dimension Each Also includes a list of candidate Benchmark IDs selected by LLM. ; Risk tolerance, including the level of consequences of errors. Density of manual supervision Reversibility .

3. The automatic selection method for large language models according to claim 1, characterized in that, In step S1, the internal logical consistency of the initial structured requirement vector is verified to obtain the structured requirement vector: the initial structured requirement vector is post-processed through code rules to ensure the internal consistency of the requirement vector; If local private deployment is selected in the requirements, that is This will automatically set restrictions that business data can only be stored domestically and require the corresponding large model to be open-source and downloadable. and Modal requirements The LLM output and keyword matching are used for dual judgment to avoid single point of failure in LLM.

4. The automatic selection method for large language models according to claim 1, characterized in that, In step S2, the dimensional uncertainty score and global uncertainty of the structured demand vector are calculated. When the global uncertainty exceeds a preset threshold or a key risk dimension is missing, the question with the greatest information gain is selected and adaptively followed up with the user. The structured demand vector is iteratively updated until convergence, including the following steps: For structured demand vectors Each dimension Calculate its uncertainty score : in The entropy of the normalized probability distribution output by the LLM in this dimension. As the importance weight of the dimension; Calculate global uncertainty : in, The set of indexes for the required dimensions for participating in global uncertainty calculations includes dimensions such as factual accuracy, reasoning depth, language quality, information processing volume, domain expertise, creative flexibility, visual understanding, and error consequences. This is the index for the aforementioned demand dimension; To extract large language models for the first The normalized probability distribution of candidate states or levels output by each demand dimension; The normalized entropy of the probability distribution ranges from [0,1]. A larger value indicates a higher degree of uncertainty in that dimension. For the first The importance weight of each demand dimension, satisfying And the sum of the weights of each dimension is 1; For the first The weighted contribution of each demand dimension to global uncertainty. This is a weighted summary of the uncertainties across various demand dimensions; A preset global uncertainty threshold θ is set based on the completeness of the text in the natural language business scenario; when U≥θ or key risk dimensions are not explicitly confirmed or modal requirements do not meet the confirmation conditions, follow-up questions are triggered; From the candidate question pool Choose the problem that maximizes the expected information gain. : The expected information gain is estimated by the LLM in a small invocation based on the current demand vector and the problem list; and After each round of follow-up questions, the user's answer Update demand vector Recalculate ;when Or ask if the number of rounds has reached the limit. Stop when the time comes.

5. The automatic selection method for large language models according to claim 1, characterized in that, In step S3, a pre-filtering process is performed on the entire large model library based on multi-level hard constraints to obtain a set of compliant candidate models, including the following steps: Input candidate model set and structured demand vector Pre-filtering is performed based on multi-level hard constraints, which include basic quality filtering, modality matching filtering, deployment method filtering, data residency compliance filtering, open source requirement filtering, and localized budget parameter filtering. The budget hard filter uses a 10% tolerance, that is... Models within the specified range are still considered candidates to avoid false rejection due to boundary values; and The candidate model set filtered through all 6 layers of hard constraints This serves as input for subsequent evidence matching and ability ranking.

6. The automatic selection method for large language models according to claim 2, characterized in that, In step S4, for each business evaluation question, a consensus evaluation benchmark with the most candidate models is selected. Valid evaluation data is screened based on the four-level evidence confidence level. The original scores are normalized and then weighted and aggregated to obtain the model's basic capability score. This includes the following steps: For each business evaluation question Statistical analysis of each candidate benchmark Number of coverage models The benchmark with the most coverage is selected as the consensus benchmark. : For each model With each business evaluation question Evidence is matched according to the following priority: Verified evidence, Claimed evidence, Inferred evidence, and Missing evidence. Among them, the evaluation data corresponding to the Verified confidence indicator is used as valid evaluation data in the score calculation, the information corresponding to Claimed and Inferred is used as reference information and is not included in the score, and Missing means that there is no available evaluation data. Each piece of evidence is recorded as: When only the evidence confidence level label corresponding to a single evaluation data point is "Verified", that is, only evidence points with status=verified participate in the ranking calculation; For each business evaluation question Collect the raw scores of all verified evidence points. According to consensus benchmark Grouping normalization: in, These are the candidate models currently to be evaluated; An index for business evaluation questions; The traversal model index used to find the maximum and minimum original scores in the same group; To establish consensus evaluation benchmarks The set of candidate models with verifiable evidence, i.e. ; For the model Regarding business evaluation issues The original evaluation score after unifying the score direction; the higher the score, the stronger the model's ability. For the model In the Normalization capability score for each business evaluation question; only those with evidence confidence status of verified and evaluation benchmark identified as [value missing]. The evaluation data were used in the above normalization calculation; If only one model has a verified evidence point for a certain business evaluation problem, or if the highest raw score of the models in the same group equals the lowest raw score, due to the lack of an effective basis for relative comparison, the corresponding model is assigned a fixed score of 0.5 for that dimension to avoid division by zero and over-rewarding of single data points; and For each model The basic capability score of the model is obtained by weighting and summing the results according to the weights of the business evaluation questions. : in, For business evaluation issues The corresponding level of the demand dimension, let's assume For business evaluation issues The corresponding demand dimension index, then , The required dimensions include capability dimensions and error consequence dimensions; when the level of the corresponding required dimension is missing or not greater than 0, a preset default weight is used.

7. The automatic selection method for large language models according to claim 6, characterized in that, In step S5, a differentiated allocation strategy is set based on business budget sensitivity to initially generate the optimal shortlist of models, including the following steps: If the business budget sensitivity is a defined budget, that is... Then: Filter to obtain a subset of budget constraints ; Recalculate the model's fundamental capability score within the budget constraint subset. Sort them and assign the strongest performer to Slot 0. Through weighted formula Calculate the global mixed score, and classify models with slight budget overruns into Slots 1 and 2. The score is the result after normalizing the monthly costs; This global blending As a score reflecting the overall capabilities of the model; If the business budget sensitivity is cost sensitive, that is... Then: A weighted formula is applied to all candidate models. Calculate the global mixture score; This global blending As a score reflecting the overall capabilities of the model; If the business budget sensitivity is that the budget is not specified, i.e. Then: Skip the budget-weighted ranking process and directly assign the model's basic capabilities output from step S4. As a score reflecting the overall capabilities of the model; and Sort the models by their overall performance scores in descending order, select the top 3 models, generate a shortlist of optimal models, and denote the set of these shortlisted models. .

8. The automatic selection method for large language models according to claim 1, characterized in that, After selecting and generating the shortlist of optimal models, the process also includes automatically assembling the basic report, supplementing business parameters through a second LLM call, and calculating the final score of the model by performing a cost closed-loop rearrangement based on the LLM output parameters from the second call. The automatic assembly of the basic report includes the following steps: Read the short list model set Assemble the following report fields in sequence: The shortlist field stores the model's strengths, evidence quality, and basic capability scores. The scoring matrix field is used to construct an evidence matrix corresponding to all candidate models and all business evaluation questions. The decision comparison field outputs the horizontal comparison content of the shortlist model under each business evaluation question; Verify the gap fields and summarize the key business dimensions where evidence is missing in the shortlist; The verification route field should be written to the three-stage process of quick verification, deep PoC, and pre-deployment confirmation; and The cost summary field estimates the monthly cost of the shortlist model based on model pricing and the default token amount.

9. The automatic selection method for large language models according to claim 8, characterized in that, The supplementary business parameters for the secondary LLM call are: taking the structured demand vector and the optimal model shortlist as input, performing a secondary large model call generates: The model adaptation recommendation conditions, selection decision branch tree, detailed description of the implementation and verification route, average input token prediction for business scenarios, average output token prediction for business scenarios, and cost_weight suggestion in the final ranking.

10. The automatic selection method for large language models according to claim 9, characterized in that, The final score of the cost closed-loop rearrangement calculation model based on the LLM output parameters called twice includes: The estimated average input token quantity and average output token quantity for the business scenario are calculated using the LLM with a second call. and model pricing Calculate monthly costs: in, This refers to the average number of daily users. This represents the average number of queries per user per day. and These represent the average number of input tokens and the average number of output tokens per query, respectively. and The models are respectively Input Token price and output Token price; when the price is denominated in USD / million Tokens, it is first converted according to the RMB to USD conversion factor. And the conversion factor for millions of tokens Convert it to RMB / Token and then substitute it into the above formula; If LLM returns Then, the shortlist of optimal models will be fine-tuned: in, and , representing the weight of cost factors in the final score of the model; For the model Basic ability score; For the model Normalized cost advantage score; in, ; A monthly budget given to the user; For the model Estimated monthly costs; and These are the minimum and maximum estimated monthly costs for models with valid cost data in the shortlist of optimal models, respectively; when Alternatively, if the model lacks effective cost data, set the model's cost advantage score to a preset neutral value of 0.5.

Images