Agent cascading routing and self-correction method based on bert low-cost large language model (LLM)

Abstract

The application discloses a kind of low-cost large language model (LLM) cascade routing and self-repairing algorithm based on BERT, belong to natural language processing and large language model application technical field.The method includes the following steps: constructing the few-sample demonstration set containing query text and corresponding label, for the query of user, first use sparse retriever to retrieve and similar example as context learning and construct prompt word as question;Obtain the initial answer of the query, a paragraph of explanation and self-statement confidence of large language model with smaller parameter quantity;Query, initial answer, explanation and self-statement confidence are spliced into trajectory sequence, input BERT model for encoding and output action probability distribution;According to probability distribution, select to execute "accept", "retry" or "upgrade" action: if accept, directly output weak model answer;If retry, construct enhanced prompt word to guide weak model self-repairing;If upgrade, call stronger large language model to generate answer.The effect of the algorithm is: for the problem of high cost of commercial large language model API call, by constructing the collaborative system of low-cost weak model and high-cost strong model, the self-repairing potential of weak model is deeply mined using intelligent decision mechanism, thereby unnecessary call to expensive strong model is significantly reduced, and the best balance between reasoning cost and task accuracy is realized under strict budget constraint.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, specifically relating to a method for using deep learning to make cascaded routing decisions for large language models (LLM), reduce inference costs, and improve model response accuracy. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, large language models have achieved revolutionary breakthroughs in the field of natural language processing. Advanced models such as GPT-4, Claude 3, and Llama 3, with their parameter scales of hundreds of billions or even trillions, have demonstrated astonishing capabilities in language understanding, generation, reasoning, and knowledge generalization. These models have been widely applied in numerous commercial scenarios, including intelligent customer service, code generation, document summarization, and machine translation. However, the performance improvement of large language models mainly follows the "scaling law," meaning that the growth of model capabilities often depends on an exponential increase in computational load, parameter count, and data volume. This expansion in scale brings high operating costs and significant inference latency. In commercial applications, especially services that rely on commercial APIs, the cost of invocation is usually proportional to the model's capabilities. For example, the API call cost of a top-tier LLM can be tens or even hundreds of times that of a lightweight LLM. For enterprises that need to handle massive user requests, relying entirely on strong LLMs for inference would result in an unbearable economic burden; while using only low-cost weak LLMs, although with fast response times and low costs, often exhibits insufficient accuracy and frequent illusions when handling complex logic, long text understanding, or knowledge-intensive tasks. Therefore, how to minimize inference overhead while ensuring task processing quality has become a core challenge that urgently needs to be addressed in the commercialization of large language models. To address the aforementioned cost-performance trade-off, industry and academia have proposed solutions such as "ModelCascades" or "Model Routing." The core idea of ​​this technology is to build a pool of models with different capabilities and cost levels (typically including a low-cost weak LLM and a high-cost strong LLM), and dynamically assign user queries to different models through a routing strategy. Ideally, simple queries are handled by the weak model, and only when a difficult query is encountered that the weak model cannot handle is the strong model "upgraded" to be called. Current model cascading routing technologies are mainly divided into the following categories: (1) Threshold routing based on confidence or rating regression: This is currently the most mainstream routing strategy. This type of method usually trains an auxiliary model (such as BERT or a lightweight classifier) ​​as a scorer, or directly uses the output confidence of the weak model itself as the basis. The system will pre-set a scalar threshold. When the predicted answer quality score or confidence is lower than the threshold, the query is routed to the strong model; otherwise, the answer from the weak model is accepted. (2) Consistency-based routing: This type of method utilizes the principle of "self-consistency" by sampling the same query multiple times (e.g., using different Dropout or temperature parameters) to observe the consistency of the weak model's output. If the answers obtained from multiple samplings are highly consistent, the answer is considered reliable; if there is a large discrepancy, the weak model is considered questionable and needs to be upgraded to a strong model. While the aforementioned methods reduce costs to some extent, significant technical bottlenecks and shortcomings remain in practical applications. Confidence-based routing heavily relies on the weak model's ability to evaluate its own answers. However, research shows that existing weak language models generally suffer from severe "overconfidence," meaning that even when generating incorrect answers, the model often assigns extremely high confidence scores. Furthermore, the illusion created by weak models is deceptive; the generated incorrect explanations are often fluent and logically sound, making it difficult for simple scorers to distinguish between true and false. This calibration bias causes routers to fail to accurately identify errors, thus missing incorrect answers for users and severely impacting system reliability. Regression-based scoring methods simplify complex routing decisions to a scalar threshold determination. Determining this "optimal threshold" is extremely challenging in practical deployments. Thresholds are typically very sensitive to data distribution; a threshold optimized on one dataset often fails to generalize to another. While consistency-based routing methods offer improved accuracy, they come at the cost of significant computational overhead. Calculating consistency scores typically requires multiple inference samples of the weak model, which defeats the purpose of model cascading to "reduce costs." Existing routing systems are typically binary decision systems: either "accept" the response from a weak model or "discard" it and upgrade to a strong model. This mechanism treats the weak model as a static generator, immediately switching to the expensive strong model once the initial response is deemed unreliable. However, research shows that weak models often possess the potential for "self-correction." In many cases, the initial error of a weak model is not due to incompetence, but rather to misunderstanding or carelessness. Given appropriate guidance or opportunities for reflection, weak models are fully capable of correcting their errors. Existing direct upgrade strategies ignore this intermediate state, leading to numerous unnecessary calls to the strong model and wasting computational and financial resources. When leveraging few-shot learning to improve model performance, existing retrieval augmentation methods (such as Top-K TF-IDF or simple cosine similarity retrieval) tend to retrieve the most semantically similar examples. However, in classification tasks, this often results in retrieved examples with single labels, lacking label diversity. The lack of contrasting "hard negatives" makes it difficult for the model to learn clear classification boundaries in context, thus limiting the performance of weak models on complex classification tasks. In summary, existing model routing techniques are mainly limited by their reliance on scalar thresholds, neglect of the self-correcting potential of weak models, and high verification costs. Therefore, there is an urgent need for a novel routing paradigm that can treat routing as a continuous, discrete decision-making process, not only judging "right" or "wrong," but also guiding the model to perform low-cost "self-correction" when potential errors are discovered, and achieving a globally optimal cost-performance tradeoff under strict budget constraints. Summary of the Invention

[0003] The primary objective of this invention is to resolve the irreconcilable contradiction between inference cost and task performance in existing Large Language Model (LLM) applications. Addressing the technical pain points of current model cascading and routing techniques, such as overconfidence in weak models leading to false negatives, difficulty in adapting scalar thresholds to dynamic environments, and the waste of expensive strong model resources by ignoring the self-correction potential of weak models, this invention provides a low-cost BERT-based method for large language model cascading routing and self-correction. Another objective of this invention is to provide a few-sample demonstration construction mechanism (Query-Label MMR) for classification tasks, which aims to solve the problem that traditional retrieval methods only focus on query similarity and ignore the diversity of label distribution, resulting in a lack of discriminative power in contextual examples. The ultimate goal of this invention is to build an end-to-end, cost-aware inference system with intelligent decision-making capabilities. Through refined decision-making in the discrete action space (accept, retry, upgrade), it maximizes the accuracy of downstream natural language processing tasks while strictly adhering to the user's preset inference budget. A low-cost, cascaded routing and self-correction algorithm for large language models based on BERT includes the following steps: (1) Obtain the query request input by the user, call the low-cost weak LLM to process the query request, and obtain the initial answer, reasoning explanation and self-status confidence. (2) The query request, initial answer, reasoning explanation and self-statistic confidence are concatenated and a classification identifier is added to construct the trajectory sequence. (3) Input the trajectory sequence into the fine-tuned BERT model for encoding, and obtain the probability distribution on the discrete action set through the classification head mapping. The discrete action set includes: accept, retry and upgrade. (4) Sample or select the action with the highest probability according to the probability distribution and perform the corresponding operation: If the action is to accept, output the initial answer directly as the final result; If the action is to retry, construct an enhanced prompt word containing the initial answer and reasoning explanation (indicating that the initial answer may be wrong), and call the weak LLM again to generate the corrected answer as the final result; If the action is to escalate, construct a prompt word and call the strong LLM with higher cost to generate the answer as the final result. The above steps are characterized by: In step (1), before calling the weak language model, the method also includes the step of constructing few-sample prompt words using the Query-Label MMR sparse retrieval algorithm: Construct a demo set containing historical query texts and corresponding answers, and build an index using TF-IDF; calculate the cosine similarity between the current query request and the samples in the demo set, and initially screen the candidate set; use the maximum boundary correlation algorithm (MMR) to iteratively select samples from the candidate set, with the selection criteria being to maximize the similarity between the sample and the query while minimizing the similarity between the selected samples, and to consider both the query text and the answer text when calculating the similarity; add the selected samples to the prompt words in the format of "input-answer" pairs. The trajectory sequence described in step (2) comprehensively considers the quality of the answer to the question, the degree of explanation, and the self-status confidence, so as to more comprehensively evaluate the weak LLM's understanding of the question and the reliability of the answer, and thus make a more appropriate intelligent decision. In step (4), when the action is to retry, the specific steps for constructing the enhanced prompt include: concatenating the original query request, the initial answer of the weak LLM, and the reasoning explanation to construct a feedback prompt containing a self-reflection instruction; the feedback prompt explicitly instructs the weak LLM to use its previously generated explanation as a reference context, indicating that its initial answer is likely to be wrong, guiding it to critically review and diagnose its own reasoning logic, and then perform self-correcting reasoning to generate a corrected answer. The fine-tuning training of the BERT model employs a cost-aware and entropy-regularized objective function. Defined as: ,in Rewards for answering the task. For reasoning cost, Let the probability entropy of the policy be . This is the cost control coefficient. is the entropy regularization coefficient. This objective function, while jointly optimizing task performance and inference cost, utilizes the entropy regularization term to promote diversified policy exploration, effectively preventing policy collapse, that is, avoiding the model from getting trapped in local optima and only performing a single action, thereby ensuring the effectiveness of the routing mechanism. The cost control coefficient During training, the update is adaptively performed based on budget constraints, and the update rule is as follows: in This represents the training round, and η is the update step size. B is the current average inference cost, and B is the preset target budget. This indicates that a non-negative value is taken. The expected reward Positive rewards or negative penalties are assigned based on the correctness of the model's final output answer; the expected inference cost Calculated based on the model type and number of calls for different actions. The concept of this invention is as follows: (1) Design concept of model input This invention selects the complete trajectory, including both "inference explanation" and "self-stated confidence," as the input to BERT. Since most existing routing methods rely solely on confidence scores or simple answer consistency, these scalar signals often fail to distinguish between "overconfidence illusions" and "careless errors." To minimize reliance on the internal parameters of the black-box model, this invention requires the weak model to output an inference explanation and then uses the BERT model to interpret this explanation. If the explanation is logically flawed or inconsistent with the question, BERT can capture the anomaly at the semantic level, which is more robust than simply relying on numerical features. (2) Design concept of model structure This invention, based on the "Agent" concept in deep learning, designs a discrete action decision model. Existing models are mostly regression models (predicting a quality score), requiring manually set thresholds and unable to distinguish between "need to retry" and "need to upgrade" using a single threshold. To achieve more refined control, this invention models routing as action selection, with BERT acting as an agent-like entity, directly outputting discrete actions. This invention specifically designs a "retry" action branch. Traditional cascaded systems are binary, ignoring the self-correcting ability of weak models under prompting. By identifying errors with "correction potential" (such as calculation errors or missing instructions) through BERT, the system can solve problems at extremely low cost without the intervention of expensive strong models. This is a high-performance, cost-effective intermediate state between "acceptance" and "upgrade." (3) Design ideas for optimizing the objective function This invention designs a cost-aware optimization objective based on the real-world constraints of commercial API calls. Existing methods typically only optimize accuracy or treat cost as a fixed hyperparameter. This invention models the budget constraint as a hard constraint and transforms it into an unconstrained optimization problem using Lagrange duality. This means that users can directly specify the budget, and the trained model will automatically adjust its policy to fit that budget. Furthermore, to avoid the policy collapse problem common in reinforcement learning (i.e., the model always chooses the most expensive path for safety or the worst path for cost-saving), this invention introduces the idea of ​​Maximum Entropy Reinforcement Learning (Maximum Entropy RL), adding an entropy regularization term to the objective function to encourage the model to explore diverse routing strategies, thereby finding the optimal solution for cost and performance globally. Attached Figure Description Figure 1 The flowchart of this invention Figure 2 Flowchart built for few-sample demonstration Figure 3 Flowchart of the objective function Detailed Implementation A low-cost, cascaded routing and self-correction method for large language models based on BERT, the overall framework of which is as follows: Figure 1 As shown in the figure. This embodiment uses a classification task in natural language processing as an example for illustration. (1) Dataset preprocessing and construction of few-sample examples This method requires constructing three datasets: a training set for fine-tuning BERT and making routing decisions within a given budget; a test set for testing the model's performance; and a demonstration set for retrieving suitable samples to construct a few-shot example dataset. (1.1) Construction of few-sample examples Before using a large language model for reasoning, it is necessary to first construct high-quality few-shot cue words. To address the problem that traditional retrieval methods, which rely solely on query similarity, result in single, undifferentiated example labels, this embodiment employs the proposed "Query-Label Maximum Boundary Relevance (Query-Label MMR)" sparse retrieval algorithm. The algorithm flow is as follows: Figure 2 As shown. Let the demonstration set be ,in For querying text, The corresponding real labels. To capture both semantic and label information during the retrieval phase, this embodiment concatenates the query and labels into text. Specifically, each sample undergoes word segmentation to construct the joint text. The TF-IDF vectorizer is used to map the joint text into sparse high-dimensional vectors. This leads to the construction of a vector index library containing all demonstration samples. For the new query to be reasoned upon input by the user First, calculate its corresponding embedding vector. .calculate Compared with all sample vectors in the index library Cosine similarity between them: (1-1) Score based on similarity Sort the samples from highest to lowest quality and select the top M samples to form a candidate pool. To select a final set of examples from the candidate pool that is both relevant and has diverse labels An iterative greedy strategy is employed. In each iteration, a sample is selected. join in The selection criterion is to maximize the following MMR scores: (1-2) The first item measures the relevance of the candidate sample to the current query, and the second item measures the redundancy (i.e., similarity) between the candidate sample and the selected sample set. To adjust the parameters, repeat this step until the desired result is selected. Examples. The selected ones. The examples are arranged in the format "Input: [Query]\nTag: [Tag]" and system instructions are added to form the final prompt words for the weak model. (2) Model training Using the Prompt constructed above, each query in the training set... Input to weak LLM Pre-inference is performed. The weak model is required to output a trajectory containing the following three elements: initial answer. Reasoning and Explanation Confidence level of self-report For each sample in the training set, we obtained a quadruple dataset: (2.1) BERT Input Layer Construction This invention constructs an action decision model based on the BERT architecture. In order for BERT to understand the reasoning state of the weak model, the above quadruple needs to be serialized. Constructing the trajectory sequence τ :τ ⊕ ⊕ ⊕ ⊕ ⊕ (2-1) in For category embedding bits, This serves as the field separator. This design allows the model to simultaneously encode "question difficulty," "response quality," and "explanation logic." (2.2) Action output layer The output of the BERT encoder vector The input is fed into a fully connected layer and mapped to a discrete action space via Softmax. probability distribution (2.3) BERT model training This invention utilizes reinforcement learning to optimize performance, aiming to maximize expected rewards under budget constraints. Figure 3 As shown. reward function If the action leads to the correct final answer Give positive rewards If an error is made, a negative reward will be given. Cost function Based on the actual API pricing. Among them, The selected action. The training objective is to maximize the following function : (2-2) in, It is the expected accuracy under the strategy; It is the expected consumption under the strategy; It is the action probability distribution entropy, used to prevent policy collapse, that is, to prevent the model from choosing only a single action in order to save money or pursue accuracy to the extreme. To strictly meet the user's preset average budget This embodiment introduces the Lagrange multiplier method for dynamic adjustment. At the end of each training epoch, calculate the average cost of the current model on the training set. And update the coefficients according to the following rules: (2-3) If the current model "overspends" ( If λ increases, it increases the penalty for high-cost actions, forcing the model to be more inclined to ACCEPT or RETRY in the next round of training; conversely, it decreases λ, encouraging the model to use ESCALATE. Through the training process described above, Action BERT automatically learns how to operate within a given budget. Below, how to use semantic features (especially interpretation) This helps determine when to retry and when to upgrade. (3) Online reasoning When a new user query is received Using the same Query-Label MMR algorithm as in step (1.1), retrieve from the database A best-in-class demo case for building prompt words. Calling a weak LLM ,enter Weak LLM returns the initial answer. Reasoning and explanation and confidence level (3.1) BERT policy action selection Will spliced ​​into a trajectory sequence The sequence trajectory is then serialized. This sequence is input into the trained BERT to calculate the action probability distribution. And select the action with the highest probability. (3-2) Action branches according to Execute different logics to achieve a balance between cost and performance: Branch 1: Accept: BERT's explanation for the weak model is logically sound and its confidence level is reliable. Therefore, it returns directly. This action has the lowest cost and fastest response time, making it the optimal action when a weak LLM is sufficient to handle user queries. Branch 2: Retry: BERT determines that logical jumps or minor misunderstandings of instructions exist in its interpretation, constituting a "careless" error. Therefore, enhanced hints are constructed. The template is: "Question: Your answer: ,explain: This answer may be incorrect. Please reflect on the logic in the above explanation and answer again. Then, call the weak LLM again. To the revised answer This action corrected the error at a low cost (only one additional weak model call) and tapped into the potential capabilities of the weak model. Branch 3: ESCALATE: BERT determines that the problem falls into a knowledge blind spot of a weak LLM (such as severe hallucinations), and retrying is meaningless. Therefore, the query will be... Forward to strong model Get high-quality answers Although this action is costly, it ensures the system's accuracy in providing backup. (4) Model training parameters In the specific experimental implementation, this invention constructs a routing policy network based on the HuggingFace Transformers open-source library, and the model initialization weights use a bert-base-uncased pre-trained model. Considering the stability and efficiency of model training, the batch size is set to 32, and the total training epochs are set to 50. To prevent overfitting and avoid being limited to local optima, this embodiment uses the AdamW optimizer to update the network parameters, with an initial learning rate set to... In the optimization process, this invention introduces an entropy regularization term coefficient. Its value is set to 0.05 to encourage diverse strategy exploration; at the same time, an initial cost penalty coefficient is set. The value is 0.01. To dynamically adapt to budget constraints during training, this embodiment employs an adaptive cost coefficient update strategy, which dynamically adjusts the cost coefficient at the end of each training cycle based on the difference between the average inference cost of the current batch and the target budget. The step size η is set to 0.01. During the few-sample prompt word construction phase, the number of retrieved examples k is set to 4. Objective function of model training This invention integrates task accuracy rewards, inference cost penalties, and policy entropy to maximize expected returns. After all training cycles are completed, data from the test set is selected to evaluate the final trained routing policy. This invention selects three key metrics to comprehensively measure the system's performance: task accuracy, average inference cost, and performance gap recovery rate (PGR). To quantify economic benefits, this embodiment simulates API pricing based on a 1:10 cost ratio, setting a single-call cost for weak LLM. Strong model single call cost Total cost of a single query Calculated based on the action sequence selected by Action BERT. To facilitate standardized comparisons across tasks of varying difficulty, this invention employs Performance Gap Recovered (PGR) to measure the extent to which the routing system bridges the performance gap between weak and strong models. The calculation formula is shown in Equation (4-1): In the above formula, This indicates the overall system performance of the present invention. This indicates the performance using only the weak model. This indicates the performance (i.e., the performance ceiling) when using only the strong model. The higher the PGR value, the closer the routing system's performance is to the strong model while maintaining low cost.

Claims

1. A low-cost, cascaded routing and self-correction algorithm for large language models based on BERT, comprising the following steps: (1) Obtain the query request input by the user, call the low-cost weak LLM to process the query request, and obtain the initial answer, reasoning explanation and self-status confidence. (2) The query request, initial answer, reasoning explanation and self-statistic confidence are concatenated and a classification identifier is added to construct the trajectory sequence. (3) Input the trajectory sequence into the fine-tuned BERT model for encoding, and obtain the probability distribution on the discrete action set through the classification head mapping. The discrete action set includes: accept, retry and upgrade. (4) Sample or select the action with the highest probability according to the probability distribution and perform the corresponding operation: If the action is to accept, output the initial answer directly as the final result; If the action is to retry, construct an enhanced prompt word containing the initial answer and reasoning explanation (indicating that the initial answer may be wrong), and call the weak LLM again to generate the corrected answer as the final result; If the action is to escalate, construct a prompt word to call the strong LLM with higher cost to generate the answer as the final result.

2. The method according to claim 1, characterized in that, In step (1), before calling the weak language model, the method also includes the step of constructing few-sample prompt words using the Query-Label MMR sparse retrieval algorithm: constructing a demo set containing historical query texts and corresponding answers, and constructing an index using TF-IDF; calculating the cosine similarity between the current query request and the samples in the demo set, and initially screening the candidate set; using the maximum boundary correlation algorithm (MMR) to iteratively select samples from the candidate set, with the selection criteria being to maximize the similarity between the sample and the query while minimizing the similarity between the selected samples, and considering both the query text and the answer text when calculating the similarity; and adding the selected samples to the prompt words in the format of "input-answer" pairs.

3. The method according to claim 1, characterized in that, The trajectory sequence described in step (2) comprehensively considers the quality of the answer to the question, the degree of explanation, and the self-status confidence, so as to more comprehensively evaluate the weak LLM's understanding of the question and the reliability of the answer, and thus make a more appropriate intelligent decision.

4. The method according to claim 1, characterized in that, In step (4), when the action is to retry, the specific steps for constructing the enhanced prompt include: concatenating the original query request, the initial answer of the weak LLM, and the reasoning explanation to construct a feedback prompt containing a self-reflection instruction; the feedback prompt explicitly instructs the weak LLM to use its previously generated explanation as a reference context, indicating that its initial answer is likely to be wrong, guiding it to critically review and diagnose its own reasoning logic, and then perform self-correcting reasoning to generate a corrected answer.

5. The method according to claim 1, characterized in that, The fine-tuning training of the BERT model employs a cost-aware and entropy-regularized objective function. Defined as: J φ = E R φ x y - λ c o s t C φ x + α e n t H φ x ,in Rewards for answering the task. For reasoning cost, Let the probability entropy of the policy be . This is the cost control coefficient. is the entropy regularization coefficient. This objective function, while jointly optimizing task performance and inference cost, utilizes the entropy regularization term to promote diversified policy exploration, effectively preventing policy collapse, that is, avoiding the model from getting trapped in local optima and only performing a single action, thereby ensuring the effectiveness of the routing mechanism.

6. The method according to claim 5, characterized in that, The cost control coefficient During training, the update is adaptively performed based on budget constraints, and the update rule is as follows: in This represents the training round, and η is the update step size. B is the current average inference cost, and B is the preset target budget. This indicates that a non-negative value is taken.

7. The method according to claim 5, characterized in that, The expected reward Positive rewards or negative penalties are assigned based on the correctness of the model's final output answer; the expected inference cost Calculated based on the model type and number of calls for different actions.

Images