Language model reasoning-oriented multi-model routing method, device and system
By identifying the type and estimating the difficulty of user requests, automatically selecting the language model with the lowest overhead, and enabling a deep thinking mechanism, the problems of high cost, high latency, and poor stability in multi-model routing scenarios are solved, achieving cost minimization and improved response speed, while ensuring service reliability and scalability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2025-12-01
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies suffer from high computational costs, long response latency, and difficulty in ensuring stability when processing user requests. In particular, they struggle to balance task diversity, dynamic load, and global optimization of accuracy and cost in multi-model orchestration/routing scenarios.
By identifying the type and estimating the difficulty of user requests, jointly evaluating the overhead and predicting the success rate, the system automatically selects the language model with the lowest overhead and determines whether to enable the deep thinking mechanism based on the probability of correct answering in question-answering and coding tasks, thereby minimizing reasoning costs and latency.
It significantly reduces average token consumption and latency, balances accuracy and throughput, provides a full-chain engineering solution, facilitates integration and operation and maintenance, and ensures stable service capabilities under abnormal conditions.
Smart Images
Figure CN122021864A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a multi-model routing method, apparatus, and system for language model inference, belonging to the field of artificial intelligence natural language processing technology, and applied to user request model routing and traffic splitting scenarios. Background Technology
[0002] With the widespread application of Large Language Models (LLMs) in tasks such as dialogue, content creation, subject-specific reasoning, and code generation, the problems of high computational costs, long response times, and difficulty in guaranteeing stability caused by processing requests with a single model are becoming increasingly prominent. The industry is gradually shifting towards a multi-model orchestration / routing paradigm, which involves deploying models of different capabilities and scales on the same inference platform and using strategies to distribute requests, aiming to complete the task with the least costly model capable of solving the problem. Existing solutions often rely on heuristic rules or manually preset thresholds, making it difficult to simultaneously address task diversity, dynamic load, and the global optimization of accuracy and cost.
[0003] Therefore, how to select the language model with the lowest overhead rate based on the difficulty of user requests and the model's capabilities has become an urgent problem to be solved. Summary of the Invention
[0004] The purpose of this invention is to address the technical problem of selecting the language model with the lowest overhead rate based on the difficulty of user requests and model capabilities. It proposes a multi-model routing method, apparatus, and system for language model inference. By identifying the type and estimating the difficulty of user requests, and jointly evaluating overhead and predicting success rates, the system automatically selects the lowest-overhead model capable of resolving the request. Furthermore, in question-answering and coding tasks, it determines whether to enable a deep thinking mechanism based on the probability of a correct answer, thereby significantly reducing inference costs and latency while ensuring quality.
[0005] The objective of this invention is achieved through the following technical solution:
[0006] On one hand, the present invention provides a multi-model routing method for language model inference, applied to a system comprising one server and multiple clients, wherein the server is used to deploy the multi-model routing method; and the clients are used to deploy a large language model; comprising the following steps:
[0007] Step 1: Deploy on both the server and client sides;
[0008] Step 1.1: Deploy the request task classifier on the server;
[0009] Step 1.2: Deploy the candidate model and difficulty estimator on the client;
[0010] Step 1.2.1: Select the casual conversation model, creative writing model, subject-based question-and-answer model, and code generation model as candidate models. Among them, the subject-based question-and-answer model and the code generation model have deep thinking capabilities.
[0011] Step 1.2.2: Configure the difficulty estimator for the subject-specific answering model and the code generation model;
[0012] Step 2: The server receives the task requests sent by the user and classifies the task requests using a trained request task classifier.
[0013] Step 2.1: Construct a request task classifier that uses Qwen3-8B-base as the semantic encoder and employs a multilayer perceptron to classify the fixed-dimensional semantic vectors mapped by the semantic encoder.
[0014] Step 2.1.1: Label the task type of the user request data, use the user request data and request task as supervision samples, freeze the Qwen3-8B-base semantic encoder, and use a multilayer perceptron to classify the fixed-dimensional semantic vector mapped by the semantic encoder.
[0015] Step 2.1.2: Train the request task classifier using cross-entropy loss and AdamW optimizer to obtain the trained request task classifier;
[0016] Step 2.2: Classify task requests using the trained request task classifier;
[0017] Step 3: Set a confidence threshold, and route user requests to the corresponding candidate models based on the comparison between the task classification probability and the confidence threshold;
[0018] Step 4: When the candidate model is a subject-based question-answering model or a code generation model, a semantic vector is generated by the model encoding request; the semantic vector is input into the trained difficulty estimator; the decision to enable deep thinking is made based on the task answer probability obtained by the trained difficulty estimator.
[0019] Step 4.1: Construct a difficulty estimator that uses a subject-based question-answering model or code generation model as a semantic encoder and employs a multilayer perceptron to estimate the difficulty of the fixed-dimensional semantic vector mapped by the semantic encoder;
[0020] Step 4.1.1: Without enabling deep thinking, label the correctness of the response to the user request data, use the user request data and the correctness of the response as supervision samples, freeze the semantic encoder of the subject answering model or code generation model, and use a multilayer perceptron to estimate the difficulty of the fixed-dimensional semantic vector mapped by the semantic encoder.
[0021] Step 4.1.2: Train the request task classifier using cross-entropy loss and the AdamW optimizer to obtain a trained difficulty estimator;
[0022] Step 4.2: Input the semantic vector into the trained difficulty estimator to obtain the probability P_correct of correctly answering the task without enabling in-depth thinking;
[0023] Step 4.3: Set a constraint threshold T. When P_correct < T, enable in-depth thinking;
[0024] Step 5: Call the corresponding model according to the task classification and the determination of enabling in-depth thinking to respond to the user request, and return the model request to the user;
[0025] On the other hand, to achieve the object of the present invention, based on the above method, the present invention further proposes a multi-model routing system for language model reasoning, including a request access and preprocessing module, a task classification module, and a user request routing and response module;
[0026] The request access and preprocessing module is used for user request compliance identification and takes the compliant user request as the input of the task classification module;
[0027] The task classification module includes a semantic encoder and a task classifier, and is used for task classification of user requests; and takes it as the input of the user request routing and response module;
[0028] Further, the semantic encoder is used for vector encoding of user requests; and takes it as the input of the task classifier;
[0029] Further, the task classifier is used for task classification of the vector encoding of user requests; and takes the task classification result as the output of the task classification module;
[0030] The user request routing and response module includes a candidate model and a difficulty estimator, and is used for determining the enabling status of in-depth thinking and responding to user requests; returns the response of the user request to the user and takes it as the output of the system;
[0031] A multi-model routing device for language model reasoning disclosed by the present invention is used to implement the multi-model routing method for language model reasoning described above. A multi-model routing device for language model reasoning disclosed by the present invention includes a client and a server;
[0032] The client consists of a high-performance parallel computing processing unit for determining the enabling status of in-depth thinking and the model to respond to user requests and a communication interface for returning the response of the user request, and realizes the determination of the enabling status of the in-depth thinking model and the model to respond to user requests and return the response result;
[0033] The server comprises a communication interface for receiving user requests and a high-performance parallel computing processing unit for task classification, thereby enabling the classification of user-requested tasks.
[0034] Compared with existing technologies, it has the following beneficial effects:
[0035] 1. Cost minimization: By jointly optimizing the trainable difficulty estimation and the cost function, the "minimum cost model that can solve the problem" is selected, which significantly reduces the average token consumption and latency.
[0036] 2. Adjustable quality and cost: In the question-answering / coding scenario, deep thinking is started and stopped with the probability of answering correctly as the core indicator, taking into account both accuracy and throughput.
[0037] 3. Practical for implementation: Provides a complete engineering solution from type identification and difficulty assessment to route execution, rollback and adaptive learning, which is easy to integrate and maintain.
[0038] 4. Security and Reliability: Introducing result verification and tiered rollback to ensure steady-state service capability under abnormal conditions.
[0039] 5. Scalable: Model pool, threshold, cost weights and strategies are all configurable, making it easy to quickly integrate new models or adapt to new scenarios. Attached Figure Description
[0040] Figure 1 This is a flowchart illustrating the present invention;
[0041] Figure 2 This is a schematic diagram of the device of the present invention. Detailed Implementation
[0042] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.
[0043] Example
[0044] like Figure 1 As shown, in one aspect, a multi-model routing method for language model inference in this embodiment is applied to a system comprising one server and multiple clients, wherein the server is used to deploy the multi-model routing method; and the clients are used to deploy a large language model; the specific implementation steps are as follows:
[0045] Step 1: Deploy on both the server and client sides;
[0046] Step 1.1: Deploy the request task classifier on the server;
[0047] Step 1.2: Deploy the candidate model and difficulty estimator on the client;
[0048] Step 1.2.1: Select the casual conversation model, creative writing model, subject-based question-and-answer model, and code generation model as candidate models. Among them, the subject-based question-and-answer model and the code generation model have deep thinking capabilities.
[0049] Step 1.2.2: Configure the difficulty estimator for the subject-specific answering model and the code generation model;
[0050] In this embodiment, a set of large language models with different capabilities are deployed on the client, including but not limited to: a large model for casual conversation (e.g., GPT-4o), a large model for creative writing (e.g., GPT-4.5), a large model for subject-specific question answering (e.g., Qwen3-30B-A3B-Thinking-2507), and a large model for code generation (e.g., Qwen3-Coder-30B-A3B-Instruct). The subject-specific question answering model and the code generation model both support the optional enabling of a deep thinking mode.
[0051] Step 2: The server receives the task requests sent by the user and classifies the task requests using a trained request task classifier.
[0052] Step 2.1: Construct a request task classifier that uses Qwen3-8B-base as the semantic encoder and employs a multilayer perceptron to classify the fixed-dimensional semantic vectors mapped by the semantic encoder.
[0053] Step 2.1.1: Label the task type of the user request data, use the user request data and request task as supervision samples, freeze the Qwen3-8B-base semantic encoder, and use a multilayer perceptron to classify the fixed-dimensional semantic vector mapped by the semantic encoder.
[0054] Step 2.1.2: Train the request task classifier using cross-entropy loss and AdamW optimizer to obtain the trained request task classifier;
[0055] Step 2.2: Classify task requests using the trained request task classifier;
[0056] In this embodiment, the task type of user request data is collected and labeled. Qwen3-8B-base is selected as the semantic encoder, and each text is mapped to a semantic vector h∈R^d of fixed dimension (d can be 1024 / 2048, etc.). A multilayer perceptron (MLP) is connected on the vector to complete four classification categories: casual conversation, creative writing, subject-based question answering, and code generation. With the goal of minimizing cross-entropy loss, AdamW is used for optimization, and early stopping and class imbalance reweighting training are performed.
[0057] Step 3: Set a confidence threshold, and route user requests to the corresponding candidate models based on the comparison between the task classification probability and the confidence threshold;
[0058] In this embodiment, a threshold and confidence level are compared. If the probability of the largest class is lower than the set threshold, the route is routed to the casual conversation consultation model.
[0059] Step 4: When the candidate model is a subject-based question-answering model or a code generation model, a semantic vector is generated by the model encoding request; the semantic vector is input into the trained difficulty estimator; the decision to enable deep thinking is made based on the task answer probability obtained by the trained difficulty estimator.
[0060] Step 4.1: Construct a difficulty estimator that uses a subject-based question-answering model or code generation model as a semantic encoder and employs a multilayer perceptron to estimate the difficulty of the fixed-dimensional semantic vector mapped by the semantic encoder;
[0061] Step 4.1.1: Without enabling deep thinking, label the correctness of the response to the user request data, use the user request data and the correctness of the response as supervision samples, freeze the semantic encoder of the subject answering model or code generation model, and use a multilayer perceptron to estimate the difficulty of the fixed-dimensional semantic vector mapped by the semantic encoder.
[0062] Step 4.1.2: Train the request task classifier using cross-entropy loss and AdamW optimizer to obtain the trained difficulty estimator;
[0063] In this embodiment, the correctness of the model's response to user requests is collected and labeled. A subject-based question-answering model or a code generation model is selected as the semantic encoder. Each text is mapped to a semantic vector h∈R^d (d is the encoder model dimension). A multilayer perceptron (MLP) is externally connected to the vector to complete the binary classification task. AdamW is used for optimization with the goal of minimizing cross-entropy loss, and early stopping is performed.
[0064] Step 4.2: Input the semantic vector into the trained difficulty estimator to obtain the probability of answering the task correctly without enabling deep thinking, P_correct.
[0065] Step 4.3: Set the constraint threshold T. When P_correct < T, enable in-depth thinking;
[0066] Step 5: According to the task classification and the determination of enabling in-depth thinking, call the corresponding model to respond to the user request, and return the model request to the user;
[0067] On the other hand, as Figure 2 shown, to achieve the purpose of the present invention, according to the above method, this embodiment further proposes a multi-model routing system for language model reasoning, including a request access and preprocessing module, a task classification module, and a user request routing and response module;
[0068] The request access and preprocessing module is used for user request compliance identification, and takes the user request after compliance identification as the input of the task classification module;
[0069] The task classification module includes a semantic encoder and a task classifier, and is used for task classification of user requests; and takes it as the input of the user request routing and response module;
[0070] Further, the semantic encoder is used for vector encoding of user requests; and takes it as the input of the task classifier;
[0071] Further, the task classifier is used for task classification of user request vector encoding; and takes the task classification result as the output of the task classification module;
[0072] The user request routing and response module includes a candidate model and a difficulty estimator, and is used for determining the enabling status of in-depth thinking and responding to user requests; returns the response of the user request to the user and takes it as the output of the system;
[0073] A multi-model routing device for language model reasoning in this embodiment is used to implement the multi-model routing method for language model reasoning described above. A multi-model routing device for language model reasoning in this embodiment includes a client and a server;
[0074] The client consists of a high-performance parallel computing processing unit for determining the enabling status of in-depth thinking and the model responding to user requests and a communication interface for returning the user request response, and realizes the determination of the enabling status of the in-depth thinking model and the model responding to user requests and returning the response result;
[0075] The server consists of a communication interface for receiving user requests and a high-performance parallel computing processing unit for task classification, and realizes task classification of user requests.
Claims
1. A multi-model routing method for language model inference, applied to a system comprising one server and multiple clients, wherein... The server is used to deploy a multi-model routing method; The client is used to deploy a large language model, which is characterized by the following steps: Step 1: Deploy on the server and the client respectively; Step 2: The server receives the task request sent by the user and classifies the task request by using the trained request task classifier; Step 2.1: Construct a request task classifier that uses Qwen3-8B-base as the semantic encoder and uses a multi-layer perceptron to classify the fixed-dimensional semantic vectors mapped by the semantic encoder; Step 2.2: Classify the task request by using the trained request task classifier; Step 3: Set a confidence threshold, compare the task classification probability with the confidence threshold, and route the user request to the corresponding candidate model; Step 4: When the candidate model is a subject answering model or a code generation model, generate a semantic vector from the model encoding request; input the semantic vector into the trained difficulty estimator; determine whether to start deep thinking according to the task correct answer probability obtained by the trained difficulty estimator; Step 4.1: Construct a difficulty estimator that uses a subject answering model or a code generation model as the semantic encoder and uses a multi-layer perceptron to estimate the difficulty of the fixed-dimensional semantic vectors mapped by the semantic encoder; Step 4.2: Input the semantic vector into the trained difficulty estimator to obtain the task correct answer probability P_correct without starting deep thinking; Step 4.3: Set a constraint threshold T, and enable deep thinking when P_correct < T; Step 5: Call the corresponding model to respond to the user request according to the task classification and the determination of starting deep thinking, and return the model request to the user.
2. The multi-model routing method for language model inference as described in claim 1, characterized in that: The implementation method of Step 1 is as follows: Step 1.1: Deploy the request task classifier on the server; Step 1.2: Deploy the candidate model and the difficulty estimator on the client.
3. The multi-model routing method for language model inference as described in claim 2, characterized in that: The implementation method of Step 1.2 is as follows: Step 1.2.1: Use the chatting consultation model, creative writing model, subject answering model, and code generation model as candidate models, where the subject answering model and the code generation model have the ability of deep thinking; Step 1.2.2: Configure a difficulty estimator for the subject answering model and the code generation model.
4. The multi-model routing method for language model inference as described in claim 1, characterized in that: The implementation method of Step 2.1 is as follows: Step 2.1.1: Label the task type of the user request data, use the user request data and the request task as supervised samples, freeze the Qwen3-8B-base semantic encoder, and use a multi-layer perceptron to classify the fixed-dimensional semantic vectors mapped by the semantic encoder; Step 2.1.2: Train the request task classifier by using the cross-entropy loss and the AdamW optimizer to obtain the trained request task classifier.
5. The multi-model routing method for language model inference as described in claim 1, characterized in that: The implementation method of Step 4.1 is as follows: Step 4.1.1: Under the condition of not starting deep thinking, label the response correctness of the user request data, use the user request data and the response correctness as supervised samples, freeze the semantic encoder of the subject answering model or the code generation model, and use a multi-layer perceptron to estimate the difficulty of the fixed-dimensional semantic vectors mapped by the semantic encoder; Step 4.1.2: Train the request task classifier using cross-entropy loss and AdamW optimizer to obtain the trained difficulty estimator.
6. A multi-model routing system for language model inference, implementing the method as described in claim 1, characterized in that: This includes a request access and preprocessing module, a task classification module, and a user request routing and response module; The request access and preprocessing module is used for user request compliance authentication, and uses the user requests for compliance authentication as input to the task classification module. The task classification module includes a semantic encoder and a task classifier, used to classify user requests into tasks. It will serve as input to the user request routing and response module; The user request routing and response module includes a candidate model and a difficulty estimator, used to determine the activation status of deep thinking and respond to user requests; the response to the user request is returned to the user and used as the output of the system.
7. A multi-model routing system for language model inference as described in claim 6, characterized in that: The semantic encoder is used to vectorize user requests and will serve as input to the task classifier. The task classifier is used to classify user request vector encodings into tasks. The task classification results are used as the output of the task classification module.
8. A multi-model routing device for language model inference, characterized in that: Includes both client and server; The client consists of a high-performance parallel computing processing unit that performs the deep thinking activation status determination and model response to user requests, and a communication interface that returns the user request response, thereby realizing the determination of the deep thinking model activation status and the model response to user requests and the return of response results. The server comprises a communication interface for receiving user requests and a high-performance parallel computing processing unit for task classification, thereby enabling the classification of user-requested tasks.