Dynamic routing method and device of large language model based on context learning model representation, readable storage medium

Abstract

The application relates to a large language model dynamic routing method and device based on a context learning model representation, a readable storage medium, a projection model is used to embed and map a query into a language model input space and realize semantic alignment, a representative evaluation set covering multi-dimensional capabilities is automatically screened from a benchmark question library, performance characteristics of the model on the evaluation set are efficiently obtained at one time, and a high-quality context model capability representation is formed; subsequently, real-time query embedding and the context model capability representation are combined, a lightweight routing language model is used for supervised learning, thereby realizing fine-grained model distinguishing capability, and an incremental embedding updating mechanism is designed, when a new model is accessed or an old model is upgraded, cold start can be quickly completed through only a small amount of fixed questions, calculation and maintenance costs are greatly reduced, and the accuracy, real-time performance and flexible expansibility of model routing are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of computer technology and natural language processing technology, and in particular to a dynamic routing method and device for large language models based on context learning model representation, as well as a readable storage medium. Background Technology

[0002] With the rapid expansion of the large-scale model ecosystem, the collaborative reasoning paradigm of "model pool + router" has become a key technical approach for the industry to reduce reasoning costs and improve answer quality: the system first builds capability representations for each candidate model, and then the router selects the best model from among the models in real time based on the user query. Current large language model routing methods are based on a three-step process of "query → router → optimal model". By building capability representations for the model pool and training (or designing) a query-aware selector, overall performance is significantly improved while keeping the cost of a single reasoning session controllable.

[0003] In existing work, the LLM-Blender ensemble framework combines pairwise comparison scoring with Top-k fusion, setting a precedent for driving online routing with offline capability assessment. However, the pre-computation cost increases quadratically with the model pool size because it requires comparing all models for each sample. The ZOOTER framework obtains labels that approximate human preferences through reward model distillation and then trains a lightweight router to achieve millisecond-level decision-making, significantly reducing inference latency. However, its upstream reward model still relies on a large amount of high-quality preference data, which is prone to performance degradation during domain migration. The RouterDC routing framework further introduces "dual contrastive learning," aligning both "sample-model" and "sample-sample" representations simultaneously. The EmbedLLM framework maintains a more stable discrimination on the external data, but this construction of positive and negative pairs still requires a large number of offline inference results, and the cold start pressure remains. The EmbedLLM framework compresses each model into a compact vector and concatenates it with the query vector to predict the accuracy. It has fewer parameters and the vectors can be stored incrementally directly, but it must run the benchmark question set for the entire pool model in advance to obtain high-quality embeddings. The ModelSAT framework attempts to get rid of pure vector representation. It first selects a batch of representative questions to obtain the model output, and then automatically generates natural language "ability instructions" covering multi-dimensional abilities for the micro-large language model to route and retrieve, which improves interpretability. However, it is still difficult to achieve accurate differentiation in subdivided skills due to the instruction template constructed by the fixed dataset.

[0004] In summary, current routing systems typically rely solely on embedding vectors generated from user queries to represent the model. However, queries often lack a comprehensive description of the model's output behavior, resulting in high-dimensional but sparse model embeddings that fail to reflect performance differences between models. Consequently, model selection can only be performed at a coarse-grained level. Furthermore, when faced with the integration of new models or the iteration of existing model versions, the industry commonly employs a method of re-inference on a massive and complete query set to generate embeddings. Since query sets often range from tens of thousands to hundreds of thousands of records, this process not only consumes significant computing resources but also significantly extends cold start time and iteration cycles, hindering the flexible expansion of the model pool and increasing operational costs. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing a dynamic routing method and device for large language models based on context learning model representation, as well as a readable storage medium, to solve or partially solve the problem of low efficiency and high computational resource consumption in routing user queries to the appropriate candidate model in scenarios with multiple candidate models deployed.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] One aspect of the present invention provides a dynamic routing method for large language models based on context learning model representations, comprising the following steps:

[0008] Query reconstruction pre-training: The projection model is used to map the input query embedding vector to a high-dimensional semantic space. The mapped embedding vector is then input into the model router. The projection model and the model router are trained with the goal of restoring the user query embedding vector.

[0009] Context-based learning-based model representation routing training: Based on a pre-built representative evaluation set with multiple difficulty levels, the query vectors in the representative evaluation set are injected into multiple candidate models using a trained projection model. For each candidate model, the correct and incorrect labels under each query vector are aggregated to obtain the context model capability representation. The context model capability representation and the user query are used as inputs, and the correct and incorrect labels corresponding to the query are used as supervision signals to train a model router based on a lightweight language model.

[0010] Online inference phase: User queries and the contextual model capability representations of each candidate model are used as inputs to the trained model router to obtain the adaptation scores of each candidate model. Dynamic routing of the large language model is then implemented based on the adaptation scores.

[0011] As a preferred technical solution, the process of constructing the representative evaluation set includes the following steps:

[0012] Based on the accuracy rate of the questions, the difficulty levels were divided into several equally wide ranges from extremely difficult to highly divergent. An equal number of questions were drawn from each range to construct a representative assessment set covering different difficulty levels.

[0013] As a preferred technical solution, the following are also included:

[0014] Incremental update process: In response to the access of new candidate models or the upgrading of existing candidate models, the new candidate models and / or upgraded candidate models are inferred on the representative evaluation set to obtain the corresponding context model capability representation.

[0015] As a preferred technical solution, based on average pooling, max pooling, weighted pooling, or attention mechanisms, the correct and incorrect labels and computational cost penalty terms under each query are aggregated to obtain the context model capability representation.

[0016] As a preferred technical solution, in the process of constructing the representative evaluation set, questions are selected based on the difficulty and information entropy of the questions, as well as the difference indicators between candidate models, so as to realize the representation of the model's ability by the questions.

[0017] As a preferred technical solution, the projection model achieves semantic alignment between the query embedding and the model input space through a linear or nonlinear neural network, and the model router is trained based on contrastive learning or reinforcement learning.

[0018] As a preferred technical solution, during the training process of model representation routing, the projection model and the model router are optimized synchronously by minimizing the reconstruction loss, thereby achieving semantic alignment between the projection model and the model router.

[0019] As a preferred technical solution, the high-dimensional semantic space is the same as the original input of the candidate model.

[0020] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned dynamic routing method for large language models based on context learning model representations.

[0021] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the aforementioned dynamic routing method for large language models based on context learning model representations.

[0022] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0023] (1) Achieving efficient user query routing under multiple candidate models: This invention uses a projection model to map the query embedding to a semantic space consistent with the language model input; then, by having the language model reconstruct the query embedding, the alignment of query semantics and model representation space is achieved. Subsequently, a small number of challenging questions covering multi-dimensional capabilities are automatically selected from the benchmark question bank, and the answer performance of each candidate model is collected at once on a representative evaluation set. The performance of the model on the representative evaluation set is compressed into a context model capability representation, and supervised learning is performed with the corresponding lightweight routing language model of the user query input. This allows the router to simultaneously grasp both query semantics and model performance information, thereby obtaining fine-grained discrimination capability. During the inference process, only one query encoding and one small language model inference are required to complete the model selection, ensuring a fast response. This invention maintains real-time performance while improving routing accuracy, shortening the deployment cycle, and can still stably distinguish model capabilities in cross-domain scenarios, providing a scalable and low-overhead dynamic routing solution for large-scale model pools.

[0024] (2) Low cold start and iteration costs: When a new model is accessed or an old model is upgraded, or when the model pool is expanded or an old model is iterated, the present invention can quickly generate new capability representations by performing reasoning once on a small number of evaluation questions of a fixed scale. There is no need to perform full reasoning and retrain the model router, which shortens the model cold start time and maintenance iteration cycle and significantly reduces operation and deployment costs.

[0025] (3) Low cost of building model capability representations: This invention automatically selects a small number of challenging problems from the benchmark question bank that cover different difficulty gradients and can fully reflect the differences in model performance, forming an efficient evaluation set, which significantly reduces the cost of acquiring capability representations and the cost of building model capability representations. Compared with the existing technology that requires offline reasoning or pairwise comparison on a large number of benchmark queries, this invention only requires a small number of representative questions to complete the generation of model capability representations, effectively reducing the demand for computing resources and time consumption.

[0026] (4) Improve the fine-grained recognition capability of routing decisions: This invention uses a projection model to map the user query embedding vector to the input semantic space of the language model, and reconstructs the query embedding through the language model to achieve accurate alignment of the two in the semantic space. This avoids the problem of high-dimensional but sparse query embedding in the prior art, making the model capability representation more accurate and compact, thereby significantly improving the fine-grained recognition capability of routing decisions.

[0027] (5) Strong robustness of domain transfer: This invention compresses the performance of candidate models on representative evaluation sets into contextual capability representations, and inputs them together with query vectors into a lightweight routing language model for supervised learning, thereby achieving fine-grained differentiation of model performance and improving the robustness of domain transfer. When applied across scenarios, it can more stably maintain the differentiation effect of model performance differences and enhance the adaptability of the system in real complex scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart of the dynamic routing method for a large language model based on context learning model representation in the embodiment;

[0029] Figure 2 This is a schematic diagram of the query reconstruction pre-training stage in the embodiment;

[0030] Figure 3 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] Example 1

[0033] To address the shortcomings of existing large-scale language model routing methods in terms of fine-grained recognition, cold-start time, and model pool expansion costs, this embodiment provides a dynamic routing method for large language models based on context-learning model representations. This method utilizes a projection model to map query embeddings to the language model input space and achieve semantic alignment. Then, it automatically selects representative evaluation sets covering multi-dimensional capabilities from a benchmark question bank, efficiently acquiring the model's performance features on the evaluation set in one go, forming a high-quality context model capability representation. Subsequently, combining real-time user queries and context model capability representations, a lightweight routing language model is used for supervised learning, thereby achieving fine-grained model discrimination capabilities. An incremental embedding update mechanism is designed so that when a new model is added or an old model is upgraded, a cold start can be quickly completed with only a very small number of fixed questions, significantly reducing computational and maintenance costs and effectively improving the accuracy, real-time performance, and flexible scalability of model routing. This method can be extended to other fields that require dynamic selection of optimal models or knowledge sources, such as recommendation systems, search engines, or knowledge question answering systems.

[0034] See Figure 1 The method mainly includes the following three stages:

[0035] S1, Query Reconstruction Pre-training Stage: The projection model is used to map the input query embedding vector to a high-dimensional semantic space. The mapped embedding vector is then input into the model router. The goal is to train the projection model and the model router to reconstruct the user query embedding vector.

[0036] For details, see Figure 2 The query reconstruction pre-training phase includes S101-S103:

[0037] S101, invoke the query embedding model to generate a vector representation of the user query. The query embedding model is implemented using an existing model.

[0038] S102. Considering that the user query generated vector is located in a general encoding space, it would be difficult to retain sufficient semantic information if it were directly processed by a language model. This step sets up a projection model to map the query vector to the same high-dimensional semantic space as the original input of the candidate model, thus obtaining the query embedding vector.

[0039] The projection model can employ linear projection or nonlinear neural networks, such as multilayer perceptrons, attention mechanisms, or recurrent neural networks, to achieve semantic alignment between the query embedding and the model input space.

[0040] In step S103, the mapped query embedding vector and the original user query are fed into the model router. To ensure the language model truly understands the semantics implied by the mapped vector, the model router is required to "reconstruct" the original user query vector at its output. During training, with the aim of minimizing the reconstruction loss, the projection model and the model router are simultaneously optimized to achieve a close semantic alignment. The model router is essentially a language model.

[0041] In the query reconstruction pre-training phase, given a query set First, the queries to be processed Input query embedding model in sequence With two fully connected projection models The model router is obtained. Vector representation of semantic space Subsequently, Only with As a condition, generate a response to the original query. Reconstruct the text and minimize the loss in the following formula. ,in Indicates the training sample number. This indicates that the model router is in a given vector Generating the original query The conditional probability, , express , Trainable parameters, This indicates that the average value is taken from all training samples.

[0042]

[0043] By optimizing the negative log-likelihood objective (i.e., the reconstruction loss), the projection model is constrained to establish a correspondence between the embedding space and the semantic subspace of the Large Language Model (LLM), while the LLM learns the semantic information carried by the parsed vectors during backpropagation. Thus, any query can be compressed into a high-density vector with inserted hints, significantly reducing the context length and laying a unified representation foundation for subsequent routing stages.

[0044] After completing the query reconstruction pre-training phase, a projection model that can map query vectors to the language model representation space is obtained, laying the foundation for subsequent training.

[0045] S2, Context-based model representation routing training phase: Based on a pre-built representative evaluation set with multiple difficulty levels, the query vectors in the representative evaluation set are injected into multiple candidate models using a trained projection model. For each candidate model, the correct and incorrect labels under each query vector are aggregated to obtain the context model capability representation. The context model capability representation and the query vector are used as inputs, and the correct and incorrect labels corresponding to the user query are used as supervision signals to train a model router based on a lightweight language model.

[0046] Specifically, the context-based learning model representation routing training phase includes steps S201-S203:

[0047] S201 first divides the difficulty into several equally wide intervals based on the accuracy of questions, ranging from the extremely difficult end ("only one model answers correctly") to the highly divergent end ("about half of the models answer correctly, with the highest information entropy"). Then, an equal number of questions are randomly selected from each interval to construct a representative evaluation set covering different difficulty levels. Next, using the query and model performance of each question as input, a query embedding model is used to obtain the query embedding vector corresponding to each question.

[0048] In the process of constructing the representative evaluation set, dynamic screening is carried out based on difficulty and information entropy, and topic selection is carried out based on the difference indicators between models (such as mutual information, variance or entropy) to ensure that the representative topics more accurately represent the model's ability.

[0049] S202 utilizes the query to reconstruct the projection model obtained in the pre-training stage. The query vector for each difficult problem is injected into all candidate models in the model pool, and each model's answer to the problem is recorded as "correct" or "incorrect." For the same candidate model, the performance results across all difficult problems are aggregated and concatenated into a vector sequence, which is defined as the context model capability representation. Specifically, the context model capability representation... Characterized as:

[0050]

[0051] In the formula, For the original query Vector representation, For the first The representation of whether the answer of each candidate model is correct.

[0052] The aggregation of contextual capability representations is achieved using sequence concatenation, pooling (such as average pooling, max pooling, or weighted pooling), or attention mechanisms. A cost penalty term is introduced into the contextual capability representation to achieve a balance between cost and performance. Specifically, the computational cost of each model (such as inference time, resource consumption, or energy consumption) is quantified as a penalty term and integrated into the model's contextual capability representation or the optimization objective of the model router, ensuring that routing decisions consider not only performance differences but also resource consumption.

[0053] S203 trains a model router composed of lightweight language models, using contextual model capability representations and query vectors as input and the correctness / incorrectness labels of candidate models for the query as supervision signals. Since the training samples contain both query semantics and model performance information, the model router, after convergence, can distinguish the merits of multiple models at a fine-grained capability level and output the most suitable route selection. Specifically, for the training samples, the sample data consists of contextual model capability representations and query vectors, with labels representing the candidate models to be assigned. Model performance is built into the contextual capability representations.

[0054] The training strategies for lightweight language models employ supervised learning, contrastive learning, or reinforcement learning.

[0055] During the routing training phase, the correct and incorrect records of each candidate model on a small batch of representative queries are encoded into a contextual model capability representation. This representation is then input into the model router along with the new query, and a binary cross-entropy loss is applied. To fit the model Can you answer the question correctly? The probability of [the outcome]. By minimizing the loss. The model router learns to establish a fine-grained mapping between query features and model capabilities.

[0056]

[0057] in, This indicates all queries and candidate models Take the average above, Representation Model In the query The performance label on it Predict the correct probability for the model router.

[0058] The context-based learning model representation routing training phase aims to incorporate the capability features of each candidate model into the vector representation and train the model router.

[0059] S3, Online Inference Stage: The user query and the contextual model capability representation of each candidate model are used as inputs to the trained model router to obtain the adaptation score of each candidate model. Based on the adaptation score, dynamic routing of the large language model is realized.

[0060] Specifically, the online inference phase includes steps S301-S302:

[0061] S301: Call the query embedding model to encode the questions in the incorrect question set into vector representations.

[0062] S302: The user query and the contextual capability representation of each candidate model are sequentially processed by the model router. An adaptation score is generated for each candidate model, and the model with the highest score is selected as the final executor to forward the user's question and return the answer.

[0063] Since the reasoning process involves only one query encoding and one small language model inference, and the context length is strictly controlled, the overall response latency is kept at an extremely low level, which can meet the requirements for real-time interaction in high-concurrency scenarios.

[0064] Furthermore, to support the continuous expansion and version updates of the model pool, this embodiment provides an incremental update mechanism. When a new model is added or an old model is upgraded, only one inference operation is needed on a representative evaluation set of a fixed size to obtain the model's performance vectors for each problem and aggregate them into a new context model capability representation. The routing language model can then immediately reference this representation without reprocessing the complete benchmark dataset or fine-tuning the model router. Since the offline computational load is directly proportional to the size of the problem set, the entire cold start process can be completed in a short time, thereby significantly reducing computational power consumption and operational costs.

[0065] The dynamic routing method provided in this embodiment improves routing accuracy while maintaining real-time performance, shortens the deployment cycle, and can still stably distinguish model capabilities in cross-domain scenarios, providing a scalable and low-overhead dynamic routing solution for large-scale model pools.

[0066] Example 2

[0067] Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the dynamic routing method for large language models based on context learning model representation as described in Embodiment 1.

[0068] like Figure 3 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method described herein. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0069] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0070] Example 3

[0071] Based on the foregoing embodiments, this embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for executing the dynamic routing method for a large language model based on context learning model representation as described in Embodiment 1.

[0072] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0073] This invention maps query embeddings to a semantic space consistent with the language model input using a projection model, and reconstructs the mapped query embeddings with the help of the language model, achieving alignment between query semantics and model representation space. Simultaneously, it automatically selects a representative evaluation set from a benchmark question bank that fully reflects the performance differences between models. High-quality contextual model capability representations can be generated by performing only one inference on each candidate model on this small set of challenging questions. During the online routing phase, supervised learning is performed by inputting real-time queries and contextual model representations into a lightweight routing language model, thereby simultaneously capturing query semantics and model output features, achieving precise differentiation of fine-grained model capabilities. When a new model is added or a model version is iterated, there is no need to re-infer or retrain the model router on the complete query set; only one inference on a fixed representative evaluation set is required to quickly update the contextual model capability representations, effectively reducing cold start and maintenance costs, and achieving a balance between real-time performance, accuracy, flexible scalability, and low-cost maintenance.

[0074] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic routing method for large language models based on context-learning model representations, characterized in that, When applied to recommendation systems, search engines, or knowledge-based question-answering systems, the following steps are included: Query reconstruction pre-training: The projection model is used to map the input query embedding vector to a high-dimensional semantic space. The mapped embedding vector is then input into the model router. The projection model and the model router are trained with the goal of restoring the user query embedding vector. Training of a model router based on context learning: Based on a pre-built representative evaluation set with multiple difficulty levels, the query vectors in the representative evaluation set are injected into multiple candidate models using a trained projection model. For each candidate model, the correct and incorrect labels under each query vector are aggregated to obtain the context model capability representation. The context model capability representation and the user query are used as inputs, and the correct and incorrect labels corresponding to the query are used as supervision signals to train a model router based on a lightweight language model. Online inference phase: User queries and the contextual model capability representations of each candidate model are used as inputs to the trained model router to obtain the adaptation scores of each candidate model. Dynamic routing of the large language model is then implemented based on the adaptation scores.

2. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, The process of constructing the representative evaluation set includes the following steps: Based on the accuracy rate of the questions, the difficulty levels were divided into several equally wide ranges from extremely difficult to highly divergent. An equal number of questions were drawn from each range to construct a representative assessment set covering different difficulty levels.

3. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, Also includes: Incremental update process: In response to the access of new candidate models or the upgrading of existing candidate models, the new candidate models and / or upgraded candidate models are inferred on the representative evaluation set to obtain the corresponding context model capability representation.

4. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, Based on average pooling, max pooling, weighted pooling, or attention mechanisms, the correct / incorrect labels and computational cost penalty terms for each query are aggregated to obtain the context model capability representation.

5. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, In constructing the representative evaluation set, questions are selected based on the difficulty and information entropy of the questions, as well as the difference indicators between candidate models, so as to represent the ability of the models by the questions.

6. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, The projection model achieves semantic alignment between the query embedding and the model input space through linear or nonlinear neural networks, and the model router is trained based on contrastive learning or reinforcement learning.

7. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, During the training of the model representation route, the projection model and the model router are optimized synchronously by minimizing the reconstruction loss, thereby achieving semantic alignment between the projection model and the model router.

8. The dynamic routing method for large language models based on context learning model representation according to claim 1, characterized in that, The high-dimensional semantic space is the same as the original input of the candidate model.

9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the dynamic routing method for large language models based on context learning model representations as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the dynamic routing method for large language models based on context learning model representations as described in any one of claims 1-8.

Images