Model routing method, system and device based on decoupled model representation

Abstract

The application discloses a model routing method, system and device based on decoupling model representation, and belongs to the technical field of artificial intelligence. The method encodes user query text into query features by using a pre-trained query text encoder to effectively represent the user query text, and introduces prior information and non-prior information to form decoupling model representation information at the model representation level to effectively represent a large language model. Then, a routing model selects a large language model based on a structured attention mechanism according to the query features and the decoupling model representation information of each large language model to generate an answer for the input user query text, which can enhance the response capability of the cluster system to different inputs, so that the model routing process can fully consider the diversity and complexity of the input, thereby flexibly adjusting the routing decision when processing different types of inputs to accurately select a large language model suitable for processing the current input user query text.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a model routing method, system, and device based on decoupled model representation. Background Technology

[0002] As large language models (LLMs) have demonstrated powerful performance in natural language processing fields such as everyday conversation, logical reasoning, mathematical problem solving, and code generation, the concept of model clusters has been proposed, aiming to build a cluster system (i.e., a large language model pool) with comprehensive capabilities in natural language tasks across different natural language processing domains.

[0003] However, in the process of implementing this application, the inventors found that the current model routing methods mainly adopt static routing strategies and lack effective representation methods for user query text and large language models, which makes it difficult to fully utilize the potential of the model in specific natural language tasks, resulting in poor overall performance of the cluster system. Summary of the Invention

[0004] The purpose of this application is to provide a model routing method, system, and device based on decoupled model representation, which can solve the problem that current model routing methods cannot fully utilize the potential of models in specific natural language tasks, resulting in poor overall performance of the cluster system.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a model routing method based on decoupled model representation, the method comprising: In response to receiving a user query text, the user query text is encoded using a pre-trained query text encoder to obtain query features; Obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. Using a pre-trained routing model and based on a structured attention mechanism, the target model features of each large language model are determined according to the query features and the decoupled model representation information of each large language model. The target model features are used to represent the intrinsic relationship between the user query text and the large language model. Based on the target model features of each of the large language models, a pre-trained neural network is used to determine the estimated score of each of the large language models. The largest language model with the highest predicted score is used to generate an answer to the user's query text.

[0006] Secondly, embodiments of this application provide a model routing system based on decoupled model representation. This system includes a query model feature embedding module, a structured attention capture module, and a model selection module, wherein: The query model feature embedding module is configured to, in response to receiving user query text, encode the user query text using a pre-trained query text encoder to obtain query features; and to obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. The structured attention capture module is configured to use a pre-trained routing model, based on a structured attention mechanism, to determine the target model features of each large language model according to the query features and the decoupled model representation information of each large language model. The target model features are used to characterize the intrinsic relationship between the user query text and the large language model. The model selection module is configured to determine the estimated score of each of the large language models using a pre-trained neural network based on the target model features of each of the large language models; and to generate an answer for the user query text using the large language model with the highest estimated score.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the model routing method based on decoupled model representation as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, implement the steps of the model routing method based on decoupled model representation as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the model routing method based on decoupled model representation as described in the first aspect.

[0010] In this embodiment, a pre-trained query text encoder is used to encode user query text into query features, thereby achieving effective representation of user query text and improving the accuracy and efficiency of semantic understanding of user query text. At the model representation level, prior information and non-prior information are introduced to form decoupled model representation information, namely, a pair of model decoupling features—text-encoded model prior features and learnable non-prior dynamic model features—to achieve effective representation of the large language model. Thus, by providing efficient and accurate representation methods for user query text and the large language model, and then using a routing model based on a structured attention mechanism to select the large language model to generate answers for the input user query text based on query features and the decoupled model representation information of each large language model, the responsiveness of the cluster system to different inputs can be enhanced. This allows the model routing process to fully consider the diversity and complexity of inputs, enabling flexible adjustment of routing decisions to accurately select the large language model suitable for processing the current input user query text when handling different types of inputs. This allows for full utilization of the model's potential in specific natural language tasks, improving the overall performance of the cluster system. Attached Figure Description

[0011] Figure 1 A flowchart illustrating the implementation of a model routing method based on decoupled model representation provided in this application embodiment; Figure 2 This is a schematic diagram illustrating an embodiment of the present application that uses an optimal model to answer a user query text. Figure 3 A schematic diagram illustrating a target structured representation provided in an embodiment of this application; Figure 4 A schematic diagram illustrating a feature embedding process provided in an embodiment of this application; Figure 5 A schematic diagram illustrating a model selection process provided in an embodiment of this application; Figure 6 This application provides a model routing system based on decoupled model representation. Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] First, to facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application will be briefly explained below.

[0015] Large language models, also known as large models, refer to a class of language models that are composed of artificial neural networks with a large number of parameters (usually in the billions to trillions) and are obtained through unsupervised or self-supervised training based on text.

[0016] KL divergence (KLD) is an indicator that measures the degree of matching between two probability distributions.

[0017] As the scale and complexity of large language models increase, the challenges of deploying model clusters, such as computational costs, latency, and scalability, become increasingly severe. Therefore, model routing methods that aim to save computational overhead on model clusters and dynamically allocate specific large language models to handle user query texts become particularly important.

[0018] However, in the process of implementing this application, the inventors discovered that current model routing methods are unable to fully utilize the potential of the model in specific natural language tasks, resulting in poor overall performance of the cluster system. The reasons for this are mainly as follows: 1. Static Routing Strategy: Current model routing methods primarily employ static routing strategies, meaning that routing decisions are fixed when processing different types of input. This approach ignores the diversity and complexity of inputs, making it difficult to fully utilize the model's potential for specific natural language tasks, resulting in poor overall performance of the cluster system.

[0019] 2. Lack of effective representation: Current model routing methods lack effective representation of user query text and large language models, making it difficult for subsequent model routing processes to effectively capture the intrinsic relationship between user query text and large language models. This limits the responsiveness of the cluster system to different inputs, making it difficult for the model routing process to fully consider the diversity and complexity of inputs. Consequently, it is difficult to fully utilize the potential of the model in specific natural language tasks, resulting in poor overall performance of the cluster system.

[0020] To address the problem that existing model routing methods suffer from fixed routing strategies and a lack of effective representation of user queries and large language models, leading to insufficient utilization of model potential in specific natural language tasks and consequently poor overall performance of the cluster system, this application provides a model routing scheme based on decoupled model representation. This scheme offers efficient and accurate representation of user query text and large language models, and leverages a structured attention mechanism to effectively capture the intrinsic connections between user query text and large language models. This allows the model routing process to fully consider the diversity and complexity of inputs, enabling flexible adjustments to routing decisions when processing different types of inputs to accurately select the large language model suitable for processing the current user query text. This allows for full utilization of model potential in specific natural language tasks, improving the overall performance of the cluster system.

[0021] The following description, in conjunction with the accompanying drawings, details a model routing method, system, and device based on decoupled model representation provided in this application, through specific embodiments and application scenarios.

[0022] See Figure 1 The diagram shown is an implementation flowchart of a model routing method based on decoupled model representation provided in this application. The method may include the following steps: Step S101: In response to receiving user query text, the user query text is encoded using a pre-trained query text encoder to obtain query features.

[0023] The query text encoder can be a text encoder such as a BERT encoder or a RoBERTa encoder; when applied to a system that supports multiple encoders, the query text encoder can include multiple text encoders.

[0024] In this embodiment of the application, the user query text is used to describe the instructions that the user wants the cluster system to complete for natural language tasks (such as question answering tasks, logical reasoning tasks, mathematical problem solving tasks, and code generation tasks), such as "generate financial advice based on your recent personal financial situation".

[0025] After receiving the user's query text, the query text is fed into a pre-trained query text encoder to encode it. The vectorized text representation output by the query text encoder is then used as query features to improve the accuracy and efficiency of semantic understanding of the user's query text. This process can be represented by the following formula:

[0026] in, This represents the user's query text. This represents a pre-trained text encoder (i.e., a query text encoder). Indicates query characteristics.

[0027] Step S102: Obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase.

[0028] The descriptive text encoder can be an mDeBERTaV3-base encoder, a BERT encoder, or a RoBERTa encoder, etc. When applied to a system that supports multiple encoders, the descriptive text encoder can include multiple text encoders. In practical applications, the descriptive text encoder and the query text encoder can be the same text encoder or different text encoders.

[0029] In this embodiment of the application, a pre-constructed system containing A large language model pool As a cluster system.

[0030] To introduce prior information at the model representation level, K model description texts are generated in advance for each large language model in the large language model pool as prior information for the corresponding large language model; wherein, K≥1, the K model description texts can be used to describe the number of parameters, input modality, output modality, whether it has been fine-tuned in a professional domain (i.e., a specific natural language processing domain or a specific natural language task) and the corresponding professional domain, the supported context length, and other information of the corresponding large language model.

[0031] Then, a pre-trained descriptive text encoder is used to encode the K model descriptive texts of each large language model to obtain the model prior features of each large language model and store them so that the model prior features of each large language model can be directly retrieved from the storage when performing model routing in the future.

[0032] In some embodiments, the K model description texts of the large language model can be concatenated and input into a pre-trained description text encoder, and then the vectorized text representation output by the description text encoder can be used as the model prior features of each large language model.

[0033] In some embodiments, in response to detecting that there are multiple model description texts for the large language model (corresponding to the case K > 1), a pre-trained description text encoder is used to encode the multiple model description texts to obtain multiple vectorized text representations; the average value of the multiple vectorized text representations is determined as the model prior feature of the large language model. The aforementioned process can be represented by the following formula:

[0034] in, This represents a pre-trained text encoder (i.e., a descriptive text encoder). Represents the i-th large language model The j-th model description text, express The model's prior features, K represents The number of texts describing the model. This indicates the number of large language models in the large language model pool.

[0035] Therefore, by using a unified text encoder to encode different model description texts of a large language model and calculate the average value to obtain the model prior features of the large language model, the efficiency and accuracy of model representation can be guaranteed even when there are multiple model description texts for the large language model.

[0036] To introduce non-prior information at the model representation level, learnable, prior-free model embedding features (without model-related information) are randomly initialized for each large language model in the large language model pool beforehand. These prior-free model embedding features are then optimized during the training phase to obtain the prior-free dynamic features of the corresponding large language model. For example, during end-to-end training of all models involved in the model routing process (such as the routing model for the structured attention capture stage, the neural network for the model selection stage, etc.), the randomly initialized prior-free model embedding features for each large language model can be simultaneously optimized. The prior-free model embedding features of each large language model obtained after end-to-end training are then used as the prior-free dynamic features of each large language model. This allows the decoupled model representation information of the various language models in the large language model pool to be obtained. ,in, Let represent the model prior features of the i-th large language model. Let represent the prior-free model dynamic features of the i-th large language model. This indicates the number of large language models in the large language model pool.

[0037] Understandably, the model routing method in this application is based on the decoupled representation of large language models. It introduces prior and non-prior information at the model representation level, specifically a pair of decoupled features: text-encoded prior features and learnable non-prior dynamic features. The prior features describe the model's inherent characteristics, while the learnable non-prior dynamic features describe the auxiliary decision-making information needed for subsequent model routing (i.e., information learned during training on how to select the appropriate large language model based on the user query text). Therefore, this application achieves efficient, accurate, and robust model feature embedding in the model routing process based on decoupled representation. Combining this with query features provides a complete and effective representation of the user query text and the large language model, laying a solid foundation for subsequent model routing. This enhances the overall cluster system's responsiveness to different inputs and ensures good discriminability among different large language models in the feature space, thereby improving the effectiveness and generalization of the model routing method.

[0038] Step S103: Using a pre-trained routing model, based on a structured attention mechanism, determine the target model features of each large language model according to the query features and the decoupled model representation information of each large language model. The target model features are used to represent the intrinsic relationship between the user query text and the large language model.

[0039] In this embodiment, the query features and the decoupled model representation information of each large language model are used as the initial features in the structured attention capture stage. A pre-trained routing model (such as a lightweight Transformer architecture or other related models that can implement the structured attention mechanism) is used to optimize the initial features based on the structured attention mechanism. This can effectively capture the intrinsic relationship between the user query text and the large language model, thereby laying the foundation for improving the accuracy, efficiency and robustness of the model routing method.

[0040] Step S104: Based on the target model features of each of the large language models, use a pre-trained neural network to determine the estimated score of each of the large language models.

[0041] In this embodiment of the application, after obtaining the target model features of each large language model through the structured attention capture stage, the target model features of each large language model are input into a pre-trained neural network (such as a feedforward neural network) to obtain the predicted score of each large language model output by the neural network.

[0042] In some embodiments, the output layer of a neural network can be constructed based on the softmax function. In the process of transforming the features of the target model through the neural network, the softmax function can be further combined to calculate the score distribution, thereby ensuring the computational efficiency of the predicted score and improving the reliability of the system response.

[0043] Step S105: Generate an answer for the user query text using the large language model with the highest estimated score.

[0044] In the embodiments of this application, reference is made to Figure 2 The diagram shown illustrates how the optimal model is used to answer a user's query text. The large language model with the highest estimated score is selected as the optimal model (i.e., the model obtained from the final routing). Subsequently, the user's query text can be input into the optimal model so that the optimal model can generate an answer for the user's query text.

[0045] In some embodiments, the predicted scores of the neural network outputs are compared, and the aforementioned process can be characterized by the following formula:

[0046] in, This indicates the number of the optimal model obtained from the routing. This represents the predicted score of the i-th (i.e., the one numbered i) large language model. This indicates the operation of retrieving the ID of the large language model with the highest predicted score.

[0047] As can be seen from the above technical solutions, using a pre-trained query text encoder to encode user query text into query features enables effective representation of user query text, improving the accuracy and efficiency of semantic understanding of user query text. At the model representation level, prior information and non-prior information are introduced to form decoupled model representation information, namely, a pair of model decoupling features—text-encoded model prior features and learnable non-prior dynamic model features—to achieve effective representation of the large language model. Therefore, by providing efficient and accurate representation methods for user query text and the large language model, and then using a routing model based on a structured attention mechanism to select the large language model to generate answers for the input user query text based on query features and the decoupled model representation information of each large language model, the responsiveness of the cluster system to different inputs can be enhanced. This allows the model routing process to fully consider the diversity and complexity of inputs, enabling flexible adjustment of routing decisions to accurately select the large language model suitable for processing the current input user query text when handling different types of inputs. This allows for full utilization of the model's potential in specific natural language tasks, improving the overall performance of the cluster system.

[0048] In some embodiments, the step of using a pre-trained routing model, based on a structured attention mechanism, to determine the target model features of each of the large language models according to the query features and the decoupled model representation information of each of the large language models includes: Using a pre-trained routing model, a target structured representation is generated based on the query features and the decoupled model representation information of each of the large language models. The target structured representation includes: a radial topology consisting of relay nodes and satellite nodes adjacent only to the relay nodes; each relay node corresponds to the user query text, and the initial state of each relay node is determined based on the query features; the context information of each relay node is determined based on the states of the relay node and each of the satellite nodes; each satellite node corresponds to each of the large language models, and the initial state of each satellite node is determined based on the decoupled model representation information of each of the large language models; the context information of each satellite node is determined based on the states of the relay node and each of the satellite nodes, as well as the decoupled model representation information of each of the large language models. Using a pre-trained routing model, based on a multi-head attention mechanism and a preset number of iterations, the state of each satellite node is iteratively updated according to the context information and state of each satellite node in the target structured representation, and the state of the relay node is iteratively updated according to the context information and state of the relay node in the target structured representation. After the final iteration, the state of each satellite node is determined as the target model feature of each large language model.

[0049] In this embodiment, considering that current model routing methods often suffer from low computational efficiency in practical applications, leading to significant waste of computing resources and reduced overall throughput of the cluster system when handling a large number of concurrent user requests, this application introduces a radial topology based on relay nodes and satellite nodes in the structured attention capture stage to support the combined application of the structured attention mechanism. This effectively simplifies the process of capturing the relationship between query features and model features, and compared to the complete attention mechanism used in related technologies, this application can effectively improve computational efficiency.

[0050] Specifically, in combining query features and After the decoupled model representation information of each of the large language models is input into the routing model, the routing model generates a target structured representation that represents the intrinsic relationship between the user query text corresponding to the query feature and each large language model; refer to Figure 3 The diagram shows a target structured representation, which (also known as a structured attention representation) consists of one relay node (also known as a query node). and Satellite nodes (also known as model nodes) Composition, corresponding to the user query text and Several large language models, and the initial state and context information of each node are based on query features and The decoupled model representation information of each large language model is initialized.

[0051] In this target structured representation, each satellite node is adjacent to only one relay node, forming a radial topology. This highly simplified topology helps improve computational efficiency, enabling the routing module to use a multi-head attention mechanism to iteratively update each node in the target structured representation. This allows for optimized learning of the decoupled model representation information of the query features and each large language model based on the structured attention mechanism, thereby obtaining the target model features of each large language model (i.e., the final state of each satellite node in the target structured representation obtained through optimized learning). This effectively improves the efficiency of capturing the intrinsic relationship between the user query text and the large language model, achieving fast routing.

[0052] Understandably, this target structured representation can effectively improve the accuracy of model routing and optimize the targeting of model selection, thereby reducing performance loss caused by inaccurate routing. Furthermore, the use of a multi-head attention mechanism to capture and update features between adjacent nodes of the target structured representation can enhance the routing model's ability to process information flow, thus enabling it to more accurately capture the inherent relationship between user query text and the large language model.

[0053] In some embodiments, the target model features of each of the large language models include: a first state of the corresponding satellite node obtained by iterative state update based on prior model features, and a second state of the corresponding satellite node obtained by iterative state update based on dynamic model features without prior knowledge; the relay node and the satellite node update their states using the following formula:

[0054]

[0055] Wherein, the context information of the i-th satellite node in the t-th iteration. When determining the first state, For the prior features of the large language model corresponding to the i-th satellite node, when determining the second state... For the priori dynamic features of the large language model corresponding to the i-th satellite node; , This represents the state of the i-th satellite node in the (t-1)-th iteration and the t-th iteration. When determining the first state, the initial state of the i-th satellite node is the model prior feature of the large language model corresponding to the i-th satellite node. When determining the second state, the initial state of the i-th satellite node is the model dynamic feature without prior information of the large language model corresponding to the i-th satellite node. MultiHeadAttention represents the multi-head attention mechanism. , This represents the state of the relay node in the (t-1)th iteration and the tth iteration. This represents the context information of the relay node in the t-th iteration. This represents the state of all satellite nodes in the t-th iteration.

[0056] In the embodiments of this application, reference is made to Figure 4 The schematic diagram of the feature embedding process shown illustrates that, in order to ensure the processing effect of the decoupled model representation (i.e., model prior features and model dynamic features without prior knowledge), a method based on structured attention mechanism is adopted to calculate the model prior features and model dynamic features without prior knowledge separately, so as to avoid mutual interference between the model prior features and model dynamic features without prior knowledge.

[0057] Specifically, the query features are used as the initial state of the relay nodes; when determining the first state, the model prior features of the large language model corresponding to each satellite node are used as the initial state of each satellite node (the initial state of the i-th satellite node when determining the first state can be represented as...). At this point, the initial context information of each satellite node includes the prior features of the large language model corresponding to the satellite node, the initial state of the corresponding satellite node, and the initial state of the relay node. When determining the second state, the prior-free dynamic features of the large language model corresponding to each satellite node are used as the initial state of each satellite node (the initial state of the i-th satellite node when determining the second state can be represented as...). At this point, the initial context information of each satellite node includes the prior-free dynamic features of the large language model corresponding to the satellite node, the initial state of the corresponding satellite node, and the initial state of the relay node.

[0058] In the first iteration, using the aforementioned satellite node state update formula, the state of each satellite node associated with the first and second states is updated once from its initial state, based on the initial context information of each satellite node associated with the first and second states respectively. The initial state of the relay node is then summarized with the updated states of all satellite nodes associated with the first and second states respectively, yielding the initial context information of the relay nodes associated with the first and second states. Then, using the aforementioned relay node state update formula, the state of each relay node associated with the first and second states is updated once from its initial state, based on the initial context information of each relay node associated with the first and second states respectively. Next, the context information of each node is updated according to the updated node states, and a new iteration is entered in a similar manner. After iteration T (the value of which is flexibly set according to actual needs), the first and second states of each satellite node are obtained (i.e., the T-cycle updated states of each satellite node associated with the first and second states respectively). ).

[0059] In some embodiments, determining the estimated score of each large language model using a pre-trained neural network based on the target model features of each large language model includes: Using a pre-trained neural network, the first potential score and the second potential score of each large language model are determined based on the first state and the second state in the target model features of each large language model. The weighted sum of the first and second latent scores of each of the large language models is calculated to obtain the predicted score of each of the large language models.

[0060] In this embodiment of the application, during the model selection stage, the first state and the second state of the target model features of each large language model are respectively input into a pre-trained neural network (which can be the same pre-trained neural network, or different neural networks pre-trained for the first state and the second state respectively) to obtain a first potential score determined based on the first state (which represents the preference for selecting a suitable model based on the model's own characteristics) and a second potential score determined based on the second state (which represents the auxiliary information for model selection decision obtained through dynamic optimization learning).

[0061] The weights corresponding to the first and second potential scores are flexibly set according to actual needs. Then, based on these weights, a weighted sum is calculated for the first and second potential scores of each large language model to obtain the predicted score for each model. (Refer to...) Figure 5 The diagram shown illustrates the model selection process, which can be represented by the following formula:

[0062] in, Let represent the predicted score of the i-th large language model. and This represents the first and second states in the target model features of the i-th large language model. This represents the neural network used to output the first and second potential scores. and This represents the weights corresponding to the first and second potential scores, respectively.

[0063] In some embodiments, the method further includes: During the training phase, based on the following constraints Perform parameter optimization:

[0064] in, and Let represent the probability distribution associated with the predicted score and the probability distribution associated with the actual score, respectively. express and KL divergence loss between them This indicates the parameters that need to be optimized during the training phase. The parameters that need to be optimized during the training phase include the parameters of the routing model, the parameters of the neural network, and the prior-free model embedding features of each of the large language models.

[0065] In some embodiments, The expression is as follows:

[0066] Where n represents the number of large language models in the large language model pool. This represents the predicted score of the i-th large language model output by the neural network. This represents the true score of the i-th large language model.

[0067] In this embodiment, KL divergence loss is used to constrain the model routing process, enabling the relevant models used in the routing process (such as routing models and neural networks) to learn model selection probabilities from the true route probability distribution based on query features during end-to-end training. This allows the routing decision (i.e., the route probability predicted by the model) to approximate the probability direction determined by the true score index, thus aligning with the actual performance of large language models. Therefore, by introducing the aforementioned constraint term based on KL divergence loss into end-to-end training, the predicted route probability distribution is effectively aligned with the true score distribution, allowing the model to learn the optimal routing strategy more accurately and improving its robustness.

[0068] In some embodiments, the method further includes: Based on the performance requirements and computational overhead constraints of the target natural language processing scenario, adjust the first weight associated with the performance score and the second weight associated with the computational overhead. Based on the first weight and the second weight, a weighted sum is calculated on the performance scores and computational costs of different large language models on different natural language tasks in the target natural language processing scenario to obtain the probability distribution associated with the true scores in the target natural language processing scenario.

[0069] In this embodiment of the application, considering that the supervision signal during the training phase is the "score" of different large language models on different user query texts, this application designs that the score is obtained by weighting the performance score (such as accuracy) and computational cost (which can be converted from relevant indicators such as inference time or computational amount) of the relevant models (such as routing models and neural networks) used in the model routing process. This allows for the adjustment of the weighting ratio (i.e., adjusting the first weight and the second weight) to obtain supervision signals with different preferences (for example, if performance is emphasized, the first weight associated with the performance score is increased, and vice versa). Then, combined with the above-mentioned constraint terms based on KL divergence loss, relevant models that meet different preferences are trained.

[0070] Therefore, by using KL divergence loss to supervise the training process and further adjusting the first and second weights, the model performance and computational cost can be dynamically adjusted. This allows for an effective trade-off between performance and computational cost, thereby improving the efficiency, robustness, and accuracy of the model routing method in natural language processing scenarios with different performance requirements and computational cost constraints. This makes the model routing method applicable to a wide range of application scenarios (such as resource-constrained environments and situations with higher requirements for response speed).

[0071] In some embodiments, after receiving the user's query text, the method further includes: In response to the detection of a newly added or changed large language model in the large language model pool, the prior features of each large language model in the large language model pool are obtained, and a large language model is selected from the large language models to generate an answer for the user query text based on the query features and the prior features of each large language model.

[0072] In this embodiment, considering the lack of effective means to represent dynamically changing large language models in related technologies, this limits the ability of model routing methods to capture the intrinsic relationship between user query text and large language models and their generalization. To solve the aforementioned problem, this application designs a method that, when there are new or changed large language models in the large language model pool, automatically skips the dynamic features of the model without prior knowledge and directly uses the model's prior features (obtained by text description and encoding based on model characteristics) to perform the aforementioned structured attention capture stage and model selection stage (e.g., calculating the first state and selecting the model accordingly). This avoids the need to retrain the relevant models and parameters every time a new model is added to the large language model pool or an existing model is updated. As a result, the model routing method can adapt to dynamic scenarios of model changes or the introduction of new models, thereby improving generalization.

[0073] Based on the above embodiments, this application can improve the accuracy and efficiency of model routing methods by introducing decoupled model representations and improved training strategies. It can not only optimize the performance of model routing methods, but also provide more efficient and flexible solutions for various complex application scenarios, which will help promote the further development of intelligent model clusters.

[0074] This application provides a model routing system based on decoupled model representation, such as... Figure 6 As shown, the system includes a query model feature embedding module, a structured attention capture module, and a model selection module, wherein: The query model feature embedding module is configured to, in response to receiving user query text, encode the user query text using a pre-trained query text encoder to obtain query features; and to obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. The structured attention capture module is configured to use a pre-trained routing model, based on a structured attention mechanism, to determine the target model features of each large language model according to the query features and the decoupled model representation information of each large language model. The target model features are used to characterize the intrinsic relationship between the user query text and the large language model. The model selection module is configured to determine the estimated score of each of the large language models using a pre-trained neural network based on the target model features of each of the large language models; and to generate an answer for the user query text using the large language model with the highest estimated score.

[0075] As can be seen from the above technical solutions, using a pre-trained query text encoder to encode user query text into query features enables effective representation of user query text, improving the accuracy and efficiency of semantic understanding of user query text. At the model representation level, prior information and non-prior information are introduced to form decoupled model representation information, namely, a pair of model decoupling features—text-encoded model prior features and learnable non-prior dynamic model features—to achieve effective representation of the large language model. Therefore, by providing efficient and accurate representation methods for user query text and the large language model, and then using a routing model based on a structured attention mechanism to select the large language model to generate answers for the input user query text based on query features and the decoupled model representation information of each large language model, the responsiveness of the cluster system to different inputs can be enhanced. This allows the model routing process to fully consider the diversity and complexity of inputs, enabling flexible adjustment of routing decisions to accurately select the large language model suitable for processing the current input user query text when handling different types of inputs. This allows for full utilization of the model's potential in specific natural language tasks, improving the overall performance of the cluster system.

[0076] Optionally, the structured attention capture module is further configured to perform the following steps: Using a pre-trained routing model, a target structured representation is generated based on the query features and the decoupled model representation information of each of the large language models. The target structured representation includes: a radial topology consisting of relay nodes and satellite nodes adjacent only to the relay nodes; each relay node corresponds to the user query text, and the initial state of each relay node is determined based on the query features; the context information of each relay node is determined based on the states of the relay node and each of the satellite nodes; each satellite node corresponds to each of the large language models, and the initial state of each satellite node is determined based on the decoupled model representation information of each of the large language models; the context information of each satellite node is determined based on the states of the relay node and each of the satellite nodes, as well as the decoupled model representation information of each of the large language models. Using a pre-trained routing model, based on a multi-head attention mechanism and a preset number of iterations, the state of each satellite node is iteratively updated according to the context information and state of each satellite node in the target structured representation, and the state of the relay node is iteratively updated according to the context information and state of the relay node in the target structured representation. After the final iteration, the state of each satellite node is determined as the target model feature of each large language model.

[0077] Optionally, the target model features of each of the large language models include: the first state of the corresponding satellite node obtained by iterative state update based on prior model features, and the second state of the corresponding satellite node obtained by iterative state update based on dynamic model features without prior knowledge; the relay node and the satellite node update their states using the following formula:

[0078]

[0079] Wherein, the context information of the i-th satellite node in the t-th iteration. When determining the first state, For the prior features of the large language model corresponding to the i-th satellite node, when determining the second state... For the priori dynamic features of the large language model corresponding to the i-th satellite node; , This represents the state of the i-th satellite node in the (t-1)-th iteration and the t-th iteration. When determining the first state, the initial state of the i-th satellite node is the model prior feature of the large language model corresponding to the i-th satellite node. When determining the second state, the initial state of the i-th satellite node is the model dynamic feature without prior information of the large language model corresponding to the i-th satellite node. MultiHeadAttention represents the multi-head attention mechanism. , This represents the state of the relay node in the (t-1)th iteration and the tth iteration. This represents the context information of the relay node in the t-th iteration. This represents the state of all satellite nodes in the t-th iteration.

[0080] Optionally, the model selection module is further configured to perform the following steps: Using a pre-trained neural network, the first potential score and the second potential score of each large language model are determined based on the first state and the second state in the target model features of each large language model. The weighted sum of the first and second latent scores of each of the large language models is calculated to obtain the predicted score of each of the large language models.

[0081] Optionally, the system further includes: The training module is configured to operate during the training phase based on the following constraints. Perform parameter optimization:

[0082] in, and Let represent the probability distribution associated with the predicted score and the probability distribution associated with the actual score, respectively. express and KL divergence loss between them This indicates the parameters that need to be optimized during the training phase. The parameters that need to be optimized during the training phase include the parameters of the routing model, the parameters of the neural network, and the prior-free model embedding features of each of the large language models.

[0083] Optionally, the training module is also configured to perform the following steps: Based on the performance requirements and computational overhead constraints of the target natural language processing scenario, adjust the first weight associated with the performance score and the second weight associated with the computational overhead. Based on the first weight and the second weight, a weighted sum is calculated on the performance scores and computational costs of different large language models on different natural language tasks in the target natural language processing scenario to obtain the probability distribution associated with the true scores in the target natural language processing scenario.

[0084] Optionally, the system further includes: The dynamic scene processing module is configured to, in response to the detection of a newly added or changed large language model in the large language model pool, obtain the model prior features of each large language model in the large language model pool, and select a large language model from the large language models to generate an answer for the user query text based on the query features and the model prior features of each large language model.

[0085] Optionally, the query model feature embedding module is further configured to perform the following steps: In response to the detection that there are multiple model description texts for the large language model, a pre-trained description text encoder is used to encode the multiple model description texts to obtain multiple vectorized text representations. The average value of the multiple vectorized text representations is determined as the model prior feature of the large language model.

[0086] Optionally, this application embodiment also provides an electronic device, including a processor 110, a memory 109, and a program or instructions stored in the memory 109 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described model routing method embodiment based on decoupled model representation and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0087] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0088] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0089] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0090] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0091] The processor 110 is used to perform the following steps: In response to receiving a user query text, the user query text is encoded using a pre-trained query text encoder to obtain query features; Obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. Using a pre-trained routing model and based on a structured attention mechanism, the target model features of each large language model are determined according to the query features and the decoupled model representation information of each large language model. The target model features are used to represent the intrinsic relationship between the user query text and the large language model. Based on the target model features of each of the large language models, a pre-trained neural network is used to determine the estimated score of each of the large language models. The largest language model with the highest predicted score is used to generate an answer to the user's query text.

[0092] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model routing method embodiment based on decoupled model representation and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0093] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0094] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described model routing method embodiment based on decoupled model representation, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0095] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0096] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model routing method based on decoupled model representation, characterized in that, The method includes: In response to receiving a user query text, the user query text is encoded using a pre-trained query text encoder to obtain query features; Obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. Using a pre-trained routing model and based on a structured attention mechanism, the target model features of each large language model are determined according to the query features and the decoupled model representation information of each large language model. The target model features are used to represent the intrinsic relationship between the user query text and the large language model. Based on the target model features of each of the large language models, a pre-trained neural network is used to determine the estimated score of each of the large language models. The largest language model with the highest predicted score is used to generate an answer to the user's query text.

2. The method according to claim 1, characterized in that, The pre-trained routing model, based on a structured attention mechanism, determines the target model features of each large language model according to the query features and the decoupled model representation information of each large language model, including: Using a pre-trained routing model, a target structured representation is generated based on the query features and the decoupled model representation information of each of the large language models. The target structured representation includes: a radial topology consisting of relay nodes and satellite nodes adjacent only to the relay nodes; each relay node corresponds to the user query text, and the initial state of each relay node is determined based on the query features; the context information of each relay node is determined based on the states of the relay node and each of the satellite nodes; each satellite node corresponds to each of the large language models, and the initial state of each satellite node is determined based on the decoupled model representation information of each of the large language models; the context information of each satellite node is determined based on the states of the relay node and each of the satellite nodes, as well as the decoupled model representation information of each of the large language models. Using a pre-trained routing model, based on a multi-head attention mechanism and a preset number of iterations, the state of each satellite node is iteratively updated according to the context information and state of each satellite node in the target structured representation, and the state of the relay node is iteratively updated according to the context information and state of the relay node in the target structured representation. After the final iteration, the state of each satellite node is determined as the target model feature of each large language model.

3. The method according to claim 2, characterized in that, The target model features of each of the large language models include: the first state of the corresponding satellite node obtained by iterative state update based on prior model features, and the second state of the corresponding satellite node obtained by iterative state update based on dynamic model features without prior knowledge; the relay node and the satellite node update their states using the following formula: Wherein, the context information of the i-th satellite node in the t-th iteration. When determining the first state, For the prior features of the large language model corresponding to the i-th satellite node, when determining the second state... For the priori dynamic features of the large language model corresponding to the i-th satellite node; , This represents the state of the i-th satellite node in the (t-1)-th iteration and the t-th iteration. When determining the first state, the initial state of the i-th satellite node is the model prior feature of the large language model corresponding to the i-th satellite node. When determining the second state, the initial state of the i-th satellite node is the model dynamic feature without prior information of the large language model corresponding to the i-th satellite node. MultiHeadAttention represents the multi-head attention mechanism. , This represents the state of the relay node in the (t-1)th iteration and the tth iteration. This represents the context information of the relay node in the t-th iteration. This represents the state of all satellite nodes in the t-th iteration.

4. The method according to claim 3, characterized in that, The step of determining the estimated score of each large language model using a pre-trained neural network based on the target model features of each large language model includes: Using a pre-trained neural network, the first potential score and the second potential score of each large language model are determined based on the first state and the second state in the target model features of each large language model. The weighted sum of the first and second latent scores of each of the large language models is calculated to obtain the predicted score of each of the large language models.

5. The method according to claim 1, characterized in that, The method further includes: During the training phase, based on the following constraints Perform parameter optimization: in, and Let represent the probability distribution associated with the predicted score and the probability distribution associated with the actual score, respectively. express and KL divergence loss between them This indicates the parameters that need to be optimized during the training phase. The parameters that need to be optimized during the training phase include the parameters of the routing model, the parameters of the neural network, and the prior-free model embedding features of each of the large language models.

6. The method according to claim 5, characterized in that, The method further includes: Based on the performance requirements and computational overhead constraints of the target natural language processing scenario, adjust the first weight associated with the performance score and the second weight associated with the computational overhead. Based on the first weight and the second weight, a weighted sum is calculated on the performance scores and computational costs of different large language models on different natural language tasks in the target natural language processing scenario to obtain the probability distribution associated with the true scores in the target natural language processing scenario.

7. The method according to any one of claims 1-6, characterized in that, After receiving the user's query text, the method further includes: In response to the detection of a newly added or changed large language model in the large language model pool, the prior features of each large language model in the large language model pool are obtained, and a large language model is selected from the large language models to generate an answer for the user query text based on the query features and the prior features of each large language model.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to the detection that there are multiple model description texts for the large language model, a pre-trained description text encoder is used to encode the multiple model description texts to obtain multiple vectorized text representations. The average value of the multiple vectorized text representations is determined as the model prior feature of the large language model.

9. A model routing system based on decoupled model representation, characterized in that, The system includes a query model feature embedding module, a structured attention capture module, and a model selection module, wherein: The query model feature embedding module is configured to, in response to receiving user query text, encode the user query text using a pre-trained query text encoder to obtain query features; and to obtain the decoupled model representation information of each large language model in the large language model pool. The decoupled model representation information includes: model prior features and model dynamic features without prior knowledge. The model prior features are obtained by encoding the model description text of the large language model using a pre-trained description text encoder. The model dynamic features without prior knowledge are obtained by optimizing the randomly initialized model embedding features without prior knowledge during the training phase. The structured attention capture module is configured to use a pre-trained routing model, based on a structured attention mechanism, to determine the target model features of each large language model according to the query features and the decoupled model representation information of each large language model. The target model features are used to characterize the intrinsic relationship between the user query text and the large language model. The model selection module is configured to determine the estimated score of each of the large language models using a pre-trained neural network based on the target model features of each of the large language models; and to generate an answer for the user query text using the large language model with the highest estimated score.

10. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the model routing method based on decoupled model representation as described in any one of claims 1-8.

Images