Large model routing method and device combined with reward model

Abstract

The invention provides a large model routing method and device combined with a reward model, and relates to the technical field of artificial intelligence model dynamic routing, and the method comprises the steps: obtaining a question inputted by a user, and determining a candidate model set according to the user authority and a preset priority rule; performing sensitive information judgment on the question through a sensitive detection model, directly returning a preset answer if sensitive information is detected, and otherwise, entering the next step; carrying out reward evaluation on each model in the candidate model set based on a routing function, training the routing function through a knowledge distillation technology, and optimizing a model selection strategy by taking KL divergence as a loss function; and according to an evaluation result of the routing function, dynamically distributing the problem to a target model with the highest reward value for reasoning, and outputting a processing result of the target model. According to the method, the calculation overhead of large model integration can be effectively reduced, the intelligent level of routing decision is improved, and the system resource utilization rate and the response efficiency are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of dynamic routing technology for artificial intelligence models, and in particular to a large-scale model routing method and apparatus that combines a reward model. Background Technology

[0002] Large Language Models (LLMs), as an important technological carrier in the field of artificial intelligence, are widely used in industries such as insurance and finance where response speed and resource efficiency are critical. With the exponential growth in the number and types of LLM parameters, traditional ensemble methods (such as weighted averaging and serial collaboration) improve performance through parallel inference of multiple models, but they suffer from significant technical bottlenecks. Specifically, existing technologies cover the entire process from model training to result fusion, including key steps such as candidate model set construction, output weighting calculation, and evaluation metric optimization. While the serial collaboration fusion method disclosed in CN117708274A can reduce random errors through model concatenation, its core drawback lies in its reliance on a structurally consistent model set, and its implementation... The weighted calculation method results in a linear relationship between computational resource consumption and the number of models.

[0003] However, existing LLM integration methods directly employ parallel inference architectures without considering the adaptability of tasks and models, which may lead to wasted computing resources and increased response latency. Specifically, this manifests in several ways: 1) In scenarios with millions of concurrent requests, the linearly increasing computational overhead significantly reduces system throughput; 2) Model structure consistency requirements limit the integration possibilities of heterogeneous models (such as hybrid architectures of Transformer and RNN); 3) Static routing strategies cannot adapt to dynamically changing business needs and model performance fluctuations. Therefore, traditional methods face technical challenges such as excessive server load and insufficient response speed in the real-time requirements of insurance business (such as auto insurance damage assessment and claims review). There is an urgent need to construct a multi-dimensional decision-making system that includes access control, sensitivity detection, and dynamic routing. The loss function in this form enables intelligent optimization of model selection. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a large-model routing method that incorporates a reward model.

[0006] The second objective of this invention is to propose a large-scale routing device that incorporates a reward model.

[0007] The third objective of this invention is to provide an electronic device.

[0008] The fourth objective of this invention is to provide a computer-readable storage medium.

[0009] The fifth objective of this invention is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of the present invention proposes a large model routing method combining a reward model, comprising: S1: Obtain the user's input question and determine the candidate model set based on user permissions and preset priority rules; S2, the question is judged for sensitive information by a sensitivity detection model. If sensitive information is detected, the preset answer is returned directly; otherwise, proceed to the next step. S3, evaluate the reward of each model in the candidate model set based on the routing function. The routing function is trained by knowledge distillation technique and uses KL divergence as the loss function to optimize the model selection strategy. S4. Based on the evaluation result of the routing function, the problem is dynamically distributed to the target model with the highest reward value for inference, and the processing result of the target model is output.

[0011] Optionally, the step of obtaining user input and determining a candidate model set based on user permissions and preset priority rules further includes: S11, Based on the user's role and access permissions, use a role-based access control (RBAC) model to filter the set of candidate models; S12, when multiple models meet the permission requirements, the candidate models are sorted according to a preset priority rule, which includes a weighted combination of model response speed, historical accuracy and resource utilization.

[0012] Optionally, the step of evaluating the rewards of each model in the candidate model set based on the routing function, wherein the routing function is trained using knowledge distillation technology and optimizes the model selection strategy using KL divergence as the loss function, further includes: S31, the knowledge distillation technique optimizes the model selection strategy by using the inference result of each model in the candidate model set as the output of the teacher model, calculating the KL divergence loss function. The calculation formula is: , in, It is a loss function. It is a student model. It is a model The output is the raw logits, without processing. The original score, It is a teacher model. The teacher model output process The probability distribution after that; S32, the routing function training process adopts a multi-round iterative mechanism, and each round of iteration updates the model selection probability distribution according to the latest reward value.

[0013] Optionally, the step of dynamically distributing the problem to the target model with the highest reward value for inference based on the evaluation result of the routing function, and outputting the processing result of the target model, further includes: S41, when the difference in reward values ​​of multiple models is less than a preset threshold, a round-robin strategy is used to select the target model; S42, dynamically adjust the reward value and calculate the weight based on the real-time system load. The weight adjustment formula is as follows:

[0014] in For load sensitivity coefficient, This represents the current load of the i-th model.

[0015] Optionally, the method further includes: S5 employs a multi-level sensitivity detection mechanism to perform secondary verification of issues, including: S51, a rule-based sensitive word matching model is used for preliminary detection; S52, for issues that are initially identified as sensitive, contextual semantic analysis is performed using a deep learning model to confirm the existence of sensitive information.

[0016] To achieve the above objectives, a second aspect of the present invention provides a large-model routing apparatus incorporating a reward model, comprising: The user input acquisition module is used to acquire user input questions and determine a set of candidate models based on user permissions and preset priority rules. The sensitivity detection module is used to determine the sensitivity of the question using a sensitivity detection model. If sensitive information is detected, a preset answer is returned directly; otherwise, the process proceeds to the next step. The reward evaluation module is used to evaluate the reward of each model in the candidate model set based on the routing function. The routing function is trained by knowledge distillation technology and uses KL divergence as the loss function to optimize the model selection strategy. The dynamic distribution module is used to dynamically distribute the problem to the target model with the highest reward value for inference based on the evaluation result of the routing function, and output the processing result of the target model.

[0017] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0018] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0019] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0020] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by a processor, implements the method described in any one of the first aspects.

[0021] The present invention provides a large model routing method, device, electronic device and storage medium that combines a reward model, which can effectively reduce the computational overhead of large model integration, support dynamic routing selection of heterogeneous models, and improve system resource utilization and response efficiency.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a large model routing method combining a reward model, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a large model routing device that combines a reward model, as provided in an embodiment of the present invention. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0025] This invention provides a large-model routing method that combines a reward model. Figure 1 This is a flowchart illustrating a large-model routing method combining a reward model, provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S1: Obtain the question input by the user and determine the candidate model set according to the user's permissions and preset priority rules.

[0026] Specifically, in this application proposal, the step of "obtaining the user's input question and determining the candidate model set according to the user's permissions and preset priority rules" is a key preliminary step in the entire large model routing system. Its technical implementation involves user identification, permission verification, rule matching, and a dynamic filtering mechanism for the model set.

[0027] At the technical implementation level, this step first obtains user input through the user authentication module. The system extracts user identifiers (such as user ID, role tags, access tokens, etc.) for subsequent permission checks. The system's built-in permission control policies are typically implemented as a rule engine, supporting multi-level permission configurations based on user roles, business scenarios, access frequency, and other dimensions. For example, a regular user might only be able to access the basic model set. Advanced users or specific business systems can access an extended collection containing specialized models. In addition, the system also supports the setting of priority rules. For example, in some high-priority tasks, the system will prioritize models with low latency and fast response, rather than simply relying on model performance.

[0028] At the parameter level, access control policies are typically stored in rule tables, containing fields such as user_role, model_access_level, task_type, and priority_flag. Priority rules can be based on model response time. Reasoning resource consumption Historical task matching degree The ranking is based on weighted indicators. For example, the priority calculation formula can be:

[0029] in, These are adjustable weight parameters used to balance the impact of different metrics on model priority.

[0030] At the application level, this step is widely applicable to intelligent service platforms with multiple tenants and business scenarios. For example, in intelligent customer service systems in the insurance industry, different users (such as ordinary customers, internal employees, and partners) have different access permissions and task priority requirements for models. The system needs to dynamically adjust the candidate model set according to the user's identity to ensure service compliance and response efficiency.

[0031] The technical effect of this step is that, by combining access control and priority rules, the system can efficiently select a set of models suitable for the current user and task while ensuring data security and business compliance. This provides a precise input space for the selection of models for subsequent routing functions, thereby significantly reducing the waste of computing resources caused by invalid model calls and improving the overall system response speed and resource utilization.

[0032] Furthermore, S1 includes: S11. Based on the user's role and access permissions, a set of candidate models is selected using a role-based access control (RBAC) model.

[0033] Specifically, in some implementations, a key preliminary step in the routing and distribution mechanism of this invention is to use a role-based access control (RBAC) model to filter the candidate model set (M) based on the user's role and access permissions. The core technical principle of this step is to dynamically determine the subset of models that the user can access by matching user identity information with predefined permission policies, thereby achieving fine-grained management and security control of model resources.

[0034] In its implementation, the system first obtains user role information (such as administrator, regular user, audit user, etc.) from the user authentication module and then determines permissions based on rules in the access control policy library (such as priority rules, idle polling rules, etc.). Under priority rules, if a user role has high priority (such as VIP user or specific business role), the set of models they can access will prioritize models with high performance and fast response. Under idle polling rules, the system distributes user requests to models with lower load based on the current load status of the models to achieve balanced resource utilization. Furthermore, the system supports multi-level access control, such as fine-grained policy configuration based on departments, positions, or business scenarios, ensuring the compliance and security of model access.

[0035] At the parameter level, RBAC policy libraries typically contain a Role-Permission Mapping Table, the structure of which is as follows: ,in Indicates the first Each role This indicates the set of models that the role can access. The system also supports dynamic policy updates, such as real-time updates of permission rules via API interfaces or a configuration center, to adapt to business changes. Access control response time should be controlled within [timeframe missing]. This is to ensure that the timeliness of subsequent model inference is not affected.

[0036] In application scenarios, this step is widely applicable to multi-tenant systems, enterprise-level AI service platforms, and other scenarios that require access control for models. For example, in the business systems of the insurance industry, different business departments (such as claims, underwriting, and customer service) may only be authorized to access large, specialized models relevant to their specific business to prevent data leaks and resource abuse.

[0037] The technical effect of this step is that by implementing permission isolation and dynamic control for model access through the RBAC model, the system's security is improved, and the waste of computing resources caused by invalid model calls is effectively reduced. This provides an accurate set of candidate models for efficient decision-making in subsequent routing functions, thereby significantly improving the overall system's resource utilization and response efficiency.

[0038] S12, when multiple models meet the permission requirements, the candidate models are sorted according to a preset priority rule, which includes a weighted combination of model response speed, historical accuracy and resource utilization.

[0039] Specifically, in some implementations, when multiple models meet the permission requirements, the system will rank the candidate models based on a preset priority rule. This priority rule is a weighted combination of model response speed, historical accuracy, and resource utilization, used to select the optimal model for the inference task while satisfying permission constraints. This step is a key decision-making mechanism for routing functions in practical applications, and its technical implementation is based on quantitative evaluation of model performance indicators and a dynamic weighting strategy.

[0040] From a technical implementation perspective, the system first uses an access control module to filter out a set of candidate models that users can access. Then, statistical analysis was performed on the performance of each model in historical tasks. Model response speed Historical accuracy is typically measured in milliseconds (ms) and represents the average time it takes for a model to go from receiving input to outputting a result; Expressed as a percentage, the calculation formula is as follows: ,in For the model The number of samples correctly predicted on the test set. Total number of samples; resource utilization rate CPU utilization or memory usage percentage is usually used as an indicator to reflect the consumption of computing resources by the model during inference.

[0041] At the parameter level, the priority ranking formula can be expressed as:

[0042] in, For the model The overall priority score, , , These are the weighting coefficients for response speed, accuracy, and resource utilization, respectively, and they satisfy... . This represents the upper limit of resource usage in the system and is used for normalization.

[0043] In application scenarios, this step is widely applicable to intelligent service platforms with multi-model deployments, such as customer service Q&A, risk assessment, and text generation. In these scenarios, user requests may have different timeliness, accuracy requirements, and resource budgets. Through dynamic weighted sorting, the system can achieve optimal resource allocation while ensuring service quality.

[0044] Furthermore, the technical benefits of this step lie in significantly improving the intelligence level of model routing and the overall operational efficiency of the system. By introducing a weighted ranking mechanism, the system can quickly identify the optimal model from multiple candidate models, thereby reducing unnecessary computational overhead, improving response speed, and achieving more efficient model scheduling in resource-constrained environments. In practical deployments, this technical solution can effectively reduce service latency, improve user experience, and enhance the scalability and stability of the system.

[0045] Step S2: Use a sensitivity detection model to determine the sensitive information in the question. If sensitive information is detected, return the preset answer directly; otherwise, proceed to the next step.

[0046] Specifically, in this application proposal, the step "judging the sensitive information of the question through a sensitivity detection model, and directly returning the preset answer if sensitive information is detected, otherwise proceeding to the next step" is a key pre-processing step in the entire routing system. Its technical implementation is based on the text classification and information filtering mechanism in natural language processing (NLP), aiming to screen the user's input question for sensitivity before model inference, thereby improving the security and compliance of the system.

[0047] At the technical implementation level, sensitivity detection models typically employ deep learning-based text classification architectures, such as BERT, RoBERTa, or custom models based on Transformer. During training, these models use pre-labeled sensitive information datasets covering categories such as privacy information, prohibited content, and sensitive topics. The model learns the semantic features and contextual relationships of the text to determine the sensitivity of the input question. During deployment, the model is embedded as an independent module into the routing system to receive user-input questions. and output the sensitivity classification results. ,in This indicates that sensitive information has been detected. This indicates that nothing was detected.

[0048] At the parameter level, the performance of sensitivity detection models is typically evaluated using metrics such as accuracy, recall, and F1 score. The input text to the model needs to undergo standardized preprocessing, including word segmentation, stop word removal, and standardization of encoding (e.g., UTF-8), and is represented using word vectors or word embeddings (e.g., WordPiece). The confidence threshold for the model output can be set as follows: When the classification confidence exceeds this threshold, the system determines it as a sensitive question and triggers the preset answer return mechanism.

[0049] At the application level, this step is widely applicable to industries with high requirements for data security and compliance, such as finance, healthcare, and government. For example, in an insurance customer service system, if a user's question contains personal identification information (such as ID card number or bank card number), the sensitivity detection model will immediately identify it and return a preset privacy protection prompt, avoiding risks arising from subsequent model processing of sensitive data.

[0050] The technical advantage of this step lies in its ability to effectively prevent the misprocessing and leakage of sensitive data by introducing a sensitive information detection mechanism before routing decisions, thereby improving the system's security and stability. Simultaneously, this mechanism reduces invalid calls to subsequent models, lowers overall computational resource consumption, and improves response efficiency.

[0051] Step S3: Based on the routing function, evaluate the reward of each model in the candidate model set. The routing function is trained using knowledge distillation technology and the KL divergence is used as the loss function to optimize the model selection strategy.

[0052] Specifically, this step involves evaluating the rewards of each model in the candidate model set based on a routing function. The routing function is trained using knowledge distillation techniques and optimized using Kullback-Leibler Divergence as the loss function, thereby improving the accuracy and efficiency of the model selection strategy. This step is one of the core components of achieving efficient and intelligent model routing in this invention proposal.

[0053] At the technical implementation level, routing functions Its purpose is to provide each input question The calculation is assigned to the candidate model The probability of . This function takes the Softmax form, and its expression is:

[0054] in, Indicate the problem With model The matching score is calculated by a pre-trained matching model or feature extractor. The training process of the routing function is based on knowledge distillation, which aligns the outputs of multiple candidate models on the training set with the reward values ​​assigned by the reward model, thereby "distilling" the complex multi-model decision-making process into a lightweight routing model.

[0055] At the parameter level, reward value The reward model is used for each question and model Output results The allocated values ​​are in the following form:

[0056] in Indicates the size of the candidate model set. This represents the number of training problems. The loss function uses KL divergence, which is defined as:

[0057] This loss function measures the probability distribution of the routing function's output. The difference between the routing function and the softmax distribution output by the reward model is minimized so that the routing function can more accurately reflect the performance of each model on different tasks.

[0058] At the application level, this step is suitable for scenarios in business systems that require dynamically selecting the optimal large model for inference, such as customer service, risk assessment, and text generation tasks. In these tasks, the system needs to select the most suitable model from multiple heterogeneous models based on user input and permission configurations. Through this reward evaluation mechanism, the system can significantly reduce the consumption of computing resources while ensuring response quality.

[0059] The technical advantage of this step lies in the fact that by introducing knowledge distillation and KL divergence optimization mechanisms, the routing function can learn the performance characteristics of the model on different tasks, thereby achieving more accurate model selection. Compared with the weighted fusion of all models in existing technologies, this solution only calls the optimal model, significantly improving inference efficiency while avoiding the limitations caused by model structure consistency, and enhancing the flexibility and scalability of the system.

[0060] Furthermore, S3 includes: S31, the knowledge distillation technique calculates the KL divergence loss function by using the reasoning result of each model in the candidate model set as the output of the teacher model.

[0061] Specifically, in this application, knowledge distillation is used in the training process of the routing function. Its core lies in using the inference results of each model in the candidate model set as the output of the teacher model, and guiding the routing function to learn the optimal model selection strategy by calculating the KL divergence loss function. This step is technically implemented based on probability distribution alignment, aiming to enable the routing function to accurately predict which model is most likely to output high-quality results for a given problem.

[0062] In some implementations, each candidate model Input problem Perform inference and output its prediction results. Subsequently, the reward model is based on the question. With model output The degree of matching is used to assign a reward value to each model. This forms a reward vector. ,in This represents the size of the candidate model set. The reward vector reflects the performance of each model on the current problem.

[0063] Furthermore, routing functions By calculating the exponential function of the reward value for each model and normalizing it, a probability distribution is output to represent the problem. The probability of being assigned to each model. Specifically, the routing function is defined as:

[0064] During training, KL divergence is used as the loss function to measure the difference between the probability distribution output by the routing function and the soft-max probability distribution output by the reward model. The loss function is defined as follows:

[0065] The formula for calculating the KL divergence is:

[0066] This step improves the accuracy and consistency of model selection by minimizing the KL divergence, making the output distribution of the routing function as close as possible to the optimal model distribution evaluated by the reward model. In practical applications, this technique can be deployed in business systems to dynamically route user queries to the most suitable model, and is particularly suitable for scenarios involving the integration of multiple heterogeneous models. Its technical advantages include significantly reducing inference costs while improving model output quality, solving problems such as high computational overhead and limited model structure in traditional integration methods.

[0067] S32, the routing function training process adopts a multi-round iterative mechanism, and each round of iteration updates the model selection probability distribution according to the latest reward value.

[0068] Specifically, a multi-round iterative mechanism is employed during the training of the routing function to dynamically optimize the model selection probability distribution. This mechanism is based on the reward learning concept in reinforcement learning. By updating the output probability distribution of the routing function in each iteration based on the reward value obtained by the candidate model on the current problem, the accuracy and efficiency of routing decisions are gradually improved.

[0069] At the technical implementation level, each iteration begins with the routing function as the input problem. From the candidate model set Assign a probability distribution to the middle , indicating a problem Assigned to model The probability of [the problem]. Subsequently, all candidate models [are applied to the problem]. The system performs inference and outputs results. The reward model assigns a reward value to each model based on the degree of match between the output result and the true label. ,in This represents the total number of candidate models. The reward value reflects the performance of the model on the problem and is a key basis for optimizing the routing function.

[0070] At the parameter level, the update of the routing function depends on the KL divergence loss function, which is defined as:

[0071] This loss function measures the probability distribution of the current routing function output. This difference stems from the softmax distribution calculated based on reward values. By minimizing this loss, the routing function learns a policy that more closely approximates the optimal model selection. During training, the Adam optimizer is typically used to ensure model convergence and the acquisition of a stable routing policy.

[0072] At the application level, this multi-round iterative mechanism is suitable for scenarios where multiple heterogeneous large models are dynamically routed in a business system. For example, in an insurance business question-and-answer system, different models may have different advantages in sub-tasks such as legal matters, claims, and product recommendations. By updating the routing function through multiple rounds of iteration, the system can significantly reduce the consumption of computing resources while ensuring service quality.

[0073] The technical advantage of this step lies in the fact that, by introducing a reward model and an iterative update mechanism, the routing function can adaptively learn the optimal model selection strategy, thereby achieving efficient utilization of computing resources while ensuring the quality of model output. Compared with traditional model ensemble methods, this scheme avoids redundant operations of inference for all models, improves the system's response speed and resource utilization, and has significant innovation and practicality.

[0074] Step S4: Based on the evaluation result of the routing function, the problem is dynamically distributed to the target model with the highest reward value for inference, and the processing result of the target model is output.

[0075] Specifically, in this application proposal, the step of "dynamically distributing the problem to the target model with the highest reward value for inference based on the evaluation result of the routing function, and outputting the processing result of the target model" is the core step in achieving efficient and accurate model routing. This step is based on the routing function. The output probability distribution for the user input question. Perform model selection and inference distribution.

[0076] At the technical implementation level, routing functions Essentially, it's a classifier trained based on knowledge distillation, with the user question as its input. The output is the candidate models to which the problem belongs. The probability distribution of the task domain. In actual deployment, the system selects the model with the highest reward value for inference based on this probability distribution. Reward value. The reward value is obtained by evaluating each problem and model combination using the reward model, reflecting the model's performance on a specific problem. The system compares the reward values ​​of all candidate models and selects the model with the highest reward value as the target model, thereby achieving optimal routing for the problem.

[0077] At the parameter level, the training of the routing function relies on the KL divergence loss function. This loss function measures the difference between the output of the routing function and the distribution of reward values. It ensures that the routing function accurately learns the decision logic of the reward model. Furthermore, the system supports parallel deployment of multiple models and a model ensemble. It can include multiple heterogeneous large models, and the inference latency, accuracy, recall and other metrics of each model are all included in the evaluation system of the reward model, thereby achieving dynamic and adaptive model selection.

[0078] At the application level, this step is widely applicable to tasks such as intelligent question answering, text generation, and intent recognition involving multi-model collaboration. For example, in a business system, the questions entered by users may involve multiple professional fields such as law, medicine, and finance. The system uses a routing function to distribute the questions to the models that are best suited for that field, thereby improving response quality and reducing overall computing resource consumption.

[0079] The technical benefits of this step are that it significantly improves the efficiency and accuracy of model inference. By calling only the optimal model instead of all models, inference latency can be reduced by more than 50%, while maintaining good task adaptability and system scalability in multi-model heterogeneous environments. This solves the technical bottlenecks of traditional ensemble methods, such as high computational overhead and limited model structure.

[0080] Furthermore, S4 includes: S41, when the difference in reward values ​​among multiple models is less than a preset threshold, a round-robin strategy is used to select the target model.

[0081] Specifically, when the difference in reward values ​​among multiple models is less than a preset threshold, a round-robin strategy is used to select the target model. This is one of the key optimization mechanisms in the routing module's model selection process in this invention. This step aims to address how to achieve load balancing and improved resource utilization without significantly increasing computational overhead when model performance is similar.

[0082] At the technical implementation level, this step is based on a reward model for the candidate model set. Each model in the current problem The output is evaluated to obtain the corresponding reward value. If the difference between the maximum reward value and the second-best reward value is less than a preset threshold... ,Right now If the performance of multiple models on the task is not significantly different, and the optimal model cannot be clearly distinguished by the reward value, then the system will no longer rely on the output of the reward model for decision-making. Instead, it will adopt a round-robin strategy to select the next candidate model as the target model for inference.

[0083] At the parameter level, preset threshold The settings need to be adjusted based on the model performance fluctuation range in the actual business scenario. Typically, The value of is between [0.01, 0.1], depending on the output scale of the reward model. For example, if the reward model output is a normalized floating-point number (such as the range [0, 1]), then Can be set to This ensures that the polling mechanism is triggered when model performance is similar. Furthermore, the scheduling frequency and model selection order of the polling strategy can be implemented through a queue mechanism, ensuring that each model receives a fair opportunity to be used under the same conditions.

[0084] At the application level, this step is suitable for high-concurrency inference scenarios in multi-model deployment environments, especially when model performance differences are small but system resources are limited. For example, in a business system, multiple large models may perform similarly in common tasks (such as text classification and summary generation). In this case, using a round-robin strategy can effectively distribute the computational load, avoid overloading a single model, and maintain the stability of the overall response quality.

[0085] From a technical perspective, this step effectively reduces the uncertainty of model selection and avoids frequent model switching caused by minor differences in reward values, thereby improving the system's inference efficiency and stability. Simultaneously, the introduction of the polling strategy enhances the system's load balancing capability, improves resource utilization, and reduces hardware costs and energy consumption, demonstrating significant engineering practical value.

[0086] S42, the reward value is dynamically adjusted and the weight is calculated based on the real-time system load. The weight adjustment formula is as follows:

[0087] in For load sensitivity coefficient, This represents the current load of the i-th model.

[0088] Specifically, in some implementations, dynamically adjusting the reward value calculation weights based on real-time system load is one of the key steps in the routing function optimization process proposed in this application. The core technical principle of this step is to introduce the system's runtime load status as a dynamic adjustment factor to weight the reward value output by the reward model, thereby balancing task matching and reasonable allocation of system resources during model selection. Specifically, the system uses a monitoring module to collect the running status of each candidate large model in real time, including but not limited to key performance indicators (KPIs) such as CPU utilization, memory usage, request queue length, and response latency, and maps these indicators to model availability scores. During reward value calculation, this score is used as a weighting factor and weighted in conjunction with the model's predicted reward value on a specific task to generate the final model selection criteria.

[0089] Optionally, the system load sampling frequency can be set to once per second or adaptively adjusted according to business needs to ensure the real-time performance and stability of weight adjustment. Furthermore, the weight adjustment formula can be designed based on the standardized value of the system load, such as incorporating the model's availability score. Compared with the original reward value Combined, a weighted reward value is formed. ,in Representation Model Availability normalized value, For the model in the task The original reward value is calculated based on the model's performance. This formula ensures that, given similar model performance, models with lower loads and more abundant resources are prioritized, thereby improving the overall system's throughput and response efficiency.

[0090] In practical applications, this step is suitable for scenarios where multiple models are deployed in parallel within a business system. Especially in high-concurrency, multi-task business environments, it effectively alleviates model resource contention and improves system stability. Through dynamic weight adjustment, the routing function can perceive and respond to the model resource status during both the training and inference phases, thereby optimizing system resource utilization while ensuring task quality.

[0091] Step S5: A multi-level sensitive detection mechanism is used to perform secondary verification of the problem.

[0092] Specifically, in some implementations, the step of "using a multi-level sensitive detection mechanism to perform secondary verification of the problem" in this application proposal is a key security and compliance control step in the routing system before model distribution. This mechanism uses a layered detection strategy to verify the user-input problem. Performing sensitivity assessments ensures that inputs that may contain privacy, illegal, or inappropriate content are filtered out before model inference, thereby improving the security and stability of the system.

[0093] At the technical implementation level, this sensitivity detection mechanism typically consists of multiple sub-models, including but not limited to a rule-based sensitive word matching module, a deep learning-based semantic sensitivity analysis model, and a context-aware sensitive information recognition model. The rule module uses a predefined sensitive word library for regular expression matching, while the semantic model is fine-tuned based on pre-trained language models such as BERT and RoBERTa to identify potential sensitive intentions or implicit information in the input. The context model further combines dialogue history or user identity information to determine whether the current question is sensitive in a specific context.

[0094] In terms of parameter metrics, the output of a sensitivity detection model is typically a confidence vector. ,in This represents a sensitive category (such as privacy, illegality, inappropriate speech, etc.), with each dimension ranging from [0,1], representing the probability that the issue belongs to the corresponding sensitive category. The confidence level for any category... (For example When this occurs, the system will trigger a sensitive response mechanism, returning a preset compliant answer or a denial-of-service response.

[0095] At the application level, this mechanism is widely used in industries with high requirements for data security and compliance, such as finance, insurance, and government. For example, in an insurance customer service system, users may enter questions containing personal identification information or sensitive business content. The system needs to perform checks before routing to prevent the leakage or improper handling of sensitive information.

[0096] From a technical perspective, this multi-level sensitive detection mechanism significantly improves the system's security and compliance, preventing sensitive issues from being incorrectly distributed to models lacking processing capabilities, thereby reducing the risk of data leakage and legal compliance. Simultaneously, through its layered design, the mechanism balances detection accuracy and response speed, ensuring highly sensitive information identification without impacting overall routing efficiency.

[0097] Furthermore, S5 includes: S51 uses a rule-based sensitive word matching model for preliminary detection.

[0098] Specifically, in some implementations, the step "using a rule-based sensitive word matching model for preliminary detection" is a crucial preprocessing step in this application to ensure the compliance of the model's output content. The core technical principle of this step is to use a pre-defined sensitive word database and matching algorithm to analyze the user-input question... The system performs real-time scanning and analysis to identify whether it contains sensitive information that violates content security policies, such as keywords or phrases related to politics, pornography, violence, or illegality. Based on a rule engine, the model employs efficient text matching techniques such as regular expression matching, TrieTree structures, or the Aho-Corasick multi-modal matching algorithm to ensure rapid identification of large-scale vocabulary with low latency.

[0099] In its implementation, the sensitive word matching model first loads a pre-processed sensitive word library, typically stored in UTF-8 encoding, containing multiple hierarchical categories of sensitive words, each corresponding to a different sensitivity type. In practical applications, this model can be deployed at the front end of the routing module as the first filtering mechanism. When the user inputs a question... At that time, the system will The words are segmented according to the word segmentation rules, and then matched word by word or phrase by phrase with the sensitive word database. If a match is successful, a response is made according to the preset strategy (such as returning a preset answer, denying service, logging, etc.) to avoid generating illegal outputs in the subsequent model inference process.

[0100] At the parameter level, the sensitive word database is typically updated daily or hourly to ensure the matching model can respond promptly to the latest policy changes. The time complexity of the matching algorithm is ideally O(n log n). ,in The length of the input question should be specified in characters. Furthermore, the recall and false positive rate of the sensitive word matching model are key performance indicators, typically evaluated using a confusion matrix. A recall rate of at least 98% and a false positive rate below 2% are required to balance security and user experience.

[0101] This step is widely used in business scenarios with high content compliance requirements, such as finance, government affairs, and media. For example, in an insurance customer service system, users may ask questions involving sensitive topics. The system needs to quickly identify and respond to these questions before routing to prevent the spread of sensitive information. Through this step, this application proposal achieves preliminary content security assurance before model inference, reduces the compliance risks of subsequent model processing, and avoids the waste of resources caused by inference on all models, thereby improving the overall system efficiency and security.

[0102] S52, for issues that are initially identified as sensitive, contextual semantic analysis is performed using a deep learning model to confirm the existence of sensitive information.

[0103] Specifically, in this application, the step of "conducting contextual semantic analysis through a deep learning model to confirm the existence of sensitive information for questions that are initially identified as sensitive" is a crucial step in achieving an efficient and secure large-scale model routing mechanism. The core objective of this step is to further verify whether the question contains sensitive information through deep semantic analysis, thereby determining whether to directly return a preset answer, avoiding routing sensitive questions to inappropriate models for processing, and ensuring the compliance and security of the system.

[0104] At the technical implementation level, this step employs a deep learning-based contextual semantic analysis model, typically a variant of a pre-trained language model (such as BERT, RoBERTa, etc.), fine-tuned for a specific sensitive information recognition task. The model input is the user's original question. The model also includes sensitivity markers output by the initial detection module. The model uses a multi-layer attention mechanism through the encoder to process the problem and extract semantic feature vectors. It also considers the context (such as entities, keywords, and syntactic structures in the question) to identify sensitive information. The output is the sensitivity confidence score. ,like (in If the preset threshold is used (usually set to 0.8), it is considered a sensitive question, and the system will return a preset answer and no longer perform model routing.

[0105] At the parameter level, the performance of sensitivity analysis models is typically evaluated using metrics such as accuracy, recall, and F1 score. The model is trained using a cross-entropy loss function, and the input data must be labeled with sensitive and non-sensitive categories. Furthermore, the response latency during model inference should be controlled within a specified timeframe. ms, to ensure the real-time performance of the overall routing process.

[0106] At the application level, this step is widely applicable to industries with high data sensitivity requirements, such as finance, healthcare, and government. For example, in insurance, users may input questions involving personal privacy or sensitive events. The system needs to perform semantic-level sensitivity verification before routing to prevent sensitive information from being improperly handled or leaked.

[0107] The technical advantage of this step lies in its significant improvement in the accuracy and robustness of sensitive information identification by introducing a semantic analysis mechanism, thus avoiding misjudgments based on keyword matching. Simultaneously, this step effectively reduces the resource consumption of subsequent model inference, improving the overall efficiency and security of the system.

[0108] To achieve the above embodiments, the present invention also proposes a large model routing device that combines a reward model. Figure 2 This is a schematic diagram of a large-model routing device combining a reward model, provided as an embodiment of the present invention. Figure 2 As shown, the device includes: The user input acquisition module 100 is used to acquire the question input by the user and determine the candidate model set according to the user's permissions and preset priority rules. Sensitivity detection module 200 is used to judge the sensitive information of the question through a sensitivity detection model. If sensitive information is detected, a preset answer is returned directly; otherwise, the process proceeds to the next step. The reward evaluation module 300 is used to evaluate the reward of each model in the candidate model set based on the routing function. The routing function is trained by knowledge distillation technology and uses KL divergence as the loss function to optimize the model selection strategy. The dynamic distribution module 400 is used to dynamically distribute the problem to the target model with the highest reward value for inference based on the evaluation result of the routing function, and output the processing result of the target model.

[0109] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0110] To implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0111] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0112] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0113] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0114] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0115] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0116] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0117] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0118] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0120] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0122] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0123] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0124] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0125] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A large-scale model routing method combining a reward model, characterized in that, Includes the following steps: S1: Obtain the user's input question and determine the candidate model set based on user permissions and preset priority rules; S2, the question is judged for sensitive information by a sensitivity detection model. If sensitive information is detected, the preset answer is returned directly; otherwise, proceed to the next step. S3, evaluate the reward of each model in the candidate model set based on the routing function. The routing function is trained by knowledge distillation technique and uses KL divergence as the loss function to optimize the model selection strategy. S4. Based on the evaluation result of the routing function, the problem is dynamically distributed to the target model with the highest reward value for inference, and the processing result of the target model is output.

2. The method according to claim 1, characterized in that, The process involves obtaining user input and determining a set of candidate models based on user permissions and preset priority rules, and also includes: S11, Based on the user's role and access permissions, use a role-based access control (RBAC) model to filter the set of candidate models; S12, when multiple models meet the permission requirements, the candidate models are sorted according to a preset priority rule, which includes a weighted combination of model response speed, historical accuracy and resource utilization.

3. The method according to claim 1, characterized in that, The candidate model set is evaluated for rewards based on a routing function, which is trained using knowledge distillation techniques and optimized using KL divergence as a loss function. The evaluation also includes: S31, the knowledge distillation technique optimizes the model selection strategy by using the inference result of each model in the candidate model set as the output of the teacher model, calculating the KL divergence loss function. The calculation formula is: ， in, It is a loss function. It is a student model. It is a model The output is the raw logits, without processing. The original score, It is a teacher model. The teacher model output process The probability distribution after that; S32, the routing function training process adopts a multi-round iterative mechanism, and each round of iteration updates the model selection probability distribution according to the latest reward value.

4. The method according to claim 1, characterized in that, Based on the evaluation result of the routing function, the problem is dynamically distributed to the target model with the highest reward value for inference, and the processing result of the target model is output. The method also includes: S41, when the difference in reward values ​​of multiple models is less than a preset threshold, a round-robin strategy is used to select the target model; S42, dynamically adjust the reward value and calculate the weight based on the real-time system load. The weight adjustment formula is as follows: in For load sensitivity coefficient, This represents the current load of the i-th model.

5. The method according to claim 1, characterized in that, Also includes: S5 employs a multi-level sensitivity detection mechanism to perform secondary verification of issues, including: S51, a rule-based sensitive word matching model is used for preliminary detection; S52, for issues that are initially identified as sensitive, contextual semantic analysis is performed using a deep learning model to confirm the existence of sensitive information.

6. A large-scale routing device combining a reward model, characterized in that, include: The user input acquisition module is used to acquire user input questions and determine a set of candidate models based on user permissions and preset priority rules. The sensitivity detection module is used to determine the sensitivity of the question using a sensitivity detection model. If sensitive information is detected, a preset answer is returned directly; otherwise, the process proceeds to the next step. The reward evaluation module is used to evaluate the reward of each model in the candidate model set based on the routing function. The routing function is trained by knowledge distillation technology and uses KL divergence as the loss function to optimize the model selection strategy. The dynamic distribution module is used to dynamically distribute the problem to the target model with the highest reward value for inference based on the evaluation result of the routing function, and output the processing result of the target model.

7. The apparatus according to claim 6, characterized in that, The user input acquisition module is used for: Based on the user's role and access permissions, a set of candidate models is selected using a role-based access control (RBAC) model. When multiple models meet the permission requirements, the candidate models are sorted according to a preset priority rule, which includes a weighted combination of model response speed, historical accuracy and resource utilization.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images