Routing decision model training method, routing request method, electronic equipment and medium

Abstract

The invention discloses a routing decision model training method, a routing request method, electronic equipment and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a pre-constructed training data set and a to-be-trained routing decision model; if the small model answer result meets a preset minimum expected condition, a label corresponding to the training interaction data is set to be a first preset value indicating to route the request to the edge small model, and the preset minimum expected condition includes that the result satisfaction degree corresponding to the small model answer result is larger than or equal to a preset minimum expected satisfaction degree; if the answer result of the small model does not meet the preset lowest expected condition, setting a label corresponding to the training interaction data as a second preset value for indicating to route the request to the cloud large model; and training a routing decision model based on the training interaction data and the label corresponding to the training interaction data. According to the invention, the utilization rate of end-side resources is improved, and the balance degree of cloud resources and end-side resource allocation is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a routing decision model training method, a request routing method, an electronic device and a medium. BACKGROUND

[0002] In recent years, large language model (LLM) technology has developed rapidly, and with its powerful text understanding and generation capabilities, it has become an important breakthrough in the field of artificial intelligence. However, cloud-based LLM services generally have high response delays and large computational overheads, making it difficult to meet the needs of real-time interactive applications. Therefore, lightweight models with smaller parameter sizes and higher computational efficiency have gradually emerged and can be deployed on edge devices such as mobile phones and embedded devices. The "end-cloud collaboration" architecture in which cloud large models and edge small models work together has therefore received widespread attention.

[0003] Common end-cloud collaboration request routing mechanisms rely on direct comparison of the capabilities of large and small models. Such methods compare the quality of responses to the same request by large and small edge models by constructing a test set, and then develop a routing strategy: if the response quality of the edge model meets the standard of the large model, the request is processed locally; otherwise, it is forwarded to the cloud. This judgment is usually based on human annotation or automated evaluation indicators (such as accuracy and fluency), with the goal of digesting as many requests as possible on the edge side without significantly affecting user experience.

[0004] However, such routing methods rely heavily on the performance gap between models, and small edge models have limited performance on complex tasks, resulting in a large number of interactive requests being forwarded to the cloud large model, insufficient utilization of edge resources, and unbalanced allocation of cloud and edge resources, making it difficult to achieve efficient collaborative reasoning.

[0005] Therefore, how to improve the utilization of edge resources to improve the balance of cloud and edge resource allocation is a technical problem that needs to be solved. SUMMARY

[0006] The main purpose of the present application is to provide a routing decision model training method, a request routing method, an electronic device and a medium, which aims to solve the problem of how to improve the utilization of edge resources to improve the balance of cloud and edge resource allocation.

[0007] To achieve the above purpose, the present application provides a routing decision model training method, which comprises:

[0008] obtaining a pre-constructed training data set and a to-be-trained routing decision model, wherein the training data set comprises training interaction data;

[0009] if the small model answer result meets the preset minimum expectation condition, setting the label corresponding to the training interaction data as a first preset value indicating that the request is routed to the edge small model, wherein the preset minimum expectation condition includes that the result satisfaction degree corresponding to the small model answer result is greater than or equal to the preset minimum expectation satisfaction degree, and the small model answer result is an answer result output after inputting the training interaction data into the edge small model;

[0010] if the small model answer result does not meet the preset minimum expectation condition, setting the label corresponding to the training interaction data as a second preset value indicating that the request is routed to the cloud large model;

[0011] training the routing decision model based on the training interaction data and the label corresponding to the training interaction data until a preset training end condition is met to obtain a trained routing decision model.

[0012] In an embodiment, after the step of obtaining the pre-constructed training data set and the routing decision model to be trained, the method further includes:

[0013] respectively inputting the training interaction data into the cloud large model and the edge small model to output a large model answer result and a small model answer result;

[0014] respectively inputting the large model answer result and the small model answer result into a preset adjudication model to output a large model answer result satisfaction degree and a small model answer result satisfaction degree;

[0015] if any of the conditions that the large model answer result satisfaction degree is less than the preset minimum expectation satisfaction degree, the small model answer result satisfaction degree is greater than or equal to the large model answer result satisfaction degree, and the small model answer result satisfaction degree is greater than or equal to the preset minimum expectation satisfaction degree is met, it is determined that the small model answer result meets the preset minimum expectation condition;

[0016] if all of the conditions that the large model answer result satisfaction degree is less than the preset minimum expectation satisfaction degree, the small model answer result satisfaction degree is greater than or equal to the large model answer result satisfaction degree, and the small model answer result satisfaction degree is greater than or equal to the preset minimum expectation satisfaction degree are met, it is determined that the small model answer result does not meet the preset minimum expectation condition.

[0017] In an embodiment, the routing decision model includes a probability prediction network and a decision layer, and the step of training the routing decision model based on the training interaction data and the label corresponding to the training interaction data includes:

[0018] training the probability prediction network based on the training interaction data and the label corresponding to the training interaction data;

[0019] In each round of iterative training of the probability prediction network, an intermediate probability prediction network obtained in the current round of iterative training is obtained;

[0020] Iterate through each preset candidate classification threshold and evaluate the performance evaluation index value corresponding to the intermediate routing decision model through a preset validation dataset. The intermediate routing decision model is a model composed of an intermediate probability prediction network and an intermediate decision layer connected in sequence. The intermediate decision layer is a decision layer whose classification threshold is set to the candidate classification threshold of the iteration.

[0021] The optimal intermediate classification threshold is selected from the candidate classification thresholds based on the values ​​of each performance evaluation index.

[0022] After the preset training termination condition is met, the target optimal classification threshold is selected from all intermediate optimal classification thresholds.

[0023] The target probability prediction network and the target decision layer, which are connected in sequence, are defined as the route decision model after training. The target probability prediction network is the probability prediction network obtained when the preset training termination condition is met, and the target decision layer is the decision layer with the classification threshold set to the target optimal classification threshold.

[0024] In one embodiment, the step of selecting the intermediate optimal classification threshold from candidate classification thresholds based on various performance evaluation index values ​​includes:

[0025] The maximum value among all performance evaluation index values ​​is determined as the intermediate optimal evaluation index value;

[0026] The candidate classification threshold corresponding to the intermediate optimal evaluation index value is determined as the intermediate optimal classification threshold.

[0027] In one embodiment, the step of selecting the target optimal classification threshold from all intermediate optimal classification thresholds includes:

[0028] The maximum value among all intermediate optimal evaluation index values ​​is determined as the target optimal evaluation index value;

[0029] The intermediate optimal classification threshold corresponding to the optimal evaluation index value of the target is determined as the optimal classification threshold of the target.

[0030] In one embodiment, the step of evaluating the performance evaluation index value corresponding to the intermediate routing decision model using a preset validation dataset includes:

[0031] The average probability of successfully solving the problem, the probability of routing to the smallest model, and the average inference latency were evaluated using a pre-set validation dataset.

[0032] Obtain the application scenarios of the routing decision model and the corresponding fusion weights for the application scenarios. The fusion weights include a first weighting factor for weighting the average probability of successfully solving the problem, a second weighting factor for weighting the probability of routing to the small model, and a third weighting factor for weighting the average inference delay.

[0033] The performance evaluation index value corresponding to the intermediate routing decision model is calculated by weighted summation of the average probability of successful problem solving, the probability of routing to a small model, and the average reasoning delay based on the fusion weight.

[0034] In addition, to achieve the above object, the application also provides a request routing method, the request routing method comprising:

[0035] receiving an interaction request of a user, and obtaining interaction data according to the interaction request;

[0036] inputting the interaction data into a pre-trained routing decision model, and outputting a decision result, wherein the routing decision model is a model trained by the routing decision model training method described above;

[0037] if the decision result indicates that the interaction request is routed to a cloud large model, inputting the interaction data into the cloud large model, and outputting an interaction result;

[0038] if the decision result indicates that the interaction request is routed to an edge small model, inputting the interaction data into the edge small model, and outputting an interaction result.

[0039] In an embodiment, the routing decision model comprises an encoding layer, a fusion layer, a prediction layer, and a decision layer, and the step of inputting the interaction data into the pre-trained routing decision model and outputting a routing decision result comprises:

[0040] encoding the interaction data by the encoding layer to obtain a first encoding vector, and encoding statistical information corresponding to the interaction data to obtain a second encoding vector;

[0041] fusing the first encoding vector and the second encoding vector by the fusion layer to obtain a target encoding vector;

[0042] converting the target encoding vector into a context semantic representation in a high-dimensional space by the prediction layer to obtain a predicted competence probability vector, and performing activation processing on the predicted competence probability vector to obtain a predicted competence probability;

[0043] comparing, by the decision layer, whether the predicted competence probability is greater than or equal to a set classification threshold, and if yes, outputting a first preset value as the routing decision result; and if no, outputting a second preset value as the routing decision result.

[0044] In an embodiment, the interaction data is multi-modal data, the encoding layer comprises an encoding block for encoding each type of modal data, and the step of encoding the interaction data by the encoding layer to obtain a first encoding vector comprises:

[0045] Each kind of modal data in the interaction data is encoded respectively by corresponding encoding blocks in the encoding layer, to obtain a first encoding vector corresponding to each kind of modal data.

[0046] In an embodiment, the method is applied to an edge device connected to a cloud server, a cloud large model is deployed in the cloud server, and an edge small model is deployed in the edge device. The step of inputting the interaction data into the cloud large model to output the interaction result includes:

[0047] The interaction request is forwarded to the cloud server, so that the cloud server inputs the interaction data corresponding to the interaction request into the cloud large model to output the interaction result, and returns the interaction result to the edge device.

[0048] In addition, to achieve the above-mentioned purposes, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned method.

[0049] In addition, to achieve the above-mentioned purposes, the present application also provides a readable storage medium, which is a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned method.

[0050] The present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned method.

[0051] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0052] The application effectively identifies interactive requests that can be independently processed by edge small models by introducing a label setting mechanism based on small model answer result satisfaction, thereby significantly improving end-side resource utilization and alleviating the problem of imbalance in cloud-side and end-side resource allocation. Specifically, when the small model answer result meets the preset minimum expectation condition, the label corresponding to the training data is set to a first preset value, indicating that the request is routed to the edge small model for processing. Otherwise, if it does not meet the condition, the label corresponding to the training data is set to a second preset value, indicating that the request is routed to the cloud-side large model for processing. This labeling strategy makes routing decisions based on the performance of small models in actual interactive tasks, avoiding the misjudgment problem caused by relying on static capability comparison between models in traditional methods, allowing a large number of requests that can be processed by edge small models to be correctly identified and routed to the end-side, thereby improving end-side resource utilization. Further, by training the routing decision model using the above labeled data, the routing decision model can gradually acquire the ability to identify requests within the capability range of edge small models during the training process, so that in actual application, using the trained routing decision model for routing decision can route more tasks to edge small models for processing, fully utilize the capabilities of edge small models, reduce dependence on cloud-side large models, alleviate cloud-side load pressure, and achieve efficient balancing of end-side and cloud-side resources. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0055] Figure 1 A schematic diagram of a large model deployed on an end-side for processing requests in the prior art;

[0056] Figure 2 A schematic diagram of a large model deployed on a cloud-side for processing requests in the prior art;

[0057] Figure 3 A flowchart of a first embodiment of the routing decision model training method of the present application;

[0058] Figure 4 A schematic diagram of the routing decision model architecture involved in an embodiment of the routing decision model training method of the present application;

[0059] Figure 5 A schematic diagram of the overall flow involved in an embodiment of the routing decision model training method of the present application;

[0060] Figure 6 The classification threshold search process schematic diagram involved in the routing decision model training method embodiment of the present application;

[0061] Figure 7 The request routing mode schematic diagram involved in the request routing method embodiment of the present application;

[0062] Figure 8 The device structure schematic diagram of the hardware running environment involved in the routing decision model training method device in the embodiment of the present application.

[0063] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0064] In order to make the above object, features and advantages of the present application more apparent, clear and complete, the technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0065] In recent years, LLM models have shown strong capabilities in text understanding and text generation, and are an important direction for the development of new generations of artificial intelligence. Large Multimodal Model (LMM) based on the development of LLM has also made significant progress and breakthroughs, opening up a new technology paradigm for natural interaction between humans and machines. More and more human-computer interaction applications use the powerful capabilities of LMM to build multimodal agents to achieve human intent understanding and accurate responses, such as AI (Artificial Intelligence) education, AI assistants, etc.

[0066] However, current methods mostly rely on the powerful LMM in the cloud, and models with a parameter quantity greater than a certain value (such as 10B), such as GPT-4 (Generative Pre-trained Transformer 4), which makes the inference delay very long and the token cost very large. These factors hinder the application of LMM in natural human-computer interaction. However, in practical applications, most of the questions that users ask AI are simple instructions, and many times do not depend on the powerful LMM in the cloud, and a small model on the edge can respond to simple instructions. Therefore, there are currently many small multimodal models (SMMs) suitable for end-side deployment, which are models with a parameter quantity less than or equal to a certain value (such as 10B), such as InternVL2_5-1B, phi4-7B, Qwen-7B, etc. These SMMs can be well deployed in end-side computing platforms such as local computers, VR (Virtual Reality) headsets, and mobile phones. Therefore, to maximize the application of multimodal models in the field of natural human-computer interaction, we should make greater use of these small models that can be deployed on the edge.

[0067] However, the existing interaction request processing methods either refer to Figure 1 , in which the end-side computing device transmits the user request to the cloud, accesses the large language model using the cloud server cluster in the cloud to obtain an answer; or refer to Figure 2 , in which the answer is directly obtained using the large language model deployed on the end side. These two schemes cannot balance the computing resources for simple and difficult problems. For the scheme that calls the cloud model, simple problems consume unnecessary computing resources and have relatively slow response speed. For the scheme that directly calls the end-side model, the deployed large language model has a small parameter quantity and poor problem-solving ability for complex problems.

[0068] Even if the end-cloud collaborative mechanism with routing strategy is introduced, the traditional routing decision scheme mostly compares the problem-solving ability of the large model and the small model to determine whether the answer of the small model is stronger than that of the large model, which makes a high proportion of problems routed to the large model, leading to an imbalance in the allocation of cloud resources and end-side resources, and the inability to fully utilize the capabilities of the end-side model.

[0069] Based on this, the main solution of the present application is: obtaining a pre-constructed training data set and a to-be-trained routing decision model, wherein the training data set includes training interaction data; if the small model answer result meets the preset minimum expectation condition, setting the label corresponding to the training interaction data as a first preset value indicating that the request is routed to the edge small model, wherein the preset minimum expectation condition includes that the result satisfaction degree corresponding to the small model answer result is greater than or equal to the preset minimum expectation satisfaction degree, and the small model answer result is the answer result obtained by inputting the training interaction data into the edge small model; if the small model answer result does not meet the preset minimum expectation condition, setting the label corresponding to the training interaction data as a second preset value indicating that the request is routed to the cloud large model; training the routing decision model based on the training interaction data and the label corresponding to the training interaction data until the preset training end condition is met to obtain a trained routing decision model.

[0070] The present application sets the label as the first preset value when the small model answer result meets the preset minimum expectation condition, indicating that the request is routed to the edge small model for processing; otherwise, if it does not meet, the label is set to the second preset value, indicating that the request is routed to the cloud large model for processing. This labeling strategy makes routing decisions based on the performance of the small model in actual interaction tasks, avoiding the misjudgment problem caused by relying on the static ability comparison between models in traditional methods, allowing a large number of requests that can be handled by edge small models to be correctly identified and routed to the end side, thereby improving the utilization rate of end side resources. Further, by training the routing decision model using the above labeled data, the routing decision model can gradually acquire the ability to identify requests within the scope of the edge small model's ability during training, so that in actual application, when using the trained routing decision model for routing decision, more tasks can be routed to the edge small model for processing, fully utilizing the capabilities of the edge small model, reducing the dependence on the cloud large model, relieving the cloud load pressure, and achieving efficient balance of end and cloud resources.

[0071] It should be noted that the execution subject of each embodiment of the routing decision model training method of the present application can be a computing service device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, such as a VR headset. The routing decision model training method of the present application does not make specific limitations on this.

[0072] Based on this, the present application proposes a routing decision model training method of the first embodiment, as shown in Figure 3 The routing decision model training method includes the following steps S10-S40:

[0073] At step S10, a pre-constructed training data set and a to-be-trained routing decision model are obtained, where the training data set includes training interaction data.

[0074] The training data set is constructed for training the routing decision model. Specifically, relevant personnel can construct the training data set based on actual needs. For example, an open-source MMLU data set can be used in a picture-text understanding scenario, and a corresponding medical image and medical report data set can be used in a medical scenario, and the present embodiment does not make a specific limitation in this regard.

[0075] It should be noted that the training data set includes a plurality of training interaction data, and each piece of training interaction data represents a structured interaction request instance. For each piece of training interaction data, a corresponding label can be set through a subsequent step to set the label corresponding to each piece of training interaction data.

[0076] The to-be-trained routing decision model is a trainable machine learning classification model, which takes the interaction data corresponding to the interaction request as input and outputs a binary classification routing decision result. For example, the routing decision model includes a probability prediction network and a decision layer connected in sequence. The probability prediction network is used to calculate a competence probability value representing whether the edge small model can successfully process the request based on the input interaction data. The decision layer is used to compare the probability value with a set classification threshold value and output a final binary classification routing decision result, i.e., routing to the edge small model or the cloud large model.

[0077] Further, the probability prediction network can include an encoding layer, a fusion layer and a prediction layer connected in sequence. The encoding layer is configured to encode the original input data and / or the corresponding statistical information, respectively. The original input data refers to the input data received by the model, such as text, image or other modal original information; the statistical information refers to low-dimensional numerical features extracted from the original data for quantifying the content complexity and processing requirements, such as the number of words, the number of special characters, the number of sentences in the text modal, the length and width pixel values, color types (RGB / monochrome) in the image modal, etc. The encoding layer can select appropriate pre-training encoders and linear transformation layers based on different modalities: for example, text data can be extracted by a BERT (Bidirectional Encoder Representations from Transformers) text encoder to obtain semantic vectors and then reduced by a linear layer; image data can be extracted by a ViT (Vision Transformer) visual encoder to obtain features and then processed by a linear layer; audio data can be extracted by a Whisper (Web-scale Supervised Pre-training for Speech Recognition) model audio encoder to obtain features and then processed by a linear layer, and the statistical information is usually directly mapped to an encoded vector by a linear transformation layer.

[0078] Preferably, the encoding process of single modal data can analyze the context semantic information of each modal. For different scenarios of routing tasks, not only does the model need to understand the difficulty of the input content, but also the complexity of the input content. For example, for text input, the more the number of words, the slower the model inference speed, and for time-requirement scenarios, a small model will be more likely to be called. Therefore, by using mathematical statistical information of different modalities to represent the statistical complexity of the modalities, the encoding layer encodes the input data and the statistical information corresponding to the input data to enhance the model's perception of the complexity of the input request and the processing requirements. This design enables the model to simultaneously perceive the semantic context and statistical complexity of the input data, providing a more comprehensive information basis for routing decisions.

[0079] The fusion layer is configured to integrate the modal vectors and statistical feature vectors output by the encoding layer, for example, a splicing operation can be used to fuse them into a unified intermediate representation vector.

[0080] The prediction layer takes the intermediate vector output by the fusion layer as input, and calculates the estimated probability value that the edge small model can successfully process the request through a neural network structure, such as a Transformer encoder.

[0081] The decision layer compares the probability value output by the prediction layer with a set classification threshold value, and if the probability value is greater than or equal to the threshold value, a decision result routing to the edge small model is output; otherwise, a decision result routing to the cloud large model is output.

[0082] It should be noted that the routing decision model supports flexible modal expansion and architecture customization. Different model structures and coding strategies can be adapted for different application scenarios. For example, in the graphic text understanding application scenario, the specific structure of the constructed routing decision model is as shown in Figure 4 As shown in the figure, in the coding layer, for the text modal, the BERT model and the linear layer are sequentially used to complete the encoding processing of the text data to obtain the text vector V text ; for the image modal data, the ViT model and the linear layer are sequentially used to complete the encoding processing of the image data to obtain the picture vector V image ; and the statistical information of the input data is encoded by the linear layer to obtain the statistical vector V stat . In the fusion layer, V text , V image and V stat are spliced to obtain the intermediate vector V inter , i.e., V inter = [V text , V image , V stat ]. In the prediction layer, the encoder of the Transformer is used for prediction, the input is Vinter, and the output is the probability Prob that the edge small model can perform the reasoning task: P θ (L satisfy |q, inputs) = σ(ω θ V output +b θ ), ω θ represents the weight parameter, Voutput is the output vector of the last layer of the encoder, and σ is the sigmoid function. In the decision layer, by comparing the competence probability Prob and the classification threshold value τ, when Prob ≥ τ, the classification result indicating routing to the edge small model is output, and otherwise, the classification result indicating routing to the cloud large model is output.

[0083] In step S20, if the small model answer result meets the preset minimum expectation condition, the label corresponding to the training interaction data is set to a first preset value indicating that the request is routed to the edge small model, wherein the preset minimum expectation condition includes that the result satisfaction degree corresponding to the small model answer result is greater than or equal to the preset minimum expectation satisfaction degree, and the small model answer result is the answer result output after inputting the training interaction data into the edge small model;

[0084] For each piece of training interaction data, first input it into the edge small model to perform inference, and output the corresponding small model answer result. The edge small model is a large language model with relatively small parameters, which is suitable for deployment on an end-side edge device, denoted as M edge .

[0085] Subsequently, the satisfaction degree of the answer result is evaluated, which can be obtained by a pre-defined metric or an automated evaluation based on a large language model (such as GPT-4), and the value is within a certain continuous or discrete range, such as [0, 10]. If the satisfaction degree is greater than or equal to a preset minimum expected satisfaction degree, which can be set by relevant personnel according to business needs, for example, it can be set to 7, it is determined that the small model answer result meets the minimum expected condition, and the label of the current training interaction data is set to a first preset value (such as 1), representing that the request is routed to the edge small model for processing.

[0086] Considering that in many application scenarios, the user's demand for response speed is greater than the demand for result quality, especially in real-time interactive environments, the user only needs to get the correct answer, and does not need the agent to give a very beautiful and complete answer. Therefore, the concept of user preset minimum expected satisfaction degree (MES) is introduced, which is the minimum satisfaction threshold that the user expects the agent to achieve. When the score of the small model answer result is not less than MES, the label is set to the first preset value. This strategy can route more simple tasks or very difficult tasks to the small model, greatly reducing the proportion of routing to the cloud large model, and fully exerting the ability of the edge small model.

[0087] It should be noted that the preset minimum expected condition is a condition indicating that the small model answer result meets the user's minimum expectation, specifically, it includes but is not limited to the result satisfaction degree corresponding to the small model answer result being greater than or equal to the preset minimum expected satisfaction degree, that is, it can be understood that at least in the case where the result satisfaction degree corresponding to the small model answer result is greater than or equal to the preset minimum expected satisfaction degree, the label is set to the first preset value.

[0088] Step S30, if the small model answer result does not meet the preset minimum expected condition, the label corresponding to the training interaction data is set to a second preset value indicating that the request is routed to the cloud large model;

[0089] If the result satisfaction degree of the small model answer result does not meet the preset minimum expected condition, the label of the current training interaction data is set to a second preset value (such as 0), representing that the request is routed to the cloud large model for processing. The cloud large model is a large language model with relatively large parameters, which is suitable for deployment on a cloud service device, denoted as M cloud .

[0090] Step S40, based on the training interaction data and the labels corresponding to the training interaction data, the routing decision model is trained until the pre-set training end condition is met to obtain the trained routing decision model.

[0091] After the labels corresponding to the training interaction data are set, the labeled training data can be used to supervise the training of the routing decision model. Specifically, the training interaction data can be used as input, and the corresponding routing label can be used as a supervision signal. The model parameters are optimized by minimizing the cross-entropy loss or a similar objective function. The training continues until the pre-set iteration end condition is met, and the trained routing decision model is obtained.

[0092] The training end condition can be a pre-set condition, such as reaching a predetermined number of iterations, the loss function value being lower than a predetermined threshold, running out of computing resources, reaching a time limit, the accuracy being a predetermined threshold, etc. The present embodiment does not make specific limitations.

[0093] In the actual application stage, when receiving a user-initiated interaction request, the user's actual input interaction data is input into the trained routing decision model. The model performs inference based on the input data and outputs a binary classification decision result. The result indicates that the current request should be routed to an edge small model or a cloud large model for processing. According to the decision result, the corresponding routing operation is performed, thereby realizing the automatic distribution of the request.

[0094] It should be noted that the deployment method of the routing decision model can be flexibly selected according to the actual system architecture. If the model is trained on a cloud device, a lightweight version or inference required parameters of the model can be deployed to an edge device, so that the edge device has the ability to independently perform routing decision. If the model itself is trained on the edge side, the edge device can directly use the locally trained model to perform real-time routing decision.

[0095] Based on the first embodiment of the routing decision model training method of the present application, in the second embodiment of the routing decision model training method of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be repeated hereinafter. On this basis, after the step of obtaining the pre-constructed training data set and the routing decision model to be trained, the method further comprises:

[0096] Step A10, respectively inputting the training interaction data into the cloud large model and the edge small model to output the large model answer result and the small model answer result;

[0097] The large model answer result is the answer result output by the cloud large model after the training interaction data is input into the cloud large model; and the small model answer result is the answer result output by the edge small model after the training interaction data is input into the edge small model.

[0098] Step A20, respectively, the large model answer result and the small model answer result are input into the preset judging model, and the large model answer result satisfaction and the small model answer result satisfaction are output;

[0099] The large model answer result satisfaction is the satisfaction output by the preset judging model after the large model answer result is input into the preset judging model; and the small model answer result satisfaction is the satisfaction output by the preset judging model after the small model answer result is input into the preset judging model.

[0100] The preset judging model can be a large language model, such as GPT4-o, Byte Doubao, Tencent Yuanbao, etc., which evaluates the quality of the input answer result based on a preset unified satisfaction standard and outputs a quantitative satisfaction score, i.e., the large model answer result satisfaction and the small model answer result satisfaction. The satisfaction is generally a discrete or continuous numerical value, which is used to objectively measure the quality of the answer.

[0101] In an exemplary embodiment, the satisfaction score standard adopted by the preset judging model is an integer scale of 1 to 10, wherein:

[0102] 1 point means completely unable to answer the task, the answer content has nothing to do with the question, or refuses to answer;

[0103] 2 points indicate that the attempt to answer deviates from the theme seriously, there are obvious understanding errors, and the information is completely wrong;

[0104] 3 points indicate that the question is partially understood but the answer quality is extremely poor, there are multiple major errors, and the logic is chaotic;

[0105] 4 points indicate that the question is basically understood but the answer is not accurate, there are some important errors, and there is a lack of key information;

[0106] 5 points indicate that the question is understood and relevant answers are given, but the accuracy is insufficient, and there are obvious defects;

[0107] 6 points indicate that the answer is basically correct and can perform the task but the effect is general, reaching the passing line, and there are some small errors;

[0108] 7 points indicate that the answer is accurate and has certain logicality, and can perform the task well, with good quality;

[0109] 8 points indicate that the answer is accurate, detailed and logically clear, with almost no errors, and the quality is excellent;

[0110] 9 points indicate that the answer is perfect, basically consistent with the reference answer, with high accuracy, and the expression is clear and complete;

[0111] 10 points indicate that the answer not only meets the requirements of the reference answer, but also considers more aspects, is more comprehensive and complete, and exceeds expectations.

[0112] It should be noted that the accuracy of the result satisfaction evaluation can be improved by using multiple judge models (such as GPT, Doubao, Yuanbao, etc.) or setting the same judge model to perform satisfaction scoring at different temperature coefficients, and calculating the average value of all obtained satisfaction scores as the final answer result satisfaction of the answer result.

[0113] Step A30, if any of the conditions that the large model answer result satisfaction is less than the preset minimum expected satisfaction, the small model answer result satisfaction is greater than or equal to the large model answer result satisfaction, and the small model answer result satisfaction is greater than or equal to the preset minimum expected satisfaction is met, it is determined that the small model answer result meets the preset minimum expected condition.

[0114] Let the large model answer result satisfaction be Score cloud , and the small model answer result satisfaction be Score edge . When Score edge ≥ MES, Score edge ≥ Score cloud , and Score cloud < MES, it is determined that the small model answer result meets the preset minimum expected condition, indicating that M edge is capable of performing the current task, that is, it is equivalent to Score edge ≥ min(Score cloud , MES) or Score cloud < MES.

[0115] This determination strategy means that when the edge small model itself performs up to standard, is not inferior to the cloud large model, or the large model also cannot handle the request well, it is considered that the edge side has the processing capability, so the request is preferentially routed to the edge small model, thereby improving the utilization rate of the end side resources.

[0116] Step A40, if all of the conditions that the large model answer result satisfaction is less than the preset minimum expected satisfaction, the small model answer result satisfaction is greater than or equal to the large model answer result satisfaction, and the small model answer result satisfaction is greater than or equal to the preset minimum expected satisfaction are met, it is determined that the small model answer result does not meet the preset minimum expected condition.

[0117] Score edge < MES, Score edge < Score cloud , and Score cloudWhen all three conditions in the MES are met, it is determined that the small model answer result does not meet the preset minimum expected condition. This rule ensures that the request is routed to the cloud only when the large model performs well and the small model performs significantly worse, maximizing the effective use of edge computing resources while ensuring service quality.

[0118] Based on the first embodiment and / or the second embodiment of the route decision model training method of the present application, in the third embodiment of the route decision model training method of the present application, the same or similar content as the above-mentioned embodiments one and two can be referred to the above introduction, and will not be repeated hereinafter. On this basis, the route decision model includes a probability prediction network and a decision layer. Based on the training interaction data and the labels corresponding to the training interaction data, the steps of training the route decision model include:

[0119] Step B10, training the probability prediction network based on the training interaction data and the labels corresponding to the training interaction data;

[0120] The probability prediction network is used to predict the competence probability of the edge small model to successfully process the request according to the input training interaction data, and the probability value is usually a continuous scalar; the decision layer is a comparison logic unit based on a set classification threshold, which is used to make a routing classification decision according to the competence probability value output by the probability prediction network, i.e. to judge whether it belongs to the first preset value (routing to the edge small model) or the second preset value (routing to the cloud large model). Specifically, if the predicted competence probability is greater than or equal to the classification threshold, the first preset value is output, otherwise the second preset value is output. The decision process can be represented by the formula:

[0121]

[0122] Where q represents the user's question, <input1,..., inputn> represents the training interaction data input to the model, p is the predicted competence probability, and τ is the classification threshold.

[0123] After the training interaction data is input into the probability prediction network, the probability prediction network outputs a predicted competency probability. A loss value, such as a cross-entropy loss value, between the predicted competency probability and a label corresponding to the training interaction data can be calculated. Based on the loss value, the probability prediction network is iteratively optimized, such as using a stochastic gradient descent (SGD) or Adam optimizer to adjust network weights and biases, so as to continuously reduce the loss value and improve the accuracy of probability prediction. For example, in an application scenario, let the predicted competency probability be Prob, the one-hot encoding of the first preset value be y1, and the one-hot encoding of the second preset value be y0. The loss value Loss between the predicted competency probability and the label corresponding to the training interaction data is calculated by the formula Loss = -(y1·Prob + y0·(1-Prob)). Then, the Adam optimizer is used for gradient update, and based on the gradient, back propagation and parameter update are performed.

[0124] Step B20, in each round of iterative training of the probability prediction network, an intermediate probability prediction network obtained in the current round of iterative training is obtained.

[0125] After each round of training iteration of the probability prediction network ends, the network parameters in the current iteration state are saved to form an intermediate probability prediction network, which is used for subsequent threshold search and model verification.

[0126] Step B30, each preset candidate classification threshold is traversed, and a performance evaluation index value of an intermediate routing decision model corresponding to the candidate classification threshold is evaluated by using a preset verification data set, wherein the intermediate routing decision model is a model composed of the intermediate probability prediction network and an intermediate decision layer connected in sequence, and the intermediate decision layer is a decision layer with the classification threshold set as the candidate classification threshold being traversed.

[0127] A predefined candidate classification threshold set is traversed, such as a set {0.00, 0.05, 0.10, …, 1.00} obtained by uniformly sampling in the interval [0, 1] with a step size of 0.05. For each candidate classification threshold, the current intermediate probability prediction network and a temporary decision layer are connected in series to construct an intermediate routing decision model, wherein the classification threshold of the temporary decision layer is set as the candidate classification threshold being traversed. The intermediate routing decision model is inferred using the preset verification data set, and a performance evaluation index value thereof is calculated to evaluate the performance of the routing decision model under different candidate classification thresholds.

[0128] Step B40, based on the performance evaluation index values, an intermediate optimal classification threshold is selected from the candidate classification thresholds.

[0129] After the performance of the intermediate routing decision models corresponding to all candidate classification thresholds is evaluated, the classification threshold with the best performance in the current training stage is selected according to the performance evaluation index values.

[0130] Step B50, selecting a target optimal classification threshold from all intermediate optimal classification thresholds after meeting a preset training end condition;

[0131] When the training of the probability prediction network meets a preset training end condition (such as reaching a maximum number of iterations, the performance of the model no longer significantly improves, etc.), one of the intermediate optimal classification thresholds generated in all training rounds with the best performance on the validation set can be selected as the final target optimal classification threshold.

[0132] Further, the effectiveness and robustness of the threshold in the actual scene can also be verified by an independent test data set.

[0133] Step B60, determining the target probability prediction network and the target decision layer connected in sequence as a trained routing decision model, wherein the target probability prediction network is the probability prediction network obtained when the preset training end condition is met, and the target decision layer is the decision layer with the classification threshold set as the target optimal classification threshold.

[0134] After the target optimal classification threshold is determined, the probability prediction network meeting the training end condition and the decision layer set as the target optimal classification threshold are connected in sequence to form a final trained routing decision model. This model has the ability to predict the competence probability of the edge small model according to the input interaction data and make reasonable routing decisions accordingly, and can effectively balance the allocation of cloud and end-side resources in actual application to realize efficient end-cloud collaborative inference.

[0135] In a possible implementation, the step of selecting an intermediate optimal classification threshold from the candidate classification thresholds based on the performance evaluation index values includes:

[0136] Step C10, determining the maximum value of all performance evaluation index values as an intermediate optimal evaluation index value;

[0137] In this embodiment, the larger the performance evaluation index value, the better the performance of the model, and therefore the maximum value of all performance evaluation index values is selected as the intermediate optimal evaluation index value.

[0138] Step C20, determining the candidate classification threshold corresponding to the use of the intermediate optimal evaluation index value as the intermediate optimal classification threshold.

[0139] After the intermediate optimal evaluation index value is determined, the corresponding candidate classification threshold value is traced back. That is, the classification threshold value that makes the performance of the intermediate routing decision model optimal during the training process is found and determined as the intermediate optimal classification threshold value. For example, if the intermediate optimal evaluation index value is obtained when the classification threshold value is 0.7, then 0.7 is determined as the intermediate optimal classification threshold value. The purpose of this step is to screen out the classification threshold value with the best performance, so as to further optimize the performance of the routing decision model on this basis in the subsequent training process, and ensure that the model can make more accurate and reasonable routing decisions in actual application.

[0140] In a possible implementation, the step of selecting the target optimal classification threshold value from all the intermediate optimal classification threshold values comprises:

[0141] Step D10, determining the maximum value among all the intermediate optimal evaluation index values as the target optimal evaluation index value.

[0142] In the previous steps, an intermediate optimal evaluation index value is screened out in each round of iterative training, and these values represent the best performance of the model at different training stages. Now, the intermediate optimal evaluation index values are compared to find the maximum value. This maximum value reflects the highest performance level that the model can achieve during the entire training process, and is therefore determined as the target optimal evaluation index value. For example, assuming that the intermediate optimal evaluation index values obtained in multiple rounds of iteration are 0.85, 0.88, 0.90, etc., then 0.90 is the target optimal evaluation index value.

[0143] Step D20, determining the intermediate optimal classification threshold value used corresponding to the target optimal evaluation index value as the target optimal classification threshold value.

[0144] After the target optimal evaluation index value is determined, the corresponding intermediate optimal classification threshold value is found. That is, the classification threshold value that makes the performance of the model reach the highest level during the entire training process is found and determined as the target optimal classification threshold value. For example, if the target optimal evaluation index value is obtained when the classification threshold value is 0.75, then 0.75 is determined as the target optimal classification threshold value. The purpose of this step is to screen out the classification threshold value that makes the performance of the model optimal during the entire training process from all the intermediate optimal classification threshold values, to provide a key parameter for finally determining the decision logic of the routing decision model, and to ensure that the model can make the most accurate routing decisions in actual application.

[0145] In a possible implementation, the step of evaluating the performance evaluation index value corresponding to the intermediate routing decision model by using the preset verification data set comprises:

[0146] Step E10, evaluating the average problem-solving probability, the cost advantage, and the average inference latency of the intermediate routing decision model corresponding to the problem by the preset validation data set;

[0147] The average problem-solving probability (APSP) refers to the average probability that the corresponding result of the response result has a result satisfaction degree not lower than the MES value. Specifically, the APSP can be calculated by the formula , wherein Q t and I t represent the user request and the corresponding input data of the tth validation data, respectively, represents the satisfaction degree of the answer to the problem by the M R to which the tth validation data is routed by the intermediate routing decision model R, and N represents the total number of validation data. The higher the APSP, the stronger the problem-solving capability of the current routing decision model. The value of M R is M edge or M cloud .

[0148] The cost advantage (CA) refers to the proportion of routing the problem to M edge . Specifically, the CA can be calculated by the formula , wherein M Rt represents the model to which the tth validation data is allocated after being routed by R. The higher the CA, the better the current routing decision model R can save computing resources.

[0149] The average inference latency (AIL) refers to the average delay from inputting the problem to finally giving the answer. Specifically, the AIL can be calculated by the formula . The lower the AIL, the faster the current routing decision model can respond to the user request.

[0150] Step E20, obtaining the application scenario of the routing decision model and obtaining the fusion weight corresponding to the application scenario, wherein the fusion weight includes a first weighting factor for weighting the average problem-solving probability, a second weighting factor for weighting the cost advantage, and a third weighting factor for weighting the average inference latency;

[0151] It should be noted that the correspondence between the application scenario and the fusion weight can be pre-set, so as to obtain the fusion weight corresponding to the application scenario based on the correspondence. Generally speaking, the first weighting factor is relatively large for a scenario with higher requirements for model answer quality; the second weighting factor is relatively large for a scenario with more restrictions on computing resources or access to cloud resources; and the third weighting factor is relatively large for a scenario with high requirements for response speed. In an example, the first weighting factor is denoted as a, the second weighting factor is denoted as β, and the third weighting factor is denoted as γ. In a scenario such as a question and answer system or AI creation with higher requirements for model answer quality, the fusion weight is a = 1.0, β = 0.2, and γ = -0.002; in a scenario such as traffic limitation or cloud resource scarcity with more restrictions on computing resources or access to cloud resources, the fusion weight is a = 1.0, β = 0.5, and γ = -0.002; and in a scenario such as intelligent chat or AI interaction with high requirements for response speed, the fusion weight is a = 1.0, β = 0.2, and γ = -0.005. It should be noted that the APSP, CA, and AIL have different dimensions, and therefore the relative sizes of a, β, and γ only represent the adjustment direction and relative importance.

[0152] In step E30, the performance evaluation index value corresponding to the intermediate routing decision model is obtained by weighted sum calculation of the problem success solution average probability, the routing to the small model probability, and the average reasoning delay based on the fusion weight.

[0153] The calculation of the performance evaluation index value RCS can be represented by the formula: RCS(R) = aAPSP(R) + βCA(R) + γAIL(R). Through this weighted sum method, the importance of each index in different application scenarios can be considered comprehensively to obtain a comprehensive performance evaluation index value. This index value can more comprehensively reflect the overall performance of the intermediate routing decision model in a specific application scenario, providing a scientific basis for subsequent optimization and decision-making.

[0154] The embodiment evaluates the performance of the intermediate routing decision model in terms of three key indicators: average probability of successful problem solving (APSP), routing to small model probability (CA), and average inference delay (AIL). The model performance is quantified from three dimensions: task completion quality, resource saving efficiency, and response speed. Further, according to the specific application scenario of the routing decision model, the corresponding fusion weight (i.e., the first weighting factor α, the second weighting factor β, and the third weighting factor γ) is obtained dynamically, so that the performance evaluation process can flexibly adapt to the core needs of different scenarios: in the quality-sensitive scenario, the importance of APSP is highlighted; in the resource-limited scenario, the weight of CA is increased; in the delay-sensitive scenario, the optimization direction of AIL is strengthened. Finally, based on the obtained fusion weight, the three indicators are weighted and fused to generate a comprehensive evaluation indicator value that can comprehensively and balancedly reflect the overall performance of the model in a specific scenario. This mechanism effectively overcomes the limitations of traditional single indicator or fixed weight evaluation methods, making the model selection and threshold optimization process more scenario-specific, thereby significantly improving the resource utilization, response efficiency, and service quality of the end-to-cloud collaborative system while ensuring user experience.

[0155] For example, to help understand the technical concept or technical principle of the routing decision model training method combined with the first embodiment and the second embodiment described above, a specific embodiment is listed. In this specific embodiment, referring to FIG. 1, the routing decision model training process includes: Figure 5

[0156] 1. Construct a suitable training data set according to the application scenario, such as medical scene text analysis (graph + text modal), interactive scene speech recognition (speech + graph modal), etc. Collect or download open source multi-modal data sets from the Internet according to the scene. After obtaining the data set, further construct the data set, obtain the answers of the edge small model and the cloud large model, and the scores of the judge model, and set the training interaction data corresponding to the routing label according to the application scenario.

[0157] 2. Then determine the architecture of the routing decision model Routeformer according to the input modal and the application scenario.

[0158] 3. Use the data set with label setting to iteratively train Routeformer. During the training process, the corresponding fusion weight is obtained according to the application scenario of the model.

[0159] 4. Search for the optimal classification threshold τ through the validation set.

[0160] 5. Apply the trained Routeformer to the scene task to realize the end-to-cloud collaborative invocation of the edge small model and the cloud large model.

[0161] ​Specifically, the setting of the routing label includes defining the problem of model routing, considering a pair of models <M cloud , M edge >, M cloud represents a model with a large number of cloud parameters, such as InternVL3-38B, and Medge represents a model with a small number of edge parameters, such as InternVL2_5-1B. Each model can be defined as M: <Q, I>→Response, which is a mapping function from input to output, Q represents a user question, and I represents input data, which can include multiple modalities such as text, image, audio, etc. The decision-making process of the designed Routeformer is R: <Q, I, M cloud , M edge >→M R , M R is the model selected for routing, i.e., M cloud or M edge .

[0162] The decision-making process R determines the competence of M edge according to the input features. If M edge can meet the conditions, it is preferred, otherwise M cloud is dispatched to ensure the balance between efficient response and satisfaction. The competence determination rules for M edge are defined as follows:

[0163] a. Score edge ≥ min (Score cloud , MES), i.e., when the result satisfaction score of M edge is not less than the smaller value of the result satisfaction score of M cloud and the preset minimum expected satisfaction MES, it means that M e dg e can handle the current task, and M edge will be preferred for response, and the label is set to 1.

[0164] b. Score cloud <MES, i.e., when the score of M cloud is lower than MES, it means that the model in the cloud cannot also solve the input problem well, and M edge can be called for response, and the label is also set to 1.

[0165] Conversely, when the above conditions a and b are not met, it means that M edge cannot handle the current task, and the label is set to 0.

[0166] Based on the above rules, the routing decision label L of R can be obtained, i.e., L = 1 when any one of the above rules is satisfied, i.e., Medge is competent; otherwise, L = 0, i.e., Medge is not competent.

[0167] Further, the constructed data set D is divided into a training set D train and a validation set D valid according to the proportion of the score distribution, so as to balance the score distribution, for subsequent training of the Routeformer. The training set is used for gradient update and parameter update in the training process, and the validation set is used for verifying the performance of the Routeformer parameters and searching for the optimal classification threshold τ.

[0168] The Routeformer needs to undergo multiple rounds of iterative training in the training process. Each round of iteration will undergo a training phase and a validation phase. The optimal classification threshold τ is searched through the validation phase. The overall process is shown in Figure 6 , which includes:

[0169] Training phase: the intermediate probability prediction network C train is obtained through training of each round of iteration D inter . The cross-entropy loss function is used to update the model parameters: Loss = -(y1·Prob+y0·(1-Prob)), y1 and y0 are one-hot encoding of 1 and 0, respectively. The Adam optimizer is used for gradient update in the training process.

[0170] Validation phase: the grid search is performed through D valid to find the optimal performance score indicator value RCS and its corresponding τ value. After obtaining C inter , τ takes an interval of 0.05 in [0-1], and the corresponding RCS score in D valid is calculated, and the optimal RCS and its τ value are recorded. The grid search process of the validation set is performed in each round of iterative training until the optimal RCS value corresponding to C inter and τ value is output at the end of the training. This process can enable the model to dynamically adapt to the optimal τ according to the application scenario requirements during the training process.

[0171] In addition, it is proved in the experimental stage that the results of the validation set can be effectively generalized to the test set D test .

[0172] It should be noted that the above examples are only used to assist in understanding the present embodiment and do not constitute a limitation on the training process of the routing decision model of the present embodiment. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0173] Further, the embodiments of the application also propose a request routing method. The execution subject of each embodiment of the request routing method of the application can be a computing service device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, a mobile phone, etc., or an edge device capable of realizing the above functions, such as a VR headset. Exemplarily, the description and illustration of each embodiment of the request routing method of the application are performed by taking the edge device as the execution subject.

[0174] Based on this, in the request routing method of the first embodiment, the request routing method comprises the following steps S100-S400:

[0175] Step S100, receiving an interactive request of a user, and obtaining interactive data according to the interactive request;

[0176] The interactive request initiated by the user through the client or the application program interface can contain text, image, voice or other modal input data. The structured interactive data is parsed and extracted according to the interactive request for subsequent model processing.

[0177] Further, the interactive data can be preprocessed, such as text segmentation, image normalization or voice feature extraction, to meet the model input requirements.

[0178] Step S200, inputting the interactive data into a pre-trained routing decision model to output a decision result, wherein the routing decision model is a model trained by the routing decision model training method of any one of the above embodiments;

[0179] The preprocessed interactive data is input into the routing decision model which is pre-trained. The routing decision model is obtained by the training method of any one of the above embodiments. Based on the content features and statistical properties of the input data, the internal probability prediction network calculates the probability that the edge small model is competent for the request, and the decision layer outputs the final routing decision result according to the set classification threshold, i.e. classified as "routing to cloud large model" or "routing to edge small model".

[0180] Step S300, if the decision result indicates that the interactive request is routed to the cloud large model, inputting the interactive data into the cloud large model to output an interactive result;

[0181] Step S400, if the decision result indicates that the interactive request is routed to the edge small model, inputting the interactive data into the edge small model to output an interactive result.

[0182] It should be noted that in the present embodiment, the edge device is connected with the cloud server, the edge small model is deployed in the edge device, and the cloud large model is deployed in the cloud server.

[0183] If the decision result output by the routing decision model indicates that the current interaction request needs to be routed to the cloud large model, the interaction request is forwarded to the cloud large model service end deployed in the cloud server through the network; the input data is inferred and calculated by the cloud large model to generate the corresponding interaction result (such as a text answer, an image description or structured data), and the result is returned to the edge device.

[0184] If the decision result output by the routing decision model indicates that the current interaction request can be routed to the edge small model, the interaction data is directly input into the locally deployed edge small model; the edge small model completes real-time inference on the end-side device to generate an interaction result and directly returns to the user interface, thereby realizing low-delay and low-cost localized processing.

[0185] In an application scenario, the request routing process can refer to Figure 7 As shown in the figure, the routing decision maker (Model Router) is used as a core scheduling module to schedule user requests in real time. Specifically, the routing decision maker calls the pre-trained routing decision model (denoted as Routeformer) to intelligently distinguish user requests to determine whether to send the request to the cloud server or directly execute inference on the edge end side. If the Routeformer determines that the request is a simple question, it is routed to the edge device end side for processing. At this time, the interaction data is input into the small model deployed on the end side to perform local inference and directly output the interaction result, thereby realizing low-delay response and saving cloud resources. If the Routeformer determines that the request is a complex or difficult question, the request is forwarded to the cloud server. In the cloud, the interaction data is input into the high-performance large model for deep inference, and the generated interaction result is finally returned to the user end.

[0186] Based on the first embodiment of the request routing method, in the second embodiment of the request routing method, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be described hereinafter. On this basis, the routing decision model includes an encoding layer, a fusion layer, a prediction layer and a decision layer, the step of inputting the interaction data into the pre-trained routing decision model and outputting the routing decision result includes:

[0187] Step A100, the interaction data is encoded by the encoding layer to obtain a first encoding vector, and the statistical information corresponding to the interaction data is encoded to obtain a second encoding vector;

[0188] The interaction data (such as text, image, etc. original input) is extracted by a pre-trained modal-specific encoder, and can be dimensionally adjusted and refined by a linear layer to obtain a first encoding vector containing semantic information. At the same time, the statistical information extracted from the original interaction data (such as text length, image resolution, etc. low-dimensional numerical features) is encoded and mapped by a linear layer to convert it into a second encoding vector.

[0189] Step A200, fusing the first encoding vector and the second encoding vector through a fusion layer to obtain a target encoding vector;

[0190] The first encoding vector and the second encoding vector are received by the fusion layer, and a splicing operation is used to fuse them in the feature dimension to form a new, unified target encoding vector. This vector retains both the deep semantic features of the input data and its statistical attribute information.

[0191] Step A300, converting the target encoding vector into a context semantic representation in a high-dimensional space through a prediction layer to obtain a predicted competency probability vector, and performing activation processing on the predicted competency probability vector to obtain a predicted competency probability;

[0192] The target encoding vector is nonlinearly transformed and context semantic modeled by the prediction layer, such as a Transformer encoder, and is mapped to a semantic representation vector in a high-dimensional space; then the vector is processed by an activation function (for example, using a Sigmoid function), which is compressed and converted into a scalar prediction value representing the competency probability of the edge small model, with a value range of [0, 1].

[0193] Step A400, comparing whether the predicted competency probability is greater than or equal to a set classification threshold through a decision layer, if yes, outputting a first preset value as the routing decision result; if not, outputting a second preset value as the routing decision result.

[0194] The predicted competency probability output by the prediction layer is received by the decision layer, and is compared with a pre-set classification threshold. If the probability value is greater than or equal to the classification threshold, the decision layer outputs a first preset value (such as the number 1 or a specific identifier) as the routing decision result, indicating that the request should be routed to the edge small model for processing; otherwise, a second preset value (such as the number 0 or another identifier) is output, indicating that the request needs to be routed to the cloud large model for processing.

[0195] In one possible implementation, the interaction data is multi-modal data, and the encoding layer includes an encoding block for encoding each type of modal data. The step of encoding the interaction data through the encoding layer to obtain the first encoding vector includes:

[0196] Step B100, encode each kind of modal data in the interaction data respectively by corresponding encoding blocks in the encoding layer, to obtain a first encoding vector corresponding to each kind of modal data.

[0197] The encoding layer includes a plurality of encoding blocks, each of which is responsible for processing one kind of modal data. For example, text modal data is input into a text encoding block (which can be based on a pre-trained language model such as BERT), image modal data is input into an image encoding block (which can be based on a pre-trained visual model such as ViT), and audio modal data is input into an audio encoding block (which can be based on a pre-trained audio model such as Whisper). Each encoding block performs high-level feature extraction and representation learning on the modal data input thereto, and outputs a first encoding vector of the modal semantic features. Finally, a plurality of first encoding vectors equal to the number of input modal types are obtained. In this way, the Routeformer is extended to multi-modal and has good scalability, and can solve the multi-modal input request of the user.

[0198] In addition, an electronic device is also provided in the embodiments of the present application, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the computer program is configured to implement the steps of the method as described above.

[0199] Reference Figure 8 which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can also include, but is not limited to, mobile terminals such as VR headsets, mobile phones, servers, notebook computers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 8 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0200] As Figure 8As shown, the electronic device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0201] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0202] The electronic device provided by the embodiments of the present disclosure adopts the routing decision model training method in the above embodiments, which can solve the problem of how to improve the utilization rate of the end-side resources, so as to improve the balance degree of the allocation of the cloud-side resources and the end-side resources. Compared with the prior art, the electronic device provided by the present disclosure has the same beneficial effects as the routing decision model training method provided by the above embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0203] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0204] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0205] In addition, to achieve the above object, the embodiment of the present application further provides a readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the route decision model training method in the above embodiment.

[0206] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency: radio frequency), etc., or any suitable combination of the above.

[0207] The above computer readable storage medium can be contained in an electronic device; or can exist separately without being assembled into an electronic device.

[0208] The above computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the flow steps of any of the above embodiments.

[0209] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0210] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0211] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0212] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the above-mentioned route decision model training method, and can solve how to improve the utilization rate of the end-side resource, so as to improve the balance degree of the cloud-side resource and the end-side resource allocation. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the present application are the same as those of the route decision model training method provided by the above-mentioned embodiments, which will not be repeated here.

[0213] Further, the embodiments of the present application also provide a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the above method.

[0214] The computer program product embodiment of the present application is basically the same as the above-mentioned route decision model training method, and will not be described here.

[0215] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0216] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software plus the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software sensor, which is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disk, optical disk), including a plurality of instructions for making an electronic device (which can be a mobile phone, computer, server or network device, etc.) execute the method of each embodiment of the present application.

[0218] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for training a routing decision model, characterized in that, The routing decision model training method includes the following steps: Obtain a pre-built training dataset and a routing decision model to be trained, wherein the training dataset includes training interaction data; If the small model's answer meets the preset minimum expectation condition, then the label corresponding to the training interaction data is set to a first preset value indicating that the request is routed to the edge small model. The preset minimum expectation condition includes that the result satisfaction corresponding to the small model's answer is greater than or equal to the preset minimum expected satisfaction. The small model's answer is the answer obtained after inputting the training interaction data into the edge small model. If the small model's answer does not meet the preset minimum expected condition, then the label corresponding to the training interaction data is set to a second preset value that indicates that the request should be routed to the large model in the cloud. Based on the training interaction data and the corresponding labels, the routing decision model is trained until the preset training termination condition is met, at which point the trained routing decision model is obtained.

2. The routing decision model training method as described in claim 1, characterized in that, After the step of obtaining the pre-built training dataset and the routing decision model to be trained, the method further includes: The training interaction data is input into the large cloud model and the small edge model respectively, and the output results are the answer results of the large model and the answer results of the small model. The large model's answer and the small model's answer are respectively input into the preset judge model, and the output is the satisfaction level of the large model's answer and the satisfaction level of the small model's answer. If any one of the following conditions is met: the satisfaction level of the large model's answer result is less than the preset minimum expected satisfaction level, the satisfaction level of the small model's answer result is greater than or equal to the satisfaction level of the large model's answer result, or the satisfaction level of the small model's answer result is greater than or equal to the preset minimum expected satisfaction level, then the small model's answer result is determined to meet the preset minimum expected condition. If all of the following conditions are met: the satisfaction level of the large model's response is less than the preset minimum expected satisfaction level, the satisfaction level of the small model's response is greater than or equal to the satisfaction level of the large model's response, and the satisfaction level of the small model's response is greater than or equal to the preset minimum expected satisfaction level, then it is determined that the small model's response does not meet the preset minimum expected condition.

3. The routing decision model training method as described in claim 1, characterized in that, The routing decision model includes a probabilistic prediction network and a decision layer. The step of training the routing decision model based on the training interaction data and the labels corresponding to the training interaction data includes: The probability prediction network is trained based on the training interaction data and the labels corresponding to the training interaction data. In each iteration of training of the probability prediction network, the intermediate probability prediction network obtained in that iteration is acquired. Iterate through each preset candidate classification threshold and evaluate the performance evaluation index value corresponding to the intermediate routing decision model through a preset verification dataset. The intermediate routing decision model is a model composed of the intermediate probability prediction network and the intermediate decision layer connected in sequence. The intermediate decision layer is the decision layer whose classification threshold is set to the candidate classification threshold of the iteration. Based on the performance evaluation index values, the intermediate optimal classification threshold is selected from the candidate classification thresholds; After the preset training termination condition is met, the target optimal classification threshold is selected from all the intermediate optimal classification thresholds; The target probability prediction network and the target decision layer, which are connected in sequence, are determined as the route decision model after training is completed. The target probability prediction network is the probability prediction network obtained when the preset training termination condition is met, and the target decision layer is the decision layer with the classification threshold set to the target optimal classification threshold.

4. The routing decision model training method as described in claim 3, characterized in that, The step of selecting the intermediate optimal classification threshold from the candidate classification thresholds based on the performance evaluation index values ​​includes: The maximum value among all the performance evaluation index values ​​is determined as the intermediate optimal evaluation index value; The candidate classification threshold corresponding to the intermediate optimal evaluation index value is determined as the intermediate optimal classification threshold.

5. The routing decision model training method as described in claim 4, characterized in that, The step of selecting the target optimal classification threshold from all the intermediate optimal classification thresholds includes: The maximum value among all the intermediate optimal evaluation index values ​​is determined as the target optimal evaluation index value; The intermediate optimal classification threshold corresponding to the target optimal evaluation index value is determined as the target optimal classification threshold.

6. The routing decision model training method as described in claim 3, characterized in that, The step of evaluating the performance evaluation index value corresponding to the intermediate routing decision model through a preset validation dataset includes: The average probability of successfully solving the problem, the probability of routing to the smallest model, and the average inference delay corresponding to the intermediate routing decision model are evaluated using a preset validation dataset. The application scenario of the routing decision model is obtained, and the fusion weight corresponding to the application scenario is obtained. The fusion weight includes a first weighting factor for weighting the average probability of successfully solving the problem, a second weighting factor for weighting the probability of routing to the small model, and a third weighting factor for weighting the average inference delay. Based on the fusion weights, the average probability of successfully solving the problem, the probability of routing to the smallest model, and the average inference delay are weighted and summed to obtain the performance evaluation index value corresponding to the intermediate routing decision model.

7. A request routing method, characterized in that, The request routing method includes the following steps: Receive user interaction requests and obtain interaction data based on the interaction requests; The interactive data is input into a pre-trained routing decision model, and a decision result is output. The routing decision model is a model trained using the routing decision model training method as described in any one of claims 1 to 6. If the decision result indicates that the interaction request should be routed to the cloud-based large model, then the interaction data is input into the cloud-based large model, and the interaction result is output. If the decision model indicates that the interaction request should be routed to the edge mini-model, then the interaction data is input into the edge mini-model and the interaction result is output.

8. The request routing method as described in claim 7, characterized in that, The routing decision model includes an encoding layer, a fusion layer, a prediction layer, and a decision layer. The step of inputting the interaction data into the pre-trained routing decision model and outputting the routing decision result includes: The interaction data is encoded through the encoding layer to obtain a first encoding vector, and the statistical information corresponding to the interaction data is encoded to obtain a second encoding vector. The target encoding vector is obtained by fusing the first encoding vector and the second encoding vector through the fusion layer. The prediction layer converts the target encoding vector into a contextual semantic representation in a high-dimensional space to obtain a predicted competence probability vector, and then performs activation processing on the predicted competence probability vector to obtain the predicted competence probability. The decision layer compares whether the predicted competence is greater than or equal to a set classification threshold. If so, the first preset value is output as the routing decision result; otherwise, the second preset value is output as the routing decision result.

9. The routing decision model training method as described in claim 8, characterized in that, The interactive data is multimodal data, and the encoding layer includes encoding blocks for each modality of data. The step of encoding the interactive data through the encoding layer to obtain a first encoding vector includes: Each modal data in the interactive data is encoded using the corresponding encoding block in the encoding layer to obtain a first encoding vector corresponding to each modal data.

10. The routing decision model training method as described in claim 7, characterized in that, The method is applied to an edge device connected to a cloud server. A large cloud model is deployed on the cloud server, and a small edge model is deployed on the edge device. The step of inputting the interaction data into the large cloud model and outputting the interaction result includes: The interaction request is forwarded to the cloud server, so that after receiving the interaction request, the cloud server inputs the interaction data corresponding to the interaction request into the cloud big model, outputs the interaction result, and returns the interaction result to the edge device.

11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 10.

Images