Efficient hybrid reasoning method, device and storage medium for large language models based on routing technology

Abstract

The present application discloses a data processing method, which relates to the technical field of efficient hybrid reasoning methods, devices and storage media for large language models based on routing technology. The efficient hybrid reasoning of large language models based on routing technology includes: if a user request is received, determining the similarity between the user request and each historical request in the cache library; if there is no similarity exceeding the similarity threshold, determining the scoring indicators of each candidate processing model based on the accuracy scoring model, the scoring indicators including accuracy, response time and interface cost; determining the comprehensive score of each candidate processing model according to the scoring indicators; selecting a target model from each candidate processing model according to the comprehensive score, and routing the user request to the target model. The present application achieves the technical effect of optimizing the selection of large language models to achieve efficient response to user requests.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an efficient hybrid reasoning method, device, and storage medium for a large language model based on routing technology. Background Art

[0002] Large language models offer superior natural language processing capabilities and are therefore used to handle complex language understanding and generation tasks. However, due to their sheer size, large models require extremely high computational and storage requirements, often requiring deployment on expensive cloud servers, resulting in high operating costs. Smaller models, which can be deployed on edge devices, while less expensive, lag behind large models in response quality and accuracy. Furthermore, different large language models differ in their task focus. Consequently, the diversity of language models makes it difficult to select the right model for both model selection and task assignment.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide an efficient hybrid inference method, device and storage medium for a large language model based on routing technology, aiming to solve the technical problem of how to optimize the selection of a large language model to achieve efficient response to user requests.

[0005] To achieve the above objectives, this application proposes an efficient hybrid reasoning method for large language models based on routing technology. The efficient hybrid reasoning method for large language models based on routing technology includes:

[0006] If a user request is received, determining the similarity between the user request and each historical request in the cache;

[0007] If there is no similarity that exceeds the similarity threshold, determining the scoring indicators of each candidate processing model based on the accuracy scoring model, the scoring indicators including accuracy, response time and interface cost;

[0008] Determine the comprehensive score of each of the candidate processing models according to the scoring index;

[0009] A target model is selected from each of the to-be-selected processing models according to the comprehensive score, and the user request is routed to the target model.

[0010] In one embodiment, if a user request is received, the step of determining the similarity between the user request and each historical request in the cache library includes:

[0011] Segmenting the user request and each of the historical requests, and converting them into a vocabulary set, to construct a user request set and a historical request set;

[0012] Determining an intersection of the user request set and each of the historical request sets based on words that exist in both the user request set and each of the historical request sets;

[0013] Merging the words in the user request set and each of the historical request sets, and removing duplicate words, to determine a union of the user request set and each of the historical request sets;

[0014] The similarities are calculated according to the size of the intersections and the size of the unions.

[0015] In one embodiment, before the step of determining scoring indicators for each candidate processing model based on an accuracy scoring model if no similarity exceeds the similarity threshold, wherein the scoring indicators include accuracy, response time, and interface cost, the method further includes:

[0016] Collect open-source Q&A data and real-world business Q&A data to build a dataset;

[0017] Classify and label each question and answer data in the data set according to each of the to-be-selected processing models;

[0018] Loading a pre-trained accuracy scoring model, and adjusting the classification head of the accuracy scoring model to the number of classification tasks corresponding to the to-be-selected processing model;

[0019] The accuracy scoring model is trained using the classified and labeled question-answer data.

[0020] In one embodiment, the step of classifying and labeling each question and answer data in the dataset according to each of the to-be-selected processing models includes:

[0021] Inputting the question in each piece of the question-and-answer data into each of the candidate processing models, performing reasoning and answering through each of the candidate processing models, and generating at least one answer result;

[0022] According to the answer results of each of the candidate processing models, a weighted score of each of the candidate processing models is calculated, and the candidate processing model with the highest weighted score is used as the classification label of the question and answer data.

[0023] In one embodiment, the step of calculating a weighted score for each of the candidate processing models based on the answer results of each of the candidate processing models, and using the candidate processing model with the highest weighted score as the classification label for the question and answer data includes:

[0024] Compare each of the answer results generated by each of the candidate processing models with the standard answer in the question and answer data to calculate the accuracy score;

[0025] For each of the to-be-selected processing models, calculating the variance according to the accuracy score, and calculating the weighted score according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance;

[0026] The weighted scores of the candidate processing models are compared, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0027] In one embodiment, the step of training the accuracy scoring model using the categorized and labeled question and answer data includes:

[0028] Inputting the classified and labeled question-answer data into the accuracy scoring model, performing word segmentation processing, and converting it into a fixed-length vector representation;

[0029] The accuracy scoring model is trained using a standard training loop, with the cross-entropy loss function measuring the difference between the model's predicted probability distribution and the true label;

[0030] An optimization algorithm is used to update the parameters of the accuracy scoring model to minimize the loss function.

[0031] In one embodiment, if there is no similarity exceeding the similarity threshold, the step of determining the scoring indicators of each candidate processing model based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time, and interface cost, includes:

[0032] Predicting the accuracy score of each of the to-be-selected processing models using the accuracy score model;

[0033] Performing a stress test on the interface of each of the to-be-selected processing models to obtain the response time;

[0034] The computing power cost required for each of the to-be-selected processing models is calculated to determine the interface fee.

[0035] In one embodiment, the step of predicting the accuracy score of each of the candidate processing models using the accuracy score model includes:

[0036] Inputting the user request into the accuracy scoring model to extract a feature vector of the user request;

[0037] The feature vector is converted into a probability distribution through the classification head of the accuracy scoring model, and the probability distribution is used as the accuracy score of each of the to-be-selected processing models and output.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a large language model efficient hybrid reasoning device based on routing technology, the device including: a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the large language model efficient hybrid reasoning method based on routing technology as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the efficient hybrid reasoning method for large language models based on routing technology as described above are implemented.

[0040] The present application provides an efficient hybrid reasoning method for large language models based on routing technology. The present application determines the similarity between the user request and each historical request in the cache library when a user request is received; if no similarity exceeds a similarity threshold, the accuracy scoring model is used to determine the scoring indicators of each candidate processing model, including accuracy, response time, and interface cost; the comprehensive score of each candidate processing model is determined based on the scoring indicators; the target model is selected from each candidate processing model based on the comprehensive score, and the user request is routed to the target model. The present application first calculates the similarity between the user request and the historical requests in the cache library. When the similarity exceeds a set threshold, the historical answer corresponding to the historical request is directly returned, avoiding repeated calculation and query processes, reducing unnecessary computing resource consumption, and significantly improving response speed. When the similarity does not meet the conditions, the accuracy of each candidate processing model is determined by the accuracy scoring model, and the accuracy, interface response time, and cost of multiple candidate processing models are comprehensively evaluated to select the optimal model for processing, ensuring reasonable allocation and efficient utilization of resources. The present application achieves the technical effect of optimizing the selection of large language models to achieve efficient response to user requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 A flowchart illustrating the first embodiment of the present invention's efficient hybrid reasoning method for a large language model based on routing technology;

[0044] Figure 2 A flowchart illustrating the second embodiment of the present invention's method for efficient hybrid reasoning of a large language model based on routing technology;

[0045] Figure 3 This is a flowchart of the third embodiment of the efficient hybrid reasoning method for a large language model based on routing technology of this application;

[0046] Figure 4 A flowchart illustrating a fourth embodiment of the present invention's method for efficient hybrid reasoning of a large language model based on routing technology;

[0047] Figure 5 A brief flowchart of the efficient hybrid reasoning method for large language models based on routing technology provided by this application;

[0048] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the efficient hybrid reasoning method for a large language model based on routing technology in an embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solutions of the embodiments of this application are:

[0053] Currently, large language models, due to their sheer size and high computing and storage requirements, typically need to be deployed on expensive cloud servers, resulting in high operating costs. Small language models, which can be deployed on edge devices, while less expensive, lag behind large models in response quality and accuracy. Furthermore, different large language models differ in their task focus. Consequently, the diversity of language models makes it difficult to select the right model for the right task.

[0054] This application calculates the similarity between user requests and historical requests in the cache. When the similarity exceeds a set threshold, it directly returns the historical answer corresponding to the historical request, avoiding repeated calculations and queries, reducing unnecessary computing resource consumption, and significantly improving response speed. When the similarity does not meet the conditions, the accuracy of each large language model to be processed is determined through an accuracy scoring model. The accuracy, interface response time, and cost of multiple language models are comprehensively evaluated to select the optimal model for processing, ensuring the rational allocation and efficient use of resources.

[0055] It should be noted that the execution entity of this embodiment can be a large language model efficient hybrid reasoning system based on routing technology, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a control device of a large language model efficient hybrid reasoning system based on routing technology that can implement the above functions, etc. This embodiment does not specifically limit this. The following uses the large language model efficient hybrid reasoning system based on routing technology as an example to illustrate this embodiment and the following embodiments.

[0056] Example 1

[0057] Based on this, this application proposes a first embodiment of a large language model efficient hybrid reasoning method based on routing technology, please refer to Figure 1 The routing-based large language model efficient hybrid reasoning method includes:

[0058] Step S10: If a user request is received, determine the similarity between the user request and each historical request in the cache.

[0059] The purpose of calculating the similarity between the user request and the historical requests in the cache is to quickly match the user's current request with the historical requests already in the cache. If a highly similar historical request is found, the response result corresponding to the historical request can be directly used without the need for complex reprocessing or calculation. This can significantly improve the system's response speed, reduce the consumption of computing resources, and enhance the user experience.

[0060] In this embodiment, a user request is a request sent to the system through a web form, an API (Application Programming Interface), or other means. It contains the information the user wants to obtain or the action the user wants to perform. The cache is a database that stores historical requests and their corresponding responses. This cache allows for rapid responses to similar future requests. Similarity is a metric that measures the degree of similarity between a user request and historical requests.

[0061] As an optional implementation, convert the user request and historical requests into text vectors, and perform preprocessing operations such as word segmentation, stop word removal, and stemming on the text data in the user request to improve the accuracy of similarity calculation. Select a similarity algorithm according to the specific type and characteristics of the data, and calculate the similarity between the user request and each historical request in the cache library. Set a similarity threshold according to actual needs to determine whether the similarity between the user request and the historical request meets the requirements. Similarity algorithms include cosine similarity, Jaccard similarity coefficient, Euclidean distance, etc.

[0062] Optionally, step S10 includes:

[0063] Step S11, perform word segmentation on the user request and each historical request, and convert them into a set of words, and construct a user request set and a historical request set.

[0064] In this embodiment, word segmentation is to split the text into independent lexical units, which is the basis for text similarity calculation. The set of words is a set composed of non-repeating words. Convert both the user request and historical requests into sets of words to facilitate subsequent set operations.

[0065] As an optional implementation, use a Chinese word segmentation tool to perform word segmentation on the user request and historical requests, remove the stop words in the word segmentation results, convert the processed words into set form, and construct a user request set and a historical request set respectively. Stop words include commonly used words such as "of" and "is" that contribute little to the meaning of the text.

[0066] Step S12, determine the intersection of the user request set and each historical request set according to the words that exist in both the user request set and each historical request set.

[0067] Finding the intersection is to find the common words between the user request and historical requests, and these words reflect the similarity between the two.

[0068] Step S13, merge the words in the user request set and each historical request set, remove the duplicate words, and determine the union of the user request set and each historical request set.

[0069] Finding the union is to find all the non-repeating words between the user request and historical requests, and these words reflect the overall coverage between the two.

[0070] Step S14, calculate each similarity according to the size of each intersection and the size of each union.

[0071] Calculating the similarity is to quantify the degree of similarity between the user request and the historical requests, so as to determine whether the response result corresponding to the historical request can be directly applied to the user request.

[0072] In this embodiment, the similarity is a numerical index measuring the degree of similarity between the user request and the historical requests, and its value range is between 0 and 1. The higher the similarity, the more similar the user request and the historical requests are.

[0073] Exemplarily, the Jaccard similarity formula is used to calculate the similarity between the user request and the historical requests.

[0074] As an optional implementation manner of this embodiment, the Chinese text of the user request is segmented into words, splitting the text into separate words or lexical units. The stop word list is loaded to remove the stop words in the text. Stop words refer to words that frequently appear in the text but contribute little to the meaning of the text, such as "de" (的), "shi" (是), etc. The text after word segmentation and stop word removal is converted into a lexical set, ensuring that the elements in the set are not repeated. The words that exist in both sets are found as the intersection of the two sets; all non-repeated words in the two sets are found as the union of the two sets. The similarity is calculated through the Jaccard similarity formula, "Jaccard similarity = size of the intersection / size of the union".

[0075] Exemplarily, text A is the user request, and text B is the historical request in the cache library. Calculate the similarity between text A and text B. Text A is "I like machine learning and artificial intelligence", and text B is "Learning artificial intelligence requires a foundation in machine learning". First, perform word segmentation. The word segmentation results are "A = {I, like, machine learning, learning, artificial intelligence}" and "B = {learning, artificial intelligence, need, machine learning, foundation}". Remove the stop words from the word segmentation results. The stop word list can be loaded, and then the stop words are removed through the stop word list. The word segmentation results after removing the stop words are "A = {like, machine learning, artificial intelligence}" and "B = {artificial intelligence, need, machine learning, foundation}". Calculate the intersection of the two sets according to the word segmentation results after removing the stop words as "Intersection = {machine learning, artificial intelligence}", and the size of the intersection is 2. Then calculate the union of the two sets as "Union = {like, need, foundation, machine learning, artificial intelligence}", and the size of the union is 5. Calculate the Jaccard similarity according to the size of the intersection and the size of the union, "Jaccard similarity = 2 / 5 = 0.4". If the set similarity threshold is 0.89 at this time, the similarity between text A and text B does not reach the threshold. It is necessary to calculate the similarity between the text A corresponding to the user request and other historical requests in the cache library. If the similarities do not reach the threshold, it is further distributed to the large language model through the router for response.

[0076] Optionally, if the similarity exceeds a similarity threshold, the historical answer corresponding to the historical request is directly returned.

[0077] When the similarity between a user request and a historical request exceeds a preset similarity threshold, in order to improve system efficiency and user experience, the historical answer corresponding to the historical request is directly returned instead of reprocessing the user request, which can save system resources and speed up the response.

[0078] In this embodiment, the similarity threshold is a preset value used to determine whether the similarity between the user request and the historical request is high enough, thereby deciding whether to directly return the historical answer.

[0079] Optionally, if the similarity between the user request and a historical request in the cache exceeds a set threshold, the response result corresponding to the historical request is returned directly; if the similarity between the user request and all historical requests is lower than the threshold, further processing or calculation is required to generate a response result.

[0080] Exemplarily, the calculated similarity is compared with a preset similarity threshold. If the similarity is greater than or equal to the threshold, the historical answer corresponding to the historical request is retrieved from the cache library, and the retrieved historical answer is directly returned to the user.

[0081] Optionally, an initial similarity threshold is set based on historical experience and industry standards, and the threshold is dynamically adjusted based on the system's operating conditions and changes in the dataset.

[0082] Step S20: If there is no similarity exceeding the similarity threshold, the scoring indicators of each candidate processing model are determined based on the accuracy scoring model, and the scoring indicators include accuracy, response time and interface cost.

[0083] When the similarity between a user request and a historical request does not exceed the preset similarity threshold, a comprehensive evaluation method is used to select the most appropriate language model to process the user request. This method not only considers the accuracy of the language model, but also takes into account the response time and interface costs, thereby ensuring that while meeting the accuracy requirements, it also achieves better performance and cost-effectiveness.

[0084] In this embodiment, the accuracy scoring model is a model used to evaluate the accuracy of different large language models when processing tasks corresponding to user requests. It is trained based on historical data and can output the accuracy of each large language model when processing user requests. The selected processing model is a large language model. The large language model is a machine learning model that can generate or process natural language text, and can process user requests and generate corresponding answers. The response time refers to the time interval from the time the selected processing model receives the user request to the time the selected processing model returns the processing result, which is used to measure the performance of the selected processing model. The interface fee is the fee required to use the selected processing model, which is related to factors such as the complexity of the selected processing model and the amount of computing resources used.

[0085] As an optional implementation method, user requests whose similarity with all historical requests in the cache library does not reach a threshold are input into the accuracy scoring model. Through the accuracy scoring model, the accuracy of each candidate processing model in processing the user request is output based on the content of the user request. The response time of each candidate processing model within the billing cycle is calculated and normalized to the interval of "[0, 1]", where 1 represents an immediate response without delay and 0 represents the maximum possible delay. The interface fees generated during the billing cycle of the candidate processing model are counted, and the average fee for each request is calculated based on the interface fees: "Average request fee = interface fee / number of requests".

[0086] Step S30: determining the comprehensive score of each of the candidate processing models according to the scoring index.

[0087] By combining the accuracy score, interface response time, interface fee, and preset cost weights and speed weights, a comprehensive model score is calculated to comprehensively evaluate the performance, efficiency, and economy of each large language model.

[0088] In this embodiment, the comprehensive score is a score that comprehensively considers accuracy, response time, and interface cost, and is used to evaluate the overall performance of different language models in processing user requests.

[0089] In this embodiment, the cost weight is the weight ratio of interface costs in calculating the model's comprehensive score. A higher cost weight indicates a greater impact of costs in the comprehensive evaluation. The speed weight is the weight ratio of interface response time in calculating the model's comprehensive score. A higher speed weight indicates a greater impact of processing speed in the comprehensive evaluation.

[0090] Optionally, the response time is normalized and calculated as "response time = (actual response time - minimum response time) / (maximum response time - minimum response time)", where the minimum response time and maximum response time refer to the minimum and maximum values of all response times recorded within the billing period.

[0091] Optionally, the average cost of processing each user request is calculated using the formula "average request cost = interface cost / number of requests".

[0092] For example, the interface response time is normalized to the normalized response time using accuracy, response time, interface cost, and their corresponding weights, and the interface cost is reflected by the average request cost. The model's comprehensive score is calculated as "Comprehensive score = Accuracy × (1 - Cost Weight × Average Request Cost - Speed Weight × (1 - Response Time))."

[0093] Optionally, you can set cost and speed weights based on your business needs. By adjusting these weights, you can balance accuracy, cost, and speed. If cost or response time significantly impacts the overall score, you can adjust the weights accordingly to balance the accuracy, cost, and speed factors in the overall score.

[0094] Step S40 , selecting a target model from each of the candidate processing models according to the comprehensive score, and routing the user request to the target model.

[0095] Based on the accuracy, interface response time and interface cost of multiple candidate processing models, the optimal model is selected as the target model by calculating the comprehensive score.

[0096] In this embodiment, the comprehensive score is a score calculated based on preset weights and formulas, combined with the accuracy, interface response time, and interface cost of the candidate processing model, and is used to measure the overall performance of the model.

[0097] As an optional implementation, among all the to-be-selected processing models, the model with the highest comprehensive score is selected as the target model, and the user request is sent to the target model for processing.

[0098] Optionally, an inference answer corresponding to the user request is generated through the target model, and the inference answer of the target model is returned to the user in text form based on routing technology.

[0099] This embodiment provides an efficient hybrid inference method for large language models based on routing technology. This method first calculates the similarity between a user request and historical requests in the cache. When the similarity exceeds a set threshold, it directly returns the historical answer corresponding to the historical request. This avoids repeated calculations and queries, reduces unnecessary computing resource consumption, and significantly improves response speed. If the similarity does not meet the requirements, the accuracy of each large language model is determined using an accuracy scoring model. The accuracy, response time, and interface costs of multiple language models are comprehensively evaluated, and the optimal model is selected for processing, ensuring the rational allocation and efficient use of resources.

[0100] Based on Example 1, Example 2 of this application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 2 , before step S30, including:

[0101] Step S50: Collect open-source question-and-answer data and real business question-and-answer data to construct a data set.

[0102] Open source question-and-answer data provides a wide range of question types and answer examples, while real-world business question-and-answer data reflects user needs and business logic in real application scenarios. The combination of the two can build a comprehensive and targeted dataset.

[0103] In this example, open-source Q&A data refers to question-and-answer pairs obtained from public Q&A platforms, forums, and communities on the internet. These data are user-generated and cover a wide range of topics and question types. Real-world Q&A data is a record of actual user Q&A interactions in real business scenarios, reflecting user needs, question types, and answering habits. A dataset is a collection of multiple sets of Q&A data used to train, validate, or test machine learning models. The Q&A data format is "question-standard answer."

[0104] Optionally, when obtaining open source question and answer data and real business question and answer data, it is necessary to obtain the corresponding authorization of the user.

[0105] As an optional implementation method, first collect open source Q&A data, select relevant open source Q&A platforms based on the target field or theme of the needs, and use crawler technology or API interfaces to crawl question and answer pairs from the selected Q&A platform. During the crawling process of question and answer pairs, it is necessary to comply with the platform's usage agreement and data privacy policy. The crawled data is preprocessed by removing duplicates, filtering spam, and standardizing the text format. Secondly, collect real business Q&A data and extract log records of user interactions from the business system, including user questions and system answers. When processing real business Q&A data, ensure the protection of user privacy and sensitive information, and perform necessary data desensitization. Integrate open source Q&A data and real business Q&A data to form a unified data set.

[0106] Step S60: Classify and label each question and answer data in the data set according to each of the to-be-selected processing models.

[0107] In this embodiment, the processing model to be selected is a large language model with strong natural language processing capabilities that can understand and generate coherent text. Classification labeling is the process of assigning a label to each question-answer pair in the dataset.

[0108] As an optional implementation method, for each question and answer data in the dataset, each large language model is used to perform inference and answer, and the inference answers of the candidate processing models are comprehensively scored to evaluate the performance of each candidate processing model. The candidate processing model with the highest comprehensive score is selected as the selected model for the question and answer data, and the selected model is classified and labeled on the data.

[0109] Exemplarily, for each candidate processing model that generates multiple model answers under each parameter setting, the quality scores of the model answers and the corresponding standard answers in the question and answer data are calculated respectively. Different weights are assigned to the quality scores obtained under different parameters according to actual needs, and the weighted average quality score is calculated as the comprehensive score.

[0110] Step S70: load the pre-trained accuracy scoring model, and adjust the classification head of the accuracy scoring model to the corresponding number of classification tasks of the to-be-selected processing model.

[0111] In this embodiment, the pretrained accuracy scoring model, a model trained on a large-scale dataset, is used to evaluate the accuracy of the text output by the candidate processing models. The classification head is the network component responsible for outputting the classification results in the neural network model. The output dimensions of the classification head need to be adjusted to match the number of candidate processing models.

[0112] As an optional implementation, a pre-trained accuracy scoring model is loaded using a deep learning framework, and the output dimension of the classification head in the accuracy scoring model is modified according to the preset number of models to be processed.

[0113] Exemplarily, the output dimension of the classification head in the accuracy scoring model is modified by modifying the size of the weight matrix of the fully connected layer in the classification head.

[0114] Step S80: training the accuracy scoring model using the classified and labeled question and answer data.

[0115] The accuracy scoring model is trained with classified and labeled question-answering data to improve the model's performance on question-answering tasks, enabling it to more accurately evaluate the degree of match between the answers generated by the candidate processing model and the questions, and then select the appropriate candidate processing model to answer.

[0116] In this embodiment, classification labeling is the process of using the candidate processing model that best matches the answer of the candidate processing model as the classification label of the question and answer data for each question and answer data in the data set.

[0117] For example, the dataset is divided into a training set, a validation set, and a test set. The accuracy model is trained using the classified and annotated training set. The accuracy of the accuracy scoring model on the validation set is minimized by iteratively adjusting the model parameters. The accuracy scoring model is evaluated using the test set data, and the model performance is measured by calculating metrics such as accuracy, recall, and F1 score. The model hyperparameters are adjusted based on the evaluation results.

[0118] Optionally, step S80 includes:

[0119] Step S81: Input the categorized and labeled question-answer data into the accuracy scoring model, perform word segmentation processing, and convert it into a fixed-length vector representation.

[0120] Preprocess the categorized and labeled question-answering data into a format acceptable to the model so that it can be input into the accuracy scoring model for training.

[0121] In this embodiment, since machine learning models generally require input data to have a fixed dimension and length, the text data needs to be converted into a fixed-length vector representation.

[0122] As an optional implementation, use a word segmentation tool to segment the content of the categorized and annotated question and answer data. Then, by counting the number of occurrences of each word in the text, a fixed-length vector is generated, where each element corresponds to the number of occurrences of a word. This vector representation of the preprocessed question and answer data is input into the accuracy scoring model.

[0123] As another optional implementation of converting to a fixed-length vector representation, each word is mapped to a high-dimensional vector space through a pre-trained word embedding model, and the average or weighted sum of the vectors of all words in the text is calculated as the vector representation of the text.

[0124] Optionally, if the lengths of the vectorized text vectors are inconsistent, length normalization is required to ensure that all input data have the same dimensions and lengths. This can be done by truncating overly long vectors or padding shorter vectors to the required length.

[0125] Step S82: The accuracy scoring model is trained using a standard training cycle, and the difference between the model's predicted probability distribution and the true label is measured using a cross-entropy loss function.

[0126] In this embodiment, the standard training loop is an iterative training process consisting of forward propagation, loss calculation, backpropagation, and parameter updates. In each iteration, the accuracy scoring model makes predictions based on the input data. It then calculates the difference between the predictions and the true labels, i.e., the loss. This loss is then used to update the accuracy scoring model's parameters. This iterative process is repeated until the accuracy scoring model's performance on the validation set stabilizes or the pre-set number of training rounds is reached.

[0127] In this example, the cross-entropy loss function is a commonly used loss function used to measure the difference between the probability distribution predicted by the model and the true label in classification tasks, i.e., the loss. The cross-entropy loss function calculates the relative entropy between the predicted probability of each class by the accuracy scoring model and the true label, and averages these results. A smaller cross-entropy loss indicates that the accuracy scoring model's prediction is closer to the true label.

[0128] Optionally, key parameters such as the number of iterations of the training loop, batch size, and learning rate are pre-set.

[0129] As an optional implementation, in each iteration of the training process, a batch of question-and-answer data in the training set is input into the accuracy scoring model, the predicted probability distribution of the accuracy scoring model is calculated through forward propagation, and the cross-entropy loss function is used to calculate the difference between the probability distribution predicted by the accuracy scoring model and the true label.

[0130] Step S83: Using an optimization algorithm to update the parameters of the accuracy scoring model to minimize the loss function.

[0131] An optimization algorithm is used to adjust the parameters of the accuracy scoring model to minimize the loss function. In machine learning and deep learning, the loss function is a key metric for measuring the difference between the accuracy scoring model's predictions and the actual labels. By minimizing the loss function, the model gradually learns the inherent patterns of the data and improves the accuracy of its predictions.

[0132] In this embodiment, an optimization algorithm is used to adjust the parameters of the accuracy scoring model to minimize the loss function. Optimization algorithms include Stochastic Gradient Descent (SGD), Momentum, RMSprop, Adam, and the like.

[0133] As an optional implementation, based on the gradient information of the loss function, the error signal is passed back to each layer of the accuracy scoring model through the backpropagation algorithm, and the parameters of the accuracy scoring model are updated based on the backpropagated gradient information using optimization algorithms such as stochastic gradient descent and Adam.

[0134] Optionally, after each iteration, the performance of the accuracy scoring model is evaluated using the validation set. If the performance of the accuracy scoring model does not improve significantly over multiple consecutive iterations, the early stopping strategy is used to terminate the training process to avoid overfitting. At the same time, hyperparameters such as learning rate and batch size are adjusted based on the performance on the validation set.

[0135] This embodiment provides an efficient hybrid inference method for a large language model based on routing technology. This embodiment first increases the diversity of the data set by collecting open source and real-world business question and answer data. By using various language models for classification and labeling, the accuracy scoring model can more accurately evaluate the degree of match between the question and answer data and the large language model, further determining the most suitable large language model for the requests in the question and answer data, thereby improving the overall efficiency and accuracy of the system. Through targeted training data, the model can converge to the optimal solution more quickly, thereby improving training efficiency.

[0136] Based on the first and second embodiments, the third embodiment of the present application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 3 , step S60 includes:

[0137] Step A10: Input the question in each piece of the question-answer data into each of the to-be-selected processing models, perform reasoning and answering through each of the to-be-selected processing models, and generate at least one answer result.

[0138] In this embodiment, question-and-answer data is a dataset containing questions and their corresponding answers, used to train and evaluate the performance of the accuracy scoring model. The candidate processing model is a large language model with strong language understanding and generation capabilities, capable of processing and understanding complex natural language text. Inference answering is the process by which the candidate processing model, based on the input question, uses its internal mechanisms to understand and analyze it, ultimately generating an answer.

[0139] As an optional implementation method, a question-and-answer dataset is collected and organized, and multiple large language models are selected as candidate processing models. Each question in the question-and-answer dataset is input into each candidate processing model in turn. Each candidate processing model performs reasoning based on the input question and generates at least one answer result.

[0140] Step A20: Calculate the weighted scores of the candidate processing models according to the answer results of the candidate processing models, and use the candidate processing model with the highest weighted score as the classification label of the question and answer data.

[0141] In this embodiment, a weighted score is a score calculated based on the candidate model's answer and preset weights, used to evaluate the candidate model's performance in answering user requests. Classification labeling involves assigning each question-and-answer data point in the dataset to the best-performing candidate model as a classification label.

[0142] Optionally, weights of various evaluation indicators are set according to task requirements and evaluation criteria.

[0143] As an optional implementation method, based on the answer results of each candidate processing model and the preset weight, the weighted score of each candidate processing model on the question and answer dataset is calculated, the weighted scores of different candidate processing models are compared, and the candidate processing model with the highest score is selected as the classification label of the question and answer data.

[0144] For example, by calculating the accuracy, recall rate, F1 score and other evaluation indicators of the candidate processing model on each question, and performing weighted summation according to the weights, the weighted score of each candidate processing model is obtained, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0145] Optionally, step A20 includes:

[0146] In step A21, each of the answer results generated by each of the candidate processing models is compared with the standard answer in the question and answer data to calculate the accuracy score.

[0147] In this embodiment, the answer is the output generated by the candidate processing model based on the input question and can be in text, numeric, or other formats. The standard answer is the answer provided for each question in the question-and-answer data. The accuracy score is the degree to which the answer generated by the candidate processing model matches the standard answer.

[0148] As an optional implementation method, various language models are used to infer historical requests in the question-and-answer data to generate answer results. The answer results of each large language model are compared with the standard answer, and the accuracy score is calculated through methods such as string matching and semantic similarity calculation.

[0149] Step A22: For each of the to-be-selected processing models, the variance is calculated according to the accuracy score, and the weighted score is calculated according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance.

[0150] In this embodiment, variance is a measure of the degree of dispersion in the data distribution and is used to assess the fluctuations in the accuracy scores of the candidate processing model on different problems. The score weight is the proportion of the accuracy score used in calculating the weighted score. The variance weight is the proportion of the variance used in calculating the weighted score and is used to balance the accuracy and stability of the candidate processing model. The weighted score is a composite score calculated by combining the accuracy score and variance, along with their respective weights, and is used to evaluate the overall performance of the candidate processing model.

[0151] Optionally, weights for accuracy score and variance are set according to task requirements and evaluation criteria.

[0152] For example, the score weight and variance weight values are dynamically adjusted based on the query type or difficulty. For example, for complex queries, the variance weight can be increased to make the router more inclined to select models with large variances; while for simple queries, the score weight can be increased to prioritize models with higher scores.

[0153] As an optional implementation, for each candidate processing model, the variance of the accuracy score of the candidate processing model on all historical requests is calculated, and a weighted score of each model is calculated by combining the accuracy score, variance, and their respective weights.

[0154] Exemplarily, the formula for calculating the weighted score is “weighted score = accuracy score × score weight + variance × variance weight”.

[0155] Step A23: compare the weighted scores of the candidate processing models, and use the candidate processing model with the highest weighted score as the classification label of the corresponding question and answer data.

[0156] Exemplarily, the weighted scores of all candidate processing models are compared to find the candidate processing model with the highest score, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0157] As an example of this embodiment, for each piece of data in the dataset, a weighted score is calculated using four different large language models (S1, S2, L1, and L2). Each large language model uses three different sets of temperature parameters (0, 7, and 10) for inference on a question. The latter two sets of parameters each generate 10 multi-sampled results, resulting in a total of 21 model responses. These 21 responses are compared with the standard response and scored using the Bart score to obtain 21 accuracy scores. The variance of these 21 scores is also calculated. The overall score for each large language model is calculated using the weighted score formula: "Weighted Score = α × p + β × Variance." "Weighted Score" is the weighted score, "p" is the accuracy score of the large language model, "Variance" is the variance of the large language model's score under different conditions, and "α" and "β" are weighting parameters used to adjust the influence of the score and variance on the overall score. "α" represents the score weight, and "β" represents the variance weight. The large language model with the highest score is selected as the classification label for the data piece. For example, if the “L1” model has the highest score, the classification label of the data is “(S1, S2, L1, L2) = (0, 0, 1, 0)”.

[0158] This embodiment provides an efficient hybrid inference method for large language models based on routing technology. This embodiment first calculates accuracy scores to intuitively understand the accuracy of each large language model in answering user requests. By calculating the variance of the large language model when answering different requests, the stability of the large language model is determined. Then, based on the accuracy and stability of the large language models, the performance of different large language models is quantitatively compared.

[0159] Based on Example 1, Example 4 of this application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 4 , step S30 includes:

[0160] Step S31: predicting the accuracy score of each of the to-be-selected processing models using the accuracy score model.

[0161] Using the trained accuracy scoring model, the accuracy of each candidate processing model in processing user requests is predicted and scored. The score can reflect the candidate processing model's ability to understand user requests and the accuracy of its answers, thereby selecting the most appropriate candidate processing model to answer, thereby improving the overall performance of the system and user satisfaction.

[0162] In this embodiment, the accuracy score is a quantitative assessment of the accuracy of the candidate processing model in processing the user request. The higher the score, the more accurate the model's understanding and response to the user request.

[0163] As an optional implementation, the user request is input into the accuracy scoring model, and the parameters and logic of the accuracy scoring model are used to predict the possible answers given by each candidate processing model, and the corresponding accuracy score is calculated. After the accuracy scoring model prediction is completed, the accuracy score of each candidate processing model is output.

[0164] Optionally, step S31 includes:

[0165] Step B10: input the user request into the accuracy scoring model to extract the feature vector of the user request.

[0166] The user request is input into the trained accuracy scoring model, and the feature vector of the user request is extracted through the model. By extracting the feature vector, the user request can be converted into a form that the model can understand, providing a basis for subsequent accuracy scoring and model selection.

[0167] In this embodiment, the feature vector is a multidimensional vector, each component of which is a numerical representation of a user request on a certain feature, including lexical features, grammatical features, semantic features, etc., which are used to capture key information of the user request.

[0168] As an optional implementation, user requests are cleaned, segmented, and stop-word removed before being fed into the accuracy scoring model. This process converts the user request into a point in a high-dimensional space, where each dimension represents a feature. After feature extraction, the accuracy scoring model outputs a feature vector containing the representation of the user request in the learned feature space.

[0169] Step B20: Convert the feature vector into a probability distribution through the classification head of the accuracy scoring model, and output the probability distribution as the accuracy score of each of the to-be-selected processing models.

[0170] In this embodiment, the classification head is a key component in the accuracy scoring model, responsible for converting feature vectors into probability distributions. The classification head contains one or more layers of neural networks, which use learned weights and biases to map feature vectors to probability space. Feature vectors are numerical vectors extracted from user requests that can represent their key information. Feature vectors are an important part of the model input and contain key information for evaluating the accuracy of the candidate processing model. The probability distribution is a prediction of the accuracy of each candidate processing model for processing user requests. Each value in the probability distribution corresponds to a candidate processing model, and the size of the value represents the predicted accuracy value of the model processing the request.

[0171] As an optional implementation, the extracted feature vector of the user request is input into the classification head of the accuracy scoring model. The classification head uses the learned weights and biases to perform linear transformation and nonlinear activation operations on the feature vector, calculates the accuracy prediction value corresponding to each candidate processing model, and normalizes it into a probability distribution. The calculated probability distribution is output as the accuracy score of each candidate processing model.

[0172] Step S32: performing a stress test on the interface of each of the to-be-selected processing models to obtain the response time.

[0173] Stress-test the interfaces of each candidate processing model to evaluate their performance and stability under real-world loads. By simulating scenarios such as high-concurrency requests or large amounts of data processing, we measure the interface response time to assess performance and user experience. Faster interface response times mean users spend less time waiting for results, resulting in a better user experience.

[0174] In this embodiment, the interface is the API or service endpoint provided by each candidate processing model. Through these interfaces, the functions of the candidate processing model can be called. Stress testing, also known as load testing or performance testing, is a testing method that evaluates the performance of an interface by simulating actual or expected load conditions. The purpose is to discover the behavior of the interface under extreme conditions, including key performance indicators such as response time, throughput, and error rate. Response time is the time it takes from a request being sent to an interface to the time the interface returns a response, and is a key indicator of interface performance.

[0175] As an optional implementation, based on the test objective of measuring response time under load, design test scenarios, including request type, request frequency, and data size, to simulate actual or expected load conditions. Execute the test scripts using a stress testing tool to perform stress testing and monitor the response time of each candidate processing model interface.

[0176] Step S33: Calculate the computing power cost required for each of the to-be-selected processing models and determine the interface fee.

[0177] Evaluate and calculate the computing power cost required for each candidate processing model in actual application, and then determine the interface service fees provided for these models. Computing power cost is an indispensable resource consumption indicator when the candidate processing model runs and provides services, and is directly related to operating costs.

[0178] In this example, computing power costs refer to the computing resources required to run the candidate processing model, including hardware resources such as the CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, and storage, as well as indirect costs such as cloud services and data center operations. Interface fees are the fees charged during the billing cycle for using the candidate processing model to perform inference and answer questions.

[0179] As an optional implementation, a comprehensive performance evaluation of each candidate processing model is conducted, including metrics such as processing speed, accuracy, and memory usage. During actual operation, resource consumption by each candidate processing model is monitored in real time, including CPU utilization, GPU memory usage, and network bandwidth. Based on this resource consumption monitoring data, a computing power cost calculation model is constructed. This computing power cost calculation model estimates interface fees based on factors such as the unit cost of hardware resources, resource utilization, and model runtime.

[0180] Optionally, for large language models provided by third-party model suppliers, the interface fee can be calculated directly based on the fees charged by the third-party model supplier.

[0181] This embodiment provides an efficient hybrid inference method for large language models based on routing technology. This embodiment first predicts the accuracy scores of each language model through an accuracy scoring model, providing a quantitative evaluation indicator for model performance. By stress testing the interfaces of each language model, the response time of the interface under different loads can be obtained to evaluate the processing speed of the model. The interface fee is determined by calculating the computing power cost required for each language model, and the cost is further controlled. According to the accuracy score, interface response time and interface fee, as well as the preset fee weight and speed weight, the comprehensive score of the model is calculated. By comparing the comprehensive scores of different models, the large language model that best suits the user's request can be selected.

[0182] For example, to help understand the implementation process of the high-efficiency hybrid reasoning method for large language models based on routing technology obtained by combining this embodiment with the above embodiment 1, please refer to Figure 5 , Figure 5 This paper provides a brief flowchart of an efficient hybrid inference method for large language models based on routing technology. Specifically:

[0183] When receiving a user request input by the user to the system, the cache is first checked to determine whether a similar user request has been processed. If there is a matching historical user request in the cache, the historical answer corresponding to the cached historical user request is directly returned.

[0184] If no matching historical user request is found in the cache, the query is sent to the router.

[0185] The router uses the routing service to decide which model to send the user request to based on the characteristics of the user request.

[0186] Optionally, set the dataset space for user requests to X and the dataset space for inference answers to Z. "L1: X->Z" indicates that the larger model completes the inference service, while "S1: X->Z" indicates that the smaller model completes the inference service. "r1: X->(0, 1)" indicates routing, where 1 indicates routing to S1 and 0 indicates routing to L1.

[0187] For example, "x" represents a user request in the user request dataset X, and the output of "r1(x)" is a probability value, indicating which model the user request should be routed to. When "r1(x)=0", the user request will be sent to the larger model L1; if "r1(x)=1", the user request will be sent to another smaller model S1.

[0188] Furthermore, we use "q:z->R" to represent the scoring function "q(z)", and use the BART Score method to score the answer "z", where "z" represents an inference answer in the inference answer dataset Z. BART Score is a scoring method based on the BART model, which is used to evaluate the performance of natural language generation systems.

[0189] Furthermore, the inference answer of the model with the highest score is returned to the user through the router.

[0190] The entire process selects the most appropriate model to generate high-quality answers based on the characteristics of user requests, and improves efficiency through a caching mechanism.

[0191] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the present application's efficient hybrid reasoning method for large language models based on routing technology. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0192] The present application provides a large language model efficient hybrid reasoning device based on routing technology. The large language model efficient hybrid reasoning device based on routing technology includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model efficient hybrid reasoning method based on routing technology in the above-mentioned embodiment one.

[0193] Reference below Figure 6, which shows a schematic diagram of the structure of a large language model efficient hybrid reasoning device based on routing technology suitable for implementing the embodiments of the present application. The large language model efficient hybrid reasoning device based on routing technology in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The large language model efficient hybrid inference device based on routing technology shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0194] like Figure 6 As shown, the routing-based large language model efficient hybrid inference device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the routing-based large language model efficient hybrid inference device. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the routing-based large language model efficient hybrid inference device to communicate wirelessly or wired with other devices to exchange data. Although the diagram illustrates the routing-based large language model efficient hybrid inference device with various systems, it should be understood that implementation or presence of all illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0195] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0196] The routing-based high-efficiency hybrid reasoning device for large language models provided in this application, which employs the routing-based high-efficiency hybrid reasoning method for large language models in the aforementioned embodiments, can solve the technical problem of optimizing the selection of large language models to achieve efficient responses to user requests. Compared to the prior art, the routing-based high-efficiency hybrid reasoning device for large language models provided in this application has the same beneficial effects as the routing-based high-efficiency hybrid reasoning method for large language models provided in the aforementioned embodiments. Other technical features of the routing-based high-efficiency hybrid reasoning device for large language models are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0197] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0198] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0199] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the efficient hybrid reasoning method for large language models based on routing technology in the above-mentioned embodiment.

[0200] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0201] The computer-readable storage medium may be included in the large language model efficient hybrid reasoning device based on routing technology; or may exist independently without being assembled into the large language model efficient hybrid reasoning device based on routing technology.

[0202] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the large language model efficient hybrid reasoning device based on routing technology, the large language model efficient hybrid reasoning device based on routing technology can be written in one or more programming languages or a combination thereof to perform the operations of the present application. The programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, via the Internet using an Internet service provider).

[0203] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0204] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0205] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned routing-based efficient hybrid inference method for large language models. This computer-readable storage medium addresses the technical problem of optimizing the selection of large language models to efficiently respond to user requests. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the routing-based efficient hybrid inference method for large language models provided in the aforementioned embodiments, and are not further elaborated here.

[0206] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An efficient hybrid reasoning method for large language models based on routing technology, characterized by: The efficient hybrid reasoning method for large language models based on routing technology includes: If a user request is received, determining the similarity between the user request and each historical request in the cache; Collect open-source Q&A data and real-world business Q&A data to build a dataset; Input the question in each question-answering data into each candidate processing model, perform reasoning and answering through each candidate processing model, and generate at least one answer result; Compare each of the answer results generated by each of the candidate processing models with the standard answer in the question and answer data to calculate the accuracy score; For each of the to-be-selected processing models, calculating the variance according to the accuracy score, and calculating a weighted score according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance; Comparing the weighted scores of the candidate processing models, and using the candidate processing model with the highest weighted score as the classification label of the corresponding question-and-answer data; Loading a pre-trained accuracy scoring model, and adjusting the classification head of the accuracy scoring model to the number of classification tasks corresponding to the to-be-selected processing model; Training the accuracy scoring model using the classified and labeled question-answer data; If there is no similarity that exceeds the similarity threshold, determining a scoring index for each of the to-be-selected processing models based on the accuracy scoring model, the scoring index including accuracy, response time, and interface cost; Determine the comprehensive score of each of the candidate processing models according to the scoring index; A target model is selected from each of the to-be-selected processing models according to the comprehensive score, and the user request is routed to the target model.

2. The efficient hybrid reasoning method for large language models based on routing technology according to claim 1, characterized in that: If a user request is received, the step of determining the similarity between the user request and each historical request in the cache library includes: Segmenting the user request and each of the historical requests, and converting them into a vocabulary set, to construct a user request set and a historical request set; Determining an intersection of the user request set and each of the historical request sets based on words that exist in both the user request set and each of the historical request sets; Merging the words in the user request set and each of the historical request sets, and removing duplicate words, to determine a union of the user request set and each of the historical request sets; The similarities are calculated according to the size of the intersections and the size of the unions.

3. The efficient hybrid reasoning method for large language models based on routing technology according to claim 1, characterized in that: The step of training the accuracy scoring model using the classified and labeled question and answer data includes: Inputting the classified and labeled question-answer data into the accuracy scoring model, performing word segmentation processing, and converting it into a fixed-length vector representation; The accuracy scoring model is trained using a standard training loop, with the cross-entropy loss function measuring the difference between the model's predicted probability distribution and the true label; An optimization algorithm is used to update the parameters of the accuracy scoring model to minimize the loss function.

4. The efficient hybrid reasoning method for large language models based on routing technology according to claim 1, characterized in that: If there is no similarity exceeding the similarity threshold, the step of determining the scoring indicators of each of the to-be-selected processing models based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time, and interface cost, comprises: Predicting the accuracy score of each of the to-be-selected processing models using the accuracy score model; Performing a stress test on the interface of each of the to-be-selected processing models to obtain the response time; The computing power cost required for each of the to-be-selected processing models is calculated to determine the interface fee.

5. The efficient hybrid reasoning method for large language models based on routing technology according to claim 4, characterized in that: The step of predicting the accuracy score of each of the to-be-selected processing models by using the accuracy score model comprises: Inputting the user request into the accuracy scoring model to extract a feature vector of the user request; The feature vector is converted into a probability distribution through the classification head of the accuracy scoring model, and the probability distribution is used as the accuracy score of each of the to-be-selected processing models and output.

6. An efficient hybrid inference device for large language models based on routing technology, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large language model efficient hybrid reasoning method based on routing technology according to any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large language model efficient hybrid reasoning method based on routing technology according to any one of claims 1 to 5 are implemented.

Images