Large language model efficient hybrid reasoning method and device based on routing technology and storage medium

Abstract

The invention discloses data processing, and relates to the technical field of a large language model efficient hybrid reasoning method and device based on a routing technology and a storage medium, and the large language model efficient hybrid reasoning method based on the routing technology comprises the steps: if a user request is received, determining the similarity between the user request and each historical request in a cache library; if the similarity exceeding a similarity threshold does not exist, scoring indexes of the to-be-selected processing models are determined based on an accuracy rate scoring model, and the scoring indexes comprise accuracy rates, response time and interface costs; determining a comprehensive score of each to-be-selected processing model according to the scoring indexes; and selecting a target model from the to-be-selected processing models according to the comprehensive score, and routing the user request to the target model. According to the method and the device, the technical effect of optimizing selection of the large language model to realize efficient response of the user request is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an efficient hybrid reasoning method, device and storage medium for a large language model based on routing technology. Background Art

[0002] Large language models have excellent natural language processing capabilities and are therefore used to handle complex language understanding and generation tasks. However, due to their large size, large models have extremely high computing and storage requirements and usually need to be deployed on expensive cloud servers, which have high operating costs. Small models that can be deployed on edge devices, although lower in cost, lag behind large models in response quality and accuracy. In addition, there are differences in the task focus of different large language models. Therefore, the diversity of language models makes it difficult to choose a suitable model when performing model selection and task assignment.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a large language model efficient hybrid reasoning method, device and storage medium based on routing technology, aiming to solve the technical problem of how to optimize the selection of large language models to achieve efficient response to user requests.

[0005] To achieve the above objectives, the present application proposes an efficient hybrid reasoning method for a large language model based on routing technology. The efficient hybrid reasoning method for a large language model based on routing technology includes: If a user request is received, determining the similarity between the user request and each historical request in the cache; If there is no similarity exceeding the similarity threshold, determining the scoring indicators of each candidate processing model based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time and interface cost; Determine the comprehensive score of each of the processing models to be selected according to the scoring index; A target model is selected from each of the to-be-selected processing models according to the comprehensive score, and the user request is routed to the target model.

[0006] In one embodiment, if a user request is received, the step of determining the similarity between the user request and each historical request in the cache library includes: Process the user request and each of the historical requests by word segmentation, and convert them into a vocabulary set, and construct a user request set and a historical request set; Determine an intersection of the user request set and each of the historical request sets according to words that exist in both the user request set and each of the historical request sets; Merging the words in the user request set and each of the historical request sets, and removing duplicate words, to determine a union of the user request set and each of the historical request sets; The similarities are calculated according to the size of each intersection and the size of each union.

[0007] In one embodiment, before the step of determining the scoring indicators of each candidate processing model based on the accuracy scoring model if there is no similarity exceeding the similarity threshold, wherein the scoring indicators include accuracy, response time, and interface cost, the method includes: Collect open source question-answering data and real business question-answering data to build a data set; According to each of the to-be-selected processing models, classify and label each question and answer data in the data set; Loading a pre-trained accuracy scoring model, and adjusting the classification head of the accuracy scoring model to a corresponding number of classification tasks of the to-be-selected processing model; The accuracy scoring model is trained using the question-answer data that is labeled by classification.

[0008] In one embodiment, the step of classifying and labeling each question and answer data in the data set according to each of the to-be-selected processing models includes: Inputting the question in each of the question-answer data into each of the processing models to be selected, and performing reasoning and answering through each of the processing models to be selected to generate at least one answer result; According to the answer results of each of the candidate processing models, a weighted score of each of the candidate processing models is calculated, and the candidate processing model with the highest weighted score is used as the classification label of the question and answer data.

[0009] In one embodiment, the step of calculating the weighted score of each of the candidate processing models according to the answer results of each of the candidate processing models, and using the candidate processing model with the highest weighted score as the classification label of the question and answer data includes: Compare each of the answer results generated by each of the candidate processing models with the standard answer in the question and answer data to calculate the accuracy score; For each of the processing models to be selected, the variance is calculated according to the accuracy score, and the weighted score is calculated according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance; The weighted scores of the candidate processing models are compared, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0010] In one embodiment, the step of training the accuracy scoring model using the question-answer data with classification annotations includes: Input the classified and labeled question-answer data into the accuracy scoring model, perform word segmentation processing, and convert it into a vector representation of a fixed length; The accuracy scoring model is trained using a standard training loop, and the difference between the model's predicted probability distribution and the true label is measured using a cross-entropy loss function; The parameters of the accuracy scoring model are updated using an optimization algorithm to minimize the loss function.

[0011] In one embodiment, if there is no similarity exceeding the similarity threshold, the step of determining the scoring indicators of each candidate processing model based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time and interface cost, comprises: Predicting the accuracy score of each of the processing models to be selected by using the accuracy score model; Performing stress testing on the interface of each of the processing models to be selected to obtain the response time; The computing power cost required for each of the processing models to be selected is calculated to determine the interface fee.

[0012] In one embodiment, the step of predicting the accuracy score of each of the to-be-selected processing models by using the accuracy score model comprises: Inputting the user request into the accuracy scoring model to extract a feature vector of the user request; The feature vector is converted into a probability distribution through the classification head of the accuracy scoring model, and the probability distribution is used as the accuracy score of each of the to-be-selected processing models and output.

[0013] In addition, to achieve the above-mentioned objectives, the present application also proposes a large language model efficient hybrid reasoning device based on routing technology, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the large language model efficient hybrid reasoning method based on routing technology as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the efficient hybrid reasoning method for a large language model based on routing technology as described above are implemented.

[0015] The present application provides a large language model efficient hybrid reasoning method based on routing technology. The present application determines the similarity between the user request and each historical request in the cache library if a user request is received; if there is no similarity exceeding the similarity threshold, the scoring indicators of each candidate processing model are determined based on the accuracy scoring model, and the scoring indicators include accuracy, response time and interface cost; the comprehensive score of each candidate processing model is determined according to the scoring indicator; the target model is selected from each candidate processing model according to the comprehensive score, and the user request is routed to the target model. The present application first calculates the similarity between the user request and the historical request in the cache library. When the similarity exceeds the set threshold, the historical answer corresponding to the historical request is directly returned, avoiding repeated calculation and query processes, reducing unnecessary computing resource consumption, and significantly improving the response speed. When the similarity does not meet the conditions, the accuracy of each candidate processing model is determined by the accuracy scoring model, and the accuracy, interface response time and cost of multiple candidate processing models are comprehensively evaluated, and the optimal model is selected for processing, ensuring the reasonable allocation and efficient use of resources. The present application achieves the technical effect of optimizing the selection of large language models to achieve efficient response to user requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A flowchart diagram of the first embodiment of the efficient hybrid reasoning method for a large language model based on routing technology of the present application; Figure 2 A flow chart of the second embodiment of the efficient hybrid reasoning method for a large language model based on routing technology provided by the present application; Figure 3 A flowchart diagram of the third embodiment of the large language model efficient hybrid reasoning method based on routing technology of the present application; Figure 4 A flowchart diagram of the fourth embodiment of the large language model efficient hybrid reasoning method based on routing technology of the present application; Figure 5 A brief flowchart of the efficient hybrid reasoning method for large language models based on routing technology provided by this application; Figure 6This is a schematic diagram of the device structure of the hardware operating environment involved in the efficient hybrid reasoning method for a large language model based on routing technology in an embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solutions of the embodiments of this application are: Currently, large language models are large in size and have extremely high requirements for computing and storage. They usually need to be deployed on expensive cloud servers, which have high operating costs. Small language models that can be deployed on edge devices, although lower in cost, lag behind large models in response quality and accuracy. In addition, there are differences in the task focus of different large language models. Therefore, the diversity of language models makes it difficult to choose a suitable model when performing model selection and task assignment.

[0023] This application calculates the similarity between user requests and historical requests in the cache library. When the similarity exceeds the set threshold, the historical answer corresponding to the historical request is directly returned, avoiding repeated calculations and query processes, reducing unnecessary computing resource consumption, and significantly improving the response speed. When the similarity does not meet the conditions, the accuracy of each large language model to be processed is determined through the accuracy scoring model, and the accuracy, interface response time and cost of multiple language models are comprehensively evaluated to select the optimal model for processing, ensuring the reasonable allocation and efficient use of resources.

[0024] It should be noted that the execution subject of this embodiment can be a large language model efficient hybrid reasoning system based on routing technology, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a control device of a large language model efficient hybrid reasoning system based on routing technology that can realize the above functions, etc. This embodiment does not specifically limit this. The following takes the large language model efficient hybrid reasoning system based on routing technology as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0025] Embodiment 1 Based on this, this application proposes a first embodiment of an efficient hybrid reasoning method for a large language model based on routing technology, please refer to Figure 1The efficient hybrid reasoning method for a large language model based on routing technology includes: Step S10: If a user request is received, determine the similarity between the user request and each historical request in the cache library.

[0026] The purpose of calculating the similarity between the user request and the historical requests in the cache is to quickly match the user's current request with the historical requests already in the cache. If a highly similar historical request is found, the response result corresponding to the historical request can be directly used without re-processing or calculating complexly, which can significantly improve the system's response speed, reduce the consumption of computing resources, and improve the user experience.

[0027] In this embodiment, a user request is a request sent by a user to the system through a web form, an API (Application Programming Interface) call, etc., which contains the information the user wants to obtain or the operation the user wants to perform. The cache library is a database that stores historical requests and their corresponding response results, and quickly responds to future similar requests based on the historical requests and response results in the cache library. Similarity is an indicator that measures the degree of similarity between a user request and a historical request.

[0028] As an optional implementation, user requests and historical requests are converted into text vectors, and preprocessing operations such as word segmentation, stop word removal, and stem extraction are performed on the text data in the user request to improve the accuracy of similarity calculation. A similarity algorithm is selected according to the specific type and characteristics of the data, and the similarity between the user request and each historical request in the cache is calculated. A similarity threshold is set according to actual needs to determine whether the similarity between the user request and the historical request meets the requirements. Similarity algorithms include cosine similarity, Jaccard similarity coefficient, Euclidean distance, etc.

[0029] Optionally, step S10 includes: Step S11, word segmentation processing of the user request and each of the historical requests, and converting them into a vocabulary set, and constructing a user request set and a historical request set.

[0030] In this embodiment, word segmentation is to divide the text into independent vocabulary units, which is the basis for calculating text similarity. A vocabulary set is a set of non-repeating words. Both user requests and historical requests are converted into vocabulary sets to facilitate subsequent set operations.

[0031] As an alternative implementation, the user request and the historical requests are segmented by a Chinese word segmentation tool, and the stop words in the segmentation results are removed. The processed words are converted into a set form, and a user request set and a historical request set are constructed respectively. Stop words include common words such as "de" (的) and "shi" (是) that contribute little to the meaning of the text.

[0032] Step S12: Determine the intersection of the user request set and each of the historical request sets based on the words that exist in both the user request set and each of the historical request sets.

[0033] Finding the intersection is to identify the common words between the user request and the historical requests, and these words reflect the similarity between the two.

[0034] Step S13: Combine the words in the user request set and each of the historical request sets, remove the duplicate words, and determine the union of the user request set and each of the historical request sets.

[0035] Finding the union is to identify all the non-duplicate words between the user request and the historical requests, and these words reflect the overall coverage between the two.

[0036] Step S14: Calculate each similarity based on the size of each intersection and the size of each union.

[0037] Calculating the similarity is to quantify the degree of similarity between the user request and the historical requests, so as to determine whether the response result corresponding to the historical request can be directly applied to the user request.

[0038] In this embodiment, the similarity is a numerical index measuring the degree of similarity between the user request and the historical requests, and its value range is between 0 and 1. The higher the similarity, the more similar the user request and the historical requests are.

[0039] Exemplarily, the Jaccard similarity formula is used to calculate the similarity between the user request and the historical requests.

[0040] As an alternative implementation of this embodiment, the Chinese text of the user request is segmented, and the text is split into individual words or lexical units. A stop word list is loaded to remove the stop words in the text. Stop words refer to words that frequently appear in the text but contribute little to the meaning of the text, such as "de" (的) and "shi" (是). The text after segmentation and removal of stop words is converted into a word set to ensure that the elements in the set are non-duplicate. The words that exist in both sets are found as the intersection of the two sets; all the non-duplicate words in the two sets are found as the union of the two sets. The similarity is calculated through the Jaccard similarity formula, "Jaccard similarity = size of the intersection / size of the union".

[0041] Exemplarily, text A is a user request, text B is a historical request in the cache library, and the similarity between text A and text B is calculated. Text A is "I like machine learning and artificial intelligence", and text B is "Learning artificial intelligence requires a machine learning foundation". First, word segmentation is performed, and the word segmentation results are "A={I, like, machine learning, learning, artificial intelligence}" and "B={learning, artificial intelligence, need, machine learning, foundation}". Stop words in the word segmentation results can be removed. A stop word list can be loaded, and then stop words can be removed through the stop word list. The word segmentation results after removing stop words are "A={like, machine learning, artificial intelligence}" and "B={artificial intelligence, need, machine learning, foundation}". According to the word segmentation results after removing stop words, the intersection of the two sets is calculated as "intersection={machine learning, artificial intelligence}", and the size of the intersection is 2. Then the union of the two sets is calculated as "union={like, need, foundation, machine learning, artificial intelligence}", and the size of the union is 5. The Jaccard similarity is calculated based on the size of the intersection and the size of the union, "Jaccard similarity=2 / 5=0.4". If the similarity threshold set at this time is 0.89, the similarity between text A and text B does not reach the threshold. It is necessary to calculate the similarity between text A corresponding to the user request and other historical requests in the cache. If the similarity does not reach the threshold, it will be further distributed to the large language model for response through the router.

[0042] Optionally, if the similarity exceeds a similarity threshold, the historical answer corresponding to the historical request is directly returned.

[0043] When the similarity between a user request and a historical request exceeds a preset similarity threshold, in order to improve system efficiency and user experience, the historical answer corresponding to the historical request is directly returned instead of reprocessing the user request, which can save system resources and speed up response.

[0044] In this embodiment, the similarity threshold is a preset value used to determine whether the similarity between the user request and the historical request is high enough, thereby deciding whether to directly return the historical answer.

[0045] Optionally, if the similarity between the user request and a historical request in the cache exceeds a set threshold, the response result corresponding to the historical request is directly returned; if the similarity between the user request and all historical requests is lower than the threshold, further processing or calculation is required to generate a response result.

[0046] Exemplarily, the calculated similarity is compared with a preset similarity threshold. If the similarity is greater than or equal to the threshold, the historical answer corresponding to the historical request is retrieved from the cache library, and the retrieved historical answer is directly returned to the user.

[0047] Optionally, an initial similarity threshold is set based on historical experience and industry standards, and the threshold is dynamically adjusted based on the operation of the system and changes in the data set.

[0048] Step S20: If there is no similarity exceeding the similarity threshold, the scoring indicators of each candidate processing model are determined based on the accuracy scoring model, and the scoring indicators include accuracy, response time and interface cost.

[0049] When the similarity between the user request and the historical request does not exceed the preset similarity threshold, a comprehensive evaluation method is used to select the most appropriate language model to process the user request. This method not only considers the accuracy of the language model, but also combines the response time and interface cost, thereby ensuring that while meeting the accuracy requirements, it can also achieve better performance and cost-effectiveness.

[0050] In this embodiment, the accuracy scoring model is a model for evaluating the accuracy of different large language models in processing tasks corresponding to user requests. It is trained based on historical data and can output the accuracy of each large language model in processing user requests. The selected processing model is a large language model, which is a machine learning model that can generate or process natural language text, and can process user requests and generate corresponding answers. The response time refers to the time interval from the time the selected processing model receives the user request to the time the selected processing model returns the processing result, which is used to measure the performance of the selected processing model. The interface fee is the fee required to use the selected processing model, which is related to factors such as the complexity of the selected processing model and the usage of computing resources.

[0051] As an optional implementation, user requests whose similarity with all historical requests in the cache library does not reach a threshold are input into the accuracy scoring model. Through the accuracy scoring model, the accuracy of each candidate processing model in processing the user request is output according to the content of the user request. The response time of each candidate processing model within the billing cycle is calculated and normalized to the interval of "[0, 1]", where 1 represents an immediate response without delay and 0 represents the maximum possible delay. The interface fees generated during the billing cycle of the candidate processing model are counted, and the average cost of each request is calculated based on the interface fees: "Average request cost = interface fee / number of requests".

[0052] Step S30, determining the comprehensive score of each of the to-be-selected processing models according to the scoring index.

[0053] By combining the accuracy score, interface response time, interface fee, and preset cost weights and speed weights, a comprehensive model score is calculated to comprehensively evaluate the performance, efficiency, and economy of each large language model.

[0054] In this embodiment, the comprehensive score is a score that comprehensively considers accuracy, response time, and interface cost, and is used to evaluate the overall performance of different language models in processing user requests.

[0055] In this embodiment, the cost weight is the weight ratio of the interface cost when calculating the comprehensive score of the model. The higher the cost weight, the greater the impact of the cost in the comprehensive evaluation. The speed weight is the weight ratio of the interface response time when calculating the comprehensive score of the model. The higher the speed weight, the greater the impact of the processing speed in the comprehensive evaluation.

[0056] Optionally, the response time is normalized and the calculation formula is "response time = (actual response time - minimum response time) / (maximum response time - minimum response time)", where the minimum response time and the maximum response time refer to the minimum and maximum values ​​of all response times recorded in the billing period.

[0057] Optionally, the average cost of processing each user request is calculated, and the calculation formula is "average request cost = interface cost / number of requests".

[0058] For example, through accuracy, response time, interface cost and corresponding weights, the interface response time is normalized into normalized response time, and the interface cost is reflected by the average request cost. The calculation formula of the model comprehensive score is "comprehensive score = accuracy × (1-cost weight × average request cost-speed weight × (1-response time))".

[0059] Optionally, you can set cost weights and speed weights based on business needs. By adjusting the cost weights and speed weights, you can make a trade-off between accuracy, cost, and speed. If the cost or response time has too much impact on the overall score, you can balance the accuracy, cost, and speed factors in the overall score by adjusting the corresponding weights.

[0060] Step S40, selecting a target model from each of the to-be-selected processing models according to the comprehensive score, and routing the user request to the target model.

[0061] Based on the accuracy, interface response time and interface cost of multiple candidate processing models, the optimal model is selected as the target model by calculating the comprehensive score.

[0062] In this embodiment, the comprehensive score is a score calculated based on preset weights and formulas, combined with the accuracy, interface response time, and interface cost of the processing model to be selected, and is used to measure the overall performance of the model.

[0063] As an optional implementation, among all the processing models to be selected, the model with the highest comprehensive score is selected as the target model, and the user request is sent to the target model for processing.

[0064] Optionally, an inference answer corresponding to the user request is generated through the target model, and the inference answer of the target model is returned to the user in text form based on routing technology.

[0065] This embodiment provides an efficient hybrid reasoning method for a large language model based on routing technology. This embodiment first calculates the similarity between the user request and the historical request in the cache library. When the similarity exceeds the set threshold, the historical answer corresponding to the historical request is directly returned, avoiding repeated calculations and query processes, reducing unnecessary computing resource consumption, and significantly improving the response speed. When the similarity does not meet the conditions, the accuracy of each large language model is determined by the accuracy scoring model, and the accuracy, response time and interface costs of multiple language models are comprehensively evaluated to select the optimal model for processing, ensuring the reasonable allocation and efficient use of resources.

[0066] Based on Example 1, Example 2 of the present application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 2 , before step S30, including: Step S50, collect open source question and answer data and real business question and answer data to build a data set.

[0067] Open source question-and-answer data provides a wide range of question types and answer examples, while real business question-and-answer data reflects user needs and business logic in real application scenarios. The combination of the two can build a comprehensive and targeted dataset.

[0068] In this embodiment, the open source Q&A data is a Q&A pair obtained from public Q&A platforms, forums, communities and other channels on the Internet. It is spontaneously generated by users and covers a wide range of topics and question types. Business real Q&A data is a record of real Q&A interactions of users in actual business scenarios, reflecting user needs, question types and answering habits. A data set is a collection of multiple groups of Q&A data, which is used to train, verify or test machine learning models. The format of the Q&A data is "question-standard answer".

[0069] Optionally, when obtaining open source question and answer data and real business question and answer data, it is necessary to obtain the corresponding authorization of the user.

[0070] As an optional implementation method, first collect open source Q&A data, select relevant open source Q&A platforms according to the target field or theme of the needs, and use crawler technology or API interface to crawl question and answer pairs from the selected Q&A platform. In the process of crawling the question and answer pairs, it is necessary to comply with the platform's usage agreement and data privacy policy. The captured data is preprocessed by removing duplicates, filtering spam, and standardizing the text format. Secondly, collect real business Q&A data and extract log records of user interactions from the business system, including user questions and system answers. When processing real business Q&A data, ensure the protection of user privacy and sensitive information, and perform necessary data desensitization. Integrate open source Q&A data and real business Q&A data to form a unified data set.

[0071] Step S60: classify and label each question and answer data in the data set according to each of the to-be-selected processing models.

[0072] In this embodiment, the processing model to be selected refers to a large language model with strong natural language processing capabilities, which can understand and generate coherent text. Classification labeling is the process of assigning a label to each question-answer pair in a dataset.

[0073] As an optional implementation, for each question and answer data in the data set, each large language model is used to perform inference and answer, and the inference answers of the candidate processing models are comprehensively scored to evaluate the performance of each candidate processing model, and the candidate processing model with the highest comprehensive score is selected as the selected model for the question and answer data, and the selected model is classified and labeled on the data.

[0074] Exemplarily, for each candidate processing model that generates multiple model answers under each parameter setting, the quality scores of the model answers and the corresponding standard answers in the question and answer data are calculated respectively, different weights are assigned to the quality scores obtained under different parameters according to actual needs, and the weighted average quality score is calculated as a comprehensive score.

[0075] Step S70, loading a pre-trained accuracy scoring model, and adjusting the classification head of the accuracy scoring model to the classification tasks corresponding to the number of the processing model to be selected.

[0076] In this embodiment, the pre-trained accuracy scoring model is a model that has been trained on a large-scale data set and is used to evaluate the accuracy of the text output by the candidate processing model. The classification head is the network part responsible for outputting the classification results in the neural network model, and the output dimension of the classification head needs to be adjusted to match the number of candidate processing models.

[0077] As an optional implementation, a pre-trained accuracy scoring model is loaded using a deep learning framework, and the output dimension of the classification head in the accuracy scoring model is modified according to the preset number of models to be processed.

[0078] Exemplarily, the output dimension of the classification head in the accuracy scoring model is modified by modifying the size of the weight matrix of the fully connected layer in the classification head.

[0079] Step S80, training the accuracy scoring model using the classified and labeled question and answer data.

[0080] The accuracy scoring model is trained through classified and labeled question-answering data to improve the performance of the model on question-answering tasks, so that it can more accurately evaluate the degree of match between the answers generated by the candidate processing model and the questions, and then select the appropriate candidate processing model to answer.

[0081] In this embodiment, classification labeling is the process of taking the candidate processing model that best matches the answer of the candidate processing model as the classification label of the question and answer data for each question and answer data in the data set.

[0082] For example, the data set is divided into a training set, a validation set, and a test set. The accuracy model is trained using the classified and labeled training set. The loss function is minimized by iteratively adjusting the model parameters to improve the accuracy of the accuracy scoring model on the validation set. The accuracy scoring model is evaluated using the test set data. The performance of the model is measured by calculating indicators such as accuracy, recall, and F1 score, and the model hyperparameters are adjusted based on the evaluation results.

[0083] Optionally, step S80 includes: Step S81, input the classified and labeled question and answer data into the accuracy scoring model, perform word segmentation processing, and convert it into a vector representation of a fixed length.

[0084] Preprocess the classified and labeled question-answering data into a format acceptable to the model so that it can be input into the accuracy scoring model for training.

[0085] In this embodiment, since the machine learning model usually requires the input data to have a fixed dimension and length, the text data needs to be converted into a vector representation of a fixed length.

[0086] As an optional implementation, a word segmentation tool is used to segment the content of the categorized and labeled question and answer data, and then a fixed-length vector is generated by counting the number of occurrences of each word in the text, in which each element corresponds to the number of occurrences of a word. The vector representation of the preprocessed question and answer data is input into the accuracy scoring model.

[0087] As another optional implementation of converting to a fixed-length vector representation, each word is mapped into a high-dimensional vector space through a pre-trained word embedding model, and the average or weighted sum of the vectors of all words in the text is calculated as the vector representation of the text.

[0088] Optionally, if the lengths of the vectorized text vectors are not consistent, length normalization is required to ensure that all input data have the same dimensions and lengths, either by truncating overlong vectors or padding shorter vectors to the required length.

[0089] Step S82, training the accuracy scoring model using a standard training cycle, and measuring the difference between the model's predicted probability distribution and the true label using a cross entropy loss function.

[0090] In this embodiment, the standard training cycle is an iterative training process, including steps such as forward propagation, loss calculation, back propagation, and parameter update. In each iteration, the accuracy scoring model makes predictions based on the input data, then calculates the difference between the prediction result and the true label, i.e., the loss, and uses this loss to update the parameters of the accuracy scoring model. The iterative process is repeated multiple times until the performance of the accuracy scoring model on the validation set reaches stability or the preset training round ends.

[0091] In this embodiment, the cross entropy loss function is a commonly used loss function used to measure the difference between the probability distribution predicted by the model and the true label in the classification task, that is, the loss. The cross entropy loss function calculates the relative entropy between the predicted probability of each category and the true label of the accuracy scoring model and averages them. The smaller the cross entropy loss, the closer the prediction of the accuracy scoring model is to the true label.

[0092] Optionally, key parameters such as the number of iterations of the training loop, batch size, learning rate, etc. are pre-set.

[0093] As an optional implementation, in each iteration of the training process, a batch of question-answering data in the training set is input into the accuracy scoring model, the predicted probability distribution of the accuracy scoring model is calculated through forward propagation, and the cross-entropy loss function is used to calculate the difference between the probability distribution predicted by the accuracy scoring model and the true label.

[0094] Step S83: Use an optimization algorithm to update the parameters of the accuracy scoring model to minimize the loss function.

[0095] Use an optimization algorithm to adjust the parameters of the accuracy scoring model to minimize the loss function. In machine learning and deep learning, the loss function is a key indicator to measure the difference between the prediction results of the accuracy scoring model and the actual label. By minimizing the loss function, the model can gradually learn the inherent laws of the data and improve the accuracy of the prediction.

[0096] In this embodiment, the optimization algorithm is used to adjust the parameters of the accuracy scoring model to minimize the loss function. The optimization algorithm includes Stochastic Gradient Descent (SGD), Momentum, RMSprop, Adam, etc.

[0097] As an optional implementation, according to the gradient information of the loss function, the error signal is transmitted back to each layer of the accuracy scoring model through the back propagation algorithm, and the parameters of the accuracy scoring model are updated according to the back-propagated gradient information using optimization algorithms such as stochastic gradient descent and Adam.

[0098] Optionally, after each iteration, the performance of the accuracy scoring model is evaluated using the validation set. If the performance of the accuracy scoring model does not improve significantly in multiple consecutive iterations, the early stopping strategy is used to terminate the training process to avoid overfitting. At the same time, hyperparameters such as learning rate and batch size are adjusted according to the performance on the validation set.

[0099] This embodiment provides an efficient hybrid reasoning method for a large language model based on routing technology. This embodiment first improves the diversity of the data set by collecting open source and real business question and answer data. By using various language models for classification and labeling, the accuracy scoring model can more accurately evaluate the degree of matching between the question and answer data and the large language model, and further determine the most suitable large language model for the request in the question and answer data, thereby improving the overall efficiency and accuracy of the system. Through targeted training data, the model can converge to the optimal solution more quickly, thereby improving training efficiency.

[0100] Based on the first and second embodiments, the third embodiment of the present application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 3 , step S60 comprises: Step A10, inputting the question in each of the question-answer data into each of the processing models to be selected, performing inference and answering through each of the processing models to be selected, and generating at least one answer result.

[0101] In this embodiment, the question-answering data is a data set containing questions and corresponding answers, which is used to train and evaluate the performance of the accuracy scoring model. The candidate processing model is a large language model with strong language understanding and generation capabilities, which can process and understand complex natural language texts. The reasoning answer is the process in which the candidate processing model understands and analyzes the input question through its internal mechanism and finally generates an answer.

[0102] As an optional implementation method, a question-and-answer dataset is collected and organized, multiple large language models are selected as candidate processing models, and each question in the question-and-answer dataset is input into each candidate processing model in turn. Each candidate processing model performs reasoning based on the input question and generates at least one answer result.

[0103] Step A20, calculating the weighted score of each of the candidate processing models according to the answer results of each of the candidate processing models, and taking the candidate processing model with the highest weighted score as the classification label of the question and answer data.

[0104] In this embodiment, the weighted score refers to the score calculated based on the answer result of the candidate processing model and the preset weight, which is used to evaluate the performance of the candidate processing model in answering the user request. Classification labeling refers to assigning each question and answer data in the data set to the best performing candidate processing model as a classification label.

[0105] Optionally, weights of various evaluation indicators are set according to task requirements and evaluation criteria.

[0106] As an optional implementation method, based on the answer results of each candidate processing model and the preset weight, the weighted score of each candidate processing model on the question and answer data set is calculated, the weighted scores of different candidate processing models are compared, and the candidate processing model with the highest score is selected as the classification label of the question and answer data.

[0107] Exemplarily, by calculating the accuracy, recall rate, F1 score and other evaluation indicators of the candidate processing model on each question, and performing weighted summation according to the weights, the weighted score of each candidate processing model is obtained, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0108] Optionally, step A20 includes: Step A21, compare each of the answer results generated by each of the candidate processing models with the standard answer in the question and answer data, and calculate the accuracy score.

[0109] In this embodiment, the answer result is the output generated by the processing model to be selected based on the input question, which can be text, number or other forms. The standard answer is the answer provided for each question in the question and answer data. The accuracy score is the degree to which the answer result generated by the processing model to be selected matches the standard answer.

[0110] As an optional implementation, various language models are used to infer historical requests in the question-and-answer data to generate answer results, the answer results of each large language model are compared with the standard answer, and the accuracy score is calculated through string matching, semantic similarity calculation and other methods.

[0111] Step A22, for each of the to-be-selected processing models, calculate the variance according to the accuracy score, and calculate the weighted score according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance.

[0112] In this embodiment, variance is an indicator to measure the degree of dispersion of data distribution, and is used to evaluate the fluctuation of accuracy scores of the candidate processing model on different issues. The score weight is the proportion of the accuracy score when calculating the weighted score. The variance weight is the proportion of the variance when calculating the weighted score, which is used to balance the accuracy and stability of the candidate processing model. The weighted score is a comprehensive score calculated by combining the accuracy score and variance, as well as their respective weights, and is used to evaluate the overall performance of the candidate processing model.

[0113] Optionally, weights for accuracy score and variance are set according to task requirements and evaluation criteria.

[0114] Exemplarily, the values ​​of the score weight and the variance weight are dynamically adjusted according to the type or difficulty of the query. For example, for complex queries, the variance weight can be increased so that the router is more inclined to select a model with a large variance; while for simple queries, the score weight value is increased to give priority to selecting a model with a higher score.

[0115] As an optional implementation, for each candidate processing model, the variance of the accuracy score of the candidate processing model on all historical requests is calculated, and the weighted score of each model is calculated by combining the accuracy score, the variance and their respective weights.

[0116] Exemplarily, the formula for calculating the weighted score is “weighted score=accuracy score×score weight+variance×variance weight”.

[0117] Step A23, comparing the weighted scores of the candidate processing models, and taking the candidate processing model with the highest weighted score as the classification label of the corresponding question and answer data.

[0118] Exemplarily, the weighted scores of all candidate processing models are compared to find the candidate processing model with the highest score, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

[0119] As an example of this embodiment, for each piece of data in the data set, four different large language models "S1, S2, L1, L2" are used to calculate the weighted score respectively. Each large language model uses three different sets of temperature parameters (0, 7, 10) for reasoning on a question, where the latter two sets of parameters each generate 10 multi-sampling results, and a total of 21 sets of model answers are obtained. These 21 sets of answers are compared with the standard answers, scored using bart score, 21 accuracy scores are obtained, and the variance of these 21 scores is calculated. The comprehensive score of each large language model is calculated using the weighted score formula, and the formula is "Weighted Score=α×p+β×Variance", "Weighted Score" is the weighted score, "p" is the accuracy score of the large language model, "Variance" is the variance of the score of the large language model under different conditions, "α" and "β" are weight parameters used to adjust the impact of the score and variance in the comprehensive score, "α" represents the score weight, and "β" represents the variance weight. The large language model with the highest score is selected as the classification label of the data. For example, if the “L1” model has the highest score, the classification label of the data is “(S1, S2, L1, L2) = (0, 0, 1, 0)”.

[0120] This embodiment provides an efficient hybrid inference method for a large language model based on routing technology. This embodiment first calculates the accuracy score to intuitively understand the accuracy of each large language model in answering user requests, and determines the stability of the large language model by calculating the variance of the large language model when answering different requests. Then, based on the accuracy and stability of the large language model, the performance of different large language models is quantitatively compared.

[0121] Based on Example 1, Example 4 of the present application proposes an efficient hybrid reasoning method for a large language model based on routing technology, referring to Figure 4 , step S30 comprises: Step S31, predicting the accuracy score of each of the to-be-selected processing models through the accuracy score model.

[0122] Using the trained accuracy scoring model, the accuracy of each candidate processing model in processing user requests is predicted and scored. The score can reflect the candidate processing model's ability to understand user requests and the accuracy of its answers, thereby selecting the most appropriate candidate processing model to answer and improve the overall performance of the system and user satisfaction.

[0123] In this embodiment, the accuracy score is a quantitative evaluation of the accuracy of the selected processing model in processing the user request. The higher the score, the more accurate the model's understanding and response to the user request.

[0124] As an optional implementation, the user request is input into the accuracy scoring model, and the parameters and logic of the accuracy scoring model are used to predict the possible answers given by each candidate processing model, and the corresponding accuracy score is calculated. After the accuracy scoring model prediction is completed, the accuracy score of each candidate processing model is output.

[0125] Optionally, step S31 includes: Step B10: input the user request into the accuracy scoring model to extract the feature vector of the user request.

[0126] The user request is input into the trained accuracy scoring model, and the feature vector of the user request is extracted through the model. By extracting the feature vector, the user request can be converted into a form that the model can understand, providing a basis for subsequent accuracy scoring and model selection.

[0127] In this embodiment, the feature vector is a multidimensional vector, each component of which is a numerical representation of a user request on a certain feature, including lexical features, grammatical features, semantic features, etc., which are used to capture key information of the user request.

[0128] As an optional implementation, the user request is cleaned, segmented, and stop words are removed, and the preprocessed user request is input into the accuracy scoring model. The user request is converted into a point in a high-dimensional space, where each dimension represents a feature. After feature extraction, the accuracy scoring model outputs a feature vector that contains the representation of the user request in the feature space learned by the model.

[0129] Step B20: convert the feature vector into a probability distribution through the classification head of the accuracy scoring model, and output the probability distribution as the accuracy score of each of the to-be-selected processing models.

[0130] In this embodiment, the classification head is a key component in the accuracy scoring model, which is responsible for converting the feature vector into a probability distribution. The classification head contains one or more layers of neural networks, which use the learned weights and biases to map the feature vector to the probability space. The feature vector is a numerical vector extracted from the user request that can represent its key information. The feature vector is an important part of the model input and contains key information for evaluating the accuracy of the candidate processing model. The probability distribution is a prediction of the accuracy of each candidate processing model for processing the user request. Each value in the probability distribution corresponds to a candidate processing model, and the size of the value represents the predicted value of the accuracy of the model processing the request.

[0131] As an optional implementation, the extracted feature vector of the user request is input into the classification head of the accuracy scoring model. The classification head uses the learned weights and biases to perform linear transformation and nonlinear activation operations on the feature vector, calculates the accuracy prediction value corresponding to each candidate processing model, and normalizes it to a probability distribution. The calculated probability distribution is output as the accuracy score of each candidate processing model.

[0132] Step S32, performing a stress test on the interface of each of the processing models to be selected to obtain the response time.

[0133] Stress test the interface of each candidate processing model to evaluate its performance and stability under actual load. By simulating scenarios such as high concurrent requests or large amounts of data processing, measure the interface response time to measure interface performance and user experience. The shorter the interface response time, the less time users have to wait for results, and the better the experience.

[0134] In this embodiment, the interface is an API or service endpoint provided by each candidate processing model, through which the functions of the candidate processing model can be called. Stress testing, also known as load testing or performance testing, is a testing method that evaluates the performance of an interface by simulating actual or expected load conditions, aiming to discover the behavior of the interface under extreme conditions, including key performance indicators such as response time, throughput, and error rate. Response time is the time required from when a request is sent to an interface to when the interface returns a response, and is a key indicator for measuring interface performance.

[0135] As an optional implementation, according to the test goal of measuring the response time under load, design a test scenario, including request type, request frequency, data size, etc., to simulate actual or expected load conditions. Execute the test script through the stress testing tool to perform stress testing and monitor the response time of each candidate processing model interface.

[0136] Step S33, calculating the computing power cost required for each of the processing models to be selected, and determining the interface fee.

[0137] Evaluate and calculate the computing power cost required for each candidate processing model in actual application, and then determine the interface service fees provided for these models. The computing power cost is an indispensable resource consumption indicator when the candidate processing model runs and provides services, and is directly related to the operating cost.

[0138] In this embodiment, the computing power cost refers to the computing resource consumption required to run the processing model to be selected, including the cost of hardware resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, storage, etc., as well as indirect costs such as cloud services and data center operations involved. The interface fee is the fee within the billing period for using the processing model to be selected for reasoning and answering.

[0139] As an optional implementation, the performance of each candidate processing model is comprehensively evaluated, including indicators such as processing speed, accuracy, and memory usage. In the actual operation process, the resource consumption of each candidate processing model when processing requests is monitored in real time, including CPU usage, GPU memory usage, network bandwidth, etc. Based on the resource consumption monitoring data, a computing power cost calculation model is constructed. Through the computing power cost calculation model, the interface fee is estimated based on factors such as the unit cost of hardware resources, resource utilization, and model running time.

[0140] Optionally, for a large language model provided by a third-party model supplier, the interface fee may be calculated directly based on the fees charged by the third-party model supplier.

[0141] This embodiment provides an efficient hybrid reasoning method for large language models based on routing technology. This embodiment first predicts the accuracy scores of each language model through an accuracy scoring model, providing a quantitative evaluation indicator for model performance. By stress testing the interfaces of each language model, the response time of the interface under different loads can be obtained to evaluate the processing speed of the model. The interface fee is determined by calculating the computing power cost required for each language model to further control the cost. The comprehensive score of the model is calculated based on the accuracy score, interface response time and interface fee, as well as the preset cost weight and speed weight. By comparing the comprehensive scores of different models, the large language model that best suits the user's request can be selected.

[0142] For example, to help understand the implementation process of the high-efficiency hybrid reasoning method for a large language model based on routing technology obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5 A brief flowchart of an efficient hybrid reasoning method for a large language model based on routing technology is provided. Specifically: When receiving a user request input by the user to the system, the cache is first checked to determine whether a similar user request has been processed. If there is a matching historical user request in the cache, the historical answer corresponding to the cached historical user request is directly returned.

[0143] If no matching historical user request is found in the cache, the query is sent to the router.

[0144] The router uses the routing service to decide which model to send the user request to based on the characteristics of the user request.

[0145] Optionally, set the dataset space of user requests to X, and the dataset space of reasoning answers to Z. "L1: X->Z" indicates that a larger model completes the reasoning service, and "S1: X->Z" indicates that a smaller model completes the reasoning service. "r1: X->(0, 1)" indicates a routing service, where 1 represents routing to S1 and 0 represents routing to L1.

[0146] Exemplarily, "x" represents a user request in the user request dataset X, and the output of "r1(x)" is a probability value indicating which model the user request should be routed to. When "r1(x)=0", the user request will be sent to the larger model L1; if "r1(x)=1", the user request will be sent to another smaller model S1.

[0147] Furthermore, “q:z->R” is used to represent the scoring function “q(z)”, and the answer “z” is scored using the BART Score method, where “z” represents an inference answer in the inference answer dataset Z. BART Score is a scoring method based on the BART model, which is used to evaluate the performance of natural language generation systems.

[0148] Furthermore, the inference answer of the model with the highest score is returned to the user through the router.

[0149] The entire process selects the most appropriate model to generate high-quality answers based on the characteristics of user requests, and improves efficiency through a caching mechanism.

[0150] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the efficient hybrid reasoning method of large language models based on routing technology of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0151] The present application provides a large language model efficient hybrid reasoning device based on routing technology, and the large language model efficient hybrid reasoning device based on routing technology includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model efficient hybrid reasoning method based on routing technology in the above-mentioned embodiment one.

[0152] Reference below Figure 6 , which shows a schematic diagram of the structure of a large language model efficient hybrid reasoning device based on routing technology suitable for implementing the embodiment of the present application. The large language model efficient hybrid reasoning device based on routing technology in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDA, Personal Digital Assistant), tablet computers (PAD, Portable Application Description), portable multimedia players (PMP, Portable Media Player), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The large language model efficient hybrid reasoning device based on routing technology shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Figure 6As shown, the large language model efficient hybrid reasoning device based on routing technology may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM, Random Access Memory) 1004. In the random access memory 1004, various programs and data required for the operation of the large language model efficient hybrid reasoning device based on routing technology are also stored. The processing device 1001, the read-only memory 1002 and the random access memory 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the large language model efficient hybrid reasoning device based on routing technology to communicate wirelessly or wired with other devices to exchange data. Although the large language model efficient hybrid reasoning device based on routing technology with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0154] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The large language model efficient hybrid reasoning device based on routing technology provided by the present application adopts the large language model efficient hybrid reasoning method based on routing technology in the above embodiment, which can solve the technical problem of how to optimize the selection of large language models to achieve efficient response to user requests. Compared with the prior art, the beneficial effects of the large language model efficient hybrid reasoning device based on routing technology provided by the present application are the same as the beneficial effects of the large language model efficient hybrid reasoning method based on routing technology provided by the above embodiment, and the other technical features of the large language model efficient hybrid reasoning device based on routing technology are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0158] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the efficient hybrid reasoning method for large language models based on routing technology in the above-mentioned embodiment.

[0159] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM, CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0160] The computer-readable storage medium may be included in the large language model efficient hybrid reasoning device based on routing technology; or may exist independently without being assembled into the large language model efficient hybrid reasoning device based on routing technology.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the large language model efficient hybrid reasoning device based on routing technology, the large language model efficient hybrid reasoning device based on routing technology can be written in one or more programming languages ​​or a combination thereof to perform computer program codes for the operation of the present application. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0164] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large language model efficient hybrid reasoning method based on routing technology, and can solve the technical problem of how to optimize the selection of large language models to achieve efficient response to user requests. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the large language model efficient hybrid reasoning method based on routing technology provided in the above-mentioned embodiment, and will not be repeated here.

[0165] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An efficient hybrid reasoning method for large language models based on routing technology, characterized in that: The efficient hybrid reasoning method for a large language model based on routing technology includes: If a user request is received, determining the similarity between the user request and each historical request in the cache; If there is no similarity exceeding the similarity threshold, determining the scoring indicators of each candidate processing model based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time and interface cost; Determine the comprehensive score of each of the processing models to be selected according to the scoring index; A target model is selected from each of the to-be-selected processing models according to the comprehensive score, and the user request is routed to the target model.

2. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 1, characterized in that: If a user request is received, the step of determining the similarity between the user request and each historical request in the cache library comprises: Process the user request and each of the historical requests by word segmentation, and convert them into a vocabulary set, and construct a user request set and a historical request set; Determine an intersection of the user request set and each of the historical request sets according to words that exist in both the user request set and each of the historical request sets; Merging the words in the user request set and each of the historical request sets, and removing duplicate words, to determine a union of the user request set and each of the historical request sets; The similarities are calculated according to the size of each intersection and the size of each union.

3. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 1, characterized in that: Before the step of determining the scoring indicators of each candidate processing model based on the accuracy scoring model if there is no similarity exceeding the similarity threshold, wherein the scoring indicators include accuracy, response time and interface cost, the method includes: Collect open source question-answering data and real business question-answering data to build a data set; According to each of the to-be-selected processing models, classify and label each question and answer data in the data set; Loading a pre-trained accuracy scoring model, and adjusting the classification head of the accuracy scoring model to a corresponding number of classification tasks of the to-be-selected processing model; The accuracy scoring model is trained using the question-answer data that is labeled by classification.

4. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 3, characterized in that: The step of classifying and labeling each question and answer data in the data set according to each of the to-be-selected processing models comprises: Inputting the question in each of the question-answer data into each of the processing models to be selected, and performing reasoning and answering through each of the processing models to be selected to generate at least one answer result; According to the answer results of each of the candidate processing models, a weighted score of each of the candidate processing models is calculated, and the candidate processing model with the highest weighted score is used as the classification label of the question and answer data.

5. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 4, characterized in that: The step of calculating the weighted scores of the candidate processing models according to the answer results of the candidate processing models, and using the candidate processing model with the highest weighted score as the classification label of the question and answer data includes: Compare each of the answer results generated by each of the candidate processing models with the standard answer in the question and answer data to calculate the accuracy score; For each of the processing models to be selected, the variance is calculated according to the accuracy score, and the weighted score is calculated according to the score weight corresponding to the accuracy score and the variance weight corresponding to the variance; The weighted scores of the candidate processing models are compared, and the candidate processing model with the highest weighted score is used as the classification label of the corresponding question and answer data.

6. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 3, characterized in that: The step of training the accuracy scoring model using the question-answer data with classification annotations includes: Input the classified and labeled question-answer data into the accuracy scoring model, perform word segmentation processing, and convert it into a vector representation of a fixed length; The accuracy scoring model is trained using a standard training loop, and the difference between the model's predicted probability distribution and the true label is measured using a cross-entropy loss function; The parameters of the accuracy scoring model are updated using an optimization algorithm to minimize the loss function.

7. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 1, characterized in that: If there is no similarity exceeding the similarity threshold, the step of determining the scoring indicators of each candidate processing model based on the accuracy scoring model, wherein the scoring indicators include accuracy, response time and interface cost comprises: Predicting the accuracy score of each of the processing models to be selected by using the accuracy score model; Performing stress testing on the interface of each of the processing models to be selected to obtain the response time; The computing power cost required for each of the processing models to be selected is calculated to determine the interface fee.

8. The efficient hybrid reasoning method for large language models based on routing technology as claimed in claim 7, characterized in that: The step of predicting the accuracy score of each of the to-be-selected processing models by using the accuracy score model comprises: Inputting the user request into the accuracy scoring model to extract a feature vector of the user request; The feature vector is converted into a probability distribution through the classification head of the accuracy scoring model, and the probability distribution is used as the accuracy score of each of the to-be-selected processing models and output.

9. An efficient hybrid reasoning device for large language models based on routing technology, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large language model efficient hybrid reasoning method based on routing technology as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large language model efficient hybrid reasoning method based on routing technology as described in any one of claims 1 to 8 are implemented.

Images