Problem classification optimization method and device of large language model and storage medium

Abstract

According to the problem classification optimization method for the large language model, in the current period, each target problem is input into the current large language model, and the initial prediction category of each target problem is obtained; for each target problem and the initial prediction category thereof, recalling a first historical data set according to the target problem in the current case database according to the total number of the current problem category pairs, and recalling a second historical data set according to the initial prediction category; for each target problem, inputting the target problem and the first historical data set and the second historical data set corresponding to the target problem into the current large language model to obtain a target prediction category of the target problem; if it is determined that the overall correct rate of each target prediction category in the current period is smaller than a preset threshold value, the total number of the current problem category pairs is increased and then serves as a new total number of the current problem category pairs, and the next period is entered. In this way, the problem that errors are concentrated and continuously exist in large model classification is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and storage medium for optimizing question classification for a large language model. Background Art

[0002] In intelligent customer service or user question classification scenarios, large models are often used to categorize user questions into specific types and modules to quickly respond to user needs. Currently, large model classification demonstrates high accuracy in practical applications.

[0003] However, with the increasing complexity of business scenarios and the diversification of user needs, certain classification errors still occur. While these errors can be corrected through fine-tuning of large models, fine-tuning typically requires a learning cycle of several hours or even longer, making it difficult to meet the demand for rapid adjustments. Furthermore, existing training processes are mostly conducted on a fixed cycle, which, regardless of the number of questions, fails to match the changing trends of actual online questions. Consequently, large language models struggle to respond promptly when classification errors occur in droves, causing them to persist for a period of time, impacting the overall accuracy of online classification. Summary of the Invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect in the prior art that large language models are difficult to respond in a timely manner when classification errors occur in a cluster, resulting in the classification errors persisting for a period of time, thereby affecting the overall accuracy of online classification.

[0005] In a first aspect, the present application provides a method for optimizing question classification of a large language model, the method comprising:

[0006] In the current cycle, each target question is input into the current large language model to obtain the initial predicted category of each target question;

[0007] For each target question and its initial predicted category, according to the total number of current question category pairs, recall a first historical data set based on the target question in the current case database, and recall a second historical data set based on the initial predicted category, wherein the first historical data set and the second historical data set both include historical questions and their predicted probabilities and true categories in each category;

[0008] For each target question, input the target question and its corresponding first historical data set and second historical data set into the current large language model to obtain a target prediction category for the target question;

[0009] If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the total number of current problem category pairs is incremented as the new total number of current problem category pairs and the next cycle is entered.

[0010] In one embodiment, the step of recalling a first historical data set in the current case database according to the target problem includes:

[0011] Calculate the text similarity between the target question and each historical question in the current case database;

[0012] In descending order of the text similarity of each historical question, the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category are selected as the first historical data set.

[0013] In one embodiment, the step of recalling a second historical data set according to the initial predicted category includes:

[0014] Obtain the predicted probability of each historical question in the current case database in the initial prediction category;

[0015] In descending order of the predicted probability of each historical question in the initial predicted category, select the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category as the second historical data set.

[0016] In one embodiment, the process of determining the overall accuracy of each target prediction category in the current cycle includes:

[0017] Get the true category of each target question in the current cycle;

[0018] Compare the target predicted category and the true category of each target question, and calculate the overall accuracy based on the comparison results.

[0019] In one embodiment, the step of incrementing the total number of current question category pairs as a new total number of current question category pairs includes:

[0020] Determine the increment of question categories based on the overall correct rate;

[0021] The sum of the problem category pair increment and the current total number of problem category pairs is taken as the new current total number of problem category pairs.

[0022] In one embodiment, the method further comprises:

[0023] If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the offline training of the large language model is triggered. When the trained large language model meets the preset training end requirements, the trained large language model will be used as the current large language model.

[0024] In one embodiment, the offline training process of a large language model includes:

[0025] Determine the true category of each target question in the current cycle, and use each target question in the current cycle and its latest predicted probability and true category in each category, as well as the current case database, to perform offline training on the current large language model.

[0026] In a second aspect, the present application provides a question classification optimization device for a large language model, the device comprising:

[0027] The initial prediction category acquisition module is used to input each target question into the current large language model in the current cycle to obtain the initial prediction category of each target question;

[0028] a historical data set acquisition module, configured to, for each target question and its initial predicted category, recall a first historical data set based on the target question in the current case database according to the total number of current question category pairs, and recall a second historical data set based on the initial predicted category, wherein both the first historical data set and the second historical data set include historical questions and their predicted probabilities and true categories in each category;

[0029] A target prediction analogy acquisition module is used to input each target question and its corresponding first historical data set and second historical data set into the current large language model to obtain a target prediction category for the target question;

[0030] The total number of question category pairs updating module is used to increment the total number of current question category pairs as the new total number of current question category pairs if it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, and enter the next cycle.

[0031] In a third aspect, the present application provides a storage medium: the storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the question classification optimization method for a large language model as in any of the above embodiments.

[0032] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0033] Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by one or more processors, the steps of the question classification optimization method for a large language model in any of the above embodiments are performed.

[0034] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0035] In the problem classification optimization method, device and storage medium of the large language model provided in the present application, by combining the historical data of the target problem, especially the first historical data set and the second historical data set including the predicted probability and the true category, it is possible to provide the large language model with richer contextual information, which helps the model to more accurately determine the target prediction category of the target problem, thereby significantly improving the classification accuracy; when the overall accuracy is lower than the preset threshold, the total number of problem category pairs is dynamically adjusted, which enables the classification method to adapt to changes in the online problem distribution, solves the limitation that the traditional fixed-cycle training process cannot quickly respond to problem changes, and improves the flexibility and real-time adaptability of classification; the method quickly discovers and solves classification errors through the adjustment process within the cycle, rather than waiting for the fixed cycle to end and then uniformly handle them, thereby significantly shortening the existence time of classification errors. In summary, this method realizes the ability to quickly respond to changes in problem distribution through the combination of dynamic adjustment mechanism, historical data recall and model re-prediction, solves the pain point of concentrated and persistent errors in large model classification, and reduces the dependence on model fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A flowchart of a method for optimizing question classification using a large language model according to an embodiment of the present application;

[0038] Figure 2 An example diagram of the question classification optimization method for a large language model provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a large language model question classification optimization device provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] This application provides a method for optimizing question classification of a large language model. The following embodiments are described using the method applied to a computer device as an example. It is understood that the computer device can be any device with data processing capabilities, including but not limited to a single server, a server cluster, a personal laptop computer, a desktop computer, etc. Figure 1 As shown, the method may include the following steps:

[0043] S101: In the current cycle, each target question is input into the current large language model to obtain an initial predicted category for each target question.

[0044] The target question is a user question or input content to be classified, which is classified by the large language model to identify its category. The current large language model is a pre-trained language model currently in use. It has the ability to understand and classify natural language and is used to predict the initial category of the target question. The initial predicted category is the category output by the large language model after the first classification of the target question, without further processing or optimization.

[0045] In this step, after receiving the target question, the target question is used as input to call the inference interface of the current large language model. The model's forward propagation process is executed through the computing device's hardware to generate the corresponding output. The model outputs a classification probability distribution or category label, from which the category with the highest probability is extracted as the initial predicted category. After receiving the target question, it can be converted into a format acceptable to the model and checked for invalid characters or additional cleaning, which is called preprocessing of the target question.

[0046] It can be understood that the initial predicted category can reflect the performance of the large language model in its current state, providing a basic result for subsequent optimization steps. It can be further adjusted in combination with historical data to quickly calibrate the classification effect.

[0047] S102: For each target question and its initial predicted category, according to the total number of current question category pairs, recall a first historical data set based on the target question in the current case database, and recall a second historical data set based on the initial predicted category, wherein the first historical data set and the second historical data set both include historical questions and their predicted probabilities and true categories in each category.

[0048] The total number of current question category pairs represents the total number of question category pairs required to be retrieved from the case database during the current cycle. The case database stores historical questions and their classification information, including data such as predicted probabilities and true categories. The first historical data set consists of historical questions and related classification data recalled from the case database based on the target question, helping to analyze the similarity distribution of the target question. The second historical data set consists of historical questions and classification data recalled from the case database based on the initial predicted category of the target question, reflecting the distribution of that predicted category.

[0049] In this step, recall refers to retrieving data from the case database that meets specific criteria. Specifically, retrieval criteria can be constructed based on the characteristics of the target question, such as text content or keywords, to query the first historical data set, and query criteria can be generated based on the initial predicted category of the target question to query the second historical data set. The current case database is queried, matching the above criteria, and filtering out historical data related to the target question or the initial predicted category. The query results include information such as the text of the historical question, the category prediction probability, and the actual category. The retrieved historical data is then grouped and organized to form the first and second historical data sets, respectively.

[0050] For example, assuming the target question is "How do I apply for an invoice?" and the initial prediction category is "financial problem", the case database is queried for questions related to "how to apply for an invoice" and the predicted probability and true category in each category to form a first historical data set, and the case database is queried for questions belonging to the "financial problem" category and their classification information to form a second historical data set.

[0051] It can be understood that by recalling the first historical data set, similar cases can be found based on the contextual characteristics of the target problem, helping the large language model to refer to the handling methods of similar problems when classifying. By recalling the second historical data set, the consistency of the initial prediction category in the historical data can be verified, thereby enhancing the credibility of the classification results. The recall operation allows the large language model to reuse the experience in historical data in real time during the classification process, avoiding accidental errors that may be caused by a single prediction. By analyzing the true category distribution of historical problems, the large language model can identify which categories are more likely to be misclassified, thereby adjusting the weight or feature selection strategy to reduce the probability of misclassification in complex scenarios.

[0052] S103: For each target question, the target question and its corresponding first historical data set and second historical data set are input into the current large language model to obtain a target prediction category for the target question.

[0053] Among them, the target prediction category is based on the target problem and its corresponding historical data set, and the final classification result is obtained by re-inference through the large language model as the optimized output.

[0054] In this step, the target question, the first historical data set, and the second historical data set must be integrated into an input format acceptable to the model. This can be done by combining a unified text input, such as the question text + similar questions + category information, or by constructing a structured input, such as the target question embedding + historical data feature vectors. The inference interface of the current large language model is then called, and the integrated input data is fed into the model for forward propagation. During inference, the large language model combines the characteristics of the historical data with the target question to generate a prediction. The large language model outputs a classification probability distribution, and the category with the highest probability is selected as the target predicted category.

[0055] It is understandable that relying solely on the target question for prediction may lead to misclassification due to the ambiguity of the question expression. Attaching the corresponding historical data set when inputting the target question can provide a richer semantic context for the large language model, calibrate the initial prediction of the target question, ensure that the classification result conforms to the actual distribution characteristics of the question, and make up for the problem of insufficient information in a single prediction.

[0056] S104: If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the total number of current question category pairs is incremented as the new total number of current question category pairs, and the next cycle is entered.

[0057] In this step, the classification results are analyzed to determine whether the overall accuracy is below the preset threshold. If so, the result is marked as "not meeting the requirements." Otherwise, the result is marked as "meeting the requirements." If the overall accuracy is below the preset threshold, the total number of new category pairs is calculated based on the incremental value, which serves as a reference value for the next cycle.

[0058] As you can see, by evaluating the overall accuracy and adjusting the number of category pairs immediately when accuracy falls below a threshold, we can quickly respond to clusters of misclassification errors and prevent them from persisting for a period of time. By incrementally adjusting the total number of question category pairs, we can introduce more fine-grained classification rules, making it easier to distinguish between different categories of the target question and reducing the occurrence of classification confusion.

[0059] In the above embodiment, by combining the historical data of the target problem, especially the first historical data set and the second historical data set including the predicted probability and the true category, it is possible to provide richer contextual information for the large language model, which helps the model to more accurately determine the target prediction category of the target problem, thereby significantly improving the classification accuracy; when the overall accuracy is lower than the preset threshold, the total number of question category pairs is dynamically adjusted, which enables the classification method to adapt to changes in the online question distribution, solves the limitation that the traditional fixed-cycle training process cannot quickly respond to problem changes, and improves the flexibility and real-time adaptability of classification; this method quickly discovers and solves classification errors through the adjustment process within the cycle, rather than waiting for the fixed cycle to end and then uniformly handle them, thereby significantly shortening the existence time of classification errors. In summary, this method achieves the ability to quickly respond to changes in question distribution through the combination of dynamic adjustment mechanism, historical data recall and model re-prediction, solves the pain point of concentrated and persistent errors in large model classification, and reduces the dependence on model fine-tuning.

[0060] In one embodiment, the step of recalling a first historical data set in the current case database according to the target problem includes:

[0061] Calculate the text similarity between the target question and each historical question in the current case database;

[0062] In descending order of the text similarity of each historical question, the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category are selected as the first historical data set.

[0063] Specifically, the target question can be cleaned and converted into a vector. Then, using cosine similarity or Euclidean distance, the similarity between the target question and each historical question vector in the case database is calculated. Based on the calculated text similarity, all historical questions are sorted from largest to smallest. The top N most similar historical questions are selected from the sorted list based on the total number of current question categories. The text content, predicted probability, and true category of these historical questions are extracted to form the first historical data set.

[0064] In this example, by comparing the semantic similarity between the target question and historical questions, we can find the most valuable reference examples from the case database for the current question. By selecting historical questions that are most similar to the target question, we can obtain more targeted historical data, providing more effective contextual support for the large language model and improving classification accuracy.

[0065] In one embodiment, the step of recalling the second historical data set according to the initial predicted category includes:

[0066] Obtain the predicted probability of each historical question in the current case database in the initial prediction category;

[0067] In descending order of the predicted probability of each historical question in the initial predicted category, select the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category as the second historical data set.

[0068] Specifically, locate all historical question records in the case database and extract the predicted probability field for the initial predicted category corresponding to each question. For example, read the value for category A from the probabilities column in the structured database. Map all predicted probabilities to the historical question ID. Sort the historical questions from highest to lowest predicted probability in the initial predicted category. Based on the total number of current question category pairs, select the top N historical questions after sorting. Extract the complete predicted probability and true category data for each selected question to form the second historical data set. For example, suppose the initial predicted category of the target question is "account issue." There are 500 historical questions in the database, each containing predicted probabilities for categories A, B, and C. The total number of current question category pairs is 20. Extract the predicted probabilities for the "account issue" category from these 500 historical questions, sort these probabilities from highest to lowest, and select the top 20 questions as the second historical data set.

[0069] In this example, a high prediction probability indicates that the question has a high classification relevance under the initial prediction category, providing an effective reference for further classification of the target question. Filtering historical questions from highest to lowest prediction probability of the initial prediction category ensures that the selected sample data has a high classification relevance to the target question, reducing the impact of noise data.

[0070] In one embodiment, the process of determining the overall accuracy of each target prediction category in the current cycle includes:

[0071] Get the true category of each target question in the current cycle;

[0072] Compare the target predicted category and the true category of each target question, and calculate the overall accuracy based on the comparison results.

[0073] Specifically, the true category of the target question can be obtained by searching for the true category field corresponding to the target question in a pre-labeled case database. For example, the true category can be returned from the database using the target question's unique identifier as a query condition. Alternatively, if the true category is stored in an external system, it can be obtained through an interface, for example, by calling a RESTful API, passing parameters based on the target question, and receiving the true category data. For each target question, its predicted target category is compared with the true category as a string or numeric value to determine whether they are equal. The number of correct entries in the comparison results is accumulated, and the number of correct classifications is divided by the total number of target questions to obtain the overall accuracy rate.

[0074] In this embodiment, comparing the predicted results with the true categories can quantify the performance of the large language model and provide data support for further optimization.

[0075] In one embodiment, the step of incrementing the total number of current question category pairs to obtain the new total number of current question category pairs includes:

[0076] Determine the increment of question categories based on the overall correct rate;

[0077] The sum of the problem category pair increment and the current total number of problem category pairs is taken as the new current total number of problem category pairs.

[0078] Specifically, if the overall accuracy is lower than the threshold, the incremental value can be dynamically generated according to the accuracy gap according to the preset rules. For example, if the overall accuracy is lower than 80%, the increment of the question category may be , K is the increment coefficient. The total number of current question category pairs and the question category pair increment are read, and these two values ​​are added together to obtain a new total number. The new total number of current question category pairs is stored in the memory or database for use in the next cycle.

[0079] In this embodiment, when the overall accuracy falls below a threshold, the granularity of the question categories may be insufficient to effectively distinguish the target questions. Therefore, by incrementally increasing the total number of category pairs, the classification resolution can be improved. Through this dynamic increment mechanism, the large language model can adapt to scenarios with varying numbers and complexities of questions, improving the level of classification refinement. Compared to fixed question category settings, dynamic adjustment can quickly respond to concentrated outbreaks of classification errors and shorten the optimization cycle of the large language model.

[0080] In one embodiment, the method further comprises:

[0081] If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the offline training of the large language model is triggered. When the trained large language model meets the preset training end requirements, the trained large language model will be used as the current large language model.

[0082] Specifically, the overall accuracy is obtained from the classification task results of the current cycle and compared with a preset threshold. If it is below the threshold, offline training begins. If it is above or equal to the threshold, offline training is not triggered. The offline training process can be selected based on actual circumstances. For example, it can extract labeled training data from a case database or online problem log, including information such as input text, predicted category, and true category. Training parameters such as the learning rate, batch size, number of training rounds, and validation set ratio are configured. The training data is then fed into the model. The model weights are updated using a backpropagation algorithm, gradually reducing the training loss. The trained model performance is evaluated on the validation set, and metrics such as accuracy are calculated to determine whether the preset training termination requirements are met, such as reaching the target accuracy or reducing the training loss below a specified value. The trained model is then loaded to replace the old current model. In an example, suppose the current model has an overall accuracy of 78%, which is below the preset threshold of 85%. After offline training is triggered, the model is optimized, achieving a validation set accuracy of 90%, and the training loss is reduced to 0.05, meeting the termination requirements. At this point, the new model is replaced with the current model.

[0083] In this embodiment, if the current large language model performs poorly, offline training can significantly improve its classification performance by optimizing model weights and parameters. Such dynamic triggering of training enables the model to absorb new data features, thereby continuously improving its understanding and classification capabilities of real problems.

[0084] In one embodiment, the offline training process of a large language model includes:

[0085] Determine the true category of each target question in the current cycle, and use each target question in the current cycle and its latest predicted probability and true category in each category, as well as the current case database, to perform offline training on the current large language model.

[0086] Specifically, the true category information is extracted from the context of the target question, user annotations, or other high-credibility sources, the target question is associated with its true category, and the record of the target question is updated. The input text and its latest predicted probability and true category are extracted from the target question, and historical questions and their category distribution are extracted from the case database to form a training set. The parameters of the current large language model are loaded as the starting point for training, and training parameters such as learning rate, batch size, and number of training rounds are set. Next, the training process of the large language model can be selected according to actual conditions. For example, the target question text can be input into the model, the category prediction probability is calculated, the cross entropy loss is calculated based on the predicted probability and the true category, the model weights are optimized using the backpropagation algorithm to make the prediction closer to the true category, the model performance indicators are calculated on the validation set, and the parameters are adjusted or training is continued until the performance meets the requirements.

[0087] In this embodiment, the target question reflects the real-time performance of the large language model in the current cycle, while the case database supplements more reference context. The combination of the two can timely identify and correct classification problems of the large language model, adapt to changes in classification requirements, maintain long-term classification performance stability, and avoid the chain reaction of incorrect large language models on predictions for the next cycle.

[0088] To facilitate understanding of the solutions of this application, specific examples are provided below for illustration.

[0089] like Figure 2 As shown in the figure, after a user submits a question description, it is sent directly to the large language model, which returns a preliminary prediction. The model then searches the case vector database and the badcase vector database to retrieve historical records similar to the user's question description. The model also retrieves the top few relevant records based on the weight of each category in the historical data. The retrieved data is inserted into the context, and the user's original instruction is rewritten. The rewritten instruction is then sent to the large model to generate an improved prediction result, which is then returned to the user.

[0090] The final prediction output is compared with the manually labeled data to calculate the accuracy of the model's predictions, also known as the confidence level. The confidence statistics module calculates the overall accuracy of the large language model output within a preset time period. When the confidence level falls below a threshold, the number of data items recalled from the database is increased to enrich the contextual input, thereby improving the accuracy of the online large language model. At the same time, the large model fine-tuning task is initiated, utilizing the full data set, including historical data and newly generated bad case data sets, for further training and optimization of model performance. By continuously and iteratively increasing the number of data items recalled and fine-tuning the large model, the accuracy of the model output is ensured to meet actual application requirements while providing users with high-quality prediction results.

[0091] The following describes the problem classification optimization device for a large language model provided by an embodiment of the present application. The problem classification optimization device for a large language model described below and the problem classification optimization method for a large language model described above can be referenced to each other. Figure 3 As shown, the present application provides a question classification optimization device for a large language model, the device comprising:

[0092] Initial prediction category acquisition module 201 is used to input each target question into the current large language model in the current cycle to obtain the initial prediction category of each target question;

[0093] A historical data set acquisition module 202 is configured to, for each target question and its initial predicted category, recall a first historical data set based on the target question and a second historical data set based on the initial predicted category from the current case database according to the total number of current question category pairs, wherein the first historical data set and the second historical data set both include historical questions and their predicted probabilities and true categories in each category;

[0094] A target prediction analogy acquisition module 203 is configured to input each target question and its corresponding first historical data set and second historical data set into the current large language model to obtain a target prediction category for the target question;

[0095] The question category pair total updating module 204 is used to increment the current question category pair total as the new current question category pair total if it is determined that the overall accuracy of each target prediction category in the current cycle is less than a preset threshold, and enter the next cycle.

[0096] In one embodiment, the historical data set acquisition module 202 includes:

[0097] A text similarity calculation method is used to calculate the text similarity between the target question and each historical question in the current case database;

[0098] The first historical data set acquisition unit is used to select the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category as the first historical data set in descending order of text similarity of each historical question.

[0099] In one embodiment, the historical data set acquisition module 202 includes:

[0100] A prediction probability acquisition unit, used to obtain the prediction probability of each historical question in the current case database in the initial prediction category;

[0101] The second historical data set acquisition unit is used to select the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category as the second historical data set in descending order of the predicted probability of each historical question in the initial predicted category.

[0102] In one embodiment, the question category pair total updating module 204 includes:

[0103] The true category acquisition unit is used to obtain the true category of each target question in the current cycle;

[0104] The overall accuracy calculation unit is used to compare the target prediction category and the true category of each target question, and calculate the overall accuracy based on the comparison results.

[0105] In one embodiment, the question category pair total updating module 204 includes:

[0106] A question category pair increment determination unit, configured to determine the question category pair increment based on the overall accuracy rate;

[0107] The question category pair total updating unit is used to take the sum of the question category pair increment and the current question category pair total as the new current question category pair total.

[0108] In one embodiment, the apparatus further comprises:

[0109] The large language model offline training module is used to trigger the offline training of the large language model if it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold. When the trained large language model meets the preset training end requirements, the trained large language model will be used as the current large language model.

[0110] In one embodiment, the large language model offline training module includes:

[0111] The large language model offline training unit is used to determine the true category of each target question in the current cycle, and use each target question in the current cycle and its latest predicted probability and true category in each category, as well as the current case database, to perform offline training on the current large language model.

[0112] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the question classification optimization method for a large language model as described in any of the above embodiments.

[0113] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the question classification optimization method for a large language model as described in any of the above embodiments.

[0114] Schematically, as Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 can be provided as a server. Figure 4Computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by memory 301 for storing instructions executable by processing component 302, such as an application. The application stored in memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, processing component 302 is configured to execute the instructions to perform the large language model question classification optimization method of any of the above embodiments.

[0115] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or the like.

[0116] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] Finally, it should be noted that, in this article, relational terms such as first and second are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further restriction, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element. Herein, "one," "said," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. A plurality refers to at least two, such as 2, 3, 5, or 8. "And / or" includes any and all combinations of the relevant listed items.

[0118] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0119] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A question classification optimization method for a large language model, characterized in that: The method comprises: In the current cycle, each target question is input into the current large language model to obtain an initial predicted category for each target question; For each target question and its initial predicted category, according to the total number of current question category pairs, recall a first historical data set based on the target question in the current case database, and recall a second historical data set based on the initial predicted category, wherein the first historical data set and the second historical data set both include historical questions and their predicted probabilities and true categories in each category; For each target question, input the target question and its corresponding first historical data set and second historical data set into the current large language model to obtain a target prediction category for the target question; If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the total number of current question category pairs is incremented as the new total number of current question category pairs, and the next cycle is entered.

2. The question classification optimization method for a large language model according to claim 1, characterized in that The step of recalling a first historical data set according to the target problem in the current case database includes: Calculate the text similarity between the target question and each historical question in the current case database; In descending order of text similarity of each historical question, the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category are selected as the first historical data set.

3. The question classification optimization method for a large language model according to claim 1, characterized in that The step of recalling the second historical data set according to the initial predicted category includes: Obtain the predicted probability of each historical question in the current case database in the initial prediction category; In descending order of the predicted probability of each historical question in the initial predicted category, select the same number of historical questions as the total number of current question category pairs and their predicted probabilities and true categories in each category as the second historical data set.

4. The method for optimizing question classification using a large language model according to claim 1, wherein: The process of determining the overall accuracy of each target prediction category in the current cycle includes: Obtain the true category of each target problem in the current cycle; The target predicted category and the true category of each target question are compared, and the overall accuracy is calculated based on the comparison result.

5. The method for optimizing question classification using a large language model according to claim 1, wherein: The step of incrementing the total number of current question category pairs as a new total number of current question category pairs includes: determining an increment of question categories according to the overall correct rate; The sum of the question category pair increment and the current total number of question category pairs is taken as the new current total number of question category pairs.

6. The method for optimizing question classification using a large language model according to any one of claims 1 to 5, characterized in that: The method further comprises: If it is determined that the overall accuracy of each target prediction category in the current cycle is less than the preset threshold, the offline training of the large language model is triggered. When the trained large language model meets the preset training end requirements, the trained large language model will be used as the current large language model.

7. The method for optimizing question classification using a large language model according to claim 6, wherein: The offline training process of the large language model includes: Determine the true category of each of the target questions in the current cycle, and use each of the target questions in the current cycle and its latest predicted probability and true category in each category, as well as the current case database, to perform offline training on the current large language model.

8. A question classification optimization device for a large language model, characterized in that: The device comprises: An initial prediction category acquisition module is used to input each target question into the current large language model in the current cycle to obtain an initial prediction category for each target question; a historical data set acquisition module, configured to, for each target question and its initial predicted category, recall a first historical data set based on the target question in the current case database according to the total number of current question category pairs, and recall a second historical data set based on the initial predicted category, wherein the first historical data set and the second historical data set both include historical questions and their predicted probabilities and true categories in each category; a target prediction analogy acquisition module, configured to input each target question and its corresponding first historical data set and second historical data set into the current large language model to obtain a target prediction category for the target question; The total number of question category pairs updating module is used to increment the total number of current question category pairs as the new total number of current question category pairs if it is determined that the overall accuracy of each target prediction category in the current cycle is less than a preset threshold, and enter the next cycle.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the question classification optimization method for a large language model as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the question classification optimization method for a large language model as described in any one of claims 1 to 7.

Images