Method, device and equipment for adjusting and optimizing intention recognition model based on LLM dialogue system and storage medium

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses an LLM dialogue system-based intention recognition model tuning method, device and equipment and a storage medium, and the method comprises the steps: obtaining first sample labeled data and sample unlabeled data input by a user in an LLM dialogue system, and obtaining an initial intention recognition model; the model is used for performing intention recognition on information input by the user to predict the real intention of the user; based on the initial intention recognition model, the sample unmarked data and the first sample marked data, second sample marked data and error sample data are obtained; generating third sample marked data based on the error sample data and the large language model; and performing optimization processing on the initial intention recognition model based on the second sample marked data and the third sample marked data to obtain a target intention recognition model. The large language model is utilized to expand the sample data, and a large amount of manual annotation of the sample data is not needed, so that the manual annotation cost is reduced, and the recognition accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for tuning an intent recognition model based on an LLM dialogue system. Background Art

[0002] In the field of artificial intelligence technology, intent recognition model is an important task in the field of natural language processing, which aims to identify the user's intention from the user's input. However, when training the intent recognition model, a large amount of labeled data is required, which makes the cost of manually labeling data high. If a small amount of labeled data is used to train the intent recognition model, the recognition effect of the trained intent recognition model will be poor. In addition, in the prior art, if the time span of the data volume for training and testing the intent recognition model is small, the generalization ability of the intent recognition model will be poor; if the data coverage of the training and fine-tuning of the intent recognition model is not comprehensive, the intent recognition model will lack world knowledge; and the intent recognition model in the prior art is similar to the bag-of-words model, which makes the intent recognition effect poor. Therefore, it can be seen that the intent recognition accuracy of the intent recognition model in the prior art is low and it requires huge costs. Summary of the Invention

[0003] The purpose of the present invention is to at least provide a method, device, equipment and storage medium for tuning an intent recognition model based on an LLM dialogue system, which can at least solve the technical problems of low accuracy of intent recognition of the intent recognition model and huge costs, and at least improve the accuracy of intent recognition of the intent recognition model.

[0004] To solve the above technical problems, at least one embodiment of the present application provides an intent recognition model tuning method based on an LLM dialogue system, including: obtaining first sample labeled data and sample unlabeled data input by a user in the LLM dialogue system, and obtaining an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels, and the initial intent recognition model is a model obtained by training based on a preset number of labeled data, which performs intent recognition on the information input by the user to predict the user's true intention; based on the initial intention recognition model, the sample unlabeled data and the first sample labeled data, second sample labeled data and error sample data are obtained; based on the error sample data and a large language model, third sample labeled data are generated; based on the second sample labeled data and the third sample labeled data, the initial intent recognition model is tuned to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

[0005] At least one embodiment of the present application also provides an intent recognition model tuning device based on an LLM dialogue system, including: an acquisition module, used to acquire first sample labeled data and sample unlabeled data input by a user in the LLM dialogue system, and obtain an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels, and the initial intent recognition model is a model obtained by training based on a preset number of labeled data, which performs intent recognition on the information input by the user to predict the user's true intention; an acquisition module, used to obtain second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data and the first sample labeled data; a generation module, used to generate third sample labeled data based on the error sample data and a large language model; a tuning module, used to tune the initial intent recognition model based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

[0006] At least one embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned intent recognition model tuning method based on the LLM dialogue system.

[0007] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for tuning the intent recognition model based on the LLM dialogue system.

[0008] An embodiment of the present application provides a method for obtaining first sample labeled data and sample unlabeled data input by a user in an LLM dialogue system, and obtaining an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels, and the initial intent recognition model is a model obtained by training based on a preset number of labeled data, which performs intent recognition on the information input by the user to predict the user's true intention; second sample labeled data and error sample data are obtained based on the initial intent recognition model, the sample unlabeled data and the first sample labeled data; third sample labeled data is generated based on the error sample data and a large language model; the initial intent recognition model is tuned based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to a solution of an intent recognition model that meets preset requirements and can be put into use. The initial intent recognition model, trained with a small amount of data, can recognize both labeled and unlabeled data. A large language model is used to augment incorrectly recognized labeled data to produce a third sample of labeled data. The correctly recognized unlabeled data is then labeled to produce a second sample of labeled data. The initial intent recognition model is then iteratively fine-tuned using the second and third labeled samples to obtain the target intent recognition model. This reduces the cost of manual labeling. The large language model is also used to expand the domain and quantity of sample data used to fine-tune the initial intent recognition model. This results in a target intent recognition model, fine-tuned using multiple domains and a large amount of sample data, possessing stronger generalization capabilities, possessing world knowledge, and achieving higher recognition accuracy.

[0009] In some optional embodiments, obtaining second sample labeled data and erroneous sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data includes: performing intent recognition on the sample unlabeled data based on the initial intent recognition model to obtain a first label; determining the first label as the true label of the sample unlabeled data based on a preset label formulation rule; using the first label as the label of the sample unlabeled data to obtain second sample labeled data; performing intent recognition on the first sample labeled data based on the initial intent recognition model to obtain a second label; and if the first label loss value between the second label and the label of the first sample labeled data is greater than a preset label loss threshold, treating the first sample labeled data and the second label as erroneous sample data. The initial intent recognition model is optimized using the second sample labeled data to improve the accuracy of the initial intent recognition model. The erroneous sample data that are incorrectly predicted in the sample labeled data predicted by the initial intent recognition model are input into a large language model to obtain more sample data, i.e., to obtain third sample labeled data. The initial intent recognition model is optimized using the third sample labeled data, so that the target intent recognition model finally obtained has world knowledge, strong generalization ability, and higher recognition accuracy.

[0010] In some optional embodiments, the determining of the first label as the true label of the sample unlabeled data based on a preset label formulation rule includes: determining a third label for the sample unlabeled data based on a preset label formulation rule; if the loss value between the third label and the second label of the first label is less than the preset label loss threshold, determining that the first label is the true label of the sample unlabeled data.

[0011] In some optional embodiments, before generating the third sample labeled data based on the erroneous sample data and the large language model, the method further includes: adjusting the preset label formulation rule based on the erroneous sample data to obtain an adjusted preset label formulation rule; and updating the preset label formulation rule using the adjusted preset label formulation rule. Proofreading the preset label formulation rule using the erroneous sample data allows for dynamic modification of erroneous portions of the preset label formulation rule, thereby improving the accuracy of the formulated labels, facilitating better labeling of samples, and improving the accuracy of the tuned target intent recognition model.

[0012] In some optional embodiments, the generation of third sample labeled data based on the erroneous sample data and the large language model includes: processing the erroneous sample data using the large language model to obtain multiple sample data that are not completely similar to the erroneous sample data; labeling each sample data in the multiple sample data to obtain predicted sample labeled data; and using the predicted sample labeled data and the erroneous sample data as the third sample labeled data. Processing the erroneous sample data using the large language model can obtain a larger number of more diverse predicted sample labeled data. Using the erroneous sample data and the predicted sample labeled data expanded by the large language model as the third sample labeled data expands the data set for tuning the initial intent recognition model and helps the initial intent recognition model correct its own errors based on the erroneous sample data. This helps improve the accuracy of the target intent recognition model.

[0013] In some optional embodiments, the method further includes: obtaining the text to be recognized; if the target intent recognition model fails to obtain the intent label of the text to be recognized, then obtaining a keyword thesaurus; using the keyword thesaurus to perform keyword matching with the text to be recognized, and if the text to be recognized contains the target keyword in the keyword thesaurus, then using the target intent label corresponding to the target keyword as the intent recognition result of the text to be recognized; if the text to be recognized does not contain any keyword in the keyword thesaurus, then inputting the text to be recognized into the large language model to obtain the intent recognition result corresponding to the text to be recognized. In the case where the target intent recognition model cannot perform intent recognition, intent recognition can be performed using a large language model and keyword matching, which provides a fallback mechanism to deal with extreme situations and improve the user experience.

[0014] In some optional embodiments, the number of the second sample labeled data is the same as the number of the third sample labeled data. The number of sample labeled data from different sources is balanced to avoid the problem of overfitting or catastrophic forgetting in the target intent recognition model, to avoid the reduction of the generalization ability of the target intent recognition model, and to improve the generalization ability and recognition accuracy of the target intent recognition model. Among them, the sample labeled data from different sources refers to the second sample labeled data that is processed after recognition based on the initial intent recognition model. The third sample labeled data includes the data obtained after recognition by the initial intent recognition model and the data expanded after processing by the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0016] Figure 1This is a flowchart of a method for optimizing an intent recognition model based on an LLM dialogue system, provided by an embodiment of the present application;

[0017] Figure 2 This is a structural diagram of a tuning process of a target intent recognition model provided by an embodiment of the present application;

[0018] Figure 3 is a structural diagram of an intent recognition process provided by another embodiment of the present application;

[0019] Figure 4 This is a structural diagram of an LLM dialogue system-based intent recognition model tuning device provided by another embodiment of the present application;

[0020] Figure 5 It is a structural diagram of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0022] It should be noted that the acquisition or use of data in the embodiments of this application requires the user's consent. The relevant data can only be obtained after the user's authorization and permission, and the acquisition or use of the data complies with the provisions of relevant laws and regulations.

[0023] In the field of artificial intelligence technology, intent recognition models are an important task in natural language processing, aiming to identify user intent from user input. However, training intent recognition models requires a large amount of labeled data, which makes the cost of manually labeling data high. If a small amount of labeled data is used to train the intent recognition model, the recognition effect of the trained intent recognition model will be poor. In the prior art, if the training set and test set for training the intent recognition model are data from the same period, the performance of the trained intent recognition model on unknown data will gradually decline over time, resulting in poor generalization ability of the intent recognition model. If the data coverage of the training and fine-tuning intent recognition model is not comprehensive, the intent recognition model will lack world knowledge. In addition, the intent recognition model in the prior art is similar to the bag-of-words model, resulting in poor intent recognition. That is, the intent recognition accuracy of the intent recognition model in the prior art is low and the cost is high.

[0024] In addition, when the existing technology only uses the intent recognition model for intent recognition, it lacks a backup module and cannot deal with some immediate problems or emergencies that occur in the intent recognition model.

[0025] In order to solve the technical problems of low recognition accuracy and high cost of the above-mentioned intent recognition model, the present invention proposes an intent recognition model tuning method based on the LLM dialogue system. The implementation details of the intent recognition model tuning method based on the LLM dialogue system of this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for the implementation of this solution.

[0026] Example 1:

[0027] The intent recognition model tuning method based on the LLM dialogue system of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 Shown, including:

[0028] Step 101: obtain the first sample labeled data and sample unlabeled data input by the user in the LLM dialogue system, and obtain an initial intention recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels. The initial intention recognition model is a model trained based on a preset number of labeled data, which performs intent recognition on the information input by the user to predict the user's true intention.

[0029] Specifically, a Large Language Model (LLM) is a deep learning model tuned using large amounts of text data. It can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and conversation.

[0030] Specifically, the LLM dialogue system is used to obtain user input and display responses corresponding to the user input. The responses corresponding to the user input can be obtained based on a server or a model installed on the server or a corresponding processing module. The model installed on the server or the corresponding processing module can be used to process the corresponding natural language task. For example, a question-and-answer task can be processed based on a question-and-answer module or question-and-answer model installed on the server, providing a corresponding answer to the user input. A text classification task can be processed based on a text classification module or text classification model installed on the server, providing a corresponding text classification result for the user input. A dialogue task can be processed based on a dialogue module or dialogue model installed on the server, providing a corresponding reply to the user input.

[0031] Specifically, the first sample labeled data includes a label and the first sample data. The label refers to the intent information corresponding to the first sample data, and the intent information refers to the user's actual intention when entering the first sample data, that is, the purpose of the user entering the first sample data. The number of the first sample labeled data can be one or more.

[0032] Specifically, the initial intent recognition model can be a BERT model, wherein the BERT model (Bidirectional Encoder Representations from Transformers, pre-trained language representation model) is a model based on the Transformer architecture, which understands text through bidirectional context and is widely used in natural language processing tasks.

[0033] Exemplarily, the initial intent recognition model can be used in an LLM-based dialogue system to identify user input information to predict user intent and, based on the predicted user intent, forward the user input information to appropriate downstream task modules, thereby enabling the invocation of different plug-ins or the adoption of different answer strategies to process the user input information. For example, if the initial intent recognition model determines based on the user input information that the user intends to perform text classification on the input information, then the user input information can be forwarded to a text classification module based on the user intent predicted by the initial intent recognition model, where the text classification is performed on the user input information. For another example, if the initial intent recognition model determines based on the user input information that the user intends to obtain an answer to the input information, then, based on the user intent predicted by the initial intent recognition model, the user input information can be forwarded to a question-and-answer module, where the question-and-answer module processes the user input information and provides an answer to the user input information. For another example, if the initial intent recognition model determines based on the user input information that the user intends to obtain a response to the input information in order to engage in a conversation, then, based on the user intent predicted by the initial intent recognition model, the user input information can be forwarded to a dialogue module, where a response to the user input information is provided.

[0034] Specifically, the number of sample unlabeled data may be one or more.

[0035] Specifically, the preset number refers to the amount of sample data required to train the initial intent recognition model. The specific number can be adjusted based on actual needs. In this solution, to save the cost of manually annotating sample data, the preset number is much smaller than the sample data required to fine-tune the target intent recognition model. The preset number is also much smaller than the sample data required to train a ready-to-use intent recognition model in the prior art. For example, the preset number in this solution can be 1.5B parameters.

[0036] For some examples, see Figure 2 The method further includes: obtaining a preset number of initial sample labeled data, wherein the initial sample labeled data is manually labeled data. Using the preset number of initial sample labeled data, an original intent recognition model is trained to obtain an initial intent recognition model, wherein the original intent recognition model is a model trained using sample data that is not labeled with intent information.

[0037] Step 102: Obtain second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data.

[0038] Specifically, see Figure 2The second sample labeled data is labeled data consisting of the intent labels with high confidence among the intent labels corresponding to the sample unlabeled data predicted by the initial intent recognition model and the corresponding sample unlabeled data. The number of the second sample labeled data can be one or more.

[0039] Specifically, see Figure 2 Error sample data refers to labeled data where the intent label corresponding to the first labeled sample data predicted by the initial intent recognition model differs in essence from the label corresponding to the first labeled sample data. In other words, error sample data consists of labeled data corresponding to an incorrect prediction result when the initial intent recognition model predicts the first labeled sample data.

[0040] In some examples, in the aforementioned step 102, obtaining the second sample labeled data and the error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data includes the following steps 1021-1025:

[0041] Step 1021: perform intent recognition on the sample unlabeled data based on the initial intent recognition model to obtain a first label;

[0042] Specifically, the first label refers to the prediction label corresponding to the sample unlabeled data obtained by the initial intent recognition model performing intent prediction on the sample unlabeled data.

[0043] Step 1022: Determine, based on a preset label formulation rule, that the first label is the true label of the sample unlabeled data;

[0044] Specifically, the preset label formulation rules refer to the rules for labeling unlabeled data.

[0045] Specifically, the true label refers to the user's true intention information corresponding to the sample unlabeled data.

[0046] In some examples, in the aforementioned step 1022, the determining of the first label as the true label of the sample unlabeled data based on the preset label formulation rule includes: determining a third label of the sample unlabeled data based on the preset label formulation rule; if the second label loss value of the third label and the first label is less than the preset label loss threshold, then determining the first label as the true label of the sample unlabeled data.

[0047] Specifically, if the second label loss value is less than the preset label loss threshold, the first label is considered to be the true label of the sample unlabeled data, and the confidence of the first label is high.

[0048] Step 1023: Use the first label as the label of the sample unlabeled data to obtain the second sample labeled data;

[0049] Step 1024: perform intent recognition on the first sample labeled data based on the initial intent recognition model to obtain a second label;

[0050] Specifically, the second label refers to the intention prediction performed by the initial intent recognition model on the first sample labeled data, and the predicted label corresponding to the first sample labeled data is obtained.

[0051] Step 1025: If the first label loss value between the second label and the label of the first sample labeled data is greater than a preset label loss threshold, the first sample labeled data and the second label are regarded as erroneous sample data.

[0052] Specifically, if the first label loss value is greater than the preset label loss threshold, it is considered that the second label is not the true label of the first sample labeled data, that is, the initial intent recognition model makes an incorrect prediction of the intent of the first sample labeled data.

[0053] Step 103: Generate third sample labeled data based on the erroneous sample data and the large language model;

[0054] Specifically, the large language model is the LLM model (Large Language Model, a natural language processing model for deep learning). The LLM model is trained using much more data than the BERT model. It can understand and generate natural language text, has strong language comprehension capabilities, extensive world knowledge, and strong generalization on positional data. It can complete a series of complex natural language processing tasks such as text classification, sentiment analysis, and entity recognition in zero-shot and few-shot situations. Among them, zero-shot learning is a machine learning method that allows the model to directly predict or classify a specific task or category without having seen it before. Few-shot learning is a machine learning method that allows the model to effectively predict or classify new tasks by training or fine-tuning on a small amount of labeled data.

[0055] Specifically, the third sample labeled data includes error sample data and sample data similar to the error sample data generated by the large language model. For details on how the large language model generates more sample data similar to the error sample data based on the error sample data, please refer to the prior art and will not be elaborated here.

[0056] In some examples, in the aforementioned step 103, before generating the third sample labeled data based on the erroneous sample data and the large language model, the method further includes: adjusting the preset label formulation rule based on the erroneous sample data to obtain an adjusted preset label formulation rule; and updating the preset label formulation rule using the adjusted preset label formulation rule.

[0057] Specifically, when erroneous sample data is obtained, since there is a large gap between the label predicted by the initial intent recognition model and the label annotated with the sample data, it is considered that the initial intent recognition model prediction is wrong. However, it may actually be because there is an error in the preset label formulation rules for annotating the sample data. Therefore, the preset label formulation rules are proofread based on the erroneous sample data, and the erroneous rules in the preset label formulation rules are adjusted. The preset label formulation rules are updated with the adjusted rules to improve the accuracy of labeling the sample data.

[0058] In some examples, in the aforementioned step 103, generating third sample labeled data based on the erroneous sample data and the large language model includes: processing the erroneous sample data using the large language model to obtain multiple sample data that are not completely similar to the erroneous sample data; labeling each sample data in the multiple sample data to obtain predicted sample labeled data; and using the predicted sample labeled data and the erroneous sample data as third sample labeled data.

[0059] Specifically, if Figure 2 As shown, the error sample data is input into the large language model to obtain the predicted sample labeled data; the predicted sample labeled data and the error sample data are used as the third sample labeled data.

[0060] In summary, we can use the error sample book as seed data and generate multiple similar sample data using a large language model. Finally, the multiple sample data generated by the large language model, the error sample data, and the labeled data for the second sample are mixed together. An iterative tuning method is used to tune the initial intent recognition model, improving the quality of the constructed sample data and continuously improving the recognition accuracy of the initial intent recognition model.

[0061] Step 104: tune the initial intent recognition model based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

[0062] Specifically, if Figure 2 As shown, the second sample labeled data and the third sample labeled data are input into the initial intent recognition model, and the initial intent recognition model is optimized in an iterative manner to obtain the target intent recognition model.

[0063] Specifically, in this solution, the initial intent recognition model is used as the base model, and the second sample labeled data and the third sample labeled data are used to tune the base model to obtain the target intent recognition model.

[0064] In some cases, to prevent overfitting of large language models, using a large amount of overfitted data to tune the initial intent recognition model can reduce the generalization ability of the resulting target intent recognition model. Furthermore, to prevent catastrophic forgetting in the target intent recognition model, this solution sets the number of the second sample labeled data to be the same as the number of the third sample labeled data. By balancing the number of the second and third sample labeled data, the generalization ability of the target intent recognition model on unseen data is improved, thus preventing catastrophic forgetting in the target intent recognition model.

[0065] In some cases, in order to deal with emergencies in the target intent recognition model, intent recognition cannot be performed, or in order to deal with real-time data, intent recognition can be performed quickly on real-time data, such as Figure 3 As shown, the method also includes: obtaining the text to be recognized; if the target intent recognition model fails to obtain the intent label of the text to be recognized, then obtaining a keyword thesaurus; using the keyword thesaurus to perform keyword matching with the text to be recognized, if the text to be recognized contains the target keyword in the keyword thesaurus, then using the target intent label corresponding to the target keyword as the intent recognition result of the text to be recognized; if the text to be recognized does not contain any keyword in the keyword thesaurus, then inputting the text to be recognized into the large language model to obtain the intent recognition result corresponding to the text to be recognized.

[0066] Specifically, if the target intent recognition model predicts the text to be recognized, and the prediction time exceeds the preset time threshold, or the target intent recognition model fails to predict the text to be recognized, it is deemed that the target intent recognition model fails to obtain the intent label of the text to be recognized.

[0067] Specifically, the intent label refers to the target intent recognition model predicting the true intention of the user who inputs the text to be recognized.

[0068] Specifically, the keyword database stores multiple target keywords and multiple target intent labels. There is a one-to-one correspondence between the target keywords and the target intent labels. Alternatively, it can be a one-to-many relationship, meaning that the target intent labels corresponding to multiple target keywords can be the same or different.

[0069] In summary, the present application proposes obtaining first sample labeled data and sample unlabeled data input by a user in an LLM dialogue system, and obtaining an initial intention recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels, and the initial intention recognition model is a model obtained by training based on a preset number of labeled data, which performs intent recognition on the information input by the user to predict the user's true intention; second sample labeled data and error sample data are obtained based on the initial intention recognition model, the sample unlabeled data and the first sample labeled data; third sample labeled data is generated based on the error sample data and a large language model; the initial intention recognition model is tuned based on the second sample labeled data and the third sample labeled data to obtain a target intention recognition model, wherein the target intention recognition model refers to a solution of an intention recognition model that meets preset requirements and can be put into use. The initial intent recognition model, trained with a small amount of data, can recognize both labeled and unlabeled data. A large language model is used to augment incorrectly recognized labeled data to produce a third sample of labeled data. The correctly recognized unlabeled data is then labeled to produce a second sample of labeled data. The initial intent recognition model is then iteratively fine-tuned using the second and third labeled samples to obtain the target intent recognition model. This reduces the cost of manual labeling. The large language model is also used to expand the domain and quantity of sample data used to fine-tune the initial intent recognition model. This results in a target intent recognition model, fine-tuned using multiple domains and a large amount of sample data, possessing stronger generalization capabilities, possessing world knowledge, and achieving higher recognition accuracy.

[0070] Specifically, a large language model can be used to obtain third-sample labeled data covering multiple domains. Simultaneously, the initial intent recognition model is fine-tuned using second-sample labeled data and third-sample labeled data, which are different from those used during the training of the initial intent recognition model. This not only imbues the target intent recognition model with world knowledge but also improves its generalization capabilities. The target intent recognition model, fine-tuned using a large amount of second-sample labeled data across multiple domains, achieves high recognition accuracy and low manual labeling costs, avoiding extensive manual labeling and improving the efficiency of tuning the target intent recognition model.

[0071] In addition, the present application also provides a fallback mechanism, which can be used to identify intent based on a keyword thesaurus and a large language model when the target intent recognition model is unable to perform intent recognition. This improves the ability to deal with emergencies and special data, and improves the user experience. If the target intent recognition model is an upstream module of multiple downstream processing modules, and different texts to be recognized need to be sent to corresponding downstream processing modules based on prediction results, the processing efficiency of the text to be recognized can be improved, and the problem of the text to be recognized being stagnant in the target intent recognition model for a long time, resulting in the downstream processing module being unable to obtain data from the upstream module, and the business process being difficult to continue, can be avoided. For example, if the target intent recognition model is used as the first link in the dialogue system to identify the intent of the text to be recognized input by the user, the text to be recognized is sent to the corresponding downstream module based on the identified intent recognition result, thereby calling different plug-ins or using different answer strategies to answer the text to be recognized to obtain an answer result.

[0072] Example 2:

[0073] Another embodiment of the present application relates to an intent recognition model tuning device based on an LLM dialogue system. The implementation details of the intent recognition model tuning device based on an LLM dialogue system of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution. The schematic diagram of the intent recognition model tuning device based on an LLM dialogue system of this embodiment can be as follows: Figure 4 As shown, it includes an acquisition module 401 , an obtaining module 402 , a generation module 403 and an optimization module 405 .

[0074] Acquisition module 401 is configured to acquire first sample labeled data and sample unlabeled data input by a user in the LLM dialogue system, and to acquire an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data without labels. The initial intent recognition model is a model trained based on a preset amount of labeled data, and performs intent recognition on user input information to predict the user's true intent.

[0075] An obtaining module 402 is configured to obtain second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data;

[0076] A generating module 403 is configured to generate third sample labeled data based on the error sample data and the large language model;

[0077] The tuning module 404 is used to tune the initial intent recognition model based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

[0078] In some embodiments, when the device is used to obtain the second sample labeled data and the erroneous sample data based on the initial intention recognition model, the sample unlabeled data and the first sample labeled data, it is specifically used to: perform intent recognition on the sample unlabeled data based on the initial intention recognition model to obtain a first label; determine the first label as the true label of the sample unlabeled data based on a preset label formulation rule; use the first label as the label of the sample unlabeled data to obtain the second sample labeled data; perform intent recognition on the first sample labeled data based on the initial intention recognition model to obtain a second label; if the first label loss value of the second label and the label of the first sample labeled data is greater than a preset label loss threshold, then treat the first sample labeled data and the second label as erroneous sample data.

[0079] In some embodiments, when the device is used to determine that the first label is the true label of the sample unlabeled data based on the preset label formulation rule, it is specifically used to: determine the third label of the sample unlabeled data based on the preset label formulation rule; if the loss value of the third label and the second label of the first label is less than the preset label loss threshold, then determine that the first label is the true label of the sample unlabeled data.

[0080] In some embodiments, before generating third sample labeled data based on the erroneous sample data and the large language model, the device is also used to: adjust the preset label formulation rules based on the erroneous sample data to obtain adjusted preset label formulation rules; and update the preset label formulation rules using the adjusted preset label formulation rules.

[0081] In some embodiments, when the device is used to generate third sample labeled data based on the erroneous sample data and the large language model, it is specifically used to: use the large language model to process the erroneous sample data to obtain multiple sample data that are not completely similar to the erroneous sample data; label each sample data in the multiple sample data to obtain predicted sample labeled data; and use the predicted sample labeled data and the erroneous sample data as the third sample labeled data.

[0082] In some embodiments, the device is also used to: obtain the text to be recognized; if the target intent recognition model fails to obtain the intent label of the text to be recognized, then obtain a keyword thesaurus; use the keyword thesaurus to perform keyword matching with the text to be recognized, and if the text to be recognized contains the target keyword in the keyword thesaurus, then use the target intent label corresponding to the target keyword as the intent recognition result of the text to be recognized; if the text to be recognized does not contain any keyword in the keyword thesaurus, then input the text to be recognized into the large language model to obtain the intent recognition result corresponding to the text to be recognized.

[0083] In some embodiments, the number of the second sample labeled data is the same as the number of the third sample labeled data.

[0084] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0085] Example 3:

[0086] Another embodiment of the present application relates to an electronic device, such as Figure 5 As shown, it includes: at least one processor 901; and a memory 902 communicatively connected to the at least one processor 901; wherein the memory 902 stores instructions that can be executed by the at least one processor 901, and the instructions are executed by the at least one processor 901 to enable the at least one processor 901 to execute the intent recognition model tuning method based on the LLM dialogue system in the above-mentioned embodiments.

[0087] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0088] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0089] Example 4:

[0090] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0091] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0092] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for optimizing an intent recognition model based on an LLM dialogue system, characterized in that: include: Obtaining first sample labeled data and sample unlabeled data input by a user in the LLM dialogue system, and obtaining an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels. The initial intent recognition model is a model trained based on a preset amount of labeled data, and performs intent recognition on the user input information to predict the user's true intent; Obtaining second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data; Generate third sample labeled data based on the error sample data and the large language model; The initial intent recognition model is tuned based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

2. The method for optimizing the intention recognition model based on the LLM dialogue system according to claim 1, characterized in that: The obtaining of second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data includes: Performing intent recognition on the sample unlabeled data based on the initial intent recognition model to obtain a first label; Determining, based on a preset label formulation rule, that the first label is a true label of the sample unlabeled data; Using the first label as the label of the sample unlabeled data to obtain the second sample labeled data; Performing intent recognition on the first sample labeled data based on the initial intent recognition model to obtain a second label; If the first label loss value between the second label and the label of the first sample labeled data is greater than a preset label loss threshold, the first sample labeled data and the second label are regarded as erroneous sample data.

3. The method for optimizing the intention recognition model based on the LLM dialogue system according to claim 2, characterized in that: The determining, based on a preset label formulation rule, that the first label is a true label of the sample unlabeled data includes: Determine a third label for the sample unlabeled data based on a preset label formulation rule; If the second label loss value of the third label and the first label is less than the preset label loss threshold, the first label is determined to be the true label of the sample unlabeled data.

4. The method for optimizing the intention recognition model based on the LLM dialogue system according to claim 2, characterized in that: Before generating third sample labeled data based on the erroneous sample data and the large language model, the method further includes: Adjusting the preset label formulation rule based on the erroneous sample data to obtain an adjusted preset label formulation rule; The preset label creation rule is updated using the adjusted preset label creation rule.

5. The method for optimizing the intention recognition model based on the LLM dialogue system according to claim 1, characterized in that: Generating third sample labeled data based on the erroneous sample data and the large language model includes: Processing the erroneous sample data using a large language model to obtain a plurality of sample data that are not completely similar to the erroneous sample data; Labeling each sample data in the plurality of sample data to obtain labeled data of the predicted sample; The predicted sample labeled data and the erroneous sample data are used as third sample labeled data.

6. The method for optimizing the intent recognition model based on the LLM dialogue system according to any one of claims 1 to 5, characterized in that: The method further comprises: Get the text to be recognized; If the target intent recognition model fails to obtain the intent label of the text to be recognized, then obtain a keyword vocabulary; Perform keyword matching on the text to be recognized using the keyword lexicon. If the text to be recognized contains a target keyword from the keyword lexicon, use the target intent tag corresponding to the target keyword as the intent recognition result of the text to be recognized. If the text to be recognized does not contain any keyword in the keyword library, the text to be recognized is input into the large language model to obtain an intent recognition result corresponding to the text to be recognized.

7. The method for optimizing an intent recognition model based on an LLM dialogue system according to any one of claims 1 to 5, characterized in that: The number of the second sample labeled data is the same as the number of the third sample labeled data.

8. An intention recognition model tuning device based on an LLM dialogue system, characterized in that: include: an acquisition module, configured to acquire first sample labeled data and sample unlabeled data input by a user in the LLM dialogue system, and to acquire an initial intent recognition model, wherein the first sample labeled data refers to data containing labels, and the sample unlabeled data refers to data not containing labels; and the initial intent recognition model is a model trained based on a preset amount of labeled data, and performs intent recognition on user input information to predict the user's true intent; an obtaining module, configured to obtain second sample labeled data and error sample data based on the initial intent recognition model, the sample unlabeled data, and the first sample labeled data; A generating module, configured to generate third sample labeled data based on the error sample data and the large language model; A tuning module is used to tune the initial intent recognition model based on the second sample labeled data and the third sample labeled data to obtain a target intent recognition model, wherein the target intent recognition model refers to an intent recognition model that meets preset requirements and can be put into use.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intention recognition model tuning method based on the LLM dialogue system as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for tuning an intent recognition model based on an LLM dialogue system according to any one of claims 1 to 7 is implemented.

Images