Intention labeling and identifying method, device, equipment, storage medium and program product

Abstract

The application relates to the technical field of artificial intelligence, and provides an intention labeling and identifying method, device, equipment, storage medium and program product. The intention labeling method comprises the following steps: obtaining sample conversation data set to be labeled, inputting the sample conversation data set into an intention output model in each round of iterative labeling process, obtaining intention label data output by the intention output model, inputting the sample conversation data set and the intention label data into an intention correction model, and obtaining intention correction data output by the intention correction model; determining an evolution rate of the sample conversation data set according to intention label consistency of the intention correction data obtained in adjacent two times of iterative labeling process, taking the evolution rate as a judgment basis for whether to continue intention correction, realizing automatic labeling of intention labels of the sample conversation data set based on a large model, improving the intention labeling efficiency, and saving manpower and material resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an intent annotation and recognition method, apparatus, device, storage medium and program product. Background Technology

[0002] A dialogue system is a system for automated human-computer interaction, widely used in various business systems and applications. Throughout the process, the system needs to understand the user's intent and generate corresponding responses. Intent recognition is crucial in the entire dialogue system, directly determining its performance; high-quality intent recognition capability is an important optimization direction for dialogue systems. Intent recognition is primarily achieved through machine learning, and the quality and quantity of data used for model training directly affect the quality of intent recognition.

[0003] Currently, the construction of intent recognition data for specific industries mostly requires manual work. Specifically, firstly, annotators need to label the sample data, then reviewers review the labeled data, and data that fails the review needs to be re-labeled. Finally, acceptance personnel accept the labeled data that has passed the review. Manual labeling is easily affected by the subjective influence of the annotators, and controlling the labeling quality is also a challenge. Continuous supervision and review are needed to ensure consistency, and this process often consumes a lot of manpower, material resources, and time. Summary of the Invention

[0004] This application provides an intent annotation and recognition method, apparatus, device, storage medium, and program product to solve the shortcomings of the prior art in manual annotation, which requires a lot of manpower, material resources, and time costs, and the annotation quality is difficult to guarantee.

[0005] This application provides an intent annotation method, including: Obtain the sample dialogue dataset to be labeled; During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1; The sample dialogue dataset and the intent label data are input into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. If, based on the evolution rate, it is determined that the intent correction data still needs correction, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is determined as the labeled data of the sample dialogue dataset.

[0006] In one embodiment, the intent correction model includes a correction suggestion sub-model and a correction execution sub-model; the step of inputting the sample dialogue dataset and the intent label data into the intent correction model to obtain the intent correction data output by the intent correction model includes: The sample dialogue dataset and the intent label data are input into the correction suggestion sub-model to obtain a correction suggestion dataset for the erroneous intent labels in the intent label data, output by the correction suggestion sub-model; the correction suggestion dataset contains the correction intent labels corresponding to the erroneous intent labels; The sample dialogue dataset, the intent label data, and the correction suggestion dataset are input into the correction execution sub-model to obtain the intent correction data output by the correction execution sub-model.

[0007] In one embodiment, after inputting the sample dialogue dataset and the intent label data into the intent correction model to obtain the intent correction data output by the intent correction model, the method further includes: The sample dialogue dataset, the intent label data, and the intent correction data are input into the intent evaluation model to obtain the score data output by the intent evaluation model; the intent evaluation model is used to score the intent label data and the intent correction data.

[0008] In one embodiment, fine-tuning the intent output model and the intent correction model includes: Based on the intent correction data, the sample dialogue dataset is divided into a first dataset and a second dataset; the first dataset consists of the first sample dialogue data corresponding to the error label, and the second dataset consists of the second sample dialogue data in the sample dialogue dataset other than the first sample dialogue data. Based on the intent label data and the second dataset, a first sample dataset is constructed; Based on the scoring data, a third dataset is selected from the first dataset; the third dataset consists of sample dialogue data where the intention score after intention correction is higher than the intention score before intention correction. Based on the correction suggestion dataset, the intent label data, and the third dataset, a second sample dataset is constructed; Based on the correction suggestion dataset, the intention correction data, and the third dataset, a third sample dataset is constructed; Using the first sample dataset, the second sample dataset, the third sample dataset, and the rating data, the intent output model, the intent correction model, and the intent evaluation model are jointly trained to fine-tune the intent output model and the intent correction model.

[0009] In one embodiment, after training the intent recognition model using the sample dataset, the method further includes: Based on the scoring data and the intent correction data, target dialogue data is filtered from the sample dialogue dataset; the target dialogue data includes high-scoring intent data and low-scoring intent data, the score of the high-scoring intent data is higher than a preset first scoring threshold, and the score of the low-scoring intent data is lower than a preset second scoring threshold, the first scoring threshold is higher than the second scoring threshold; A preference sample dataset is constructed based on the target dialogue data, and the intention output model is trained using the preference sample dataset for preference optimization.

[0010] This application also provides an intent recognition method, including: The user's dialogue data is input into the intent recognition model to obtain the user's intent corresponding to the dialogue data output by the intent recognition model. The intent recognition model is pre-trained based on a sample dataset, which is constructed by annotating the sample dialogue dataset using any of the intent annotation methods described above.

[0011] This application also provides an intent annotation device, including the following modules: The data acquisition module is used to acquire the sample dialogue dataset to be labeled; The intent labeling module is used to input the sample dialogue dataset into the intent output model during the Kth iteration of the labeling process, and obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset, and the intent output model is built based on a large model; K is a positive integer greater than 1; An intent correction module is used to input the sample dialogue dataset and the intent label data into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate calculation module is used to determine the evolution rate of the sample dialogue dataset based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. The annotation iteration module is used to fine-tune the intention output model and the intention correction model when it is determined based on the evolution rate that the intention correction data still needs to be corrected, to determine the intention correction data as the historical label data, and to return to the step of inputting the sample dialogue dataset into the intention output model to obtain the intention label data output by the intention output model; The annotation determination module is used to determine the intention correction data as the annotation data of the sample dialogue dataset when it is determined based on the evolution rate that the intention correction data does not need to be corrected.

[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intent annotation method or the intent recognition method as described above.

[0013] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intent annotation method or the intent recognition method as described above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the intent annotation method or the intent recognition method as described above.

[0015] The intent labeling and recognition method, apparatus, device, storage medium, and program products provided in this application pre-identify intents in sample dialogue datasets through an intent output model and correct erroneous intent labels in the pre-identified datasets through an intent correction model. Based on the consistency between the intent labels of the intent correction data obtained in the later iteration and the intent correction data obtained in the next iteration during two adjacent iterations of labeling, the evolution rate of the sample dialogue dataset is determined as the criterion for whether to continue intent correction. This achieves automatic labeling of intent labels in sample dialogue datasets based on a large model, improving the efficiency of intent labeling and saving human and material costs. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the intent annotation method provided in the embodiments of this application.

[0018] Figure 2 This is a schematic diagram of the intent tag tree structure provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram of the decoding trajectory based on the intent tag tree provided in the embodiments of this application.

[0020] Figure 4 This is a schematic diagram of the intention annotation device provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] This application provides an intent annotation method for annotating user dialogue data during interaction, thereby constructing sample data, training a corresponding intent recognition model, and achieving intent recognition for users in a specific domain. Specifically, the user's dialogue data can originate from a dialogue system. The intent annotation method provided in this application is used to annotate dialogue data in an intelligent customer service scenario, where the dialogue data to be annotated is the dialogue data between the user and the agent.

[0024] Based on this, refer to Figure 1 , Figure 1 This is a flowchart illustrating the intent annotation method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps: Step 100: Obtain the sample dialogue dataset to be labeled; Step 200: During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1. Step 300: Input the sample dialogue dataset and the intent label data into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; Step 400: Based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data, determine the evolution rate of the sample dialogue dataset; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. Step 500: If it is determined based on the evolution rate that the intent correction data still needs to be corrected, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. Step 600: If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is determined as the labeled data of the sample dialogue dataset.

[0025] Obtain the sample dialogue dataset to be labeled. This sample dialogue dataset can be composed of historical dialogue data between users and agents. The sample dialogue dataset to be labeled includes multiple sample dialogue subsets, each of which corresponds to a complete dialogue. It may include sample dialogue data from multiple rounds of dialogue.

[0026] The acquired sample dialogue dataset includes dialogue data from one or more users. For any given user, the acquired sample dialogue dataset includes multiple dialogues between that user and the agent, and each dialogue may include at least one round of sample dialogue data.

[0027] Optionally, by representing the sample dialogue dataset obtained in the form of a dataset, the corresponding dialogue dataset can be obtained as follows: ,in, Indicates the total number of dialogues. Indicates the first The sample dialogue subset of this dialogue can be represented as: (1) In Formula 1, the sample dialogue subset Each tuple in the code represents a response round in a dialogue between the user and the agent. This indicates the user's first interaction with the agent. The total number of dialogue rounds in this conversation. Indicates the first The first dialogue Dialogue data of user questions in each round, Indicates the first The first dialogue The response information generated in each round to user questions.

[0028] Furthermore, the annotation of the sample dialogue dataset is performed iteratively. During the Kth iteration (K is a positive integer greater than 1), the acquired sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model. This intent label data contains the intent labels of the sample dialogue data in each round of the sample dialogue dataset. The intent labels serve as labels for the sample dialogue data and are used to annotate the sample dialogue data.

[0029] The intent output model is built upon a large model. Intent label data is obtained by pre-identifying intent in sample dialogue data from each round of the sample dialogue dataset using the intent output model. In one embodiment, the intent output model performs multi-level intent decoding on the sample dialogue data from each round based on a pre-built intent label tree, obtaining the user's initial intent in each dialogue round and forming intent label data. For a complete dialogue containing multiple rounds, when performing multi-level intent decoding based on the intent label tree, the intent output model can use a context-sensitive approach to decode the intent for each dialogue round, ensuring the accuracy of intent decoding.

[0030] The pre-built intent label tree is constructed using user intents from a specific domain as nodes. It includes multiple levels of intents, and the tree depth represents the number of levels of intent. For example, refer to... Figure 2 The intent tag tree shown includes first-level intents such as information collection, business consultation, business inquiry, business unsubscription, and casual conversation such as greetings and emotional closing remarks. Further, taking business inquiry as an example, second-level intents include basic business inquiry, tariff and package service inquiry, data service inquiry, call charge service inquiry, number service inquiry, payment service inquiry, balance inquiry, package and fixed fee inquiry, out-of-package internet access fee inquiry, broadband inquiry, internet fault inquiry, local number inquiry, local number package inquiry, local number bill details inquiry, and local number data service inquiry.

[0031] Furthermore, the intent output model performs multi-level decoding of user intent based on the intent label tree. Specifically, the intent label tree serves as the decoding trajectory to achieve pre-identification of user intent. Each level of intent decoding can include multiple decoding iterations until the end character is encountered. For example, if the user inputs the question "How much is my monthly package?" during the dialogue, in the first-level intent decoding, the first decoding step involves segmenting the tags within the first-level intent set [business consultation, business inquiry, business cancellation, business processing, greeting, emotional expression], resulting in [[business, consultation], [business, inquiry], [business, cancellation], [greeting, expression], [emotion, expression]]. (See reference...) Figure 3 The intent decoding trajectory shown uses a large language model to decode the user's intent, which determines the user's primary intent as [business, query], i.e., business query.

[0032] Specifically, in the first-level intent decoding process, the first decoding involves word segmentation including user, business, emotion, greeting, and end, from which it can be determined that the business is related to the user's intent. The second decoding involves word segmentation including information under user, unsubscribe, consultation, and query under business, and language under emotion and greeting, etc. Through the second decoding, it can be determined that the query under business is related to the user's intent. Therefore, the user's first-level intent is determined to be a business query.

[0033] Furthermore, under the primary intent of the business query, including business-related word segments such as basic, package, local, tariff, and mobile, the user's secondary intent is determined to be a query for local number package through multiple decoding processes during the secondary intent decoding process. Specifically, firstly, under the primary intent of the business query, the first decoding of words such as basic, package, local, tariff, and mobile determines that the user wants to query local services. The second decoding involves word segments including services under basic, "and" and "outside" under package, number under local, "and" under tariff, and broadband and internet access under mobile, where "and," "and," and "outside" are conjunctions for the next level of word segments. The second decoding determines that it is number services under local services. The third decoding involves word segments including query under services, fixed fees under "and," internet fees under "outside," packages, query, and bills under number, packages under "and," query under broadband, and faults under internet access. Among these, the word segments under local number services include packages, query, and bills, etc. The third decoding determines that the user's intent is a package service for local number. The fourth decoding involves word segmentation including queries under fixed fees, queries under internet fees, queries under packages, details under bills, services under packages, and queries under faults. Through the fourth decoding, it is determined that the user's final intent is "to query the package of this phone number".

[0034] Intent annotation based on Large Language Models (LLMs) can significantly improve annotation efficiency, especially when dealing with large-scale datasets. However, this approach places high demands on the performance of the LLM model, particularly in understanding complex annotation rules. Conventional LLM models may exhibit domain-specific knowledge biases based on the training data, affecting the fairness of the annotation. Furthermore, LLMs may have limitations in reasoning and decision-making processes, which is a drawback for tasks requiring deep understanding and complex judgment.

[0035] As is known, large language models are generative models with a vast decoding space. Often, prompt words are needed to constrain the output of large language models, but this still cannot prevent them from outputting information outside the expected word list, causing interference, especially in classification tasks where model output needs to be constrained within predefined labels. Therefore, in this embodiment, an intent label tree is used to constrain the output of the large language model. The large language model decodes intents according to the expected tree-like trajectory, ensuring that the model output falls within the expected label system with the highest probability. Based on the pre-constructed intent label tree, a structured processing method is used to limit the model's prediction range to predefined intent categories, reducing the possibility of irrelevant or misclassified intents. This not only improves the accuracy of intent recognition but also simplifies subsequent intent correction work, improving overall intent annotation efficiency.

[0036] Furthermore, the sample dialogue dataset and intent label data are input into the intent correction model to obtain intent correction data output by the intent correction model. The intent correction model is then used to correct the intent label data output by the intent output model to obtain intent correction data. This intent correction data includes the corrected intent of the sample dialogue data, which is obtained by correcting erroneous intent labels in the intent labels of the sample dialogue data output by the intent output model.

[0037] The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent-corrected data obtained in the Kth iteration annotation process and the historical label data, where the historical label data is the intent-corrected data obtained in the (K-1)th iteration annotation process. In other words, the evolution rate of the sample dialogue dataset is determined based on the consistency of the corrected intent labels obtained in two adjacent iterations. It should be noted that the process of the (K-1)th iteration annotation is the same as that of the Kth iteration annotation.

[0038] In one embodiment, the sample dialogue dataset is iteratively labeled. In each round of iterative labeling, based on the intent correction data obtained in the Kth iteration labeling process, compared with the intent correction data obtained in the (K-1)th iteration process, the proportion of sample dialogue data in the sample dialogue dataset whose intent labels have not changed is determined, and the evolution rate of the sample dialogue dataset is obtained.

[0039] In other words, intent annotation of the sample dialogue dataset is performed iteratively. In each iteration, the sample dialogue dataset is input into the intent output model to obtain intent label data, and the sample dialogue dataset and intent label data are input into the intent correction model to obtain intent correction data. The evolution rate of the sample dialogue dataset is determined based on the consistency of the intent correction data obtained in two adjacent iterations. Specifically, based on the intent correction data of the current iteration, the sample dialogue data whose intent labels have not changed relative to the previous iteration are determined. The evolution rate of the sample dialogue dataset is obtained based on the proportion of sample dialogue data with unchanged intent labels in the sample dialogue dataset.

[0040] Based on the evolution rate, it is determined whether the intent correction data needs further correction. If it is determined that the intent correction data needs further correction, the intent output model and the intent correction model are fine-tuned, the iteration count is updated, and the intent correction data is designated as historical label data for the next round of iteration annotation. The sample dialogue dataset is then re-inputted into the intent output model to obtain the intent label data output by the intent output model. If it is determined based on the evolution rate that the intent correction data does not need further correction, the intent correction data is designated as the labeled data of the sample dialogue dataset, completing the intent annotation of the sample dialogue dataset. The fine-tuning of the intent output model and the intent correction model can be implemented based on the evolution rate, adjusting the intent output model and the intent correction model according to the evolution rate of the sample dialogue dataset. For example, the optimization objective can be maximizing the evolution rate.

[0041] Optionally, when performing intent annotation on the sample dialogue dataset using an iterative method, the evolution rate of the sample dialogue dataset is used as the criterion for determining the termination of the iteration. Specifically, a preset threshold corresponding to the evolution rate can be used as the termination condition. The evolution rate of the sample dialogue dataset is compared with the preset threshold to determine whether the termination condition is met, i.e., whether the intent correction data needs to be further corrected. In one embodiment, if the evolution rate of the sample dialogue dataset is greater than or equal to the preset threshold, the intent correction data meets the termination condition and does not need to be corrected further. Otherwise, if the evolution rate of the sample dialogue dataset is less than the preset threshold, the intent correction data does not meet the termination condition and needs to be corrected further. If the intent correction data meets the termination condition based on the evolution rate of the sample dialogue dataset, the intent correction data is determined as the labeled data of the sample dialogue dataset, thus completing the intent annotation of the sample dialogue dataset.

[0042] It should be noted that during the initial iteration of annotation on the sample dataset, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model. Then, the sample dialogue dataset and intent label data are input into the intent correction model to obtain the intent correction data output by the intent correction model. The intent output model and intent correction model are then fine-tuned. Based on the fine-tuned intent output model and intent correction model, the iteration round is updated (K value incremented by 1), and the next round of iteration annotation is performed. The intent correction data obtained in the second iteration annotation process is compared with the intent correction data obtained in the first iteration annotation process to determine their consistency, thereby determining the evolution rate of the sample dialogue dataset. Following this method, in each subsequent iteration annotation process, the evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained in the current iteration annotation process and the intent correction data obtained in the previous iteration annotation process.

[0043] Optionally, the intent correction model can also be built upon a larger model. This larger model used to construct both the intent output model and the intent correction model can be the same or different; there are no specific limitations on this. If the intent output model and the intent correction model are to be fine-tuned, this can be done by fine-tuning the larger model that constructed them.

[0044] In this embodiment, the intent output model is used to pre-identify the intent of the sample dialogue dataset, and the intent correction model is used to correct the erroneous intent labels of the pre-identified intent. Based on the consistency of intent labels between the intent correction data obtained in the later iteration and the intent correction data obtained in the previous iteration in two adjacent iterations, the evolution rate of the sample dialogue dataset is determined, and this evolution rate is used as the criterion for whether to continue intent correction. This realizes the automatic labeling of intent labels of the sample dialogue dataset based on a large model, which improves the efficiency of intent labeling and saves manpower and material costs.

[0045] In one embodiment, the intent output model and intent correction model built using a large model can perform preliminary intent recognition and correction on multi-turn dialogue data using prompt words when the model is not fine-tuned in its initial state. For example, for the intent output model, prompt words can be used to guide the intent pre-recognition task of the intent output model, annotate the input sample dialogue dataset with intent, generate intent label data, and output it. For example, the prompt word could be "As a customer service assistant, your task is to identify the user's intent in the current round of dialogue based on the dialogue history between the customer service agent and the user." Using this fourth prompt word as input to the large language model, the complete input to the large language model would be: (2) in, The large language model is in the first Complete input in each dialogue round, These are the input prompt words. It is the user's input in the current round. It is the history of dialogue, which can be represented as: (3) By analyzing the acquired historical dialogue data, the user intent for each round of dialogue can be identified, thereby enabling the construction of a multi-turn dialogue dataset containing intents. As shown in Formula 4 below: (4) in, It is the size of the dialogue data set containing intents, for In The first in the dialogue data set a set , containing total Each round of dialogue, and User intent corresponding to dialogue data from each dialogue turn. Specifically, it is expressed as follows: (5) Optionally, the intent correction model further includes a correction suggestion sub-model and a correction execution sub-model. The correction suggestion sub-model generates correction suggestions for the sample dialogue data with erroneous intent labels, and the correction execution sub-model executes the correction suggestions to correct the erroneous intent labels, thereby obtaining intent correction data. Based on this, step 200 may also include: Step 201: Input the sample dialogue dataset and the intent label data into the correction suggestion sub-model to obtain the correction suggestion dataset for the erroneous intent labels in the intent label data output by the correction suggestion sub-model; the correction suggestion dataset contains the correction intent labels corresponding to the erroneous intent labels; Step 202: Input the sample dialogue dataset, the intent label data, and the correction suggestion dataset into the correction execution sub-model to obtain the intent correction data output by the correction execution sub-model.

[0046] The sample dialogue dataset and intent label data are input into the correction suggestion sub-model to obtain the correction suggestion dataset for the erroneous intent labels in the intent label data. The erroneous intent labels may include one or more, and the correction suggestion dataset contains the correct label corresponding to each erroneous intent label, that is, the corrected intent label.

[0047] The sample dialogue dataset, intent label data, and correction suggestion dataset are input into the correction execution sub-model to obtain intent correction data output by the correction execution sub-model. The correction execution sub-model corrects the corresponding erroneous intent labels according to the correction intent labels suggested in the correction suggestion dataset to obtain intent correction data.

[0048] In one embodiment, the intention correction model's correction suggestions and execution of intention label data can also be implemented through prompt words. The corresponding prompt words are input into the intention correction model to guide the intention correction model to perform the intention correction task, thereby correcting the intention label data output by the intention output model and obtaining intention correction data.

[0049] Specifically, for the intent labels in the pre-identified intent label data, an intent correction model based on a large model is used again to determine whether the pre-identified intent labels are reasonable. For unreasonable intents, correction suggestions are generated, and further intent correction is performed based on the generated correction suggestions. The determination and correction of intent labels output by the intent output model helps improve the accuracy of intent recognition and also helps to better understand the limitations of the model.

[0050] For example, the prompts used to determine intent labels could be something like, "As a customer service assistant, your task is to determine whether a user's intent is reasonable based on the dialogue history between the customer service agent and the user, and the identified user intent. For unreasonable intents, you should provide suggestions for improvement." In the correction suggestion phase, the complete input to the intent correction model can be represented as: (6) The intent correction model determines the intent label and generates correction suggestions for irrational intents, thus obtaining a corresponding correction suggestion dataset. : (7) in: (8) In Formula 8, It is a dataset of correction suggestions The Middle A label containing intent And suggestions for correcting intent labels. A multi-turn dialogue dataset.

[0051] The intent correction suggestions generated during the intent label determination process can also be guided by prompt words to the intent correction model to perform intent correction tasks, thereby correcting the intent labels. This leverages the capabilities of the large model to correct inaccurate intents based on the intent correction suggestions, thus gradually improving the accuracy of intent recognition while ensuring that each correction brings positive results.

[0052] For example, the prompts used to correct incorrect intent labels might be such as, "As a customer service assistant, your task is to correct the initial intent based on the conversation history and initial intent between the customer service agent and the user, as well as intent correction suggestions, and output the corrected intent." In the intent correction phase, the complete input to the intent correction model is: (9) The intent correction model obtains corrected intent data by performing corresponding intent correction tasks on the intent label data. : (10) in: (11) The dataset corresponding to the number of intent corrections includes multiple dialogue datasets. Each dialogue data set It contains the intent label output by the intent output model. Intended Correction Suggestions and corrective intent A collection of multi-turn dialogue data.

[0053] Furthermore, since correcting the intent does not guarantee its complete accuracy and may introduce errors, to quantify the effectiveness of intent correction, the large model acts as a "judge," scoring the corrected intent. This approach not only simplifies the evaluation process but also maintains consistency, as the scoring criteria are the same as the large model's prediction criteria. Similarly, the scoring of the intent is based on the cue words instructing the large model to perform the corresponding scoring task. Therefore, after step 300, the following may also be included: Step 310: Input the sample dialogue dataset, the intent label data, and the intent correction data into the intent evaluation model to obtain the score data output by the intent evaluation model; the intent evaluation model is used to score the intent label data and the intent correction data.

[0054] The sample dialogue dataset, intent label data, and intent correction data are input into the intent evaluation model to obtain the score data output by the intent evaluation model. This intent evaluation model is used to score the intent labels in the intent label data and intent correction data. Optionally, the intent evaluation model can also be built based on a large model. The intent evaluation model, intent correction model, and intent output model can be the same model or different models.

[0055] In one embodiment, the intent evaluation model can also evaluate intent tags based on prompts. Prompts instruct the intent evaluation model, built on a large model, to perform intent evaluation tasks, scoring the intent tags in the intent correction data and intent tag data to obtain score data. Evaluating the intent correction data specifically involves scoring the corrected intent tags within it. Similarly, evaluating the intent tag data involves scoring each intent tag within it to obtain score data. This score data includes the intent scores of the intent tags in the intent correction data before and after intent correction.

[0056] For example, the prompts used in intent assessment might be such as "As a customer service assistant, your task is to score the initial intent and corrected intent of the current user input based on the conversation history between the customer service agent and the user." In the intent scoring phase, the complete input to the intent assessment model is: (12) The intent assessment model performs an intent assessment task to obtain a score dataset containing initial intent label scores and corrected intent scores. , can be represented as: (13) in: (14) For the rating dataset The first in A dialogue dataset, It refers to the initial intention before correction and the high-scoring intention within the corrective intention. It refers to the initial intention before correction and the low-scoring intention in the correction intention. In actual annotation, choose ( High score intention Label it as an intent tag, and reject ( Low rating intention .

[0057] Furthermore, by performing tasks such as intent pre-identification, intent correction suggestion, intent correction, and intent scoring, various types of intent recognition data are obtained. This allows for joint training of multiple models—the intent output model, intent correction model, and intent evaluation model—or joint training of a large model that constructs these models across multiple tasks, achieving model fine-tuning and optimization. Different tasks require different sample data during model training; therefore, separate sample datasets need to be constructed. Based on this, when the evolution rate determines that the intent correction data still needs further correction, a sample dataset is constructed to jointly train the intent recognition model and intent correction model, achieving model fine-tuning and optimization. Therefore, in step 500, the intent output model and intent correction model are fine-tuned, specifically including: Step 501: Divide the sample dialogue dataset into a first dataset and a second dataset according to the intent correction data; the first dataset consists of the first sample dialogue data corresponding to the error label, and the second dataset consists of the second sample dialogue data in the sample dialogue dataset other than the first sample dialogue data; Step 502: Based on the intent label data and the second dataset, construct a first sample dataset; Step 503: Based on the scoring data, select a third dataset from the first dataset; the third dataset consists of sample dialogue data where the intention score after intention correction is higher than the intention score before intention correction. Step 504: Construct a second sample dataset based on the correction suggestion dataset, the intent label data, and the third dataset; Step 505: Construct a third sample dataset based on the correction suggestion dataset, the intention correction data, and the third dataset; Step 506: Using the first sample dataset, the second sample dataset, the third sample dataset, and the scoring data, jointly train the intent output model, the intent correction model, and the intent evaluation model to fine-tune the intent output model and the intent correction model.

[0058] Based on the intent correction data, the first dataset that underwent intent correction and the second dataset that did not underwent intent correction were identified in the sample dialogue dataset. The first dataset contains the first sample dialogue data that underwent intent correction corresponding to the erroneous intent labels, and the second dataset contains the second sample dialogue data that did not undergo intent correction.

[0059] Based on the second dataset without intent correction and intent label data, a first sample dataset for training the intent pre-recognition task is constructed. Since the intent labels of the uncorrected sample dialogue data have high confidence, the first sample dataset for training the intent pre-recognition task is constructed based on the sample dialogue data with high confidence.

[0060] Furthermore, based on the scoring data, a third dataset is selected from the second dataset, which consists of the first sample dialogue data that underwent intent correction, and shows that the intent score after intent correction is higher than the intent score before intent correction. A second sample dataset for training the intent correction suggestion task is constructed based on the intent suggestion dataset, intent label data, and the third dataset. Intent correction suggestion data where the intent correction model proposed intent correction suggestions, the intent correction was performed on the sample dialogue data during the intent correction process, and the intent score after correction is higher in the scoring stage is extracted to form the second sample dataset for training the intent correction suggestion task.

[0061] Based on the correction suggestion dataset, intent correction data, and the third dataset, a third sample dataset is constructed for training the intent correction task. Sample dialogues that underwent intent correction and achieved higher intent label scores compared to the intent label data during the scoring phase are used as the third sample dataset for intent correction task training.

[0062] Using the constructed first, second, and third sample datasets, along with rating data, the intent output model, intent correction model, and intent evaluation model are jointly trained, thereby fine-tuning the intent output and intent correction models. Based on the fine-tuned intent output and intent correction models, the sample dialogue dataset is re-input into the trained intent output model to obtain the intent label data output by the trained intent output model, achieving iterative annotation of the sample dialogue dataset. A fourth sample dataset is constructed based on the rating data for training the intent evaluation task.

[0063] In one embodiment, the joint training of the intent output model, intent correction model, and intent evaluation model specifically involves training the intent output model using a first sample dataset, fine-tuning the correction suggestion sub-model in the intent correction model using a second sample dataset, fine-tuning the correction execution sub-model in the intent correction model using a third sample dataset, and fine-tuning the intent evaluation model using a fourth sample mutual dataset constructed from scoring data.

[0064] In another embodiment, the intent output model, intent correction model, and intent evaluation model are fine-tuned using a first sample dataset following the iterative annotation process of the sample dialogue dataset; the same three models are fine-tuned using a second sample dataset following the iterative annotation process of the sample dialogue dataset; and finally, the same three models are fine-tuned using a third sample dataset following the iterative annotation process of the sample dialogue dataset. In other words, the intent output model, intent correction model, and intent evaluation model are jointly trained using the first, second, and third sample datasets, respectively, following the iterative annotation process of the sample dialogue dataset, thus achieving model fine-tuning.

[0065] The sample dialogue data in the second dataset consists of sample dialogue data with high confidence selected through intent correction. The sample dialogue data in the second dataset is then labeled using intent label data to obtain the corresponding dataset, which can be represented as follows: (15) in, The first in A set of dialogue data , can be represented as: (16) Accordingly, the loss function for the intent pre-recognition task during the model training phase is: (17) in, This indicates the model strategy corresponding to the large model.

[0066] Based on the data set obtained through intent correction suggestions, intent correction, and intent assessment mentioned above. , and The intent correction suggestions that underwent intent correction during the intent correction phase and achieved higher intent scores after correction during the intent evaluation phase are extracted to form the training sample dataset for the intent correction suggestion task, as shown below: (18) in, The first in A set of dialogue data , can be represented as: (19) Accordingly, the loss function for the intent correction suggestion task during the model training phase is: (20) Furthermore, the dataset obtained through intent correction and intent assessment and The sample dialogue data that underwent intent correction and received higher scores than the intent label data during the intent evaluation phase were used as training samples for the intent correction task, as shown below: ;(twenty one) in, The first in A set of dialogue data , can be represented as: ;(twenty two) Accordingly, the loss function for the intent correction task during the model training phase is: ;(twenty three) For the intent assessment task, training samples are constructed using scoring data. The training objective for the intent assessment task is the cross-entropy loss function of the language model. Therefore, the output target of the model is a string, and the constructed training samples can be represented as follows: ;(twenty four) in, The first in A set of dialogue data , can be represented as: (25) Accordingly, the loss function for the intent correction task during the model training phase is: (26) The loss function for multi-task joint training is: (27) In one embodiment, to enable the model to output more accurate intent labels, Direct Preference Optimization (DPO) using rating data helps the model output intents with higher ratings. Based on this, after step 506, the method further includes: Step 601: Based on the scoring data and the intent correction data, filter target dialogue data from the sample dialogue dataset; the target dialogue data includes high-scoring intent data and low-scoring intent data, wherein the score of the high-scoring intent data is higher than a preset first scoring threshold, and the score of the low-scoring intent data is lower than a preset second scoring threshold, wherein the first scoring threshold is higher than the second scoring threshold. Step 602: Construct a preference sample dataset based on the target dialogue data, and use the preference sample dataset to perform preference optimization training on the intent output model.

[0067] Based on the scoring data and intent correction data, target dialogue data is selected from the sample dialogue dataset. This target dialogue data includes high-scoring intent data and low-scoring intent data. Further, high-scoring intent data refers to sample dialogue data whose intent score is higher than a preset first scoring threshold, and low-scoring intent data refers to sample dialogue data whose intent score is lower than a preset second scoring threshold, wherein the first scoring threshold is higher than the second scoring threshold.

[0068] Furthermore, a preference sample dataset is constructed based on the target dialogue data, and this preference sample dataset is used to train the intent output model for preference optimization.

[0069] In each iteration of the intent output model, intent correction model, and intent evaluation model, multiple tasks such as intent pre-identification, intent correction suggestion, intent correction, and intent evaluation are performed based on the large model. Through multiple iterations, the performance of the large model on various tasks is improved. To quantify the optimization effect of the model and prevent inaccurate predictions due to insufficient training or overfitting due to excessive training, the evolution rate is used as the criterion for model iteration optimization. Specifically, the evolution rate is the proportion of sample dialogue data with unchanged intent labels in the current iteration relative to the total amount of data in the sample dialogue dataset. In other words, it is the ratio of the amount of sample dialogue data with unchanged intent labels in the current iteration to the amount of data participating in model optimization. The evolution rate is compared with a preset threshold to determine whether it meets the iteration termination condition. If the evolution rate does not meet the termination condition, the next iteration continues until the evolution rate meets the termination condition.

[0070] In this embodiment, the tree structure based on intent tags constrains the decoding trajectory of the large language model, avoiding unexpected word tags from the generative large language model output, and to a certain extent solving the problem of unstable output of the large language model in classification tasks.

[0071] Furthermore, based on an intent annotation process that includes intent pre-identification, intent correction suggestion, intent correction, intent evaluation, multi-task joint optimization, and direct preference optimization, automated data annotation and optimization are achieved, with annotation quality gradually improved through multiple iterations. Moreover, an evolution rate metric is used to control the number of iterations in model training and the data annotation effect, thereby quantifying the quality of data annotation.

[0072] This application also provides an intent recognition method, including the following steps: S1, Input the dialogue data of the user's question into the intent recognition model to obtain the user intent corresponding to the dialogue data output by the intent recognition model; The intent recognition model is pre-trained based on a sample dataset, which is constructed by annotating the sample dialogue dataset using the intent annotation methods described in the above embodiments.

[0073] The user's question dialogue data is input into the intent recognition model to obtain the user intent corresponding to the dialogue data output by the intent recognition model. The intent recognition model is pre-trained based on a constructed sample dataset, which is constructed by annotating the sample dialogue dataset with intent using the intent annotation methods described in the above embodiments, and will not be repeated here.

[0074] Optionally, the intent recognition model is built on a large model. By using the intent annotation methods described in the above embodiments, the sample dialogue dataset is annotated to obtain the annotated data of the sample dialogue dataset. Based on this, a sample dataset is constructed, and the intent recognition model built on the large model is pre-trained to obtain an intent recognition model for intent recognition. This improves the model training efficiency and reduces the manual cost of the model training stage. When this intent recognition model is applied to dialogue systems such as customer service seats, it can accurately identify the user's intent based on the dialogue data of the user's questions, thereby generating response information.

[0075] The intent annotation apparatus provided in the embodiments of this application is described below. The intent annotation apparatus described below can be referred to in correspondence with the intent annotation method described above.

[0076] Reference Figure 4 The intent annotation device provided in this application includes: Data acquisition module 10 is used to acquire the sample dialogue dataset to be labeled; The intent labeling module 20 is used to input the sample dialogue dataset into the intent output model during the Kth iteration of the labeling process, and obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset, and the intent output model is built based on a large model; K is a positive integer greater than 1; The intent correction module 30 is used to input the sample dialogue dataset and the intent label data into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate calculation module 40 is used to determine the evolution rate of the sample dialogue dataset based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. The annotation iteration module 50 is used to fine-tune the intention output model and the intention correction model when it is determined based on the evolution rate that the intention correction data still needs to be corrected, to determine the intention correction data as the historical label data, and to return to the step of inputting the sample dialogue dataset into the intention output model to obtain the intention label data output by the intention output model; The annotation determination module 60 is used to determine the intention correction data as the annotation data of the sample dialogue dataset when it is determined based on the evolution rate that the intention correction data does not need to be corrected.

[0077] In one embodiment, the intent correction model includes a correction suggestion sub-model and a correction execution sub-model; the intent correction module 20 is further configured to: The sample dialogue dataset and the intent label data are input into the correction suggestion sub-model to obtain a correction suggestion dataset for the erroneous intent labels in the intent label data, output by the correction suggestion sub-model; the correction suggestion dataset contains the correction intent labels corresponding to the erroneous intent labels; The sample dialogue dataset, the intent label data, and the correction suggestion dataset are input into the correction execution sub-model to obtain the intent correction data output by the correction execution sub-model.

[0078] In one embodiment, the intent annotation device further includes an intent evaluation module, used for: The sample dialogue dataset, the intent label data, and the intent correction data are input into the intent evaluation model to obtain the score data output by the intent evaluation model; the intent evaluation model is used to score the intent label data and the intent correction data.

[0079] In one embodiment, the annotation iteration module 50 is further configured to: Based on the intent correction data, the sample dialogue dataset is divided into a first dataset and a second dataset; the first dataset consists of the first sample dialogue data corresponding to the error label, and the second dataset consists of the second sample dialogue data in the sample dialogue dataset other than the first sample dialogue data. Based on the intent label data and the second dataset, a first sample dataset is constructed; Based on the scoring data, a third dataset is selected from the first dataset; the third dataset consists of sample dialogue data where the intention score after intention correction is higher than the intention score before intention correction. Based on the correction suggestion dataset, the intent label data, and the third dataset, a second sample dataset is constructed; Based on the correction suggestion dataset, the intention correction data, and the third dataset, a third sample dataset is constructed; Using the first sample dataset, the second sample dataset, the third sample dataset, and the rating data, the intent output model, the intent correction model, and the intent evaluation model are jointly trained to fine-tune the intent output model and the intent correction model.

[0080] In one embodiment, the intent annotation device further includes a direct preference optimization module, used for: Based on the scoring data and the intent correction data, target dialogue data is filtered from the sample dialogue dataset; the target dialogue data includes high-scoring intent data and low-scoring intent data, the score of the high-scoring intent data is higher than a preset first scoring threshold, and the score of the low-scoring intent data is lower than a preset second scoring threshold, the first scoring threshold is higher than the second scoring threshold; A preference sample dataset is constructed based on the target dialogue data, and the intention output model is trained using the preference sample dataset for preference optimization.

[0081] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an intent annotation method, which includes: Obtain the sample dialogue dataset to be labeled; During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1; The sample dialogue dataset and the intent label data are input into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. If, based on the evolution rate, it is determined that the intent correction data still needs correction, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is determined as the labeled data of the sample dialogue dataset.

[0082] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the intent annotation method provided by the above methods, the method including: Obtain the sample dialogue dataset to be labeled; During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1; The sample dialogue dataset and the intent label data are input into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. If, based on the evolution rate, it is determined that the intent correction data still needs correction, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is determined as the labeled data of the sample dialogue dataset.

[0084] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the intent annotation methods provided by the methods described above, the method comprising: Obtain the sample dialogue dataset to be labeled; During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1; The sample dialogue dataset and the intent label data are input into the intent correction model to obtain intent correction data output by the intent correction model; the intent correction data includes intent labels of each of the sample dialogue datasets, and the intent correction model is used to correct erroneous intent labels in the intent label data; The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. If, based on the evolution rate, it is determined that the intent correction data still needs correction, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is determined as the labeled data of the sample dialogue dataset.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An intent annotation method, characterized in that, include: Obtain the sample dialogue dataset to be labeled; During the Kth iteration of annotation, the sample dialogue dataset is input into the intent output model to obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset; the intent output model is built based on a large model; K is a positive integer greater than 1; The sample dialogue dataset and the intent label data are input into the intent correction model to obtain the intent correction data output by the intent correction model. The intent correction data includes intent labels for each of the sample dialogue data, and the intent correction model is used to correct erroneous intent labels in the intent label data. The evolution rate of the sample dialogue dataset is determined based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. If, based on the evolution rate, it is determined that the intent correction data still needs correction, the intent output model and the intent correction model are fine-tuned, the intent correction data is determined as the historical label data, and the step of inputting the sample dialogue dataset into the intent output model to obtain the intent label data output by the intent output model is returned. If it is determined based on the evolution rate that the intent correction data does not need to be corrected, the intent correction data is identified as the labeled data of the sample dialogue dataset.

2. The intent annotation method according to claim 1, characterized in that, The intent correction model includes a correction suggestion sub-model and a correction execution sub-model; the step of inputting the sample dialogue dataset and the intent label data into the intent correction model to obtain the intent correction data output by the intent correction model includes: The sample dialogue dataset and the intent label data are input into the correction suggestion sub-model to obtain a correction suggestion dataset for the erroneous intent labels in the intent label data, output by the correction suggestion sub-model; the correction suggestion dataset contains the correction intent labels corresponding to the erroneous intent labels; The sample dialogue dataset, the intent label data, and the correction suggestion dataset are input into the correction execution sub-model to obtain the intent correction data output by the correction execution sub-model.

3. The intent annotation method according to claim 1, characterized in that, After inputting the sample dialogue dataset and the intent label data into the intent correction model to obtain the intent correction data output by the intent correction model, the method further includes: The sample dialogue dataset, the intent label data, and the intent correction data are input into the intent evaluation model to obtain the score data output by the intent evaluation model; the intent evaluation model is used to score the intent label data and the intent correction data.

4. The intent annotation method according to claim 3, characterized in that, The fine-tuning of the intent output model and the intent correction model includes: Based on the intent correction data, the sample dialogue dataset is divided into a first dataset and a second dataset; the first dataset consists of the first sample dialogue data corresponding to the error label, and the second dataset consists of the second sample dialogue data in the sample dialogue dataset other than the first sample dialogue data. Based on the intent label data and the second dataset, a first sample dataset is constructed; Based on the scoring data, a third dataset is selected from the first dataset; the third dataset consists of sample dialogue data where the intention score after intention correction is higher than the intention score before intention correction. Based on the correction suggestion dataset, the intent label data, and the third dataset, a second sample dataset is constructed; Based on the correction suggestion dataset, the intention correction data, and the third dataset, a third sample dataset is constructed; Using the first sample dataset, the second sample dataset, the third sample dataset, and the rating data, the intent output model, the intent correction model, and the intent evaluation model are jointly trained to fine-tune the intent output model and the intent correction model.

5. The intent annotation method according to claim 4, characterized in that, After jointly training the intent output model, the intent correction model, and the intent evaluation model using the first sample dataset, the second sample dataset, the third sample dataset, and the rating data, the method further includes: Based on the scoring data and the intent correction data, target dialogue data is filtered from the sample dialogue dataset; the target dialogue data includes high-scoring intent data and low-scoring intent data, the score of the high-scoring intent data is higher than a preset first scoring threshold, and the score of the low-scoring intent data is lower than a preset second scoring threshold, the first scoring threshold is higher than the second scoring threshold; A preference sample dataset is constructed based on the target dialogue data, and the intention output model is trained using the preference sample dataset for preference optimization.

6. An intent recognition method, characterized in that, include: The user's dialogue data is input into the intent recognition model to obtain the user's intent corresponding to the dialogue data output by the intent recognition model. The intent recognition model is obtained by pre-training on a sample dataset, which is constructed by annotating the sample dialogue dataset with intent using the intent annotation method described in any one of claims 1 to 5.

7. An intent annotation device, characterized in that, include: The data acquisition module is used to acquire the sample dialogue dataset to be labeled; The intent labeling module is used to input the sample dialogue dataset into the intent output model during the Kth iteration of the labeling process, and obtain the intent label data output by the intent output model; the intent label data includes the intent labels of the sample dialogue data in each round of the sample dialogue dataset, and the intent output model is built based on a large model; K is a positive integer greater than 1; The intent correction module is used to input the sample dialogue dataset and the intent label data into the intent correction model to obtain the intent correction data output by the intent correction model. The intent correction data includes intent labels for each of the sample dialogue data, and the intent correction model is used to correct erroneous intent labels in the intent label data. The evolution rate calculation module is used to determine the evolution rate of the sample dialogue dataset based on the consistency between the intent correction data obtained during the Kth iteration annotation process and the historical label data; the historical label data is the intent correction data obtained during the (K-1)th iteration annotation process. The annotation iteration module is used to fine-tune the intention output model and the intention correction model when it is determined based on the evolution rate that the intention correction data still needs to be corrected, to determine the intention correction data as the historical label data, and to return to the step of inputting the sample dialogue dataset into the intention output model to obtain the intention label data output by the intention output model; The annotation determination module is used to determine the intention correction data as the annotation data of the sample dialogue dataset when it is determined based on the evolution rate that the intention correction data does not need to be corrected.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the intent annotation method as described in any one of claims 1 to 5, or the intent recognition method as described in claim 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the intent annotation method as described in any one of claims 1 to 5, or the intent recognition method as described in claim 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the intent annotation method as described in any one of claims 1 to 5, or the intent recognition method as described in claim 6.

Images