Intention clarification method of man-machine conversation system
By utilizing statistical analysis and generative language models in the human-computer dialogue system, the problem of initiating clarification at the appropriate time was solved, improving the user experience and the accuracy and fluency of the dialogue system.
Patent Information
- Application Number
- CN202411030668.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-12
AI Technical Summary
Existing technologies in human-computer dialogue systems make it difficult to initiate clarification at appropriate times and reduce the number of candidate intents, leading to a decline in user experience.
By performing statistical analysis on the intent classification validation set, doubt quantiles are determined. A candidate intent list is generated using a pre-trained Chinese BERT model, and different strategies are applied based on the number and confidence of the candidate intents, including direct response, restatement, and clarification. Clarification questions are constructed using a generative language model.
It improves the rationality of the timing of question clarification in the dialogue system, reduces misjudgments and the number of candidate intentions, and enhances user experience and dialogue fluency.
Smart Images

Figure CN118939775A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an intention clarification method for a human-computer dialogue system, and belongs to the technical field of natural language processing. Background Art
[0002] At present, the human-computer dialogue system is a system in which people use natural language to communicate with computers, which can help people answer questions, query information and even shop online. A typical human-computer dialogue system based on text interaction includes several core modules such as natural language understanding, dialogue management and natural language generation. The natural language understanding module needs to convert the text input by the user into an information form that is easy for the computer to understand and process, so that the dialogue management system can adopt appropriate strategies according to the information the user wants to query or the task he wants to complete, and mobilize the natural language generation module to respond or execute the corresponding instructions.
[0003] Specifically, after receiving the user input text, the natural language understanding module needs to determine the domain involved in the user's speech, the intention contained, the additional information mentioned, and even the potential emotional tendencies. For a dialogue system in a single vertical field, its domain is fixed, and common intentions have been pre-defined based on business knowledge. Therefore, the level of system capabilities is mainly reflected in the accuracy of intention judgment and the recognition of additional information.
[0004] Intent discrimination in dialogue systems is usually handled as a text classification task, so there are already a large number of classification algorithms and models available. However, due to the diversity and inevitable ambiguity of natural language dialogues, some user inputs are difficult to be clearly classified as a certain intent by the classifier, which poses a challenge to the dialogue system. A common strategy is to spend energy on improving the performance of the intent classifier, collecting a large number of cases that are difficult to discriminate or fail to discriminate, annotating them for classifier iteration, or replacing more powerful classification models. Due to the existence of the marginal diminishing effect, the cost of this route is very high, the improvement is not limited, and in special cases, it may be wrong to always choose the intent with the highest probability; another strategy is to select the top-k intents for users to choose in cases where the user is not sure enough, but there is a lack of reasonable optimization strategy, which easily returns too many questions and too many options to users to choose, thereby reducing user satisfaction.
[0005] Therefore, how to initiate clarification at a reasonable time and how to reduce the candidate options while covering the real intention when clarifying become technical challenges. Summary of the invention
[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for clarifying the intention of a human-computer dialogue system.
[0007] The purpose of the present invention is achieved through the following technical solutions:
[0008] The intention clarification method of the human-computer dialogue system is characterized by comprising the following steps:
[0009] S1) receiving speech content input by the user;
[0010] In the human-computer dialogue system, in addition to the system actively initiating a dialogue with the user, the user is the first to initiate the dialogue and receive the content of the user's input speech;
[0011] S2) generating a predicted intent list for user utterances;
[0012] After receiving the user's speech, the core of natural language understanding is to understand the intention, determine the user's specific intention, and convert the problem into a list of predicted intentions that contains the true intention at a predetermined confidence level;
[0013] S3) respond to various candidate intent lists with different strategies;
[0014] The number of candidate intent lists varies with different user speech samples. The classifier assigns at least one classification label to any input speech. In order to distinguish between slightly confusing intents and situations that are difficult to understand, a threshold th for the number of candidate intents is set. Different strategies are used to handle different numbers of candidate intent lists, as follows:
[0015] a) If there is only one candidate intent in the list, the conversation engine responds in a preset manner based on the intent;
[0016] b) If the number of candidate intents is greater than the threshold th, it indicates that there are too many ambiguities to determine the true intent, and the dialogue engine uses specific words to ask the user to restate it;
[0017] c) If the number of candidate intents is greater than 1 and less than or equal to the threshold th, and there is a confidence level of not less than d that the user's true intent is in the candidate list, the dialogue engine asks a clarification question and allows the user to select from the candidate intent list;
[0018] The conversation from user speech input to the dialogue system response grasps the user's intention by clarifying and questioning ambiguous intentions.
[0019] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, in which, in step S1), the user inputs the text of the dialogue through the keyboard, directly inputs the text using the voice input method, or directly speaks to the system in the form of voice, or expresses it in the form of a picture with text or sign language; for the dialogue form of non-text input, it is converted into text form using information collection hardware and recognition software.
[0020] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, step S2) comprises the following steps:
[0021] S21) setting a desired confidence level;
[0022] Set the confidence level d, and expect the confidence that the correct intent is included in the candidate intent list in unseen samples to be no less than d;
[0023] S22) determining the doubt degree quantile on the validation set;
[0024] When the data set for intent classification is randomly divided, it can be considered that the samples of each category in the validation set, test set, and training set are identically distributed, and statistical analysis is performed on the prediction results of the validation set;
[0025] Select some samples from the validation set to calculate the confusion in batches. The confusion s(Xi) of the true label Yi of the selected i-th sample Xi is calculated by formula (1);
[0026] s(Xi) = 1- f(Xi) Yi (1)
[0027] Among them, f(Xi) Yi It is the Softmax score of the true intent label Yi corresponding to Xi that is finally calculated in the classifier f;
[0028] The doubt quantile v that can satisfy the requirement that the true intent coverage is not less than the confidence level d is obtained by formula (2);
[0029] v = Ceiling( (N+1) ×d) / N (2)
[0030] Wherein, Ceiling is a rounding function, d is the confidence level preset in step S21), and N is the number of selected samples;
[0031] S23) determining a list of candidate intents using a pre-trained classifier;
[0032] Use pre-trained classifiers;
[0033] In step S22), the confusion quantile v is obtained. For the received speech sample X and each y in the total set of intent labels Y, the candidate intent list L(X) consists of the intents whose confusions predicted by the classifier for X are not higher than v, as shown in formula (3);
[0034] L(X) = { y: s(X,y) ≤ v} (3)
[0035] Where s(X,y) is the doubt level of the intent label y on X, calculated by formula (1).
[0036] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, in step S21), the confidence level d is set to 0.9, 0.95, 0.97 or 0.99.
[0037] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, in step S23), a pre-trained classifier is selected as a Chinese BERT model pre-trained on the domain intention corpus.
[0038] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, in step S3), the threshold th is 2-7.
[0039] Furthermore, in the above-mentioned intention clarification method of the human-computer dialogue system, in step S3), the construction method of the clarification question includes using pre-designed questions, using question templates with variable elements to construct, and generating them using a generative language model.
[0040] Compared with the prior art, the present invention has significant advantages and beneficial effects, which are specifically embodied in the following aspects:
[0041] ① The present invention obtains quantiles by statistically analyzing the distribution of true intent doubts on the intent classification verification set and generalizing it to the prediction of new samples, thereby ensuring that the true intent appears in the candidate intent list with sufficient confidence; by diverting and processing different numbers of candidate intent lists, the boundary between the robot's direct answer and the need for clarification, as well as the boundary between the need for clarification and the incomprehensible is reasonably distinguished, making the timing of the dialogue system to initiate clarification questions more reasonable, avoiding the drawbacks of too many clarifications and too many misjudgments of the highest probability intent directly;
[0042] ② Clarification questions use generative language models to assist in generation, improving the fluency and diversity of sentences;
[0043] ③ The processing method of the present invention can reduce the blindness of mechanical selection of top-k and the number of necessary candidate intents in a vertical field human-computer dialogue system with numerous intents, while ensuring the confidence as much as possible, thereby reducing dialogue turns and improving user experience.
[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the specific embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 : Schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.
[0048] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, directional terms and order terms are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0049] The present invention provides a more balanced and reasonable method for clarifying the intention of a human-computer dialogue system. By statistically analyzing the verification set, the doubt degree quantile is obtained and generalized to the prediction of new samples, thereby ensuring that the true intention will appear in the candidate intention list with sufficient confidence. By separately processing different situations in the candidate intention list, the timing of initiating clarification questions by the dialogue system is made more reasonable, avoiding the drawbacks of too many clarifications and too many misjudgments of the highest probability intention directly. In addition, a generative language model can be used to assist in generating clarification questions to improve the fluency and diversity of sentences.
[0050] like Figure 1 As shown, the intention clarification method of the human-computer dialogue system includes the following steps:
[0051] S1) receiving speech content input by the user;
[0052] In a human-computer dialogue system, except for the scenario where the system actively initiates a dialogue with the user, a typical dialogue is initiated by the user and the system receives the user's input utterances;
[0053] The implementation methods in human-computer dialogue systems with different interfaces are slightly different. Users can input the text of the dialogue through the keyboard, directly input text using the voice input method, or speak directly to the system in the form of voice, or express themselves in the form of pictures with text or sign language; for dialogue forms that are not input in text, information collection hardware and recognition software are used to convert them into text form for subsequent module processing; for example, the content of user speech is received in text form;
[0054] S2) generating a predicted intent list for user utterances;
[0055] After receiving the user's speech, the core of natural language understanding lies in the understanding of intent and determining the specific intent of the user. The prediction of the intent classifier is uncertain. The present invention transforms the problem into generating a predicted intent list containing the true intent under a predetermined confidence level. The specific steps include:
[0056] S21) setting a desired confidence level;
[0057] First, the confidence level d is set, which means that the confidence that the correct intent is included in the candidate intent list in the unseen samples is expected to be no less than d;
[0058] According to the characteristics of the data set and actual needs, it is set to 0.9, 0.95, 0.99, etc. Specifically, in the embodiment of the present invention, the expected confidence d is set to 0.97;
[0059] S22) determining the doubt degree quantile on the validation set;
[0060] When the data set for intent classification is randomly divided, it is assumed that the samples of each category in the validation set, test set, and training set are identically distributed. By performing statistical analysis on the prediction results of the validation set, it is helpful for the prediction of unseen samples.
[0061] Select some samples from the validation set to calculate the confusion in batches. The confusion s(Xi) of the true label Yi of the selected i-th sample Xi is calculated by formula (1);
[0062] s(Xi) = 1- f(Xi) Yi (1)
[0063] Among them, f(Xi) Yi It is the Softmax score of the true intent label Yi corresponding to Xi that is finally calculated in the classifier f;
[0064] The doubt quantile v that can satisfy the requirement that the true intent coverage is not less than the confidence level d is obtained by formula (2);
[0065] v = Ceiling( (N+1) ×d) / N (2)
[0066] Wherein, Ceiling is a rounding function, d is the confidence level preset in step S21), and N is the number of selected samples; specifically, the selected number N is 1000;
[0067] It should be noted that step S21) and step S22) are preparatory steps and only need to be performed once at the beginning of the task, and do not need to be performed in each round of dialogue;
[0068] S23) determining a list of candidate intents using a pre-trained classifier;
[0069] Intent classification, as a special text classification task, has many algorithms and models that can be used, including SVM, multi-layer perceptron, convolutional neural network, etc. In addition, its training and reasoning are also commonly used, and are not closely related to the core steps of the present invention. The pre-trained classifier is directly selected;
[0070] Specifically, the Chinese BERT model pre-trained with domain intent corpus was selected;
[0071] In step S22), the confusion quantile v is obtained. For the received speech sample X and each y in the total set of intent labels Y, the candidate intent list L(X) consists of the intents whose confusions predicted by the classifier for X are not higher than v, as shown in formula (3);
[0072] L(X) = { y: s(X,y) ≤ v} (3)
[0073] Where s(X,y) is the doubt level of the intent label y on X, calculated by formula (1);
[0074] S3) respond to various candidate intent lists with different strategies;
[0075] The number of candidate intent lists varies with different user speech samples; since the current mainstream classifier will assign at least one classification label to any input speech, the case where the candidate intent list is empty is not considered here. In order to distinguish between slightly confusing intents and difficult to understand situations, a threshold th for the number of candidate intents needs to be set, usually between 2 and 7. Specifically, th is set to 5;
[0076] For different numbers of candidate intent lists, the present invention processes them with different strategies, and the specific operations are as follows:
[0077] a) If there is only one candidate intent in the list, the conversation engine responds in a preset manner based on the intent;
[0078] b) If the number of candidate intents is greater than the threshold th, it indicates that there are too many ambiguities to determine the true intent. The dialogue engine uses specific words to ask the user to restate the intent. For example, a sentence like "I'm sorry I didn't understand what you meant. Can you say it again in a clear and understandable way?" is used;
[0079] c) If the number of candidate intents is greater than 1 and less than or equal to the threshold th, and there is a confidence level of not less than d that the user's true intent is in the candidate list, the dialogue engine will ask a clarification question to allow the user to select from the candidate intent list;
[0080] There are many ways to construct clarification questions, such as using pre-designed questions, using question templates with variable elements, and using generative language models to generate them;
[0081] There are many Chinese large language models with public weights or API interfaces available for selection, including Baichuan2-7B, Qwen1.5-7B-Chat, etc. ChatGLM3-6B can be selected in combination with corresponding prompts for generation.
[0082] A round of conversation in which the user's speech is input and the dialogue system responds. By clarifying and questioning ambiguous intentions, the dialogue system can more accurately grasp the user's intentions and thus provide high-quality information and services.
[0083] In summary, the present invention obtains quantiles by statistically analyzing the distribution of true intent doubts on the intent classification verification set and generalizing it to the prediction of new samples, ensuring that the true intent appears in the candidate intent list with sufficient confidence; by diverting and processing different numbers of candidate intent lists, the boundaries between direct robot answers and those that need clarification, as well as the boundaries between those that need clarification and those that cannot be understood are more reasonably distinguished, making the timing of the dialogue system to initiate clarification questions more reasonable, avoiding the drawbacks of too many clarifications and too many misjudgments of the highest probability intent directly; in addition, clarification questions can be assisted in generation using a generative language model to improve the fluency and diversity of sentences. The processing method of the present invention can reduce the blindness of mechanical top-k selection and reduce the number of necessary candidate intents in a vertical field human-computer dialogue system with many intents, while ensuring confidence as much as possible, thereby reducing dialogue rounds and improving user experience. Therefore, the method of the present invention has a wide range of application value in human-computer dialogue systems in various vertical industry fields.
[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0085] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A method for clarifying intentions in a human-computer dialogue system, characterized in that: The following steps are involved: S1) receiving speech content input by the user; In the human-computer dialogue system, in addition to the system actively initiating a dialogue with the user, the user is the first to initiate the dialogue and receive the content of the user's input speech; S2) generating a predicted intent list for user utterances; After receiving the user's speech, the core of natural language understanding is to understand the intention, determine the user's specific intention, and convert the problem into a list of predicted intentions that contains the true intention at a predetermined confidence level; S3) respond to various candidate intent lists with different strategies; The number of candidate intent lists varies with different user speech samples. The classifier assigns at least one classification label to any input speech. In order to distinguish between slightly confusing intents and situations that are difficult to understand, a threshold th for the number of candidate intents is set. Different strategies are used to handle different numbers of candidate intent lists, as follows: a) If there is only one candidate intent in the list, the conversation engine responds in a preset manner based on the intent; b) If the number of candidate intents is greater than the threshold th, it indicates that there are too many ambiguities to determine the true intent, and the dialogue engine uses specific words to ask the user to restate it; c) If the number of candidate intents is greater than 1 and less than or equal to the threshold th, and there is a confidence level of not less than d that the user's true intent is in the candidate list, the dialogue engine asks a clarification question and allows the user to select from the candidate intent list; The conversation from user speech input to the dialogue system response grasps the user's intention by clarifying and questioning ambiguous intentions.
2. The intention clarification method of the human-computer dialogue system according to claim 1, characterized in that: Step S1), the user inputs the text of the dialogue through the keyboard, directly inputs the text using the voice input method, or speaks directly to the system in the form of voice, or expresses in the form of pictures with text or sign language; for non-text input dialogue forms, the information collection hardware and recognition software are used to convert them into text form.
3. The intention clarification method of the human-computer dialogue system according to claim 1, characterized in that: Step S2) comprises the following steps: S21) setting a desired confidence level; Set the confidence level d, and expect the confidence that the correct intent is included in the candidate intent list in unseen samples to be no less than d; S22) determining the doubt degree quantile on the validation set; When the data set for intent classification is randomly divided, it can be considered that the samples of each category in the validation set, test set, and training set are identically distributed, and statistical analysis is performed on the prediction results of the validation set; Select some samples from the validation set to calculate the confusion in batches. The confusion s(Xi) of the true label Yi of the selected i-th sample Xi is calculated by formula (1); s(Xi) = 1- f(Xi) Yi (1) Among them, f(Xi) Yi It is the Softmax score of the true intent label Yi corresponding to Xi that is finally calculated in the classifier f; The doubt quantile v that can satisfy the requirement that the true intent coverage is not less than the confidence level d is obtained by formula (2); v = Ceiling( (N+1) ×d) / N (2) Wherein, Ceiling is a rounding function, d is the confidence level preset in step S21), and N is the number of selected samples; S23) determining a list of candidate intents using a pre-trained classifier; Use pre-trained classifiers; In step S22), the confusion quantile v is obtained. For the received speech sample X and each y in the total set of intent labels Y, the candidate intent list L(X) consists of the intents whose confusions predicted by the classifier for X are not higher than v, as shown in formula (3); L(X) = { y: s(X,y) ≤ v} (3) Where s(X,y) is the doubt level of the intent label y on X, calculated by formula (1).
4. The intention clarification method of the human-computer dialogue system according to claim 3, characterized in that: In step S21), the confidence level d is set to 0.9, 0.95, 0.97 or 0.
99.
5. The intention clarification method of the human-computer dialogue system according to claim 3, characterized in that: Step S23), select the Chinese BERT model pre-trained by the pre-trained classifier for the domain intent corpus.
6. The intention clarification method of the human-computer dialogue system according to claim 1, characterized in that: Step S3), the threshold th is 2 to 7.
7. The intention clarification method of the human-computer dialogue system according to claim 1, characterized in that: Step S3), clarifying that the construction method of the question includes using a pre-designed question, using a question template with variable elements, and generating it using a generative language model.
