Cue-based learning dialog act recognition system

Abstract

The application belongs to the technical field of dialogue behavior recognition, and specifically relates to dialogue behavior recognition based on prompt learning, which improves the accuracy and generalization ability of dialogue behavior recognition by designing and optimizing specific prompt information. Prompt learning is introduced, a mixed template for dialogue behavior recognition is constructed by combining a hand-designed template and a pre-trained language model, some parts of the template are regarded as adjustable parameters (SoftTokens), and the adjustable parameters are updated and fine-tuned in the training process, so that the accuracy and robustness of the model are improved. Language transformer technology is also used to convert labels into natural language descriptions, further improving the explainability and generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of dialogue behavior recognition technology. Specifically, it utilizes the Prompt Learning method to improve the accuracy and generalization ability of dialogue behavior recognition by designing and optimizing specific prompt information. Background Technology

[0002] Dialogue action recognition is an important research area in natural language processing. Its main purpose is to automatically identify different actions in dialogue text, such as statements, questions, and commands. Over the past few decades, many methods and techniques have been proposed to solve this problem.

[0003] Early dialogue behavior recognition methods were primarily based on manually designed features and rules, identifying different dialogue behaviors by analyzing grammatical and semantic information in the text. However, this approach struggles to handle complex semantic information and requires significant human labor and time.

[0004] With the development of machine learning and deep learning technologies, methods based on these technologies have gradually become the mainstream approaches for dialogue action recognition. Among them, traditional machine learning methods such as Support Vector Machines (SVM) and Naive Bayes classifiers are widely used in dialogue action recognition tasks, training classifiers to categorize text. However, these methods still require manually designed features and struggle to handle complex semantic information.

[0005] In recent years, with the development of deep learning technology and the emergence of pre-trained language models, significant progress has been made in dialogue action recognition tasks. For example, pre-trained language models such as BERT and RoBERTa have achieved remarkable results in dialogue action recognition tasks and are gradually becoming the mainstream methods for dialogue action recognition. These models utilize large-scale corpora for pre-training and then fine-tuning them on specific tasks, which can significantly improve the accuracy and generalization ability of dialogue action recognition.

[0006] Besides pre-trained language models, some deep learning-based methods have been applied to dialogue behavior recognition tasks, such as those based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These methods utilize neural networks to automatically extract features and information from text, thereby achieving dialogue behavior recognition.

[0007] In conclusion, with the continuous development and application of deep learning technology, dialogue behavior recognition has significant research importance and application value in the field of natural language processing. Continuous innovation and development of new algorithms and models will help improve the accuracy and generalization ability of dialogue behavior recognition tasks, providing more intelligent and efficient solutions for applications in human-computer interaction, sentiment analysis, and intelligent customer service.

[0008] However, dialogue behavior recognition still faces some challenges. For example, traditional methods often rely on large amounts of manually labeled data, which is very costly to obtain. Furthermore, dialogue data may contain interference such as synonyms and colloquial language, which can negatively impact the model's predictions and reduce its accuracy. Summary of the Invention

[0009] To address the high cost of data annotation and the interference inherent in dialogue data in dialogue action recognition, this invention introduces Prompt Learning into the task. By constructing templates, domain knowledge and experiential information are injected into the model, thereby improving the accuracy and generalization ability of dialogue action recognition. Compared to traditional fine-tuning, Prompt Learning can reduce the quantity and quality requirements of labeled data through the design of reasonable templates, and improve the model's performance in new domains, exhibiting better scalability and adaptability. Therefore, this invention provides new ideas and solutions for the research and application of dialogue action recognition tasks, and improves the accuracy of dialogue action recognition based on single-sentence dialogue text.

[0010] The technical solution of this invention is to introduce prompt learning, combining hand-designed templates and pre-trained language models to construct a hybrid template for dialogue behavior recognition. Certain parts of the template are treated as adjustable parameters (soft tokens), which are updated and fine-tuned during training to improve the model's accuracy and robustness. Furthermore, this invention employs a verbalizer technique to convert tags into natural language descriptions, further enhancing the model's interpretability and generalization ability. The specific steps include the following three parts:

[0011] (1) Design a learning prompt template.

[0012] Designing a prompt learning template is one of the core steps in the Prompt Learning method. In this step, we need to design a template that accurately guides the model's learning for a specific downstream task. For the Dialogue Act Classification task, we adopted a prompt learning template called a mixed template. A mixed template is a flexible prompt learning template composed of multiple text segments, each containing one or more tokens. Each token can be labeled as hard or soft. Hard tokens must be fixed during training and inference, while soft tokens can be replaced with other words during training and inference. Each text segment in a mixed template can contain one or more hard and soft tokens, allowing the template to adapt to different input and output requirements. The flexibility of the mixed template allows the model to learn task-related information more accurately, while also making the template easier to design and adjust. In the Dialogue Act Classification task, we use a mixed template to constrain the model's input and output formats and design appropriate hard and soft tokens according to the task requirements.

[0013] (2) Pre-trained model BERT

[0014] Pre-trained BERT models are another core component of prompt learning methods. In dialogue action recognition tasks, we use pre-trained BERT models to extract feature representations of the input text and leverage these features for training and prediction. By utilizing the language modeling capabilities of pre-trained BERT models, we can train on smaller datasets and achieve better model generalization. In Prompt Learning, we use a pre-trained BERT model as the base model and guide it to learn specific tasks by adding custom prompt text to its input. In practice, the BERT model can convert the input text into vector representations, providing better input representations for downstream tasks.

[0015] (3) Construct a Verbalizer

[0016] The Verbalizer is another key component of cue learning methods. It transforms the model's output into natural language descriptions, making the model's predictions easier to understand and interpret. In dialogue action recognition tasks, we use the Verbalizer to convert the model's predicted categories into natural language descriptions of dialogue actions, such as "greeting," "asking," and "ending the conversation."

[0017] Beneficial effects

[0018] Compared with traditional single-sentence dialogue behavior recognition methods, this invention utilizes cue learning to better leverage a large amount of unlabeled data to improve the model's generalization ability and performance. At the same time, it designs a flexible mixed template and a soft token-based language converter, which can better utilize human prior knowledge and the grammatical features of natural language, thereby improving the accuracy and efficiency of dialogue behavior recognition.

[0019] This invention employs a prompting-based learning method that uses templates and relevant knowledge to guide model learning. Templates standardize the input and output formats of the model and guide it to learn task-related knowledge and information, improving the model's interpretability. The significance of templates lies in enabling us to more efficiently apply the model to different application scenarios and better understand the model's performance on a specific task. By appropriately setting templates, we can allow the model to better adapt to task requirements, improving its effectiveness and performance.

[0020] In Prompt Learning, using a pre-trained BERT model can effectively alleviate the problem of insufficient data for downstream tasks, while improving the model's generalization ability and performance. Therefore, the role of pre-trained BERT in Prompt Learning is to provide better text representations, alleviate the problem of insufficient data for downstream tasks, and enable the model to better utilize general knowledge of language, thereby improving the effectiveness and performance of natural language processing tasks.

[0021] Verbalizers provide intuitive natural language descriptions of model predictions, making the model's output more readable and understandable to human users. Verbalizers are an important tool for improving the interpretability and operability of models. In this process, the parameters of the Soft Verbalizer (i.e., the embedding of each word in the Label Words) are fixed, and therefore can be shared across multiple different inputs. During computation, the classifier's parameters do not directly use the embedding of each word in the Label Words, but rather the calculated average embedding. Therefore, the classifier's parameters can be updated during training.

[0022] This method introduces prompting learning into the dialogue behavior recognition task, which can effectively improve the accuracy and generalization ability of the dialogue behavior recognition task, while also reducing data annotation costs and mitigating interference problems in dialogue data. Attached Figure Description

[0023] To more clearly illustrate the technical solution of this invention, the accompanying drawings used in the implementation are briefly described below:

[0024] Figure 1 We introduce a dialog behavior recognition model framework based on prompting learning.

[0025] Figure 2 Flowchart for updating a Mixed Template. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the experimental process and results. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] Building upon the general single-sentence dialogue action recognition model BERT-Linear, this invention proposes a novel solution for the single-sentence dialogue action recognition task by introducing a prompt learning method. This solution includes designing a flexible mixed template, fine-tuning a pre-trained BERT model, and employing a soft token-based language converter.

[0028] The specific experimental plan and key points include the following three aspects:

[0029] (1) Build and update the mixed template.

[0030] Mixed Templates are a type of Prompt Learning template used to guide models in learning task-related knowledge and information, thereby improving model performance and interpretability. In the design of Mixed Templates, the input is first text composed of mixed tokens, which is then parsed and converted into a dictionary containing each mixed token. A mixed token contains a special token and some free text strings, for example: "{"soft":"Given the sentence spoken by speaker"},{"soft":"classify the dialog act"}.{"soft":"The dialog act is"}:{"mask"}". The "soft" key-value pair represents the free text of the soft token, which can be used for subsequent BERT fine-tuning. The corresponding hard token is the fixed token in the template text. Besides the special token inherent in the template text, there are other key-value pairs in subsequent processing, such as the "soft_id" key-value pair, which indicates which hard token the current soft token should correspond to.

[0031] After the mixed token parsing is complete, we can prepare to generate the soft token word embeddings for the input. For each soft token, its embedding is initialized using the input embedding from the BERT pre-trained model. This embedding is trainable because it can be backpropagated during fine-tuning. During computation, for the current input batch, its input data (input_ids) is first passed through the BERT raw embedding layer to obtain its input embedding. Then, another embedding layer, the soft_embedding layer, is used to generate the embeddings for all soft tokens. This embedding layer is initialized from the raw embedding layer in the BERT pre-trained model but will be adjusted during subsequent fine-tuning. Next, based on the information in the soft token encoding (soft_token_ids), we select which embedding to use. If the value in a token's soft_token_ids is 0, then the raw embedding is used; otherwise, the generated soft embedding is used. The final output is a combination of the input embeddings.

[0032] If we let e iorig Let e ​​represent the original embedding of the i-th token. i soft Let represent the soft embedding of the i-th token. For each token in the input batch, its final embedding can be represented as:

[0033]

[0034] During computation, `soft_token_ids` is continuously updated to ensure that each soft token correctly matches its corresponding hard token. Specifically, based on the soft token information in the template, a unique ID is assigned to each soft token and added to an identity list (`id_list`). If multiple soft tokens correspond to a single hard token, they are placed in the same `id_list`. Then, `id_list` is added to `soft_token_ids` for selection during computation. For updating soft IDs, a dictionary `idx_mp` records the `id_list` corresponding to each soft ID. If this `id_list` already exists, it is used directly; otherwise, a new `id_list` is generated and the dictionary `idx_mp` is updated to ensure that each soft token correctly matches its corresponding hard token. During fine-tuning, `soft_embedding` is adjusted to allow soft tokens to better participate in model training.

[0035] Let s i Let be the unique ID corresponding to the i-th soft token. The corresponding id_list can be found using the dictionary idx_mp. If id_list already exists, then s... i Add to id_list; otherwise, create a new id_list and add s. i The process of updating soft_token_ids can be represented as:

[0036]

[0037] In this way, the model can continuously update the correspondence between soft and hard tokens during the fine-tuning process, thereby improving the performance of dialogue action recognition tasks. This method of using mixed templates for prompt learning helps the model better learn task-related knowledge and information, and improves the model's accuracy and interpretability in dialogue action recognition.

[0038] (2) Construct a soft verbal converter and perform calculations.

[0039] Soft Verbalizer is a method for generating labels from model predictions. It takes the model's hidden states as input, uses a feedforward network to map the hidden states to a vector space of label logits, and outputs a logit for each class. In its implementation, parameters are generated and initialized using corresponding label words. During computation, the hidden states are input into the feedforward network, and the output is used as the logit to predict the class. These logits are updated via backpropagation during subsequent fine-tuning. Here, we design the label words for each dialogue action label in detail for the SWDA dataset we use.

[0040] The calculation process of the Soft Verbalizer can be divided into the following steps:

[0041] The first step is to use the embedding of each word in Label Words as parameters for the classifier. Specifically, for each label, all words in the label are converted into corresponding token IDs using a tokenizer, and these token IDs are then grouped into a sequence. Because the token sequence lengths may differ for different labels, insufficient sequences are padded with zeros to ensure they are of uniform length. For each label, its corresponding token sequence is passed through the embedding layer of a pre-trained model to obtain its embedding vector, and these vectors are then averaged and used as parameters for the classifier.

[0042] The second step is to extract the hidden state of the last layer from the output of the pre-trained model. Specifically, for an input, it is passed through the encoder of the pre-trained model to obtain all its hidden states, and the hidden state of the last layer is selected as the representation of the input.

[0043] The third step is to pass the hidden state obtained in the second step into the classifier to obtain the label logits output by the classifier. Specifically, the hidden state obtained in the second step is passed through the linear layer of the classifier to obtain the label logits output by the classifier.

[0044] The fourth step is to use the label logits obtained in the third step as the model's prediction result. During model training, the loss can be calculated based on the label logits and ground truth, and backpropagation can be performed to update the parameters.

[0045] In this process, the parameters of the Soft Verbalizer (i.e., the embedding of each word in the Label Words) are fixed, and therefore can be shared with multiple different inputs. During computation, the classifier's parameters do not directly use the embedding of each word in the Label Words, but rather the calculated average embedding. Therefore, the classifier's parameters can be updated during training.

[0046] (3) Experimental Dataset

[0047] This experiment uses the SWDA (The Switchboard Dialogue Act Corpus) dataset. SWDA is a corpus of multi-party telephone conversations containing transcribed texts of 2000 English voice calls from native English speakers across various regions of the US Census. The SWDA dataset is labeled using a DialogueAct-based labeling system, which identifies the purpose of utterances by classifying them. In the experiment, this invention splits the 216,954 dialogues in the SWDA dataset into training, validation, and test sets, as shown in Table 1.

[0048] SWDA 192225 20218 4514

[0049] Table 1. Experimental Database Division and Basic Information

[0050] The specific model training parameter settings in this invention are shown in Table 2.

[0051] The experiment in this invention used one RTX2080Ti GPU.

[0052]

[0053] Table 2 Model Training Parameters

[0054] Baseline experiment 73.5 Experiment of this invention 74.8

[0055] Table 3 Comparison of experimental results of the present invention and baseline experiments.

[0056] The baseline experiment used in this invention is based on the BERT-Linear model. This invention is based on the BERT-Linear model and introduces prompt learning into it. After experimental comparison, it was found that the accuracy of the model proposed in this invention is improved compared with the baseline model on the SWDA test set. As shown in Table 3, the accuracy of the model proposed in this invention is 1.3% higher than that of the baseline model.

Claims

1. A dialogue behavior recognition system based on cue learning, characterized in that, Includes the following steps: (1) Design of prompt learning template: For the dialogue behavior recognition task, a mixed template prompt learning template is adopted, which consists of multiple text fragments. Each text fragment contains one or more word tokens. Each token is marked as hard or soft. Hard tokens must be fixed during training and inference, while soft tokens are replaced with other words during training and inference. Each text fragment in the mixed template can contain one or more hard tokens and soft tokens. Use mixed templates to constrain the input and output formats of the model, and design appropriate hard and soft tokens according to task requirements; (2) Prompt learning to use templates and relevant knowledge to guide model learning; (3) Pre-trained BERT model: Use a pre-trained BERT model to extract feature representations of the input text, and use these features for training and prediction; Using a pre-trained BERT model as the base model, the model is guided to learn specific tasks by adding custom prompt text to its input; (4) Constructing a language converter: In cue-based learning, the Verbalizer refers to the process of mapping the model output to natural language text. After the mixed token parsing is completed, the soft token word embeddings for the input can be prepared. For each soft token, its embedding is initialized using the input embedding from the BERT pre-trained model. During the computation, for the current input batch, its input data input_ids are first processed through the original embedding layer of BERT to obtain its input embedding. Then, another embedding layer, namely the soft_embedding layer, is used to generate the embeddings for all soft tokens. The initialization of this embedding layer comes from the original embedding layer in the BERT pre-trained model, but it will be adjusted during the subsequent fine-tuning process. Finally, based on the information in the soft token encoding soft_token_ids, the appropriate embedding to use is selected.

2. The dialogue behavior recognition system based on cue learning according to claim 1, characterized in that, The training data for pre-trained BERT models comes from large-scale unlabeled text data, and the language model capabilities learned from it can be used in downstream tasks.

3. The dialogue behavior recognition system based on cue learning according to claim 1, characterized in that, If the value of soft_token_ids in a token is 0, then the original embedding is used; otherwise, the generated soft embedding is used, and the final output is a combination of the input embeddings. If we let e i orig Let e ​​represent the original embedding of the i-th token. i soft Let represent the soft embedding of the i-th token. For each token in the input batch, its final embedding can be represented as: （1） During the calculation process, soft_token_ids will be continuously updated so that each soft token can be correctly matched with its corresponding hard token; Specifically, based on the soft token information in the template, a unique ID is assigned to each soft token and placed in the identity list id_list. If multiple soft tokens correspond to one hard token, they are placed in the same id_list. Then, the id_list is added to soft_token_ids for selection during the calculation process. For updating the soft ID, the dictionary idx_mp records the id_list corresponding to each soft ID. If this id_list already exists, it can be used directly; otherwise, a new id_list is generated and the dictionary idx_mp is updated to ensure that each soft token can correctly match its corresponding hard token. During the fine-tuning process, the soft_embedding is adjusted so that the soft tokens can better participate in the model training. make Let i be the unique ID corresponding to the i-th soft token. The corresponding id_list can be found using the dictionary idx_mp. If id_list already exists, then... Add to id_list; otherwise, create a new id_list and add it. The process of updating soft_token_ids can be represented as: （2） In this way, the model can continuously update the correspondence between soft tokens and hard tokens during the fine-tuning process.

4. The dialogue behavior recognition system based on cue learning according to claim 1, characterized in that, The method of prompt learning is introduced into the dialogue behavior recognition task. The specific strategy of step (3) is: manually define the label words corresponding to each dialogue behavior category in dialogue behavior recognition. Specifically, prompt learning is to adapt the downstream task, namely the dialogue behavior recognition task referred to here, to the characteristics of the upstream pre-trained model, namely BERT, design corresponding prompts and apply them to the pre-trained model. In the upstream pre-trained model BERT, the model needs to predict a masked word from a given text. The language converter maps the BERT vocabulary space to the vocabulary space defined by the Verbalizer through the predefined Label Words and the corresponding dialogue behavior labels. When the masked word predicted by the model corresponds to a word defined in the Verbalizer, it is replaced with the dialogue behavior label corresponding to that word.

5. The dialogue behavior recognition system based on cue learning according to claim 1, characterized in that, The calculation process of SoftVerbalizer can be divided into the following steps: The first step is to use the embedding of each word in Label Words as the parameter of the classifier. Specifically, for each label, all words in the label are converted into corresponding token IDs through the tokenizer, and these token IDs are combined into a sequence. For each label, its corresponding token sequence is passed through the embedding layer of the pre-trained model to obtain their embedding vectors. Then, these vectors are averaged and used as the parameter of the classifier. The second step is to extract the hidden state of the last layer through the output of the pre-trained model. Specifically, for an input, it is passed through the encoder of the pre-trained model to obtain all its hidden states, and the hidden state of the last layer is selected as the representation of the input. The third step is to pass the hidden state obtained in the second step into the classifier to obtain the label logits output by the classifier. Specifically, the hidden state obtained in the second step is passed through the linear layer of the classifier to obtain the label logits output by the classifier. The fourth step is to use the label logits obtained in the third step as the result of the model prediction. During the model training process, the loss can be calculated based on the label logits and the ground truth, and backpropagation can be performed to update the parameters.

Images