Power grid customer service dialogue generation method and device based on external knowledge

Abstract

This invention discloses a method and device for generating power grid customer service dialogues based on external knowledge. The method includes: an encoder module encodes the current round of user messages and a knowledge base; a knowledge selection module, based on the encoding results, performs a topic transfer calculation or a topic inheritance calculation to select corresponding candidate knowledge from the knowledge base; and a generator module generates responses for the current round based on the selected candidate knowledge. This invention can more accurately construct query vectors, improving the accuracy of customer service responses.

Description

Technical Field

[0001] The present invention relates to the field of open domain dialogue generation-natural language processing technology, and in particular to a method and device for generating power grid customer service dialogue based on external knowledge. Background Art

[0002] In open-domain dialogue generation technology, external knowledge-based dialogue generation has received widespread attention in recent years. This technology, also known as knowledge-based dialogue generation, consists of two subtasks: knowledge selection and response generation. Existing work focuses on two main areas of improvement: improving the accuracy of knowledge selection and enhancing knowledge-aware response generation modules. The first area of ​​improvement improves knowledge selection accuracy by mining latent features in conversation history and introducing a posteriori knowledge. For example, Meng et al. designed a knowledge selection network based on the dual learning paradigm (Chuan Meng, Pengjie Ren, Zhumin Chen, Weiwei Sun, Zhaochun Ren, Zhaopeng Tu, and Maarten de Rijke. 2020. Dukenet: A dual knowledge interaction network for knowledge-grounded conversation. In Proceedings of the 43rd International ACM SIGIRconference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1151–1160. ACM), treating knowledge tracking and knowledge transfer as a pair of dual tasks, providing feedback information to each other during training to achieve a joint improvement in the performance of the two modules. Zheng et al. considered the differences in knowledge between two rounds of conversation (Chujie Zheng, Yunbo Cao, Daxin Jiang, and Minlie Huang. 2020. Difference-aware knowledge selection for knowledge-grounded conversation generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 115–125, Online. Association for Computational Linguistics.). The transfer of knowledge in a conversation should neither be too abrupt nor completely consistent and repetitive. Using this feature, the authors designed a knowledge difference-aware selector.Kim et al. designed a sequential latent variable model based on the idea of ​​conditional variational autoencoder (CVAE) (Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential latent knowledge selection for knowledge-grounded dialogue. In 8th International Conference on Learning Representations, ICLR2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.), treating the entire dialogue turn as a process of sequential knowledge selection, thereby improving the diversity of knowledge selection. Zhan et al. found that not only can the knowledge selection stage improve diversity by sampling latent variables from the latent space, but using this technique in the dialogue generation stage can also improve the diversity of response generation. Therefore, they proposed a collaborative latent variable model (Haolan Zhan, LeiShen, Hongshen Chen, Hainan Zhang. 2021. CoLV: A Collaborative Latent Variable Model for Knowledge-Grounded Dialogue Generation. EMNLP (1) 2021: 2250-2261). The second category of improvements considers how to better utilize the knowledge selected by the model in the first stage and efficiently incorporate this knowledge into the responses. Because the golden knowledge contains term-level noise, which can reduce the quality of response generation, Zheng et al. designed a term-level denoising method (Wen Zheng, Natasa Milic-Frayling, and Ke Zhou. 2021. Knowledge-grounded dialogue generation with term-level de-noising. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2972–2983, Online. Association for Computational Linguistics.). By improving the loss function and utilizing an attention mechanism, the model focuses more on knowledge terms that appear in the standard responses, improving generation quality.Zhao et al. introduced the pre-trained generative model GPT-2 as a generator and marked the knowledge, dialogue history, and responses in the segment embedding layer to improve the generation effect (Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020b. Knowledge Grounded Dialogue Generation with Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3377–3390. Association for Computational Linguistics).

[0003] Most existing knowledge selection techniques directly pass the entire conversation history unprocessed to the knowledge selector to construct a query vector and use this query vector to select candidate knowledge. This approach treats all historical information equally. However, different pieces of information in the conversation history play different roles. In different conversation rounds, the same statement may contain critical information or introduce negative noise that impacts knowledge selection in the current round. Relying solely on models makes it difficult to extract the most appropriate information from the complex historical information to retrieve the knowledge for the current round. Therefore, it is necessary to introduce some prior knowledge to assist the model in filtering out noisy information. As can be seen from the invention of the work of Meng et al. (Chuan Meng, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tengxiao Xi, and Maarten de Rijke. 2021. Initiative-Aware Self-Supervised Learning for Knowledge-Grounded Conversations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'21), July 11–15, 2021, Virtual Event, Canada. ACM, New York, NY, USA, 11 pages), by introducing dominant features to decouple the knowledge selectors, each knowledge selector only utilizes a portion of the information in the conversation history, which can effectively improve the model's efficiency in applying historical information and reduce noise interference. In a multi-round conversation, there are two parties: the user and the system. Generally speaking, the user is the more active party, leading the direction of the topic; the system, or the agent, needs to pick up the topics raised by the user to continue the conversation and provide as informative and interesting responses as possible. In the user's active conversation round, the user's current message contains more key information and should be used to construct the query vector; otherwise, a similar topic should be found based on the historical knowledge in the previous conversation round to continue the conversation.

[0004] Based on the above research and observations, this application found that there are still several issues that need to be improved in the current research on knowledge-based dialogue.

[0005] 1) The dominant feature in the conversation is not the intrinsic reason that affects the knowledge selection process. Since dominance needs to be judged based on the changes in the topic between two consecutive rounds, the topic is the fundamental reason that affects knowledge selection.

[0006] 2) Dominant labels are difficult to label through experience and can only be marked manually one by one, which is too labor-intensive in specific applications. In addition, the accuracy of the self-supervision methods in previous inventions is generally low. Summary of the Invention

[0007] To address the above issues, the present invention discloses a method and device for generating power grid customer service dialogues based on external knowledge. By designing a topic transfer / topic inheritance knowledge selector, the method can more accurately construct query vectors and improve the accuracy of customer service responses.

[0008] The technical contents of the present invention include:

[0009] A customer service dialogue generation method based on external knowledge, the method comprising:

[0010] User message X for round t in the customer service conversation t , get the knowledge base The knowledge of the history round {K′1,…,K′ t-1} and high-dimensional representation of historical knowledge

[0011] Send user message X t and knowledge base Each candidate knowledge Splice them separately and use the BERT encoder to calculate the latent vector representation of the flag bit [CLS] of each splicing result Candidate Knowledge High-dimensional semantic representation of and user message X t High-dimensional semantic representation of Among them, i represents the knowledge base The sequence number of the candidate knowledge in

[0012] Based on the high-dimensional semantic representation of the user message A high-dimensional representation of the historical knowledge Predict the probability of topic shift in this round of conversation To determine the topic transfer calculation or topic inheritance calculation, get the user message X t Corresponding probability distribution of candidate knowledge;

[0013] Combining the probability distribution of the candidate knowledge with the dialogue X t , generate the response for round t.

[0014] Furthermore, the user message X t and knowledge base Each candidate knowledge Splice them separately and use the BERT encoder to calculate the latent vector representation of the flag bit [CLS] of each splicing result Each candidate knowledge High-dimensional semantic representation in latent space and user message X t High-dimensional semantic representation in latent space include:

[0015] Message to user X t Splice with each candidate knowledge in the knowledge base separately, and add the flag bit [CLS] and flag bit [SEP] in BERT to get the splicing result

[0016] Using BERT encoder, the splicing result Encoded as a latent vector representation, the latent vector representation of the flag bit [CLS] of each splicing result is obtained User MessageX t The latent vector representation of The hidden vector representation of the flag bit [SEP] of each splicing result And each candidate knowledge The latent vector representation of

[0017] The latent vector represents And the latent vector representation Perform average pooling operation to obtain candidate knowledge High-dimensional semantic representation in latent space and user message X t High-dimensional semantic representation of

[0018] For the high-dimensional semantic representation After the nonlinear transformation, the attention mechanism is used to calculate the high-dimensional semantic representations Weight To get the user message X t High-dimensional semantic representation in latent space

[0019] Furthermore, the high-dimensional semantic representation based on the user message A high-dimensional representation of the historical knowledge Predict the probability of topic shift in this round of conversation To determine the topic transfer calculation or topic inheritance calculation, including:

[0020] The high-dimensional semantic representation After a feed-forward layer, the user message X is obtained t The information contained in the topic shift

[0021] Represent the historical knowledge in a high-dimensional way After splicing, pass through the TransformerEncoder layer to obtain the comprehensive feature information in historical knowledge

[0022] Splicing the information With the comprehensive feature information Get a high-dimensional vector

[0023] Splicing the information High-dimensional representation of this round of knowledge With the comprehensive feature information Get a high-dimensional vector

[0024] The high-dimensional vector Input the student topic transfer discriminator to obtain the probability of topic transfer in this round of dialogue The structure of the student topic transfer discriminator includes: a fully connected layer and a softmax layer;

[0025] According to the probability Determine whether to perform topic inheritance calculation or topic transfer calculation.

[0026] Furthermore, the training process of the student topic transfer discriminator includes:

[0027] The high-dimensional vector Input the teacher topic transfer discriminator to obtain the probability of topic transfer in this round of dialogue The structure of the teacher topic transfer discriminator includes: a fully connected layer and a softmax layer;

[0028] Based on the probability represents the binary cross entropy loss function of the student topic transfer discriminator, based on the probability Represents the binary cross entropy loss function of the teacher topic transfer discriminator, and adopts KL divergence as the loss function of knowledge distillation to make the probability With the probability Approaching each other.

[0029] To obtain topic transfer and topic inheritance labels, we use the search terms in the correct knowledge selected in each round when constructing the dataset as topic words, and compare the similarities and differences between the topic words in each adjacent round to obtain the label of the topic transfer in each round of dialogue;

[0030] By using the curriculum learning method and knowledge distillation method, the noise that may be contained in the labels is reduced to improve the robustness of the model.

[0031] Furthermore, in the case of determining to perform topic transfer calculation, the user message X is obtained t The corresponding candidate knowledge probability distribution includes:

[0032] The latent vector represents After a fully connected layer activated by ReLU, each candidate knowledge is extracted Message X with the user t High-dimensional representation of semantic associations

[0033] Based on the high-dimensional semantic representation Candidate knowledge Attention calculation is performed between the candidate knowledge points, and the high-dimensional representation of the differences between the candidate knowledge points is extracted through the activation function and the fully connected layer.

[0034] The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the key vector in the attention mechanism is constructed

[0035] The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the query vector Q in the attention mechanism is constructed shift ;

[0036] The key vector With the query vector Q shift After attention calculation according to the additive model, the probability distribution of candidate knowledge is obtained through the softmax layer.

[0037] Furthermore, in the case of determining to perform topic inheritance calculation, the user message X is obtained t The corresponding candidate knowledge probability distribution includes:

[0038] Represent the historical knowledge in a high-dimensional way After splicing, pass through the TransformerEncoder layer to obtain the comprehensive feature information in historical knowledge

[0039] The latent vector represents After a fully connected layer activated by ReLU, each candidate knowledge is extracted Message X with the user t High-dimensional representation of semantic associations

[0040] Based on the high-dimensional semantic representation Candidate knowledge Attention calculation is performed between the candidate knowledge points, and the high-dimensional representation of the differences between the candidate knowledge points is extracted through the activation function and the fully connected layer.

[0041] The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the key vector in the attention mechanism is constructed

[0042] The high-dimensional representation With the comprehensive feature information After concatenation, the dimension is transformed through the fully connected layer and the query vector Q in the attention mechanism is constructed inherit ;

[0043] The key vector With the query vector Q inherit After attention calculation according to the additive model, the probability distribution of candidate knowledge is obtained through the softmax layer.

[0044] Furthermore, the comprehensive candidate knowledge probability distribution and the dialogue X t , generate the response for round t, including:

[0045] Based on the probability distribution of the candidate knowledge in the knowledge base Select candidate knowledge;

[0046] Define a new token in the gpt2 tokenizer dictionary; wherein the new token includes: <context> 、 <response>and <knowledge>;

[0047] Use the new token as segment embedding to mark the conversation X t Different information components in the selected candidate knowledge;

[0048] Mark the conversation after X t The selected candidate knowledge is input into the generator designed with gpt2 as the backbone to obtain the response of the tth round.

[0049] Furthermore, the new token is used as a segment embedding to mark the conversation X t Different information components in the selected candidate knowledge include:

[0050] Will <context>Spliced ​​into the dialogue X t Before <knowledge>Before being spliced ​​into the selected candidate knowledge and during the training phase <response>Before splicing into the current round of responses, it helps distinguish different information input into the gpt2 model.

[0051] A storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.

[0052] An electronic device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0053] Compared with the prior art, the present invention has the following positive effects:

[0054] 1) This invention models the knowledge selection process based on topic transfer characteristics and designs a topic transfer / topic inheritance knowledge selector. Before each round of dialogue begins, the topic transfer discriminator uses the dialogue history and user messages to determine whether the current round of dialogue will experience topic transfer or topic inheritance. This design can better capture key information in the dialogue history and more accurately construct query vectors, thereby improving the accuracy of knowledge selection.

[0055] 2) To address the difficulty of labeling, this paper designs a labeling method based on remote supervision. This method treats the search terms used to retrieve candidate knowledge when constructing the dataset as topic terms and labels topic transfer pseudo-labels based on whether the topic terms appear in historical conversational rounds. Considering that pseudo-labels obtained through remote supervision introduce a certain amount of noise, this paper combines curriculum learning with knowledge distillation to mitigate the negative impact of this noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of a knowledge-based conversation generation model with topic transition awareness.

[0057] Figure 2 Schematic diagram of the encoder module.

[0058] Figure 3 Schematic diagram of the knowledge selection module.

[0059] Figure 4 Schematic diagram of the generator module. DETAILED DESCRIPTION

[0060] In order to make the purpose, invention and advantages of the present invention more clearly understood, the present invention is further described in detail using experiments conducted on real data sets as an example. It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0061] The present invention's method for generating power grid customer service dialogues based on external knowledge, at the beginning of each round of dialogue, the topic transfer discriminator determines whether the topic of this round of dialogue will be transferred based on the dialogue history information. If the user introduces a new topic that causes the topic to differ from the previous round, the model will use the topic transfer knowledge selector to select the knowledge for this round; otherwise, the model will use the topic inheritance knowledge selector and select a similar topic based on the topic of the historical round of dialogue to continue the dialogue. For each sub-selector, the model will be fed with different types of input information. During the training phase, the present invention designs a pseudo-label annotation method based on remote supervision to help the model more accurately capture the occurrence of topic transfer in the dialogue during inference.

[0062] like Figure 1 As shown, the present invention includes an encoder module, a knowledge selector module, and a decoder module. The specific functions of each module are as follows:

[0063] 1. Encoder module

[0064] The present invention adopts BERT as the encoder, such as Figure 2 As shown, given the current round user message X t and knowledge base Message to user X t After concatenating it with each candidate knowledge in the knowledge base and adding the flags [CLS] and [SEP] in BERT, the result is as follows:

[0065]

[0066] Afterwards Pass the BERT encoder and encode it into a latent vector representation. According to the subscript position, the latent vector representation of [CLS], X t The latent vector representation of [SEP] and The latent vector representation of :

[0067]

[0068] for and Because their lengths are the preset sentence lengths, the inconsistent lengths are not conducive to constructing the query variable query for attention, so we perform an average pooling operation and get:

[0069]

[0070] Through the above operations, each candidate knowledge in the knowledge base can be obtained High-dimensional semantic representation in latent space for Perform a nonlinear transformation in high-dimensional space to obtain Then use the attention mechanism to calculate the weight The current round of user messages X under weighted comprehensive D knowledge perception t Obtain high-dimensional semantic representation in latent space

[0071]

[0072]

[0073]

[0074] Where W c , are trainable parameters in the model.

[0075] 2. Knowledge Selector Module

[0076] like Figure 3 As shown in Figure 3, this module is mainly divided into three sub-modules: topic transfer discriminator, topic transfer knowledge selector, and topic inheritance knowledge selector.

[0077] 1. Topic Transfer Discriminator

[0078] The topic transfer discriminator can determine whether the topic transfer will occur in each round of dialogue. Since the present invention combines knowledge distillation technology, the topic transfer discriminator consists of two submodules: teacher topic transfer discriminator and student topic transfer discriminator. Given the high-dimensional representation of the current round of user messages High-dimensional representation of historical knowledge The topic transfer discriminator can predict the probability of topic transfer in this round of dialogue The teacher discriminator has an additional high-dimensional representation of the golden knowledge of this round, which belongs to the posterior information. The teacher discriminator only assists in training the student discriminator during the training phase and does not work during the inference phase.

[0079] High-dimensional representation of user messages in the current round Pass it through a feed-forward layer to try to extract information from user messages indicating topic shifts

[0080]

[0081] The present invention concatenates the high-dimensional latent vector representations of historically selected knowledge and passes them through the TransformerEncoder layer. This layer is composed of several layers of stacked transformer encoder blocks and includes a unique positional embedding layer that marks the conversational turn number of the historical knowledge. This positional relationship helps improve the model's perception of sequential knowledge information and enables it to better extract features transmitted across multiple rounds of conversation. Furthermore, these encoder blocks include unidirectional left-to-right self-attention masks that ensure that knowledge at each position can only reference knowledge from previous rounds and is not visible to knowledge from future rounds. This helps improve the model's generalization capabilities and prevents a sudden drop in performance during the inference phase.

[0082]

[0083] Afterwards, the information contained in this round of messages is and comprehensive characteristic information in historical knowledge After simple processing, splicing is done to obtain a high-dimensional vector As the input of the feed-forward layer of the student model, it is as follows:

[0084]

[0085] For the teacher model, the vector is multi-dimensional representation of the current round of knowledge

[0086]

[0087] After passing through the fully connected layer and normalized by the softmax layer, the output of the teacher topic transfer discriminator is Represents the probability of sub-topic transfer in this round, and its value is between 0 and 1. This probability value is retained without processing and is only used as a softened label.

[0088]

[0089]

[0090] The loss function of the student topic transfer discriminator module is the binary cross entropy loss function:

[0091]

[0092] The loss function of the teacher topic transfer discriminator module is the binary cross entropy loss function:

[0093]

[0094] The present invention uses KL divergence as the loss function for knowledge distillation. This loss can constrain the probability distributions obtained by the teacher model and the student model to be close to each other:

[0095]

[0096] 2. Topic Transfer Knowledge Selector

[0097] According to the survey, topic transfer is usually caused by the new topic mentioned in the message of the user in the conversation, so the present invention uses the current round of user messages for knowledge selection. High-dimensional representation of each candidate knowledge in the current round knowledge base And the high-dimensional representation of the flag [CLS] The present invention first After a fully connected layer with ReLU activation, the semantic association between the knowledge sentence and the current round of user messages is extracted.

[0098]

[0099] Afterwards, the present invention uses the attention mechanism between candidate knowledge and extracts the correlation information between candidate knowledge through activation function and fully connected layer. The model can use the difference information to better select knowledge.

[0100]

[0101]

[0102]

[0103]

[0104] in W k are trainable parameters in the model.

[0105] After the above two feature extraction layers, a high-dimensional representation of the semantic association between user messages and knowledge can be obtained. High-dimensional representation of differences between candidate knowledge The former is concatenated with the corresponding candidate knowledge representation, and then the dimension is transformed through the fully connected layer to construct the key vector (key) in the attention mechanism; the latter is concatenated with the high-dimensional representation of the user's current round of messages and then the dimension is transformed through the fully connected layer to construct the query vector (query) in the attention mechanism. The present invention performs the attention mechanism on the two according to the additive model, and then passes the softmax layer to obtain the probability distribution of the candidate knowledge.

[0106]

[0107]

[0108]

[0109]

[0110] in W and U are trainable parameters in the model.

[0111] 3. Topic inheritance knowledge selector

[0112] In the case of topic inheritance, the user side is likely to agree with or confirm the system side's message in this round and not raise a new topic. At this time, the system side needs to select a suitable topic from the previous conversation to continue the conversation. In this case, the model needs to make full use of the knowledge selected in the previous round to make knowledge selection in this round. The present invention first uses the same network as the topic transfer knowledge selector to extract the semantic associations in [CLS] and extract the association information between candidate knowledge.

[0113] Afterwards, the present invention uses a module with the same structure as that in the topic transition discriminator to extract comprehensive feature information from historical knowledge, but this module does not share parameters with the previous module.

[0114]

[0115] As for the attention mechanism, it is similar to the previous one. This module is also implemented using the additive model, and the distribution of candidate knowledge can be obtained in the end.

[0116]

[0117]

[0118]

[0119]

[0120] Combining the above modules, our invention first determines whether the topic will shift in the current round based on the topic shift discriminator. If so, we select the knowledge with the highest probability based on the distribution of the topic shift knowledge selector for the subsequent generation steps. Otherwise, we select the knowledge with the highest probability based on the distribution of the topic inheritance knowledge selector.

[0121] 3. Generator module

[0122] Given the selected knowledge and dialogue history K′1,…,K′ t-1 Afterwards, Figure 4 As shown, the decoder synthesizes the two parts of information to generate a response. This paper uses GPT-2 as the backbone to design the generator and defines new tokens in the dictionary of gpt2 tokenizer: <context> "," <response> "," <knowledge>"And use them as segment embeddings to mark different information components in the input. The generator module is trained using cross-entropy loss. It is important to note that only the loss is calculated for generating responses. The previous knowledge and dialogue history are masked when constructing the target.

[0123]

[0124] In addition, the task of generating knowledge-based dialogues faces the problem of topic labels not being present in the dataset. To address this difficulty, the present invention uses a remotely supervised data annotation method, which labels the data by assuming that the entity words of the candidate knowledge sentences in the retrieved knowledge base are topic words. However, the above method is highly hypothetical and noisy, so the present invention improves it by combining it with a noisy label learning method. Specifically, by combining curriculum learning with knowledge distillation methods, the model can mitigate the harm caused by noisy labels during the training phase. Experimental results show that topic information in multi-round dialogues can indeed help model the knowledge selection process.

[0125] In summary, the present invention models the knowledge selection process based on topic transfer characteristics and designs a topic transfer / topic inheritance knowledge selector. Before each round of dialogue begins, the topic transfer discriminator uses the dialogue history and user messages to determine whether the current round of dialogue will experience topic transfer or topic inheritance. This design can utilize key information in the dialogue history to more accurately construct query vectors, thereby improving the accuracy of knowledge selection.

[0126] To address the difficulty of labeling, this paper designs a labeling method based on remote supervision. This method treats the search terms used to retrieve candidate knowledge when constructing the dataset as topic terms and labels topic transfer pseudo-labels based on whether the topic terms appear in historical conversational rounds. Considering that pseudo-labels obtained through remote supervision introduce a certain amount of noise, this paper combines curriculum learning with knowledge distillation to mitigate the negative impact of this noise.

[0127] In the experimental verification, the dataset used is the Wizard of Wikipedia (WoW). The WoW dataset was obtained by the Facebook team through crowdsourcing from data collection websites. The specific method for constructing the dataset is as follows:

[0128] When collecting data, both the wizard and the apprentice are played by users. The difference is that the wizard can use the knowledge retrieved from Wikipedia as support to generate knowledge-rich replies, while the wizard user will be prompted to combine knowledge when constructing a reply, but is not allowed to directly copy knowledge.

[0129] At the beginning of a conversation, either the wizard or the apprentice selects a topic and sends a message. The other user receives the topic and message and responds, proceeding in an orderly fashion. When the apprentice receives a message, they can offer any response, but they must ensure that the topic is discussed in depth. Upon receiving the message, the wizard selects a piece of knowledge from the system's knowledge base and responds accordingly. Of course, if the system-provided knowledge item is irrelevant, the wizard can choose not to use external knowledge. This is indicated in the dataset with a corresponding knowledge label. The conversation continues until one party terminates the conversation, and the number of conversation rounds reaches at least the minimum limit specified by the system (4-5 rounds for each). The collected dataset is divided into training, validation, and test sets according to the appropriate ratio. The dataset contains a rich and diverse range of conversation topics, including 1,365 different topics. The data volumes in the three sets are: 18,430, 1,948, and 1,933. The test set is divided into test seen and test unseen. The conversation topics in the test seen set have appeared in the training set, while the conversation topics in the test unseen set are completely new and out-of-domain data, which is more difficult.

[0130]

[0131] Table 1

[0132] As shown in Table 1, the baseline model of the present invention includes some state-of-the-art models for knowledge-based dialogue generation: MemNet, SKLS, DukeNet, DiffKS, MIKe, CoLV, and KnowledGPT. The present invention uses accuracy, BLEU, ROUGE, and METEOR to evaluate the performance of the models. The table above shows the automatic evaluation results of the present invention and the baseline models on the Wizard of Wikipedia dataset, with the best results indicated in bold. As can be seen, on both test sets, the present invention surpasses the baseline models in virtually all metrics. For the knowledge selection metric, ACC, the present invention achieves particularly significant improvements on the Test Unseen dataset, achieving an accuracy 3.3% higher than the best baseline result of 22.5%. Moderate improvements are also achieved on the Test Seen dataset. This demonstrates the effectiveness of decoupling the knowledge selector and selecting a more appropriate selector for each sample based on the topic. The pseudo-labels obtained from distant supervision are instructive. Furthermore, the label noise learning method improves the robustness of the model. The present invention achieves significant improvements on all generation metrics on both test sets, particularly in BLEU-1 and METEOR. Part of this improvement comes from the pre-trained GPT-2 module's better ability to incorporate knowledge; another part comes from the knowledge selector selecting more appropriate knowledge, which is incorporated into the decoder to generate responses that are closer to the ground-truth. This confirms the importance of the knowledge selection module in the KGC task.

[0133] Experimental results show that this application has a higher accuracy rate in knowledge selection, the quality of model-generated responses is better, and the response content is closer to real-person responses, which are not only fluent but also rich in knowledge.

[0134] The above description is only one embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / knowledge> < / response> < / context> < / response> < / knowledge> < / context> < / knowledge> < / response> < / context>

Claims

1. A method for generating customer service dialogue based on external knowledge, the method comprising: User message X for round t in the customer service conversation t , get the knowledge base The knowledge of the history round {K′1,…,K′ t-1 } and high-dimensional representation of historical knowledge Send user message X t and knowledge base Each candidate knowledge Splice them separately and use the BERT encoder to calculate the latent vector representation of the flag bit [CLS] of each splicing result Candidate Knowledge High-dimensional semantic representation of and user message X t High-dimensional semantic representation of Among them, i represents the knowledge base The sequence number of the candidate knowledge in ; Based on the high-dimensional semantic representation of the user message A high-dimensional representation of the historical knowledge Predict the probability of topic shift in this round of conversation To determine the topic transfer calculation or topic inheritance calculation, get the user message X t Corresponding probability distribution of candidate knowledge; Combining the probability distribution of the candidate knowledge and the user message X t , generate the response for round t.

2. The method according to claim 1, wherein The user message X t and knowledge base Each candidate knowledge Splice them separately and use the BERT encoder to calculate the latent vector representation of the flag bit [CLS] of each splicing result Each candidate knowledge High-dimensional semantic representation in latent space and user message X t High-dimensional semantic representation in latent space include: Message to user X t Splice with each candidate knowledge in the knowledge base separately, and add the flag bit [CLS] and flag bit [SEP] in BERT to get the splicing result Using BERT encoder, the splicing result Encoded as a latent vector representation, the latent vector representation of the flag bit [CLS] of each splicing result is obtained User MessageX t The latent vector representation of The hidden vector representation of the flag bit [SEP] of each splicing result And each candidate knowledge The latent vector representation of The latent vector represents And the latent vector representation Perform average pooling operation to obtain candidate knowledge High-dimensional semantic representation in latent space and user message X t High-dimensional semantic representation of For the high-dimensional semantic representation After the nonlinear transformation, the attention mechanism is used to calculate the high-dimensional semantic representations Weight To get the user message X t High-dimensional semantic representation in latent space 3. The method according to claim 1, wherein The high-dimensional semantic representation based on the user message A high-dimensional representation of the historical knowledge Predict the probability of topic shift in this round of conversation To determine the topic transfer calculation or topic inheritance calculation, including: The high-dimensional semantic representation After a feed-forward layer, the user message X is obtained t The information contained in the topic shift The high-dimensional representation set of the historical knowledge After splicing, pass through the TransformerEncoder layer to obtain the comprehensive feature information in historical knowledge Splicing the information With the comprehensive feature information Get a high-dimensional vector Splicing the information High-dimensional representation of this round of knowledge With the comprehensive feature information Get a high-dimensional vector The high-dimensional vector Input the student topic transfer discriminator to obtain the probability of topic transfer in this round of dialogue The structure of the student topic transfer discriminator includes: a fully connected layer and a softmax layer; According to the probability Determine whether to perform topic inheritance calculation or topic transfer calculation.

4. The method according to claim 3, wherein The training process of the student topic transfer discriminator includes: The high-dimensional vector Input the teacher topic transfer discriminator to obtain the probability of topic transfer in this round of dialogue The structure of the teacher topic transfer discriminator includes: a fully connected layer and a softmax layer; Based on the probability represents the binary cross entropy loss function of the student topic transfer discriminator, based on the probability Represents the binary cross entropy loss function of the teacher topic transfer discriminator, and adopts KL divergence as the loss function of knowledge distillation to make the probability With the probability approaching each other; To obtain topic transfer and topic inheritance labels, we use the search terms in the correct knowledge selected in each round when constructing the dataset as topic words, and compare the similarities and differences between the topic words in each adjacent round to obtain the label of the topic transfer in each round of dialogue; By using the curriculum learning method and knowledge distillation method, the noise that may be contained in the labels is reduced to improve the robustness of the model.

5. The method according to claim 1, wherein In the case of determining to perform topic transfer calculation, the user message X is obtained t The corresponding candidate knowledge probability distribution includes: The latent vector represents After a fully connected layer activated by ReLU, each candidate knowledge is extracted Message X with the user t High-dimensional representation of semantic associations Based on the high-dimensional semantic representation Candidate knowledge Attention calculation is performed between the candidate knowledge points, and the high-dimensional representation of the differences between the candidate knowledge points is extracted through the activation function and the fully connected layer. The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the key vector in the attention mechanism is constructed The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the query vector Q in the attention mechanism is constructed shift ; The key vector With the query vector Q shift After attention calculation according to the additive model, the probability distribution of candidate knowledge is obtained through the softmax layer.

6. The method according to claim 1, wherein In the case of determining to perform topic inheritance calculation, the user message X is obtained t The corresponding candidate knowledge probability distribution includes: The high-dimensional representation set of the historical knowledge After splicing, pass through the TransformerEncoder layer to obtain the comprehensive feature information in historical knowledge The latent vector represents After a fully connected layer activated by ReLU, each candidate knowledge is extracted Message X with the user t High-dimensional representation of semantic associations Based on the high-dimensional semantic representation Candidate knowledge Attention calculation is performed between the candidate knowledge points, and the high-dimensional representation of the differences between the candidate knowledge points is extracted through the activation function and the fully connected layer. The high-dimensional representation With the high-dimensional semantic representation After concatenation, the dimension is transformed through the fully connected layer and the key vector in the attention mechanism is constructed The high-dimensional representation With the comprehensive feature information After concatenation, the dimension is transformed through the fully connected layer and the query vector Q in the attention mechanism is constructed inherit ; The key vector With the query vector Q inherit After attention calculation according to the additive model, the probability distribution of candidate knowledge is obtained through the softmax layer.

7. The method according to claim 1, wherein The combination of the candidate knowledge probability distribution and the dialogue X t , generate the response for round t, including: Based on the probability distribution of the candidate knowledge in the knowledge base Select candidate knowledge; Define a new token in the gpt2 tokenizer dictionary; wherein the new token includes: <context> 、 <response>and <knowledge> ；< / knowledge> < / response> < / context> Use the new token as segment embedding to mark the conversation X t Different information components in the selected candidate knowledge; Mark the conversation after X t The selected candidate knowledge is input into the generator designed with gpt2 as the backbone to obtain the response of the tth round.

8. The method according to claim 7, wherein The new token is used as segment embedding to mark the conversation X t Different information components in the selected candidate knowledge include: Will <context>Spliced ​​into the dialogue X t Before <knowledge>Before being spliced ​​into the selected candidate knowledge and during the training phase <response> Before splicing into the current round of responses, it helps distinguish different information input into the gpt2 model.< / response> < / knowledge> < / context> 9. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images