Knowledge-centered reply screening method and system in open domain dialogue

Abstract

The present application belongs to the field of man-machine conversation technology in natural language processing, and particularly relates to a knowledge-centered reply screening method and system in an open domain conversation field. Step 1, constructing required training data by using existing conversation data and encyclopedic knowledge data, and constructing a pseudo data algorithm; Step 2, performing three-stage training on a BERT model by using the training set constructed in step 1; Step 3, using the sum of scores of the three stages in step 2 as the final score during testing. The present application is used to solve the problems of low knowledge screening accuracy and inconsistency between knowledge screening results and screened replies existing in the prior art non-structured knowledge enhancement method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-computer dialogue in natural language processing, and particularly relates to a knowledge-centered reply screening method and system for an open-domain dialogue field. BACKGROUND

[0002] Dialogue system is an important branch of natural language processing, which aims to generate meaningful responses in the process of computer-human communication. Current dialogue system research usually uses external knowledge to improve the quality of responses (Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR, 2019. and Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. A knowledge-grounded neural conversation model. AAAI, 2018.). Among them, knowledge is defined as the result and experience of human exploration of the objective world, which is reusable information, such as legal provisions, dictionaries, Wikipedia, Baidu Encyclopedia, news reports, historical documents, literary works, etc. External knowledge is information that can be used outside the dialogue context. Dialogue systems can use this external information to construct more informative responses and enhance the logic and interest of the dialogue (Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR, 2019.). External knowledge can be divided into unstructured knowledge (raw text without processing, such as text on the web) and structured knowledge (unstructured text processed into knowledge entries with fixed structure, such as knowledge graphs in the form of triples). Compared with structured knowledge that needs special processing, unstructured knowledge is easier to obtain, more timely, and has a larger data volume.An example of a Chinese unstructured knowledge enhanced dialogue (Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zhumin Chen, Zhaochun Ren, and Maarten de Rijke. Wizard of Search Engine: Access to Information Through Conversations with Search Engines. In SIGIR 2021.) is shown in Table 1.

[0003] Table 1: Unstructured knowledge enhanced dialogue example

[0004]

[0005]

[0006] In the example in Table 1, the reply uses the external knowledge item (3) to construct a reply that meets the requirements of the conversation context. In practical applications, the amount of external knowledge can vary depending on the system's computing power. Unstructured knowledge-enhanced dialogue systems first need to address the knowledge selection problem. This involves using the conversation context to select the most suitable knowledge items for constructing a reply from a large amount of external knowledge (Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipedia: Knowledge-powered conversational agents. In the 7th International Conference on Learning Representations, ICLR, 2019). After selecting the appropriate knowledge items, the dialogue system uses the conversation context and the retrieved knowledge items to construct a reply (the reply can be generated directly by the dialogue system or selected from a large number of candidates). The retrieval system in the dialogue system based on unstructured knowledge enhancement (Longxuan Ma, Mingda Li, Wei-Nan Zhang, et al. Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey [J]. ACM Transactions on Information Systems (TOIS), 2021.) is a system that selects the best response from multiple candidate responses given the previous dialogue context. After solving the problem of knowledge screening, this type of system needs to further solve the problem of response screening, that is, using the previous dialogue context and retrieved external knowledge to select the most appropriate response from a large number of candidate responses. The two current modes of retrieval-based dialogue systems enhanced by unstructured knowledge are as follows:

[0007] 1) If Figure 1 The conversation-centered retrieval model shown

[0008] This mode first filters useful external knowledge from external knowledge sources using the dialogue context, and then filters the dialogue context and the filtered external knowledge (Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR, 2019. and Chongyang Tao, Changyu Chen, Jiazhan Feng, Ji-Rong Wen, and Rui Yan. A pre-training strategy for zero-resource response selection in knowledge-grounded conversations. ACL / IJCNLP 2021.). On the one hand, this method separates the two tasks of knowledge filtering and reply filtering, ignoring the relationship between the two sub-tasks. On the other hand, the output of knowledge filtering is taken as the input of reply filtering, which is easy to produce cascading errors and thus reduce the accuracy of the entire system.

[0009] 2) as shown in the reply-centered retrieval mode Figure 2

[0010] This mode takes external knowledge as a whole, calculates two types of scores (the relevance score of the candidate reply to the dialogue context, and the relevance score of the candidate reply to the external knowledge), and integrates the two types of scores to obtain the final score of the candidate reply (Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, and Quan Liu. Dually interactive matching network for personalized response selection in retrieval-based chatbots. EMNLP-IJCNLP, 2019.). This method does not perform knowledge filtering, so it cannot provide the accuracy of knowledge filtering, and in the scenario where the external knowledge contains a large number of entries, it cannot remove redundant knowledge.

[0011] ​In summary, the existing unstructured knowledge enhancement method models knowledge filtering and reply filtering separately, breaks the connection between the two, and is prone to cascade errors, resulting in two problems: 1) low accuracy of knowledge filtering; and 2) the result of knowledge filtering is inconsistent with the filtered reply, that is, the reply does not use the retrieved knowledge. SUMMARY

[0012] The present application provides an open domain dialogue field knowledge-centered reply filtering method and system to solve the problems of low accuracy of knowledge filtering and inconsistency between the result of knowledge filtering and the filtered reply existing in the existing unstructured knowledge enhancement method.

[0013] The present application is implemented by the following technical solutions:

[0014] An open domain dialogue field knowledge-centered reply filtering method, the reply filtering method comprising the following steps:

[0015] Step 1, using existing dialogue data and encyclopedic knowledge data to construct required training data, and constructing a pseudo-data algorithm;

[0016] Step 2, using the training set constructed in step 1 to perform three-stage training on the BERT model;

[0017] Step 3, using the sum of the scores of the three stages in step 2 as the final score when testing.

[0018] Further, the step 1 of constructing required training data specifically comprises,

[0019] According to the dialogue data in the data set ζ, the mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) are established, and specifically,

[0020] Data set {D i , C i , R i , y i} is the ith dialogue data in the data set ζ, N represents the total number of dialogue data in the data set ζ, wherein, D i represents the document in the ith dialogue data, represents the first knowledge item in the document, represents the second knowledge item in the document, represents the d i th knowledge item in the document, d i represents the number of knowledge items in D i , C i represents the dialogue context except the reply in the ith dialogue data, Representative C i In the first round of dialogue, Representative C i In the second round of dialogue, Representative C i c i Round dialogue, c i Representative C i The number of rounds in the dialogue, R i For C i Reply, y i =1 represents R i C i The correct answer is y i =0 represents R i Not C i The correct response;

[0021] The dialogue above C i The last round of dialogue Defined as the query Q of the i-th conversation data i , the dialogue above C i The other rounds except the last round of dialogue are defined as the dialogue history of the i-th dialogue data Establish mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) according to the data set ζ;

[0022] Among them, Q is the query, K is the knowledge item, H is the conversation history, and R is the reply.

[0023] Furthermore, the step 1 further includes constructing a training set according to the established mapping relationship, specifically,

[0024] Step 1.1, the mapping relationship G1(Q, K) represents the relevance between the knowledge item and the query, (Q, K) is used as the input of the training set, and the relevance score G1(Q, K) is used as the corresponding output;

[0025] Step 1.2: The mapping relationship G2(K, R) represents the relevance between the knowledge item and the response. (K, R) is used as the input of the training set, and the relevance score G2(K, R) is used as the corresponding output.

[0026] Step 1.3: The mapping relationship G3(K, H, Q, R) represents the relevance of knowledge items, conversation history, queries, and responses. (K, H, Q, R) is used as the input of the training set, and the relevance score G3(K, H, Q, R) is used as the corresponding output.

[0027] Furthermore, the pseudo data construction algorithm in step 1 is specifically as follows:

[0028] Input: Reddit data DR Wikipedia data D w , threshold γ;

[0029] Output: pseudo DGD data Dp;

[0030] 1: initialize Dp as an empty set;

[0031] 2: for each pair of conversation context C and reply R in D R , perform operations 3-6;

[0032] 3: calculate the similarity between the reply and each knowledge entry in Wikipedia data D w using BM25 algorithm;

[0033] 4: if the similarity score is greater than threshold γ;

[0034] 5: then put the found knowledge entry together with C and R into a group and put it into Dp;

[0035] 6: after traversing all data in D R , get the pseudo data Dp.

[0036] Further, step 2 is specifically that segment embedding encoding is defined for document D, conversation data C and reply R respectively, and the last layer hidden state representation of special marker [CLS] located at the beginning of the input sequence is denoted as E cls , E cls passes through two nonlinear layers, and outputs the relevance score G(input):

[0037] G(input) = σ(W2μ(W1E cls +b1) + b2)

[0038] Wherein, W1, W2, b1, b2 are trainable parameters, σ represents sigmoid activation function, and μ represents tank activation function; In the three relevance tasks, the same BERT model is used for training, and the calculation of the relevance score uses the above formula.

[0039] Further, the loss function L final used in step 2 is:

[0040] L final = L1+L2+L3

[0041] Wherein:

[0042]

[0043] In the formula, e is the base of natural logarithm, N1 represents the number of positive examples corresponding to the mapping relationship G1(Q, K), a correlation score of the i-th positive example, a correlation score of the i-th negative example, and a1 represents a correlation score of each positive example a number of corresponding negative examples, a correlation score of the j-th negative example, j = 1, 2,..., a1;

[0044]

[0045] wherein N2 represents a number of positive examples corresponding to the mapping relationship G2(K, R), a correlation score of the i-th positive example, a correlation score of the i-th negative example, and a2 represents a correlation score of each positive example a number of corresponding negative examples, a correlation score of the j-th negative example, j = 1, 2,..., a2;

[0046]

[0047] wherein N3 represents a number of positive examples corresponding to the mapping relationship G3(K, H, Q, R), a correlation score of the i-th positive example, and a3 represents a correlation score of each positive example a number of corresponding negative examples, a correlation score of the j-th negative example, j = 1, 2,..., a3: λ represents different scores assigned to each negative example.

[0048] Further, the step 3 is specifically screening the knowledge entries before constructing the inputs corresponding to the three mapping relationships in the step 1, and screening the top m knowledge entries;

[0049] When the number of knowledge entries is large, the knowledge entries are combined, that is, the top m knowledge entries are screened first during knowledge screening, and then the three scores are calculated; the value of m is selected within the allowable range of error and is different according to different tasks; the value of m is consistent during training and testing.

[0050] An open-domain dialogue field knowledge-centered reply screening system, which uses the reply screening method according to claim 1, and the reply screening system comprises,

[0051] A construction module, which uses existing dialogue data and encyclopedic knowledge data to build required training data and construct a pseudo-data algorithm;

[0052] A training module, which performs three-stage training on the BERT model by using the training set constructed by the construction module;

[0053] The test module uses the sum of the three-stage scores generated by the training module as the final score when testing.

[0054] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method as described above when executing the computer program.

[0055] A computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the method as described above.

[0056] The beneficial effects of the present application are:

[0057] The present application uses knowledge items as a bridge to jointly model the knowledge screening and reply screening two tasks, the knowledge items need to have high relevance to the dialogue context and the candidate reply at the same time, which improves the accuracy of knowledge screening;The reply screening and knowledge screening are carried out at the same time, which enhances the relevance between the two tasks of knowledge screening and reply screening, so as to improve the consistency of the knowledge screening result and the reply screening result.

[0058] Compared with the existing method, the present application can improve the accuracy of knowledge screening by more than 5%, and improve the consistency of knowledge screening and reply screening results by more than 6% using the same size model on three public standard evaluation data sets. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a schematic diagram of the existing dialogue-centered retrieval mode;

[0060] Figure 2 is a schematic diagram of the existing reply-centered retrieval mode;

[0061] Figure 3 is a schematic diagram of the knowledge-centered retrieval mode of the present application;

[0062] Figure 4 is a schematic diagram of the dynamic multi-element matching method of the present application. DETAILED DESCRIPTION

[0063] In the following description, specific details are set forth such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art should understand that the present application can be implemented in other embodiments without these specific details. In other cases, well-known systems, devices, circuits, and methods have not been described in detail in order not to obscure the description of the present application with unnecessary detail.

[0064] It should be understood that the term "include" as used in this specification and the following claims indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0065] It should also be understood that the terminology used in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this specification and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0066] The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0067] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details that are set forth in the description, and it is understood that persons having ordinary skill in the art can make and use other implementations of the present application without departing from the scope of the present application. Accordingly, the present application is not limited to the embodiments described herein.

[0068] Embodiment one

[0069] The embodiment provides a domain open dialogue field knowledge-centered reply screening method, and has the characteristics that the reply screening method comprises the following steps:

[0070] Step 1, constructing required training data by using existing dialogue data and encyclopedic knowledge data, and constructing a pseudo-data algorithm, and the specific algorithm is shown in Table 2;

[0071] Step 2, using the training set constructed in step 1 to perform three-stage training on the BERT model, and a training schematic diagram is shown in FIG. 2; Figure 4

[0072] Step 3, using the sum of the scores of the three stages in step 2 as the final score during testing.

[0073] Further, the step 1 of constructing required training data specifically comprises: establishing mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) according to dialogue data in the data set ζ, and then constructing a training set according to the established mapping relationships;

[0074] ​The mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) are established based on the conversation data in the data set ζ as follows:

[0075] Dataset {D i , C i , R i ,y i} is the i-th conversation data in the dataset ζ, N represents the total number of conversation data in the dataset ζ, where D i Represents the document in the i-th conversation data, Represents the first knowledge entry in the document. Represents the second knowledge entry in the document. Indicates the dth i Knowledge items, d i Representative D i The number of knowledge items in C i Represents the conversation context excluding the reply in the i-th conversation data, Representative C i In the first round of dialogue, Representative C i In the second round of dialogue, Representative C i c i Round dialogue, c i Representative C i The number of rounds in the dialogue, R i For C i Reply, y i =1 represents R i C i The correct answer is y i =0 represents R i Not C i The correct response;

[0076] The dialogue above C i The last round of dialogue Defined as the query Q of the i-th conversation data i , the dialogue above C i The other rounds except the last round of dialogue are defined as the dialogue history of the i-th dialogue data Establish mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) according to the data set ζ;

[0077] Among them, Q is the query, K is the knowledge item, H is the conversation history, and R is the reply.

[0078] Further, the step 1 further comprises constructing a training set according to the established mapping relationship, specifically,

[0079] Step 1.1, the mapping relationship G1(Q, K) represents the relevance of the knowledge entry and the query, taking (Q, K) as the input of the training set, and taking the relevance score G1(Q, K) as the corresponding output;

[0080] Step 1.2, the mapping relationship G2(K, R) represents the relevance of the knowledge entry and the reply, taking (K, R) as the input of the training set, and taking the relevance score G2(K, R) as the corresponding output;

[0081] Step 1.3, the mapping relationship G3(K, H, Q, R) represents the relevance of the knowledge entry, the dialogue history, the query and the reply, taking (K, H, Q, R) as the input of the training set, and taking the relevance score G3(K, H, Q, R) as the corresponding output.

[0082] Further, in view of the problem that artificial annotation data is limited and expensive in actual application scenarios, a pseudo data annotation method is designed, two kinds of easily obtained data resources are used to construct pseudo data, and the pseudo data is used for knowledge-centered reply retrieval training, so as to verify the effect of the application in the actual application scenario when the labeled data is lacking. The specific implementation scheme is as follows:

[0083] Taking the construction of pseudo data in English as an example. First, a large amount of unlabeled dialogue data is selected, such as multi-turn dialogue data on English forums (Reddit), the last sentence of the multi-turn dialogue is taken as the reply, and the dialogue before the reply is taken as the dialogue context. Based on the automatic evaluation index BM25, external knowledge is added to each sentence of the reply, the length of the knowledge entry is selected from the English Wikipedia document, the knowledge entry is aligned with the reply. For different actual tasks, different lengths of knowledge entries and different threshold values of BM25 algorithm can be set. In this way, a triple (C, K, R) of dialogue context, external knowledge and reply is obtained, so that the dynamic multi-element matching method of the application can be used for model training.

[0084] The pseudo data algorithm constructed in the step 1 is specifically,

[0085] Input: Reddit data D R , Wikipedia data D w , threshold value γ;

[0086] Output: pseudo data DGD data Dp;

[0087] 1: initialize Dp as an empty set;

[0088] 2: for D REach group of conversation context C and reply R in the data is operated as follows (3-6 steps);

[0089] 3: Calculate the similarity between the reply and the Wikipedia data D using the BM25 algorithm w of each knowledge entry in the data D;

[0090] 4: If the similarity score is greater than the threshold value γ;

[0091] 5: Then the found knowledge entry is combined with C and R to form a group and put into Dp;

[0092] 6: After traversing all the data in D R , the pseudo data Dp is obtained.

[0093] Similarly, for Chinese data, conversation data without knowledge labels can be obtained from Chinese forums, and knowledge entries can be obtained from Chinese encyclopedic documents, so as to obtain Chinese pseudo-labeled data (C, K, R). A large number of experiments are carried out on the public evaluation set using the method of the application, which proves that the method is more effective than the existing method in the zero-resource / low-resource / full-data scenario. Zero resource is defined as not using real labeled training data for training, but only using pseudo-labeled data for training; low resource is defined as using only a small amount of real labeled data for training; full data is defined as using all available labeled data for training. After the training is completed, the above three cases are tested on the real test set.

[0094] When constructing pseudo data, the selected conversation data can be replaced with other publicly released conversation corpus, and the selected external knowledge can be replaced with other knowledge base containing unstructured knowledge entries. The automatic evaluation index (BM25) used to connect the reply with the external knowledge can be replaced with other automatic evaluation indexes or models capable of calculating the similarity between two text sequences.

[0095] Further, step 2 is specifically defined as defining segment embedding encoding for document D, conversation data C and reply R respectively, and the last layer hidden state representation of the special marker [CLS] located at the beginning of the input sequence is denoted as E cls , E cls After passing through two nonlinear layers, the output correlation score G(input) is:

[0096] G(input) = σ(W2μ(W1E cls +b1) + b2)

[0097] Where W1, W2, b1, b2 are trainable parameters, σ represents the sigmoid activation function, and μ represents the tanh activation function; the BERT baseThe input of BERT contains three different vector representations: word vectors, position vectors, and segment vectors. Unlike traditional BERT, three different segment embedding encodings are defined for D, C, and R to distinguish them as different parts. In the three correlation tasks, the same BERT model is used for training, and the correlation score is calculated using the above formula.

[0098] In the present application, BERT is used as the basic model for correlation modeling. Other alternative basic models include RoBERTa (Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs / 1907.11692 (2019).), ELECTRA (Clark, K.; Luong, M.; Le, Q. V.; and Manning, C. D. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In ICLR 2020.), and other mask rule-based language models. These models come in different sizes, such as BERT base with about 110M parameters, BERT large with about 340M parameters. They can all be trained using the method designed in the present application to obtain models with the same reply retrieval function.

[0099] Further, the loss function L final used in step 2 is:

[0100] L final = L1+L2+L3

[0101] Where:

[0102]

[0103] In the formula, e is the base of the natural logarithm, N1 represents the number of positive examples corresponding to the mapping relationship G1(Q, K), represents the correlation score of the i-th positive example, represents the correlation score of the i-th negative example, and a1 represents the number of corresponding negative examples for each positive example ​represents the relevance score of the jth negative example, j = 1, 2, ..., α1;

[0104]

[0105] Where N2 represents the number of positive examples corresponding to the mapping relationship G2(K, R), represents the relevance score of the i-th positive example, represents the relevance score of the i-th negative example, and α2 represents the relevance score of each positive example The corresponding number of negative examples, represents the relevance score of the jth negative example, j = 1, 2, ..., α2;

[0106]

[0107] Where N3 represents the number of positive examples corresponding to the mapping relationship G3(K, H, Q, R), represents the relevance score of the i-th positive example, and α3 represents the relevance score of each positive example The corresponding number of negative examples, Represents the relevance score of the jth negative example, j = 1, 2, ..., α3: λ represents the different scores assigned to each negative example, and the values ​​of different negative examples are as follows Figure 4 As shown in:

[0108] By jointly training the three subtasks, the model can effectively model the relationship between external knowledge and the conversation context and responses: Figure 4 All negative examples in are randomly determined and dynamically constructed during the training process: therefore, the method of the present invention is called a dynamic multi-element matching method.

[0109] Furthermore, the step 3 specifically includes screening the knowledge items before constructing the input corresponding to the three mapping relationships in step 1, and screening out the knowledge items ranked in the top m positions;

[0110] When the number of knowledge items is large, the knowledge items are combined. That is, when screening knowledge, the knowledge ranked in the top m is screened first, and then the three scores are calculated. The m value is selected within the allowable error range and varies according to different tasks. The m value remains consistent during training and testing.

[0111] Specifically, the application proposes a dynamic multi-element matching method to train a knowledge-centered reply retrieval model, which simultaneously models the relationships among dialogue context-knowledge, knowledge-reply and dialogue context-knowledge-reply in a unified framework, so that the framework has the ability to screen knowledge and reply based on knowledge. The model learns a mapping relationship G(C, D, R) for a new sample (D, C, R) that does not appear in the data set, and the mapping relationship G can predict whether R is the correct reply to C.

[0112] Embodiment two

[0113] This embodiment uses the method as in embodiment one, which is a further limitation of embodiment one,

[0114] In step 1, the relevance score of the positive example (Q + , K) is 1, Q + represents a query matched with K;

[0115] The relevance score of the negative example (Q - , K) is 0, Q - represents a query not matched with K.

[0116] Q - is randomly sampled from other dialogue samples in the data set ζ.

[0117] In step 2, the relevance score of the positive example (K, R + ) is 1, R + is the corresponding original reply R in the dialogue data, i.e. R + is a reply matched with K;

[0118] The relevance score of the negative example (K, R - ) is 0, R - is a reply not matched with K.

[0119] R - is randomly sampled from other dialogue data in the data set ζ.

[0120] In step 3, the relevance score of the mutually matched original dialogue data (K, H + , Q + , R + ) as the positive example is 1, H + represents that the dialogue history, the knowledge item, the query and the reply are matched;

[0121] The relevance score of the negative example (K, H - , Q + , R + ) is 0.9, H -representing a dialogue history that does not match a knowledge item, query, or reply;

[0122] negative example (K, H + , Q - , R + ) has a relevance score of 0.1, Q - representing a query that does not match a dialogue history, knowledge item, or reply;

[0123] negative example (K, H - , Q - , R + ) has a relevance score of 0;

[0124] negative example (K, H + , Q + , R - ) has a relevance score of 0, representing a reply that does not match a dialogue history, knowledge item, or query;

[0125] negative example (masked, H + , Q + , R + ) has a relevance score of 0.5, masked representing a query that uses special symbols <mask>instead of each word in the knowledge entry;

[0126] The relevance score for the negative example (masked, H + , Q + , R - ) is 0.

[0127] Each word in the external knowledge entry is replaced with a special symbol <mask>Instead, the model learns the importance of external knowledge for relevance matching.

[0128] Embodiment Three

[0129] This embodiment uses the method as described in Embodiments One to Two, specifically,

[0130] The present application can be directly applied to an open-domain chatbot system, and is a core module of a chatbot. The central control module of the chatbot system transmits the dialogue context to the module where the present application is located. The module where the present application is located screens the most suitable item from a large number of knowledge items searched by the dialogue context on a local server or network, and selects the most suitable reply from a large number of candidate replies using the search result, and transmits the reply to the central control module to complete an open-domain dialogue.

[0131] In terms of deployment, the technology can be independently used as a computing node and deployed on a cloud computing platform such as Ali Cloud or Meituan Cloud. Communication between other modules can be performed by binding an IP address and a port number.

[0132] In terms of specific implementation, the technology uses the deep learning open source framework Pytorch. If necessary, other frameworks such as the open source tensorflow or PadlePadle can be used.

[0133] Experimental Part

[0134] In terms of test data, the data shown in Table 3 is used. Among them, LIGHT (Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim External knowledge used by LIGHT (Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to Speak and Act in a Fantasy Text Adventure Game. EMNLP-IJCNLP 2019.) and CMU_DoG (Kangyan Zhou, Shrimai Prabhumoye, and Alan W. Black. A Dataset for Document Grounded Conversations. EMNLP 2018.) are documents composed of multiple sentences without knowledge labels, part of the test data of LIGHT is manually annotated; external knowledge used by WoW (Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR, 2019.) and WISE (Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zhumin Chen, Zhaochun Ren, and Maarten de Rijke. Wizard of Search Engine: Access to Information Through Conversations with Search Engines. In SIGIR 2021.) are knowledge entries with knowledge labels. Except that WISE is in Chinese, the rest of the data is in English.

[0135] Table 3: Statistics of test datasets

[0136]

[0137]

[0138] On the above public datasets, using the same size of model (BERT-base), compare the accuracy of knowledge filtering with existing methods, the experimental results on LIGHT, WoW, WISE datasets (the average number of candidate knowledge entries on the three datasets are 15, 61, 245 respectively) are shown in Table 4:

[0139] Table 4: Knowledge screening effect on test set

[0140]

[0141] Zero resource, low resource, full data respectively use 0%, 10%, 100% of the data of the row data set for training, and the WoW and WISE data sets have labeled knowledge labels for training data. Since the training data of LIGHT does not have knowledge labels, only zero-resource test experiments are performed on the labeled test set.

[0142] On the above disclosed data sets, the same size model (BERT-base) is used to compare the reply screening accuracy with the existing method. The experimental results on the LIGHT, CMU_DoG, WoW, and WISE data sets are shown in Table 5 (the number of candidate replies on the four data sets is 20, 100, 20, and 20, respectively):

[0143] Table 5: Reply screening effect on test set

[0144]

[0145]

[0146] In order to verify whether the present application better solves the inconsistency problem of screening knowledge and replies, the proportion of simultaneously selecting correct knowledge and correct replies in the WoW test set is compared, as shown in Table 6:

[0147] Table 6: Proportion of simultaneously selecting correct knowledge and correct replies on the WoW test set

[0148]

[0149] The experimental results of Table 4, Table 5 and Table 6 prove that, under fair comparison, the method proposed in the present application has obvious improvement in the accuracy of knowledge screening and reply screening under the conditions of zero resource, low resource and full data, among which, in knowledge screening, the average improvement is 5 percentage points; in reply screening, the average improvement is 4 percentage points; in the consistency of reply and knowledge screening results, the average improvement is 6 percentage points. It is proved that the method of the present application is universal and can be transferred between different fields.

[0150] Example Four

[0151] The present embodiment provides an open domain dialogue field knowledge-centered reply screening system using the method as described in Example One, which comprises a construction module, a training module and a test module.

[0152] The construction module uses existing dialogue data and encyclopedic knowledge data to construct required training data, and constructs a pseudo data algorithm, and the specific algorithm is shown in Table 2.

[0153] The training module uses the training set constructed by the construction module to perform three-stage training on the BERT model, and a training schematic diagram is shown in Figure 2. Figure 4

[0154] The test module uses the sum of the three-stage scores generated by the training module as the final score during testing.

[0155] The construction module uses the method in Embodiment One for construction.

[0156] The training module uses the method in Embodiment One for training.

[0157] The test module uses the method in Embodiment One for testing.

[0158] Embodiment Five

[0159] The electronic device provided in the embodiments of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor performs various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected through a bus. Specifically, the processor realizes any step in Embodiment One by running the computer program stored in the memory.

[0160] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0161] The memory can include read-only memory, flash memory, and random access memory, and provide instructions and data to the processor. Part or all of the memory can also include non-volatile random access memory.

[0162] ​As can be seen from the above, the electronic device provided by the embodiment of the present application can realize the reply screening method as described in the embodiment one by running the computer program, the model simultaneously models the connections of the dialogue context-knowledge, knowledge-reply and dialogue context-knowledge-reply three combinations in a unified framework, so that the framework has the ability of knowledge screening and reply screening based on knowledge. The model learns a mapping relationship G(C, D, R), and for a new sample (D, C, R) that does not appear in the data set, the mapping relationship G can predict whether R is the correct reply of C.

[0163] It should be understood that the integrated modules / units described above, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, and can also be completed by instructing related hardware through a computer program. The above-mentioned computer program can be stored in a computer readable storage medium, and when the processor executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The above-mentioned computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the above-mentioned computer program code. It should be noted that the content contained in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0164] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0166] It should be noted that the method and its details provided by the above-mentioned embodiments can be combined into the device and equipment provided by the embodiments, and mutual reference will not be described here.

[0167] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0168] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the above-mentioned device / device embodiment is only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0169] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.< / mask> < / mask>

Claims

1. A knowledge-centric reply screening method for open-domain dialogue, characterized in that: The reply screening method comprises the following steps: Step 1: Use existing conversation data and encyclopedia knowledge data to build the required training data and construct a pseudo-data algorithm; Step 2: Use the training set constructed in step 1 to perform three-stage training on the BERT model; Step 3: During the test, use the sum of the scores from the three stages of Step 2 as the final score; The step 1 of constructing the required training data is specifically to establish mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) based on the dialogue data in the dataset ζ. Specifically, Dataset is the i-th conversation data in the dataset ζ, N represents the total number of conversation data in the dataset ζ, where D i Represents the document in the i-th conversation data, Represents the first knowledge entry in the document. Represents the second knowledge entry in the document. Indicates the dth i Knowledge items, d i Representative D i The number of knowledge items in C i Represents the conversation context excluding the reply in the i-th conversation data, Representative C i In the first round of dialogue, Representative C i In the second round of dialogue, Representative C i c i Round dialogue, c i Representative C i The number of rounds in the dialogue, R i For C i Reply, y i =1 represents R i C i The correct answer is y i =0 represents R i Not C i The correct response; The dialogue above C i The last round of dialogue Defined as the query Q of the i-th conversation data i , change the dialogue C i The other rounds except the last round of dialogue are defined as the dialogue history of the i-th dialogue data Establish mapping relationships G1(Q, K), G2(K, R) and G3(K, H, Q, R) according to the data set ζ; Among them, Q is the query, K is the knowledge item, H is the conversation history, and R is the reply; The step 1 also includes constructing a training set according to the established mapping relationship, specifically, Step 1.1, the mapping relationship G1(Q, K) represents the relevance between the knowledge item and the query, (Q, K) is used as the input of the training set, and the relevance score G1(Q, K) is used as the corresponding output; Step 1.2: The mapping relationship G2(K, R) represents the relevance between the knowledge item and the response. (K, R) is used as the input of the training set, and the relevance score G2(K, R) is used as the corresponding output. Step 1.3: The mapping relationship G3(K, H, Q, R) represents the relevance between knowledge items, conversation history, and query and reply. (K, H, Q, R) is used as the input of the training set, and the relevance score G3(K, H, Q, R) is used as the corresponding output. The specific algorithm for constructing pseudo data in step 1 is: Input: Reddit data D R Wikipedia data D w ;Threshold γ; Output: pseudo DGD data Dp; 1: Initialize Dp to an empty set; 2: To D R For each conversation in the data, with context C and response R, perform operations 3-6. 3: Use BM25 algorithm to calculate the response and Wikipedia data D w The similarity of each knowledge item in 4: If the similarity score is greater than the threshold γ; 5: Then group the found knowledge items with C and R and put them into Dp; 6: After traversing D R After the data in, the pseudo data Dp is obtained; Specifically, step 2 is to define segment embedding codes for document D, conversation data C, and reply R, respectively, and denote the last hidden state of the special identifier [CLS] at the beginning of the input sequence as E cls , E cls After two nonlinear layers, the output correlation score G(input) is: G(input)=σ(W2μ(W1E cls +b1)+b2) Among them, W1, W2, b1, b2 are trainable parameters, σ represents the sigmoid activation function, and μ represents the tanh activation function. In the three relevance tasks, the same BERT model is used for training, and the relevance scores are calculated using the above formula.

2. The reply screening method according to claim 1, wherein: The loss function L used in step 2 is final for: L final =L1+L2+L3 in: Where e is the base of the natural logarithm, N1 represents the number of positive examples corresponding to the mapping relationship G1(Q, K), represents the relevance score of the i-th positive example, represents the relevance score of the i-th negative example, and α1 represents the relevance score of each positive example The corresponding number of negative examples, represents the relevance score of the jth negative example, j = 1, 2, ..., α1; Where N2 represents the number of positive examples corresponding to the mapping relationship G2(K, R), represents the relevance score of the i-th positive example, represents the relevance score of the i-th negative example, and α2 represents the relevance score of each positive example The corresponding number of negative examples, represents the relevance score of the jth negative example, j = 1, 2, ..., α2; Where N3 represents the number of positive examples corresponding to the mapping relationship G3(K, H, Q, R), represents the relevance score of the i-th positive example, and α3 represents the relevance score of each positive example The corresponding number of negative examples, represents the relevance score of the jth negative example, j = 1, 2, ..., α3: λ represents the different scores assigned to each negative example.

3. The reply screening method according to claim 1, wherein: Specifically, step 3 comprises constructing inputs corresponding to the three mapping relationships in step 1 based on the current conversation, knowledge item, and candidate response, and inputting the constructed inputs into the trained BERT model to obtain three relevance scores output by the BERT model. Sum the three relevance scores and the result is the score of the candidate response; After traversing each candidate response, the candidate response with the highest score is taken as the final response.

4. A knowledge-centric response screening system for open-domain dialogue, characterized by: The reply screening system uses the reply screening method according to claim 1, and the reply screening system includes: The construction module uses existing conversation data and encyclopedia knowledge data to build the required training data and construct pseudo-data algorithms; The training module uses the training set constructed by the construction module to perform three-stage training on the BERT model; The testing module, during testing, uses the sum of the three-stage scores generated by the training module as the final score.

5. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images