Open domain dialogue generation method, device, equipment and medium based on hard selective context application

Abstract

The present invention discloses an open domain dialogue generation method, device, equipment and medium based on hard selective context application, the method includes: recording the latest utterance in the historical dialogue as a query, and recording the remaining utterances as context; dividing the query into a strong query or a weak query according to the information density; using different response generation modules to generate responses for the strong and weak queries respectively; the strong query response generation module is constructed based on the "encoding-decoding" framework; in the weak query response generation module, firstly, the context and the weak query are encoded respectively, and then the semantic interaction module is used to fuse the context and the semantics of the weak query to obtain a semantic interaction vector, and the weak query decoder is used to generate a response according to the semantic interaction vector; the weak query response generation module adopts a multi-task learning framework based on remote supervision of co-reference relationship. The present invention can perform flexible and accurate selective context application according to the specific situation of the historical dialogue, so as to generate a response.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent dialogue data processing, and in particular relates to an open domain dialogue generation method, device, equipment and medium based on hard selective context application. Background Art

[0002] Open-domain dialogue systems are able to have natural conversations with humans on a variety of topics. Their purpose is to maximize user engagement through anthropomorphic interaction, and then build long-term and stable connections with users. Dialogue is a spoken text that is generated by a dynamic information exchange process between multiple interlocutors. Unlike monologue texts that maintain a high degree of focus on the topic and a rigorous language style, dialogues are more casual in expression. Although there is a relatively stable dialogue background on a global scale, the focus of ongoing local topics is dynamically changing. This results in the semantics of discourse in open-domain dialogue systems being heavily dependent on the context environment, so how to properly use the dialogue context is the key to generating high-quality responses.

[0003] Existing technical solutions for using conversation context can be divided into two categories:

[0004] (1) Non-Selective Context Utilization (NonSCU). NonSCU methods treat the dialogue context as a monologue text, sequentially connect all context utterances to form a unified text sequence, and then generate responses through a traditional neural network-based encoding and decoding framework, or select responses from a candidate library through semantic matching. This type of method is common in early open-domain dialogue system research and response generation methods based on large-scale pre-trained models.

[0005] (2) Selective Context Utilization (SCU). SCU methods encode the conversation context as multiple independent utterances or extract key words from the context, and then design various weighted fusion strategies to semantically fuse multiple utterances or key words. The key to the fusion process is to highlight the role of those utterances or topics that are more relevant to the response to be generated. Currently, most open domain dialogue generation methods adopt the SCU approach.

[0006] The essence of NonSCU methods is to use the idea of ​​single-round dialogue generation to handle multi-round dialogues. The disadvantages are: on the one hand, the dynamic semantic changes across multiple rounds of interaction in open-domain dialogues are ignored; on the other hand, the context is often a long text after splicing, and the traditional encoding and decoding framework has the problem of long-term memory dissipation. Although the rich language knowledge in the pre-trained model helps to improve the long text encoding ability of NonSCU methods, ignoring the semantic changes in the context will still cause the model to be unable to capture the current focus of the dialogue. This disadvantage cannot be compensated even by large-scale pre-training.

[0007] Existing SCU-type methods essentially assume that all conversation contexts are necessary for response generation, and rely on weighted fusion of context semantic representations to generate responses. This idea is "soft selective context application" (SoftSCU), and its disadvantage is that it cannot really filter out the noise in the context. Irrelevant words may directly overwhelm relevant words, resulting in irrelevant responses. Summary of the invention

[0008] The present invention provides an open domain dialogue generation method, device, equipment and medium based on hard selective context application, which can flexibly, accurately and effectively generate responses according to historical dialogues.

[0009] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0010] A method for generating open-domain dialogues based on hard selective context application, comprising:

[0011] The latest utterance in the historical conversation is recorded as the query, and the remaining utterances are recorded as the context;

[0012] If the information density of the query exceeds a preset density value, the query is marked as a strong query, otherwise it is marked as a weak query; the preset density value can be set to the average query information density of the data set;

[0013] For strong queries, an "encoder-decoder" architecture is used to generate responses based on the strong queries; the "encoder-decoder" architecture can be constructed using a recurrent neural network or a generative pre-trained model T5;

[0014] For weak queries, a weak query response generation module is used to generate responses, and the weak query response generation module includes: a context encoder, a weak query encoder, a semantic interaction module and a response decoder for weak queries; the specific process of response generation is: first, the context encoder and the weak query encoder are used to encode the context and the weak query respectively to obtain the context semantic representation and the weak query semantic representation; then, the semantic interaction module is used to perform semantic interaction fusion on the context semantic representation and the weak query semantic representation to obtain a semantic interaction vector; finally, the weak query response decoder is used to generate responses according to the semantic interaction vector.

[0015] Furthermore, the semantic interaction module includes a self-attention mechanism layer, an interactive attention mechanism layer and a residual connection layer; the context semantics and weak query semantics are respectively enhanced internally through a self-attention mechanism layer; then the interactive attention mechanism layer is used to supplement and enhance the weak query semantic representation using the context semantic representation; and then the residual connection layer is used to perform a residual connection between the weak query semantic representation and the vector output by the interactive attention mechanism layer to obtain a semantic interaction vector.

[0016] Furthermore, the model parameters of the weak query response generation module are obtained by training using a multi-task learning framework based on co-reference relationship remote supervision; the multi-task learning framework based on co-reference relationship remote supervision includes the weak query response generation module and the context selection module; the multi-task learning framework includes:

[0017] Obtain “context-query-response” triple samples based on conversation data;

[0018] For each triple sample, the true label selected by the context is determined based on whether there is a co-reference relationship between each utterance in the context and the query;

[0019] The context and query in the triple sample are used as the input of the context encoder and the weak query encoder respectively; the outputs of the context encoder and the weak query encoder are both used as the input of the semantic interaction module; the semantic interaction vector output by the semantic interaction module is used as the input of the response decoder and the context selection module at the same time; finally, the loss of the weak query decoder is calculated according to the predicted output of the weak query response decoder and the true response, the loss of the context selection module is calculated according to the predicted output of the context selection module and the true label of the context selected, and the sum of the loss of the weak query response decoder and the loss of the context selection module is taken as the total loss to train the multi-task learning framework.

[0020] Furthermore, the loss calculation function of the weak query response decoder is:

[0021]

[0022] in, The loss of the decoder for weak query responses; For the real response U m+1 The length of p(w n ) is the true response U m+1 The unique word vector at the nth position has the dimension of the entire vocabulary size |V|; p(y n ) is the probability of the word distribution at the nth position predicted by the weak query decoder, and the dimension is also |V|;

[0023] The loss calculation function of the context selection module is:

[0024]

[0025] in, is the loss of the context selection module; m is the number of utterances included in the historical dialogue, m-1 is the number of utterances included in the context; α is the true label selected by the context, Predict the context selection probability of the output for the context selection module.

[0026] Furthermore, the context selection module is implemented using a multi-layer perceptron.

[0027] Furthermore, the true label of the selected context is determined by the following method: first, the context C and query X in the triple data sample are concatenated into a text sequence to form a monologue text; second, the co-reference resolution model of AllenNLP open source is used to resolve the monologue text and extract the co-reference relationship and the co-reference parties; third, the query X and each utterance U in the context C are found. i (U i ∈C, i=1,...,m-1) whether there is a coreference relationship, if so, the utterance U i Selected tag α i Marked as 1, i.e. α i =1; otherwise marked as 0, i.e. α i =0; Finally, the label values ​​of all utterances in context C constitute the true label of the context selected α = {α1,...,α m-1}.

[0028] Furthermore, the information density refers to the number of words contained in the discourse, or refers to the average word-level information entropy of the discourse; wherein the calculation formula for the average word-level information entropy of the discourse is:

[0029]

[0030] Where U represents the utterance for which the average word-level information entropy is to be calculated, H(U) represents the average word-level information entropy of utterance U, and w nrepresents the nth word of discourse U, N represents the number of words contained in discourse U; p(w n ) represents word w n The distribution probability in the dialogue corpus is expressed by w n The frequency of occurrence in the corpus is approximately calculated.

[0031] An open domain dialogue generation device based on hard selective context application, comprising:

[0032] An information density evaluation module is used to compare the information density of a query with the average query information density of the data set, and mark queries that exceed the average query information density of the data set as strong queries, and mark queries that do not exceed the average query information density of the data set as weak queries; wherein the query refers to the latest utterance in the historical dialogue used for dialogue generation, and other utterances are contexts; the preset density value can be set to the average query information density of the data set;

[0033] Strong query encoder and decoder, used to generate responses based on strong queries;

[0034] A weak query response generation module is used to generate responses according to weak queries; wherein, the weak query response generation module includes: a context encoder, a weak query encoder, a semantic interaction module and a weak query decoder; the specific process of response generation is: first, use the context encoder and the weak query encoder to encode the context and the weak query respectively to obtain context semantics and weak query semantics; then use the semantic interaction module to perform semantic interaction on the context semantics and the weak query semantics to obtain a semantic interaction vector; finally, use the weak query decoder to generate a response according to the semantic interaction vector.

[0035] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements any of the above-mentioned methods for generating an open domain dialogue based on hard selective context application.

[0036] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for generating an open domain dialogue based on hard selective context utilization.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention first divides the latest utterance in the historical conversation into strong queries or weak queries according to the information density; if it is a strong query, it means that it contains enough information for response generation. At this time, the context can be ignored and the latest utterance can be directly input into the traditional "encoding-decoding" framework for response generation, thereby avoiding the noise introduced by the context; if it is a weak query, it means that the amount of information contained in it for response generation is insufficient and it is necessary to review the content in the context. Therefore, a semantic interaction module is used to selectively inject context semantics into the semantics of the latest utterance, and then a response is generated according to the obtained semantic interaction vector, thereby improving the accuracy of response generation.

[0039] Moreover, in the present application, the network parameters in the weak query response generation module are obtained by training using a multi-task learning framework based on remote supervision of co-reference relationships. The semantic interaction vector output by the semantic interaction module is input into the context selection module, and the context selection real label calculation loss is obtained based on the co-reference relationship between the context and the query. The loss and the loss of the weak query response generation module together constitute the loss function of the multi-task learning framework training, thereby optimizing the semantic interaction vector input to the weak query response decoder, so that the weak query response decoder can generate responses and outputs more accurately and effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a framework diagram of the method described in the embodiment of the present application;

[0041] Figure 2 It is a framework diagram of the weak query response generation module part in the method described in the embodiment of the present application;

[0042] Figure 3 is the average query information density of the two data sets provided in the experiment described in the embodiment of the present application;

[0043] Figure 4 It is a performance comparison of the response generation of each model of the DailyDialog dataset in the experiment described in the embodiment of the present application;

[0044] Figure 5 It is a performance comparison of each model of the PersonaChat dataset in response generation in the experiment described in the embodiment of the present application;

[0045] Figure 6 This is Example 1 of generating responses of each model in the experiment described in the embodiment of the present application;

[0046] Figure 7 This is Example 2 of generating responses of each model in the experiment described in the embodiments of the present application. DETAILED DESCRIPTION

[0047] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0048] In real-world conversations, generally speaking, when the other party's current words are difficult to understand independently, or are vague or ambiguous, we need to review the previous conversation content and use it as background information to supplement the other party's current words. If the other party's current words are semantically independent, contain sufficient information, or are even a question that is very clear in terms of response orientation, then reviewing the history may not be necessary.

[0049] Based on the reflection on the real-world dialogue process and the shortcomings of existing context application research, the present invention provides an open domain dialogue generation method based on Hard Selective Context Application (HardSCU). The basic idea of ​​Hard Selective Context Application (HardSCU) is that dialogue is a process in which subsequent utterances respond to preceding utterances, and the essence of response generation is feedback on the user's latest utterance. Therefore, how to use the context should be based on the user's latest utterance.

[0050] like Figure 1 As shown, the HardSCU model in the present invention first defines and measures the information density of the latest utterance (hereinafter referred to as query) in the user's historical conversation. If the information density of the query exceeds the overall average of all queries in the data set, it is marked as a "strong query", otherwise it is marked as a "weak query". For strong queries, the encoding and decoding framework is directly input for response generation. At this time, the model chooses not to use the context; for weak queries, context representation and query representation are obtained on the basis of independent encoding, and an interaction vector is obtained through a semantic interaction module, and response generation is performed based on this. In order to further reflect the "selectivity", the semantic interaction vector is used as input, and a context selection module based on a deep network is used to predict the probability of each context utterance being selected. At the same time, the coreference resolution model is used to extract the coreference relationship between the context and the query, form a remote supervision label, and promote the learning of the context selection module, thereby reversely optimizing the representation of the semantic interaction vector, and then improving the accuracy of weak query response generation.

[0051] The technical solution of the present invention is explained in detail below.

[0052] 1. Formal Definition

[0053] The essence of the conversation context application task is to understand the conversation history semantically, extract the global topic background and local topic focus of the conversation, and then generate responses. The evaluation of the quality of the context application model focuses on response generation. Therefore, the formal description is as follows:

[0054] Given a dialogue D = {U1,...,U M}(M≥2), in the mth round (m∈[1,M)), define C={U1,...,U m-1} is the context, X=U m For query, U m+1 For the response, when m=1, C=X. The context application model generates a response Y based on the semantics of C and X, with the goal of making Y as close as possible to the true response U m+1 .

[0055] 2. Information density of the query

[0056] Information density is the basis for judging strong and weak queries, and its core is to measure the amount of information in the query. Given a query X = {x1,...,x NX}, where N X It represents the number of words contained, and its information density is defined as InfoD(X). Given a dialogue dataset, we first calculate the information density of each query corresponding to each sample, and then sum and average them to get the average query information density, which is recorded as

[0057] if Then X is determined to be a strong query;

[0058] if Then X is determined to be a weak query.

[0059] The query information density in the present invention may be measured by the length of the utterance or by the average word-level information entropy.

[0060] (1) Utterance length

[0061] Utterance length refers to the number of words contained in an utterance. The more words an utterance contains, the more information it contains. Using utterance length to measure information density is very intuitive and simple, but it also has its drawbacks. There are many high-frequency words in natural language, such as personal pronouns, quantitative pronouns, conjunctions, prepositions, etc. These words are necessary components of utterances and appear frequently in various conversations, but they are often difficult to carry effective semantics, especially unique semantics that help identify utterances. Therefore, sometimes there will be a situation where an utterance is long but the amount of information is not high, such as "I am sorry. I am not sure what you are saying."

[0062] (2) Average word-level information entropy

[0063] In response to the disadvantage of discourse length, the average word-level information entropy is used from the perspective of information theory to judge the information content of each word in the discourse by the distribution probability of each word in the corpus, and then the information density of the entire discourse is obtained by averaging.

[0064] Given a utterance U = {w1,...,w N}, its average word-level information entropy H(U) is calculated as follows:

[0065]

[0066] Where p(w n ) represents word w n The distribution probability in the dialogue corpus can be obtained by w n The frequency of occurrence in the corpus is approximated.

[0067] 3. Response generator for “strong query samples”

[0068] like Figure 1 As shown, when the query utterance of the dialogue is judged to be a strong query, it is directly input into the traditional "encoding-decoding" framework for response generation without considering the context.

[0069] Given a strong query First, the encoder Encoder converts each word x of X n Convert to vector representation Then the decoder Decoder according to Generate response Y. The model loss function is the predicted response Y and the true response U m+1 The cross entropy between

[0070]

[0071] in For the real response U m+1 The length of p(w n ) is the true response U m+1 The unique word vector at the nth position has a dimension equal to the entire vocabulary size |V|. p(y n ) is the word distribution probability of the predicted response Y at the nth position, and the dimension is also |V|.

[0072] In the specific implementation of Encoder and Decoder, various models with text representation and generation can be used, such as recurrent neural network, Transformer, autoencoder, pre-trained model, etc. This technical solution provides two: one is based on bidirectional recurrent neural network (BiGRU); the other is based on T5 pre-trained model, using Huggingface's open source T5-base version as the pre-trained model. In the subsequent experimental phase, the performance of the two structures will be compared.

[0073] 4. Response generator for “weak query samples”

[0074] like Figure 2As shown in the figure, for dialogues with weak queries, we use semantic interaction to drive the query, integrate context-related semantics, and then generate responses. The semantic interaction between context and query is the key here. In order to optimize the representation ability of the semantic interaction vector, we also introduce an auxiliary learning task of context selection. On the remote supervision label based on co-reference relationship, we perform multi-task learning with response generation.

[0075] 4.1 Weak Query Encoder

[0076] Considering that weak queries are generally limited in length, their encoding is performed using a traditional recurrent neural network, specifically a bidirectional gated recurrent neural network BiGRU. The encoding process is formally defined as follows:

[0077]

[0078] in Is a weak query The initial word vector representation of . Indicates a weak query.

[0079] 4.2 Context Encoder

[0080] The context generally contains multiple utterances, and the utterances have a temporal order. Since further selective context application is to be implemented later, context encoding must be able to extract the semantic content of a single utterance on the one hand, and capture the temporal relationship in the process of dialogue formation on the other hand. Therefore, context encoding adopts the classic hierarchical encoding method based on recurrent neural networks, that is, first use a recurrent neural network to encode the utterances one by one, and then use another recurrent neural network to perform temporal integration on the independent representations of each utterance, and finally obtain the representation of each context utterance. The context encoding process is formally defined as follows:

[0081] Given a context C = {U1,...,U m-1}, use BiGRU to encode each utterance in C as follows:

[0082]

[0083] in It is the discourse i The nth word in The word vector of , 1≤n≤N i , N i For discourse U i The total number of words contained. is the coded word Considering the temporal encoding characteristics of recurrent neural networks, we set U i The vector representation of the last word in is the vector representation of the entire discourse, denoted as:

[0084]

[0085] Then, the time sequence of each discourse in C is integrated, and the formula is as follows:

[0086]

[0087] This gives the representation of context C

[0088] 4.3 “Weak Query-Context” Semantic Interaction Module

[0089] The core of semantic interaction is to imitate the human dialogue understanding process. When the weak query is insufficient in information or semantically ambiguous, relevant content is absorbed from the context to supplement it. Figure 2 As shown in the figure, the “weak query-context” semantic interaction module mainly includes three layers: self-attention mechanism layer, interactive attention mechanism layer and residual connection layer.

[0090] (1) Self-attention mechanism layer

[0091] The context representation and weak query representation are each enhanced through a self-attention mechanism layer to emphasize the relatively more important content in their respective semantics. The formal definition is as follows:

[0092]

[0093]

[0094] (2) Interactive Attention Mechanism Layer

[0095] Based on the output of the self-attention mechanism layer, the interactive attention mechanism layer uses contextual semantics to enhance the weak query semantics. The essence is to selectively inject contextual semantics into the weak query. The formal representation is as follows:

[0096]

[0097] (3) Residual Connection Layer

[0098] Based on the output of the interactive attention mechanism layer, the residual connection layer further emphasizes the original semantics of the weak query. The calculation formula is as follows:

[0099]

[0100] In summary, the input of the semantic interaction module is context C and weak query X, and the output is Defined as "Semantic Interaction Vector".

[0101] In the above calculation process, VecAdd represents the vector addition operation at the corresponding position. The specific implementation process of SelfAttention and InterAttention is as follows:

[0102]

[0103] head h =Attention(q h ,k h ,v h ),h=1,...,H,

[0104]

[0105] 4.4 Weak Query Response Decoder

[0106] The response decoder generates the response Y word by word in an autoregressive manner, and is implemented using recurrent decoding based on a gated recurrent neural network GRU and a common attention mechanism.

[0107] When generating the nth word, the decoder calculates the word distribution probability:

[0108]

[0109] in is the predicted word distribution probability, is the word predicted from the dictionary, W dec is the mapping matrix. is the hidden state of the decoder output at the nth position, and its calculation process is as follows:

[0110]

[0111] Yes word The word vector of n It can be understood as a high-level representation of the conversation, which is obtained through the attention mechanism. The formula is as follows:

[0112]

[0113] ρ n,j Represents the semantic interaction vector The importance of the nth word. tanh It is a fully connected network with tanh as the activation function.

[0114] The loss function of the weak query response generation module is defined as follows:

[0115]

[0116] In the above formula, p(yn ) is the probability of the word distribution at the nth position predicted by the weak query response decoder, that is,

[0117] 4.5 Context Selection

[0118] In order to further improve the representation ability and performance of the model in context selection, in addition to the main optimization goal of response generation, a module dedicated to context selection is also set up. As input, through a multi-layer nonlinear perceptron, the probability of each context utterance being selected is predicted. The calculation process is defined as follows:

[0119]

[0120]

[0121] Here, MLPs are composed of multiple nonlinear fully connected networks cascaded. in Represents the context utterance U i The probability of being selected.

[0122] 4.6 Multi-task Learning Framework Based on Co-reference Distant Supervision

[0123] A multi-task learning framework is adopted, and the optimization objective includes the loss generated by response generation and the loss of the additionally introduced context selection module.

[0124] (1) Construction of supervised labels based on coreference relations in dialogue history

[0125] The context selection probability is obtained by extracting the co-reference relationship between each utterance in the conversation history, especially between the query and the context. supervision labels.

[0126] First, the conversation history D = {C; X} (including context and query) is concatenated into a text sequence to form a common monologue text; secondly, the coreference resolution model of AllenNLP open source is used to resolve the coreference of the conversation history in the form of monologue text, and the coreference relationship and the coreference parties are extracted; thirdly, the coreference relationship between X and U is found. i (U i ∈C) whether there is a coreference relationship, if so, then U i Marked as 1, corresponding to the selection label α i =1; otherwise, marked as 0, α i =0.

[0127] α={α1,...,α m-1} will be used as context selection probability The true label is used to calculate the training loss and perform gradient backpropagation.

[0128] (2) Multi-task learning framework

[0129] Based on the multi-task learning model composed of the weak query response generation module and the context selection module, the total loss calculation formula of the training process is as follows:

[0130]

[0131] in The calculation is as follows:

[0132]

[0133] 5. Performance Testing

[0134] In order to verify the effectiveness of the present invention, performance tests were conducted on the present invention and typical prior art solutions on two recognized dialogue data sets to demonstrate the technical effects of the present invention.

[0135] 5.1 Experimental Setup

[0136] 5.1.1 Dataset and Preprocessing

[0137] This experiment selected two widely used datasets, DailyDialog and PersonaChat, for experimental verification, where:

[0138] DailyDialog contains about 10k conversations between people from daily life, covering different topics such as culture, travel, etc.

[0139] PersonaChat contains more than 12k character-conditioned conversations. Each conversation is produced by two crowdsourced workers assuming a predefined role.

[0140] Both datasets are chat corpora containing natural multi-round interactions between people. To some extent, compared with PersonaChat, the topics in DailyDialog are more diverse and the language style is closer to real life.

[0141] For the dataset, the official version of the training / test split is adopted. Given a dialogue sample {U1,...,U M}, construct the "context-query-response" triple data. If U is set m is a query, then {U1,...,U m-1} is the context, U m+1is the response, where m=1,...,M-1. When m=1, the query and context are the same. For triple data, the maximum number of utterances in the conversation context is set to 20, and the maximum number of words in each utterance is 50. Any sample exceeding the above limits will be truncated. Given the triple data "context-query-response", the information density of the query is calculated and compared with the average query information density in the training set. If it is greater than the average level, it is judged as a "strong query", otherwise it is a "weak query".

[0142] Figure 3 The table shown gives the average query information density of the two datasets.

[0143] 5.1.2 Comparison Models

[0144] The core of the present invention is the use of conversation context. In order to conduct a fair and comprehensive performance comparison, the following two types of baseline models are used, including NonSCU class and SoftSCU class.

[0145] (1) NonSCU: Non-selective contextual application model

[0146] The NonSCU model treats the conversation context as a monologue text, sequentially connects all context utterances to form a unified text sequence, and then generates a response through an "encode-decode" framework. The following two methods are selected as baselines:

[0147] T5: A pre-trained model based on the Transformer structure. It introduces a unified framework to convert most natural language tasks into text generation tasks. It only needs to add a prefix that indicates the nature of the downstream task, i.e. "response_generation", to the original dialogue input to fine-tune the downstream task. In the experimental comparison, the T5-base version open sourced by Huggingface was used for fine-tuning on the dialogue dataset.

[0148] DialoFlow: A pre-trained model specifically for dialogue based on the GPT structure. It introduces a dynamic flow mechanism to model the semantic structure in the dialogue history and designs three loss methods to promote context modeling and response generation through a multi-task learning framework. In the experimental comparison, the author's open source DialoFlow-base version was used for fine-tuning on the DailyDialog and PersonaChat datasets.

[0149] (2) SoftSCU: Soft Selective Context Utilization Model

[0150] The SoftSCU model assumes that all conversation context is necessary for response generation and relies on the weighted fusion of context semantic representation to generate responses. The following methods are selected as baselines

[0151] HRED: A classic soft selective context-exploitation method that sequentially integrates the conversation history to obtain context representation and then generates responses.

[0152] VHRED: A method based on HRED and latent variable model, which introduces latent variables in the response generation process, aiming to improve the diversity of generated responses.

[0153] SD-HRED: A model based on HRED and dynamic and static attention mechanisms. Static attention measures the importance of each utterance in the historical dialogue, and dynamic attention measures the importance of each utterance relative to the response.

[0154] HRAN: A hierarchical attention-based model that models the importance of each utterance in context and each word relative to the response separately.

[0155] TA-S2S: First, LDA is used to extract key topic words in the conversation history, and then they are input into the "encode-decode" framework together with the conversation history. During the generation process, a topic attention mechanism is designed to inject the semantics of topic words into the response generation.

[0156] The HardSCU model includes the following two models, the main difference is the choice of strong query response generator:

[0157] CorefSCU-R: A strong query response generator adopts an encoding-decoding framework based on a recurrent neural network.

[0158] CorefSCU-T: A strong query response generator using a T5-based encoding and decoding framework.

[0159] 5.1.3 Evaluation Metrics

[0160] (1) Response relevance

[0161] Response relevance refers to the semantic relevance of the generated response to the given conversation history. It is usually measured by comparing the similarity between the generated response and the real response. The more similar the two are, the better the relevance of the generated response. BLEU and topic similarity based on word vectors are mainly used to evaluate the relevance of the response.

[0162] BLEU. BLEU evaluates the N-gram overlap between the ground truth and the generated response, where N-gram represents the number of consecutive words. It has been widely used to evaluate models for response generation, question generation, and machine translation. Here, BLEU-2 is used as a representative.

[0163] · Topic similarity index based on word vectors. First, map each word in the response to the word vector space, then calculate the word vector representation of the entire response, and then calculate the cosine distance between the generated response and the real response in the word vector space to determine the semantic similarity between the two. Specifically, it includes Average, Extreme, and Greedy.

[0164] (2) Response information volume

[0165] The amount of response information refers to the amount of information carried by the response itself, which is mainly used to distinguish those responses that are common in conversations but have ambiguous semantics, such as "Yes", "OK", "I am not sure", etc. The information entropy of the discourse H is used. utt The binary version is used to measure the amount of response information, and the calculation formula is as follows:

[0166]

[0167] Where N is the number of words contained in the response, w n is the word in the response, H(w n ) is calculated as follows:

[0168] H(w n )=-p(w n |w n-1 )logp(w n |w n-1 ),

[0169] Where p(w n |w n-1 ) can be obtained by training the binary set {w n-1 ,w n} is approximated by the probability of occurrence.

[0170] (3) Response diversity

[0171] Distinct-2 is used to measure the percentage of different bigrams in the predicted response. The higher the value of Distinct-2, the more diverse the responses can be generated by the model.

[0172] 5.2 Experimental Results and Discussion

[0173] 5.2.1 Performance Analysis of HardSCU in Response Generation

[0174] First, we focus on the results on the DailyDialog dataset. Figure 4 The following figure shows the performance comparison of various models in response generation on the DailyDialog dataset.

[0175] (1) Baseline model performance

[0176] In terms of response relevance, HRAN achieves the best performance on Average, Extreme, Greedy, and BLEU-2, even better than the pre-trained models T5 and DialoFlow, which shows the important role of selective context in response generation. At the same time, TA-S2S outperforms DialoFlow in the relevance index. This further proves that there is noise in the conversation context. The practice of using all contexts without distinction can easily affect the relevance of the generated response. Even the pre-trained model with prior knowledge cannot make up for this negative impact. In the Soft-SCU type baseline model, some models perform well in response relevance (such as HRAN), while others are not ideal (such as SD-HRED). This uneven performance reflects the challenge faced by the Soft-SCU type method, that is, "What is the basis for weighted fusion context?" If this basis is not found well or the weighting process is unreasonable, the truly relevant context will be submerged by noise, and irrelevant responses will be generated. TA-S2S performs better than most Soft-SCU based on discourse semantics. This may be because: the topic is a high-order semantic condensed from the original discourse, and the condensation process is actually a process of filtering contextual noise. Therefore, it is more refined and accurate than the original semantics.

[0177] In the response information volume (H utt ), TA-S2S is the best baseline, which shows that topic information helps to generate more informative responses.

[0178] In terms of response diversity (Distinct-2), DialoFlow is the best baseline. Surprisingly, T5, which is also a pre-trained model, achieved the worst results in Distinct-2. Manually reviewing the responses generated by T5, it was found that T5 tends to generate similar responses for similar conversation histories. For example, if two conversations have exactly the same context but different query contents, T5 cannot recognize this difference and will automatically generate duplicate responses. DialoFlow, on the other hand, has targetedly modeled the context structure based on pre-training, improving the model's sensitivity to query discourse.

[0179] (2)HardSCU model

[0180] Compared with the baseline models, all HardSCU models outperform the best baseline (HRAN) on Average, Extreme, Greedy, and BLEU-2; uttOn , CorefSCU-R is the best model, and CorefSCU-T outperforms most baseline models; on Distinct-2, HardSCU has a very obvious advantage, with a significant improvement over the baseline. The above results prove that the HardSCU model can effectively improve the quality of responses and outperform the listed baselines in context application.

[0181] Let’s look at the results on the PersonaChat dataset. Figure 5 shown.

[0182] On the PersonaChat dataset, the best baselines and best models on various metrics are different. None of the models show a relatively stable and consistent advantage. Some models even have polarized performance. For example, VHRED has the highest H utt , but its score on Distinct-2 is the lowest. For example, the Average value of T5 is the highest among the baselines, but its score on H utt The top is the lowest.

[0183] Compared with the DailyDialog dataset, the performance of each model on the PersonaChat dataset has also changed. For example, HRAN is the best baseline on the DailyDialog dataset in terms of response relevance, but it does not show a clear advantage on the PersonaChat dataset.

[0184] In terms of response relevance, except for Average, all other indicators of the HardSCU model are lower than the best baseline; in terms of response information, some HardSCU models lose to the best baseline VHERD; in terms of response diversity, the HardSCU model shows a stable and overwhelming advantage.

[0185] Based on the above results, we can draw the following conclusions:

[0186] In the context application task, SCU methods are generally better than NoSCU methods. The prior knowledge learned by the pre-trained model on a large scale may help filter out some noise in the dialogue history, but this still cannot ensure the quality of its response generation. This also shows that adding a selective context application strategy to the pre-trained model may improve its performance in the dialogue generation task.

[0187] ·Soft-SCU methods show uneven performance, which shows that the approach of "weighted fusion context" is fundamentally effective, but it is difficult to determine the "weight calculation basis". Improper weighted fusion will cause the semantics of irrelevant context to directly cover the relevant one.

[0188] The Hard-SCU idea is effective in response generation tasks, especially in the DailyDialog dataset, where all the models in this chapter show obvious advantages. In terms of response diversity, they show a significant improvement over the baseline model across the two datasets.

[0189] 5.2.2 HardSCU response generation example

[0190] like Figure 6 In the example 1 of each model response generation, the query utterance contains two sentences with detailed content and rich information. From the perspective of human understanding, the query semantics are independent enough even without considering the context. In this case, the responses generated by the baseline model are relatively appropriate and can be used as feedback for a given conversation. Among them, the responses generated by T5 and TA-S2S are more informative and semantically clearer. The HardSCU model gave appropriate responses. It is worth mentioning that the responses generated by CorefSCU-T (entropy) and CorefSCU-R (entropy) are rich in information and have diverse language organization, while maintaining good consistency and resonance with the given conversation. CorefSCU-T (token) also introduces a new related word aperitif (appetizer). Overall, compared with the baseline model, the response generated by HardSCU has a higher overall quality. The query in this example is a typical strong query, which means that HardSCU does not use the context when generating the response, but still gives satisfactory results, which shows that the conversation context is not always necessary.

[0191] like Figure 7 In the example 2 of response generation of each model shown, the query is a weak query with relatively vague semantics and contains a reference relationship. In this case, the responses given by most baseline models are inappropriate. Among them, although T5 and DialoFLow can capture relevant words such as course and major, the overall consistency of the response with the given dialogue is low. Compared with the baseline model, since the given query is a weak query, HardSCU uses the CorefSCU model for response generation, so the results of CorefSCU-T and CorefSCU-R are the same. It can be seen that the responses they generate are very consistent with the real responses and accurately capture the relevant context. On the one hand, this example well illustrates the rationality of using co-reference relationships as context to select remote supervision labels; on the other hand, it also shows the accuracy of HardSCU in context application.

[0192] The solution of the present invention: 1) Focus on the utterance (query) just spoken by the user and use it to determine how to use the conversation context. 2) First, the information density of the utterance is defined, and two measurement methods based on the length of the utterance and the average word-level information entropy are given. The conversation is further divided into strong query samples or weak query samples according to the information density of the query. 3) For strong query samples, HardSCU directly inputs the strong query into the "encoding-decoding" framework for response generation. 4) For weak query samples, HardSCU enables the query to selectively introduce relevant semantics from the context to enrich itself, and uses the existing coreference relationship in the conversation as a remote supervision label to drive the model to mine the context selection relationship hidden in the conversation data. 5) In the HardSCU model, the so-called "hard style" means that the model directly judges "whether context is needed" based on the information density of the query, which is a clear and unambiguous 0 / 1 decision. The so-called "selective context application" actually has two meanings. One is to choose "use or not use context" according to the information density of the query, and the other is to select "relevant context" in weak query samples through the semantic interaction of context and query. 6) HardSCU is simple but effective and can be flexibly extended to assemble various powerful dialogue models to generate ideal responses.

[0193] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A method for generating open-domain dialogues based on hard selective context, characterized in that: include: The latest utterance in the historical conversation is recorded as the query, and the remaining utterances are recorded as the context; If the information density of the query exceeds the preset density value, the query is marked as a strong query, otherwise it is marked as a weak query; for strong queries, the "encoder-decoder" architecture is used to generate responses based on the strong queries; For weak queries, a weak query response generation module is used to generate responses, and the weak query response generation module includes: a context encoder, a weak query encoder, a semantic interaction module, and a response decoder for weak queries; the specific process of response generation is: first, the context encoder and the weak query encoder are used to encode the context and the weak query respectively to obtain the context semantic representation and the weak query semantic representation; then, the semantic interaction module is used to perform semantic interaction fusion on the context semantic representation and the weak query semantic representation to obtain a semantic interaction vector; finally, the response decoder for weak queries is used to generate responses according to the semantic interaction vector; The model parameters of the weak query response generation module are obtained by training using a multi-task learning framework based on co-reference relationship remote supervision; the multi-task learning framework based on co-reference relationship remote supervision includes the weak query response generation module and the context selection module; the multi-task learning framework includes: Obtain "context-query-response" triple samples based on conversation data; For each triple sample, the true label selected by the context is determined based on whether there is a co-reference relationship between each utterance in the context and the query; The context and query in the triple sample are used as the input of the context encoder and the weak query encoder respectively; the output of the context encoder and the weak query encoder are both used as the input of the semantic interaction module; the semantic interaction vector output by the semantic interaction module is used as the input of the response decoder and the context selection module at the same time; finally, the loss of the weak query response decoder is calculated based on the predicted output of the weak query response decoder and the true response, the loss of the context selection module is calculated based on the predicted output of the context selection module and the true label of the context selected, and the sum of the loss of the weak query response decoder and the loss of the context selection module is taken as the total loss to train the multi-task learning framework; The loss function of the weak query response decoder is calculated as: in, is the training loss for the weak query response decoder; For the real response U m+1 The length of p(w n ) is the true response U m+1 The unique word vector at the nth position has the dimension of the entire vocabulary size |V|; p(y n ) is the probability of the word distribution at the nth position of the weak query response decoder prediction output, and the dimension is also |V|; The loss calculation function of the context selection module is: in, is the training loss of the context selection module; m is the number of utterances in the historical dialogue, m-1 is the number of utterances in the context; α is the true label selected for the context, Predict the context selection probability of the output for the context selection module.

2. The method according to claim 1, characterized in that: The semantic interaction module includes a self-attention mechanism layer, an interactive attention mechanism layer and a residual connection layer; the context semantic representation and the weak query semantic representation are respectively enhanced internally by a self-attention mechanism layer; then the interactive attention mechanism layer is used to supplement and enhance the weak query semantic representation using the context semantic representation; and then the residual connection layer is used to perform a residual connection between the weak query semantics and the vector output by the interactive attention mechanism layer to obtain a semantic interaction vector.

3. The method according to claim 1, characterized in that The context selection module is implemented using a multi-layer perceptron.

4. The method according to claim 1, characterized in that The method used to determine the true label of the selected context is as follows: first, the context C and query X in the triple sample are concatenated into a text sequence to form a monologue text; second, the open-source co-reference resolution model of AllenNLP is used to resolve the monologue text and extract the co-reference relationship and the two parties; third, the query X is found to be related to each utterance U in the context C. i (U i ∈C, i=1,…,m-1) whether there is a coreference relationship, if so, the utterance U i Selected tag α i Marked as 1, i.e. α i =1; Otherwise it is marked as 0, i.e. α i =0; Finally, the label values ​​of all utterances in context C constitute the true label of the context selected α = {α1,...,α m-1 }.

5. The method according to claim 1, characterized in that The information density refers to the number of words contained in a discourse, or refers to the average word-level information entropy of a discourse; wherein the calculation formula for the average word-level information entropy of a discourse is: Where U represents the utterance for which the average word-level information entropy is to be calculated, H(U) represents the average word-level information entropy of utterance U, and w n represents the nth word of discourse U, N represents the number of words contained in discourse U; p(w n ) represents word w n The distribution probability in the dialogue corpus is expressed by w n The frequency of occurrence in the corpus is approximately calculated.

6. An open domain dialogue generation device based on hard selective context application, characterized in that: include: An information density evaluation module, used to compare the information density of a query with a preset density value, and mark queries exceeding the preset density value as strong queries, and mark queries not exceeding the preset density value as weak queries; wherein a query refers to the latest utterance in the historical dialogue used to generate a response, and other utterances are contexts; Strong query encoder and decoder, used to generate responses based on strong queries; A weak query response generation module is used to generate a response according to a weak query; wherein the weak query response generation module includes: a context encoder, a weak query encoder, a semantic interaction module and a weak query decoder; the specific process of response generation is: first, the context encoder and the weak query encoder are used to encode the context and the weak query respectively to obtain a context semantic representation and a weak query semantic representation; then, the semantic interaction module is used to perform semantic interaction fusion on the context semantic representation and the weak query semantic representation to obtain a semantic interaction vector; finally, the weak query decoder is used to generate a response according to the semantic interaction vector; The model parameters of the weak query response generation module are obtained by training using a multi-task learning framework based on co-reference relationship remote supervision; the multi-task learning framework based on co-reference relationship remote supervision includes the weak query response generation module and the context selection module; the multi-task learning framework includes: Obtain "context-query-response" triple samples based on conversation data; For each triple sample, the true label selected by the context is determined based on whether there is a co-reference relationship between each utterance in the context and the query; The context and query in the triple sample are used as the input of the context encoder and the weak query encoder respectively; the output of the context encoder and the weak query encoder are both used as the input of the semantic interaction module; the semantic interaction vector output by the semantic interaction module is used as the input of the response decoder and the context selection module at the same time; finally, the loss of the weak query response decoder is calculated based on the predicted output of the weak query response decoder and the true response, the loss of the context selection module is calculated based on the predicted output of the context selection module and the true label of the context selected, and the sum of the loss of the weak query response decoder and the loss of the context selection module is taken as the total loss to train the multi-task learning framework; The loss function of the weak query response decoder is calculated as: in, is the training loss for the weak query response decoder; For the real response U m+1 The length of p(w n ) is the true response U m+1 The unique word vector at the nth position has the dimension of the entire vocabulary size |V|; p(y n ) is the probability of the word distribution at the nth position of the weak query response decoder prediction output, and the dimension is also |V|; The loss calculation function of the context selection module is: in, is the training loss of the context selection module; m is the number of utterances in the historical dialogue, m-1 is the number of utterances in the context; α is the true label selected for the context, Predict the context selection probability of the output for the context selection module.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images