Emotion and knowledge-based reply generation method of double potential variables

Abstract

The invention discloses a reply generation method of double potential variables based on emotion and knowledge, and belongs to the technical field of dialogue systems. Comprising the following steps: constructing an emotional condition variation auto-encoder unit for generating emotional feature distribution; constructing a knowledge condition variation auto-encoder unit for generating knowledge feature distribution; a dialogue reply generation unit is constructed and used for generating a complete reply statement; the emotion condition variation self-encoder unit, the knowledge condition variation self-encoder unit and the dialogue reply generation unit are integrated into an EKDLVG reply generation model; a loss function is determined, parameters of the EKDLVG reply generation model are adjusted through the loss function until the loss function is convergent, and a final EKDLVG reply generation model is obtained; and replying a dialogue generation result obtained by the generation model by using the final EKDLVG. According to the invention, the emotion and knowledge expression consistency can be realized in the emotion-enhanced knowledge-driven dialogue generation model.

Description

Technical Field

[0001] The present invention relates to the technical field of dialogue systems, and more particularly to a response generation method based on dual latent variables of emotion and knowledge. Background Art

[0002] The goal of the open domain dialogue system is to improve the interactive experience and meet the potential needs of users.

[0003] However, dialogue systems tend to produce boring and bland responses. More and more researchers have begun to use external knowledge to enhance open-domain dialogue systems. Lowe et al. and Dodge et al. were the first to try to introduce external knowledge into dialogue systems and proposed knowledge-driven dialogue generation models. Knowledge-driven dialogue generation models aim to use different forms of knowledge to enhance dialogue systems to generate informative and meaningful responses. In the emotion-enhanced knowledge-driven dialogue generation model, response generation is also an important step in the emotion-enhanced knowledge-driven dialogue generation task. In the knowledge-driven dialogue generation model, in order to increase the diversity and information content of responses, introducing external knowledge for response generation has become a current research hotspot. Researchers have constructed a large number of knowledge graphs for this task and proposed a variety of dialogue generation models. There are two basic requirements for introducing external knowledge into response generation: first, the response needs to be consistent with the dialogue history and fluent; second, the generated response needs to accurately express the integrated external knowledge information. In the emotion-enhanced knowledge-driven dialogue generation model, it is necessary not only to consider incorporating knowledge into response generation, but also to consider incorporating both knowledge features and emotional features into response generation. When generating responses by combining emotions and knowledge, the imbalance between the fusion of emotional features and knowledge features causes the dialogue generation model to be too blunt or exaggerated in expressing emotions. The responses generated by the model lack realism and naturalness, and the expressions of emotions and knowledge are inconsistent. Figure 1 As shown in the figure, due to the unbalanced fusion of emotional features and knowledge features in the dialogue generation model, the model cannot correctly generate nostalgic replies, resulting in inconsistent expression of emotions and knowledge.

[0004] Therefore, how to integrate emotional features and knowledge features into dialogue generation in a balanced manner and generate responses that are consistent with emotional and knowledge expressions is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a response generation method based on dual latent variables of emotion and knowledge to solve the technical problems existing in the above-mentioned prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A response generation method based on dual latent variables of emotion and knowledge includes the following steps:

[0008] S100: Construct an emotion conditional variational autoencoder unit to generate emotion feature distribution;

[0009] S200: construct a knowledge conditional variational autoencoder unit to generate knowledge feature distribution;

[0010] S300: constructing a reply generation unit for a dialogue, for generating a complete reply sentence;

[0011] S400: Integrate the emotion conditional variational autoencoder unit, the knowledge conditional variational autoencoder unit, and the dialogue response generation unit into the EKDLVG response generation model;

[0012] S500: determining a loss function, and adjusting the parameters of the EKDLVG recovery generation model by using the loss function until the loss function converges, thereby obtaining a final EKDLVG recovery generation model;

[0013] S600: Dialogue generation result obtained using the final EKDLVG response generation model.

[0014] Furthermore, the emotion conditional variational autoencoder unit passes the emotion state through an emotion attention and a linear layer Liner to obtain mean and variance parameters. The dialogue sentence features are represented by average attention and passed through a linear layer Liner to obtain mean and variance parameters. By calculating the KL divergence, the generated emotion latent variables are constrained to be close to the emotion prior distribution, ensuring that the generated emotion feature distribution meets the preset conditions.

[0015] Furthermore, the knowledge conditional variational autoencoder unit passes the knowledge features through an average attention and a linear layer Liner to obtain mean and variance parameters. The dialogue sentence features are represented by the average attention and passed through a linear layer Liner to obtain mean and variance parameters. By calculating the KL divergence, the generated knowledge latent variables are constrained to be close to the knowledge prior distribution, so that the knowledge features are reasonably distributed in the latent space.

[0016] Furthermore, the emotion conditional variational autoencoder unit shares parameters with the knowledge conditional variational autoencoder unit.

[0017] Furthermore, the response generation unit of the dialogue takes the emotion latent variables and knowledge latent variables obtained from the emotion conditional variational autoencoder unit and the knowledge conditional variational autoencoder unit as input, assists in generating natural language responses with emotion and knowledge logic, embeds the dialogue sentences, combines the information of the knowledge graph with the current dialogue context through the knowledge fusion module KnowledgeFusion, and then generates candidate responses through the decoder TransDec, points to the generated candidate responses through the PointerNetwork module, and generates a complete response sentence.

[0018] Furthermore, the decoder TransDec adopts a multi-layer network structure, including: a Masked Self-Attention layer, a CrossAttention layer, and a FeedForwardNetwork layer;

[0019] Among them, the Masked Self-Attention layer is used to process the relationship within the context, the Cross Attention layer combines the input information and generates candidate responses, and the FeedForwardNetwork layer is a feedforward neural network layer that extracts deeper features through nonlinear transformations.

[0020] Furthermore, the loss function determined in S500 is:

[0021] L=αL EK +βL null +λ||θ|| 2

[0022] Among them, α and β are trainable coefficients, L EK is the divergence loss, L nll Generate loss for reply,λ||θ|| 2 For L nll Regularization term, λ is the regularization term coefficient, and θ is the weight matrix of the EKDLVG reply generation model.

[0023] Furthermore, the S600 includes:

[0024] The current user's dialogue sentences, dialogue history and external knowledge are obtained, and external knowledge that conforms to the current dialogue context is selected through the dialogue sentences, dialogue history and external knowledge. The emotional component in the dialogue is enhanced by the emotion conditional variational autoencoder unit, so that the generated reply is emotionally consistent with the previous text; and the knowledge conditional variational autoencoder unit is used to generate a reply that is more in line with the context and emotion. The reply generation unit of the dialogue is used to generate a complete reply sentence, and rounds of iterations are performed. After T rounds of dialogue, the EKDLVG reply generation model generates a complete reply sentence consistent with the emotional expression.

[0025] Furthermore, given the dialogue history D and the external knowledge K that matches the current dialogue context s , and with K s Expressing consistent emotions The EKDLVG response generation model generates complete responses consistent with emotional expressions. The statement is, if It is considered that there is a problem of inconsistency in the expression of emotions and knowledge, and the conversation continues.

[0026] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a reply generation method based on dual latent variables of emotion and knowledge. The EKDLVG reply generation model of the present invention can generate emotional responses that are consistent with the expression of emotion and knowledge. The conditional variational autoencoder (CVAE) is changed to a standard Transformer model structure, and the decoder adopts the decoder in the Transformer model. The EKDLVG reply generation model contains a dual latent space of two potential Gaussian distributions: the latent distribution of the emotion space determined by the given emotion and the latent distribution of the knowledge space determined by external knowledge. The CVAE latent space is standardized by introducing the dual latent space of knowledge and emotion. The present invention improves the content diversity and emotion accuracy of the generated responses, as well as the consistency of the expression of emotion and knowledge by learning emotional features and external knowledge features. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0028] Figure 1 Effect diagrams for expressing inconsistency between emotions and knowledge;

[0029] Figure 2 It is a schematic diagram of the structure of the EKDLVG reply generation model of the present invention;

[0030] Figure 3 It is a schematic diagram of the method flow of the present invention;

[0031] Figure 4 A schematic diagram of the KF1 index of the comparison model under different data sets provided in Example 2 of the present invention;

[0032] Figure 5 This is a schematic diagram of the EmoSA index of the comparison model under different data sets provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The purpose of the present invention is to provide a response generation method based on dual latent variables of emotion and knowledge, including: constructing an emotion conditional variational autoencoder unit for generating emotion feature distribution; constructing a knowledge conditional variational autoencoder unit for generating knowledge feature distribution; constructing a dialogue response generation unit for generating a complete response sentence; integrating the emotion conditional variational autoencoder unit, the knowledge conditional variational autoencoder unit and the dialogue response generation unit into an EKDLVG response generation model; determining a loss function, adjusting the parameters of the EKDLVG response generation model by the loss function until the loss function converges, and obtaining a final EKDLVG response generation model; using the final EKDLVG response generation model to obtain a dialogue generation result. In order to achieve consistent expression of emotion and knowledge in the emotion-enhanced knowledge-driven dialogue generation model, the present invention better integrates the selected knowledge features and the identified emotion features into the dialogue generation to achieve emotion-enhanced knowledge-driven dialogue generation. The model structure is as follows: Figure 2 As shown. The EKDLVG model consists of three parts, namely, the Emotion Conditional Variational Autoencoder Unit (ECVAEU), the Knowledge Conditional Variational Autoencoder Unit (KCVAEU) and the Dialogue Response Generation Unit (DRGU). The present invention improves the CVAE and Transformer decoder to enable the emotion-enhanced knowledge-driven dialogue generation model to generate responses that are consistent with emotion and knowledge expression.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1:

[0037] See also Figure 3Embodiment 1 of the present invention discloses a response generation method based on dual latent variables of emotion and knowledge, comprising the following steps:

[0038] S100: Construct an emotion conditional variational autoencoder unit to generate emotion feature distribution;

[0039] S200: construct a knowledge conditional variational autoencoder unit to generate knowledge feature distribution;

[0040] S300: constructing a reply generation unit for a dialogue, for generating a complete reply sentence;

[0041] S400: Integrate the emotion conditional variational autoencoder unit, the knowledge conditional variational autoencoder unit, and the dialogue response generation unit into the EKDLVG response generation model;

[0042] S500: determining a loss function, and adjusting the parameters of the EKDLVG recovery generation model by using the loss function until the loss function converges, thereby obtaining a final EKDLVG recovery generation model;

[0043] S600: Dialogue generation result obtained using the final EKDLVG response generation model.

[0044] In a specific embodiment, the emotion conditional variational autoencoder unit passes the emotion state through an emotion attention and a linear layer Liner to obtain the mean and variance parameters. The dialogue sentence features are represented by average attention and passed through a linear layer Liner to obtain the mean and variance parameters. By calculating the KL divergence, the generated emotion latent variables are constrained to be close to the emotion prior distribution, ensuring that the generated emotion feature distribution meets the preset conditions.

[0045] In a specific embodiment, the knowledge conditional variational autoencoder unit passes the knowledge features through an average attention and a linear layer Liner to obtain the mean and variance parameters. The dialogue sentence features are represented by the average attention and passed through a linear layer Liner to obtain the mean and variance parameters. The KL divergence is calculated to constrain the degree of closeness between the generated knowledge latent variables and the knowledge prior distribution, so that the knowledge features are reasonably distributed in the latent space.

[0046] In a specific embodiment, the emotion conditional variational autoencoder unit shares parameters with the knowledge conditional variational autoencoder unit.

[0047] Specifically, in the emotion conditional variational autoencoder unit or the knowledge conditional variational autoencoder unit, given a dialogue sentence, a Transformer encoder is used to transform each word w j,i, i∈[1,n] represents a d-dimensional embedding, where j represents the batch size and n represents the length of the sentence. The encoder is used to obtain the representation of each word in the sentence. As shown in formulas (1) to (2).

[0048]

[0049] Specifically, the emotional state e is passed through an emotional attention, and the dialogue sentence feature u x and knowledge feature u k Fusion representation through average attention. Traditional RNN / LSTM encoders usually use the last hidden state in the encoder to represent the semantic representation of the sequence level, while BERT uses special tokens to represent the semantic representation of the sequence level. However, it is believed that the semantic latent variable space obtained on this basis is not sufficient to summarize multiple feature information. In the response generation model based on dual latent variables of sentiment and knowledge, the output of the encoder layer is a vector sequence with the length of the input sentence. In order to make full use of the representation of each word in the sentence, an average attention block is defined, which can combine n vectors of the same length into one vector. The average attention block is essentially an example of multi-head self-attention. Create a queryable matrix Q=q avg ∈R d , and denote the hidden state output of the encoder as K, V, which is expressed by the multi-head attention formula (3).

[0050]

[0051] Among them, Q,K,V∈R b×d is the original input embedding that participates in self-attention, and the hidden layer representation of the last layer is obtained at the sequence level, and then the vector representation is transferred to the CVAE network to predict the prior and posterior distribution space.

[0052] Specifically, two encoders with shared parameters are implemented and used to obtain representations for reply generation. The prior encoder is used to obtain the sequence embeddings of knowledge k and emotional state e, and the posterior encoder is used to obtain the sequence embeddings of dialogue sentences x. and Approximately, P(z k |k) and P(z e |e). In addition, to estimate the distribution of emotion variables and knowledge variables, the latent variable z′ is introduced e and z′ k and the posterior distribution q φ (z e |x) and q φ (z k|x). Both the prior network and the recognition network are defined by the MLP layer. The prior network and the recognition network are established using MLP, as shown in formulas (4) to (7).

[0053]

[0054] Among them, W k and W e Represents the prior network weight vectors of knowledge and emotion, b k and b e The bias vector corresponding to the knowledge and sentiment prior network. k ' and W e 'represents the recognition network weight vectors of knowledge and emotion, b' k and b e ' represents the recognition network bias vector corresponding to knowledge and emotion, μ k represents the mean of the knowledge prior distribution; μ e represents the mean of the emotion prior distribution; μ' k Represents the mean of the posterior distribution of knowledge; μ' e Represents the mean of the posterior distribution of sentiment.

[0055] Specifically, the KL difference between the prior network and the recognition network is included in the final loss function, and the loss function is rewritten as shown in formula (8).

[0056]

[0057] where KL(·) is the KL divergence.

[0058] Specifically, in order to ensure the differentiability of back propagation, the latent variables are sampled using the reparameterization technique. During the training process, the latent variable z′ is extracted from the emotion and knowledge features. e and z′ k However, in inference, the latent variable Z e and Z k is sampled from the previous sentiment and knowledge features. Then, the latent variable z′ e and z′ k (training process) or Z e and Z k (Generation process) is input into the linear layer and then transferred to the reply generation unit of the dialogue.

[0059] In a specific embodiment, the dialogue response generation unit takes the emotion latent variables and knowledge latent variables obtained from the emotion conditional variational autoencoder unit and the knowledge conditional variational autoencoder unit as input, assists in generating natural language responses with emotion and knowledge logic, embeds the dialogue sentences, combines the information of the knowledge graph with the current dialogue context through the knowledge fusion module KnowledgeFusion, and then generates candidate responses through the decoder TransDec, points to the generated candidate responses through the Pointer Network module, and generates a complete response sentence.

[0060] In a specific embodiment, the decoder TransDec adopts a multi-layer network structure, including: a MaskedSelf-Attention layer, a Cross Attention layer and a Feed Forward Network layer;

[0061] In a specific embodiment, the Masked Self-Attention layer is used to process the relationship within the context, the Cross Attention layer combines the input information and generates a candidate response, and the Feed Forward Network layer is a feedforward neural network layer. Through nonlinear transformation, deeper features are extracted, so that the model can generate more natural and smooth responses.

[0062] Specifically, in the dialogue response generation unit, z obtained by the emotion conditional variational autoencoder unit and the knowledge conditional variational autoencoder unit is converted into e and z k As the input of the dialogue response generation unit. The dialogue response generation unit is built based on the Transformer decoder and finally generates the response R through the pointer network. T .

[0063] Specifically, R T Output of the overall pipeline for the EKDLVG response generation model.

[0064] Specifically, the dialogue sentences are Where L is the sequence length of the dialogue sentence, and the dialogue sentence is passed through the embedding layer to obtain E U , E U Input to Transformer encoder to obtain dialogue sentence representation H U , as shown in formula (9).

[0065] H U =TransEnc(E U ) (9)

[0066] Among them, TransEnc is an encoder based on Transformer output size d, H U ∈R L×d .

[0067] Specifically, the path of the common sense knowledge graph KG is converted into a sequence, such as two triples in the common sense knowledge graph<home,RelatedTo,heart> ,<heart,UsedFor,love> It can be transformed into "Home is related to heart, Heart is used for love". Then the sentence transformed into natural language is input into the two-layer BiGRU to obtain the knowledge representation H K , and finally the knowledge representation H K and the representation of the dialogue sentence H U By connecting, we get the representation of knowledge-enhanced dialogue sentences As shown in formulas (10) to (11).

[0068]

[0069] in, is the connection operation, H K ∈R p×2d , p is the length of the sequence from knowledge triples to natural language sentences, W K ∈R 2d×d is the weight matrix of the linear layer.

[0070] Specifically, in order to ensure that the triples are logically related to the keywords in the dialogue history, we first use the text ranking algorithm to obtain the word sequence from the dialogue history D, and then filter the words with noun, verb, adverb and adjective vocabulary attributes in the dialogue history word sequence keywords to obtain the keyword τ = {h1,h2,···,h i ,···,h cw}, h i ∈τ is obtained through the word embedding layer E i , and then use ItrEnc to obtain the conversation history semantic features δ i Then, according to the cosine similarity score, we filter the common sense knowledge graph KG to get the i The first k associated triplets, Judge h j and Are the two words the same? If they are the same, then h i and h j There is a one-hop path between And h i and h j Add the final set of keywords τr Finally, the attention layer is used to calculate τ r The attention weight vector g of each word in Add to τ r Repeat the above steps until all paths are found, as shown in formulas (12) to (13).

[0071] δ i =ItrEnc(E i ),i=1,···,cw (12)

[0072] g = Attention(τ r ) (13)

[0073] Among them, ItrEnc is the internal encoder based on Transformer with output size d, Attention is the attention layer with output size c and g∈R c .

[0074] Specifically, the dialogue sentence is embedded in E U and with the initial sequence marker E SOS The embeddings are connected and then a linear transformation is used to obtain a new embedding E' SOS , as shown in formula (14).

[0075]

[0076] Where W t ∈R d×(d+da) , here refers to the matrix space of d×(d+da), where d is the internal encoder of size d.

[0077] Specifically, using the knowledge-enhanced decoder, since the keywords between the paths are crucial for generation, this embodiment provides a multi-head keyword attention, with the output of the cross-attention layer as the query, and g and The dot product on is used as the key and value. Then TransDec outputs the hidden state H, as shown in formula (15).

[0078]

[0079] Among them, TransDec is a decoder based on Transformer, E r1:t-1 Represents the resulting token embedding.

[0080] Specifically, in order to better integrate the emotions and knowledge in the decoder into the dialogue generation, z e 、z k and H are connected at the token level, and a pointer network is used to output the probability distribution of each word in the vocabulary, as shown in formula (16).

[0081]

[0082] Among them, PointGen is the pointer generation network, and P* refers to the conditional probability.

[0083] During the training phase, the target response is prepared in advance with the [SOS] marker to generate R T , and obtain the final decoder input. The training loss is the standard negative log-likelihood loss on the target reply generation, as shown in Formula (17).

[0084]

[0085] In a specific embodiment, the training of the EKDLVG response generation model includes the response generation of the emotion and knowledge conditional variational autoencoder and the dialogue. Therefore, the loss L of EKDLVG should consist of two parts: KL divergence loss L EK and the reply generation loss L nll . For the KL divergence loss L EK Optimize the target through KL divergence; decision loss L null The model performance is optimized by using the standard negative log-likelihood loss. To ensure a smooth solution during training, this embodiment adds an additional regularization term to the total loss L. Therefore, the EKDLVG loss function can be described by formula (18).

[0086] L=αL EK +βL null +λ||θ|| 2 (18)

[0087] Among them, α and β are trainable coefficients, L EK is the divergence loss, L nll Generate loss for reply,λ||θ|| 2 For L nll Regularization term, λ is the regularization term coefficient, and θ is the weight matrix of the EKDLVG reply generation model.

[0088] In a specific embodiment, S600 includes:

[0089] Get the current user's conversation statement U T , dialogue history D and external knowledge K, select external knowledge K that is suitable for the current dialogue context through dialogue sentences, dialogue history and external knowledge s , enhance the emotion of the current dialogue sentence to get Y T , the emotion-enhanced knowledge-driven dialogue generation model obtains the most appropriate emotional response R T , through T rounds of dialogue, model generation and K s Express consistent emotions The statement.

[0090] in, Refers to the i-th word of the user's utterance in the T-th round of dialogue, with a sequence length of L. is the jth word of the reply sentence in the Tth round of dialogue, and the sequence length is n. T-1 ,R T-1}, K={K1,K2,···,K s ,···,K m}. Represents the T-th round of dialogue generation model discourse R T emotions.

[0091] Specifically, in order to generate responses that are consistent with the expression of emotions and knowledge, the present invention solves the problem of consistency between the expression of emotions and knowledge, which is defined as: given a dialogue history D, external knowledge K that conforms to the current dialogue context s , and with K s Expressing consistent emotions The sentiment expressed by the generated sentence in the model reply is like It is considered that there is a problem of inconsistency in the expression of emotions and knowledge, and the conversation continues.

[0092] Example 2

[0093] In order to better demonstrate the effect of the response generation method based on dual latent variables of emotion and knowledge provided in the above-mentioned Example 1 of the present invention, this Example 2 conducted more experiments to compare the performance of the EKDLVG response generation model of the present invention with the emotional dialogue generation model and the knowledge-driven dialogue generation model.

[0094] Specifically, the reply generation model proposed in the present invention is compared with two types of dialogue generation models, namely, the emotional dialogue generation model and the knowledge-driven dialogue generation model. The reply generation model proposed in the present invention is composed of these three units, and the whole is compared with these two types of models.

[0095] Specifically, the experiments were conducted on the emotional dialogue dataset EmpatheticDialogues and the knowledge-driven dialogue dataset Wizard-of-Wikipedia. The EmpatheticDialogues dataset is a one-to-one dialogue between 25k speakers and listeners. Each dialogue contains an emotion label and a given context. There are 32 emotion categories with a balanced distribution. The speakers talk about their situation, and the listeners try to understand the speakers' feelings and respond accordingly. The Wizard-of-Wikipedia dataset is an open domain dialogue dataset between speakers and listeners, containing multi-round knowledge-based dialogues. The speaker is a knowledge expert who can access any knowledge retrieval system to view the knowledge related to the dialogue, which contains 201999 rounds of dialogue. The hardware configuration used in the experiment is AMD Ryzen 75800H CPU, NVIDIAGeForce RTX 3060GPU, Windows operating system, CUDA11.0 software environment, and Pytorch1.7 deep learning framework.

[0096] Specifically, this embodiment 2 uses the pre-trained BERT to initialize word embedding, where the hidden vector is 768. The dimension of the knowledge graph is 300. The SGD optimizer is used during training, with an initial learning rate of 0.01, a dropout of 0.4, a batch_size of 64, and 50 iterations.

[0097] Specifically, this Example 2 conducted more experiments to compare the performance of EKDLVG with the emotional dialogue generation model and the knowledge-driven dialogue generation model. The baseline models for comparison are as follows:

[0098] Transformer: Transformer with a pointer network, trained to optimize the negative log-likelihood loss.

[0099] Multi-Transformer (Multi-Trans): A variant of Transformer that includes generation loss and sentiment classification loss in addition to jointly optimize the model.

[0100] MOEL: A model consisting of several Transformer decoders, each of which learns how to respond to emotions, and the outputs are gently combined to produce a response.

[0101] MIME: A Tramsformer-based model that adopts an imitation strategy to sample the emotion of the target reaction based on the detected user emotion. The emotions are classified into two categories (positive and negative). The model utilizes a VAE to obtain representations of simulated and non-imitation emotions.

[0102] EmpDG: A generative adversarial network that considers multi-resolution emotions and introduces a recognizer to supervise the training of semantics and emotions.

[0103] KEMP: A model that uses multi-type knowledge to help understand and express emotions and learns emotion dependencies via emotion cross-attention mechanism.

[0104] CEM: A method for conversational generation by leveraging common sense to improve understanding of the interlocutor’s context and feelings.

[0105] TMNet: A model that combines the element memory network structure and the Transformer structure to retrieve knowledge from the knowledge graph for generating responses.

[0106] PostKS+BERT: Utilizes posterior information to jointly train knowledge selection and dialogue generation, and uses BERT as the base model.

[0107] SKT: Use a sequence latent variable model to model the knowledge selection history. In order to better generate dialogues, BERT is used to encode the input source and a copy mechanism is combined to promote response generation.

[0108] KINet: Improve knowledge selection using dense representation of augmented knowledge and improve response generation by incorporating more relevant knowledge via knowledge sampler.

[0109] In a specific embodiment, in order to evaluate the effectiveness of the proposed model and determine whether the response generation model based on dual latent variables of emotion and knowledge can generate responses that express emotions and knowledge consistently, this embodiment uses the following evaluation indicators to verify the effectiveness of the model.

[0110] (1) Automatic evaluation indicators

[0111] PPL: is an indicator that measures the quality of a language model based on the model itself. The larger the sentence probability, the better the dialogue generation model and the smaller the perplexity. Calculation method: It mainly estimates the probability of a sentence appearing based on each word and normalizes it by the sentence length. The formula is shown in (19).

[0112]

[0113] Among them, C represents the candidate sentence, len(C) is the sentence length, P(w i ) is the probability of the i-th word. The first word is P(w1|w0), and w0 is START, which means the beginning of the sentence and is a placeholder.

[0114] Distinct-1 (DIST-1): measures the proportion of unique n-grams in the generated responses and is used to measure the diversity of response generation, as shown in (20).

[0115]

[0116] Among them, Count(uniquengram) represents the number of unique words in the generated reply, and Count(word) represents the total number of words in the generated reply.

[0117] Knowledge F1 (KF1): The standard F1 represents a measure of the word overlap between the responses generated by the dialogue generation model and the target responses of real humans, while Knowledge F1 is a measure of the word overlap between the responses generated by the dialogue generation model and the knowledge-based responses.

[0118] EmoSA and EmoLA: Speaker’s emotion accuracy and listener’s emotion accuracy, re-identify the emotions of two interlocutors through the emotion recognition model.

[0119] (2) Manual evaluation indicators

[0120] In order to better evaluate the performance of the response generation model based on the dual latent variables of emotion and knowledge in terms of people's subjective perception, this embodiment selected five raters to evaluate the response generation quality of the model from the following aspects.

[0121] Consistency between emotion and knowledge expression: whether the emotion expression in the response sentence is consistent with the target knowledge expression, where 0 indicates inconsistency and 1 indicates consistency.

[0122] Appropriateness of knowledge expression: whether the knowledge expression in the response sentence is appropriate, where 0 means inappropriate and 1 means appropriate.

[0123] Reasonableness of emotional expression: whether the emotional expression in the response sentence is reasonable, where 0 means unreasonable and 1 means reasonable.

[0124] Content relevance: whether the reply statement is relevant to the dialogue statement and external knowledge, where 0 means no relevance, 1 means slightly relevant, and 2 means very relevant.

[0125] In a specific embodiment, the experimental results and analysis are as follows:

[0126] (1) Model comparison experiment

[0127] The automatic evaluation comparison experiment was carried out on the test sets of the EmpatheticDialogues and Wizard-of-Wikipedia datasets. The experimental results under multiple automatic evaluation indicators are shown in Tables 1 and 2.

[0128] Table 1 Experimental results on the EmpatheticDialogues dataset

[0129] Models PPL DIST-1 KF1 EmoSA EmoLA Transformer 34.11 0.49 10.17 - - Multi-Trans 36.42 0.43 11.53 28.91 - MOEL 36.59 0.60 12.35 32.33 - MIME 37.52 0.32 14.64 34.88 - EmpDG 37.37 0.45 13.96 32.45 - KEMP 36.39 0.66 15.42 36.57 - CEM 36.11 0.66 14.95 39.07 - CCM 36.16 0.73 16.54 31.34 - TMNet 35.48 0.71 17.49 30.79 - SKT 36.10 0.83 19.16 32.94 - Postk+BERT 35.03 0.91 18.30 32.25 - KINet 34.97 1.04 20.57 30.91 - EKDLV 34.36 1.13 19.83 35.43 65.23

[0130] Specifically, Table 1 shows the experimental results of automatic evaluation on the EmpatheticDialogues test set. EKDLVG outperforms all baseline models in both PPL and DIST-1, which shows that EKDLVG has good performance in both the quality and diversity of the response generation content. This is because EKDLVG enhances the response generation of the model through external knowledge. Specifically, compared with the SOTA model MIME on the EmpatheticDialogues dataset, EKDLVG decreased by 3.16% in PPL and increased by 0.81% in DIST-1. Compared with the stronger baseline model EmpDG, EKDLVG also achieved good results. However, in terms of emotional expression, EKDLVG is not as good as the emotional dialogue generation models CEM and KEMP under the EmoSA indicator, but it is worse than the knowledge-driven dialogue generation model KINet in emotional expression. This is because the CEM and KEMP models use external knowledge to help understand and express emotions, and learn emotional dependencies through the emotional cross-attention mechanism, while in EKDLVG, emotional expression is sacrificed to improve language quality and diversity. In terms of knowledge expression, EKDLVG is not as good as the knowledge-driven dialogue generation model KINet in KF1, but it is worse than the emotional dialogue generation model in knowledge expression.

[0131] Table 2 Experimental results on the Wizard-of-Wikipedia dataset

[0132]

[0133]

[0134] Specifically, Table 2 shows the experimental results of automatic evaluation on the Wizard-of-Wikipedia test set. EKDLVG outperforms the baseline model in both PPL and DIST-1 indicators, which shows that EKDLVG has good performance in both the quality and diversity of the reply generation content. This is because EKDLVG not only integrates emotional features when generating replies, but also integrates knowledge features to enhance the model's reply generation. Specifically, compared with the SOTA model KINet on the Wizard-of-Wikipedia dataset, EKDLVG decreased by 1.4% in PPL and increased by 0.02% in DIST-1. Compared with the stronger baseline models SKT and Postk+BERT, EKDLVG also achieved good results. However, in terms of knowledge expression, EKDLVG is not as good as the KINet model in the KF1 indicator. This is because KINet improves reply generation through external knowledge, while EKDLVG understands and expresses emotions through external knowledge. In terms of emotional expression, EKDLVG is not as good as the emotional dialogue generation models CEM and KEMP under the EmoSA indicator, but it is worse than the knowledge-driven dialogue generation model KINet in terms of emotional expression.

[0135] Specifically, in order to better verify the effectiveness of the constructed EKDLVG model, this embodiment conducts comparative experiments on two data sets. The comparison of the KF1 index in the EmpatheticDialogues and Wizard-of-Wikipedia data sets is shown in the figure below: Figure 4 As shown in the figure, the comparison of EmoSA indicators in the EmpatheticDialogues and Wizard-of-Wikipedia datasets is shown in Figure 5 As shown. Figure 4 It can be seen that the KF1 index of EKDLVG in both datasets is better than that of other comparison models, but lower than that of KINet model. The KF1 index of the comparison model in Wizard-of-Wikipedia dataset is higher than that in EmpatheticDialogues dataset, which is because Wizard-of-Wikipedia dataset contains knowledge triples related to dialogue. Figure 5 It can be seen that the EmoSA index of EKDLVG in both datasets is better than that of other comparison models, but lower than that of CEM model. The EmoSA index of the comparison model in EmpatheticDialogues dataset is higher than that in Wizard-of-Wikipedia dataset.

[0136] Specifically, in summary, EKDLVG generates responses better than the emotional dialogue generation model and the knowledge-driven dialogue generation model on both the emotional dialogue dataset and the knowledge-driven dialogue dataset. The response generation model based on the dual latent variables of emotion and knowledge can not only ensure the authenticity of the response sentence, but also the sentence quality and emotional expression.

[0137] (2) Manual evaluation comparison experiment

[0138] In order to verify the performance of the response generation model based on the dual latent variables of emotion and knowledge in the problem of consistency of emotion and knowledge expression, we first used 100 groups of multi-round dialogue samples extracted from the test set to generate dialogue responses, and then invited 5 reviewers to conduct subjective manual scoring from the perspectives of consistency of emotion and knowledge expression, appropriateness of knowledge expression, appropriateness of emotion expression, and content relevance. The final evaluation results are shown in Table 3.

[0139] Table 3. Results of manual evaluation experiments

[0140]

[0141] Specifically, from the perspective of consistency of emotion and knowledge expression, EKDLVG is significantly ahead of other models, indicating that EKDLVG can better integrate emotion and knowledge features into dialogue generation, while other models cannot generate sentences with consistent emotion and knowledge expression due to the lack of integration of emotion and knowledge features. From the perspective of content relevance, EKDLVG is also ahead of other models, indicating that EKDLVG uses external knowledge to enhance the model's response generation, while other models mainly use external knowledge for emotion expression and knowledge expression. From the perspective of emotion rationality, KEMP and CEM models achieved the best scores, indicating that EKDLVG sacrifices emotion expression to make knowledge expression appropriate and content relevant, and from the perspective of knowledge expression appropriateness, KINet model achieved the best score. In summary, EKDLVG can not only express emotion and knowledge consistency, but also generate responses related to the above context. Compared with multiple baseline models, it has good performance in dialogue response generation tasks.

[0142] Specifically, in order to intuitively demonstrate the response generation effect of the emotion-enhanced knowledge-driven dialogue generation model EEKDGM, this embodiment sets two groups of dialogues from the data set for sampling tests, and records the contents of the two groups of dialogues for case analysis. The records are shown in Table 4.

[0143] Table 4. EEKDGM reply generation example

[0144]

[0145] Table 4 EEKDGM’s reply generation example (continued)

[0146]

[0147] As shown in Table 4, in the first example, through external knowledge<kid,excite> and<disneyland,time> ,EEKDGM improves its understanding of the conversation history and focuses on Disneyland during decoding, generating responses that not only acknowledge the user’s feelings, but also generate responses that are consistent in expressing emotions and knowledge. In the second example, EEKDGM recognizes the user’s emotions, distinguishes between its own and the user’s emotions, and generates a response “That is a scary situation” to acknowledge the user’s emotions.

[0148] Specifically, this embodiment discloses the problem that the dialogue generation model is more blunt or exaggerated when expressing emotions due to the imbalance between the fusion of emotional features and knowledge features, the responses generated by the model lack realism and naturalness, and the expression of emotions and knowledge are inconsistent. Inspired by CVAE, a response generation model based on dual latent variables of emotions and knowledge is proposed. Experiments were conducted on the EmpatheticDialogues and Wizard-of-Wikipedia datasets, and multiple emotional dialogue generation models and knowledge-driven dialogue generation models were compared. The results showed that the response generation model based on dual latent variables of emotions and knowledge is also superior to other models in terms of consistency between emotion and knowledge expression. At the same time, in order to further prove the performance of the knowledge-driven dialogue generation model enhanced by emotion, experiments were conducted from the aspects of consistency between emotion and knowledge expression, rationality of emotion expression, appropriateness of knowledge expression, and content relevance. The experimental results show that the knowledge-driven dialogue generation model enhanced by emotion has a good performance.

[0149] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0150] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present embodiments may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present embodiment, but will conform to the widest range consistent with the principles and novel features disclosed in the present embodiment.

Claims

1. A response generation method based on dual latent variables of emotion and knowledge, characterized in that: The following steps are involved: S100: Construct an emotion conditional variational autoencoder unit to generate emotion feature distribution; S200: construct a knowledge conditional variational autoencoder unit to generate knowledge feature distribution; S300: constructing a reply generation unit for a dialogue, for generating a complete reply sentence; S400: Integrate the emotion conditional variational autoencoder unit, the knowledge conditional variational autoencoder unit, and the dialogue response generation unit into the EKDLVG response generation model; S500: determining a loss function, and adjusting the parameters of the EKDLVG recovery generation model by using the loss function until the loss function converges, thereby obtaining a final EKDLVG recovery generation model; S600: Use the final EKDLVG response generation model to obtain the dialogue generation result.

2. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The emotion conditional variational autoencoder unit passes the emotion state through an emotion attention and a linear layer Liner to obtain the mean and variance parameters. The dialogue sentence features are represented by average attention and passed through a linear layer Liner to obtain the mean and variance parameters. By calculating the KL divergence, the generated emotion latent variables are constrained to be close to the emotion prior distribution, ensuring that the generated emotion feature distribution meets the preset conditions.

3. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The knowledge conditional variational autoencoder unit passes the knowledge features through an average attention and a linear layer Liner to obtain the mean and variance parameters. The dialogue sentence features are represented by the average attention and passed through a linear layer Liner to obtain the mean and variance parameters. By calculating the KL divergence, the generated knowledge latent variables are constrained to be close to the knowledge prior distribution, so that the knowledge features are reasonably distributed in the latent space.

4. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The emotion conditional variational autoencoder unit shares parameters with the knowledge conditional variational autoencoder unit.

5. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The reply generation unit of the dialogue takes the emotion latent variables and knowledge latent variables obtained from the emotion conditional variational autoencoder unit and the knowledge conditional variational autoencoder unit as input, assists in generating natural language replies with emotion and knowledge logic, embeds the dialogue sentences, combines the information of the knowledge graph with the current dialogue context through the knowledge fusion module KnowledgeFusion, and then generates candidate replies through the decoder TransDec, points to the generated candidate replies through the PointerNetwork module, and generates a complete reply sentence.

6. The response generation method based on dual latent variables of emotion and knowledge according to claim 5, characterized in that: The decoder TransDec adopts a multi-layer network structure, including: Masked Self-Attention layer, CrossAttention layer and FeedForwardNetwork layer; Among them, the Masked Self-Attention layer is used to process the relationship within the context, the Cross Attention layer combines the input information and generates candidate responses, and the FeedForwardNetwork layer is a feedforward neural network layer that extracts deeper features through nonlinear transformations.

7. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The loss function determined in S500 is: L=αL EK +βL null +λ||θ|| 2 Among them, α and β are trainable coefficients, L EK is the divergence loss, L nll Generate loss for reply,λ||θ|| 2 For L nll Regularization term, λ is the regularization term coefficient, and θ is the weight matrix of the EKDLVG reply generation model.

8. The response generation method based on dual latent variables of emotion and knowledge according to claim 1, characterized in that: The S600 includes: The current user's dialogue sentences, dialogue history and external knowledge are obtained, and external knowledge that conforms to the current dialogue context is selected through the dialogue sentences, dialogue history and external knowledge. The emotional component in the dialogue is enhanced by the emotion conditional variational autoencoder unit, so that the generated reply is emotionally consistent with the previous text; and the knowledge conditional variational autoencoder unit is used to generate a reply that conforms to the context and emotion. The reply generation unit of the dialogue is used to generate a complete reply sentence, and rounds of iterations are performed. After T rounds of dialogue, the EKDLVG reply generation model generates a complete reply sentence consistent with the emotional expression.

9. According to the method for generating responses based on dual latent variables of emotion and knowledge in claim 7, given a dialogue history D, external knowledge K that conforms to the current dialogue context s , and K s Expressing consistent emotions The EKDLVG response generation model generates complete responses consistent with emotional expressions. Statement, if It is considered that there is a problem of inconsistency in the expression of emotions and knowledge, and the conversation continues.

Images