A method and system for generating dialogues with enhanced text features
By extracting keywords and encoding using attention mechanisms, fusing semantic vectors to enhance dialogue feature representations, the problem of generating low-quality replies in existing dialogue systems is solved, and the diversity and accuracy of dialogue generation is improved.
Patent Information
- Application Number
- CN202211238085.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-10-11
AI Technical Summary
The existing generative dialogue system model is difficult to fully characterize the characteristics of dialogue text, resulting in the generated dialogue reply low quality or irrelevant, and all words in the text sequence have the same weight in the characteristics, so it is impossible to effectively learn attention weights, which affects the diversity of dialogues and accuracy.
The TextRank algorithm is used to extract keywords in problem text, generate rich semantic vectors through keyword encoder and semantic encoder, and integrate keyword semantic vectors and problem text vectors through multi-layer perceptrons to train dialogue generation models to improve feature representation Accuracy and quality.
By explicitly extracting keywords and coding with attention mechanisms, the representation ability of text features is enhanced, the quality and diversity of dialogue generation is improved, and the problem of not having keywords occupy high weight is avoided. The generated dialogue is more semantic and accurate. and diversity.
Smart Images

Figure CN115495566B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer dialogue, in particular to an open domain generation model for enhancing dialogue feature expression, and specifically to a dialogue generation method and system for enhancing text features. Background Art
[0002] Human-computer dialogue is mainly divided into task-oriented and non-task dialogue (open domain) application systems. Compared with task-oriented dialogue systems, open domain dialogue systems do not need to perform specific tasks, and the responses generated are more casual than task-oriented ones. Chatbots can currently be divided into three types, namely retrieval-based, generative, and knowledge graph-based. Retrieval-based chatbots use techniques such as sorting and matching to extract the most appropriate responses from existing dialogue corpora, but this method can only generate texts that exist in the corpus and cannot achieve the diversity of dialogues. If there are more dialogues in the corpus, the speed of generating responses will slow down, affecting the chat experience.
[0003] With the in-depth development of end-to-end deep learning models, open-domain dialogue system models have solved some of the corresponding problems, making the generated dialogue responses richer. Among them, the end-to-end encoder-decoder model used by the generative chatbot encodes the dialogue into a specific feature vector, and the decoder samples each word of the generated dialogue from the vocabulary, so that dialogues that are not in the generated corpus can be generated, which makes up for the defect that the retrieval-based dialogue can only generate dialogues according to the template of the corpus, making the responses richer. However, since the generative model samples from the vocabulary and then combines the sampled words into the reply sentence in the sampling order, the representation of the dialogue features is not complete, making it very easy to produce low-quality or irrelevant semantic responses.
[0004] This type of problem is explained in detail using the Seq2seq model as an example. Seq2seq is the earliest end-to-end generation model and has made significant contributions to the field of text generation. Subsequent chatbots are basically based on the seq2seq paradigm. It contains two recurrent neural networks (RNNs), an encoder and a decoder. The encoder encodes the input sequence into a semantic vector, and the decoder decodes the semantic vector into an output sequence. However, both the RNN encoding sequence and the decoding sequence must be performed from left to right in an autoregressive manner, which is difficult to parallelize and has a high time complexity for processing long sequences. At the same time, it is difficult for RNN to establish a long-distance context dependency model, resulting in the loss of important information in the extracted features.
[0005] Existing deep models pay the same attention to words at different positions in a text sequence. Although there is an attention mechanism to solve the problem of word weights in text sequences, the current dialogue system model still uses the maximum likelihood function of the generated text and the reference text as the optimization function, and lacks a special optimization function to learn the attention weights, resulting in inaccurate importance weights between different words learned by the model. If the attention mechanism is not used, all words will have the same weight in the text features, which is not suitable for open dialogue scenarios. Summary of the invention
[0006] In order to solve the problems that the existing generative model cannot fully characterize the features of the dialogue text, the recurrent neural network causes feature loss, and all words in a text sequence have the same weight in the text features, which makes it very easy to generate some general or semantically inaccurate dialogue replies, the present invention provides a dialogue generation method and system with enhanced text features.
[0007] In a first aspect, the present invention provides a method for generating a dialogue with enhanced text features, comprising the following steps:
[0008] S1. Obtain the question text and the reply text, extract the keywords in the question text through the TextRank algorithm, and obtain the keyword sequence; obtain the question text vector of the question text by inputting the encoder;
[0009] S2. Introduce a keyword encoder, which encodes the keyword sequence through an attention mechanism to obtain a keyword vector;
[0010] S3. concatenate the keyword vector and the semantic vector and input them into the first multi-layer perceptron to obtain a keyword semantic vector containing rich semantics;
[0011] S4. concatenate the keyword semantic vector and the question text vector and pass them through the second multi-layer perceptron to obtain an input vector;
[0012] S5. Train the dialogue generation model according to the input vector and the reply text, calculate the loss value using the loss function and perform back propagation, and adjust the parameters of the dialogue generation model;
[0013] S6. Input the text to be replied into the trained dialogue generation model to generate a dialogue.
[0014] Furthermore, step S4 further includes:
[0015] Using the output encoder to obtain a reply text vector of the reply text;
[0016] The keyword semantic vector and the question text vector are concatenated and passed through the second layer perceptron to obtain the first fusion feature, and the first fusion feature is input into the prior network to obtain the prior distribution parameter;
[0017] The keyword semantic vector, question text vector and reply text vector are concatenated and passed through the third layer perceptron to obtain the second fusion feature, and the second fusion feature is input into the recognition network to obtain the approximate posterior distribution parameter;
[0018] The parameters of the approximate posterior distribution are reparameterized to obtain latent variables, which are initialized through linear transformation to obtain the input vector.
[0019] Furthermore, the keyword encoder encodes the keyword sequence through the attention mechanism to obtain the keyword vector, including:
[0020] h t =Enc key (e(K))
[0021] Enc key (e(k i ))=LSTM(input i )
[0022]
[0023]
[0024] Among them, Enc key () represents the keyword encoder, K = k1, k2, ..., k t represents the keyword sequence, k i represents the i-th keyword, h t represents the keyword vector, e() represents the word vector of the calculated word, input i Indicates keyword k i The weighted vector of LSTM() represents the long short-term memory network LSTM, α(e(k i ),e(k j )) indicates keyword k i With keyword k j The attention weight between them, and 1≤j<i.
[0025] Furthermore, a semantic encoder is used to obtain a semantic vector, including:
[0026] h s =Enc sem (e(S),(h0,c0))
[0027] Among them, Enc sem () represents the semantic encoder, h s represents the semantic vector, h0 represents the initial hidden state of the semantic encoder, c0 represents the initial cell state of the semantic encoder, and S represents the semantic text corresponding to the question text.
[0028] In a second aspect, based on the method proposed in the first aspect, the present invention provides a dialogue generation system for enhancing text features, including a sample module, a keyword extraction module, an encoding module, a fusion module, and a training module, wherein:
[0029] A sample module is used to obtain multiple groups of dialogue samples, where the dialogue samples include question texts and response texts;
[0030] A keyword extraction module is used to extract multiple keywords from the question text to form a keyword sequence;
[0031] The encoding module includes an input encoder, an output encoder, a keyword encoder and a semantic encoder, which are used to encode the data of the sample module and the keyword extraction module;
[0032] The fusion module includes a first fusion module and a second fusion module:
[0033] The first fusion module is used to fuse the keyword vector and the semantic vector to obtain a keyword semantic vector containing rich semantics;
[0034] The second fusion module is used to fuse the keyword semantic vector and the question text vector;
[0035] The training module is used to train the dialogue generation model. It uses the loss function to calculate the loss and back-propagate to adjust the parameters of the dialogue generation model.
[0036] Furthermore, it also includes a third fusion module, a priori network and a recognition network:
[0037] The third fusion module is used to fuse the keyword semantic vector, the question text vector and the reply text vector;
[0038] A priori network, used to receive the output result of the second fusion module and obtain prior distribution parameters;
[0039] The recognition network is used to receive the output result of the third fusion module and obtain the approximate posterior distribution parameters.
[0040] Beneficial effects of the present invention:
[0041] The present invention proposes an open dialogue generation method for enhancing the feature expression of dialogue text. When ordinary RNN is used to encode text, the front part of the text is easily ignored, and the later the content of the text, the greater its weight, so ordinary RNN cannot encode long text. The long text dependency problem can be solved by using LSTM, but all words in the text have the same weight; if only the attention mechanism is used to solve this problem, since there is no optimization function specifically optimizing attention, but the maximum likelihood optimization function generated by the dialogue is borrowed, it is extremely difficult to learn the correct weight, which may cause the key text to have a low weight and the non-key text to have a high weight, thereby misleading the model optimization. Therefore, the present invention chooses to use a keyword extraction method to explicitly extract keywords from the text, and encodes the keywords through the attention mechanism instead of the entire text. Specifically, the keyword features encoded using the attention mechanism are calculated with the semantic category features, and then the results are used as part of the text features and merged into the text features. In this way, the weight of the more critical words in the text sequence in the text features is increased. Such a method can not only correctly enhance the weight of the key text, but also avoid the model from learning the wrong weight, resulting in non-critical words occupying a relatively high weight. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A flow chart of a method for generating a dialogue with enhanced text features according to the present invention;
[0043] Figure 2 A schematic diagram of a Seq2seq model according to an embodiment of the present invention;
[0044] Figure 3 Schematic diagram of a CAVE model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] The present invention provides a method and system for generating dialogues with enhanced text features, which integrate keywords with category semantics to extract feature representations with richer semantics and more accurate weights, and fuse the integrated features with the original text features to obtain higher quality input features, train a dialogue generation model, and obtain higher quality dialogues.
[0047] In one embodiment, the Figure 2The Seq2seq model shown in FIG. 1 describes the dialog generation model training method for enhancing text features of the present invention, such as Figure 1 As shown, including:
[0048] S11. Get question text C = c1, c2, ..., c m and reply text X=x1,x2,...,x n , c u represents the u∈{1,2,...,m}th word in the question text, x v represents the v∈{1,2,...,n}th word in the reply text, m represents the number of words in the question text, and n represents the number of words in the reply text. The keywords in the question text are extracted by the TextRank algorithm to obtain the keyword sequence K=k1,k2,...,k t , k i represents the i∈{1,2,...,t}th keyword;
[0049] S12. The question text vector of the question text is obtained by inputting the encoder, and the semantic encoder obtains the semantic vector, which is expressed as:
[0050] h ci ,c ci =Enc in (e(c i ),(h ci-1 ,c ci-1 ))
[0051] h s =Enc sem (e(S),(h0,c0))
[0052] h ci ,c ci They represent the hidden state of the i-th step of the input encoder and the cell state of the i-th step, Enc in () indicates the input encoder, Enc sem () represents the semantic encoder, h s represents the semantic vector, h0 represents the initial hidden state of the semantic encoder, c0 represents the initial cell state of the semantic encoder, and S represents the semantic category of the question text;
[0053] S13. Introduce a keyword encoder, which encodes each keyword through an attention mechanism to obtain a corresponding keyword vector;
[0054] S14. The keyword vector and the semantic vector are concatenated and input into the first multi-layer perceptron to obtain a keyword semantic vector containing rich semantics, and one question text corresponds to one semantic vector, which is expressed as:
[0055] h'=MLP([h t :h s ])
[0056] h' represents the keyword semantic vector, h t represents the keyword vector, h s represents the semantic vector, MLP() represents the first multi-layer perceptron;
[0057] S15. After concatenating the keyword semantic vector and the question text vector, pass it through the second multi-layer perceptron to obtain the input vector, which is expressed as:
[0058]
[0059] represents the input vector of the decoder, that is, the initial hidden state of the decoder, h m represents the question text vector, MLP′() represents the second multi-layer perceptron;
[0060] S16. Train the dialogue generation model based on the input vector and the reply text, use the loss function to calculate the loss value and back propagate, and adjust the parameters of the dialogue generation model.
[0061] Specifically, Figure 2 As shown, the first input to the decoder is <sos>(Start of sentence) label and initial hidden state, that is, input vector The input of each subsequent step is the word x in the reply sequence t And the hidden state s output in the previous step t-1 Finally, SoftMax is used to map the output of the decoder to the vocabulary space, and the word with the highest probability in the vocabulary is taken as the generated result, which is expressed as:
[0062] s t =Dec(s t-1 ,e(x t ));
[0063]
[0064] s t represents the hidden state of the decoder at step t, Dec() represents the decoder, represents the word generated by the decoder in step t, and MLP() represents the output to the vocabulary space through multi-layer perceptron mapping;
[0065] Specifically, the Seq2seq-based dialogue generation model uses cross entropy as the loss function, expressed as:
[0066]
[0067] x represents the reference reply text, Represents the decoder generating text.
[0068] In one embodiment, the Figure 3 The CVAE model shown in the figure describes the dialogue generation model training method for enhancing text features of the present invention, including:
[0069] S21. Get question text C = c1, c2, ..., c m and reply text X=x1,x2,...,x n , c u represents the u∈{1,2,...,m}th word in the question text, x v represents the v∈{1,2,...,n}th word in the reply text, m represents the number of words in the question text, and n represents the number of words in the reply text. The keywords in the question text are extracted by the TextRank algorithm to obtain the keyword sequence K=k1,k2,...,k t , k i represents the i∈{1,2,...,t}th keyword;
[0070] S22. The question text vector of the question text is obtained by the input encoder, the semantic encoder obtains the semantic vector, and the output encoder obtains the reply text vector of the reply text, which is expressed as:
[0071] h ci ,c ci =Enc in (e(c i ),(h ci-1 ,c ci-1 ))
[0072] h xi ,c xi =Enc out (e(x i ),(h xi-1 ,c xi-1 ))
[0073] h s =Enc sem (e(S),(h0,c0))
[0074] h ci ,c ci denote the hidden state of the input encoder at step i and the cell state at step i, respectively, xi ,c xi They represent the hidden state of the i-th step of the output encoder and the cell state of the i-th step, Enc out () indicates the output encoder, Enc in () indicates the input encoder, Enc sem () represents the semantic encoder, h s represents the semantic vector, h0 represents the initial hidden state of the semantic encoder, c0 represents the initial cell state of the semantic encoder, and S represents the semantic category of the question text;
[0075] S23. Introduce a keyword encoder, which encodes each keyword through an attention mechanism to obtain a corresponding keyword vector;
[0076] S24. The keyword vector and the semantic vector are concatenated and input into the first multi-layer perceptron to obtain a keyword semantic vector containing rich semantics, which is expressed as:
[0077] h'=MLP([h t :h s ])
[0078] h' represents the keyword semantic vector, h t represents the keyword vector, h s represents the semantic vector, MLP() represents the first multi-layer perceptron;
[0079] S25. After concatenating the keyword semantic vector and the question text vector, pass them through a second multi-layer perceptron to obtain a first fusion feature, and input the first fusion feature into a priori network to obtain a priori distribution parameter;
[0080] S26. The keyword semantic vector, the question text vector and the reply text vector are concatenated and passed through the third layer perceptron to obtain a second fusion feature, and the second fusion feature is input into the recognition network to obtain an approximate posterior distribution parameter;
[0081] S27. Reparameterize the approximate posterior distribution parameters to obtain latent variables, initialize the latent variables through linear transformation, and obtain an input vector;
[0082] S28. Train the dialogue generation model based on the input vector and the reply text, use the loss function to calculate the loss value and back propagate, and adjust the parameters of the dialogue generation model.
[0083] Specifically, the CVAE-based dialogue generation model uses reconstruction loss and KL distance as the loss function, expressed as:
[0084]
[0085] Among them, q φ (z|X,C,K,S) represents the approximate posterior distribution, represents the prior distribution, represents the expectation of reconstructing the reply text X under the approximate posterior distribution, KL represents the KL divergence of the two distributions, X represents the reply text, C represents the question text, K represents the keyword text, S represents the semantic text, and z represents the latent variable.
[0086] In one embodiment, the keyword encoder encodes each keyword through an attention mechanism to obtain a corresponding keyword vector, including:
[0087] h t =Enc key (e(K))
[0088] Enc key (e(k i ))=LSTM(input i )
[0089]
[0090]
[0091] Among them, Enc key () represents the keyword encoder, K = k1, k2, ..., k t represents the keyword sequence, k i represents the i-th keyword, h t represents the hidden state of the keyword encoder at step t, which is also the keyword vector in this embodiment, e() represents the calculated word vector, and input i Indicates keyword k i The weighted vector of LSTM() represents the long short-term memory network LSTM, α(e(k i ),e(k j )) indicates keyword k i With keyword k j The attention weight between them, and 1≤j<i.
[0092] In one embodiment, the present invention provides a dialogue generation system for enhancing text features, including a sample module, a keyword extraction module, an encoding module, a fusion module, and a training module, wherein:
[0093] A sample module is used to obtain multiple groups of dialogue samples, where the dialogue samples include question texts and response texts;
[0094] A keyword extraction module is used to extract multiple keywords from the question text to form a keyword sequence;
[0095] The encoding module includes an input encoder, an output encoder, a keyword encoder and a semantic encoder, which are used to encode the data of the sample module and the keyword extraction module;
[0096] The fusion module includes a first fusion module and a second fusion module:
[0097] The first fusion module is used to fuse the keyword vector and the semantic vector to obtain a keyword semantic vector containing rich semantics;
[0098] The second fusion module is used to fuse the keyword semantic vector and the question text vector;
[0099] The training module is used to train the dialogue generation model. It uses the loss function to calculate the loss and back-propagate to adjust the parameters of the dialogue generation model.
[0100] Specifically, the system also includes a third fusion module, a priori network and a recognition network:
[0101] The third fusion module is used to fuse the keyword semantic vector, the question text vector and the reply text vector;
[0102] A priori network, used to receive the output result of the second fusion module and obtain prior distribution parameters;
[0103] The recognition network is used to receive the output result of the third fusion module and obtain the approximate posterior distribution parameters.
[0104] Table 1 Evaluation results on the DAILYDIALOGS dataset with behavior labels
[0105]
[0106] Table 2 Evaluation results of DAILYDIALOGS dataset with sentiment labels
[0107]
[0108] Table 3 Evaluation results of the EMPATHETICDIALOGUES dataset with emotion labels
[0109]
[0110] Tables 1, 2, and 3 show the automatic evaluation results of our model and other models. Table 1 shows the results of using motivational semantics on the DailyDialog dataset. Table 2 shows the results of using emotional semantics on the DailyDialog dataset. Table 3 shows the results of using emotional semantics on the EmpatheticDialogues dataset.
[0111] We can see that our model has achieved good results on three different datasets. Our model is superior to other comparison models in BLEU and METEOT, and is only lower than Transformer in Rouge. This shows that our model can generate higher quality dialogue texts, especially on different datasets, so our method has good generalization ability. However, after adding our method, the error bar of the model becomes higher, which shows that although adding our method can improve the generation effect of the dialogue model, it will make the model more complex, resulting in greater fluctuations in the model.
[0112] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.
[0113] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.< / sos>
Claims
1. A method for generating dialogues with enhanced text features, characterized in that: The following steps are involved: S1. Obtain the question text and the reply text, extract the keywords in the question text through the TextRank algorithm, and obtain the keyword sequence; obtain the question text vector of the question text by inputting the encoder; The semantic encoder is used to obtain the semantic vector, including: h s =Enc sem (e(S),(h0,c0)) Among them, Enc sem () represents the semantic encoder, h s represents the semantic vector, h0 represents the initial hidden state of the semantic encoder, c0 represents the initial cell state of the semantic encoder, and S represents the semantic text corresponding to the question text; S2. Introduce a keyword encoder, which encodes the keyword sequence through an attention mechanism to obtain a keyword vector; S3. concatenate the keyword vector and the semantic vector and input them into the first multi-layer perceptron to obtain a keyword semantic vector containing rich semantics; S4. concatenate the keyword semantic vector and the question text vector and pass them through the second multi-layer perceptron to obtain an input vector; S5. Train the dialogue generation model according to the input vector and the reply text, calculate the loss value using the loss function and perform back propagation, and adjust the parameters of the dialogue generation model; S6. Input the text to be replied into the trained dialogue generation model to generate a dialogue.
2. The method for generating a dialogue with enhanced text features according to claim 1, characterized in that: Step S4 also includes: Using the output encoder to obtain a reply text vector of the reply text; The keyword semantic vector and the question text vector are concatenated and passed through the second layer perceptron to obtain the first fusion feature, and the first fusion feature is input into the prior network to obtain the prior distribution parameter; The keyword semantic vector, question text vector and reply text vector are concatenated and passed through the third layer perceptron to obtain the second fusion feature, and the second fusion feature is input into the recognition network to obtain the approximate posterior distribution parameter; The parameters of the approximate posterior distribution are reparameterized to obtain latent variables, which are initialized through linear transformation to obtain the input vector.
3. The method for generating a dialogue with enhanced text features according to claim 1, characterized in that: The keyword encoder encodes the keyword sequence through the attention mechanism to obtain the keyword vector, including: h t =Enc key (e(K)) Enc key (e(k i ))=LSTM(input i ) Among them, Enc key () represents the keyword encoder, K = k1, k2, ..., k t represents the keyword sequence, k i represents the i-th keyword, h t represents the keyword vector, e() represents the word vector of the calculated word, input i Indicates keyword k i The weighted vector of LSTM() represents the long short-term memory network LSTM, α(e(k i ),e(k j )) indicates keyword k i With keyword k j The attention weight between them, and 1≤j<i.
4. A dialog generation system for implementing a dialog generation method for enhancing text features as claimed in any one of claims 1 to 3, characterized in that: It includes sample module, keyword extraction module, encoding module, fusion module and training module, among which: A sample module is used to obtain multiple groups of dialogue samples, where the dialogue samples include question texts and response texts; A keyword extraction module is used to extract multiple keywords from the question text to form a keyword sequence; The encoding module includes an input encoder, an output encoder, a keyword encoder and a semantic encoder, which are used to encode the data of the sample module and the keyword extraction module; The fusion module includes a first fusion module and a second fusion module: The first fusion module is used to fuse the keyword vector and the semantic vector to obtain a keyword semantic vector containing rich semantics; The second fusion module is used to fuse the keyword semantic vector and the question text vector; The training module is used to train the dialogue generation model. It uses the loss function to calculate the loss and back-propagate to adjust the parameters of the dialogue generation model.
5. A dialog generation system for enhancing text features according to claim 4, characterized in that: It also includes the third fusion module, prior network and recognition network: The third fusion module is used to fuse the keyword semantic vector, the question text vector and the reply text vector; A priori network, used to receive the output result of the second fusion module and obtain prior distribution parameters; The recognition network is used to receive the output result of the third fusion module and obtain the approximate posterior distribution parameters.
