A joint latent variable based variational auto-regressive dialog generation apparatus and method

Abstract

The application discloses a kind of variational autoregressive dialogue generation device and method based on joint latent variable, including encoder module, knowledge selection module and variational autoregressive decoder module;Encoder module is used to encode dialogue context and knowledge sentence set into vector form feature representation respectively, wherein knowledge sentence set is obtained by searching external text library to dialogue context, and is provided with several labeled knowledge sentences;The pre-training language model based on multilayer self-attention mechanism constructs word-level and sentence-level encoding representation;Knowledge selection module selects the most relevant knowledge sentence with dialogue context semantics from knowledge sentence set based on dialogue context and the given knowledge sentence set, for reply generation;Variational autoregressive decoder module includes variation layer and stacked decoding layer, and variation layer is used to calculate to obtain reply sequence latent variable, and finally generates the final reply sentence through stacked decoding layer.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a dialogue generation device and method for fusing external unstructured knowledge, knowledge selection and knowledge perception in dialogue generation. BACKGROUND

[0002] The dialogue generation technology represented by sequence-to-sequence has been relatively mature, but tends to generate short replies and ordinary replies, mainly because the dialogue system lacks knowledge. Knowledge is crucial to understanding and generating language, and external knowledge such as common sense and background knowledge is an important source of information for organizing dialogue sentences. Unstructured external knowledge such as encyclopedic articles, domain knowledge documents and social media comments is more numerous and easier to obtain than structured knowledge (knowledge graph). A dialogue system that fuses external unstructured knowledge can identify entities or topics discussed by the user and link them to real-world facts, such as retrieving relevant background information and introducing new dialogue topics to actively engage with the user. It can also be trained by continuously increasing knowledge, making it highly scalable. Therefore, it is necessary to introduce external unstructured knowledge into the dialogue system to improve the quality of dialogue replies and generate more informative and diverse dialogues.

[0003] In a dialogue system that fuses unstructured external knowledge, the typical task framework includes two sub-tasks: knowledge selection and reply generation. Existing research on knowledge selection methods includes using reply sentences as posterior information and labeled knowledge as posterior information. For example, the PostKS algorithm uses reply sentences as posterior information and does not rely on labeled knowledge to model the posterior distribution of knowledge selection. By narrowing the gap between the prior distribution and the posterior distribution of knowledge selection, the accuracy of knowledge is improved, and then the dialogue reply is generated. However, knowledge selection is based on single-turn dialogue. The SKLS algorithm models knowledge selection as a sequence hidden variable for multi-turn dialogue scenarios, and relies on dialogue context and historical selected knowledge to improve the accuracy of knowledge selection. The PIPM algorithm further improves the SKLS algorithm to address the problem that the prior knowledge selection module does not obtain posterior information, which may lead to inaccurate knowledge selection during testing, thereby affecting the knowledge-aware reply generation. It proposes to predict the information of the reply sentence as a supplement, but its posterior information for knowledge selection is still through the reply sentence. Some researchers have proposed a difference-aware knowledge selector that considers the difference between the selected knowledge in the current round and the previous round to help smooth the transition of knowledge selection. In addition, the DukeNet algorithm considers knowledge tracking and knowledge drift, and improves their accuracy based on the dual learning paradigm; the CoLV algorithm uses labeled knowledge as posterior information and considers the relationship between knowledge selection and reply generation, and improves the diversity of reply generation through a joint hidden variable model, but the hidden variable of reply generation is a global single hidden variable.

[0004] Through research and analysis, the following challenges were found in the existing technology: (1) The posterior distribution of the knowledge selection latent variables is calculated through the dialogue context and label knowledge, while the prior distribution is calculated only through the dialogue context. The gap between the prior and posterior distributions can lead to unreasonable choices in the knowledge selection stage, thereby affecting the quality of the generated response statements; (2) The latent variables of response generation are single latent variables at the global and sentence levels, which are insufficient to model complex semantics and the diversity of response generation, and may lead to the neglect of global latent variables of response generation during the autoregressive decoding process. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a variational autoregressive dialogue generation device and method based on joint latent variables. This method not only utilizes supplementary posterior information to improve the accuracy of knowledge selection, but also models the diversity of knowledge selection and response generation through a conditional variational autoencoder. At the same time, it uses a variational autoregressive decoder to model the latent variables of the responses at each step of the decoding process, making the responses more diverse and informative.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A variational autoregressive dialogue generation device based on joint latent variables includes an encoder module, a knowledge selection module, and a variational autoregressive decoder module.

[0008] The encoder module is used to encode the dialogue context and the set of knowledge sentences into feature representations in vector form, and sets several labeled knowledge sentences; a pre-trained language model based on a multi-layer self-attention mechanism is used to construct word-level and sentence-level encoding representations;

[0009] The knowledge selection module selects the knowledge sentences most semantically relevant to the dialogue context from the given set of knowledge sentences for response generation, based on the dialogue context and the current set of knowledge sentences. Specifically, it models the posterior and prior distributions of the knowledge selection latent variables, where the knowledge selection latent variables represent latent variables that follow a category distribution on the set of knowledge sentences based on the dialogue context. The posterior distribution is modeled using labeled knowledge sentences and the dialogue context. The prior distribution is modeled using predicted posterior information and the dialogue context, where predicted posterior information refers to the predicted response statement information. During the training phase, the prior and posterior distributions are made to approximate each other, enabling the use of the prior distribution to select knowledge sentences and feed them into the variational autoregressive decoder module during the testing phase.

[0010] The variational autoregressive decoder module includes a variational layer and a stacked decoding layer, the variational layer is used to calculate a reply sequence latent variable, in a training stage, the calculation of the reply sequence latent variable includes a posterior sequence latent variable and a prior sequence latent variable, and a selected reply sequence latent variable and a decoding hidden state of a prior path are fused and transmitted to the decoding layer, wherein the selected reply sequence latent variable is a posterior and prior reply sequence latent variable in a training and testing stage respectively, and the decoding hidden state of the prior path refers to a hidden state representation of a generated reply sequence through the variational layer; finally, a final reply sentence is generated through the stacked decoding layer.

[0011] Further, comprising:

[0012] S1, a pre-training language model BERT based on a self-attention mechanism is used to encode a dialogue context and a knowledge sentence set, and word-level and sentence-level encoding representations are constructed;

[0013] S2, posterior distribution and prior distribution of a knowledge selection latent variable are modeled; wherein the posterior distribution is modeled by a labeled knowledge sentence and a dialogue context, the prior distribution is modeled by predicted posterior information and a dialogue context, and a latent variable is sampled from the prior distribution to select a knowledge sentence to input into a variational autoregressive decoder module;

[0014] S3, a reply sentence is generated through a variational autoregressive decoder module; the variational autoregressive decoder module includes a variational layer and a stacked decoding layer, the variational layer calculates posterior distribution and prior distribution of a reply sequence latent variable, and a selected reply sequence latent variable and a decoding hidden state of a prior path are fused and transmitted to the decoding layer, wherein the selected reply sequence latent variable is a posterior and prior reply sequence latent variable in a training and testing stage respectively, and the decoding hidden state of the prior path refers to a hidden state representation of a generated reply sequence through the variational layer; the stacked decoding layer generates a reply sentence based on the output of the variational layer, a dialogue context and a selected knowledge.

[0015] Further, in step S1, given a current round dialogue context C t and a knowledge sentence set K t , a BERT model is used to encode to obtain corresponding word-level feature representations to obtain dialogue context word-level representation vectors and sentence-level representation vectors after average pooling For any sentence K t,l in the knowledge sentence set K t , word-level and sentence-level representation vectors and The overall sentence-level representation of the knowledge sentence set K t is denoted as L is the size of the knowledge sentence set, and d represents the dimension of the hidden state.

[0016] Further, step S2 comprises:

[0017] S2.1, calculating the posterior distribution of the knowledge selection latent variable : concatenating the dialogue context and the selected knowledge representation vector, i.e. , and performing dot product attention calculation with the sentence-level representation , obtaining the posterior latent variable representation after the softmax normalization layer

[0018] S2.2, calculating the prior distribution of the knowledge selection latent variable ; using the predicted reply sentence information to supplement the prior information of knowledge selection, performing dot product attention calculation on the dialogue context and the sentence-level representation , i.e. representing the knowledge sentence set fusion feature representation perceived by the dialogue context, Attn represents the method of calculating attention;

[0019] concatenating the dialogue context and the knowledge sentence set fusion feature representation, i.e. , obtaining the word probability distribution T of the predicted reply sentence on the BERT vocabulary through the multi-layer perception, and then performing weighted summation on the embedding representation E of the entire BERT vocabulary and the dialogue context feature representation , obtaining , and then obtaining , and then performing dot product attention calculation with the sentence-level representation , obtaining the attention score on the knowledge sentence set as the prior latent variable after the softmax normalization layer

[0020] S2.3, sampling the prior latent variable from the prior distribution of the knowledge selection latent variable to select the knowledge sentence K t,sel : that is, first sampling from the prior latent variable, and the prior latent variable is subject to a categorical distribution, , and then sending the corresponding knowledge sentence into the variational auto-regressive decoder module, ~ representing subjecting to a probability density function; the reconstruction loss of the knowledge selection task in the knowledge selection module is where K t,a represents the label knowledge, i.e.

[0021] Further, step S3 comprises:

[0022] S3.1, modeling the posterior distribution of the reply sequence latent variable through the variational layer: in the training stage, the word embedding representation and position encoding of the input reply generation sequence Y t are added bit by bit as the initial input of the variational layer, denoted as S 0Furthermore, the posterior distribution parameters of the latent variables in the response sequence are obtained through variable stratification: the mean and variance vector, respectively. and Then, the posterior latent variables at time step n are obtained by sampling using reparameterization techniques. The latent variable of the response sequence before time step n is denoted as The following formula is abbreviated as: The latent variable representing the response sequence after sampling at time step n is a posterior distribution.

[0023]

[0024] Among them, C t For the previous round of dialogue, K t,sel For the knowledge sentence, I represents the identity matrix;

[0025] S3.2, Modeling the prior distribution of latent variables in the response sequence; including the training and testing phases, the prior distribution parameters of the latent variables in the response sequence are obtained through variable stratification: the mean and variance vector, respectively. and Then, prior latent variables for time step n are obtained by sampling using reparameterization techniques. Let Y be the response sequence before time step n. t,＜n Meanwhile, the following formula is abbreviated as: The latent variable of the response sequence represents the prior distribution of sampling at time step n.

[0026]

[0027] S3.3, In variable hierarchical processing, the latent variables of the response sequence are fused with the decoded latent states of the prior path; the decoded latent states in the prior path are fused with the selected latent variables of the response sequence. Further fusion yields the representation vector S. 1 The selected response sequence latent variables are the posterior and prior latent variables, respectively, during the training and testing phases; after layer normalization, the latent state vector fed into the decoding layer is...

[0028] S3.4 The stacked decoding layer generates response statements based on variable layer output, dialogue context, and selected knowledge: The stacked decoding layer can obtain the decoding generation probability, the probability of copying selected knowledge, and the probability of copying dialogue context, and then sum the above probabilities with weights to obtain the probability distribution of the final generated word in the BERT vocabulary; the word with the highest probability in the BERT vocabulary is obtained through the argmax function, and the final response statement is obtained by concatenating the above words with the highest probability.

[0029] Preferably, the stacked decoding layer is based on the Transformer decoder layer architecture, denoted as TRS_Decs, to obtain the hidden state representation of the last layer.

[0030]

[0031] The stacked decoding layers use a copy mechanism, and the final probability distribution of generated words is a weighted sum of the probability distributions generated by the decoding layer, the copy of selected knowledge, and the copy of the dialogue context. After the generated word sequence is processed by the decoder, the hidden state at the final position and its generation probability distribution on the BERT vocabulary are obtained. The hidden state at the final position is processed with the word-level vector representations of the selected knowledge and the dialogue context to obtain the corresponding probability distribution and relevance score of the copied word. The relevance scores are then weighted and summed with the word-level vector representations of the selected knowledge and the dialogue context to obtain the attention representations of the selected knowledge and the dialogue context. The hidden state at the final position, the selected knowledge, and the attention representations of the dialogue context are then processed by a multilayer perceptron to obtain the weight coefficients of the three probability distributions. Finally, the weight coefficients are weighted and summed with the corresponding probability distributions to obtain the final probability distribution of the generated words on the BERT vocabulary.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the variational autoregressive dialogue generation method based on joint latent variables.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the variational autoregressive dialogue generation method based on joint latent variables.

[0034] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0035] 1) This invention uses a set of knowledge sentences and the context of the dialogue to predict the information of the response statement, which in turn serves as supplementary information for calculating the prior distribution of the knowledge selection latent variable. By modeling the knowledge selection task through the knowledge selection latent variable, the prior distribution of the knowledge selection latent variable approximates the posterior distribution during the training phase, enabling the selection of knowledge sentences that are more semantically relevant to the context of the dialogue, thus resulting in higher accuracy of the knowledge selection task during the testing phase.

[0036] 2) This invention uses a variational autoregressive decoder to model the latent variables of the response sequence, that is, to model the latent variables of the response at each decoding time step. Therefore, the generated response statements are more diverse and informative, and are closer to human responses. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the variational autoregressive dialogue generation method in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the operation of the variational autoregressive decoder module in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0040] This embodiment provides a variational autoregressive dialogue generation device based on joint latent variables, including an encoder module, a knowledge selection module, and a variational autoregressive decoder module. The encoder module encodes the dialogue context and the set of knowledge sentences into vector-like feature representations, where the set of knowledge sentences is obtained by retrieving the dialogue context from an external text library and includes several labeled knowledge sentences. A pre-trained language model based on a multi-layer self-attention mechanism constructs word-level and sentence-level encoding representations, which are then used in the knowledge selection module and the variational autoregressive decoder module.

[0041] The knowledge selection module selects the knowledge sentences most semantically relevant to the dialogue context and the given set of knowledge sentences for response generation. Specifically, it models the posterior and prior distributions of the knowledge selection latent variables. The knowledge selection latent variables represent latent variables that follow a category distribution on the knowledge sentence set based on the dialogue context. The posterior distribution is modeled using labeled knowledge sentences and the dialogue context. The prior distribution is modeled using predicted posterior information and the dialogue context, where predicted posterior information refers to the predicted response statement. During the training phase, the prior and posterior distributions are made to approximate each other. During the testing phase, the prior distribution is used to select knowledge sentences for input into the variational autoregressive decoder module.

[0042] The variational autoregressive decoder module generates response statements based on the dialogue context and selected knowledge sentences, specifically including a variational layer and a stacked decoding layer. The variational autoregressive decoder module uses response sequence latent variables to model the diversity of response statements, where diversity refers to the presence of several different words in the response statement; that is, each decoding time step models its corresponding latent variable. The variational layer includes the calculation of response sequence latent variables. During the training phase, the calculation of response sequence latent variables includes posterior and prior sequence latent variables, and the selected response sequence latent variables and the decoding latent states of the prior path are fused and passed to the decoding layer. The selected response sequence latent variables are the posterior and prior response sequence latent variables during training and testing, respectively. The decoding latent states of the prior path refer to the latent state representation obtained by processing the generated response sequence through the variational layer. Finally, the final response is generated through stacked decoding layers.

[0043] Step 1: Encode the dialogue context and knowledge sentence set using the pre-trained language model BERT based on the self-attention mechanism, constructing word-level and sentence-level encoding representations: Given the current dialogue context C... t and the set of knowledge sentences K t The BERT model is used to encode and obtain the corresponding word-level feature representations. For example, the special symbols [CLS] and [SEP] are added to the beginning and end of the sentences in the dialogue, respectively. This operation is denoted as SP. Then, the data is fed into the BERT model to obtain the word-level representation vectors of the dialogue. Then use AvgPool average pooling to obtain sentence-level representation vectors. As shown in the following formula:

[0044]

[0045] Knowledge Sentence Set K t Any sentence K in t,l After undergoing the same processing as the preceding dialogue text, word-level and sentence-level representation vectors are obtained. and Knowledge Sentence Set K t The overall sentence-level representation vector is The matrix is ​​L, which is the size of the knowledge sentence set, and d represents the hidden state dimension.

[0046] Step 2: Model the posterior and prior distributions of the latent variables for knowledge selection. The posterior distribution is modeled using labeled knowledge sentences and dialogue context, while the prior distribution is modeled using predicted posterior information and dialogue context. Latent variables are sampled from the prior distribution to select knowledge sentences, which are then fed into the variational autoregressive decoder module. The specific processing in this step is as follows:

[0047] Step 2.1: Calculate knowledge selection of latent variables The posterior distribution is calculated by concatenating the representation vectors of the dialogue context and the selected knowledge, where the selected knowledge is the knowledge sentence corresponding to a chosen label; a dot product attention operation is performed with the sentence-level representation vectors of the entire set of knowledge sentences, followed by softmax normalization. The posterior distribution of the latent variables chosen for knowledge is abbreviated as: As shown in the following formula:

[0048]

[0049] Where [;] and T represent vector concatenation and transpose, respectively, and W post These are learnable parameters.

[0050] Step 2.2: Calculate the prior distribution of the latent variables for knowledge selection: This invention proposes using predicted response statement information to supplement the prior information for knowledge selection. First, dot product attention is calculated by combining the sentence-level representation vectors of the dialogue context and the knowledge sentence set. Then, this score is multiplied by the overall feature representation of the knowledge sentence set.

[0051]

[0052] Where q P (K t |C t W represents the attention score between the preceding dialogue and the set of knowledge sentences. P For learnable parameters, The set of knowledge sentences perceived in the context of the dialogue is fused with feature representation.

[0053] Next, the feature representations of the dialogue context and the knowledge sentence set are concatenated and processed by a multilayer perceptron (MLP) to predict the word probability distribution T of the response statement in the vocabulary. This distribution T is then weighted and summed with the overall embedding representation E of the vocabulary to obtain a bag-of-words semantic representation of the predicted response statement, which approximates the representation of the actual response statement. This is used to narrow the gap between prior and posterior information in knowledge selection tasks. Finally, the dialogue context and the predicted response information are concatenated to form a new query vector, as shown in the following equation:

[0054]

[0055]

[0056] Where |V| represents the vocabulary size. The knowledge selected for construction refers to prior information. Here are the learnable parameters, and E represents the vocabulary embedding matrix. The prior distribution of the latent variable for knowledge selection is denoted as . This embodiment uses KL distance loss constraint knowledge to select the distribution distance between the posterior and prior distributions in the task, i.e.

[0057] Step 2.3: Select latent variables from knowledge. Sampling prior latent variables from the prior distribution to select knowledge sentence K t,sel That is, sampling is first performed from prior latent variables, which follow a categorical distribution. The corresponding knowledge sentence is then fed into the variational autoregressive decoder module, where ~ indicates that it follows a probability density function. The reconstruction loss of the knowledge selection task within the knowledge selection module is... Where K t,a Represents label knowledge, that is

[0058] Step 3: Model the latent variables of the response sequence and the response generation using the variational autoregressive decoder module. The variational autoregressive decoder module includes a hierarchical and stacked decoding layer. The hierarchical layer calculates the posterior and prior distributions of the latent variables of the response sequence and fuses the selected latent variables of the response sequence and the decoded latent states of the prior path, which are then passed to the decoding layer. The selected latent variables of the response sequence are the posterior and prior latent variables of the response sequence during the training and testing phases, respectively. The decoded latent states of the prior path refer to the latent state representation obtained by the decoder after the generated response sequence is processed. The stacked decoding layer generates the response statement based on the output of the hierarchical layer, the dialogue context, and the selected knowledge. The specific processing of this step is as follows:

[0059] Step 3.1: Model the posterior distribution of the latent variables of the response sequence through variable hierarchical modeling: During the training phase, the input response sequence Y is generated. t The word embedding representation and positional encoding are added bit-by-bit and used as the initial input for the variable hierarchical structure, denoted as S. 0 .like Figure 2 As shown, this embodiment uses different mask matrices to simultaneously calculate the parameters of the posterior and prior distributions of the latent variables in the response sequence during the training phase, without the need for reverse encoding of the response sequence.

[0060] The posterior distribution parameters of the latent variables in the response sequence are obtained through variable stratification: the mean and variance vectors, respectively. and As shown in the following formula:

[0061]

[0062]

[0063] in This represents a variable hierarchical structure based on the Transformer model. and This represents a multilayer perceptron network, where `softplus` represents the activation function to ensure that the variance is always positive. The posterior latent variables at time step `n` are then sampled using a reparameterization technique. The following formula is abbreviated as: Represents the latent variables of the response sequence from the posterior distribution sampled at time step n. As shown in the following formula:

[0064]

[0065]

[0066] Where n∈[0,N] represents the time step, and ε is derived from the standard normal distribution. The vector sampled in the middle, ⊙ denotes the element-wise multiplication of the vector. Let I represent the latent variable of the response sequence up to time step n, and let I represent the identity matrix.

[0067] Step 3.2: Modeling the prior distribution of the latent variables in the response sequence: This includes the training and testing phases. The steps are similar to calculating the posterior distribution of the latent variables in the response sequence in Step 3.1. A masking operation is required when calculating the multi-head attention. Further, the mean vector of the prior distribution of the latent variables in the response sequence is obtained. Sum of variance vectors And the prior latent variable of time n obtained by sampling through reparameterization technique. The following formula is abbreviated as: The latent variable of the response sequence represents the prior distribution of sampling at time step n.

[0068]

[0069] This embodiment uses KL distance loss to constrain the probability distribution distance between the posterior and prior distributions of the sequence latent variables in the recovery generation task, as shown in the following equation:

[0070]

[0071] Step 3.3: Fusion of latent variables in the response sequence and the decoded latent state of the prior path in the variable hierarchical process: The decoded latent state of the prior path is fused with the selected latent variables in the response sequence. Further fusion yields the representation vector S. 1 After layer normalization, the hidden state vector fed into the decoding layer is... The latent variables selected during the training and testing phases are the posterior and prior latent variables, respectively.

[0072] Step 3.4: The stacked decoding layers generate response statements based on the variable layer output, dialogue context, and selected knowledge. The stacked decoding layers are denoted as TRS_Decs, such as... Figure 2 As shown, each layer includes two cross-attention sublayers, one for focusing on the context of the dialogue and the other for focusing on the selected knowledge. The input received by the decoding layer is... After processing through stacked decoding layers, the hidden state representation of the last layer is obtained. As shown in the following formula:

[0073]

[0074] Assuming the generated word sequence is After this sequence is processed by the variational autoregressive decoder, the hidden state representation of the last position in the last layer is obtained, denoted as . After performing full connection and normalization operations, we can obtain the word y generated in step j.t,j The probability score p on the vocabulary θ (y t,j |Gen), as shown in the following formula:

[0075]

[0076] Using a decoding layer with a copy mechanism, the probability distribution of the final generated words is the weighted sum of the probability distributions generated by the decoding layer, the probability distribution of copying the selected knowledge, and the probability distribution of copying the context of the dialogue.

[0077] The hidden state is represented at decoding time step j. Word-level representation of selected knowledge Perform projection transformations to obtain new representation vectors. and The correlation score of the dot product of the two is then normalized. This score is used as the probability score of the corresponding word in the copied knowledge, and is calculated as follows:

[0078]

[0079]

[0080] in Representation of hidden state The normalized fraction of the product of the selected knowledge points. K represents the i-th word in the selected knowledge. t,sel,i With decoding state The correlation score, p θ (y t,j |CP k For decoding time step j, copy the selected knowledge with y. t,j Same word K t,sel,i The probability sum.

[0081] Similarly, the probability score p of copying words from the preceding dialogue. θ (y t,j |CP c Similarly, the hidden state will be decoded. And the word-level representation in the dialogue context Perform projection transformations to obtain new representation vectors. and The correlation score of the dot product of the two is then normalized. This score is used as the probability of copying the corresponding word in the dialogue above, and is calculated as follows:

[0082]

[0083]

[0084] in Representation of hidden state And the normalized fraction of the dot product operation mentioned above. Indicates the dialogue above C t The i-th word C t,i With decoding state The correlation score, p θ (y t,j |CP c () is the decoding time step j, copying the dialogue above and y. t,j Same word C t,i The probability sum.

[0085] The weighting coefficients λ1, λ2, and λ3, which are the sum of the three probabilities, are calculated using a multilayer perceptron (MLP). Abbreviated as p θ (y t,j ), representing the final generation probability score, as shown in the following formula:

[0086]

[0087]

[0088] p θ (y t,j )=λ1p θ (y t,j |Gen)+λ2p θ (y t,j |CP k )+λ2p θ (y t,j |CP c )

[0089] in and These represent the hidden state at the current time step. Attention is indicated in relation to the selected knowledge and the context of the dialogue. θ (y t,j ) indicates that time step j outputs the vocabulary word y. t,j The final probability score.

[0090] The loss for the response generation task is the negative log-likelihood loss, as shown in the following formula:

[0091]

[0092] Step 3.5: Model Training: The variational autoregressive dialogue generation model framework proposed in this embodiment has two sub-tasks: knowledge selection and response generation. The loss for the knowledge selection sub-task includes the KL distance loss from step 2.2. and the reconstruction loss in step 2.3 The loss from the response generation subtask includes the KL distance loss from step 3.2. And the negative log-likelihood loss in step 3.4

[0093] The optimization objective of the complete variational autoregressive dialogue generation model is the sum of the above four losses: γ1, γ2, γ3 and γ4 are the weights of each part of the loss. In this experiment, all four weights are set to 1.0.

[0094] This embodiment supplements prior information for knowledge selection with predicted response statements to improve the accuracy of knowledge selection. It uses a conditional variational autoencoder to model the diversity of knowledge selection and a variational autoregressive decoder to model latent variables in the response sequence, i.e., modeling latent variables at each decoding time step, thereby modeling higher-dimensional response statement diversity. This embodiment was tested on the publicly available open-domain knowledge-based dialogue dataset Wizard-of-Wikipedia (WoW), which contains labeled data for knowledge selection tasks. Each data sample in the WoW dataset includes the dialogue context, the corresponding set of knowledge sentences, and the response statement; sourced from https: / / parl.ai / projects / wizard_of_wikip edia / .

[0095] The knowledge in the WoW dataset comes from encyclopedia websites. In the dialogue, one party is the learner (Apprentice), and the other is the expert (Wizard). The expert can acquire unstructured knowledge (paragraphs or sentences) related to the dialogue and respond to the learner with reference to this external knowledge. The learner cannot see the knowledge content selected or used by the expert. Active interaction between both parties ensures the dialogue continues. This dataset contains 1365 open-domain dialogue topics, divided into a training set, a validation set, and two test sets. One test set contains topics that have appeared in either the training or validation set (Test Set 1 is TestSeen), while the other tests the model's zero-shot learning ability, meaning the topics in the test set have never appeared in either the training or validation set (Test Set 2 is Test Unseen). The model needs to model the expert role. This invention retains the original partitioning method of the WoW dataset, as shown in Table 1, which contains the dataset's statistical information.

[0096] Table 1

[0097]

[0098] This embodiment uses information from an external knowledge sentence set to predict response information, thereby supplementing the prior information of knowledge selection. This makes the prior distribution of knowledge selection during the training phase closer to the posterior distribution, improving the accuracy of knowledge selection. A conditional variational autoencoder is used to model the diversity of knowledge selection. At the same time, a variational autoregressive decoder is used to model the latent variables of the response sequence, that is, to model the latent variables of the response at each decoding time step, thereby modeling a higher-dimensional diversity of response statements. This enables the model to select external knowledge that is more relevant to the context of the dialogue, and the generated response content is more informative and diverse.

[0099] This embodiment compares the model with several knowledge-based dialogue generation baseline models, namely MemNet, PostKS, SKLS, DukeNet, and CoLV. The performance of the model of this invention is evaluated using accuracy (ACC), response quality evaluation BLEU-4, ROUGE-2 (RG-2), and response diversity (Dist-2) metrics. Table 2 shows the experimental results of the baseline model and the model of this invention.

[0100] Table 2

[0101]

[0102] As shown in Table 2, the variational autoregressive dialogue generation model proposed in this invention outperforms the compared baseline models (except for RG-2 in Test Unseen) on multiple metrics, demonstrating the effectiveness and strong generalization of the proposed model. Regarding knowledge selection accuracy, this invention shows a certain degree of improvement over the strong baseline CoLV model on both test sets. This is attributed to the use of supplementary posterior information in the knowledge selection task, allowing the prior distribution of the knowledge selection latent variables to learn the pattern of the posterior distribution during training. However, due to the diversity of knowledge selection, the improvement in the ACC metric is not significant, as multiple suitable knowledge points fit the current dialogue context. In the response generation task, this invention improves the BLEU-4 metric compared to all baseline models, indicating that the model can generate response sentences that conform to human habits and are more informative, further proving the effectiveness of jointly using knowledge selection and response generation latent variables. In terms of response diversity, this invention significantly outperforms the baseline model on the Dist-2 metric, indicating that introducing response sequence latent variables can improve the diversity of response sentences in the dialogue generation model. Simultaneously, the introduction of a copy mechanism allows the model to learn to copy words from the dialogue context and knowledge.

[0103] This embodiment also includes an ablation experiment to demonstrate the effectiveness of each module in the proposed model framework, as shown in Table 3, which presents the ablation experiment results. The control groups in the ablation experiment are set as follows:

[0104] (1) w / o PI: In knowledge selection latent variable modeling, posterior prediction information is not used, only the dialogue context is used;

[0105] (2) w / o Kvar: Remove the knowledge selection latent variable and use the knowledge corresponding to the highest attention score;

[0106] (3) w / o Y var-seq: The latent variable of the response sequence is changed into a single latent variable, and concatenated with the above text and sent to the decoder;

[0107] (4) w / o Y var: Remove the latent variable modeling module of the response sequence and directly decode and output;

[0108] (5) w / o all: becomes the basic encoder-decoder model;

[0109] Apart from the modifications mentioned, the other parts of the above five ablation experiment settings are the same as the complete model of this invention.

[0110] Table 3

[0111]

[0112] The ablation experiments yielded the following conclusions: After removing posterior prediction information, the accuracy of knowledge selection decreased, affecting the quality of response generation, with a slight decrease in the BLEU-4 score. Removing the knowledge selection latent variable significantly reduced the ACC score on both the Test Seen and Test Unseen test sets. This indicates that the model, relying solely on attention scores to select external knowledge, may make inappropriate choices, resulting in a lack of informative content in the responses. Changing the response sequence latent variable to a single latent variable reduced the response generation index, highlighting the importance of modeling the sequence latent variable for response diversity. Completely removing the response sequence latent variable further reduced the BLEU-4, RG-2, and Dist-2 scores, and also lowered the knowledge selection ACC score, demonstrating the effectiveness of simultaneously considering knowledge selection and response sequence latent variable modeling. Finally, with all latent variable modeling removed, the overall model of this invention becomes a basic encoder-decoder model, achieving performance close to MemNet.

[0113] The comparative and ablation experiments show that the present invention achieves improvements in knowledge selection and response generation on the publicly available knowledge-based dialogue dataset WoW compared to the baseline model. The design of each module within the model framework effectively contributes to the overall model performance. Therefore, the technical effects of the present invention are as follows: The present invention uses information from an external knowledge sentence set to predict response information, thereby supplementing the prior information of knowledge selection. This makes the prior distribution of the knowledge selection latent variables during the training phase closer to the posterior distribution, resulting in higher accuracy in knowledge selection during the testing phase. Because a variational autoregressive decoder is used to model the latent variables of the response sequence—that is, modeling the latent variables at each decoding time step—the generated response sentences are more diverse, more informative, and closer to human responses.

[0114] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps in the variational autoregressive dialogue generation method based on joint latent variables in the above embodiments. The electronic device specifically includes the following:

[0115] Processor, memory, communications interface, and bus;

[0116] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0117] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the variational autoregressive dialogue generation method based on joint latent variables in the above embodiments.

[0118] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the variational autoregressive dialogue generation method based on joint latent variables in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the variational autoregressive dialogue generation method based on joint latent variables in the above embodiments.

[0119] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0120] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially as shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A variational autoregressive dialogue generation device based on joint latent variables, characterized in that, It includes an encoder module, a knowledge selection module, and a variational autoregressive decoder module; The encoder module is used to encode the dialogue context and the set of knowledge sentences into feature representations in vector form, and sets several labeled knowledge sentences; a pre-trained language model based on a multi-layer self-attention mechanism is used to construct word-level and sentence-level encoding representations; The knowledge selection module selects the knowledge sentences most semantically relevant to the dialogue context from the given set of knowledge sentences for response generation, based on the dialogue context and the current set of knowledge sentences. Specifically, it models the posterior and prior distributions of the knowledge selection latent variables, where the knowledge selection latent variables represent latent variables that follow a category distribution on the set of knowledge sentences based on the dialogue context. The posterior distribution is modeled using labeled knowledge sentences and the dialogue context. The prior distribution is modeled using predicted posterior information and the dialogue context, where predicted posterior information refers to the predicted response statement information. During the training phase, the prior and posterior distributions are made to approximate each other, so that the prior distribution can be used to select knowledge sentences and feed them into the variational autoregressive decoder module during the testing phase. The variational autoregressive decoder module includes a variational layer and a stacked decoding layer. The variational layer is used to calculate the latent variables of the response sequence. During the training phase, the calculation of the latent variables of the response sequence includes the posterior and prior sequence latent variables. The selected latent variables of the response sequence and the decoded latent states of the prior path are fused and passed to the decoding layer. The selected latent variables of the response sequence are the posterior and prior latent variables of the response sequence during the training and testing phases, respectively. The decoded latent states of the prior path refer to the latent state representation obtained by the generated response sequence through the variational layer. Finally, the final response statement is generated through the stacked decoding layer.

2. A variational autoregressive dialogue generation method based on joint latent variables, based on the variational autoregressive dialogue generation device of claim 1, characterized in that, include: S1. The pre-trained language model BERT based on the self-attention mechanism encodes the dialogue context and knowledge sentence set to construct word-level and sentence-level encoding representations; S2. Model the posterior and prior distributions of the latent variables for knowledge selection; the posterior distribution is modeled using labeled knowledge sentences and dialogue context, and the prior distribution is modeled using predicted posterior information and dialogue context. Latent variables are sampled from the prior distribution to select knowledge sentences and send them to the variational autoregressive decoder module. S3. Generate response statements through the variational autoregressive decoder module; the variational autoregressive decoder module includes a variable-layer and stacked decoding layer. The variable-layer calculates the posterior and prior distributions of the latent variables of the response sequence and fuses the selected latent variables of the response sequence and the decoded latent states of the prior path and passes them to the decoding layer. The selected latent variables of the response sequence are the posterior and prior latent variables of the response sequence during the training and testing phases, respectively. The decoded latent states of the prior path refer to the latent state representation obtained by the generated response sequence through the variable-layer. The stacked decoding layer generates response statements based on the output of the variable-layer, the dialogue context, and the selected knowledge.

3. The variational autoregressive dialogue generation method based on joint latent variables according to claim 2, characterized in that, In step S1, given the context C of the current round of dialogue... t and the set of knowledge sentences K t The corresponding word-level feature representations are obtained by encoding using the BERT model to obtain the word-level representation vectors of the dialogue context. and the sentence-level representation vector after average pooling For the knowledge sentence set K t Any sentence K in t,l Similarly, word-level and sentence-level representation vectors are obtained. and Knowledge Sentence Set K t The overall sentence-level representation is denoted as L represents the size of the knowledge sentence set, and d represents the hidden state dimension.

4. The variational autoregressive dialogue generation method based on joint latent variables according to claim 2, characterized in that, Step S2 includes: S2.1 Calculating knowledge by selecting latent variables The posterior distribution: concatenates the dialogue context and the representation vector of the selected knowledge, i.e. Then, with sentence-level representation Perform dot product attention calculations, and obtain the posterior latent variable representation after passing through a softmax normalization layer. S2.2, Calculating knowledge by selecting latent variables The prior distribution; using predicted response information to supplement prior information for knowledge selection, combining dialogue context and sentence-level representation. Perform dot product attention calculation, i.e. The set of knowledge sentences representing the context of the dialogue is fused with feature representations, and Attn represents the method for calculating attention. The above dialogue context and knowledge sentence set are combined and their feature representations are concatenated, i.e. The word probability distribution T of the response is predicted using a multilayer perceptron on the BERT vocabulary. The word probability distribution T is then weighted and summed with the embedding representation E of the entire BERT vocabulary, and then summed with the contextual features of the dialogue. By piecing them together, we obtain After feature transformation, it is obtained Then, with sentence-level representation We perform dot product attention calculations, and then use the results through a softmax normalization layer to obtain attention scores on the knowledge sentence set, which are then used as prior latent variables. S2.3, Selecting Latent Variables from Knowledge Sampling prior latent variables from the prior distribution to select knowledge sentence K t,sel That is, sampling is first performed from prior latent variables, which follow a categorical distribution. The corresponding knowledge sentence is then fed into the variational autoregressive decoder module, where ~ indicates that it follows a probability density function; the reconstruction loss of the knowledge selection task within the knowledge selection module is... Where K t,a Represents label knowledge, that is 5. The variational autoregressive dialogue generation method based on joint latent variables according to claim 2, characterized in that, Step S3 includes: S3.1, Modeling the posterior distribution of latent variables in the response sequence using variable hierarchical modeling: During the training phase, the input response sequence Y is generated. t The word embedding representation and positional encoding are added bit-by-bit and used as the initial input for the variable hierarchical structure, denoted as S. 0 Furthermore, the posterior distribution parameters of the latent variables in the response sequence are obtained through variable stratification: the mean and variance vector, respectively. and Then, the posterior latent variables at time step n are obtained by sampling using reparameterization techniques. The latent variable of the response sequence before time step n is denoted as The following formula is abbreviated as: The latent variable representing the response sequence of the posterior distribution at time step n. Among them, C t For the previous round of dialogue, K t,sel For the knowledge sentence, I represents the identity matrix; S3.2, Modeling the prior distribution of latent variables in the response sequence; including the training and testing phases, the prior distribution parameters of the latent variables in the response sequence are obtained through variable stratification: the mean and variance vector, respectively. and Then, prior latent variables for time step n are obtained by sampling using reparameterization techniques. Let Y be the response sequence before time step n. t,＜n Meanwhile, the following formula is abbreviated as: The latent variable of the response sequence represents the prior distribution of sampling at time step n. S3.3, In variable hierarchical processing, the latent variables of the response sequence are fused with the decoded latent states of the prior path; the decoded latent states in the prior path are fused with the selected latent variables of the response sequence. Further fusion yields the representation vector S. 1 The selected response sequence latent variables are the posterior and prior latent variables, respectively, during the training and testing phases; after layer normalization, the latent state vector fed into the decoding layer is... S3.4 The stacked decoding layer generates response statements based on variable layer output, dialogue context, and selected knowledge: The stacked decoding layer can obtain the decoding generation probability, the probability of copying selected knowledge, and the probability of copying dialogue context, and then sum the above probabilities with weights to obtain the probability distribution of the final generated word in the BERT vocabulary; the word with the highest probability in the BERT vocabulary is obtained through the argmax function, and the final response statement is obtained by concatenating the above words with the highest probability.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the variational autoregressive dialogue generation method based on joint latent variables as described in any one of claims 2 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the variational autoregressive dialogue generation method based on joint latent variables as described in any one of claims 2 to 5.

Images