A multi-round dialogue response generation system and method based on relational graph attention network

Abstract

This invention belongs to the field of computer artificial intelligence for natural language generation and discloses a multi-round dialogue response generation system and method based on a relational graph attention network. The system comprises: obtaining multi-round dialogue input content and preprocessing it to obtain a semantic information representation of each utterance, then encoding the semantic information of each utterance sentence to obtain a semantic representation of the dialogue context; then using a graph attention network to capture the autocorrelations in the multi-round dialogue and the correlation features between the interlocutors, and introducing relational position encoding into the graph attention network to describe the sequence information of the utterances, thereby obtaining a high-level semantic representation of the graph encoding layer; finally, using the dialogue context semantic information representation and the high-level semantic representation of the relational graph attention encoding as input, and using a GRU model for decoding to generate the final dialogue response output representation. This invention significantly improves the quality of multi-round dialogue response generation, making the generated responses more coherent and meaningful.

Description

Technical Field

[0001] The present invention belongs to the field of computer artificial intelligence for natural language generation, and in particular relates to a multi-round dialogue response generation system and method based on a relational graph attention network. Background Art

[0002] With the booming internet and the rapid development of social media, a vast corpus of user dialogues has emerged, creating the conditions for data-driven dialogue systems. The immense research and commercial value of intelligent dialogue systems has attracted increasing attention from both academia and industry. Currently, dialogue systems can be categorized as task-driven, limited-domain dialogue systems and task-free, open-domain dialogue systems. Compared to the former, the latter offers greater practicality, scalability, and domain adaptability. Consequently, open-domain dialogue systems have become a hot topic among researchers.

[0003] Currently, systems can be categorized into retrieval-based models and generative models based on their implementation methods. Retrieval-based models use selection algorithms to select appropriate responses from a conversation corpus. While these responses are grammatically correct and factually objective, they suffer from issues such as monotony and topical restrictions. Generative models, on the other hand, employ natural language processing techniques to learn and understand the contextual information input by the user before gradually generating appropriate responses. Generative models can also be categorized into single-turn and multi-turn generative models based on whether they consider historical conversation information. Compared to single-turn generative models, multi-turn generative models require the system to understand complex contextual information, making them more challenging. However, current research in open-domain multi-turn generative models still faces numerous challenges, such as generic responses, lack of background knowledge, and inconsistency. Therefore, improving open-domain multi-turn generative systems holds great research value.

[0004] Recent research in this field has been primarily based on a sequence-to-sequence framework, focusing on how to effectively model contextual relationship information. However, previous research has rarely considered the discourse dependencies and temporal information between interlocutors. To address this issue, modeling the discourse dependencies between interlocutors is crucial, while also focusing on the temporal information of the interlocutors' discourse. This paper employs a relational graph attention network algorithm to model the discourse dependencies between interlocutors, fully exploiting the information between the different relation types of discourse uttered by the interlocutors; at the same time, relational position encoding is introduced to capture the sequence information contained in the discourse of the interlocutors, thereby making the generated responses more coherent, natural, and specific. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-round dialogue response generation system and method based on a relational graph attention network to solve the above-mentioned technical problems.

[0006] To solve the above technical problems, the specific technical solutions of the present invention, a multi-round dialogue response generation system and method based on a relationship graph attention network, are as follows:

[0007] A multi-round dialogue response generation system based on a relational graph attention network includes a sentence input encoding layer, a graph encoding layer, and a decoding layer. The sentence input encoding layer includes a word-level encoder and an utterance-level encoder; the word-level encoder encodes the words in each round of utterance input by the model, thereby obtaining the semantic representation of the utterance itself; the utterance-level encoder encodes the semantic representation of the model's utterance itself, thereby obtaining the semantic representation of the entire dialogue context; the graph encoding layer first uses a graph attention network to capture the autocorrelation of utterances in multi-round dialogues and the correlation characteristics between interlocutors, and then introduces relational position encoding to illustrate the sequence information containing utterances; the decoding layer generates a response reply based on the input contextual semantic information representation and the high-level semantic representation of the graph encoding layer.

[0008] The present invention also discloses a multi-round dialogue response generation method based on a relationship graph attention network, comprising the following steps:

[0009] Step 1: Obtain multiple rounds of dialogue input and preprocess it. Use the pre-trained BERT model to convert the word semantics in each round of utterance into corresponding vector representations, thereby obtaining the semantic information representation of each round of utterance itself. Then, use the Bi-GRU model to encode the semantic information of each round of utterance to obtain the semantic representation of the dialogue context.

[0010] Step 2: A graph attention network is used to capture the autocorrelation in multi-round conversations and the correlation features between interlocutors. Relation position encoding is introduced into the graph attention network to describe the sequence information of the utterances, thereby obtaining a high-level semantic representation at the graph encoding layer.

[0011] Step 3: Take the high-level semantic representation of the conversation context semantic information representation and the relationship graph attention encoding as input, and use the GRU model to decode and generate the final dialogue response output representation.

[0012] Furthermore, in the step 1, the process of preprocessing the input content of multiple rounds of dialogue to obtain the semantic information representation of each round of discourse itself includes: encoding the words in each round of input discourse, first using the BPE algorithm to perform sequence labeling representation on each round of discourse, and then inputting it into the pre-trained language BERT model for fine-tuning learning and training, thereby obtaining the semantic information representation of each round of discourse itself.

[0013] Furthermore, step 1 uses a word-level encoder and an utterance-level encoder for encoding, and includes the following specific steps:

[0014] The word-level encoder is used to generate a multi-turn dialogue context U={u1,...,u M}, first use the BPE algorithm to represent the sequence of each round of speech Where T i is the number of tokens in the i-th round of speech, which is then input into the pre-trained language BERT model for fine-tuning training. The word-level encoder encoding process is expressed by the following formula:

[0015]

[0016] Thus, the semantic information representation of each round of discourse is obtained

[0017] The Bi-GRU model is used as the encoding method of the discourse level encoder; the Bi-GRU model first obtains the semantic representation of each round of discourse itself from the upper word level encoder As input, each utterance is encoded by the Bi-GRU model. The encoding process of the utterance-level encoder is expressed by the following formula:

[0018]

[0019]

[0020] in is the i-th hidden layer representation in the forward GRU, is the i-th hidden layer representation in the backward GRU; concatenate the hidden layer representations of the forward GRU and the backward GRU to obtain a semantic representation containing contextual information Capture high-level feature information between multi-round dialogue utterances through a hierarchical structure.

[0021] Furthermore, obtaining the high-level semantic representation of the graph coding layer in step 2 includes the following steps:

[0022] Construct a directed graph for M sentences in a multi-round conversation and define the following:

[0023]

[0024] Define each statement in a multi-round conversation as a node v i , The relationship dependency information between each statement is defined as edge r, in The edge weight is defined as α ijr ,

[0025] (1) First, the context semantic representation output by the context encoding layer As node v i The initial vector representation of ;

[0026] (2) Based on the nodes, information edges r are constructed and the types of information edges r are defined into the following four categories: (a) self-before edge r1: the target utterance is related to the utterance before the utterance; (b) inter-before edge r2: the target utterance is related to the utterance before the utterance; (c) self-after edge r3: the target utterance is related to the utterance after the utterance; (d) inter-after edge r4: the target utterance is related to the utterance after the utterance;

[0027] (3) Relational Position Encodings is used to capture the temporal information between these four types of information. The relational position encoding process is expressed by the following formula:

[0028]

[0029] Among them, PE ijr Represents the target discourse u under relation type r i Its adjacent discourse u j The maximum relationship value is between [b, a], where b and a are the sliding window values ​​between the target discourse and other discourses. Represents the target discourse u under relation type r i Neighborhood;

[0030] (4) Calculate the weight of the relevant information edge, and the formula is as follows:

[0031]

[0032] where α ijr Represents the target discourse u under relation type r i Its adjacent discourse u j The edge weight between them, W r is the parameterized weight matrix under the attention mechanism, a r is the parameterized weight vector, T stands for transpose, LRL is the LeakyReLU activation function;

[0033] (5) By aggregating neighborhood The information is used to update the vector representation of each node The graph propagation mechanism encoding process is expressed by the following formula:

[0034]

[0035]

[0036] in is the trainable parameter weight matrix, L is the number of convolutional layers, and the final output is the high-level semantic representation of the graph encoding layer.

[0037] Furthermore, the step three includes the following steps:

[0038] The GRU model is used as the decoder to generate the response. The decoder decoding process is expressed by the following formula:

[0039]

[0040]

[0041] Where s0 is the initialization input of the decoder, W e and b e is a trainable parameter, is the concatenated representation of the last hidden layer of the forward and backward GRU in the utterance-level encoder; t is the hidden layer representation of the decoder at time t, e(r t-1 ) is the word vector representation of the output word at time t-1, It is the high-level semantic representation output by the graph encoding layer at layer L at time t-1;

[0042] Finally, according to the high-level semantic information of the graph encoding layer Combined with the decoder's hidden layer representation s at time t t To predict the output at the current moment, it is expressed by the following formula:

[0043]

[0044] Where W o , b o It is a trainable parameter, and p represents the probability of generating a word at the current moment;

[0045] Reply R = [r1, r2, ..., r T ] is the training target, and the cross entropy loss function is used Train the model parameters:

[0046]

[0047] The multi-round dialogue response generation system and method based on the relationship graph attention network of the present invention has the following advantages:

[0048] 1. The present invention is a multi-round dialogue response generation method. Compared with the traditional single-round dialogue response generation, the hierarchical structure can capture the high-level feature information between multi-round dialogue discourses, making the generated response information more relevant and diverse.

[0049] 2. This paper adopts the pre-trained language model BERT and uses a deep model to learn a better text feature, effectively solving the problem of polysemy.

[0050] 3. The present invention uses a relational graph attention network model to capture the interdependence between text sequences by constructing nodes, edges and corresponding topological structures, thereby further extracting potential feature representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a framework diagram of the multi-round dialogue response generation system based on the relationship graph attention network of the present invention;

[0052] Figure 2 A conceptual diagram of the relationship positions of the present invention;

[0053] Figure 3 A diagram of the process of position encoding for the four different relationship types of the present invention. DETAILED DESCRIPTION

[0054] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a multi-round dialogue response generation system and method based on a relationship graph attention network in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the multi-round dialogue response generation method based on the relationship graph attention network of the present invention includes the following steps:

[0056] Step 1: Obtain multiple rounds of dialogue input and preprocess it. Use the pre-trained BERT model to convert the word semantics in each round of utterance into corresponding vector representations, thereby obtaining the semantic information representation of each round of utterance itself. Then, use the Bi-GRU model to encode the semantic information of each round of utterance to obtain the semantic representation of the dialogue context.

[0057] Step 2: A graph attention network is used to capture the autocorrelation in multi-round conversations and the correlation features between interlocutors. Relation position encoding is introduced into the graph attention network to describe the sequence information of the utterances, thereby obtaining a high-level semantic representation at the graph encoding layer.

[0058] Step 3: Take the high-level semantic representation of the conversation context semantic information representation and the relationship graph attention encoding as input, and use the GRU model to decode and generate the final dialogue response output representation.

[0059] As a preferred embodiment of the present invention, in step 1, the process of preprocessing the input content of multiple rounds of dialogue to obtain the semantic information representation of each round of discourse itself includes: encoding the words in each round of input discourse, first using the BPE algorithm to perform sequence labeling representation on each round of discourse, and then inputting it into the pre-trained language BERT model for fine-tuning learning and training, thereby obtaining the semantic information representation of each round of discourse itself.

[0060] The framework diagram of the multi-round dialogue response generation system based on the relationship graph attention network of the present invention is as follows: Figure 1 As shown in Figure 1, the entire framework consists of the following components: sentence input encoding layer, graph encoding layer, and decoding layer. The following is a detailed description of each component of this system:

[0061] (1) Sentence input encoding layer

[0062] The sentence input encoding layer includes two different levels of encoders: word-level encoder and utterance-level encoder. These two encoders are described in detail below.

[0063] Word-level encoder: This encoder is designed to encode the words in each round of speech input to the model, so as to obtain the semantic representation of the speech itself. For a multi-round dialogue context U = {u1, ..., u M First, in order to fully extract the information expressed by the user's speech, the model uses the BPE algorithm to sequence mark each round of speech. Where T i is the number of tokens in the i-th round of speech, which is then input into the pre-trained language BERT model for fine-tuning learning. The encoder encoding process is expressed by the following formula:

[0064]

[0065] Thus, the semantic information representation of each round of discourse is obtained

[0066] Discourse level encoder: This encoder is designed to encode the semantic representation of the model's discourse itself, thereby obtaining the semantic representation of the entire conversation context. This paper adopts the Bi-GRU model as the encoding method of the discourse level encoder. The model first obtains the semantic representation of each round of discourse itself from the upper word level encoder. As input, each round of speech is encoded by Bi-GRU. The encoder encoding process is expressed by the following formula:

[0067]

[0068]

[0069] in is the i-th hidden layer representation in the forward GRU, is the i-th hidden layer representation in the backward GRU. Concatenate the hidden layer representations of the forward GRU and the backward GRU to obtain a semantic representation containing contextual information Capture high-level feature information between multi-round dialogue utterances through a hierarchical structure.

[0070] (2) Image Coding Layer

[0071] This encoding layer first uses a graph attention network to capture the autocorrelation of utterances in multi-round conversations and the correlation characteristics between interlocutors. In addition, this model introduces a new positional encoding (i.e., relational position encoding) in the graph attention network to account for the sequence information contained in the utterances.

[0072] Different from traditional research on response generation in multi-round dialogues, this paper constructs a directed graph for M statements in a multi-round dialogue and defines the following:

[0073]

[0074] Define each statement in a multi-round conversation as a node v i , The relationship dependency information between each statement is defined as edge r, in The edge weight is defined as α ijr , The following section introduces in detail the representation of each node in the graph, the definition of edge types, the encoding of relationship positions, the representation of edge weights, and the graph propagation mechanism.

[0075] In the graph attention network, the model first transforms the context semantic representation output by the context encoding layer into As node v i The initial vector representation of . Figure 1As shown, in order to fully display the different relational dependency information between discourses, in the information edge shown in the graph attention network coding layer, this paper defines the types of information edges r into the following four categories: (a) self-before type edge r1: the relational dependency type information between the target discourse and the discourse before the speaker of the discourse; (b) inter-before type edge r2: the relational dependency type information between the target discourse and the discourse before the speaker of the discourse; (c) self-after type edge r3: the relational dependency type information between the target discourse and the discourse after the speaker of the discourse; (d) inter-after type edge r4: the relational dependency type information between the target discourse and the discourse after the speaker of the discourse. In addition, this paper proposes a relational position encoding method to capture the temporal information between discourses of these four types of information edges. Different from the previous absolute position encoding and relative position encoding, this encoding is encoded based on the relative distance between discourses under the relational type. Figure 2 The concept of relationship position is shown. The different background colors in the relationship positions shown in the third row represent different types of information edges. Position encoding is then performed based on these four different relationship types, and this encoding information is added to the weight of the edge. Figure 3 This encoding process is shown. Therefore, the relationship position encoding process can be expressed by the following formula:

[0076]

[0077] Among them, PE ijr Represents the target discourse u under relation type r i Its adjacent discourse u j The maximum relationship value is between [b, a], where b and a are the sliding window values ​​between the target discourse and other discourses. Represents the target discourse u under relation type r i Neighborhood. Inspired by the graph attention network model and combined with the above-mentioned relationship position encoding, the edge weight is defined by the following formula:

[0078]

[0079] where α ijr Represents the target discourse u under relation type r i Its adjacent discourse u j The edge weight between them, W r is the parameterized weight matrix under the attention mechanism, a r is the parameterized weight vector, T stands for transpose, and LRL is the LeakyReLU activation function.

[0080] The graph coding layer finally aggregates the neighborhood The information is used to update the vector representation of each node The graph propagation mechanism encoding process is expressed by the following formula:

[0081]

[0082]

[0083] in is the trainable parameter weight matrix, L is the number of convolutional layers. The final output is the high-level semantic representation of the graph encoding layer.

[0084] (3) Decoding layer

[0085] The decoding layer is designed to generate a response reply based on the input contextual semantic information representation and the high-level semantic representation of the graph encoding layer. This paper uses the GRU model as a decoder to generate the reply. The decoder decoding process is expressed by the following formula:

[0086]

[0087]

[0088] Where s0 is the initialization input of the decoder, W e and b e is a trainable parameter, is the concatenated representation of the last hidden layer of the forward and backward GRU in the utterance-level encoder. t is the hidden layer representation of the decoder at time t, e(r t-1 ) is the word vector representation of the output word at time t-1, It is the high-level semantic representation output by the graph encoding layer at layer L at time t-1.

[0089] Finally, according to the high-level semantic information of the graph encoding layer Combined with the decoder's hidden layer representation s at time t t To predict the output at the current moment. It is expressed by the following formula:

[0090]

[0091] Where W o , b o It is a trainable parameter, and p represents the probability of generating a word at the current moment.

[0092] The present invention responds to a given real dialogue with R=[r1, r2, ..., r T ] is the training target, and the cross entropy loss function is used Train the model parameters:

[0093]

[0094] This invention was verified on two open source datasets, Ubuntu and Dailydialog, and the results are shown in the following table:

[0095]

[0096]

[0097] As can be seen from the table, the evaluation indicators of the method of the present invention on both datasets are substantially superior to those of other baseline models, verifying the effectiveness of the relational graph attention network method proposed in this invention. In terms of PPL, BLEU, and BERTScore indicators, the method of the present invention significantly outperforms all baseline models, indicating that the response information generated by this method is more relevant and diverse. At the same time, compared with the Ours-GCN method, the model proposed in this invention outperforms the traditional GCN model in all indicators, indicating that the attention mechanism introduced in the graph network layer can effectively capture information between sentence relation dependencies. By comparing with the Ours-NPE method, the performance of the method of the present invention has been greatly improved in terms of indicators, demonstrating the importance of relation position encoding.

[0098] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for generating multi-turn dialogue responses using a multi-turn dialogue response generation system based on a relational graph attention network. The multi-turn dialogue response generation system based on a relational graph attention network includes a sentence input encoding layer, a graph encoding layer, and a decoding layer, characterized in that: The sentence input encoding layer includes a word-level encoder and an utterance-level encoder; the word-level encoder encodes the words in each round of utterance input by the model, thereby obtaining the semantic representation of the utterance itself; the utterance-level encoder encodes the semantic representation of the model's utterance itself, thereby obtaining the semantic information representation of the entire conversation context; the graph encoding layer first uses a graph attention network to capture the autocorrelation of utterances in multiple rounds of conversation and the correlation characteristics between interlocutors, and then introduces relational position encoding to illustrate the sequence information contained between utterances; the decoding layer generates a response reply based on the input contextual semantic information representation and the high-level semantic representation of the graph encoding layer; the method is characterized in that it comprises the following steps: Step 1: Obtain multiple rounds of dialogue input and preprocess it. Use the pre-trained BERT model to convert the word semantics in each round of utterance into corresponding vector representations, thereby obtaining the semantic representation of each round of utterance itself. Then, use the Bi-GRU model to encode the semantic information of each round of utterance to obtain the semantic information representation of the dialogue context. Step 2: A graph attention network is used to capture the autocorrelation and correlation features between interlocutors in multi-round conversations. Relation position encoding is introduced into the graph attention network to account for the sequential information between utterances, thereby obtaining a high-level semantic representation at the graph encoding layer. Obtaining high-level semantic representation of the graph encoding layer includes the following steps: Construct a directed graph for M sentences in a multi-round conversation and define the following: Define each statement in a multi-round conversation as a node The relationship dependency information between each sentence is defined as edge r, (v i ,r,v j )∈ε, where The weight of an edge is defined as (1) First, the context semantic representation output by the context encoding layer As node v i The initial vector representation of ; (2) Information edges r are constructed based on the nodes, and the types of information edges r are defined into the following four categories: (a) Information edge r = 1: self-before type: the target utterance is dependent on the utterance before the utterance of the utterance speaker; (b) Information edge r = 2: inter-before type: the target utterance is dependent on the utterance before the utterance of the utterance speaker; (c) Information edge r = 3: self-after type: the target utterance is dependent on the utterance after the utterance of the utterance speaker; (d) Information edge r = 4: inter-after type: the target utterance is dependent on the utterance after the utterance of the utterance speaker; (3) Use the relational position encoding method to capture the temporal information between these four types of information; The relation position encoding process is expressed by the following formula: Among them, PE ijr Indicates the target utterance u under relation type r = 1, 2, 3, 4 i Its adjacent discourse u j The maximum relationship value is between [b, a], where b and a are the sliding window values ​​between the target utterance and other utterances. Indicates the target utterance u under relation type r = 1, 2, 3, 4 i Neighborhood; (4) Calculate the weight of the relevant information edge, and the formula is as follows: where α ijr Represents the target discourse u under relation type r i Its adjacent discourse u j The edge weight between them, W r is the parameterized weight matrix under the attention mechanism, a r is the parameterized weight vector, T stands for transpose, LRL is the LeakyReLU activation function; (5) By aggregating neighborhood The information is used to update the vector representation of each node The graph propagation mechanism encoding process is expressed by the following formula: in is the trainable parameter weight matrix, L is the number of convolutional layers, and the final output is the high-level semantic representation of the graph encoding layer. Step 3: Take the high-level semantic representation of the conversation context semantic information representation and the relationship graph attention encoding as input, and use the GRU model to decode and generate the final dialogue response output representation.

2. The method according to claim 1, characterized in that In the step 1, the process of preprocessing the input content of multiple rounds of dialogue to obtain the semantic information representation of each round of discourse itself includes: encoding the words in each round of input discourse, first using the BPE algorithm to perform sequence labeling on each round of discourse, and then inputting it into the pre-trained language BERT model for fine-tuning learning and training, thereby obtaining the semantic information representation of each round of discourse itself.

3. The method according to claim 1, characterized in that Step 1 uses a word-level encoder and an utterance-level encoder for encoding; it includes the following specific steps: The word-level encoder generates a multi-round dialogue context U of M sentences = {u1, ..., u M }, first use the BPE algorithm to represent the sequence of each round of speech Where T i is the number of tokens in the i-th round of speech, which is then input into the pre-trained language BERT model for fine-tuning training. The word-level encoder encoding process is expressed by the following formula: Thus, the semantic information representation of each round of discourse is obtained The Bi-GRU model is used as the encoding method of the discourse level encoder; the Bi-GRU model first obtains the semantic representation of each round of discourse itself from the upper word level encoder As input, each utterance is encoded by the Bi-GRU model. The encoding process of the utterance-level encoder is expressed by the following formula: in is the i-th hidden layer representation in the forward GRU, is the i-th hidden layer representation in the backward GRU; concatenate the hidden layer representations of the forward GRU and the backward GRU to obtain a semantic representation containing contextual information Capture high-level feature information between multi-round dialogue utterances through a hierarchical structure.

4. The method according to claim 1, wherein The step three includes the following steps: The GRU model is used as the decoder to generate the response. The decoder decoding process is expressed by the following formula: Where s0 is the initialization input of the decoder, W e and b e is a trainable parameter, is the concatenated representation of the last hidden layer of the forward and backward GRU in the utterance-level encoder; t is the hidden layer representation of the decoder at time t, e(r t-1 ) is the word vector representation of the output word at time t-1, It is the high-level semantic representation output by the graph encoding layer at layer l at time t-1; Finally, according to the high-level semantic information of the graph encoding layer Combined with the decoder's hidden layer representation s at time t t To predict the output at the current moment, it is expressed by the following formula: Where W o , b o It is a trainable parameter, P represents the probability of generating a word at the current moment; Reply Y=[y1,y2,…,y t …,y T ] is the training target, and the cross entropy loss function is used Train the model parameters:

Images