End-to-end task-based dialogue system based on graph knowledge enhancement

Abstract

This paper proposes an end-to-end task-based dialogue system based on graph knowledge enhancement. The system comprises a graph-structured context encoder module, which encodes the context using a graph structure to obtain a vector representation of the dialogue history; a graph-structured knowledge module, which encodes the knowledge base information using a graph structure to obtain a vector representation of the knowledge base information; and a decoder module, which calculates the entity distribution based on the vector representations of the knowledge base information and the dialogue history information in the knowledge base. This system can encode the dialogue information and the knowledge base separately through a graph structure, effectively capturing the structural information between entities and the structural information between words in the dialogue, thereby improving the model's ability to predict entities.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to an end-to-end task-based dialogue system based on graph knowledge enhancement. Background Art

[0002] Although end-to-end task-based dialogue systems based on memory networks have made significant progress in recent years thanks to their multi-hop attention mechanisms, the generated entities still exhibit some deviations. This is primarily due to the fact that the currently popular triplet representation of knowledge has improved accuracy to a certain extent, but the model may still lose information when modeling knowledge. This is because the primary drawback of using triples to represent each entity is that entities are often not independent individuals, but rather have special relationships between them that triples cannot represent. If the dependencies between entities are ignored, the model will struggle to capture the vast amount of information embedded in the knowledge base, severely impacting the accuracy of the results. Furthermore, in currently popular end-to-end task-based dialogue systems, the focus is often on exploring the contextual relationships between conversational sentences, indirectly ignoring the dependencies between individual words within a sentence. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes an end-to-end task-based dialogue system based on graph knowledge enhancement.

[0004] To achieve the above-mentioned object of the present invention, the present invention provides an end-to-end task-based dialogue system based on graph knowledge enhancement, comprising:

[0005] Graph Structure Context Encoder Module: This module encodes the context in the form of a graph structure to obtain a vector representation of the conversation history information.

[0006] Graph knowledge module: encodes knowledge base information in the form of a graph structure to obtain a vector representation of the knowledge base information;

[0007] Decoder module: Calculates entity distribution based on the vector representation of knowledge base information and the vector representation of conversation history information in the knowledge base.

[0008] Furthermore, the encoding of the knowledge base information in the form of a graph structure is performed by a graph context encoder, and the encoding process is as follows:

[0009] x t is the input word of the input sentence at time t, P(t)=(p1,p2,..,p k ) is x t A conversation graph representation of a set of preceding words, p iRepresents and x t Related words, p i ∈P(t), k is x t The total number of preceding words; a set of hidden vectors of preceding words is represented as H = (h1, h2, ..., h k ), where each h j Represents the jth preceding word p j The hidden vector, h j ∈H;

[0010] First, loop over the hidden vectors k times and calculate their reset gates respectively, and use formula (1) to calculate r j As the reset gate of the j-th hidden vector:

[0011] r j =σ(W r x t +U r h j )#(1)

[0012] Among them, W r and U r These are all trainable parameters;

[0013] σ is the sigmoid activation function;

[0014] x t and h j are the hidden vectors of the input word at the current time t and the j-th preceding word at time t;

[0015] In order to better capture all the features at time t, according to the reset gate r j and the hidden vector h j The cumulative sum of the multiplication results is then added to the current input word x t , calculate the candidate hidden state at the current moment The specific calculation process is as shown in formula (2):

[0016]

[0017] Among them, W t and U t are all learnable weight parameter matrices;

[0018] k represents x t The total number of preceding words: the number of words;

[0019] In order to obtain rich representation capabilities, we first t and hidden state h j Apply a linear transformation:

[0020]

[0021]

[0022] Among them, W l and U l are all learnable parameters;

[0023] t is the current moment;

[0024] The hidden state H′ is obtained by linearly transforming the hidden state of the previous word. and the current input x t Candidate hidden states of composition

[0025] In order to capture the different effects of these different inputs, the masked attention mechanism is applied to the model to learn the importance of the previous words at different time steps. The process of calculating the attention mechanism is as follows:

[0026]

[0027] α j =softmax([e j ] m )#(6)

[0028] Among them, W e , W w and b e Represents the learnable parameters of the model;

[0029] Represents the input word x t Apply linear transformations;

[0030] represents the jth vector in H′;

[0031] α j represents the attention weight of the j-th vector in H′;

[0032] [·] m Indicates mask operation;

[0033] Finally, the hidden state h at time t t The output vector is:

[0034]

[0035] Furthermore, in order to obtain a richer bidirectional representation of the conversation history, the forward graph representation can be calculated according to formulas (1)-(7). and reverse graph representation and That is, the vector representation of the sequential relationship; then they are spliced ​​together as the vector representation of the conversation history information It is the hidden vector output by the encoder at the last moment. The set of hidden vectors output by the encoder is

[0036] Furthermore, encoding the knowledge base information in the form of a graph structure is achieved by performing multi-hop reasoning on the graph using a graph multi-hop reasoning mechanism, including the following steps:

[0037] There are two sets of trainable embedding matrices in the graph knowledge structure, E=(E 1 , E 2 ,...,E K+1 ) and V=(V 1 , V 2 ,...,V K+1 ), E k ∈E represents the kth embedding matrix that maps entities to vector representations, V k ∈V represents the weight vector matrix for calculating the k-th self-attention coefficient, and K represents the maximum number of hops;

[0038] The self-attention mechanism is applied between nodes and the attention coefficient is calculated. The specific calculation process is shown in formula (8):

[0039]

[0040] Among them, e i，j Represents the attention coefficient between the i-th node and the j-th node in the graph knowledge;

[0041] V k Represents the weight vector matrix for calculating the k-th self-attention coefficient;

[0042] T is the transpose symbol;

[0043] Indicates the use of the embedding matrix E to obtain the continuous vector representation of the i-th node at the k-th hop in the graph knowledge;

[0044] Indicates that the embedding matrix E is used to obtain the continuous vector representation of the j-th node at the k-th hop in the graph knowledge;

[0045] Express and Perform splicing operations;

[0046] For each node i and its first-order neighboring nodes, the normalized correlation coefficient operation is performed using formula (9):

[0047]

[0048] Among them, N i represents the first-order neighboring nodes of node i;

[0049] m represents the number of first-order neighboring nodes;

[0050] exp represents the exponential function;

[0051] e i，m Represents the attention coefficient between the i-th node and the m-th node in the graph knowledge;

[0052] The representation of node i is updated using the weighted sum of its first-order neighboring nodes. The specific calculation process is shown in formula (10):

[0053]

[0054] Represents calculation Representation after weighted sum;

[0055] Indicates that the embedding matrix E is used to obtain the continuous vector representation of the j-th node at the k-th hop in the graph knowledge;

[0056] In order to read the graph structure knowledge base information, the external knowledge base requires an initialized query vector. The query vector q converts the last hidden state vector of the graph context encoder into As the initial query vector, and then use the query vector q k Update nodes in graph knowledge and calculate k-hop attention weights The specific calculation process is shown in formula (11) and formula (12):

[0057]

[0058]

[0059] q 1 Represents the first query vector;

[0060] The specific content of the k-th hop information of the knowledge base can be read out by formula (13) k :

[0061]

[0062] The update of query vector q is shown in formula (14), where the query vector q of k+1 hops is k+1The readout content of the current graph knowledge o k and the current query vector q k Calculated;

[0063] q k+1 =q k +o k #(14)

[0064] The K-hop reading and reasoning capability of the graph multi-hop reasoning mechanism is finally realized by looping through formulas (11) to (14), and finally obtaining the vector representation of the knowledge base information o K .

[0065] Furthermore, it also includes using graph knowledge pointers to check whether the word exists in the expected system response during the training model:

[0066] Define a set of vectors representing graph knowledge pointers with true labels Its value is 0 or 1, where 0 means it does not exist and 1 means it exists; The graph knowledge pointer representing the nth true label, where n represents the number of entities in the graph knowledge;

[0067] For the graph knowledge pointer G obtained by calculation, n ), whose value is between 0 and 1. Different from the vector obtained by the attention mechanism, which requires all weights to sum to 1, each element in G is an independent probability. The Sigmoid activation function is used instead of the Softmax activation function in formula (12) to calculate the graph knowledge pointer g i , according to the calculation formula (15):

[0068]

[0069] q K represents the query vector of the K-th hop;

[0070] Represents calculation Representation after weighted sum;

[0071] Indicates the use of the embedding matrix E to obtain the continuous vector representation of the i-th node at the k-th hop in the graph knowledge;

[0072] In order to further improve the ability of graph knowledge pointers, an auxiliary loss function is added for training multi-label classification tasks. Then the cross entropy loss function is used as the G label and G learning objectives, Loss g The specific calculation is as shown in formula (16):

[0073]

[0074] Labels representing graph knowledge pointers;

[0075] g i Represents a graph knowledge pointer.

[0076] Furthermore, GRU is used as the decoder to generate system responses word by word. The initial input of the decoder is mainly divided into two parts: the output of the graph knowledge base o K and the output of the graph context encoder The two partial concatenation values ​​are used as the initial input of the decoder

[0077]

[0078] o K Indicates the specific content of the K-th hop information;

[0079] At each moment, the decoder uses the word generated at the previous moment and hidden state Generates a new hidden state as input The decoder hidden state calculation formula is as shown in formula (18):

[0080]

[0081] The hidden state H of the graph context encoding e and the hidden state generated by GRU Splice them together and then calculate an attention coefficient for all hidden states The specific calculation process is shown in formula (19):

[0082]

[0083] Among them, W p and W h are all parameter weights;

[0084] tanh() is the hyperbolic tangent function;

[0085] Finally, according to the attention coefficient and hidden state Compute candidate hidden states The specific calculation can be obtained as shown in formula (20):

[0086]

[0087] According to the hidden state at the current time t and candidate hidden states The final output vocabulary distribution of the decoder is calculated as follows:

[0088]

[0089] Among them, W o is the output weight matrix.

[0090] Furthermore, the BERT model is also included: encoding the query sentence input in each conversation to obtain the aggregated hidden state representation C, which specifically includes the following steps:

[0091] The query sentence s=(s1, s2, ..., s T ), where s t ∈s represents the sentence of the t-th round of dialogue, T represents the total number of rounds of dialogue; together with the context s t-1 Round of dialogue and s t The pre-trained BERT model is used to encode the model output and obtain the aggregated hidden state representation C. The aggregated hidden representation of the BERT model output C is passed to the fully connected hidden layer to predict the entity in the input question, which can be obtained by the following formula (22):

[0092] e inp =softmax(W ent C+b ent )#(twenty two)

[0093] Among them, W ent and b ent These are all parameters of the fully connected hidden layer.

[0094] Furthermore, the aggregated hidden state representation C and the vector representation of the conversation history information are subjected to graph Laplacian operations and then input into the decoder;

[0095] Graph Laplacian operations include:

[0096] To avoid making predictions only from relations related to the input entity, the subgraph and its label are encoded together and used in the final decoding process during evaluation. inp The k-hop subgraph of is encoded using the graph Laplacian:

[0097]

[0098] in, A is the adjacency matrix;

[0099] I is the identity matrix;

[0100] D is the degree matrix;

[0101] D -1 represents the inverse matrix of D;

[0102] f in is the feature representation of the nodes and edges of the input graph;

[0103] Output result S enc is a vector whose dimension corresponds to the total number of nodes and edges in the k-hop subgraph;

[0104] The final vocabulary distribution O f is the vector representation of the subgraph and the vocabulary distribution of the decoder output The Hadamard product of is calculated as shown in formula (24):

[0105]

[0106] Finally, the Beam-Search algorithm is used to output the final response of the system.

[0107] Through graph Laplacian, we can accurately calculate the relationships between entities, filter out the less relevant relationships, and improve the accuracy of responses.

[0108] Furthermore, the objective function of the model is also included:

[0109] Use the standard cross entropy loss function as the learning function between the output system response and the true response, Loss o As shown in formula (25):

[0110]

[0111] represents the true word probability;

[0112] Represents the calculated word probability;

[0113] m represents the number of first-order neighboring nodes;

[0114] The objective function of the model is Loss o With Loss g sum:

[0115] Loss = αLoss g +βLoss i #(26)

[0116] Among them, α and β are hyperparameters;

[0117] Loss g Represents G label and the learning function between G, G represents the graph knowledge pointer obtained by the operation, G label Graph knowledge pointer representing the true label.

[0118] In summary, due to the adoption of the above technical solution, the present invention can combine dialogue information and knowledge

[0119] The knowledge base is encoded separately, effectively capturing the structural information between entities and the structural information between words in the conversation, thereby improving the model's ability to predict entities.

[0120] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0122] Figure 1 Schematic diagram of the basic structure of the GKE2Seq model of the present invention.

[0123] Figure 2 It is a schematic diagram of the process of subgraph encoding using graph Laplacian in the present invention.

[0124] Figure 3 It is an example diagram of the knowledge base information graph structure of the present invention.

[0125] Figure 4 This is an example diagram of the conversation history information graph structure of the present invention. DETAILED DESCRIPTION

[0126] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0127] Aiming at the knowledge encoding and dialogue history information encoding in an end-to-end task-based dialogue system, the present invention presents a graph-structured representation of external knowledge base information and dialogue history information, and proposes an end-to-end task-based dialogue system GKE2Seq based on graph knowledge enhancement.

[0128] The overall structure of the GKE2Seq model is as follows Figure 1 As shown, it mainly includes:

[0129] The graph knowledge module encodes the knowledge base information in the form of a graph structure to obtain a vector representation of the knowledge base information;

[0130] The graph-structured context encoder module encodes the context of the conversation history information in the form of a graph structure, obtaining a vector representation of the dependency and order relationships between words in the conversation;

[0131] The decoder module calculates the entity distribution based on the vector representation of the knowledge base information and the context representation (obtained by the graph structure context encoder module). The following sections will describe each module in detail.

[0132] The method based on the GKE2Seq model is:

[0133] S1, the GKE2Seq model first parses the dependency relationship of the conversation history information, namely Conversation history information The graph structure is then encoded into a conversation hidden vector;

[0134] S2, performs graph structure encoding and reasoning on the knowledge base to obtain the knowledge base graph information structure representation, and then encodes it into the knowledge base hidden vector;

[0135] S3, finally inputs the knowledge base graph information structure and the dialogue hidden vector as the initial vector into the decoder for decoding to obtain the decoded data;

[0136] Step S1 is to obtain the hidden vector of the conversation using the graph structure context encoder. The pre-trained BERT model can also be used to process the conversation history information to predict the entities in the conversation information, that is, entity detection, and then obtain the subgraph information corresponding to the entity. The graph Laplacian in the graph neural network is used to additionally encode the subgraph of the input query entity to obtain additional encoded data, which further improves the performance of the model.

[0137] Finally, the decoded data and the additional encoded data are concatenated, and the Beam-Search algorithm is used to improve the accuracy of the model, thereby improving the accuracy of the conversation.

[0138] The knowledge base information graph structure is as follows Figure 3 As shown in the figure. For graph-structured knowledge base information, each entity has different connections. Giving extra attention to the connections associated with the queried entity and filtering out irrelevant connections from the response helps obtain the most appropriate and correct connections. This chapter uses graph Laplacian to accurately calculate entity-related connections, filter out less relevant connections, and improve the accuracy of responses.

[0139] The structure of the conversation history information graph is as follows: Figure 4As shown in Figure 1, there are rich relationship representations between words in a sentence (such as advmod, det, and prep, etc.). In order to better learn the rich representations of these different relationships, this paper first uses the spaCy tool to extract the dependency relationships between each word in the conversation history X. Since this paper is all English datasets, the spaCy tool uses the en_co-re_web_sm model to extract the dependency relationships between words. The dependency and sequence relationship diagram is shown in Figure 1. Figure 4 shown.

[0140] Figure 4 This example shows the dependency parsing results for the sentence "how is the traffic going there?" The various dependency representations in the parsing result resemble graph-like structural information, which we refer to as a conversation graph in this article. The bilateral representations (e.g., expl_reverse and expl) allow information flow between words, from dependency words to core words and vice versa. Therefore, core words are influenced by dependency words, and vice versa. Based on these relational representations, the model can capture richer representations of the various dependencies between words. To further support bilateral representation learning, the conversation graph is split into two parts: a forward graph running from left to right and a reverse graph running from right to left. The input to the graph encoder is a combination of the dependency relationships and order relationships (next and pre) between tokens in the sentence.

[0141] By representing the dialogue history information in a graph structure, we can explore and mine various complex dependencies between sentences and learn rich representations between words, which is conducive to enhancing the reasoning ability of the dialogue system and improving the accuracy of the answers generated by the system.

[0142] 1. Graph Context Encoder

[0143] Assume x t is the input word of the input sentence at time t, P(t)=(p1,p2,..,p k ) is x t A conversation graph representation of a set of preceding words, p i Represents and x t Related words, p i ∈P(t), k is x t The total number of preceding words. A set of hidden vectors of preceding words is represented as H = (h1, h2, ..., h k ), where each h j Represents the jth preceding word p j The hidden vector, h j ∈H.

[0144] First, loop over the hidden vector k times (k preceding words), calculate their reset gates respectively, and calculate r using formula (1) j As the reset gate of the j-th hidden vector:

[0145] r j =σ(W r x t +U r h j )#(1)

[0146] Among them, W r and U r are all trainable parameters, σ is the sigmoid activation function. t and h j They are the hidden vectors of the input word at the current time t and the jth preceding word at time t. In order to better capture all the features at time t, according to the reset gate r j and the hidden vector h j The cumulative sum of the multiplication results is then added to the current input word x t , calculate the candidate hidden state at the current moment The specific calculation process is as shown in formula (2):

[0147]

[0148] Among them, W t and U t are all learnable weight parameter matrices.

[0149] k represents the number of words;

[0150] In order to obtain rich representation capabilities, we first t and hidden state h j Apply a linear transformation:

[0151]

[0152]

[0153] Among them, W l and U l are all learnable parameters, t is the current moment. The hidden state H′ is obtained by linearly transforming the hidden state of the previous word. and the current input x t Candidate hidden states of composition Since different inputs may lead to different effects, such as different preceding words in H′, in order to enable the GKE2Seq model to capture the different effects of these different inputs, the masked attention mechanism is applied to the model to learn the importance of preceding words at different time steps. The process of calculating the attention mechanism is as follows:

[0154]

[0155] α j =softmax([e j ] m )#(6)

[0156] Among them, W e , W w and b e represents the learnable parameters of the model,

[0157] Represents the input word x t Apply a linear transformation

[0158] represents the jth vector in H′,

[0159] α j represents the attention weight of the j-th vector in H′,

[0160] [·] m Represents a masking operation. If the jth hidden state is a [PAD] word, then its value is set to negative infinity. [PAD] is a placeholder. The [PAD] word can be understood as a meaningless word, which takes up a position to represent a blank word.

[0161] Finally, the hidden state h at time t t The output vector is:

[0162]

[0163] h t That is, the dependency relationship between words in the conversation. In addition, in order to obtain a richer bidirectional representation of the conversation history, according to formulas (1)-(7), the forward graph representation can be calculated separately and reverse graph representation and This is the vector representation of the sequential relationship. Then concatenate them together as the vector representation of the conversation history information It is the hidden vector output by the encoder at the last moment. As one of the input vectors of the decoder, the set of hidden vectors output by the encoder is

[0164] 2. Graph multi-hop reasoning mechanism

[0165] The graph multi-hop reasoning mechanism is used for reading and reasoning external knowledge. There are two sets of trainable embedding matrices on the graph knowledge structure, E = (E 1 , E 2 ,...,E K+1 ) and V=(V 1 , V 2 ,...,V K+1 ), E k represents the kth embedding matrix that maps entities to vector representations, V k Denotes the weight vector matrix for calculating the kth self-attention coefficient, and K represents the maximum number of hops. The self-attention mechanism is applied between nodes and the attention coefficient is calculated. The specific calculation process is shown in formula (8):

[0166]

[0167] in, Indicates using the embedding matrix E to obtain the continuous vector representation of the i-th node in the k-th hop in the graph knowledge, and then and Perform splicing operations. T is the transpose symbol.

[0168] For each node i and its first-order neighboring nodes, the normalized correlation coefficient operation is performed using formula (9):

[0169]

[0170] Among them, N i represents the first-order neighboring nodes of node i, with a total of m first-order neighboring nodes, and exp represents the exponential function. Then the weighted sum of the first-order neighboring nodes of node i can be used to update the representation of node i. The specific calculation process is shown in formula (10):

[0171]

[0172] Represents calculation Representation after weighted sum;

[0173] Indicates that the embedding matrix E is used to obtain the continuous vector representation of the j-th node at the k-th hop in the graph knowledge;

[0174] In order to read the graph structure knowledge base information, the external knowledge base requires an initialized query vector. The query vector q converts the last hidden state vector of the graph context encoder into As the initial query vector, and then use the query vector q kUpdate nodes in graph knowledge and calculate k-hop attention weights The specific calculation process is shown in formula (11) and formula (12):

[0175]

[0176]

[0177] q 1 Represents the first query vector.

[0178] The GKE2Seq model reads the specific content of the k-th hop information of the knowledge base through formula (13) k :

[0179]

[0180] The update of query vector q is shown in formula (14), where the query vector q of k+1 hops is k+1 The readout content of the current graph knowledge o k and the current query vector q k Calculated. And the final readout of the graph knowledge o k As one of the inputs to the decoder.

[0181] q k+1 =q k +o k #(14)

[0182] The ultimate realization of the K-hop reading and reasoning capability of the graph multi-hop reasoning mechanism is achieved through the process of looping formulas (11) to (14).

[0183] 3. Graph Knowledge Pointer

[0184] Define a set of vectors representing graph knowledge pointers with true labels n represents the number of entities in the graph knowledge, which is used to check whether the word exists in the expected system response. Its value is 0 or 1, where 0 indicates non-existence and 1 indicates existence. Graph knowledge pointer representing the nth true label.

[0185] For the graph knowledge pointer G obtained by GKE2Seq model operation, G = (g1, g2, ..., g n ), whose value is between 0 and 1. Different from the vector obtained by the attention mechanism, which requires all weights to sum to 1, each element in G is an independent probability. The Sigmoid activation function is used instead of the Softmax activation function in formula (12) to calculate the graph knowledge pointer g i , according to the calculation formula (15):

[0186]

[0187] q K represents the query vector of the K-th hop;

[0188] Represents calculation Representation after weighted sum;

[0189] Indicates the use of the embedding matrix E to obtain the continuous vector representation of the i-th node at the k-th hop in the graph knowledge;

[0190] In order to further improve the ability of graph knowledge pointers, an auxiliary loss function is added for training multi-label classification tasks. Then the cross entropy loss function is used as the G label and G learning objectives. Loss g The specific calculation is as shown in formula (16):

[0191]

[0192] Labels representing graph knowledge pointers;

[0193] g i Represents graph knowledge pointer;

[0194] 4 decoders

[0195] The GKE2Seq model uses GRU as a decoder to generate system responses word by word. The initial input of the decoder is mainly divided into two parts: the output of the graph knowledge base K and the output of the graph context encoder The two partial concatenation values ​​are used as the initial input of the decoder

[0196]

[0197] o K Indicates the specific content of the K-th hop information.

[0198] At each moment, the decoder uses the word generated at the previous moment and hidden state Generates a new hidden state as input The decoder hidden state calculation formula is as shown in formula (18):

[0199]

[0200] The hidden state H of the graph context encoding e and the hidden state generated by GRU Splice them together and then calculate an attention coefficient for all hidden states The specific calculation process is shown in formula (19):

[0201]

[0202] Among them, W p and W h These are all parameter weights, and finally according to the attention coefficient and hidden state Compute candidate hidden states The specific calculation can be obtained as shown in formula (20):

[0203]

[0204] According to the hidden state at the current time t and candidate hidden states The final output vocabulary distribution of the decoder is calculated as follows:

[0205]

[0206] Among them, W o is the output weight matrix.

[0207] 5. Entity Detection

[0208] BERT is a pre-trained bidirectional Transformer model that focuses on two training objectives on unlabeled data: masked language modeling and next sentence prediction. When encoding any text, BERT uses the special words [CLS] and [SEP] to indicate the beginning and end of the text, respectively. In the GKE2Seq model, BERT is used to predict the input entity in the sentence and then obtain the subgraph of the input entity.

[0209] In the GKE2Seq model, the query sentence s = (s1, s2, ..., s T ), where s t represents the sentence of the t-th round of dialogue, T represents the total number of rounds of dialogue. Together with the context s t-1 Round of dialogue and s t The pre-trained BERT model is used to encode the model output and obtain the aggregated hidden state representation C. The aggregated hidden representation of the BERT model output C is passed to the fully connected hidden layer to predict the entity in the input question, which can be obtained by the following formula (22):

[0210] e inp =softmax(W ent C+b ent )#(twenty two)

[0211] Among them, W entand b ent These are all parameters of the fully connected hidden layer.

[0212] 6 Graph Laplace

[0213] In order to avoid the GKE2Seq model from predicting only the relations related to the input entity, the subgraph and its labels are encoded together and used in the final decoding process during evaluation. inp The k-hop subgraph of is encoded using the graph Laplacian:

[0214]

[0215] in, A is the adjacency matrix, I is the identity matrix, D is the degree matrix, D -1 represents the inverse matrix of D, f in It is the feature representation of the nodes and edges of the input graph, and the output result S enc is a vector whose dimension corresponds to the total number of nodes and edges in the k-hop subgraph.

[0216]

[0217] The final vocabulary distribution O f It is the Hadamard product of the vector representation of the subgraph and the vocabulary distribution output by the decoder, and its calculation process is shown in formula (24).

[0218] Finally, the Beam-Search algorithm is used to output the final response of the system. An example of a subgraph encoding mechanism is Figure 2 As shown in Figure 2, the probability of generating words after the input sentence is calculated by the graph Laplacian.

[0219] This step helps the GKE2Seq model prioritize connections based on the query entity and filter out irrelevant relationships from the response. For example, if the input text "can you find me a chine-se_restaurant that is within 5_miles" is used, and the "poi_type" relationship of the entity "chef_chu_s" is retrieved from the graph knowledge as the response relationship, Graph Laplacian can accurately calculate the relevant relationships of the entity and filter out less relevant relationships, thereby improving the accuracy of the response.

[0220] Use the standard cross entropy loss function as the learning function between the output system response and the true response, and define Loss o As shown in formula (25):

[0221]

[0222] represents the true word probability;

[0223] Represents the calculated word probability;

[0224] m represents the number of words, that is, the number of first-order neighboring nodes;

[0225] In summary, the final objective function of GKE2Seq is Loss o With Loss g sum:

[0226] Loss = αLoss g +βLoss o #(26)

[0227] Among them, α and β are hyperparameters.

[0228] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. An end-to-end task-based dialogue system based on graph knowledge enhancement, characterized by: include: Graph Structure Context Encoder Module: Encodes the context in the form of a graph structure to obtain a vector representation of the conversation history information. The encoding of the context in the form of a graph structure is performed by the graph context encoder. The encoding process is as follows: x t is the input word of the input sentence at time t, P(t)=(p1,p2,..,p k ) is x t A conversation graph representation of a set of preceding words, p i Represents and x t Related words, p i ∈P(t), k is x t The total number of preceding words; a set of hidden vectors of preceding words is represented as H = (h1,h2,…,h k ), where each h j Represents the jth preceding word p j The hidden vector, h j ∈H; First, loop over the hidden vectors k times and calculate their reset gates respectively, and use formula (1) to calculate r j As the reset gate of the j-th hidden vector: r j =σ(W r x t +U r h j )#(1) Among them, W r and U r These are all trainable parameters; σ is the sigmoid activation function; x t and h j are the hidden vectors of the input word at the current time t and the j-th preceding word at time t; In order to capture all the features at time t, according to the reset gate r j and the hidden vector h j The cumulative sum of the multiplication results is then added to the current input word x t , calculate the candidate hidden state at the current moment The specific calculation process is as shown in formula (2): Among them, W t and U t are all learnable weight parameter matrices; k represents x t The total number of preceding words: the number of words; In order to obtain the representation capability, we first t and hidden state h j Apply a linear transformation: Among them, W l and U l are all learnable parameters; t is the current moment; The hidden state H′ is obtained by linearly transforming the hidden state of the previous word. and the current input x t Candidate hidden states of composition In order to capture the different effects of these different inputs, the masked attention mechanism is applied to the model to learn the importance of the previous words at different time steps. The process of calculating the attention mechanism is as follows: a j =softmax([e j ] m )#(6) Among them, W e , W w and b e Represents the learnable parameters of the model; Represents the input word x t Apply linear transformations; represents the jth vector in H′; α j represents the attention weight of the j-th vector in H′; [·] m Indicates mask operation; Finally, the hidden state h at time t t The output vector is: Furthermore, in order to obtain a richer bidirectional representation of the conversation history, the forward graph representation can be calculated according to formulas (1)-(7). and reverse graph representation and That is the vector representation of the order relationship; Then concatenate them together as a vector representation of the conversation history information It is the hidden vector output by the encoder at the last moment. The set of hidden vectors output by the encoder is Graph knowledge module: encodes knowledge base information in the form of a graph structure to obtain a vector representation of the knowledge base information; Decoder module: Calculates entity distribution based on the vector representation of knowledge base information and the vector representation of conversation history information in the knowledge base.

2. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 1 is characterized in that: Encoding the knowledge base information in the form of a graph structure is achieved by using a graph multi-hop reasoning mechanism on the graph to perform multi-hop reasoning, which includes the following steps: There are two sets of trainable embedding matrices in the graph knowledge structure, E=(E 1 ,E 2 ,…,E K+1 ) and V=(V 1 ,V 2 ,…,V K+1 ), E k ∈E represents the kth embedding matrix that maps entities to vector representations, V k ∈V represents the weight vector matrix for calculating the k-th self-attention coefficient, and K represents the maximum number of hops; The self-attention mechanism is applied between nodes and the attention coefficient is calculated. The specific calculation process is shown in formula (8): Among them, e i,j Represents the attention coefficient between the i-th node and the j-th node in the graph knowledge; V k Represents the weight vector matrix for calculating the k-th self-attention coefficient; T is the transpose symbol; Indicates the use of the embedding matrix E to obtain the continuous vector representation of the i-th node at the k-th hop in the graph knowledge; Indicates that the embedding matrix E is used to obtain the continuous vector representation of the j-th node at the k-th hop in the graph knowledge; Express and Perform splicing operations; For each node i and its first-order neighboring nodes, the normalized correlation coefficient operation is performed using formula (9): Among them, N i represents the first-order neighboring nodes of node i; m represents the number of first-order neighboring nodes; exp represents the exponential function; e i,m Represents the attention coefficient between the i-th node and the m-th node in the graph knowledge; The representation of node i is updated using the weighted sum of its first-order neighboring nodes. The specific calculation process is shown in formula (10): Represents calculation Representation after weighted sum; Indicates that the embedding matrix E is used to obtain the continuous vector representation of the j-th node at the k-th hop in the graph knowledge; The query vector q converts the last hidden state vector of the graph context encoder into As the initial query vector, and then use the query vector q k Update nodes in graph knowledge and calculate k-hop attention weights The specific calculation process is shown in formula (11) and formula (12): q 1 Represents the first query vector; The specific content of the k-th hop information of the knowledge base can be read out by formula (13) k : The update of query vector q is shown in formula (14), where the query vector q of k+1 hops is k+1 The readout content of the current graph knowledge o k and the current query vector q k Calculated; q k+1 =q k +o k #(14) The K-hop reading and reasoning capability of the graph multi-hop reasoning mechanism is finally realized by looping through formulas (11) to (14), and finally obtaining the vector representation of the knowledge base information o K .

3. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 1 is characterized in that: This also includes using graph knowledge pointers to check whether a word exists in the expected system response during the training model: Define a set of vectors representing graph knowledge pointers with true labels Its value is 0 or 1, where 0 means it does not exist and 1 means it exists; The graph knowledge pointer representing the nth true label, where n represents the number of entities in the graph knowledge; For the graph knowledge pointer G obtained by calculation, n ), whose value is between 0 and 1; the Sigmoid activation function is used instead of the Softmax activation function in formula (12) to calculate the graph knowledge pointer g i , according to the calculation formula (15): q K represents the query vector of the K-th hop; Represents calculation Representation after weighted sum; Indicates the use of the embedding matrix E to obtain the continuous vector representation of the i-th node at the k-th hop in the graph knowledge; Using cross entropy loss function as G label and G learning objectives, Loss g The specific calculation is as shown in formula (16): Labels representing graph knowledge pointers; g i Represents a graph knowledge pointer.

4. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 1, characterized in that: GRU is used as the decoder to generate system responses word by word. The initial input of the decoder is mainly divided into two parts: the output of the graph knowledge base o K and the output of the graph context encoder The two partial concatenation values ​​are used as the initial input of the decoder o K Indicates the specific content of the K-th hop information; At each moment, the decoder uses the word generated at the previous moment and hidden state Generates a new hidden state as input The decoder hidden state calculation formula is as shown in formula (18): The hidden state H of the graph context encoding e and the hidden state generated by GRU Splice them together and then calculate an attention coefficient for all hidden states The specific calculation process is shown in formula (19): Among them, W p and W h are all parameter weights; tanh() is the hyperbolic tangent function; Finally, according to the attention coefficient and hidden state Compute candidate hidden states The specific calculation can be obtained as shown in formula (20): According to the hidden state at the current time t and candidate hidden states The final output vocabulary distribution of the decoder is calculated as follows: Among them, W o is the output weight matrix.

5. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 1 is characterized in that: It also includes the BERT model: Encode the query sentence input in each conversation to obtain the aggregated hidden state representation C, which specifically includes the following steps: The query sentence s=(s1,s2,…,s T ), where s t ∈s represents the sentence of the t-th round of dialogue, T represents the total number of rounds of dialogue; together with the context s t-1 Round of dialogue and s t The pre-trained BERT model is used to encode the model output and obtain the aggregated hidden state representation C. The aggregated hidden representation of the BERT model output C is passed to the fully connected hidden layer to predict the entity in the input question, which can be obtained by the following formula (22): e inp =softmax(W ent C+b ent )#(22) Among them, W ent and b ent These are all parameters of the fully connected hidden layer.

6. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 5, characterized in that: Perform a graph Laplacian operation on the aggregated hidden state representation C and the vector representation of the conversation history information, and then input it into the decoder; Graph Laplacian operations include: For the input entity e inp The k-hop subgraph of is encoded using the graph Laplacian: in, A is the adjacency matrix; I is the identity matrix; D is the degree matrix; D -1 represents the inverse matrix of D; f in is the feature representation of the nodes and edges of the input graph; Output result S enc is a vector whose dimension corresponds to the total number of nodes and edges in the k-hop subgraph; The final vocabulary distribution O f is the vector representation of the subgraph and the vocabulary distribution of the decoder output The Hadamard product of is calculated as shown in formula (24): Finally, the Beam-Search algorithm is used to output the final response of the system.

7. The end-to-end task-based dialogue system based on graph knowledge enhancement according to claim 1 is characterized in that: The objective function of the model is also included: Use the standard cross entropy loss function as the learning function between the output system response and the true response, Loss o As shown in formula (25): represents the true word probability; Represents the calculated word probability; m represents the number of first-order neighboring nodes; The objective function of the model is Loss o With Loss g sum: Loss=αLoss g +βLoss o #(26) Among them, α and β are hyperparameters; Loss g Represents G label and the learning function between G, G represents the graph knowledge pointer obtained by the operation, G label Graph knowledge pointer representing the true label.

Images