A text question answering model based on knowledge-enhanced graph neural network

Abstract

This invention discloses a text question-answering model based on a knowledge-enhanced neural network to improve machine understanding of natural language. The model consists of three main modules: input, inference, and prediction. By designing a knowledge enhancement module, it integrates structured knowledge from the commonsense knowledge base ConceptNet with deep semantics from unstructured text. A graph attention network is used to select text-related knowledge for multi-hop inference based on text semantics. A multilayer perceptron outputs the prediction results for accurate text question answering. This invention demonstrates superior performance on public datasets such as DREAM, validating its excellent semantic understanding capabilities. This also shows that the text question-answering model architecture based on a knowledge-enhanced graph neural network can perform knowledge selection, fuse heterogeneous information, and improve the machine's commonsense inference ability for accurate text question answering.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text question answering model based on a knowledge-enhanced neural network. Background Art

[0002] Writing and language are important symbols of human civilization and the main carriers of human communication. In the ever-evolving historical process, humans have produced a vast amount of text materials and text information. However, due to the lack of machine understanding of natural language, complex texts are usually only physically archived as valid data, but not fully mined and utilized, which brings huge challenges to human-computer interaction. With the rapid development of the Internet, traditional information retrieval technologies such as information search based on a few keywords and shallow semantic analysis can no longer meet the current users' urgent needs for accurate processing of massive information. Therefore, researchers have proposed a series of machine reading comprehension question-answering models based on deep neural networks to help machines automatically find answers from complex data resources and answer user questions in natural language to meet the growing demand for information acquisition.

[0003] Since there are a lot of common sense, metaphors, humor, etc. that are difficult for machines to understand in natural language texts, accurate answers to questions require machines to combine some background knowledge that is not explicitly expressed in the original text. However, existing machine reading comprehension models often lack the corresponding external knowledge reserves and find it difficult to cope with complex language phenomena. Achieving a deep understanding of the intrinsic semantics of the text remains a major challenge. Therefore, how to obtain relevant knowledge from external knowledge outside the question text so that the model can understand the background information based on the contextual semantics and then make the correct answer based on reasoning is a hot topic in current question-answering model research. This is because introducing too much external knowledge often brings unnecessary noise to the model, which in turn fails to help the model achieve performance improvement;

[0004] Secondly, there are many ways to integrate knowledge. How to find the best combination of structured knowledge and unstructured text is a key issue that needs to be studied urgently. At present, knowledge-based models cannot have the human cognitive ability of the outside world. Simple antonym knowledge relationships can easily deceive current models. Therefore, it is extremely necessary to improve the reasoning and understanding ability of question-answering models. Summary of the invention

[0005] The purpose of this invention is to provide a question-answering model based on knowledge-enhanced neural networks, which aims to assist in deep semantic understanding of text by introducing world knowledge. By establishing an effective interaction between text semantics and external knowledge, external knowledge related to the text is selected and then integrated with the pre-trained language model, and the multi-head attention mechanism of the pre-trained language model is used to dynamically focus on specific content in the text, and the information aggregation algorithm in the graph neural network is used to achieve effective semantic reasoning, thereby completing the correct answer to the question.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] A text question answering model based on knowledge-enhanced neural networks uses external knowledge to help machines understand common sense background information and uses graph neural network aggregation algorithms to improve the model's cognitive reasoning ability. The model includes an input module, a reasoning module, and a prediction module, where:

[0008] The input module is composed of a common sense knowledge base, namely ConceptNet, a knowledge enhancement module and a fusion module. The knowledge enhancement module receives input sentences converted into a unified format, integrates the knowledge in the common sense knowledge base according to the entities in the input sentences, uses an attention mask matrix to store the position encoding of the sentences and knowledge, and the fusion module realizes the fusion of the input sentences and the knowledge.

[0009] The reasoning module is composed of a pre-trained language model and a knowledge-enhanced graph attention network. The pre-trained language model is used to process the input sample after knowledge fusion to obtain the representation vector of the passage-question pair. At the same time, the knowledge-enhanced graph attention network is used to complete information aggregation and knowledge reasoning to obtain a new semantic representation of the text after knowledge enhancement.

[0010] The prediction module fuses the representation vector output by the pre-trained language model and the new text semantic representation output by the knowledge-enhanced graph attention network, uses a multi-layer perceptron to perform probability prediction, and completes the training of the question-answering model by minimizing the negative log-likelihood.

[0011] The knowledge enhancement module receives input sentences converted into a unified format, and the unified format of the input sentences is: [SEP] is used to separate the article and the question-answer pair.

[0012] The knowledge enhancement module locates entities in the input sentence and uses them as clues to integrate the structured knowledge in ConceptNet to improve the common sense understanding ability of the model, including:

[0013] For all entities in the input sentence, extract their adjacent entities from ConceptNet;

[0014] Add the corresponding relationship between the central entity and the adjacent entities to form a new tree form;

[0015] Use the relative distance between each node and the root node as a soft position encoding to replace the position encoding, and use the attention mask matrix to maintain the invisibility of different branches in the tree input;

[0016] The tree-shaped structured knowledge is embedded into the input sentence using positional encoding storage and template expansion strategy, including:

[0017] Use the edge weight values ​​in ConceptNet as prior knowledge to select the neighboring entities with the greatest connection to the target entity to avoid introducing noise;

[0018] Manually design text templates and use natural language to describe the relationship between the target entity and the structured knowledge extracted from ConceptNet;

[0019] The transformed unstructured text snippets are inserted into the original input via soft positions and attention mask matrices.

[0020] The fusion module flattens the original input and structured knowledge and sends them to the pre-trained language model for processing.

[0021] The pre-trained language model is used to process input samples after knowledge fusion. The model is first pre-trained using massive texts, and then the model is fine-tuned on the downstream task, i.e., the target task, to improve the accuracy of the downstream task, specifically including:

[0022] Process the input samples after knowledge fusion to obtain the representation vector of the passage-question pair;

[0023] Use the word vector matrix, block vector matrix and position vector matrix to map the representation vector to obtain the input representation;

[0024] The input representation is then encoded in multiple layers of the encoding layer, and the semantic associations between each word in the sentence are fully learned with the help of the self-attention mechanism to obtain the contextual semantic representation of the sentence;

[0025] Finally, the choice of answer to the question is converted into a classification task. The output layer predicts the classification label corresponding to the input text through a fully connected layer, and its parameters are learned by optimizing the cross entropy loss function.

[0026] The multi-layer encoding of the encoding layer, the specific process is:

[0027] Predict the original word at the position of [MASK]. Specifically, some words in the input text are masked with [MASK], and the masked words are restored to the original words through the multi-layer Transformer encoder using the context information around the masked words;

[0028] Then use the replacement word detection task to determine whether the words predicted by the masked language model at the [MASK] position are replaced with respect to the original input text, that is, to identify whether these words are consistent with the words at the corresponding position in the original sentence.

[0029] The knowledge-enhanced graph attention network includes a graph attention network, namely GAT, a multi-layer perceptron and a self-attention mechanism, and utilizes the information propagation algorithm of a graph neural network to realize information aggregation and updating of neighbor nodes, thereby improving knowledge understanding and reasoning capabilities.

[0030] The graph attention network will automatically learn valid entities and then perform reasoning, including:

[0031] Based on the entities extracted from the input, select adjacent nodes from ConceptNet according to the edge weights;

[0032] Construct a subgraph for each entity using the selected adjacent nodes;

[0033] Use the attention mechanism to optimize the embedding representation of entities;

[0034] During the optimization process, the optimal edge weight between two entities in the subgraph is automatically learned according to the question-answering task to represent the relevance of adjacent entities to the central entity;

[0035] The multi-layer perceptron fuses the entity-level output of the graph attention network with the output of the pre-trained language model, and then performs classification to obtain the semantic features required for the question-answering task;

[0036] The self-attention module receives the semantic features output by the multi-layer perceptron, calculates the dependencies between words, learns the internal structure of the sentence, and further refines the output of the multi-layer perceptron.

[0037] The present invention has demonstrated superior properties on public datasets such as DREAM, verifying its good semantic understanding ability; this also shows that the text question answering model architecture based on knowledge-enhanced graph neural networks can perform knowledge selection, complete heterogeneous information fusion, and enhance the machine's common sense inference ability to perform accurate text question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the structure of the present invention;

[0039] Figure 2 It is the algorithm framework diagram of the present invention;

[0040] Figure 3 It is the flow chart of the input module of the present invention;

[0041] Figure 4 This is a working principle diagram of the knowledge enhancement module in the present invention;

[0042] Figure 5 This is a test result diagram of the present invention on the DREAM dataset;

[0043] Figure 6 This is a test effect diagram of the present invention on different conversation turn data in the DREAM dataset. DETAILED DESCRIPTION

[0044] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0045] The following is a detailed description of the text question answering model based on knowledge-enhanced neural network. Figure 1 , 2 As shown, the present invention includes three parts: input module, reasoning module, and prediction module, and the specific steps of its operation are as follows:

[0046] Step 1: Convert the input training instance format. For each question, assume that P is the text paragraph; Q is the question; N is the number of candidate answers; O i represents the options, where i∈{1, 2, 3, …, N}. For each training instance, O i Spliced ​​after Q, the question-answer pair is represented as QO i Next, concatenate the article text as [CLS]P[SEP]QO i [SEP], and denote the transformed input as U i . Use a one-hot vector to represent U i , and add position coding, so that for each U i With a new embedding representation Processing flow such as Figure 3 shown.

[0047] Inspired by K-BERT, this paper proposes a new knowledge enhancement module, namely Kemb, to integrate some structured knowledge in ConceptNet to improve the common sense understanding ability of the model. For all entities in the input sentence, their adjacent entities are first extracted from ConceptNet, and the corresponding relationships are added to form a new tree form. The relative distance between each node and the root node is used as a soft position encoding instead of the position encoding, and an attention mask matrix is ​​used to maintain the invisibility of different branches in the tree input, and the structured knowledge is embedded into the pre-trained language model in combination with the following strategies: First, since the edge weight value in ConceptNet represents the credibility of this information, it is used as prior knowledge to select the most relevant adjacent entities and avoid some unnecessary noise; secondly, some text templates are manually designed, and natural language is used to describe the structured entities and the relationships between them in ConceptNet, and then the converted unstructured text fragments are inserted into the original input. Finally, the flattened input can be processed by a pre-trained language model. The working principle of the knowledge enhancement module is as follows: Figure 4 shown.

[0048] Step 2: In the reasoning phase, the knowledge-enhanced graph attention network, Kegat, is used to reason based on all relevant entities, and the word-sensitive pre-trained language model Electra is used to obtain a high-level representation of the entire passage-question pair at the sentence level. The knowledge-enhanced graph attention network is mainly composed of a graph attention network, a self-attention module, and a multi-layer perceptron. Among them, the graph attention network relies on the information aggregation algorithm to effectively complete the representation of neighbor nodes to update node features, and uses the attention mechanism to learn different weights for different neighbors.

[0049] Specifically, this paper uses a novel method to select relevant structured knowledge from ConceptNet and uses a graph attention network to perform semantic reasoning based on subgraphs constructed from entities and related knowledge. First, using the N-gram method, we first extract the relevant structured knowledge from the transformed input U i All entities are extracted from ConceptNet. Second, the edge weights are used as the probability of selecting up to k neighboring nodes from ConceptNet, and these neighboring nodes are used to construct subgraphs for each extracted entity. Assuming the number of entities is n, n subgraphs will be constructed in sequence. Next, ConceptNet is used to obtain the embedding representation of the i-th entity as the initial representation. It will be optimized by the L-layer graph attention network. During the optimization process, the graph attention network automatically learns the optimal edge weights between two entities in these subgraphs according to the question-answering task, which is used to represent the relevance of adjacent entities to each central entity. In other words, for each central entity, the graph attention network only assigns higher weights to those edges that are adjacent to several most reasonable adjacent entities in the constructed subgraph, and discards some insignificant edges. Therefore, by improving the subgraph, the model's abstract semantic reasoning ability can be improved. The working principle of the graph attention network is as follows:

[0050]

[0051]

[0052] The present invention updates each entity node based on formula (1), where σ(·) represents the ELU activation function, W is the network parameter, represents the l-th layer representation of the graph attention network, Ν i represents all neighboring nodes of the i-th entity. M represents the number of independent attention mechanisms in formula (2), represents the correlation between the jth entity and the ith entity. In addition, f(·) represents the projection function that converts the vector into a real number, and [;] represents the concatenation operation. Finally, we define formula (3), where U i is the i-th statement, n represents the number of entity subgraphs in the statement, As the final entity subgraph obtained from the graph attention network

[0053]

[0054] The reasoning module of the present invention will continue to use the self-attention module and the multi-layer perceptron for feature optimization. The specific steps are as follows: for all nodes in the constructed subgraph, a shared weight matrix is ​​trained to obtain the weight of each neighbor node; when calculating the attention value, the representation of the node is mapped using the shared weight matrix respectively, and the result vectors are spliced ​​together; then the spliced ​​vector is mapped to a real number using a feedforward neural network, and the final attention coefficient is obtained after normalization after activation function processing; after obtaining the attention coefficient, the neighbor nodes can be weighted and summed to obtain the output feature of the central node. In order to enable self-attention to stably represent nodes, a multi-head attention mechanism is introduced to improve the representation ability of the model: for intermediate output features, self-attention is calculated using multiple weight matrices, and then the results obtained by the attention heads are spliced ​​to obtain the output vector, and the output vectors of each attention head are averaged to obtain the final result. At the sentence level, the present invention uses a self-attention module and a multi-layer perceptron, namely MLP, to facilitate the model to reason about entities and input sentences. First, the output of the graph attention network is converted into The output of the pre-trained language model Fusion Then the self-attention module is used to refine G Ui Therefore, the question-answer pair can be used as a reference to further improve the entity-level representation. As shown in formulas (4)-(5):

[0055]

[0056]

[0057] in, is the refined representation of , SelfAttn(·) represents the self-attention operation, and σ(·) is the activation function. Finally, and Concatenate to get the whole inference representation As shown in formula (6).

[0058]

[0059] Step 3: After being processed by the inference module, each input instance can obtain a transformed representation In the prediction module, a multi-layer perceptron is used to perform probability calculations to complete the downstream question-answering task. The principle of this process can be defined as formulas (7)-(9).

[0060]

[0061] P′=softmax(P) (8)

[0062] y=argmax(P') (9)

[0063] Where P is the MLP output and P∈R N×1 , y is the predicted result, and P' is the probability of selecting the i-th option.

[0064] In the training phase, the negative log-likelihood minimization training is completed according to formula (10)

[0065]

[0066] Among them, y * represents the one-hot vector of the optimal option, and L represents the loss function.

[0067] The specific network parameter configuration of the present invention is as follows: batch_size is set to 1 during training, batch_size is 4 for testing, training learning rate is 0.001, learning rate is 0.0000005 during fine-tuning, adam_epsilon is 0.000001, weight_decay is 0.01, iteration epochs in training is set to 2, warmup_proportion is set to 0.05, fine_tune_epochs is set to 10, and k_fold is set to 5.

[0068] In the specific experimental stage, since dialogue texts usually contain complex language features such as abbreviations, references, metaphors, etc., the DREAM dataset poses greater challenges to the question-answering model than common written texts. In order to explore the ability of this question-answering model to understand dialogue paragraphs, the experiment tested the answer selection accuracy of different question-answering models on the DREAM dataset for comparison. The baseline models used in the experiment include: random selection methods and methods based on the latest pre-trained language model architectures XLNET-large, BERT-large, BERT-large+WAE, etc. WAE refers to the Wasserstein autoencoder, which is an algorithm for constructing a data distribution generation model. It minimizes the penalty form of the Wasserstein distance between the model distribution and the target distribution, so that the encoded training distribution matches the prior distribution. It is a generalization of the adversarial autoencoder and can generate better quality samples than the variational autoencoder. The experimental results are as follows Figure 5 As shown in the figure, the present invention shows good understanding performance on the dialogue text. The system that introduces the knowledge-enhanced graph attention network has a relative improvement of 14.9% compared with the previous BERT-large+WAE model with excellent performance, and a relative improvement of 4.8% compared with the architecture based on Electra-large.

[0069] The DREAM dataset contains multi-round conversations consisting of multiple speakers. Due to the particularity of the conversation text, the random joining of participants often causes the topics to be scattered in different rounds. However, multiple rounds of conversation passages pose a considerable challenge to the machine's understanding. Therefore, the experiment tested the understanding effect of the present invention on conversations of different rounds. After training with all the Train data subsets, the test set was divided into the following data sets according to the number of conversation rounds: less than 6 rounds, less than 8 rounds, more than 13 rounds, and more than 14 rounds. The divided data subsets were tested separately. The experimental results are as follows: Figure 6As shown. From the comparison of the results in the figure, when the input dialogue rounds are controlled to be less than 8 rounds, the accuracy of the answer prediction of the present invention is 82.41%, which is 3.15% higher than the prediction of all test data; when the dialogue rounds are adjusted to less than 6 rounds, the accuracy is 82.50%, which is 3.24% higher than the accuracy of all test data. When the dialogue rounds are greater than 13 rounds, the accuracy of the model's answer prediction is 71.43%, which is 7.83% lower than the accuracy of all test data; when the dialogue rounds are adjusted to greater than 14 rounds, the accuracy is reduced by 8.81%. From this analysis, since the topic in the dialogue sentence shifts quickly and covers more content, it is challenging for the present invention to understand its content. The present invention has a better understanding ability when the dialogue rounds are less than 8 rounds.

Claims

1. A text question answering device based on knowledge-enhanced neural network, characterized in that , including input module, reasoning module and prediction module, where: The input module is composed of a common sense knowledge base, namely ConceptNet, a knowledge enhancement module and a fusion module. The knowledge enhancement module receives input sentences converted into a unified format, integrates the knowledge in the common sense knowledge base according to the entities in the input sentences, uses an attention mask matrix to store the position encoding of the sentences and knowledge, and the fusion module realizes the fusion of the input sentences and the knowledge. The reasoning module is composed of a pre-trained language model and a knowledge-enhanced graph attention network. The pre-trained language model is used to process the input sample after knowledge fusion to obtain the representation vector of the passage-question pair. At the same time, the knowledge-enhanced graph attention network is used to complete information aggregation and knowledge reasoning to obtain a new semantic representation of the text after knowledge enhancement. The prediction module fuses the representation vector output by the pre-trained language model and the new text semantic representation output by the knowledge-enhanced graph attention network, uses a multi-layer perceptron to perform probability prediction, and completes the training of the question-answering model by minimizing the negative log-likelihood; wherein: The knowledge enhancement module locates entities in the input sentence and uses them as clues to integrate the structured knowledge in ConceptNet to improve the common sense understanding ability of the model, including: For all entities in the input sentence, extract their adjacent entities from ConceptNet; Add the corresponding relationship between the central entity and the adjacent entities to form a new tree form; Use the relative distance between each node and the root node as a soft position encoding to replace the position encoding, and use the attention mask matrix to maintain the invisibility of different branches in the tree input; Use positional encoding storage and template expansion strategy to embed tree-shaped structured knowledge into input sentences; The method of using positional encoding storage and template expansion strategy to embed tree-shaped structured knowledge into input sentences specifically includes: Use the edge weight values ​​in ConceptNet as prior knowledge to select the neighboring entities with the greatest connection to the target entity to avoid introducing noise; Manually design text templates and use natural language to describe the relationship between the target entity and the structured knowledge extracted from ConceptNet; The transformed unstructured text snippets are inserted into the original input via soft positions and attention mask matrices.

2. The text question-answering device according to claim 1, characterized in that: The knowledge enhancement module receives input sentences converted into a unified format, and the unified format of the input sentences is: [SEP] is used to separate the article and the question-answer pair.

3. The text question-answering device according to claim 1, characterized in that: The fusion module flattens the original input and structured knowledge and sends them to the pre-trained language model for processing.

4. The text question-answering device according to claim 1, characterized in that The pre-trained language model is used to process the input samples after knowledge fusion. The model is first pre-trained using massive texts, and then the model is fine-tuned on the downstream task, i.e., the target task, so as to improve the accuracy of the downstream task, including: Process the input samples after knowledge fusion to obtain the representation vector of the passage-question pair; Use the word vector matrix, block vector matrix and position vector matrix to map the representation vector to obtain the input representation; The input representation is then encoded in multiple layers of the encoding layer, and the semantic associations between each word in the sentence are fully learned with the help of the self-attention mechanism to obtain the contextual semantic representation of the sentence; Finally, the choice of answer to the question is converted into a classification task. The output layer predicts the classification label corresponding to the input text through a fully connected layer, and its parameters are learned by optimizing the cross entropy loss function.

5. The text question-answering device according to claim 4, characterized in that ,The multi-layer encoding of the encoding layer, the specific process is: Predict the original word at the position of [MASK]. Specifically, some words in the input text are masked with [MASK], and the masked words are restored to the original words through the multi-layer Transformer encoder using the context information around the masked words; Then use the replacement word detection task to determine whether the words predicted by the masked language model at the [MASK] position are replaced with respect to the original input text, that is, to identify whether these words are consistent with the words at the corresponding position in the original sentence.

6. The text question-answering device according to claim 1, characterized in that The knowledge-enhanced graph attention network includes a graph attention network, namely GAT, a multi-layer perceptron and a self-attention module, and uses the information propagation algorithm of the graph neural network to realize information aggregation and update of neighbor nodes, thereby improving knowledge understanding and reasoning capabilities.

7. The text question-answering device according to claim 6, characterized in that ,The graph attention network will automatically learn valid entities and then perform reasoning, specifically including: Based on the entities extracted from the input, select adjacent nodes from ConceptNet according to the edge weights; Construct a subgraph for each entity using the selected adjacent nodes; Use the attention mechanism to optimize the embedding representation of entities; During the optimization process, the optimal edge weight between two entities in the subgraph is automatically learned according to the question-answering task to represent the relevance of adjacent entities to the central entity; The multi-layer perceptron fuses the entity-level output of the graph attention network with the output of the pre-trained language model, and then performs classification to obtain the semantic features required for the question-answering task; The self-attention module receives the semantic features output by the multi-layer perceptron, calculates the dependencies between words, learns the internal structure of the sentence, and further refines the output of the multi-layer perceptron.

Images