A method for building a question-answering system based on the BERT model and external knowledge graph

Abstract

The present invention is applicable to the fields of deep learning and natural language processing technology, and provides a method for constructing a question-answering system based on the BERT model and an external knowledge graph, comprising the following steps: performing word embedding and character embedding on the input words, obtaining an embedding vector after preprocessing; extracting features using a stacked encoder; fusing the vector obtained by extracting the features with the knowledge representation obtained using the external knowledge graph; calculating an attention score matrix, fusing the knowledge representation to obtain a question-aware context representation; using a stacked model encoder to obtain the final embedding expression, using an activation function to predict the probability of the start and end positions of the answer; and performing textual implication recognition between the predicted answer and the true answer. The present invention adopts a combination of static and dynamic embedding, which is conducive to the machine's understanding of natural language; the present invention utilizes the concept of knowledge graph to increase accuracy; and adds an answer verification component to convert the answer verification task into a textual implication recognition task.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and natural language processing, and in particular relates to a method for constructing a question-answering system based on a BERT model and an external knowledge graph. Background Art

[0002] In intelligent question-answering systems based on deep learning and embedding vectors, word embeddings are mostly static, using pre-trained models for fixed word embeddings. For example, in the GLoVe model, each word has its own fixed embedding. Since the introduction of large-scale pre-trained language models such as BERT and GPT, the performance of many downstream tasks in the field of natural language processing has been greatly improved. In intelligent question-answering tasks, the same word may have different semantics in different contexts. However, if a pre-trained model with fixed word embeddings is used, then no matter how the context of the word changes, its embedding will be the same, and therefore the semantics expressed will also be the same, which will lead to errors in understanding the article. This method chooses to use a combination of static GloVe embeddings and dynamic BERT embeddings to better represent the semantic features of words in different contexts.

[0003] The knowledge graph represents the relationship between different entities. Introducing it as external knowledge can theoretically improve the capabilities of the question-answering system. This method uses the knowledge graph as an external representation, which is different from knowledge graph question answering. For knowledge graph question answering, the entire process is carried out on the basis of the knowledge graph, and operations such as relationship extraction and semantic parsing are performed on the graph, and the answers to the questions are also obtained in the graph. For this method, its main task is to answer questions in the text direction. It only needs to extract subgraphs related to the task from the existing knowledge graph, and then integrate the features extracted from the subgraph into subsequent research. The knowledge graph can represent the relationship between different entities, so this operation can effectively utilize its advantages, integrate the features extracted from the graph into the text, increase the relevance between the context and the question, and be able to more accurately extract the answer from the context based on the question.

[0004] In real-life conversations, humans often choose not to answer questions they don't know. Similarly, when machines perform intelligent question-answering, there may be questions beyond their knowledge base, for which they are unable to provide answers. However, most existing methods give specious answers to such questions, which can be misleading and even lead to serious consequences. Therefore, this method transforms answer verification into a textual entailment recognition task by adding an answer verification component. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for constructing a question-answering system based on the BERT model and an external knowledge graph, aiming to solve the problems existing in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for building a question-answering system based on the BERT model and an external knowledge graph includes the following steps:

[0008] Step A: Perform word embedding and character embedding on the input words, and obtain embedding vectors that can be input into the neural network after preprocessing;

[0009] Step B: Extract features using a stacked encoder based on the embedding vector obtained in step A.

[0010] Step C: Use an external knowledge graph to increase the representation of the relationship between texts, recorded as the knowledge representation of context and question, and fuse the vector obtained in step B with the knowledge representation;

[0011] Step D: Based on the representation obtained in step C, use the attention mechanism to calculate the attention score matrix to obtain the question-aware context representation;

[0012] Step E: Obtain the question-aware context representation according to step D, use the stacked model encoder to obtain the final embedding representation, and use the activation function to obtain the probability prediction of the start and end positions of the answer;

[0013] Step F: Based on the predicted answer obtained in step E, perform textual implication recognition with the actual answer input in step A to determine whether the question is answerable.

[0014] Furthermore, step A adopts a combination of GloVe and BERT to perform word embedding at the same time to form a word embedding vector.

[0015] Furthermore, in step A, CNN is used to embed the characters of each word to form a character vector.

[0016] Furthermore, in step A, the specific operations of pre-processing are:

[0017] Use the highway network to concatenate the word embedding vector and the character vector to obtain an embedding vector that can be input into the neural network.

[0018] Furthermore, in step C, based on each context in the SQuAD2.0 dataset, entities and their related relationships are extracted from the knowledge graph ConceptNet5.5 to form a subgraph. The GAT network is used to learn the embedded representation of the subgraph, and the features of each node in the graph are aggregated into one node to form the final knowledge representation, which is then integrated with the context and question as the input of subsequent modules.

[0019] Furthermore, in step F, the predicted answer is used as the premise and the true answer is used as the hypothesis. The premise and the hypothesis are judged based on the textual implication, and whether the answer to the question is answerable is determined based on whether the two are in an implication relationship.

[0020] Furthermore, in step F, a bidirectional LSTM is used to perform binary classification prediction on the premise and hypothesis to determine whether the premise and hypothesis are in an implication relationship. If the result is implication, the question is considered answerable; if not, the question is unanswerable.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. This method uses a combination of static and dynamic embedding, which is more conducive to the machine's understanding of natural language.

[0023] 2. This method is only a text-oriented question answering method. Applying the concept of knowledge graph can increase the relevance between the context and the question, which is more conducive to finding the answer from the context and increasing the accuracy.

[0024] 3. This method is designed to add an answer verification part, converting the answer verification task into a text implication recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0028] like Figure 1 As shown in FIG, a method for constructing a question-answering system based on a BERT model and an external knowledge graph is provided in one embodiment of the present invention, comprising the following steps:

[0029] Step A: Perform word embedding and character embedding on the input words, and obtain embedding vectors that can be input into the neural network after preprocessing;

[0030] Step B: Extract features using a stacked encoder based on the embedding vector obtained in step A.

[0031] Step C: Use an external knowledge graph to increase the representation of the relationship between texts, recorded as the knowledge representation of context and question, and fuse the vector obtained in step B with the knowledge representation;

[0032] Step D: Based on the representation obtained in step C, use the attention mechanism to calculate the attention score matrix to obtain the question-aware context representation;

[0033] Step E: Obtain the question-aware context representation based on step D, use the stacked model encoder to obtain the final embedding expression of the context about the text, and use the classification function to predict the answer;

[0034] Step F: Perform textual implication recognition based on the predicted answer obtained in step E and the actual answer input in step A to determine whether the answer to the question is answerable.

[0035] As a preferred embodiment of the present invention, step A adopts a combination of GloVe and BERT to perform word embedding simultaneously to form a word embedding vector.

[0036] As a preferred embodiment of the present invention, in step A, CNN is used to embed the characters in each word to form a character vector.

[0037] As a preferred embodiment of the present invention, in step A, the specific operation of pre-processing is:

[0038] Use the highway network to concatenate the word embedding vector and the character vector to obtain an embedding vector that can be input into the neural network.

[0039] As a preferred embodiment of the present invention, in step C, a subgraph is obtained from the knowledge graph, and the GAT network is used to learn the embedded expression of the subgraph.

[0040] In an embodiment of the present invention, preferably, data contained in a data set is extracted from the knowledge graph of ConceptNet5.5 to form a subgraph, and the subgraph is trained using a GAT network to obtain a knowledge representation of text and questions.

[0041] As a preferred embodiment of the present invention, in step F, the predicted answer is used as the premise and the true answer is used as the hypothesis. The premise and the hypothesis are judged based on the textual implications, and whether the answer to the question is answerable is determined based on whether there is an implication relationship between the two.

[0042] As a preferred embodiment of the present invention, in step F, a bidirectional LSTM is used to perform binary classification prediction on the premise and hypothesis to determine whether the premise and hypothesis are in an implication relationship. If the result is implication, the question is considered answerable; if not, the question is unanswerable.

[0043] In the embodiments of the present invention, the following formulas are all basic formulas in the prior art neural network.

[0044] (1) Word Embedding Layer

[0045] First, we embed the input context and question into words, and use GloVe and BERT models to perform static and dynamic embedding respectively to form word vectors, which are recorded as C GloVe , C BERT , Q GloVe , Q BERT , use char-CNN to embed the input characters, denoted as C char , Q char , concatenate the three vectors obtained above to obtain formulas (1) and (2):

[0046] h c =Concat(C GloVe , C BERT , C char ) (1)

[0047] h q =Concat(Q GloVe , Q BERT , Q char ) (2)

[0048] Concat represents the vector concatenation operation. The concatenated vectors are input into the highway network. The highway network calculation is shown in formulas (3)-(5):

[0049] g=σ(W g h+b g ) (3)

[0050] t=ReLU(W t h+b t ) (4)

[0051] h′=g⊙t+(1-g)h (5)

[0052] Among them, W g and W t is the trainable weight, b g and b t is the bias, h represents h c and h q ⊙ represents the matrix dot product multiplication operation, σ represents the sigmoid operation, ReLU is the activation function, and h′ represents the output of the word embedding layer.

[0053] (2) Coding layer

[0054] This layer applies stacked encoding blocks to extract the features of the input data and reduce the feature dimension of the input data. Depthwise separable convolution, self-attention layer and fully connected layer are used here. LN is used between layers to make the data more stable.

[0055] (3) External knowledge layer

[0056] Extract the subgraph related to context and question from ConceptNet5.5, and use GAT and diffpool to aggregate features to obtain the knowledge representation k of the graph, which is concentrated in context and question. Use the softmax function to obtain the knowledge attention scores of context and question respectively, as shown in formulas (6) and (7):

[0057] c atn =softmax(h C k) (6)

[0058] q atn =softmax(h Q k) (7)

[0059] The knowledge attention score is used to calculate the knowledge-aware context representation and question representation, as shown in formulas (8) and (9):

[0060]

[0061]

[0062] The knowledge-aware representation is fused with the original representation through the gate unit to obtain the final representation and

[0063] (4) Attention Interaction Layer

[0064] The fused expression obtained above is input into the Context2Question and Question2Context attention blocks. The attention mechanism here is consistent with that in BiDAF, where Represent the i-th word in the context and the j-th word in the question respectively. The attention score si of the i-th word in the context with respect to the j-th word in the question is calculated by formula (10) j :

[0065]

[0066] Among them, Wsimilar are trainable parameters.

[0067] By s ij You can get the attention score matrix S, the attention of Context2Question According to formula (11) (12), we can get:

[0068]

[0069]

[0070] The attention of Question2Context can be obtained through formulas (13)-(15):

[0071]

[0072]

[0073]

[0074] The final question-aware context representation combines the context, question, and external knowledge graph. The calculation of the final question-aware context representation is shown in formula (16):

[0075]

[0076] (5) Stacked coding layers

[0077] Using stacked encoders to further process the above g c , the encoder here has the same structure as the encoder in the encoding layer (2).

[0078] (6) Answer Verification Layer

[0079] The judgment problem between the answer obtained by the stacked coding layer (5) and the true answer is converted into a text implication discrimination task. The predicted answer is taken as the premise P and the true answer is taken as the hypothesis H. They are input into the Bi-LSTM for encoding and the results are A and B, respectively. p and A h , and then perform a softmax operation on it to achieve a binary classification of implied and non-implied. If the predicted result is 0, it means the question is answerable, otherwise it is unanswerable.

[0080] Example 1

[0081] An embodiment of the present invention provides a method for constructing a question-answering system based on a BERT model and an external knowledge graph, comprising the following steps:

[0082] Step 1: The above context1 and question1 are the initial input data. Use GloVe and BERT to perform static and dynamic word embedding on context1 and question1 respectively to obtain C GloVe1 , C BERT1 , Q GloVe1 , Q BERT1 , which are the word vector representations of context1 and question1 respectively. Then use char-CNN to embed the input characters and get C char1 , Q char1 , which is the character vector representation of context1 and question1. Use the highway network to concatenate the above three vectors to get h c1 , h q1 .

[0083] Step 2: h obtained in step 1 c1 , h q1 Input to the stacked encoder in step 2 and get the output as h C1 , h Q1 .

[0084] Step 3: First, based on context1 and question1, extract the entity nodes related to them from ConceptNet5.5 and construct a subgraph. Then use a diffpool network to cluster the obtained subgraph to compress the obtained subgraph. Then use the GAT network to extract the features of each node and aggregate them into a node, denoted as k1, as the knowledge representation of context1 and question1, and integrate k1 into h C1 , h Q1 Further calculate the knowledge attention score c of context1 and question1 atn1 ,q atn1 , and calculate the knowledge-aware representation c of context1 and question1 through their attention scores k ,q k Finally, the perception representation and the output obtained in step 2 are fused through the gate unit to obtain the final expression and

[0085] Step 4: Get the and Input into the Context2Question and Question2Context attention blocks, calculate the mutual perception attention score matrix A, B, and fuse it with the perception representation obtained in step 3. The final question perception context representation combines the external knowledge graph part. The final output of this step is g c .

[0086] Step 5: Input the output from step 4 into the stacked encoder. This step stacks three encoders and calculates three outputs, denoted as M0, M1, and M2. M0 and M1 calculate the probability S1 of the answer's starting position, and M0 and M2 calculate the probability E1 of the answer's ending position.

[0087] Step 6: The predicted value of the answer obtained by the model is recorded as A p1 , the true value of the answer in the data set is recorded as A h1 . p1 As a premise, A h1 Treat them as hypotheses, perform textual entailment analysis on them, input them into a bidirectional LSTM for encoding, and then use the softmax function for binary classification. If the premise and hypothesis are implied, the question is considered answerable; if the relationship is not implied, the question is considered unanswerable.

[0088] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention. These should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for constructing a question-answering system based on the BERT model and an external knowledge graph, characterized in that: The following steps are involved: Step A: Perform word embedding and character embedding on the input words, and obtain the embedding vector that can be input into the neural network after preprocessing; Step B: Extract features using a stacked encoder based on the embedding vector obtained in step A. Step C: Use an external knowledge graph to increase the representation of the relationship between texts, recorded as the knowledge representation of context and question, and fuse the vector obtained in step B with the knowledge representation; Step D: Based on the representation obtained in step C, use the attention mechanism to calculate the attention score matrix to obtain the question-aware context representation; Step E: Obtain the question-aware context representation according to step D, use the stacked model encoder to obtain the final embedding representation, and use the activation function to obtain the probability prediction of the start and end positions of the answer; Step F: Based on the predicted answer obtained in step E, perform textual implication recognition with the actual answer input in step A to determine whether the question is answerable; Step A combines GloVe and BERT to perform word embedding simultaneously to form word embedding vectors; uses CNN to embed the characters of each word to form character vectors; and uses the highway network to concatenate the word embedding vectors and character vectors to obtain an embedding vector that can be input into the neural network. The specific process is as follows: First, we embed the input context and question into words, and use GloVe and BERT models to perform static and dynamic embedding respectively to form word vectors, which are recorded as , , , , use char-CNN to embed the input characters, recorded as , , concatenate the three vectors obtained above to obtain formulas (1) and (2): ， ， (1) ， ， (2) Among them, Concat represents the vector concatenation operation; the concatenated vector is input into the highway network, and the highway network calculation is shown in formulas (3)-(5): (3) (4) (5) in, and are trainable weights, and is the bias, represent and Any one of represents the matrix dot product multiplication operation, Represents sigmoid operation, ReLU is the activation function, Represents the output of the word embedding layer.

2. The method for constructing a question-answering system based on a BERT model and an external knowledge graph according to claim 1, characterized in that: In step C, based on each context in the SQuAD2.0 dataset, entities and their related relationships are extracted from the knowledge graph ConceptNet5.5 to form a subgraph. The GAT network is used to learn the embedded representation of the subgraph, and the features of each node in the graph are aggregated into one node to form the final knowledge representation. It is then integrated with the context and question as the input of subsequent modules.

3. The method for constructing a question-answering system based on a BERT model and an external knowledge graph according to claim 1, wherein: In step F, the predicted answer is used as the premise and the true answer is used as the hypothesis. The premise and the hypothesis are judged based on the textual implications. Whether the answer to the question is answerable is determined based on whether there is an implication relationship between the two.

4. The method for constructing a question-answering system based on a BERT model and an external knowledge graph according to claim 3, wherein: In step F, a bidirectional LSTM is used to perform binary classification prediction on the premise and hypothesis to determine whether the premise and hypothesis are in an implication relationship. If the result is implication, the question is considered answerable; if not, the question is unanswerable.

Images