A knowledge question answering method based on pre-trained model and graph convolutional neural network

Abstract

The present invention belongs to the field of natural language processing, and specifically relates to a knowledge question-answering method based on a pre-trained model and a graph convolutional neural network; the method comprises: constructing a knowledge subgraph according to question information; improving the knowledge subgraph by using a PageRank algorithm; encoding the question by using a pre-trained model to obtain an initial question vector; encoding the knowledge subgraph by using a graph convolutional neural network to obtain an initial entity vector; updating the knowledge subgraph according to the initial question vector and the initial entity vector; calculating the relationship scores of all relationships in the updated knowledge subgraph, and constructing a knowledge subgraph relationship set according to the relationship scores; calculating the initial question-answer matching score; calculating the final question-answer matching score between the question and each candidate answer, and selecting the candidate answer with the highest final question-answer matching score as the answer to the question; the present invention can improve the answer accuracy of multi-hop reasoning knowledge questions and answers, and has broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a knowledge question answering method based on a pre-trained model and a graph convolutional neural network. Background Art

[0002] With the rapid development of the Internet, the amount and format of network information are becoming more and more complex and diverse. On the one hand, it brings rich and colorful information to users, but on the other hand, it is becoming increasingly difficult for people to easily obtain effective information online. The cost of obtaining effective information online is increasing. Usually, people are accustomed to obtaining information through keyword retrieval methods, but this method lacks semantic understanding of the problem, resulting in differences in the expression of needs; at the same time, people need to spend a lot of time and energy to screen the returned results to eliminate useless information, which is inefficient. Knowledge question answering is an advanced way of information retrieval, which can automatically, accurately and clearly understand the questions given by users, and return the best search results by retrieving the knowledge graph, knowledge base or corpus related to the question.

[0003] Nowadays, faced with a large amount of digital and knowledge-based information, one of the key technical issues facing knowledge question answering is the understanding and reasoning of triples in large-scale knowledge graphs. For complex questions that require multi-hop reasoning, the number of entities will increase exponentially with the increase in the number of hops, and a large number of candidate answer entities will also make it difficult to find answers. At present, most knowledge question answering methods focus on finding answers on a triple connected to the question. However, in practical applications, the problems are more complicated, and the answers are usually not found through only one triple. It is often necessary to find the answers by reasoning multiple connected triples, but multi-hop reasoning leads to the problem of low accuracy of the answers found. Therefore, there is an urgent need for a multi-hop reasoning knowledge question answering method that improves the accuracy of the answers. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a knowledge question answering method based on a pre-trained model and a graph convolutional neural network, the method comprising:

[0005] S1: Obtain question information and build a knowledge subgraph based on the question information;

[0006] S2: Based on the question information, the PageRank algorithm is used to calculate the PageRank scores of all entities in the external knowledge graph; the N entities with the highest PageRank scores are added to the knowledge subgraph;

[0007] S3: Use the pre-trained model to encode the question and obtain the initial question vector;

[0008] S4: Use graph convolutional neural network to encode the knowledge subgraph and obtain the initial entity vector;

[0009] S5: Based on the initial question vector and the initial entity vector, the knowledge subgraph is updated using the PageRank control propagation mechanism to obtain an updated knowledge subgraph;

[0010] S6: Use the sigmoid function to calculate the relationship scores of all relationships in the updated knowledge subgraph, and form the relationships with relationship scores greater than 0.5 into a knowledge subgraph relationship set;

[0011] S7: Obtain the answer set of the question, select candidate answers from the answer set according to the knowledge subgraph and obtain the candidate answer entity; obtain the question-answer relationship set according to the candidate answer entity and the central entity, and obtain the initial question-answer matching score according to the knowledge subgraph relationship set and the question-answer relationship set;

[0012] S8: Calculate the final question-answer matching score between the question and each candidate answer based on the initial question-answer matching score and the candidate answers, and select the candidate answer with the highest final question-answer matching score as the answer to the question.

[0013] Preferably, the formula for calculating the PageRank score of an entity is:

[0014]

[0015] Among them, PR(u) represents the PageRank score of entity u, PR(v) represents the PageRank score of entity v, and W v represents the relationship weight between entity u and entity v, N e (u) represents the set of adjacent entities of entity u.

[0016] Preferably, the process of encoding the question using a pre-trained model includes: using the RoBERTa pre-trained model to embed the question into a 768-dimensional vector to obtain a vector representation of the question; inputting the vector representation of the question into four fully connected layers and processing it using the ReLU activation function to obtain an initial question vector.

[0017] Preferably, the formula for encoding the knowledge subgraph using a graph convolutional neural network is:

[0018]

[0019] in, represents the state of entity v after passing through the j+1th layer of graph convolutional neural network, W represents the state of the adjacent entity u of entity v after passing through the j-th layer of graph convolutional neural network, (j) represents the first trainable parameter, b (j) represents the second trainable parameter, N e(v) represents the set of adjacent entities of entity v.

[0020] Preferably, the formula for the PageRank control propagation mechanism is:

[0021]

[0022]

[0023] in, represents the initial PageRank score of entity v, |ε| represents the number of central entities in the question, and ε represents the central entity set. represents the PageRank score of entity v at the lth iteration, represents the hyperparameter, α vv′ Indicates the importance of entity v's adjacent entity to entity v, N represents the PageRank score of entity v's adjacent entity on the knowledge subgraph at the l-1th iteration, e (v) represents the set of adjacent entities of entity v.

[0024] Preferably, the formula for updating the knowledge subgraph is:

[0025]

[0026] in, represents the representation of entity v at the lth iteration, represents the representation of problem q at the l-1th iteration, α vv′ Indicates the importance of entity v's adjacent entity to entity v, represents the PageRank score of entity v's adjacent entity on the knowledge subgraph at the l-1th iteration, h r The vector representation of the relationship r between entity v and its adjacent entity v′, It represents the representation of the adjacent entity v′ of entity v at the l-1th iteration, and FFN represents the feed-forward neural network.

[0027] Preferably, the formula for calculating the relationship scores of all relationships in the knowledge subgraph is:

[0028]

[0029] Among them, S represents the relationship score, h r represents the relationship vector, h q Represents the problem vector.

[0030] Preferably, the formula for selecting a candidate answer from the answer set is:

[0031]

[0032] Among them, {a} q represents the answer set of the question, G represents the knowledge subgraph, q represents the question, σ represents the sigmoid function, represents the final representation of entity v after updating, W represents the third trainable parameter, b represents the fourth trainable parameter, Pr(v∈{a} q |G,q) represents the probability that entity v in the knowledge subgraph G is the answer to question q.

[0033] Preferably, the formula for calculating the initial question-answer matching score is:

[0034] Score=|R a ∩R a′ |

[0035] Among them, Score represents the initial question-answer matching score, R a represents the knowledge subgraph relation set, R a′ represents the question-answer relation set, |R a ∩R a′ | represents the number of elements in the intersection of the knowledge subgraph relationship set and the question-answer relationship set.

[0036] Preferably, the formula for calculating the final question-answer matching score is:

[0037]

[0038] Among them, φ represents the final question-answer matching score, h q represents the problem vector, h a Represents the candidate answer vector, and Score represents the initial question-answer matching score.

[0039] The beneficial effects of the present invention are as follows: the present invention uses the RoBERTa pre-trained model to encode questions, which can capture more contextual information and improve the quality of encoding; the present invention uses GCN to encode knowledge subgraphs, and the multi-layer GCN structure can better extract feature information in the knowledge subgraphs; entities in the knowledge subgraphs obtain information from their adjacent entities based on the questions and perform iterative updates, so that the entities can capture more information; when selecting candidate answers, the present invention not only considers the matching degree between the question and each candidate answer, but also considers the matching degree between the question and each relationship; the present invention can improve the accuracy of answers to multi-hop reasoning knowledge questions and answers, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the knowledge question answering method based on the pre-trained model and graph convolutional neural network in the present invention;

[0041] Figure 2A schematic diagram of the structure of the knowledge question answering method based on a pre-trained model and a graph convolutional neural network in the present invention;

[0042] Figure 3 Schematic diagram of the knowledge subgraph updating process in the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention proposes a knowledge question answering method based on a pre-trained model and a graph convolutional neural network. Figure 1 As shown, the method includes:

[0045] S1: Obtain question information and build a knowledge subgraph based on the question information;

[0046] S2: Based on the question information, the PageRank algorithm is used to calculate the PageRank scores of all entities in the external knowledge graph; the N entities with the highest PageRank scores are added to the knowledge subgraph;

[0047] S3: Use the pre-trained model to encode the question and obtain the initial question vector;

[0048] S4: Use graph convolutional neural network to encode the knowledge subgraph and obtain the initial entity vector;

[0049] S5: Based on the initial question vector and the initial entity vector, the knowledge subgraph is updated using the PageRank control propagation mechanism to obtain an updated knowledge subgraph;

[0050] S6: Use the sigmoid function to calculate the relationship scores of all relationships in the updated knowledge subgraph, and form the relationships with relationship scores greater than 0.5 into a knowledge subgraph relationship set;

[0051] S7: Obtain the answer set of the question, select candidate answers from the answer set according to the knowledge subgraph and obtain the candidate answer entity; obtain the question-answer relationship set according to the candidate answer entity and the central entity, and obtain the initial question-answer matching score according to the knowledge subgraph relationship set and the question-answer relationship set;

[0052] S8: Calculate the final question-answer matching score between the question and each candidate answer based on the initial question-answer matching score and the candidate answers, and select the candidate answer with the highest final question-answer matching score as the answer to the question.

[0053] like Figure 2 As shown, a preferred embodiment of the present invention is as follows:

[0054] Get question information, which includes the central entity of the labeled question and the question vector. The set of central entities is denoted as ε={e|e∈q}, where e represents the central entity and q represents the question. Add all central entities in the central entity set to the blank graph to obtain the knowledge subgraph G.

[0055] Obtain the external knowledge graph, use the PageRank algorithm to calculate the PageRank score of each entity in the knowledge graph from the inside out, and set the PageRank score of the outermost entity; the PageRank score calculation formula of the entity is:

[0056]

[0057]

[0058] Among them, PR(u) represents the PageRank score of entity u, PR(v) represents the PageRank score of entity v, and N e (u) represents the set of adjacent entities of entity u, v(r) represents the relationship vector between entities, v(q) represents the question vector, and W v is the cosine similarity between the relationship vector and the question vector, which represents the relationship weight between entity u and entity v.

[0059] Sort all entities by scores from high to low, and take the first N entities. The size of N is set according to the size of the knowledge graph, for example, N is 50; add the selected N entities, the relationships between entities, and the relationships between entities and the central entity to the knowledge subgraph G.

[0060] Preferably, in order to capture more context information and improve the quality of question encoding, the present invention uses the RoBERTa pre-trained model to encode the question. The specific process is: the RoBERTa pre-trained model is used to embed the question q into a 768-dimensional vector to obtain a vector representation of the question; the vector representation of the question is input into four fully connected layers and activated and mapped to a space of a specific dimension using the ReLU activation function to obtain a vector h q ∈R n , R n Represents n-dimensional vector space; vector h q As the initial problem vector, denoted as The pre-trained model for encoding the question is not limited to the RoBERTa pre-trained model, and other pre-trained models can also be used to encode the question. The specific process is: use the pre-trained model to process the question q to obtain a vector representation of the question; input the vector representation of the question into the fully connected layer and use the ReLU activation function to activate and map it to a space of a specific dimension to obtain an initial question vector.

[0061] The graph convolutional neural network GCN is used to encode the knowledge subgraph G. The multi-layer GCN structure can better extract the feature information in the knowledge subgraph. The formula is:

[0062]

[0063] in, represents the state of entity v after passing through the j+1th layer of graph convolutional neural network, W represents the state of the adjacent entity u of entity v after passing through the j-th layer of graph convolutional neural network, (j) represents the first trainable parameter, b (j) represents the second trainable parameter, N e (v) represents the set of adjacent entities of entity v.

[0064] The final entity vector is input into a linear layer to map the entity vector to the same dimension as the question, denoted as h v ∈R n , h v As the initial entity vector, denoted as

[0065] The attention mechanism is used to calculate the importance of one entity to another entity. The specific process is:

[0066] Calculate the entity's attention coefficient using the formula:

[0067]

[0068] Where W′ represents the attention training parameter, Represents the vector representation of entity i.

[0069] The importance of entity j to entity i is calculated based on the attention coefficient. The formula is:

[0070]

[0071] Use the softmax function to normalize the importance of entity j to entity i:

[0072]

[0073] like Figure 3As shown in Figure 1, according to the initial question vector and the initial entity vector, the knowledge subgraph is updated so that the entity aggregates the information of adjacent entities based on the question. The update formula is:

[0074]

[0075] in, represents the representation of entity v at the lth iteration, represents the representation of problem q at the l-1th iteration, α vv′ Indicates the importance of entity v's adjacent entity to entity v, represents the PageRank score of entity v's adjacent entity on the knowledge subgraph at the l-1th iteration, h r The vector representation of the relationship r between entity v and its adjacent entity v′, It represents the representation of the adjacent entity v′ of entity v at the l-1th iteration, and FFN represents the feed-forward neural network.

[0076] The iterative update of the problem is to iteratively represent the central entity in the problem, and the formula is:

[0077]

[0078] In the process of updating entities, in order to ensure that aggregation is completed only on the central entity of the current problem and the corresponding adjacent entities without being affected by other entities, the present invention uses the PageRank control propagation mechanism to control the update process of the knowledge subgraph; the formula of the PageRank control propagation mechanism is:

[0079]

[0080]

[0081] in, represents the initial PageRank score of entity v, |ε| represents the number of central entities in the question, and ε represents the central entity set. represents the PageRank score of entity v at the lth iteration, represents the hyperparameter, α vv′ Indicates the importance of entity v’s adjacent entity v to entity v.

[0082] Get the answer set of the question {a} q , select candidate answers from the answer set according to the knowledge subgraph and obtain the candidate answer entity, and map the updated entity representation vector to the binary classification through the fully connected layer:

[0083]

[0084] Among them, {a} q represents the answer set of the question, G represents the knowledge subgraph, q represents the question, σ represents the sigmoid function, represents the final representation of entity v after updating, W represents the third trainable parameter, b represents the fourth trainable parameter, Pr(v∈{a} q |G,q) represents the probability that entity v in the knowledge subgraph G is the answer to question q.

[0085] Based on the binary classification of the final representation of all entities in the knowledge subgraph, candidate answers are selected from the answer set, and the candidate answer set is denoted as a′.

[0086] The sigmoid function is used to calculate the relationship scores of all relationships in the updated knowledge subgraph. The formula is:

[0087]

[0088] Among them, S represents the relationship score, h r represents the relationship vector, h q represents the problem vector, T Represents a transpose operation.

[0089] Select all relations with a score greater than 0.5 to form the knowledge subgraph relation set R a The shortest path (relationship) between the candidate answer entity in the candidate answer set a′ and the central entity in the question in the knowledge subgraph constitutes the question-answer relationship set R a′ , define the initial question-answer matching score:

[0090] Score=|R a ∩R a′ |

[0091] Among them, |R a ∩R a′ | represents the number of elements in the intersection of the knowledge subgraph relationship set and the question-answer relationship set.

[0092] In order to improve the accuracy of the selected answer, the present invention not only considers the matching degree between the question and each candidate answer, but also considers the matching degree between the question and each relationship; the final question-answer matching score of the question and each candidate answer is calculated according to the initial question-answer matching score and the candidate answer, and the calculation formula is:

[0093]

[0094] Among them, φ represents the final question-answer matching score, h q represents the problem vector, h a Represents the candidate answer vector.

[0095] Compare the final question-answer matching scores of all candidate answers, and select the candidate answer with the highest final question-answer matching score as the answer to the question.

[0096] The present invention uses the RoBERTa pre-trained model to encode questions, which can capture more contextual information and improve the quality of encoding; uses GCN to encode knowledge subgraphs, and can better extract feature information in knowledge subgraphs through a multi-layer GCN structure; entities in knowledge subgraphs obtain information from their adjacent entities based on questions and perform iterative updates, so that entities can capture more information; when selecting candidate answers, the present invention not only considers the matching degree between the question and each candidate answer, but also considers the matching degree between the question and each relationship; the present invention can improve the accuracy of answers to multi-hop reasoning knowledge questions and answers, and has broad application prospects.

[0097] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge question answering method based on a pre-trained model and a graph convolutional neural network, characterized in that: include: S1: Obtain question information and build a knowledge subgraph based on the question information; S2: Based on the question information, the PageRank algorithm is used to calculate the PageRank scores of all entities in the external knowledge graph; the N entities with the highest PageRank scores are added to the knowledge subgraph; the formula for calculating the PageRank score of an entity is: Among them, PR(u) represents the PageRank score of entity u, PR(v) represents the PageRank score of entity v, and W v represents the relationship weight between entity u and entity v, N e (u) represents the set of adjacent entities of entity u; S3: Encode the question using the pre-trained model to obtain an initial question vector; the process of encoding the question using the pre-trained model includes: embedding the question into a 768-dimensional vector using the RoBERTa pre-trained model to obtain a vector representation of the question; inputting the vector representation of the question into four fully connected layers and processing it using the ReLU activation function to obtain an initial question vector; S4: Use graph convolutional neural network to encode the knowledge subgraph to obtain the initial entity vector; the formula for encoding the knowledge subgraph using graph convolutional neural network is: in, represents the state of entity v after passing through the j+1th layer of graph convolutional neural network, W represents the state of the adjacent entity u of entity v after passing through the j-th layer of graph convolutional neural network, (j) represents the first trainable parameter, b (j) represents the second trainable parameter, N e (v) represents the set of adjacent entities of entity v; S5: According to the initial question vector and the initial entity vector, the knowledge subgraph is updated using the PageRank control propagation mechanism to obtain an updated knowledge subgraph; the formula of the PageRank control propagation mechanism is: in, represents the initial PageRank score of entity v, |ε| represents the number of central entities in the question, and ε represents the central entity set. represents the PageRank score of entity v at the lth iteration, represents the hyperparameter, α vv′ Indicates the importance of entity v's adjacent entity to entity v. Represents the PageRank score of entity v's adjacent entity on the knowledge subgraph at the l-1th iteration; The formula for updating the knowledge subgraph is: in, represents the representation of entity v at the lth iteration, represents the representation of problem q at the l-1th iteration, represents the PageRank score of entity v's adjacent entity on the knowledge subgraph at the l-1th iteration, h r The vector representation of the relationship r between entity v and its adjacent entity v′; S6: The sigmoid function is used to calculate the relationship scores of all relations in the updated knowledge subgraph, and the relations with relationship scores greater than 0.5 form the knowledge subgraph relationship set; the formula for calculating the relationship scores of all relations in the knowledge subgraph is: Among them, S represents the relationship score, h q represents the problem vector; S7: Get the answer set of the question, select candidate answers from the answer set according to the knowledge subgraph and obtain the candidate answer entity; obtain the question-answer relationship set according to the candidate answer entity and the central entity, and obtain the initial question-answer matching score according to the knowledge subgraph relationship set and the question-answer relationship set; the formula for selecting candidate answers from the answer set is: Among them, {a} q represents the answer set of the question, G represents the knowledge subgraph, σ represents the sigmoid function, W represents the third trainable parameter, b represents the fourth trainable parameter, Pr(v∈{a} q |G,q) represents the probability that entity v is the answer to question q in the knowledge subgraph G; The formula for calculating the initial question-answer matching score is: Score=|R a ∩R a′ | Among them, Score represents the initial question-answer matching score, R a represents the knowledge subgraph relation set, R a′ represents the question-answer relation set, |R a ∩R a′ | represents the number of elements in the intersection of the knowledge subgraph relationship set and the question-answer relationship set S8: Calculate the final question-answer matching score between the question and each candidate answer based on the initial question-answer matching score and the candidate answers, and select the candidate answer with the highest final question-answer matching score as the answer to the question; the formula for calculating the final question-answer matching score is: Among them, φ represents the final question-answer matching score, h a Represents the candidate answer vector.

Images