A Relationship Selection Method Based on BERT Siamese Attention Network and Fuded Graph Embedding Features

Abstract

This invention discloses a relation selection method based on BERT Siamese Attention Network and fused graph embedding features. It employs named entity recognition technology to extract named entities from user questions, links these entities to topic entities in a knowledge graph using entity linking technology, queries the knowledge graph for three-hop relationships connecting to the entities, obtaining a candidate relation set. Relationships are selected from this set based on the question. The question and a relation are encoded through a network to obtain two vectors. The cosine similarity between the two vectors is calculated as a semantic similarity score, and the relation with the highest score is selected. The knowledge graph is then queried for the triple <topic entity, highest-scoring relation, attribute value>, and the attribute value contained in the triple is returned as the answer to the question. This invention uses the advanced BERT pre-trained model as a feature extractor for both the question and relation text, enhancing the expressive power of the representation vectors and contributing to improved relation selection performance.

Description

Technical Field

[0001] This invention relates to the field of knowledge graphs, and in particular to a method for selecting the relationship between BERT Siamese attention network and fused graph embedding features. Background Technology

[0002] In recent years, with the development of knowledge graph technology, knowledge graph-based intelligent question answering has become an important and challenging research area in automatic question answering. Its core is to identify entities and attributes in the question, retrieve the attribute values ​​corresponding to the entities and attributes on the knowledge graph, and then return them as answers to the user. Since knowledge graphs can be automatically constructed from structured, semi-structured information and text, the time required for answer configuration can be significantly reduced.

[0003] Current research on knowledge graph-based question answering systems mainly includes: ① Converting questions into corresponding query statements on the knowledge graph through syntax parsing, thus achieving knowledge graph-based question answering. Its advantages include no need to configure answers and high accuracy, but its disadvantage is that it converts user questions into graph query statements through rules, thus limiting the effectiveness of question answering to the quality and quantity of rules. ② Implementing knowledge graph-based question answering through attribute selection using deep neural network models. Its advantages are that deep neural networks can recognize diverse questions and are not limited by rules, but its disadvantage is that the model's performance still needs improvement in complex questions.

[0004] In summary, due to the disadvantages and limitations of syntax parsing methods, current research hotspots and mainstream methods are based on deep neural network models for relation selection to implement knowledge graph question answering. In implementing knowledge graph-based question answering using the current mainstream deep neural network-based relation selection method, the performance of the relation ranking and selection stage in the current algorithm model needs improvement. The difficulty of relation selection lies in the fact that the relation description in the question may differ significantly from the relation names in the knowledge graph. Selecting a relation related to the current question from the attribute set A is called relation selection. Therefore, how to design a model to effectively match questions with attributes is one of the research focuses of this invention. Summary of the Invention

[0005] The main objective of this invention is to provide a relation selection method based on BERT Siamese Attention Network and fused graph embedding features. To address this problem, a Siamese Attention Network is designed, which borrows the idea of ​​the Siamese network used in face recognition to calculate the similarity between two faces. Two BERT networks with shared weights encode the question q and relation p respectively. During the encoding of question q and relation p, an attention mechanism is used to fuse the knowledge graph embedding vectors, resulting in a semantic vector S for the question and relation. qand S p The model then calculates the cosine similarity score between question q and all candidate relations p∈A, and finally sorts them to obtain the relation with the highest score, which is the final relation.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A relation selection method based on BERT Siamese attention network and fused graph embedding features includes the following steps:

[0008] Step 1, Topic Entity Acquisition: Use named entity recognition technology to obtain named entities from the user's question, and use entity linking technology to link the named entities to topic entities in the knowledge graph;

[0009] Step 2, Generate candidate relations: Query the knowledge graph for relations within 3 hops that connect to the entity.

[0010] Obtain the candidate relation set;

[0011] Step 3, Relation Ranking and Selection: Based on the question, select relations from the candidate relation set. After encoding the question and a relation through the network, two vectors are obtained. The cosine similarity between the two vectors is calculated as the semantic similarity score, and the relation with the highest score is selected.

[0012] Step 4, Generate Answer: Query the knowledge graph for the highest-scoring relation: triple < topic entity.

[0013] The attribute value> method returns the attribute value contained in the triple as the answer to the question.

[0014] A further improvement of the present invention is that, in step three, during encoding, the knowledge graph is fused into the embedded vector through an attention mechanism, and the resulting semantic vector S of the question and relationship is obtained after encoding. q and S p .

[0015] A further improvement of the present invention is that the working steps of the relation selection model in step three are as follows: first, relation encoding is performed to obtain the relation encoding vector S. p Then, the problem is encoded to obtain the problem's encoding vector S. q Finally, similarity scores are calculated and ranked to obtain the relationship with the highest score, which is the final relationship.

[0016] A further improvement of the present invention is that the relation attribute encoding includes the following steps: first, selecting the set of relations within 3 hops of topic entity e as candidate relations, where p represents the path from topic entity e to candidate relations; then, feeding the text into the BERT model to obtain the text semantic features of relation p; and finally, fusing the text semantic features with the graph features through a multi-head attention mechanism to obtain the encoding vector of the relation.

[0017] A further improvement of the present invention is that the question encoding includes the following steps: the question text BERT feature extractor obtains the vector representation q1 to q of each word in the question. m By using a multi-head attention mechanism, textual semantic features and graph features are fused to obtain the encoding vector for the relation problem.

[0018] A further improvement of the present invention is that the similarity score calculation includes the following steps: first, the representation vector of the encoded question and the representation vector of the relation are concatenated and input into a feedforward network, and then the final similarity score is calculated; second, the model calculates the similarity score between question q and all candidate relations, and finally sorts them to obtain the relation with the highest score, which is the final relation.

[0019] A further improvement of this invention is that, during the training phase of the relation selection model, a hinge loss function is used, as shown in the following formula:

[0020]

[0021] Wherein, S(S q S p ), The score is calculated separately for the relationship between the problem and the positive example, and the relationship between the problem and the negative example. The hyperparameter γ is a positive real value that represents the interval between the positive and negative examples.

[0022] Compared with existing technologies, this invention addresses the problem of poor performance in the relation selection subtask of knowledge graph-based automatic question answering under complex problem conditions. It proposes a relation selection method based on a Siamese BERT attention network and fused graph embedding features, aiming to improve the performance of the relation selection subtask and thus enhance the performance of knowledge graph-based automatic question answering. On one hand, it employs the state-of-the-art BERT pre-trained model as a feature extractor for question and relation text. The BERT pre-trained model learns semantic information from a large amount of unlabeled text through self-supervised learning. On the other hand, it trains the knowledge graph using a graph representation learning model to obtain graph embedding representation vectors of entities and relations in the knowledge graph. Then, it uses a multi-head attention mechanism to fuse the graph embedding representation vectors with text semantics to obtain semantic knowledge representation vectors of questions and relations, enhancing the expressive power of the representation vectors and contributing to improved relation selection performance. Attached Figure Description

[0023] To more clearly illustrate the technical solution of the present invention, the drawings used in the technical description of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the overall structure of the relationship selection method based on BERT Siamese Attention Network and fused graph embedding features of the present invention.

[0025] Figure 2 This is an example knowledge graph diagram of the relationship selection method based on BERT Siamese Attention Network and fused graph embedding features proposed in this invention.

[0026] Figure 3 This invention provides an automatic question-answering process based on knowledge graphs, which is a relation selection method based on BERT Siamese Attention Network and fused graph embedding features. Detailed Implementation

[0027] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the present invention. In order to better illustrate the specific embodiments of the present invention, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the actual product size. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1

[0029] like Figure 1 As shown, a relationship selection method based on BERT Siamese attention network and fused graph embedding features includes the following steps:

[0030] Step 1, Topic Entity Acquisition: Use named entity recognition technology to obtain named entities from the user's question, and use entity linking technology to link the named entities to the topic entity e in the knowledge graph;

[0031] Step 2, Generate candidate relations: Query the 3-hop relations connected to entity e on the knowledge graph to obtain the candidate relation set A;

[0032] Step 3, Relation Ranking and Selection: Based on the question q and a relation p∈A, select relations from the candidate relation set. After encoding the question and a relation through a network, two vectors are obtained. The cosine similarity between the two vectors is calculated as the semantic similarity score, and the relation p with the highest score is selected. max ;

[0033] Step 4, Generate the answer: Query the triples on the knowledge graph. <e,p max The function returns the attribute value v contained in the triple as the answer to the question.

[0034] In this embodiment, during the encoding process in step three, the knowledge graph embedding vector is fused using an attention mechanism to obtain a semantic vector S of the question and relationship after encoding. q and S p .

[0035] In this embodiment, the working steps of the relation selection model in step three are as follows: First, relation encoding is performed to obtain the relation encoding vector S. p Then, the problem is encoded to obtain the problem's encoding vector S. q Finally, similarity scores are calculated and ranked to obtain the relationship with the highest score, which is the final relationship.

[0036] In this embodiment, relation (attribute) encoding includes the following steps: First, select the set of relations within 3 hops of the topic entity e as candidate relations, where p represents the path from the topic entity e to the candidate relation. If the path is e→r1→e1→r2, the text representation of p is obtained by concatenating the entity, entity type, and the beginning and end of the relation. Then, the text is fed into the BERT model to obtain the textual semantic features of relation p. Figure 1 p1 to p n The representation vector of a word, e p The graph embedding representation vector representing the path from topic entity e to candidate relation is the sum of the graph embedding representation vectors of all relations, while the graph embedding vector of a single relation is learned from the knowledge graph using a graph representation learning model; finally, a multi-head attention mechanism is used to integrate the text semantic features (p1 to p...). n ) and spectral features (e p The relationship encoding vector S is obtained by fusing the data. p .

[0037] In this embodiment, question encoding includes the following steps: the question text BERT feature extractor obtains the vector representation q1 to q1 of each word in the question. m e q This represents the graph embedding representation vector of the topic entity e identified from question q, which is then processed by a multi-head attention mechanism to integrate the textual semantic features (q1 to q2). m ) and spectral features (e q The encoding vector S for the relation problem is obtained by fusion. q .

[0038] In this embodiment, the similarity score calculation includes the following steps: First, the representation vector of the encoded question and the representation vector of the relation are concatenated and input into a feedforward network. Then, the final similarity score is calculated. Second, the model calculates the similarity score between question q and all candidate relations. Finally, the relation with the highest score is sorted and becomes the final relation.

[0039] In this embodiment, during the training phase of the relationship selection model, a hinge loss function is used, as shown in the following formula:

[0040]

[0041] Wherein, S(S q S p ), The hyperparameter represents the score calculated for the relationship between the problem and positive examples, and the relationship between the problem and negative examples, respectively. The hyperparameter is a positive real value, and γ represents the interval between positive and negative examples.

[0042] Example 2

[0043] Typically, a knowledge graph is a set of triples: <entity, relation, entity>. In the ideological domain knowledge graph of this invention, attributes are also considered "relations," therefore, many entity nodes in the ideological domain knowledge graph correspond to attribute values, such as... Figure 2 The example shown is a triple <Shenzhen, resident population, 17.56 million>, used as a knowledge graph example. An automatic question-answering process based on the knowledge graph is employed as follows: Figure 3 As shown, a knowledge graph search was performed to determine the number of permanent residents in Shenzhen.

[0044] Step 1, Topic Entity Acquisition: Named entities are obtained from the user's question using named entity recognition technology. The named entities are then linked to topic entities in the knowledge graph using entity linking technology. In this embodiment, the topic entity is Shenzhen.

[0045] Step 2, Generate candidate relations: Query the knowledge graph for relations within 3 hops that connect to the entity.

[0046] By obtaining the candidate relation set, we can obtain the candidate relation set for total GDP, population, and number of companies;

[0047] Step 3, Relation Ranking and Selection: Based on the question, relations are selected from the candidate relation set. The question and a relation are encoded by the network to obtain two vectors. The cosine similarity between the two vectors is calculated as the semantic similarity score. The relation with the highest score is selected and ranked using the Siamese model. The semantic similarity of population is 0.95, the semantic similarity of GDP is 0.02, and the semantic similarity of the number of companies is 0.01. The ranking order is population, GDP, and number of companies.

[0048] Step 4, Generate Answer: Query the knowledge graph for the highest-scoring relation: triple < topic entity.

[0049] The attribute value> method returns the attribute value contained in the triple as the answer to the question. Since the population number is the first digit, the answer to the question "How many permanent residents does Shenzhen have?" is 17.56 million.

[0050] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A relationship selection method based on BERT twin attention network and fusion graph embedding features, characterized in that, The method comprises the following steps, Step one, subject entity acquisition: using named entity recognition technology to obtain named entity from user's question, Link the named entity to the subject entity in the knowledge graph by using entity linking technology; Step two, generate candidate relations: query the relations within 3 hops connected with the entity on the knowledge graph, Get the candidate relation set; Step three, relationship ranking selection: select the relationship from the candidate relation set according to the question, encode the question and a relationship to get two vectors, calculate the cosine similarity of the two vectors as the semantic similarity score, and select the relationship with the highest score; Step four, generate answer: query the triple <subject entity, the highest scoring relationship, Attribute value> on the knowledge graph, and return the attribute value contained in the triple as the answer to the question; The working steps of the step three relationship selection model are as follows: first, relationship encoding is performed to obtain an encoded vector of the relationship Then, question encoding is performed to obtain an encoded vector of the question Finally, similarity score calculation is performed to sort and obtain the relationship with the highest score as the final relationship; the question encoding includes the following steps: a question text BERT feature extractor obtains a vector representation of each word in the question to The text semantic features and graph features are fused through a multi-head attention mechanism to obtain an encoded vector of the relationship question ; The similarity score calculation comprises the following steps: firstly, the representation vectors of the question and the relationship after encoding are spliced and input into a feedforward network, and then the final similarity score is calculated; secondly, the model calculates the similarity score of the question q and all candidate relations; finally, the relationship with the highest score is obtained by sorting, which is the final relationship; The graph feature represents the path of the subject entity to the candidate relationship, which is the sum average of the graph embedding representation vectors of all relations; The relationship encoding The method comprises the following steps: firstly, selecting a set of relations within 3 hops of a subject entity e as candidate relations, p representing a path from the subject entity e to the candidate relations; then, feeding the text into a BERT model to obtain text semantic features of the relations p; and finally, fusing the text semantic features and graph features through a multi-head attention mechanism to obtain an encoding vector of the relations .

2. The relationship selection method based on BERT twin attention network and fusion graph embedding features according to claim 1, characterized in that: In the step three, the knowledge graph embedding vector is fused by attention mechanism when encoding, and the encoding vector of the question is obtained after encoding and the encoding vector of the relationship .

3. The relationship selection method based on BERT twin attention network and fusion graph embedding features according to claim 1, characterized in that: In the training stage of the relationship selection model, the hinge loss function is adopted, and the formula is as follows: where, , denote the scores computed for the question and positive example relationship, question and negative example relationship respectively, hyperparameters is a positive real value indicating the separation between positive and negative examples.

Images