Open world knowledge graph completion method and device based on multi-attention mechanism

Abstract

The invention provides an open world knowledge graph completion method and device based on a multi-attention mechanism. The method comprises the following steps: constructing a plurality of to-be-discriminated triads including new head entities, relationships and candidate tail entities; the names and descriptions of the entities and the relationships are spliced into texts respectively, after word embedding is obtained through BERT, different weights are distributed for entity word embedding through a multi-attention mechanism layer, and then word embedding is aggregated to obtain respective text embedding; obtaining a relation in the knowledge graph and structure embedding of a candidate tail entity through a TransE model, and embedding and mapping a text into a graph-based embedding space by using an alignment function; the embedding representation of the entity is optimized through a gating mechanism, and then the structure embedding and text embedding of the relation and the candidate tail entity are fused; and inputting the obtained final embedded representation of the entities and the relationships into a scoring model, and selecting a triple with the highest possibility and a related new header entity to be supplemented into the knowledge graph. According to the method, the knowledge graph completion accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge graph completion, and specifically relates to an open-world knowledge graph completion method and device based on a multi-attention mechanism. Background Art

[0002] The core goal of knowledge graph completion is to predict and fill in missing triples, so that the triple information in the knowledge graph becomes more complete. Common knowledge graph completion methods include translation model-based, tensor decomposition model-based, neural network-based, and meta-learning-based. The core of the translation model-based method is to translate the triple (h, r, t) into an embedded representation, and obtain the probability that the triple is true by geometric reasoning on the embedded representation. The tensor decomposition model-based method mainly expresses entities and relationships in the form of tensors to capture the semantic relationship between entities and relationships. The neural network-based model is divided into many types according to the different neural networks selected. They have different feature extraction capabilities and can capture richer and more complex relationships in the data to better represent entities and relationships and enhance the accuracy of knowledge graph completion. The meta-learning-based method is mainly aimed at few-shot or zero-shot knowledge graph completion. The model can be trained on a small number of samples and can quickly adapt to new entities and relationships, thereby predicting the association between unknown entities. However, most of the existing knowledge graph completion methods are designed based on the closed world assumption, which assumes that the entity and relationship sets in the knowledge graph are fixed and unchanging, and only constructs new triples through the existing entity and relationship sets without considering the addition of new external entities. However, the knowledge in the real world is constantly evolving. In order to make the existing knowledge graph keep up with the changes in the outside world, people have proposed an open world knowledge graph completion method, which can link new entities to the knowledge graph and construct triples related to the new entities, thereby more comprehensively reflecting the latest knowledge status of the real world.

[0003] Among the current embedding-based knowledge graph completion methods, the ConMask model pays attention to the impact of relations in obtaining entity text representations, and uses a relation masking mechanism to focus on the content related to relations in the entity description text. However, this mechanism easily misses the target words, which will bring inaccuracy to the text embedding representation of the entity. The MIA model proposed by Lei Niu et al. solves this problem. It proposes a multi-attention mechanism to fully capture the important information generated from multiple interactions between entity names, relationship names, and entity descriptions. However, this method ignores the text description information of the relationship and does not take into account the structural information of the knowledge graph. The OWE model considers the structural information and text information of the entity, but is too rough when processing the text information of the new entity, and only considers the text description of the new entity while ignoring the text information in the knowledge graph, and does not make good use of the contextual features of the entity and relationship descriptions in the knowledge graph. Most other methods for open-world completion have also failed to make full use of the structural information and text information of the knowledge graph, resulting in the lack of semantic and structural information in the embedding representation of entities and relationships, making the embedding representation inaccurate, thereby reducing the accuracy of the model. Summary of the invention

[0004] Purpose of the invention: In view of the above shortcomings, the present invention proposes an open-world knowledge graph completion method and device based on a multi-attention mechanism, which can obtain more accurate embedding representation and overcome the shortcomings in current practical applications.

[0005] Technical solution: The present invention provides an open world knowledge graph completion method based on a multi-attention mechanism, comprising the following steps:

[0006] (1) Based on the given new head entity and its name and text description, as well as the knowledge graph to be completed, multiple triples to be judged are constructed, including the new head entity, the relationship, and the candidate tail entity;

[0007] (2) The names and descriptions of entities and relations are concatenated into texts respectively. After obtaining word embeddings through BERT, different weights are assigned to entity word embeddings according to the relations through a multi-attention mechanism layer, and then the word embeddings are aggregated to obtain their respective text embeddings.

[0008] (3) The TransE model is used to obtain the structural embeddings of relations and candidate tail entities in the knowledge graph, and the alignment function is used to map the text embeddings to the graph-based embedding space;

[0009] (4) The embedding representation of the entity is optimized through a gating mechanism, and then the structural embedding of the relationship and candidate tail entity is fused with the text embedding to integrate the structural and semantic information to obtain the final embedding representation of the entity and relationship;

[0010] (5) Input the final embedding representation of entities and relations into the scoring model to obtain the probability of the triple being true, and select the triple with the highest probability and the related new head entity to be added to the knowledge graph;

[0011] (6) We adjust the parameters in the alignment function and the gating mechanism by training and minimizing the loss function to achieve open-world knowledge graph completion.

[0012] Furthermore, the implementation process of step (1) is as follows:

[0013] Given a new head entity h' extracted from an open-world document fragment, and a knowledge graph G to be completed containing a set of relations R and a set of entities E, for the relation r∈R, a set of triples {(h',r,t i )|i=1,...,n}, where, t i ∈E, n is the number of entities in G.

[0014] Furthermore, the implementation process of step (2) is as follows:

[0015] Concatenate the names and text descriptions of entities and relations into a text sequence Txt r ,Txt h' , The text sequence Txt is converted into a text sequence through the Bert pre-trained language model r ,Txt h' , Vectorize and get the corresponding initial word embedding sequence V h' , V r ,

[0016] The word embedding sequence V for the entity is obtained through multiple attention mechanism layers h' and Assign a weight to each word embedding in , calculate the word embedding of the new head entity h'specific to the relation r, and the candidate tail entity t i The word embedding specific to the relation r and the new head entity h' is used to realize the semantic information interaction between entities and relations, where the attention function is as follows:

[0017]

[0018] in, Captures the jth word embedding in the entity text sequence The i-th word embedding in the relation text sequence The similarity of, where W is a weight matrix, n r Represents the number of words in the relation text sequence;

[0019] According to the attention function calculation, the j-th relation-aware word embedding corresponding to h' is Among them, |r| represents Text r The number of words in

[0020] Similarly, first calculate t i The corresponding j-th relation-aware word embedding: Then calculate the j-th new head entity-relationship-aware word embedding: where |h'| represents Txt h' The number of words in

[0021] Embed h' into relation-aware words and initial word embedding Concatenate the input Bi-LSTM fusion to get the new word embedding sequence h' Likewise, and Fusion obtains candidate tail entity t i New word embedding sequence where |t i | indicates The number of words in

[0022] Will and V r The average aggregation results are h' and t i and text embeddings of r and

[0023] Furthermore, the implementation process of step (3) is as follows:

[0024] Obtain the relation r and candidate tail entity t in the knowledge graph through the TransE model i The structural embedding Then use the alignment function to embed the text of entities and relations Mapped to the graph-based embedding space, we get

[0025] Furthermore, the implementation process of step (4) is as follows:

[0026] The gating mechanism formula specific to relation r is:

[0027] e r =e⊙σ(W e ·[e;r]+b e )

[0028] Among them, W e is the weight matrix, b eis the deviation, ⊙ is the Hadamard product operation; after the gating mechanism is optimized, the final embedding representation e of the new head entity is obtained h' ;

[0029] Embed the structure of r and text embedding Splicing, input Bi-LSTM for fusion to get the final embedding representation e r Similarly, Input gating mechanism, optimized to obtain Then concatenate the two and input them into Bi-LSTM to obtain the candidate tail entity t i The final embedding representation of

[0030] Furthermore, the scoring model in step (5) adopts the scoring function of TransE:

[0031] score(h,r,t)=||e h +e r -e t ||

[0032] Among them, e h is the head entity embedding, e t is the tail entity embedding, e r Embedding for relations.

[0033] Furthermore, the loss function in step (6) is as follows:

[0034]

[0035] in, refers to the kth candidate tail entity t k The text embedding and structure embedding optimized by the gating mechanism; the loss function is minimized by and The distance between them is used to adjust the parameters mentioned above, n t Refers to the number of candidate tail entities.

[0036] Furthermore, the alignment function is as follows:

[0037] map(v) = A r ·v+b

[0038] Among them, A r is the relationship weight matrix and b is the bias vector.

[0039] A device according to the present invention includes a memory and a processor, wherein:

[0040] A memory for storing computer programs that can be run on the processor;

[0041] A processor is used to execute the steps of the open-world knowledge graph completion method based on the multi-attention mechanism as described above when running the computer program.

[0042] A storage medium described in the present invention stores a computer program on the storage medium, and when the computer program is executed by at least one processor, the steps of the open-world knowledge graph completion method based on the multi-attention mechanism as described above are implemented.

[0043] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0044] For the task of completing the open world knowledge graph, the present invention makes full use of the text description information of entities and relationships, as well as the structural information of the knowledge graph, and combines the two kinds of information to better extract feature embedding; for the word embedding obtained by the Bert model, a multi-attention mechanism is adopted, and by assigning weights to the word embedding of the entity, the relationship and the entity are fully interacted with semantic information, and then the word embeddings are aggregated to obtain the optimal text embedding; the structural embedding of the relationship and the candidate tail entity in the knowledge graph is obtained, the text embedding is mapped to the graph-based embedding space through a mapping function, and then a gating mechanism specific to the relationship is used to optimize and obtain a better embedding representation; finally, the text embedding and the structural embedding are fused to fuse the structural information and the semantic information to obtain the final embedding representation; the present invention solves the problem that the current open world knowledge graph completion method for processing text data is too simple and does not fully combine the use of structural information and semantic information, thereby improving the feature representation ability of entities and relationships, improving the accuracy of the knowledge graph completion model, improving the completion efficiency, improving the association relationship between entities in the knowledge graph, constructing new triples, and adding new entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the open world knowledge graph completion method based on multi-attention mechanism;

[0046] Figure 2 Complete the open world knowledge graph;

[0047] Figure 3 It is the flowchart of acquiring the text embedding of the new head entity;

[0048] Figure 4 This is the flowchart for obtaining tail entity text embedding. DETAILED DESCRIPTION

[0049] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with the accompanying drawings.

[0050] like Figure 1 As shown in the figure, an open-world knowledge graph completion method based on a multi-attention mechanism includes the following steps:

[0051] Step 1: Given a new head entity, its name, and text description, as well as a knowledge graph to be completed, construct multiple triples to be discriminated that include the new head entity, relationships, and candidate tail entities.

[0052] As Figure 2 shown, open-world knowledge graph completion not only introduces a new entity h, but also constructs new triples (h, r, t). Given a new head entity h' extracted from an open-world document fragment and a knowledge graph G to be completed that includes a set of relationships R and a set of entities E, for a relationship r ∈ R, construct a set of triples to be discriminated {(h', r, t i )|i = 1,..., n}, where t i ∈ E and n is the number of entities in G.

[0053] For example, given the new head entity "Xingfu Reservoir" and the Baidu Encyclopedia text description of "Xingfu Reservoir" as follows:

[0054] Xingfu Reservoir is located in Xingfu Village, Wangcheng Town, Xinjian District, Nanchang City, Jiangxi Province. The water system it belongs to is Tongyuangang, a tributary of the Ganjiang River. It is 15.0 km away from the Xinjian urban area and 20.0 km away from Nanchang City. Its geographical location is at 115.42° east longitude and 28.40° north latitude. The main tourism resources of the Xingfu Reservoir Scenic Area are "six groups and one reservoir". The six groups are namely the water mill group, waterfall group, ancient tomb group, terraced field group, elephant rock group, and heron group. The one reservoir refers to the glittering Xingfu Reservoir. The Xingfu Reservoir Water Conservancy Scenic Area has been rated as a provincial water conservancy scenic area for its beautiful and magnificent natural scenery, unique traditional culture, and human charm. The scenic resources of the Xingfu Reservoir Water Conservancy Scenic Area are distinctively featured and have high natural aesthetic value, scientific value, historical and cultural value, health care value, and outdoor recreation value. It is a water conservancy scenic area integrating nature, culture, and ecology.

[0055] The knowledge graph G to be completed contains the entity set {"Three Gorges Reservoir", "Hubei", "Danjiangkou Reservoir", "Xin'anjiang Reservoir", "Jiangxi", "Zhejiang"}, the relationship set {"is located in"}, the triple set {(Three Gorges Reservoir, is located in, Hubei Province), (Danjiangkou Reservoir, is located in, Hubei), (Xin'anjiang Reservoir, is located in, Zhejiang)}, and the text descriptions of entities and relationships, which are not listed one by one here. By traversing the relationship set and entity set in G, the following set of triples to be discriminated can be constructed {(Xingfu Reservoir, is located in, Three Gorges Reservoir), (Xingfu Reservoir, is located in, Hubei), (Xingfu Reservoir, is located in, Danjiangkou Reservoir), (Xingfu Reservoir, is located in, Xin'anjiang Reservoir), (Xingfu Reservoir, is located in, Jiangxi), (Xingfu Reservoir, is located in, Zhejiang)}.

[0056] Step 2: If Figure 3 , Figure 4 As shown in the figure, the names and text descriptions of the entities and relations in the triples to be identified are concatenated into text sequences to obtain Txt r ,Txt h' , Use the Bert pre-trained semantic model to obtain word embeddings, then assign weights to entity word embeddings through multiple attention mechanism layers to obtain new word embeddings for entities, and then pass them through the average aggregator to obtain their respective text embeddings

[0057] (2.1) First, the text sequence Txt is converted into r ,Txt h' , Vectorize and get the corresponding initial word embedding sequence Among them, |h'|, |r|, |t i |Represents Txt h' ,Txt r , The number of words in the sequence.

[0058] (2.2) Then, the word embedding sequence V of the entity is constructed through multiple attention mechanism layers. h' and Assign a weight to each word embedding in , calculate the word embedding of the new head entity h'specific to the relation r, and the candidate tail entity t i The word embedding specific to the relation r and the new head entity h' is used to realize the semantic information interaction between entities and relations, where the attention function is as follows:

[0059]

[0060] in, Captures the jth word embedding in the entity text sequence The i-th word embedding in the relation text sequence The similarity of, where W is a weight matrix, n r Represents the number of words in the relation text sequence.

[0061] According to the attention function calculation, the j-th relation-aware word embedding corresponding to h' is Among them, |r| represents Text r The number of words in .

[0062] Similarly, first calculate t i The corresponding j-th relation-aware word embedding: Then calculate the j-th new head entity-relationship-aware word embedding: where |h'| represents Txt h' The number of words in .

[0063] Embed h' into relation-aware words and initial word embedding Concatenate the input Bi-LSTM fusion to get the new word embedding sequence h' Likewise, and Fusion obtains candidate tail entity t i New word embedding sequence

[0064] Finally, and V r The average aggregation is h' and t i and text embeddings of r.

[0065] Step 3: Obtain the relation r and candidate tail entity t in the knowledge graph through the TransE model i The structural embedding Then use the alignment function to embed the text of entities and relations Mapped to the graph-based embedding space, we get

[0066] The alignment function is as follows:

[0067] map(v) = A r ·v+b

[0068] Among them, A r is the relationship weight matrix and b is the bias vector.

[0069] Step 4: Embedding the entity After the gating mechanism is optimized, the structural embeddings of the relationship and candidate tail entities are fused with the text embeddings to fuse the structural and semantic information to obtain h', r, t i The final embedding representation of .

[0070] (4.1) where the gating mechanism formula specific to relation r is:

[0071] e r =e⊙σ(W e ·[e;r]+b e )

[0072] Among them, W e is the weight matrix, b e is the deviation, ⊙ is the Hadamard product operation. After the gating mechanism is optimized, the final embedding representation e of the new head entity is obtained h' .

[0073] (4.2) Embed the structure of r and text embedding Splicing, input Bi-LSTM for fusion to get the final embedding representation e r Similarly, Input gating mechanism, optimized to obtain Then concatenate the two and input them into Bi-LSTM to obtain the candidate tail entity t i The final embedding representation of

[0074] Step 5: The new head entity h', relation r and candidate tail entity t obtained in the above steps i The final embedding representation is sent to the triple scoring model to obtain the triple (h', r, t i ) scores, and select the triples that are most likely to be true and the new head entity h' to be added to G according to the scores.

[0075] The scoring function uses the TransE scoring function as follows:

[0076] score(h,r,t)=||e h +e r -e t ||

[0077] Among them, e h is the head entity embedding, e t is the tail entity embedding, e r Embedding for relations.

[0078] By inputting the embedding expressions of entities and relations in the above triples to be judged into the scoring model, the score of each triple can be obtained. The scoring function judges the score of the triple by judging the distance between vectors. When e h +e r The distance between t The closer the distance is, that is, the lower the score is, the higher the possibility that the triple is true. Finally, the set of triples to be judged is sorted, and the triple with the lowest score is selected to be added to the knowledge graph, that is, (China, capital, Beijing). The entity set of G after completion is expanded to {"Three Gorges Reservoir", "Hubei", "Danjiangkou Reservoir", "Xin'anjiang Reservoir", "Jiangxi", "Zhejiang", "Xingfu Reservoir"}, and the triple set is expanded to {(Three Gorges Reservoir, located in Hubei Province), (Danjiangkou Reservoir, located in Hubei), (Xin'anjiang Reservoir, located in Zhejiang), (Xingfu Reservoir, located in Jiangxi)}.

[0079] Step 6: Adjust the parameters in the alignment function and gating mechanism, including the weight matrix and bias vector, by training to minimize the loss function.

[0080] In step (4.2), the candidate tail entity t is obtained i Text embedding and structure embedding optimized by gating mechanism Then we can obtain the text embedding sequence of all candidate tail entities after optimization by the gating mechanism and structure embedding sequence The loss function is as follows:

[0081]

[0082] in, refers to the kth candidate tail entity t k The text embedding and structure embedding optimized by the gating mechanism; the loss function is minimized by and The distance between them is used to adjust the parameters mentioned above, n t Refers to the number of candidate tail entities.

[0083] The present invention also provides an apparatus, comprising a memory and a processor, wherein: the memory is used to store a computer program that can be run on the processor; the processor is used to execute the steps of the open-world knowledge graph completion method based on the multi-attention mechanism as described above when running the computer program.

[0084] The present invention also provides a storage medium having a computer program stored thereon, and when the computer program is executed by at least one processor, the steps of the open-world knowledge graph completion method based on the multi-attention mechanism as described above are implemented.

[0085] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and variations can be made without departing from the technical principles of the present invention, and these improvements and variations should also be considered as the protection scope of the present invention. The method of the present invention can be compiled into a program code, the code is stored in a computer storage medium, the program code is transmitted to a processor, and the method of the present invention is executed by the processor.

Claims

1. A method for completing an open-world knowledge graph based on a multi-attention mechanism, characterized in that: The following steps are involved: (1) Based on the given new head entity and its name and text description, as well as the knowledge graph to be completed, multiple triples to be judged are constructed, including the new head entity, relationship, and candidate tail entity; (2) The names and descriptions of entities and relations are concatenated into texts respectively. After obtaining word embeddings through BERT, different weights are assigned to entity word embeddings according to the relations through a multi-attention mechanism layer, and then the word embeddings are aggregated to obtain their respective text embeddings. (3) The TransE model is used to obtain the structural embeddings of relations and candidate tail entities in the knowledge graph, and the alignment function is used to map the text embeddings to the graph-based embedding space; (4) The embedding representation of the entity is optimized through a gating mechanism, and then the structural embedding of the relationship and candidate tail entity is fused with the text embedding to integrate the structural and semantic information to obtain the final embedding representation of the entity and relationship; (5) Input the final embedding representation of entities and relations into the scoring model to obtain the probability of the triple being true, and select the triple with the highest probability and the related new head entity to be added to the knowledge graph; (6) We adjust the parameters in the alignment function and the gating mechanism by training and minimizing the loss function to achieve open-world knowledge graph completion.

2. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The implementation process of step (1) is as follows: Given a new head entity h' extracted from an open-world document fragment, and a knowledge graph G to be completed containing a set of relations R and a set of entities E, for the relation r∈R, a set of triples {(h',r,t i )|i=1,...,n}, where, t i ∈E, n is the number of entities in G.

3. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The implementation process of step (2) is as follows: Concatenate the names and text descriptions of entities and relations into a text sequence Txt r ,Txt h' , The text sequence Txt is converted into a text sequence through the Bert pre-trained language model r ,Txt h' , Vectorize and get the corresponding initial word embedding sequence V h' , V r , The word embedding sequence V for the entity is obtained through multiple attention mechanism layers h' and Assign a weight to each word embedding in , calculate the word embedding of the new head entity h'specific to the relation r, and the candidate tail entity t i The word embedding specific to the relation r and the new head entity h' is used to realize the semantic information interaction between entities and relations, where the attention function is as follows: in, Captures the jth word embedding in the entity text sequence The i-th word embedding in the relation text sequence The similarity of, where W is a weight matrix, n r Represents the number of words in the relation text sequence; According to the attention function calculation, the j-th relation-aware word embedding corresponding to h' is Among them, |r| represents Text r The number of words in Similarly, first calculate t i The corresponding j-th relation-aware word embedding: Then calculate the j-th new head entity-relationship-aware word embedding: where |h'| represents Txt h' The number of words in Embed h' into relation-aware words and initial word embedding Concatenate the input Bi-LSTM fusion to get the new word embedding sequence h' Likewise, and Fusion obtains candidate tail entity t i New word embedding sequence where |t i | indicates The number of words in Will and V r The average aggregation results are h' and t i and text embeddings of r and 4. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The implementation process of step (3) is as follows: Obtain the relation r and candidate tail entity t in the knowledge graph through the TransE model i The structural embedding Then use the alignment function to embed the text of entities and relations Mapped to the graph-based embedding space, we get 5. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The implementation process of step (4) is as follows: The gating mechanism formula specific to relation r is: e r ⊙σ(W e ·[er]+b e ) Among them, W e is the weight matrix, b e is the deviation, ⊙ is the Hadamard product operation; after the gating mechanism is optimized, the final embedding representation e of the new head entity is obtained h' ; Embed the structure of r and text embedding Splicing, input Bi-LSTM for fusion to get the final embedding representation e r Similarly, Input gating mechanism, optimized to obtain Then concatenate the two and input them into Bi-LSTM to obtain the candidate tail entity t i The final embedding representation of 6. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The scoring model in step (5) adopts the TransE scoring function: score(h,r,t)=||e h +e r -e t || Among them, e h is the head entity embedding, e t is the tail entity embedding, e r Embedding for relations.

7. The open world knowledge graph completion method based on multi-attention mechanism according to claim 1, characterized in that: The loss function in step (6) is as follows: in, refers to the kth candidate tail entity t k The text embedding and structure embedding optimized by the gating mechanism; the loss function is minimized by and The distance between them is used to adjust the parameters mentioned above, n t Refers to the number of candidate tail entities.

8. The open world knowledge graph completion method based on multi-attention mechanism according to claim 4, characterized in that: The alignment function is as follows: map(v)=A r ·v+b Among them, A r is the relationship weight matrix and b is the bias vector.

9. A device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor, configured to execute the steps of the open-world knowledge graph completion method based on a multi-attention mechanism as described in claims 1 to 8 when running the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the open-world knowledge graph completion method based on a multi-attention mechanism as described in claims 1 to 8.

Images