Open world knowledge graph completion method and system based on LSTM and attention mechanism

Abstract

The application provides an open world knowledge graph completion method and system based on LSTM and an attention mechanism, relates to the technical field of knowledge graph completion, and comprises the following steps: given to-be-completed triple data, obtaining new entity names and entity description information in the triple data, and splicing the new entity names and the words in the entity description information into a symbol sequence; vectorizing the symbol sequence through a word embedding model; initializing entities and relations in a knowledge graph through a ComplEx model to form a structure embedding; aggregating the vectorized symbol sequence to obtain a text feature vector, entering an attention layer after averaging to obtain an assigned weight, and finally obtaining a final entity text feature vector through a Bi-LSTM layer; mapping the entity text feature vector into an embedding vector of a graph entity through a mapping function, aligning two vector spaces through an alignment function, and then applying the mapped feature vector to an embedding model to complete the open world knowledge graph completion. The application improves the accuracy of knowledge graph completion and is applied to a common sense knowledge graph.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph completion, in particular to an open world knowledge graph completion method and system based on LSTM and attention mechanism. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The completion of the knowledge graph is to predict the missing part in the triple, so as to make the knowledge graph more complete. Common knowledge graph completion can be divided into three categories. The knowledge graph completion method based on translation model maps entities and relationships into low-dimensional continuous vector space through knowledge representation learning, obtains low-dimensional dense vector representation, and then predicts the relationship between triples by calculating the distance between vectors. Another kind of knowledge graph completion method based on tensor decomposition solves the polysemy of entities and relationships by representing entities or relationships as tensors. In addition, neural networks are also used in the completion task of knowledge graph. Neural networks have stronger feature extraction capability, and some neural networks have the function of aggregating neighborhood information, which can better represent entities and relationships and improve the completion efficiency of knowledge graph.

[0004] However, the above model can only complete the existing entities in the knowledge graph, and cannot link new entities outside the knowledge graph, so these completion methods are classified into closed world knowledge graph completion. However, the knowledge graph in reality is constantly evolving, and large knowledge graphs such as DBpedia rely on unstructured information sources such as Wikipedia, and Wikipedia grows a large amount of new information sources every day. In order to obtain the embedding representation of new entities and relationships outside the knowledge graph and perform the completion task, it is obviously unrealistic to repeatedly train the above model. Therefore, an open world knowledge graph completion model is proposed to adapt to the new challenge of the current rapidly expanding knowledge graph. In the open world scenario, the knowledge graph completion task can obtain knowledge from open world resources.

[0005] For the task of open world knowledge graph completion, there are DKRL model, ConMask model, OWE model and other models. The DKRL model constructs joint learning of structure information and text information by adding entity description information. The ConMask model obtains entity and relationship embedding vectors from entity description through attention mechanism, and scores triples based on matching of pair vectors, but the model relies too much on text information and does not fully utilize structure information. The OWE model combines the conventional completion model learned from the knowledge graph and the word embedding learned from the text corpus. After independent training, the embedding of the entity name and description is mapped to the graph-based embedding space through conversion. The model utilizes the complete knowledge graph and does not depend on long text, and has high scalability. However, the method of processing text data and mapping in OWE is too simple, and each step is not modularized. SUMMARY

[0006] In order to solve the above problems, the application proposes an open world knowledge graph completion method and system based on LSTM and attention mechanism, which improves the OWE model, modularizes it and proposes an enhanced expansion method. LSTM and attention mechanism are introduced when aggregating word vectors, so that word sequence elements can allocate different weights according to the correlation degree with different relationships, improve the aggregation function, introduce the alignment function to align the two vector spaces, and further improve the feature representation ability of external entities, improve the knowledge graph completion accuracy, and apply it to common sense knowledge graph completion.

[0007] According to some embodiments, the application adopts the following technical solutions:

[0008] The open world knowledge graph completion method based on LSTM and attention mechanism comprises:

[0009] Given the triple data to be completed, the new entity name and entity description information in the triple data are obtained, and the new entity name and the words in the entity description information are spliced into a symbol sequence, which is vectorized by a word embedding model;

[0010] The vectorized symbol sequence is aggregated to obtain a text feature vector, which is then averaged and input into an attention layer to obtain the assigned weight, and the final entity text feature vector is obtained through an LSTM layer;

[0011] The text feature vector of the entity is mapped to the embedding vector of the knowledge graph entity by a mapping function, the two vector spaces are aligned by an alignment function, and then the mapped feature vector is applied to an embedding model to complete the open world knowledge graph completion.

[0012] According to some embodiments, the application adopts the following technical solutions:

[0013] An open world knowledge graph completion system based on LSTM and attention mechanism comprises:

[0014] An initialization module is configured to obtain new entity names and entity description information in the triple data and concatenate the new entity names and the words in the entity description information into a symbol sequence, vectorize the symbol sequence through a word embedding model word2vec, and initialize entities and relationships in the knowledge graph through a ComplEx model to form a structure embedding.

[0015] A feature acquisition module is configured to aggregate the vectorized symbol sequence to obtain a text feature vector, average the text feature vector, and then input the average text feature vector into an attention layer to obtain an assigned weight, and obtain a final entity text feature vector through an LSTM layer.

[0016] A knowledge graph completion module is configured to map the text feature vector of the entity into an embedding vector of a graph entity through a mapping function, align the two vector spaces through an alignment function, and then apply the mapped feature vector to an embedding model to complete the open world knowledge graph.

[0017] According to some embodiments, the present application adopts the following technical solutions:

[0018] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions are suitable for being loaded and executed by a processor of a terminal device, and the instructions are based on the open world knowledge graph completion method based on LSTM and attention mechanism.

[0019] According to some embodiments, the present application adopts the following technical solutions:

[0020] A terminal device comprises a processor and a computer readable storage medium, the processor is configured to implement instructions, and the computer readable storage medium is configured to store a plurality of instructions, the instructions are suitable for being loaded and executed by the processor, and the instructions are based on the open world knowledge graph completion method based on LSTM and attention mechanism.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] For the open-world knowledge graph completion task, this invention fully utilizes entity description information and assigns different weights based on different relationships using a Bi-LSTM model and attention mechanism to better extract feature embeddings, laying the groundwork for subsequent embeddings of new entities. A mapping function maps text embeddings to entity embeddings in the knowledge graph, and an alignment function aligns the two vector spaces. The mapped feature vectors are then applied to the embedding model to complete the open-world knowledge graph completion. This invention maximizes the use of entity description information, generating feature representations for new entities from text information. It addresses the problems of current open-world knowledge graph completion methods, such as overly simplistic text data processing and mapping methods, and a lack of modularization. This improves the feature representation capabilities of external entities, enhances the accuracy and efficiency of the knowledge graph completion model, improves the relationships between entities in the commonsense knowledge graph, adds new entities, and applies them to the commonsense knowledge graph. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0025] Figure 2 This is a simplified flowchart of the open-world model network results in an embodiment of the present invention. Detailed implementation method:

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Example 1

[0030] One embodiment of the present invention provides an open-world knowledge graph completion method based on LSTM and attention mechanisms, comprising:

[0031] Step 1: Given the triple data to be completed, initialize the entities and relations in the knowledge graph using the ComplEx model to form a structural embedding space. Vectorize the entity description information using the word embedding model word2vec to form a word vector space. Obtain the new entity name and entity description information from the triple data and concatenate the new entity name with the words in the entity description information to form a symbol sequence. Vectorize it using the word embedding model word2vec.

[0032] Step 2: Aggregate the vectorized symbol sequence to obtain the text feature vector, average it, and then enter the attention layer to obtain the assigned weights. Finally, obtain the entity text feature vector through the LSTM layer.

[0033] Step 3: Map the text feature vector of the entity to the embedding vector of the knowledge graph entity using the mapping function. After aligning the two vector spaces using the alignment function, apply the mapped feature vector to the embedding model to complete the open-world knowledge graph.

[0034] As one embodiment, in step 1, given the triplet data to be completed, the new entity name and entity description information in the triplet data are obtained, and the new entity name and the words in the entity description information are concatenated into a symbol sequence, which is then vectorized through a word embedding model;

[0035] Given a triple (h, r, t), where h is a new entity with associated entity description information, given a relation r, concatenate the entity name corresponding to h with the words in the entity description information to form a symbol sequence. The word embedding model word2vec is used to vectorize the text. The vectorized symbol sequence is fed into an average layer to generate a text feature vector, which is then fed into an attention layer to obtain weights. Finally, a Bi-LSTM layer is used to obtain the final entity text feature vector, which has a dimension of 300.

[0036] First, the entities and relations in the knowledge graph are initialized using the ComplEx model. ComplEx addresses the asymmetric problem by introducing complex values ​​and performs embedding extensions to better model asymmetric relations.

[0037] Word2vec and ComplEx were used to train a word vector model and a knowledge graph embedding model. The dimension of entities and relations was 300, the initial learning rate was 8e-5, and Adam optimization was used. The training set was divided into 100 batches, and the training iterations were 1000 times. The learning rate was updated every 30 iterations, the gamma was 0.95, and the final dropout of the feature vectors was 0.5.

[0038] The loss function is as follows:

[0039]

[0040] Where, when (h,r,t)∈G, l (h,r,t) =1, when (h,r,t)∈G', l (h,r,t) = -1. G represents the set of correct triples in the training set, and G' represents the set of incorrect triples in the training set. This represents the corresponding conjugate complex number.

[0041] In step 2, the average sequence information features of entity description information are captured by the averaging layer. Then, the sequence information features are introduced into the attention layer to calculate the similarity between the feature vector of entity description information and the sequence information features. The correlation between the relation and the words in the entity description is modeled. The word vectors in the entity description are multiplied by the corresponding weight coefficients and then fed into the Bi-LSTM layer to obtain the final entity text feature vector. The dimension of the entity text feature vector is 300.

[0042] Specifically, the entity name of the new entity is concatenated with words from the entity description information to form a symbol sequence. And it was initialized using the word2vec word vector generation tool. v w The data is fed into an averaging layer to obtain an average vector of entity descriptions. First, the averaging layer captures the average sequence information features of the entity descriptions.

[0043] f = AVG(v w )

[0044] Where AVG represents the averaging operation, and f represents the average feature vector of the entity description information.

[0045] Then, it is introduced into the attention layer for computation. Similarity s with f i Model the association between terms in relation and entity descriptions.

[0046] The similarity is calculated as follows:

[0047]

[0048] Attention coefficient α i for:

[0049]

[0050] Multiply the word vectors in the entity description by their corresponding weight coefficients to obtain the entity description information feature vector u. T :

[0051]

[0052] Where, α i It is the attention coefficient. These are word vectors in entity description information.

[0053] will u T = (u1, u2, ..., u i The sequence information features are fed into the Bi-LSTM layer to capture the final entity description information.

[0054] u f =ReLU(Bi-LSTM(u T ))

[0055] ReLU is the activation function.

[0056] In step 3, the text feature vector of the entity is input into the transformation mapping layer through the mapping function and mapped into the embedding vector of the knowledge graph entity. After the two vector spaces are aligned using the alignment function, the mapped feature vector is applied to the embedding model to complete the open world knowledge graph.

[0057] The process of mapping the text feature vector of an entity to the embedding vector of a knowledge graph entity through a mapping function includes: inputting the final entity text feature vector into a transformation mapping layer, mapping the text-based entity embedding to the knowledge graph entity embedding, and performing the mapping through an affine transformation mapping function to map the entity text feature vector to the real number space.

[0058] Specifically, the final entity description features are input into the transformation mapping layer, embedding the text-based entities into u. f Mapped to knowledge graph entity embedding u e , that is ψ r (u f )≈u e Mapping function through affine transformation To perform mapping, where M and b map entity description features to the real number space, M... i b i Projected onto the imaginary space.

[0059] Training the parameters in the mapping function requires pre-training a word vector model (word2vec) and a knowledge graph embedding model to obtain the embedding representations and word vectors of existing entities in the knowledge graph. Then, entities with relevant descriptions are selected from the entity set ε to form a subset ε. t Taking the case where the head entity is the new entity as an example, let all head entities be ε. t The triples of the elements in the middle are extracted as the training set. Finally, training is performed by minimizing the loss:

[0060]

[0061] The initial learning rate was 1e-4, the training batch size was 256, the Adam optimizer was used, gamma was set to 0.5, and a total of 100 iterations were performed.

[0062] Then, the word vectors in the entity description vector space are aligned with the word vectors in the knowledge graph triple vector space using an alignment function:

[0063] The expression for the alignment function is as follows:

[0064]

[0065] Furthermore, the embedding representation generated through joint training in both spaces will be more explicit. The total loss function is trained using batch stochastic gradient descent, and the total loss function is as follows:

[0066] L ALL =(L+λL) reg )+(αL proj +βL X )

[0067] Among them, L reg Let λ represent the structured embedding second-order normal form, λ represent the regularization parameter, and α and β represent the hyperparameters.

[0068] The mapped feature vectors are then applied to the scoring function in the embedded model:

[0069]

[0070] The commonsense knowledge graph was expanded. Adam optimization was used with a learning rate of 0.00005. β was always set to 1, and α to 0.001.

[0071] Example 2

[0072] One embodiment of the present invention provides an open-world knowledge graph completion system based on LSTM and attention mechanisms, comprising:

[0073] The initialization module is used to obtain the new entity name and entity description information from the triple data to be completed, and concatenate the new entity name and the words in the entity description information into a symbol sequence. It then vectorizes the symbol sequence using a word embedding model and initializes the entities and relations in the knowledge graph using the ComplEx model to form a structured embedding.

[0074] The feature acquisition module is used to aggregate the vectorized symbol sequence to obtain the text feature vector, average it, and then enter the attention layer to obtain the assigned weights. Finally, the final entity text feature vector is obtained through the LSTM layer.

[0075] The knowledge graph completion module is used to map the text feature vectors of entities to the embedding vectors of knowledge graph entities through a mapping function. After aligning the two vector spaces through an alignment function, the mapped feature vectors are applied to the embedding model to complete the open-world knowledge graph.

[0076] By improving the OWE model, modularizing and extending it into a framework capable of generating embeddings for new entities, this approach fully utilizes entity description information. By introducing LSTM and attention mechanisms, it more accurately represents entity text information, addressing the current limitations of open-world knowledge graphs in their simplistic use of entity description information and lack of modularity. Furthermore, an alignment function aligns the entity description information and knowledge graph embedding spaces, applying this to open-world knowledge graph completion tasks. After multiple training iterations, more accurate entity and relation embeddings are generated, improving the accuracy of knowledge graph completion.

[0077] Example 3

[0078] One embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the steps of the open-world knowledge graph completion method based on LSTM and attention mechanism.

[0079] Example 4

[0080] One embodiment of the present invention provides a terminal device, including a processor and a computer-readable storage medium. The processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, which are adapted to be loaded by the processor and executed by the processor to perform the steps of the open-world knowledge graph completion method based on LSTM and attention mechanism.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An open world knowledge graph completion method based on LSTM and attention mechanism, characterized in that, Comprise: Given the triple data to be completed, obtain the new entity name and entity description information in the triple data, and concatenate the new entity name and the words in the entity description information into a symbol sequence, and vectorize it through a word embedding model; Aggregate the vectorized symbol sequence to obtain a text feature vector, average it and then enter the attention layer to obtain the assigned weight, and obtain the final entity text feature vector through the LSTM layer, specifically: The sequence information feature of the entity description information is captured by an average layer, the sequence information feature is introduced into an attention layer after averaging, the similarity between the feature vector of the entity description information and the sequence information feature is calculated, the association between the relationship and the word in the entity description is modeled, the word vector in the entity description is multiplied by the corresponding weight coefficient, and the final entity text feature vector is obtained through Bi-LSTM an average layer. Map the entity text feature vector to the embedding vector of the knowledge graph entity through a mapping function, align the two vector spaces after using an alignment function, and apply the transformed feature vector to the embedding model to complete the completion of the open world knowledge graph. The mapping function is by a mapping function of an affine transformation Transforming, wherein M , b Mapping the entity description features to a real number space, M i , b i Projecting to an imaginary number space. 2.The LSTM and attention mechanism based open world knowledge graph completion method of claim 1, wherein, After obtaining the given triple data to be completed, the ComplEx model is used to initialize the entities and relationships in the knowledge graph. ComplEx solves the non-symmetry problem by introducing complex values and extends the embedding to model non-symmetric relationships. 3.The LSTM and attention mechanism based open world knowledge graph completion method of claim 1, wherein, The entity name of the new entity is concatenated with the words in the entity description information to form a symbol sequence, which is vectorized through a word2vec word vector embedding model. 4.The LSTM and attention mechanism based open world knowledge graph completion method of claim 1, wherein, Train the parameters in the mapping function, pretrain the word vector model and the knowledge graph embedding model, and obtain the embedding representation of the existing entities in the knowledge graph and the word vector. 5.The LSTM and attention mechanism based open world knowledge graph completion method of claim 1, wherein, Select entities with related descriptions from the entity set to form a subset, train by minimizing the loss, align the word vectors in the entity description vector space with the word vectors in the knowledge graph triple vector space through an alignment function, and generate embedding expressions through joint training of the two spaces. Apply the transformed feature vector to the embedding model to complete the completion of the open world knowledge graph.

6. An open world knowledge graph completion system based on LSTM and attention mechanism, characterized in that, Comprise: The initialization module is configured to, given the triple data to be completed, obtain the new entity name and entity description information in the triple data, and concatenate the new entity name and the words in the entity description information into a symbol sequence, and vectorize it through a word embedding model; The feature acquisition module is configured to aggregate the vectorized symbol sequence to obtain a text feature vector, average the text feature vector, and then input the text feature vector into an attention layer to obtain an assigned weight. Bi-LSTM The layer obtains a final entity text feature vector, and the final entity text feature vector is obtained in the following specific manner: The sequence information feature of the entity description information is captured by an average layer, the sequence information feature is introduced into an attention layer after averaging, the similarity between the feature vector of the entity description information and the sequence information feature is calculated, the association between the relationship and the words in the entity description is modeled, the word vector in the entity description is multiplied by the corresponding weight coefficient, and the final entity text feature vector is obtained through Bi-LSTM an average layer. The knowledge graph completion module is configured to map the entity text feature vector to the embedding vector of the graph entity through a mapping function, align the two vector spaces after using an alignment function, and apply the transformed feature vector to the embedding model to complete the completion of the open world knowledge graph. The mapping function is by a mapping function of an affine transformation transforming, wherein M , b mapping the entity description features to a real number space, M i , b i projecting to an imaginary number space.

7. A computer readable storage medium characterized in that, A plurality of instructions are stored therein, and the instructions are adapted to be loaded and executed by the processor of the terminal device to implement the open world knowledge graph completion method based on LSTM and attention mechanism according to any one of claims 1-5.

8. A terminal device, comprising: Comprise a processor and a computer readable storage medium, the processor is used to realize each instruction; the computer readable storage medium is used to store a plurality of instructions, the instructions are adapted to be loaded and executed by the processor to implement the open world knowledge graph completion method based on LSTM and attention mechanism according to any one of claims 1-5.

Images