A knowledge representation learning method and system

Abstract

The application provides a knowledge representation learning method and system, and relates to the field of representation learning, wherein a constructed knowledge representation model is trained by using a knowledge base to be modeled, knowledge representation of triples in the knowledge base is learned, and modeling of the knowledge base is completed; the knowledge representation model comprises an information extraction module and an encoder; the information extraction module is used for extracting relation path information representation, text description information representation and entity hierarchical type information representation of the triples; the encoder takes the relation path information representation, the text description information representation, the entity hierarchical type information representation and structured information representation of the triples as input, and outputs vector representation of entities and relations of the triples; by fusing the relation path, the text description information and the hierarchical type information of the entity and considering other effective information in the triples, the application effectively solves the problem of incomplete semantic representation of the entities and the relations, improves the effect of knowledge representation learning, and efficiently realizes knowledge graph link prediction and triple classification tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of representation learning, and particularly relates to a knowledge representation learning method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the advent of the Web3.0 era, the knowledge base containing a large amount of structured knowledge has become an important part of many semantic applications (such as intelligent question answering and search services), and more and more enterprises and organizations are committed to constructing large knowledge bases; the structured knowledge in the knowledge base is represented in the form of triples (head entity, relation, tail entity); although the existing knowledge base already contains a large number of triples, due to the massive and constantly changing information, there are still missing entities and relations in the knowledge base; how to complete the missing relations between entities in the knowledge base is a key problem in the construction of the knowledge base.

[0004] The knowledge base completion task needs to model the entities and relations in the knowledge base; symbolic representation is a kind of knowledge base modeling method, which uses a graph-based method to model the knowledge base, but with the increase of knowledge base information, this method is low in efficiency and has a data sparsity problem. Knowledge representation is also a kind of knowledge base modeling method, which maps entities and relations to a low-dimensional dense vector space, and models the knowledge base through vector operations, which well overcomes the disadvantages of symbolic representation.

[0005] In the existing knowledge representation learning method, a translation model is usually used to model the entities and relations in the knowledge base, wherein the TransE model is a very representative method, which regards the relation as a translation vector between the head entity and the tail entity, but the TransE model is too simple in structure, and has limitations in handling complex relations such as one-to-many, many-to-one and many-to-many, and cannot effectively distinguish different entities with the same relation; the TransH model assumes that each relation has a hyperplane, and projects the entity onto the hyperplane to realize different representations of the entity in different relations; the TransR model proposes to construct entity and relation embeddings in separate entity space and relation space, project the entity from the entity space to the corresponding relation space, and then establish translation between the projected entities to learn the embedding, which focuses on the semantic difference of different attributes of the entity and the relation, so that the representation ability of the model is stronger, but the translation model only focuses on the information in the triple, only uses the semantic information of the structure of the knowledge graph itself, and is seriously affected by the data sparsity, and the computational complexity also becomes higher.

[0006] In knowledge graphs, besides the original structured triples, there is a lot of related multi-source heterogeneous information that cannot be fully utilized. This information can help fill in the missing information in the knowledge graph, so some knowledge representation learning models that integrate additional information have emerged, improving the representation effect of traditional translation models. The DKRL model, based on the TransE model, integrates entity description information and uses continuous bag-of-words encoding and convolutional neural network models to encode the entity description text. The PTransE model is a path-based model that uses a path constraint resource algorithm to measure the confidence of relation paths and combines the obtained relation paths for representation. The TKRL model integrates hierarchical type information. For each relation, it limits the domain information and type information of the corresponding head and tail entities, thus forming the auxiliary information of the entities in the triples. The IKRL model integrates entity image information into knowledge representation learning, learning the representation of knowledge by combining triple information and image information.

[0007] Existing knowledge representation learning methods that integrate additional information have relatively simple integration methods and a low degree of integration of additional information in the knowledge graph. They do not make full use of the additional information in the knowledge graph, resulting in poor knowledge representation learning performance. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention provides a knowledge representation learning method and system. By integrating relational paths, textual description information, and hierarchical type information of entities, and considering other effective information in triples, it effectively solves the problem of incomplete semantic representation of entities and relations, improves the effect of knowledge representation learning, and efficiently realizes knowledge graph link prediction and triple classification tasks.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] The first aspect of this invention provides a knowledge representation learning method;

[0011] A knowledge representation learning method, comprising:

[0012] The knowledge representation model is trained using the knowledge base to be modeled, learning the knowledge representation of triples in the knowledge base, and thus completing the modeling of the knowledge base.

[0013] The knowledge representation model includes an information extraction module and an encoder;

[0014] The information extraction module is used to extract the relation path information representation, text description information representation, and entity hierarchy type information representation of triples; wherein, a weighted hierarchical encoder in TKRL is used to construct a projection matrix to obtain the entity hierarchy type information representation;

[0015] The encoder takes as input the relation path information representation of triples, the text description information representation, the entity hierarchy type information representation, and the structured information representation of triples, and outputs vector representations of entities and relations of triples.

[0016] Furthermore, the relationship path information represents the relationship between entity pairs, and the specific formula is as follows:

[0017]

[0018]

[0019] Where p∈P(h,t) represents the possible relational paths between traversing entity pairs, R(p|h,t) represents the reliability of a relational path p between a given entity pair (h,t), and Z is the normalization factor.

[0020] Furthermore, the extraction steps for the text description information representation are as follows:

[0021] Textual description information is transformed into vectors of various forms through word embedding, segmentation embedding, and position embedding.

[0022] After concatenating vectors of various forms, the data is input into the BERT model to extract textual descriptive information features and generate sentence vectors.

[0023] The sentence vectors are averaged to obtain the text description information representation.

[0024] Furthermore, the constructed projection matrix is ​​as follows:

[0025]

[0026] Where n represents the number of levels in the hierarchy type z, z (i) It is the i-th subtype of z. Indicate z (i) The projection matrix, α i express The weight.

[0027] Furthermore, the acquisition of the entity hierarchy type information representation specifically includes:

[0028] After obtaining the projection matrix, the head entity and tail entity are projected into their respective specific spaces using the TransR model to obtain entity hierarchy type information representations.

[0029] Furthermore, the weighted encoding employs an encoder constructed based on a graph attention network to calculate the influence weights of different adjacency information in the structured information on the representation of relational path information, textual description information, and entity hierarchical type information. Based on these influence weights, the structured information representation, relational path information representation, textual description information representation, and entity hierarchical type information representation are weighted and encoded to obtain vector representations of entities and relations.

[0030] Furthermore, the encoder is trained using the maximum margin method, and the loss function is defined as:

[0031]

[0032] Where γ>0 is a preset interval parameter, and T` is the negative sample corresponding to the triple in the knowledge base after replacing the head or tail entity with an entity from the entity set.

[0033] A second aspect of the present invention provides a knowledge representation learning system.

[0034] A knowledge representation learning system trains a constructed knowledge representation model using a knowledge base to be modeled, learns the knowledge representation of triples in the knowledge base, and completes the modeling of the knowledge base. The system includes an information extraction module and an encoder.

[0035] The information extraction module is used to extract the relation path information representation, text description information representation, and entity hierarchy type information representation of triples; wherein, a weighted hierarchical encoder in TKRL is used to construct a projection matrix to obtain the entity hierarchy type information representation;

[0036] The encoder takes as input the relation path information representation of triples, the text description information representation, the entity hierarchy type information representation, and the structured information representation of triples, and outputs vector representations of entities and relations of triples.

[0037] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a knowledge representation learning method as described in the first aspect of the present invention.

[0038] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a knowledge representation learning method as described in the first aspect of the present invention.

[0039] The above one or more technical solutions have the following beneficial effects:

[0040] This invention effectively solves the problem of incomplete semantic representation of entities and relations by integrating relational paths, textual description information, and hierarchical type information of entities, and taking into account other effective information in triples. The PDMRL model has made great progress in various indicators in both link prediction and triple classification tasks, demonstrating its superiority.

[0041] Following the principle that understanding the world is typically hierarchical, this invention integrates entity hierarchical type information to represent entity hierarchical structures with different granularities. This helps people automatically discover the relationships between different entities, and the hierarchical type information of entities can also be used to pre-estimate the features of entities. The hierarchical type information possessed by entities can provide systematic type knowledge based on the triple information of knowledge graphs, which helps to build more efficient representation models.

[0042] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0044] Figure 1 This is a flowchart of the method in the first embodiment.

[0045] Figure 2 This is a schematic diagram of the relationship path in the first embodiment.

[0046] Figure 3 This is a flowchart illustrating the extraction process of text description information in the first embodiment.

[0047] Figure 4 , Figure 5 The results are experimental results for link prediction in the first embodiment.

[0048] Figure 6 This is a comparison chart of the classification accuracy of triples in the first embodiment. Detailed Implementation

[0049] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0050] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0051] Example 1

[0052] In one or more embodiments, a knowledge representation learning method, PDMRL, is disclosed. This method integrates structured information, relational path information, textual description information, and entity hierarchy type information to perform knowledge representation learning on a knowledge base, establishing a knowledge representation model, PDMRL. The PDMRL model accurately represents the entities and relations in triples, facilitating efficient knowledge graph link prediction and triple classification tasks. The accuracy of the knowledge representation depends on the knowledge representation capability of the PDMRL model, which is trained through knowledge representation learning. In this embodiment, the knowledge base to be modeled is used to train the constructed knowledge representation model, PDMRL, to learn the knowledge representation of triples in the knowledge base, thus completing the modeling of the knowledge base. The knowledge representation model, PDMRL, includes an information extraction module and an encoder. Figure 1 This is a flowchart of knowledge representation learning methods, such as... Figure 1 As shown, it includes the following steps:

[0053] Step S1: Extract information from the knowledge base to be modeled, including the relational path information, text description information, and entity hierarchy type information for each triple in the knowledge base.

[0054] Specifically, information extraction transforms relational path information, textual description information, and entity hierarchical type information into vector representations. For relational path information representation, the Path Ranking Algorithm (PRA) is used to obtain the relational path information between entity pairs in the knowledge graph. For textual description information representation, the BERT model is used to represent the textual description information as vectors. When representing entity hierarchical type information, a hierarchical type matrix is ​​constructed.

[0055] Relationship path information representation

[0056] In a knowledge graph, relational paths can provide relationships between seemingly unrelated entity pairs. Path ranking algorithms are used to obtain the relational path information between entity pairs in a knowledge graph. Assume there are multiple relational paths P = {p1, p2, ..., p...} between entity pairs (h, t). N One of the relational paths can be represented as p = (r1, e1, ... e1). l-1 ,r l),like Figure 2 As shown, this represents a relation path with l relations and l-1 entities. Then, the relation path is represented as a vector: the vector representations of all the relation paths are added together using addition to obtain the vector representation p of the entire relation path. * :

[0057] p * =r1+e1+...+e l-1 +r l (1)

[0058] For a single path, its energy function can be defined by referring to the energy function of the TransE model, that is, E(h,p,t)=||h+pt||.

[0059] For the energy function of multiple relation paths, referring to existing path-constrained resource allocation algorithms for fact triples (h,r,t), the comprehensive energy of multiple relation paths is defined as:

[0060]

[0061]

[0062] Where p∈P(h,t) represents the possible relational paths between traversed entity pairs, R(p|h,t) represents the reliability of a relational path p between a given entity pair (h,t), and Z is a normalization factor; in this embodiment, only the relational path information in steps 2 and 3 is considered to avoid computational complexity.

[0063] Text description information representation

[0064] Existing translation models like DKRL extract keywords from descriptive information and then use CBOW and CNN to encode and represent them. However, these representations do not contain all the semantic information of entity descriptions, which will cause some semantic loss. Therefore, this embodiment uses the BERT model to represent text description information as vectors. The BERT model is a multi-layer Transformer model structure that can better capture bidirectional relationships and longer-distance dependencies in sentences.

[0065] Figure 3 This is a flowchart of the extraction process for text description information, such as... Figure 3 As shown, the specific steps are as follows:

[0066] Textual description information is transformed into vectors of various forms through word embedding, segmentation embedding, and position embedding.

[0067] After concatenating vectors of various forms, the data is input into the BERT model, which uses a multi-layer Transformer structure to transform these vectors, thereby extracting textual descriptive information features and generating sentence vectors.

[0068] Averaging these sentence vectors yields the text description information representation vector, expressed by the formula:

[0069]

[0070] in, S represents the vector representing the text description information. i (i = 1, 2, ..., n) represents the sentence vector of the i-th sentence, and n represents the number of sentence vectors.

[0071] Combining textual description information with existing triplet information allows for better learning of vector representations between entities and relations. Therefore, the energy function for textual description information representation is defined as follows:

[0072]

[0073] in, and It is a representation of structured information; and Represents textual description information. E dd The energy function indicates that both the head and tail entities are represented based on textual description information; E ds The head entity represents a text-based representation, and the tail entity represents an energy function based on a structural representation; E sd The head entity represents an energy function based on structural representation, while the tail entity represents an energy function based on textual description.

[0074] Representation of entity hierarchy type information

[0075] Entity hierarchical type information refers to the hierarchical structure of entities with different granularities. This information can help people automatically discover relationships between different entities and can also be used to pre-estimate the features of entities. The meaning of the same entity can vary greatly in different contexts, so each entity typically contains multiple hierarchical types. Different granularities of semantic concepts are considered subtypes within different hierarchical types; that is, each hierarchical type has many layers. For a hierarchical type z with n layers, z... (i) It is the i-th subtype of z.

[0076] The method for extracting entity hierarchy type information is as follows:

[0077] (1) Using the weighted hierarchical encoder in TKRL, construct the projection matrix M. z :

[0078]

[0079] Where n represents the number of levels in the hierarchy type z, z (i) It is the i-th subtype of z. Indicate z (i) The projection matrix, α i express The weights of each layer are calculated using the following formula:

[0080] α i :α i+1 =(1-η):η

[0081]

[0082] Where η∈(0,0.5) is the weight coefficient. For an entity, there are multiple hierarchical types, and the hierarchical type with more specific description has a higher weight.

[0083] (2) Obtain the entity hierarchy type information representation

[0084] After obtaining the projection matrix, the head and tail entities are projected into their respective specific spaces using the TransR model. For two types of entity information with the same relationship, the same matrix is ​​used for calculation to obtain the entity hierarchy type information representation, expressed by the energy function:

[0085] E z =||M z h+rM z t|| (8)

[0086] Among them, E z M is the energy function representing entity hierarchy type information. z It is a projection matrix, where h and t represent entity vectors and r represents relation vectors.

[0087] Step S2: Train the constructed knowledge representation encoder using the knowledge base after information extraction to complete the modeling of the knowledge base; wherein, the knowledge representation encoder takes the relation path information representation of triples, text description information representation, entity hierarchy type information representation and structured information representation as input, and outputs vector representations of entities and relations of triples.

[0088] Specifically, path information, text description information, hierarchical type information, and the original triplet structure information are combined with the relation vector and input into an encoder with a graph attention network as the main structure. The graph attention mechanism is used to capture the features of the entity's neighbor nodes, find the mutual influence between each entity and its neighbor nodes, and use it as the influence weight. Based on the influence weight, the structured information representation, relation path information representation, text description information representation, and entity hierarchical type information representation are weighted and encoded to obtain the vector representation of the entity and the relation, expressed by the formula:

[0089] E = λ s E s +λ p E p +λ d E d +λ z E z (9)

[0090] Where E is the energy function of the vector representation of entities and relations, E s It is an energy function based on structured information representation, E s =||h+rt||, E p It is the energy function based on relational path information representation, E d E is an energy function based on textual description information. z It is the energy function based on entity hierarchy type information representation, λ s , λ p , λ d and λ z Corresponding to E respectively s E p E d E z The influence weight.

[0091] like Figure 1 As shown in the encoder section, the encoder mainly consists of a graph attention mechanism. Structural information representation, relational path information representation, textual description information representation, and entity hierarchical type information representation are combined with relation vectors and input into the graph attention network. Graph attention is used to capture the critical point features of entities and find the mutual influence between each entity and its neighboring nodes.

[0092] During training, the encoder can be viewed as being based on the TransE model, except that the vector representations of the head and tail entities and relations have undergone complex transformations. The scoring function of the encoder model is:

[0093]

[0094] Where h,t represents the entity vector that integrates the four types of information, and r represents the relation vector; h,t,r are calculated using L1 or L2 norm.

[0095] During training, the maximum margin method is used, and the loss function is defined as:

[0096]

[0097] Where γ>0 is a preset interval parameter, and T` is the negative sample corresponding to the triple in the knowledge base after replacing the head or tail entity with an entity from the entity set.

[0098] After training, the knowledge representation model PDMRL, as the result of knowledge base learning, is used to complete the missing information in the input triples. For example, it is used in the link prediction task and triple classification task of knowledge graph. The link prediction task predicts the missing entity when only an entity and relation are given in the triple, while the triple classification task is used to verify the model's ability to identify positive triples.

[0099] The PDMRL model was evaluated using link prediction and triplet classification tasks on the benchmarks WN18, FB15K, and FB15K-237, and the gap between the PDMRL model and the current state-of-the-art models was compared. Figure 4 , 5 6 are the experimental results. Figure 4 and Figure 5 These are the experimental results of the PDMRL model and the baseline model for link prediction on the WN18 and FB15K datasets. Figure 6 It represents the triplet classification accuracy of the PDMRL model compared to the baseline model.

[0100] The experimental data above demonstrates that the PDMRL model is highly competitive compared to existing models and methods. By integrating relational paths, textual descriptions, and hierarchical type information of entities, and considering other effective information in triples, it effectively solves the problem of incomplete semantic representation of entities and relations. Therefore, the PDMRL model has made significant progress in all metrics of both link prediction and triple classification tasks, demonstrating its superiority.

[0101] Example 2

[0102] In one or more embodiments, a knowledge representation learning system is disclosed, which trains a constructed knowledge representation model using a knowledge base to be modeled, learns the knowledge representation of triples in the knowledge base, and completes the modeling of the knowledge base, including an information extraction module and an encoder:

[0103] The information extraction module is used to extract the relation path information representation, text description information representation, and entity hierarchy type information representation of triples; wherein, a weighted hierarchical encoder in TKRL is used to construct a projection matrix to obtain the entity hierarchy type information representation;

[0104] The encoder takes as input the relation path information representation of triples, the text description information representation, the entity hierarchy type information representation, and the structured information representation of triples, and outputs vector representations of entities and relations of triples.

[0105] Example 3

[0106] The purpose of this embodiment is to provide a computer-readable storage medium.

[0107] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a knowledge representation learning method as described in Embodiment 1 of this disclosure.

[0108] Example 4

[0109] The purpose of this embodiment is to provide an electronic device.

[0110] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a knowledge representation learning method as described in Embodiment 1 of this disclosure.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge representation learning method, characterized in that, The knowledge representation model is trained using the knowledge base to be modeled, learning the knowledge representation of triples in the knowledge base, and thus completing the modeling of the knowledge base. The knowledge representation model includes an information extraction module and an encoder; The information extraction module is used to extract the relational path information representation, text description information representation, and entity hierarchy type information representation of the triples; The relationship path information represents the relationship between entity pairs, and the specific formula is as follows: in, This indicates the possible relationship paths between entity pairs. Represents a given pair of entities There is a relationship path between them. Reliability, It is a normalization factor; The method for extracting the entity hierarchy type information is as follows: The projection matrix is ​​constructed using the weighted hierarchical encoder in TKRL. : Where n represents the number of levels in the hierarchy type z. It is the i-th subtype of z. express The projection matrix, express The weights; After obtaining the projection matrix, the head and tail entities are projected into their respective specific spaces using the TransR model. For two types of entity information with the same relationship, the same matrix is ​​used for calculation to obtain the entity hierarchy type information representation, expressed by the energy function: in, It is the energy function representing entity hierarchy type information. It is a projection matrix, where h and t represent entity vectors and r represents relation vectors; The encoder takes as input the relation path information representation, text description information representation, entity hierarchy type information representation, and structured information representation of the triples, and outputs vector representations of entities and relations of the triples. The path information, text description information, hierarchy type information, and the original triple structure information are combined with the relation vectors and input into an encoder with a graph attention network as its main structure. The graph attention mechanism captures the features of entity neighbor nodes, identifies the mutual influence between each entity and its neighbor nodes as influence weights, and based on these influence weights, the structured information representation, relation path information representation, text description information representation, and entity hierarchy type information representation are weighted and encoded to obtain the vector representations of entities and relations, expressed by the formula: in, E It is the energy function representing the vector representation of entities and relations. E s It is an energy function based on structured information representation. , E p It is an energy function based on relational path information representation. E d It is an energy function based on textual description information. E z It is an energy function based on entity hierarchy type information representation. Corresponding to E s、 E p、 E d、 E z The influence weight.

2. The knowledge representation learning method as described in claim 1, characterized in that, The steps for extracting the text description information are as follows: Textual description information is transformed into vectors of various forms through word embedding, segmentation embedding, and position embedding. After concatenating vectors of various forms, the data is input into the BERT model to extract textual descriptive information features and generate sentence vectors. The sentence vectors are averaged to obtain the text description information representation.

3. The knowledge representation learning method as described in claim 1, characterized in that, The encoder is trained using the maximum margin method, and the loss function is defined as: in, The preset interval parameter, It is the negative sample corresponding to the triple in the knowledge base after replacing the first or last entity with an entity from the entity set.

4. A knowledge representation learning system, characterized in that, The method described in any one of claims 1-3 includes an information extraction module and an encoder: The information extraction module is used to extract the relation path information representation, text description information representation, and entity hierarchy type information representation of triples; wherein, a weighted hierarchical encoder in TKRL is used to construct a projection matrix to obtain the entity hierarchy type information representation; The encoder takes as input the relation path information representation of triples, the text description information representation, the entity hierarchy type information representation, and the structured information representation of triples, and outputs vector representations of entities and relations of triples.

5. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-3.

6. A storage medium characterized in that it non-transitory stores computer-readable instructions, wherein, When the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-3 are executed.

Images