A Temporal Knowledge Graph Reasoning Method Based on Hyperrelation Subgraphs

Abstract

This invention discloses a temporal knowledge graph reasoning method based on super-relation subgraphs. First, a temporal knowledge graph reasoning model based on super-relation subgraphs is constructed and trained. Then, the trained model is used to realize temporal knowledge graph reasoning. This temporal knowledge reasoning model, which uses a two-layer graph attention network, models the constraints of the relative position of relations and solves the "information silo" problem in relation evolution, thereby improving the accuracy of future time entity prediction and relation prediction tasks.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and more specifically, relates to a temporal knowledge graph reasoning method based on super relational subgraphs. Background Technology

[0002] Against the backdrop of the accelerated development of artificial intelligence, knowledge graphs, as a key technology for structured knowledge representation, have played a central role in scenarios such as semantic understanding and intelligent interaction. Although knowledge graphs have made significant progress in scale in recent years, typically encompassing tens of millions of entities and hundreds of millions of triples, much implicit knowledge remains difficult to directly represent. This limitation leads to two major problems prevalent in knowledge graphs: sparsity and incompleteness. Taking the Freebase knowledge base as an example, although it contains over 3 million person entities, approximately 75% lack nationality information, and 71% lack the crucial detail of birthplace. These sparsity and incompleteness issues severely restrict the effectiveness of knowledge graphs in practical applications. To address this challenge, Knowledge Graph Completion (KGC) technology has been proposed and developed. Knowledge graph completion uses known explicit knowledge to infer implicit knowledge not yet explicitly recorded in the graph, thereby compensating for the incompleteness of the knowledge graph.

[0003] Traditional knowledge graphs describe static knowledge in the form of triples, that is, entity relationships that do not change over time. They are typically used to describe static graphs where facts do not change over time. However, in the real world, a large amount of knowledge has temporal characteristics; for example, the occurrence of events and changes in entity attributes are closely related to time.

[0004] To more accurately describe and process this dynamically changing knowledge, temporal knowledge graphs have been proposed. To capture dynamically evolving knowledge, researchers innovatively embed time constraints into triple structures, forming quadruples (head entity, relation, tail entity, timestamp) or (entity, attribute, attribute value, timestamp), thus supporting temporal reasoning tasks. Similar to traditional static knowledge graphs, temporal knowledge graphs also suffer from incompleteness and missing information. Furthermore, temporal knowledge graphs involve predicting quadruples at future timestamps. Compared to static knowledge graph reasoning tasks, temporal knowledge graph reasoning tasks require additional consideration of temporal information and its integration into the knowledge representation process, making temporal knowledge graph reasoning more challenging. While static knowledge graph reasoning has relatively mature research, and temporal knowledge graphs also have broad application prospects, related research is relatively scarce, warranting further in-depth study.

[0005] Currently, temporal knowledge graph reasoning tasks can generally be categorized into three methods: translation-based, tensor decomposition-based, and neural network-based. With the development of deep learning, neural network-based methods are increasingly becoming the mainstream approach.

[0006] Know-Evolve, first proposed in 2017, combines deep learning methods to learn temporally nonlinear evolutionary entity representations. It models the occurrence of a fact as a multivariate point process, defines a bilinear relation score that captures the multi-relational interactions between entities, adjusts the intensity function of this point process, and dynamically updates the entity representation using the intensity function under different facts. This model was the first to predict both the occurrence of a fact and the timing of its occurrence.

[0007] DyRep proposed a dual-timescale deep temporal process model to model the global and local evolution of dynamic graph networks, improving the ability to capture the interleaved dynamics of observation processes. The RE-GCN model effectively models the correlation features between synchronous facts by stacking multiple graph convolutional network layers and introduces a gated recurrent structure to efficiently extract long-term temporal features from historical facts through a temporal information transfer mechanism. It also utilizes residual connection design to alleviate the gradient decay problem in deep networks. Gao et al. proposed SToKE based on the pre-trained language model BERT, constructing a hierarchical event evolution tree to transform entity relationships and their temporal evolution paths in the knowledge graph into sequential event representations, thereby fusing structural correlations and temporal dynamic features to achieve joint modeling of structural and temporal features.

[0008] While the aforementioned models have greatly improved the accuracy of temporal knowledge graph reasoning tasks by combining deep learning technology, they still suffer from insufficient modeling of relative positional constraints of relationships and the existence of "information silos" in relationship evolution.

[0009] The insufficient modeling of relative position constraints in relationships refers to the fact that, in knowledge graphs, two adjacent relationships have four different relative positions relative to the central shared entity based on the direction of the head entity, tail entity, and the orientation of the shared entity. Previous models have rarely modeled the relative positions of these relationships. The "information silo" problem in relationship evolution refers to the fact that the evolutionary information of relationships during dynamic processes is not transmitted across time, forming "information silos." Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a temporal knowledge graph reasoning method based on super relational subgraphs. The method models the constraints of the relative positions of relations through a two-layer graph attention network temporal knowledge reasoning model and solves the "information silo" problem in relation evolution, thereby improving the accuracy of future time entity prediction and relation prediction tasks.

[0011] To achieve the above-mentioned objectives, the present invention provides a temporal knowledge graph reasoning method based on super-relational subgraphs, characterized by comprising the following steps;

[0012] (1) Download and initialize the dynamic knowledge graph dataset;

[0013] (2) Construct a super-relationship subgraph;

[0014] (3) Construct a temporal knowledge graph reasoning model based on super-relational subgraphs;

[0015] (4) Train a temporal knowledge graph reasoning model based on super-relational subgraphs;

[0016] (5) Temporal knowledge graph reasoning based on super-relational subgraphs;

[0017] The objective of this invention is achieved as follows:

[0018] This invention presents a temporal knowledge graph reasoning method based on super-relational subgraphs. First, a temporal knowledge graph reasoning model based on super-relational subgraphs is constructed and trained. Then, the trained model is used to perform temporal knowledge graph reasoning. This method models the constraints of the relative positions of relations through a two-layer graph attention network temporal knowledge reasoning model and solves the "information silo" problem in relation evolution, thereby improving the accuracy of future time entity prediction and relation prediction tasks.

[0019] Meanwhile, the temporal knowledge graph reasoning method based on super-relational subgraphs of this invention also has the following beneficial effects:

[0020] (1) By adding the definition and training of the super relation subgraph, the relative position constraints between relations are effectively learned, and information capture based on graph attention mechanism is realized at the level of historical subgraph and super relation subgraph.

[0021] (2) By adding an LSTM-based relation evolution update mechanism, the LSTM layer that captures relation evolution information is designed into the relation update mechanism, which successfully alleviates the temporal redundancy problem of the GRU-LSTM cascade structure and alleviates the "information silo" problem of relation evolution, resulting in a significant improvement in the MRR index and Hits@k index of the knowledge reasoning task compared with baseline models such as RE-GCN and Trend. Attached Figure Description

[0022] Figure 1 This is a flowchart of the temporal knowledge graph reasoning method based on super-relational subgraphs of the present invention;

[0023] Figure 2 This is a schematic diagram of a knowledge graph mapped into a super-relationship subgraph;

[0024] Figure 3It is a temporal knowledge graph reasoning model based on super-relational subgraphs;

[0025] Figure 4 This is a structure diagram of the attention module at the historical subgraph level;

[0026] Figure 5 This is a structural diagram of the relationship update mechanism module;

[0027] Figure 6 It is a diagram of the attention module structure at the super relational subgraph level. Detailed Implementation

[0028] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0029] Example

[0030] In this embodiment, as Figure 1 As shown, the present invention provides a temporal knowledge graph reasoning method based on a super-relational subgraph, characterized by comprising the following steps;

[0031] (1) Download and initialize the dynamic knowledge graph dataset;

[0032] Thus, the dynamic knowledge graph dataset can be downloaded from the ICEWS18 dataset. , Represents the number of discrete-time samples. represent A knowledge graph of moments;

[0033] The knowledge graph at each time step consists of several triples, and each triple is represented as (head entity) ,relation Tail entity ), where the head entity ,relation Tail entity All data is text, for example: triples (Mr. Li, award, Nobel Peace Prize);

[0034] Knowledge graph for each moment Word embedding processing based on word2vec, thereby... Each entity and relation in the vector is mapped to a multi-dimensional dense vector, where the head entity and tail entity are denoted uniformly as . The relationship is denoted as , For the number of entities, Set the entity vector dimension to 200. For the number of relations, Set the dimension of the relation vector to 200;

[0035] (2) Construct a super-relationship subgraph;

[0036] Traversal The relation of all triples, such as Figure 2 As shown, each relation is mapped to a twin super-relation subgraph. One of the nodes, and define four types of super relations. As a super-relational subgraph The edge;

[0037] According to the above definition, The super-relation subgraph at time step is also composed of several triples, each triple being represented as (head relation) Super-relationship Tail Relationship ), the four types of super relations are ,in, Represents two relationships and The intermediate entity serves as the head entity in both triples corresponding to the two relations; Represents two relationships and The intermediate entity is the tail entity in both triples; Representing the intermediate entity in the head relationship The corresponding triple contains the head entity, and the tail relation contains the tail entity. The corresponding triple contains the tail entity; Then the intermediate entity is in the head relation The triple is the tail entity, in the tail relation The triplet contains the head entity;

[0038] In this embodiment, taking the triple (Mr. Li, Award, Nobel Peace Prize) and (Mr. Li, Position, Doctor) as an example, the corresponding super-relation triple in the super-relation subgraph is (Award, oo, Position).

[0039] Therefore, in this embodiment, a set of super-relationship subgraphs corresponding to each other is constructed based on the historical subgraph set of the ICEWS18 dataset.

[0040] (3) Construct a temporal knowledge graph reasoning model based on super-relational subgraphs;

[0041] In this embodiment, as Figure 3 As shown, the temporal knowledge graph reasoning model based on super relational subgraphs includes an encoder module and a decoder module;

[0042] Among them, such as Figure 3 As shown in the left half, the encoder module includes a history subgraph level attention module, a relation update mechanism module, and a super relation subgraph level attention module.

[0043] like Figure 4 As shown, in the attention module at the history subgraph level, the entity from the previous time step is embedded. and relational embedding as well as The multidimensional dense vectors at each time step are input into a two-layer GAT network with an improved attention weight calculation method, resulting in an aggregated embedding of historical subgraph structure information. Its improved attention weight is ,in and Entities and entity , For entities and entity The relationship between them and This is the weight matrix. For learning functions; then aggregate embeddings Embedded entity from the previous moment As input, it enters a GRU network and obtains... Entity embedding at time ;

[0044] like Figure 5 As shown, in the relationship update mechanism module, the aggregation at the current moment is embedded. With Relationship All directly connected entities are embedded Perform average pooling together, then combine with the initial values ​​of relation embedding. The data is then pieced together to obtain the data for updating. Moment-by-moment relational evolution embedding Then, a Long Short-Term Memory (LSTM) module is used to capture... Moment-by-moment relational evolution embedding The temporal information is used to obtain relational evolution embeddings with temporal information. Finally, by passing Time-based relational embedding and relational evolution embedding with temporal information Concatenate the data to obtain the updated relationship embedding. ;

[0045] In the attention module at the super relational subgraph level, , Furthermore, dense relation vectors are input into a two-layer GAT network with improved attention weights to obtain the structural information aggregation embedding of the super relation subgraph. Its improved attention weight is ,in, For head relationship Tail relationship The shared central entity and This is the weight matrix. For learning functions; then aggregate embeddings Embedded relationship with the previous moment Input into the GRU network, obtain Moment-based relationship embedding .

[0046] (4) Train a temporal knowledge graph reasoning model based on super-relational subgraphs;

[0047] (4.1) Capture information at the historical subgraph level;

[0048] (4.1.1) The knowledge graph dataset Knowledge graphs at various times As a historical subgraph, from the dataset Selected from The historical subgraph at each moment is input into the temporal knowledge graph reasoning model based on the hyperrelationship subgraph;

[0049] Improving the graph attention network module using historical subgraphs Aggregation and embedding of historical subgraph structure information at any given time ;

[0050]

[0051] in, and These are the entity embedding and relation embedding for the previous timestamp, respectively. and It is a multidimensional dense vector;

[0052] (4.1.2) The timing information of the historical subgraph is captured using a gated recurrent unit (GRU) to obtain... Entity embedding at time ;

[0053]

[0054] (4.2) Relationship embedding update;

[0055] (4.2.1) Computational relation evolution embedding;

[0056] Embed the current moment's aggregation and relationships at the present moment Embedding of all directly connected entities Perform average pooling together, then combine with the initial values ​​of relation embedding. The data is then pieced together to obtain the data for updating. Moment-by-moment relational evolution embedding ;

[0057]

[0058] in, This is an average pooling operation;

[0059] (4.2.2) Capture the temporal information of relational evolution embedding;

[0060] Using a Long Short-Term Memory (LSTM) network to capture relational evolution embeddings The temporal information is used to obtain relational evolution embeddings with temporal information. ;

[0061]

[0062] in, for The temporal evolution embedding of relationships, with time-series information. and These are LSTM networks in Time and Cellular state at any given moment;

[0063] (4.3) Update relation embedding;

[0064] Will Moment-based relationship embedding and Moment-by-moment relational evolution embedding The relationships are concatenated to obtain the updated relationship embeddings.

[0065]

[0066] Here, || represents the concatenation operation. Embedding of the updated relationship;

[0067] (4.4) Capturing information at the super-relationship subgraph level;

[0068] (4.4.1) Improving the Graph Attention Network Hyper-GAT using Hyper-Relational Subgraphs to obtain... Aggregation and embedding of super-relational subgraph structural information at any given time ;

[0069]

[0070] (4.4.2) The timing information of the super-relational subgraph is captured using a gated recurrent unit (GRU) to obtain... Moment-based relationship embedding ;

[0071]

[0072] in, Embedding the relationship between the timestamp and the previous timestamp;

[0073] (4.5) Embedding entities and relational embedding Input the Conv-TransE network and calculate the probability vector of each candidate entity in the entity prediction task. The probability vector of each candidate relation in the relation prediction task ;

[0074] (4.6) Calculate the total loss after this round of training;

[0075] (4.6.1) Calculate the loss for the entity prediction task and the relationship prediction task;

[0076]

[0077]

[0078] in, and The loss for entity prediction and relationship prediction tasks. Represents timestamp All fact triples below, and They represent the first time. In this embodiment, k is 6, representing the label vectors for the entity prediction task and the relationship prediction task at each time step.

[0079] (4.6.2) Calculate the total loss;

[0080]

[0081] in, These are weighting coefficients, taken as 0.7 and 0.3 respectively;

[0082] (4.7) Total loss The vector initialization method in step (1) is fed into the Adam optimizer and optimized; then step (4) is repeated until the overall loss function of the model is calculated. Convergence signifies the completion of the model training task.

[0083] (5) Temporal knowledge graph reasoning based on super-relational subgraphs;

[0084] (5.1) The part to be completed The dynamic knowledge graph reasoning model inputs the query triplet at each moment to obtain the corresponding embedding vector;

[0085] (5.2) If the query triple to be completed is an entity prediction, the embedding vector is input into the Conv-TransE network, and then the probability of the candidate entity is calculated according to step (4.5), and the entity with the highest probability is selected as the output.

[0086] (5.3) If the query triple to be completed is a relation prediction, the embedding vector is input into the Conv-TransE network, and then the probability of the candidate relation is calculated according to step (4.5), and the relation with the highest probability is selected as the output.

[0087] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A temporal knowledge graph reasoning method based on hyperrelation subgraphs, characterized in that, Comprising the following steps: (1) downloading and initializing dynamic knowledge graph dataset; Downloading dynamic knowledge graph datasets , representing a number of discrete time samples, representing a knowledge graph at a time instant; The knowledge graph at each moment is composed of a plurality of triples, each triple is represented as (head entity , relation , tail entity ), Wherein, the head entity , the relationship , the tail entity are all text data; Knowledge graph at each time word2vec-based word embedding processing, so as to map each entity and relationship in into a multi-dimensional dense vector, wherein the head entity and the tail entity are uniformly denoted as , and the relationship is denoted as , is the number of entities, is the entity vector dimension, is the number of relationships, is the relationship vector dimension. (2) constructing hyper-relation sub-graph; traversing all the triples in the relation, mapping each relation to a twin hyperrelation subgraph a node in the relation, and defining four hyperrelations as edges of the hyperrelation subgraph ​ According to the above definition, The hyper-relation subgraph of a time instant is also composed of several triples, each of which is represented as (head relation , hyper-relation , tail relation ), and the four kinds of hyper-relation are , where represents that the intermediate entity of two relations and is the head entity in the triples corresponding to the two relations; represents that the intermediate entity of two relations and is the tail entity in the two triples; represents that the intermediate entity is the head entity in the triple corresponding to the head relation , and is the tail entity in the triple corresponding to the tail relation ; represents that the intermediate entity is the tail entity in the triple corresponding to the head relation , and is the head entity in the triple corresponding to the tail relation ; (3) constructing time-series knowledge graph reasoning model based on hyper-relation sub-graph; (4) training time-series knowledge graph reasoning model based on hyper-relation sub-graph; (4.1) capturing historical sub-graph level information; (4.1.1), the knowledge graph dataset knowledge graph at each time As a historical subgraph, the historical subgraph at a time selected from the dataset is input into the hyper-relational subgraph-based temporal knowledge graph reasoning model; Improving graph attention network module with historical subgraph aggregated embedding of historical subgraph structure information at the time instant ; ； wherein, and are an entity embedding and a relation embedding, respectively, of the previous timestamp, and are multi-dimensional dense vectors; (4.1.2), capturing the time sequence information of the historical subgraph by using a gated recurrent unit (GRU) to obtain entity embedding of the moment ; ； (4.2) relationship embedding update; (4.2.1) calculating relationship evolution embedding; the current time , and the embeddings of all entities directly connected with the relation at the current time are averaged together, and concatenated with the initial value of the relation embedding to obtain the updated relation evolution embedding at the current time ; ； wherein is an average pooling operation; (4.2.2) capturing time-series information of relationship evolution embedding; Capturing temporal information of relationship evolution embedding using long short-term memory network (LSTM) to obtain relationship evolution embedding with temporal information ​ ； wherein, is the relationship evolution embedding of the time-stamped information at time and are the cell states of the LSTM network at time and time respectively. (4.3) updating relationship embedding; will be described below. temporal relationship embedding and temporal relationship evolution embedding The splicing is performed to obtain an updated relationship embedding. ； where || denotes a concatenation operation, is the updated relational embedding; (4.4) capturing hyper-relation sub-graph level information; (4.4.1), Hyper-GAT is used to improve the graph attention network Hyper-GAT to obtain aggregated embedding of hyper-relational subgraph structure information at the moment ; ； (4.4.2), capturing the timing information of the hyper-relational subgraph by using a gated recurrent unit (GRU) to obtain relationship embedding at the time instant ; ； wherein, is a relationship embedding for the time of the previous timestamp; (4.5), embedding the entities and the relation embedding inputting the Conv-TransE network to respectively calculate a probability vector of each candidate entity in the entity prediction task and a probability vector of each candidate relation in the relation prediction task ​ (4.6) calculating total loss after this round of training; (4.6.1) calculating loss of entity prediction task and relationship prediction task; ； ； wherein, and are the losses for the entity prediction task and the relation prediction task, denotes a timestamp under which all fact triples, and denote the label vector for the entity prediction task and the label vector for the relation prediction task, respectively, at the moment in time. (4.6.2) calculating overall loss; ； wherein are weighting coefficients; (4.7), the total loss into Adam optimizer, optimize the vector initialization in step (1); then repeat step (4) until the model overall loss function converges, that is, the model training task is completed; (5) time-series knowledge graph reasoning based on hyper-relation sub-graph; (5.1), the to-be-completed first input the query triple of the moment into the dynamic knowledge graph inference model, so as to obtain the corresponding embedding vector; (5.2) if the query triple to be completed is entity prediction, input the embedding vector into the Conv-TransE network, then calculate the probability of the candidate entity according to step (4.5), and select the entity with the highest probability as the output; (5.3) if the query triple to be completed is relationship prediction, input the embedding vector into the Conv-TransE network, then calculate the probability of the candidate relationship according to step (4.5), and select the relationship with the highest probability as the output. 2.The method of claim 1, wherein, The time-series knowledge graph reasoning model based on hyper-relation sub-graph comprises an encoder module and a decoder module; The encoder module further comprises a historical sub-graph level attention module, a relationship update mechanism module and a hyper-relation sub-graph level attention module. In the historical sub-graph attention module, the entity embedding of the previous time is input into the GAT network with improved attention weight calculation method and relation embedding and The multi-dimensional dense vector input of the time is aggregated into the improved GAT network with attention weight The improved attention weight is , wherein and are the entity and the entity , is the relation between the entity and the entity , and are weight matrices, is a learning function; then the aggregated embedding and the entity embedding of the previous time are input into a GRU network to obtain the entity embedding of the time ;​ In the relationship update mechanism module, the aggregation at the current moment is embedded. With Relationship All directly connected entities are embedded Perform average pooling together, then combine with the initial values ​​of relation embedding. The data is then pieced together to obtain the data for updating. Moment-by-moment relational evolution embedding Then, a Long Short-Term Memory (LSTM) module is used to capture... Moment-by-moment relational evolution embedding The temporal information is used to obtain relational evolution embeddings with temporal information. Finally, by passing Time-based relational embedding and relational evolution embedding with temporal information Concatenate the data to obtain the updated relationship embedding. ; In the super-relation sub-graph level attention module, the , and the relation dense vector are input into the GAT network with improved attention weight to obtain the structure information aggregation embedding of the super-relation sub-graph , the improved attention weight of which is , wherein is the center entity shared by the head relation and the tail relation , and are weight matrices, is a learning function; then the aggregation embedding and the relation embedding at the previous moment are input into the GRU network to obtain the relation embedding at the moment . .

Images