Time sequence knowledge graph multi-hop path reasoning method based on attention mechanism and reinforcement learning

Abstract

The invention discloses a time sequence knowledge graph multi-hop path reasoning method based on an attention mechanism and reinforcement learning. Firstly, after time information is mapped, entities and relationships in a time sequence knowledge graph are dynamically represented by using a time sequence embedding method based on an attention mechanism, and the time information is embedded and combined with the entities and the relationships, so that accurate capture of time sequence features is realized. On the basis, path exploration is guided by adopting a strategy network based on an attention mechanism, action selection of each step is optimized through reinforcement learning, and multi-hop reasoning is carried out step by step. Secondly, combining time constraint and reasoning step number limitation, optimizing path exploration by using a reward mechanism, and ensuring that path reasoning conforms to time limitation and path length limitation in the time sequence knowledge graph; and finally, training the model by using a strategy gradient algorithm of reinforcement learning, and continuously updating and improving the reasoning capability of the strategy network. The method has the advantages that the attention mechanism and the reinforcement learning method are combined, the multi-hop reasoning path in the time sequence knowledge graph can be effectively explored, the time factor is considered in the reasoning process, the interpretable reasoning path can be provided, and the reasoning accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to a multi-hop path reasoning method for a temporal knowledge graph based on an attention mechanism and reinforcement learning. The present invention is a multi-hop path reasoning method for the quality of temporal knowledge graph data, and belongs to the intersection of engineering application and information science. Background Art

[0002] Knowledge graph is a graphical data structure used for structured storage and representation of knowledge. It expresses entities in the real world and their relationships in the form of nodes and edges. This representation of knowledge can be presented in the form of triples (entity 1, relationship, entity 2). The main features of knowledge graph include structured representation, scalability and semantic association, which makes it not only used for knowledge storage, but also for intelligent reasoning and information retrieval. Temporal knowledge graph introduces the time dimension on the basis of traditional knowledge graph, so that knowledge representation can reflect dynamic facts that change over time. It describes the existence state of a fact at a specific time point or time period by expanding the triple structure to a quadruple (entity 1, relationship, entity 2, time). Temporal knowledge graph is widely used in temporal reasoning, event modeling and time-sensitive question-answering systems to help understand and predict the law of fact evolution over time. The existing mainstream multi-hop reasoning methods of temporal knowledge graph are mainly based on embedding, rules and graph neural networks.

[0003] Embedding-based reasoning methods embed entities, relations, and time information into a low-dimensional vector space and perform reasoning in the embedded space. This method can effectively process large-scale data and capture complex temporal relationships through time-aware vector representations. However, when faced with complex reasoning tasks, the embedding method has weak interpretability.

[0004] Rule-based reasoning methods explicitly model the logical relationships in the time series knowledge graph and perform multi-hop reasoning using predefined rules and time constraints. This method can capture clear time series patterns and has strong interpretability. However, this type of method relies on manually designed rules and is difficult to handle large-scale or complex time series data, and has limited applicability.

[0005] The reasoning method based on graph neural network regards entities and relationships as nodes and edges in the graph, and models time dependencies through the propagation mechanism of graph neural network. This method can automatically learn the temporal patterns in the graph structure and is suitable for complex temporal reasoning tasks, but its training cost is high and the reasoning process is not easy to explain, especially when facing large-scale graphs, the amount of calculation is very large.

[0006] The hybrid method combines the interpretability of logical reasoning with the pattern recognition ability of deep learning, and can capture both explicit and implicit information in the knowledge graph. This method can achieve good results in temporal reasoning. However, due to the high complexity of the model, the training process is cumbersome, and there are still certain limitations in the processing of large-scale data.

[0007] In early studies, these multi-hop reasoning methods for temporal knowledge graphs modeled temporal relationship features in a relatively rough manner and failed to fully capture the details of temporal dynamic changes in the graph. Specifically, many methods ignore the complex interactions between entities and relations, between time and relations, and between relations in the temporal reasoning process. For example, in practical applications, events occurring at different time points and their mutual correlations have an important impact on the reasoning results. However, most existing models fail to deeply capture the potential complexity of temporal relations, resulting in insufficient utilization of time-dependent features in multi-hop paths. In addition, these methods generally do not model the representation of relations in a refined manner, which is crucial for the dynamic reasoning process, especially when predicting future events and discovering new time-sensitive facts. This limitation is particularly evident. Summary of the invention

[0008] Purpose of the invention: To address some problems existing in traditional multi-hop reasoning methods of temporal knowledge graphs: First, existing models usually model temporal relationships in a rough manner, relying only on basic triples or simplified time information, and failing to deeply explore the complex interactions between entities, between relationships, and between time and relationships. Second, the mining of temporal information is insufficient, and many models tend to only focus on a single temporal aspect, and fail to fully consider temporal reasoning as a dynamic process that changes over time, which involves the complex temporal characteristics of entities and relationships. Third, most existing methods only focus on one aspect of the temporal knowledge graph (such as time information, relationship embedding or graph neural network), so when faced with complex temporal reasoning tasks, the robustness and accuracy of the model are insufficient. The purpose of the present invention is to provide a multi-hop path reasoning method for a temporal knowledge graph based on an attention mechanism and reinforcement learning, which is used to solve these problems existing in current temporal knowledge graph reasoning methods.

[0009] Technical solution: To achieve the above-mentioned purpose, the present invention proposes a multi-hop path reasoning method for time series knowledge graph based on attention mechanism and reinforcement learning. First, the Transformer model is used to embed the time series data, including the embedding of entities and relationships, to capture the complex dependencies and deep-level features in the time series. Next, a time mapping layer is introduced to map the date and time information from the "year-month-day" format to independent "year", "month" and "day" representations, so as to process the time data more finely. Then, using the relational attention layer, the attention mechanism is extended to the relational level, not only paying attention to the entities, but also giving dynamic weights to the relationships between entities, further improving the accuracy of reasoning. In addition, a dynamic completion module is introduced, in which the action space includes original actions and potential actions to adapt to changes and uncertainties in time series reasoning. Finally, these modules are integrated, and the reasoning strategy is optimized through the reinforcement learning algorithm to achieve efficient multi-hop reasoning of the time series knowledge graph. Its specific technical solution includes the following steps:

[0010] Step 1: Time mapping and relation embedding.

[0011] First, entities, relations, and time are represented as vectors through a pre-trained temporal knowledge graph embedding model to capture the complex features and dependencies in temporal data.

[0012] (1) Time mapping: The time feature in time series data is an important component of multi-hop reasoning. This paper vectorizes the timestamp, decomposes the time information into time granularities such as "year", "month", and "day", and embeds it into a unified time mapping space.

[0013] (2) Entity and relationship embedding: In order to better capture the deep dependencies between entities and relationships in time series, we use the Transformer model to embed entities and relationships. Transformer effectively handles long-distance dependencies in time series data through the self-attention mechanism, and can aggregate and propagate information between different time steps, thereby obtaining fine-grained features of the time series knowledge graph.

[0014] Step 2: Multi-hop reasoning and action selection.

[0015] In order to perform multi-hop reasoning on the temporal knowledge graph, this paper introduces a reinforcement learning policy network, which enables the model to dynamically select the reasoning path.

[0016] (1) Dynamic completion of latent action space: To address the problem of limited action space in traditional reasoning methods, this paper introduces a dynamic completion mechanism. Based on the pre-trained temporal knowledge graph embedding model, the latent action space is learned and the completed action set is integrated into the original action set, thus enhancing the agent’s ability to explore multiple possible paths during the reasoning process.

[0017] (2) LSTM historical information encoding: In order to record the historical path information during the reasoning process, the long short-term memory network LSTM is used to encode the past reasoning steps and capture the historical state of the previous reasoning steps. This historical information is used to update the current state representation and help the policy network select the optimal path in subsequent reasoning.

[0018] Step 3: Strategy network and reward mechanism.

[0019] (1) Policy network: In the reinforcement learning framework, the policy network is responsible for mapping the current state to the action space and outputting the selection probability of each action through the normalized exponential function softmax. The input includes the embedded representation of entities, relations, and timestamps, and the output is the selection probability of the path.

[0020] (2) Reward mechanism: A three-part reward mechanism is designed to evaluate the quality of the reasoning path, including hit reward, embedding soft reward, and path embedding soft reward. The hit reward is used to evaluate whether the correct entity is selected; the embedding soft reward scores the predicted target entity through the scoring function of the temporal embedding model; the path embedding soft reward ensures the rationality of the path by accumulating the relations and time embedding vectors selected in the reasoning path.

[0021] Step 4: Model training and optimization.

[0022] (1) Pre-training stage: First, the temporal knowledge graph embedding model is pre-trained so that the model can grasp the complex associations between entities, relationships, and time, providing support for subsequent reasoning.

[0023] (2) Reinforcement learning training: The reinforcement learning stage is trained through the policy gradient Reinforce algorithm. The agent dynamically selects the reasoning path through interaction with the environment. The model optimizes the parameters of the policy network and improves the reasoning accuracy.

[0024] (3) Path optimization and dynamic adjustment: By combining the dynamic completion mechanism of the latent action space with the reinforcement learning of the policy network, the model can gradually optimize the reasoning path selection to ensure that the correct target entity is found or the reasoning is terminated within a limited step length.

[0025] Beneficial effects: In response to the data quality problem of knowledge graphs, the present invention proposes a multi-hop path reasoning method for temporal knowledge graphs based on attention mechanism and reinforcement learning. This method not only focuses on the temporal dynamic features in the knowledge graph, but also introduces the attention mechanism at the relationship level to capture complex relationship interactions and temporal evolution laws, thereby solving the problem of decreased reasoning performance caused by incomplete temporal and relational information in large-scale knowledge graphs. In addition, most existing reasoning methods only perform simple embedding representations of entities or relationships, fail to fully explore the details in temporal information, and many methods only support one-hop reasoning, resulting in potential multi-hop information being ignored. In order to solve these problems, the present invention adopts the Transformer model to embed entities and relationships in combination with temporal features, deeply mines potential time-dependent features from temporal data, and combines reinforcement learning to optimize the reasoning strategy, thereby effectively improving the performance of the reasoning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The overall flow chart of the multi-hop path reasoning method of temporal knowledge graph based on attention mechanism and reinforcement learning.

[0027] Figure 2 It is a method flow chart.

[0028] Figure 3 It is a graph of the relation-oriented attention mechanism.

[0029] Figure 4 It is a strategy network module diagram. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings.

[0031] The present invention is a multi-hop path reasoning method for temporal knowledge graph based on attention mechanism and reinforcement learning. The overall flow chart is shown in the attached figure. Figure 1 As shown. First, the time mapping layer is used to map the date and time information from the "year-month-day" format to independent "year", "month", and "day" representations in order to process time data more finely. Then, the relational attention layer is used to extend the attention mechanism to the relational level, not only paying attention to the entities, but also giving dynamic weights to the relationships between entities, further improving the accuracy of reasoning. In addition, a dynamic completion module is used, in which the action space includes original actions and potential actions to adapt to changes and uncertainties in temporal reasoning. Finally, these modules are fused and the reasoning strategy is optimized through a reinforcement learning algorithm. A multi-hop path reasoning method for temporal knowledge graphs based on attention mechanism and reinforcement learning mainly includes four parts: 1) time-relationship attention module; 2) policy module; 3) policy network optimization module, 4) training module. The specific architecture of the model is shown in the attached Figure 2 , the specific implementation steps are as follows.

[0032] 1. Time-Relationship Mapping Module

[0033] (1) Time Mapping Module

[0034] When building an event understanding model, the time mapping module is a crucial component that specifically processes the time information of events to enhance the model's understanding and prediction capabilities of event sequences. First, in order to reduce the parameters of timestamps, the traditional approach is to treat the complete timestamp (such as "year-month-day") as a single value, which may lead to an explosive growth of model parameters. To solve this problem, the present invention adopts a novel splitting strategy to decompose each timestamp into three independent parts: year, month, and day. This splitting method can model the three dimensions of time separately without adding too many parameters. Then, t is concatenated using vector splicing. y , t m , and t d ∈R k The timestamp vector is combined as shown in formula (1).

[0035] T t =t y +t m +t d (1)

[0036] where t y t m and t d Vectors representing year, month, and day respectively.

[0037] Then, this paper uses v s and T t Calculate the transformation matrix W and construct the mapping space, as shown in formula (2).

[0038]

[0039] where v s , t t ∈R k are the vectors of entity s and timestamp t respectively, and T represents the transpose operation.

[0040] Considering that only the information of s is known, the related relationships of s are clustered into chain A e , and then each relationship vector Mapping to the constructed space W∈R k×k , the calculation process is shown in formula (3).

[0041]

[0042] in It is a relationshipi vector, W∈R k×k is a linear transformation matrix based on the entity and timestamp vectors.

[0043] (2) Relational Attention Module

[0044] like Figure 3 The figure shows the relation-oriented attention mechanism. In the attention module, the present invention adopts the “scaled dot product attention” and “multi-head attention” proposed by Vaswani et al. to calculate the attention between relations. In fact, the query consists of the vector v′ of the relation r. r Indicates that the keys and values ​​of the attention mechanism are based on chain A e Forming a matrix and The output of the attention layer is expressed as The calculation process is shown in formula (4):

[0045]

[0046] Among them, attn() is the calculation process of the attention mechanism.

[0047] Refer to "Scaled Dot Product Attention", for each r i ∈A e Using v′ r and v′ i The multiplication between them is used to calculate the attention of the relationship, and the value is divided by Scaling is performed, where n′ is the dimension of time embedding, and the calculation process is shown in formula (5).

[0048]

[0049] Among them, v′ r and Represent r and r respectively i The mapping vector of , T represents the transpose operation.

[0050] The attention value represents each relationship r i For the importance of the source relation r, use α i Relative attention λ i It is calculated by the Softmax function on all attention values, as shown in formula (6).

[0051]

[0052] where α i represents the attention value, λ i Represents the relative attention value.

[0053] The new vector of r is the sum of all relationship vectors in the chain weighted by their attention values. The calculation process is shown in formula (7)(8).

[0054]

[0055] in Represents r i The mapping vector, λ i Represents r i Relative attention value.

[0056]

[0057] where v' r Represents the mapping vector of r.

[0058] In addition, as suggested by Busbridge et al., “multi-head attention” is used to encapsulate more information about the relationship in the learning process. H independent attention heads are used to calculate the embedding, which is then concatenated, and the calculation process is shown in Equation (9).

[0059]

[0060] Where П represents a serial operation.

[0061] 2. Strategy Module

[0062] When using a random strategy, the action space A constrained by temporal information t For each action in, for each state s t There is a certain probability of selection. In order to guide the agent in the current state s t Next, we select an action for the t+1th jump using a reinforcement learning strategy network module, such as Figure 4 As shown. The agent is in state s t Next selected action a t It can be expressed as a combination of relation embedding vector, time embedding vector and entity embedding vector, that is,

[0063]

[0064] where r t+1 、tim t+1 and e t+1 They represent the relationship, time and entity of t+1 hop respectively.

[0065] The long short-term memory network (LSTM) is used to encode historical information into a continuous vector. The historical reasoning path information, h t ={e s , r1:tim1,e1...,e t-1 , r t :timt , e t}, including past observations and action sequences, where some action a i may not belong to TKG, i.e. These actions come from the latent action space obtained by the embedding model.

[0066] Hidden state h t It can be based on the previous hidden state h t-1 and currently in a t-1 However, for the initial step size t=0, no previous hidden state is available. In the present invention, a zero vector is selected as the initial input, and the LSTM calculates the initial hidden state h0 according to its weights and biases. Specifically, it can be expressed as formula (11).

[0067] h0=LSTM(0,0) (11)

[0068] Where h0 represents the initial hidden state, and 0 here represents the zero vector.

[0069] The dynamic update process of LSTM can be expressed as formula (12).

[0070] h t =LSTM(h t-1 , a t-1 ) (12)

[0071] where h i-1 represents the hidden state at the t-1th jump, a t-1 Indicates the action at jump t-1. The tth state s t It can be expressed as formula (13).

[0072] s t =[r q ;e t ;h t ;tim] (13)

[0073] where r q Indicates the query relationship, e t An entity representing a jump in time t.

[0074] Then, to encode the action space, we transform A t The embeddings of all actions in are stacked together to form a matrix Here, R represents the real number domain, and d represents the dimensions of the entity vector, relation vector, and timestamp vector. In deep reinforcement learning, the agent is represented as a neural network. The neural network transforms the current state s t As input, it generates all possible actions A for a given state tThe probability distribution of the policy network π is defined as shown in formula (14).

[0075] π θ (a t |s t )=σ(A t ×W1RELU(W2|s t )) (14)

[0076] Where σ represents the softmax operator, W1 and W2 are two parameters of the reinforcement learning policy network.

[0077] 3. Policy network optimization module

[0078] (1) Dynamic completion module

[0079] Based on the embedding model, a set of potential actions t is created for each state, and these actions that may find the correct entity are merged into the original action space A. t Therefore, for the current state, the definition of the potential action candidate set is as shown in formula (15).

[0080]

[0081] where e p Represents the entity prediction probability vector obtained by the temporal knowledge graph embedding model.

[0082] Since the state is affected by the potential action, we define the state as s t =(e p , r q , e t ,h t , tim), so the probability calculation process of the potential action in the current state is shown in formula (16).

[0083] p((r p , e p , tim p )|s t )=p((r p , tim p )|s t )p(e p |(r p , tim p ), s t ) (16)

[0084] where s t Represents the current state space.

[0085] First, the current state information is fed into a multi-layer perceptron (MLP) to calculate the highest probability relationship and timestamp combination. Then, based on the selected relationship and timestamp combination, the entity with the highest probability is selected. The specific operation is to use a three-layer MLP to calculate the attention vector of the current state, and select the most significant relationship and timestamp combination based on the highest attention value in the attention vector, as shown in formula (17).

[0086] S t =MLP(s t ) (17)

[0087] where s t Represents the current state space.

[0088] Finally, only the top k entities with the highest probability among these combinations are retained. Therefore, the attention vector w for each relationship and timestamp combination is defined as shown in formula (18).

[0089]

[0090] in Represents the combination of the jth relation and timestamp.

[0091] The potential action N is calculated by using formula (19) add The number of relationships and timestamps is selected, and the top x relationships and timestamp combinations with the highest attention values ​​in the attention vector w are formed to form a new set of relationship-timestamp combinations.

[0092]

[0093] Where N(s t ) indicates state s t The size of the action space, M(s t ) indicates state s t The maximum number of high-quality potential actions, α t Used to adjust potential actions in a given state, It is rounded up.

[0094] Use the pre-trained temporal knowledge graph model to obtain quadruple queries (e t , r i ,? ,tim i ) for each relation timestamp combination The probability distribution of the tail entity (where And e t In state t ). The probability distribution of the model predicting the tail entity is defined as p(e|(r i , tim i ), s t ). To simplify the action space, only keep each The top k tail entities with the highest probability generate k potential actions In this way, each In status t The k potential actions under this condition together constitute the potential action space The parameter k is determined by the state s t The number of potential actions N under add (s t ) and the number of tail entities k to be retained, and the calculation process is shown in formula (20).

[0095] x=N add (s t ) / k (20)

[0096] Where N add (s t ) represents the number of potential actions.

[0097] By using a dynamic completion strategy, the temporal embedding model can provide the agent with comprehensive prediction information of TKG under various conditions. In addition, this approach also provides superior latent actions, thereby improving the reasoning ability of the TKG multi-hop path reasoning model.

[0098] (2) Reward Module

[0099] The reward module consists of three parts: hit reward, embedding soft reward and path embedding soft reward.

[0100] The hit reward is the basic reward. The agent needs to select a specific action from a large number of actions. If the selected action can reach the correct entity e, it will receive a reward of 1; if it cannot reach the correct entity e T , then there is no reward, that is, 0 reward. The calculation process of the hit reward is shown in formula (21).

[0101]

[0102] Where R1 represents the first part of the reward.

[0103] The embedded soft reward is used to measure the four groups (e s , r q , e T ,tim) is obtained according to the scoring function of the temporal embedding model. T is the correct target entity, the agent will receive a reward of 1; otherwise, the agent is divided into four groups (e s ,r q ,e T ,tim) to obtain the reward. The calculation process of embedded soft reward is shown in formula (22).

[0104] R2=f(e s , r q , e T , tim) (22)

[0105] Where R2 represents the second part of the reward embedded in the soft reward.

[0106] In traditional embedding methods, the embedding of the correct target entity should be similar to the embedding of the predicted entity at the last step of the reasoning path, i.e., e o ≈e T In addition, since the final prediction relies on a multi-hop path reasoning method, the embedding of the predicted entity should be equivalent to the embedding of the source entity plus the embedding of the query relation and query time, i.e. The embedding vector of the query relation and query time should be equal to the cumulative embedding vector sum of each selected relation and time in the entire reasoning path, denoted as Based on the above vector relationship, the calculation process of the path-based embedded soft reward function is shown in formula (23).

[0107]

[0108] in represents the expected score function of path embedding, and D(·) represents the cosine similarity function.

[0109] Finally, the hit reward, embedded soft reward and path embedded soft reward are comprehensively considered to obtain a more accurate reward function. The calculation process is shown in formula (24).

[0110] R=R1+(1-R1)(λR2+(1-λ)R3) (24)

[0111] Where λ is a constant factor, satisfying λ < 1, and R3 represents the path embedding soft reward.

[0112] 4. Training Module

[0113] The training is divided into two stages: the first stage is the pre-training stage, which uses the time embedding model to obtain prediction information, and the second stage is to train the reinforcement learning agent so that the agent can select actions from the action space to find the correct entity within the specified step size or fail to query the entity that meets the requirements.

[0114] By maximizing the expectation J(θ RS ) to train the policy network. The expected expectation J(θ) for each query quadruple in the training set RS ) is calculated as shown in formula (25).

[0115]

[0116] where θRS is the parameter of the policy network, S T represents the state space during T jumps.

[0117] Policy network parameters θ RS The stochastic gradient The calculation process is shown in formula (26).

[0118]

[0119] Where π represents the policy network.

Claims

1. A multi-hop path reasoning method for temporal knowledge graph based on attention mechanism and reinforcement learning, characterized in that: The following steps are involved: (1) Knowledge graph embedding: Use the temporal embedding method to embed entities and relations in the temporal knowledge graph. First, map the time information, then initialize the representation of entities and relations, and use the attention mechanism to further represent the relations. (2) Path exploration: A policy network based on an attention mechanism is used to guide path exploration. First, the embedded representation of the current state is combined with the policy network in reinforcement learning to select actions. In multi-hop reasoning, each action selection corresponds to a relationship and the next entity in the knowledge graph until the target entity is reached. (3) Reinforcement learning training: The policy network is trained through a reinforcement learning framework, and the time constraints and reward mechanisms in the temporal knowledge graph are used to optimize the selected path. The policy network is updated during the training process based on the criterion of maximizing the cumulative reward.

2. According to claim 1, a multi-hop path reasoning method for temporal knowledge graph based on attention mechanism and reinforcement learning is characterized in that: In step (1), the knowledge graph is embedded in the following way: (1-1) Statically initialize entities and relations, using random initialization or pre-trained embedding vectors; (1-2) The attention mechanism is applied to the relationship to obtain the embedding vector of the relationship. The time information is combined with the embedding vectors of entities and relationships to obtain a dynamic embedding that represents the changes in entities and relationships at different time points.

3. According to claim 1, a multi-hop path reasoning method for temporal knowledge graph based on attention mechanism and reinforcement learning is characterized in that: In step (2), a strategy network based on an attention mechanism is used for path exploration, and its implementation method includes: (2-1) Use the attention mechanism to weight the candidate relationships. The attention mechanism determines the priority of the next action by calculating the correlation between the current state and the candidate relationship. (2-2) In each time step, combined with the policy network in reinforcement learning, the optimal relationship and entity are selected based on the embedded representation of the current state and the historical path information.

4. According to claim 1, a multi-hop path reasoning method for temporal knowledge graph based on attention mechanism and reinforcement learning is characterized in that: In step (3), the policy network is trained using a reinforcement learning framework. The training process includes: (3-1) Design a reward function to give rewards or penalties based on the final result of path exploration. The reward function is designed based on whether the reasoning result is correct and the length of the path; (3-2) Use the policy gradient algorithm to optimize the policy network and update the network parameters based on the principle of maximizing cumulative rewards; (3-3) During the training process, consider setting the maximum number of search steps to ensure that the selected path meets the path length limit of the temporal knowledge graph.

Images