Time sequence knowledge graph reasoning method based on fine-grained reinforcement learning

Through fine-grained reinforcement learning method and adaptive reward mechanism, the problems of sparse rewards and performance fluctuations in timing knowledge graph inference are solved, and effective mining and interpretable reasoning of timing dependencies are realized, which improves the stability and efficiency of the model.

CN118964633APending Publication Date: 2024-11-15SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411019362.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing time-series knowledge graph inference method based on reinforcement learning has problems of sparse rewards and performance fluctuations, and it is difficult to effectively utilize the complex timing dependencies between query problems and historical facts.

Method used

The fine-grained reinforcement learning method is adopted to design an adaptive reward mechanism and a time-sequential displacement encoder to optimize the learning process of the inference, generate fine-grained real-time rewards, and enhance the stability and efficiency of the inference.

Benefits of technology

It effectively solves the problems of sparse rewards and performance fluctuations, improves the stability and reasoning efficiency of the model, can explore the timing dependencies of entities and the time evolution laws of events, and provides explainable inference paths.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964633A_ABST
    Figure CN118964633A_ABST
Patent Text Reader

Abstract

The invention discloses a time sequence knowledge graph reasoning method based on fine-grained reinforcement learning, which relates to the field of time sequence knowledge graph reasoning and comprises the following steps of: acquiring an entity embedding set containing time information by using a time sequence displacement encoder; acquiring state information of the entity information in each exploration step, and generating a candidate action set corresponding to actions in the exploration steps so as to select an optimal action from the candidate action set; performing characterization processing on the state information of each exploration step to obtain entity information and relation characterization in the query problem; calculating the probability distribution of all actions according to the correlation between the state information after representation processing and the candidate action set, and selecting the action with the highest probability; and each exploration step executes the above steps, so that when the exploration steps are reached, a prediction result and a reasoning path of each exploration step are obtained. According to the method, the accuracy of time sequence knowledge graph reasoning can be remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of temporal knowledge graph reasoning, and in particular to a temporal knowledge graph reasoning method based on fine-grained reinforcement learning. Background Art

[0002] The temporal knowledge graph stores a large amount of entity historical event data with timestamps in the form of a graph structure. The network provides rich data support for various downstream tasks. In this type of network, in addition to the structural information of the traditional knowledge graph, the timestamp of the historical events of each entity is also recorded. The temporal knowledge graph can be regarded as a sequence of static knowledge graphs corresponding to multiple timestamps. The entity historical events in the temporal knowledge graph are represented as a quadruple (e s ,r,e o ,t), where e s is the main entity, e o is the target entity, t represents the timestamp of the event, and r represents the connection to e at timestamp t s and e o Although the temporal knowledge graph can describe time-aware facts in the real world, incomplete facts may reduce the accuracy of the model in downstream applications. Therefore, temporal knowledge graph reasoning becomes an important method to solve the incomplete knowledge. The main purpose of temporal knowledge graph reasoning is to infer the query question (e s ,r q ,? ,t q ), where t q is a timestamp in the future.

[0003] Representation learning techniques are used to characterize temporal knowledge graphs because of their outstanding ability in mining the structure and temporal features of temporal knowledge graphs. Existing embedding-based temporal knowledge graph reasoning methods can be roughly divided into time vector-guided, recurrent neural network-based, and reinforcement learning-based methods. The first method represents the elements in the temporal knowledge graph by constructing an embedding model and constructing a fusion algorithm to integrate the embeddings of timestamps and entities. However, it is difficult for such methods to exploit the relational structure of the temporal knowledge graph to mine the long-term dependencies of facts at different timestamps. On the other hand, recurrent neural network-based methods have also been proposed to explore the properties of historical facts at different timestamps. They usually use graph neural networks and recurrent models to encode and integrate the knowledge graph representations of all timestamps to mine the sequential dependencies between entities. However, most of these methods work in a black-box manner, where the missing elements of the quadruple are usually predicted by an embedding-based likelihood function. Since they cannot explicitly show which historical facts play a role in the prediction, their predicted results lack interpretability, which makes them unsuitable for practical applications. Recently, deep reinforcement learning has been applied to temporal knowledge graph reasoning tasks to generate reasoning paths for query questions. For example, Figure 1 As shown, by analyzing the logical chains in these paths, they can explain the reasons behind the model reasoning, thereby achieving explainable reasoning. Specifically, these methods usually combine graph neural network models and recursive models (a model category specifically designed to process sequence data) to encode and integrate the temporal knowledge graph data of each timestamp, thereby mining the sequential dependencies of entities in the time dimension. Then, a reasoner based on the structural and temporal characteristics of the entity is designed. It guides the agent to transfer from the starting entity of the query question to the target entity based on the correlation between entities, thereby constructing a reasoning path. In addition, in order to optimize the performance of the reasoner, these methods also design a binary final reward or auxiliary reward mechanism to provide rewards to the reasoner.

[0004] Optimizing the reasoner with binary final rewards or auxiliary rewards based on the agent state usually leads to the problem of sparse rewards, that is, the reasoner becomes unstable and may fall into local optimality. Fusion methods based on recurrent models have difficulty in exploiting the complex temporal dependencies between query questions and historical facts. These methods usually rely on the temporal order of events, which cannot mine the evolutionary patterns of events.

[0005] Therefore, technicians in this field are committed to developing a temporal knowledge graph reasoning method based on fine-grained reinforcement learning. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is the problem of sparse rewards and performance fluctuations in the reward mechanism.

[0007] To achieve the above objectives, the present invention proposes an improved reward mechanism that optimizes the reward distribution and feedback mechanism to enhance the stability and efficiency of the reasoner learning process. Specifically, the mechanism generates fine-grained rewards by analyzing the logic of the reasoner's exploration path and the semantic relevance between the relationship between the exploration path and the relationship between the query problem. This reward mechanism can provide real-time rewards that are closely related to the existing reasoning path for each step of the reasoner's reasoning, thereby effectively solving the problems of sparse rewards and performance fluctuations.

[0008] The present invention discloses a temporal knowledge graph reasoning method based on fine-grained reinforcement learning, which comprises the following steps:

[0009] S1: Use a temporal displacement encoder to obtain an entity embedding set containing temporal information;

[0010] S2: embedding the entity information in the entity set to obtain prediction results and reasoning paths using an action selection network;

[0011] The step S2 includes the following sub-steps:

[0012] S21: obtaining state information of the entity information in each exploration step, generating a candidate action set corresponding to the action in the exploration step, and selecting the best action from the candidate action set;

[0013] S22: performing characterization processing on the state information of each exploration step to obtain the characterization of the entity information and relationship in the query question;

[0014] S23: Calculating the correlation between the state information after the characterization process and the candidate action set to obtain the probability distribution of all actions, so that the action with the highest probability is selected;

[0015] S24: Each exploration step executes the above steps S21 to S23, so that when the exploration step is reached, the prediction result and the reasoning path of each exploration step are obtained.

[0016] Furthermore, the step S1 includes the following sub-steps:

[0017] S11: using a temporal displacement encoder to obtain a temporal displacement representation of the query question and the historical information;

[0018] S12: The time information and the entity information are fused by using the temporal displacement representation and the distance scoring function to obtain the entity embedding set.

[0019] Furthermore, the inference method further comprises step S3:

[0020] Each action of the action selection network obtains an adaptive reward, and the adaptive reward is obtained by summing a logical reward function and a semantic reward function.

[0021] Furthermore, the timing displacement encoder is:

[0022] Ψ(Δt)=σ(W t Δt+b)

[0023] σ is a Sigmoid activation function, and is a matrix with trainable parameters, m is the matrix dimension, and the time series displacement Δt=t q -t i+1 , t q is the timestamp of the query question, t i+1 Timestamp for historical issues.

[0024] Furthermore, the fusion of the time information and entity information is:

[0025] Temporal information is fused with entity information by minimizing the Euclidean distance of all quadruple pairs in the embedding space and the numerical order of temporal shifts:

[0026]

[0027] Among them, i refers to the i-th entity e i , j refers to the jth entity e j ;X + represents the set of true quadruple in the knowledge graph, X - By replacing X + A false quadruple obtained from the main entity or target entity; Refers to X + The real quadruple obtained by sampling, Refers to X - The false quadruple obtained by sampling;

[0028] Among them, G(s i )=x i ⊙Ψ(Δt)+rx j ⊙Ψ(Δt), ⊙ represents the Hadamard product; T refers to the number of temporal shifts; ||·|| represents the L2 vector norm; timestamp t i+1 e i Characterization By e i Characterization of X i Multiplying with the representation of the time series shift gives:

[0029] Further, the step S22 is:

[0030]

[0031]

[0032] in is a trainable parameter matrix, m is the matrix dimension; Refers to the timestamp t l Time Entity l The representation of Refers to the timestamp t q Time Entity q The representation of q Refers to the relationship q The representation of l Refers to the relationship l Representation; | | represents the concatenation of two vectors by column; ReLU represents a nonlinear activation function.

[0033] Further, the step S23 is:

[0034] The action selection network is based on the current state s l With candidate action A l The correlation between them calculates the probability distribution of all actions:

[0035]

[0036] Where softmax is a normalization function that calculates the probability of selecting each action; <·> represents the similarity function, and X′ and R′ refer to A l Embeddings of all actions in .

[0037] Furthermore, the logic reward function in step S3 is:

[0038] R logic (a l )=F fc (H l ), where F fc (·) is a multi-layer perceptron, where

[0039] Among them, H l-1 is the exploration path h in the l-1th reasoning step l-1 representation.

[0040] Furthermore, the semantic reward function in step S3 is:

[0041] The adaptive reward is:

[0042] R adapt (a l )=R logic (a l )+R sem (al ).

[0043] Furthermore, the knowledge graph reasoning method trains the reasoning model to maximize the expected value of the cumulative adaptive reward, and the expected value of the adaptive reward is:

[0044]

[0045] Q train Refers to the set of four tuples used to train the model, a1,...,a k ~π refers to the action a1,...,a k belongs to the action set π.

[0046] Technical Effects

[0047] The concept, specific structure and technical effects of the present invention are further described in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention.

[0048] The present invention uses the complex temporal dependencies between entities to represent entities and mines the temporal evolution of events. The present invention proposes a temporal displacement-enhanced dynamic embedding model with a distance scoring function. The temporal displacement-enhanced dynamic embedding model characterizes entities by utilizing the complex structural and temporal dependencies between entities, thereby retaining the useful characteristics of entities from multiple perspectives and mining the evolution patterns of events. The model uses the complex temporal dependencies between entities to characterize entities, thereby mining the temporal evolution patterns of events. The temporal displacement-enhanced dynamic embedding model designs a temporal displacement encoder to encode the temporal displacement between query questions and historical facts, thereby revealing the potential temporal evolution patterns of entities. At the same time, the model also designs a distance scoring function to integrate the temporal and structural information of entities by limiting the distance of quadruplets in the embedding space, thereby mining the semantic and structural dependencies between entities.

[0049] In addition, the present invention also designs a fine-grained adaptive reward mechanism. This mechanism can analyze the logic and semantics of the exploration path to optimize the reasoner and alleviate the problems of sparse rewards and performance fluctuations in the existing reward mechanism. This mechanism generates fine-grained adaptive rewards by analyzing the relationship between the logic of the exploration path and the comparison query question and the semantic relevance of the exploration path. This reward mechanism alleviates the problems of untimely sparse rewards and weak deep semantic relevance between auxiliary rewards and the exploration path, thereby improving the stability of the model.

[0050] According to the temporal knowledge graph reasoning method based on deep reinforcement learning of the present invention, the structure and temporal information of entities are mined, and the high-quality representation of entities is learned by analyzing the temporal displacement and semantic dependency between entities, effectively retaining the structural characteristics of the knowledge graph and the characteristics of rich entity information, thereby mining the evolution pattern of event periodicity and repetitiveness. This mechanism optimizes the reasoner by analyzing the logic and semantics of the exploration path and generating real-time rewards to avoid the agent getting lost in the early exploration stage, thereby solving the problem of unstable reasoner performance caused by sparse rewards. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a reasoning diagram of a temporal knowledge graph reasoning model based on deep reinforcement learning in the prior art;

[0052] Figure 2 It is a schematic framework diagram of the overall process of the temporal knowledge graph reasoning method based on fine-grained reinforcement learning implemented according to the present invention;

[0053] Figure 3 It is a flowchart of the steps of an implementation method of a temporal knowledge graph reasoning method based on fine-grained reinforcement learning implemented according to the present invention;

[0054] Figure 4 It is a specific implementation method of reasoning in one of the specific implementation methods of the temporal knowledge graph reasoning method based on fine-grained reinforcement learning implemented according to the present invention;

[0055] Figure 5 This is an example diagram comparing the effects of the temporal knowledge graph reasoning method based on fine-grained reinforcement learning implemented in accordance with the present invention with other methods. DETAILED DESCRIPTION

[0056] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0057] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0058] The present invention proposes a temporal knowledge graph reasoning method based on fine-grained reinforcement learning to solve the problem that existing reinforcement learning-based reasoning methods are difficult to utilize the temporal dependency between query questions and historical facts, and will fall into local optimality under sparse rewards.

[0059] like Figures 2-3 As shown in , it is a schematic diagram of the framework flow of the temporal knowledge graph reasoning method based on fine-grained reinforcement learning according to the present invention,

[0060] The temporal knowledge graph reasoning method based on fine-grained reinforcement learning implemented in accordance with the present invention comprises the following steps:

[0061] S1: Use a temporal displacement encoder to obtain an entity embedding set containing temporal information;

[0062] S2: Use the action selection network to obtain prediction results and reasoning paths for the entity information in the entity embedding set;

[0063] Step S2 includes the following sub-steps:

[0064] S21: obtaining state information of the entity information in each exploration step, generating a candidate action set corresponding to the action in the exploration step, and selecting the best action from them;

[0065] S22: Perform characterization processing on the state information of each exploration step to obtain the representation of entity information and relations in the query question;

[0066] S23: Obtain the probability distribution of all actions by calculating the correlation between the state information after characterization and the candidate action set, so that the action with the highest probability is selected;

[0067] S24: Each exploration step executes the above steps S21 to S23, so that when the exploration step is reached, the prediction result and the reasoning path of each exploration step are obtained.

[0068] Step S1 includes the following sub-steps:

[0069] S11: using a temporal displacement encoder to obtain a temporal displacement representation of the query question and the historical information;

[0070] S12: Use temporal displacement representation and distance scoring function to fuse time information and entity information to obtain entity embedding set.

[0071] The reasoning method further comprises step S3:

[0072] Each action of the action selection network receives an adaptive reward, which is obtained by summing the logical reward and the semantic reward.

[0073] Given the graph structure and query questions of the temporal knowledge graph, the present invention adopts a graph attention network to learn the representation of entities and relations. Then, the present invention designs a temporal displacement encoder to represent the temporal features by using the temporal displacement between the query question and the historical facts. Next, the distance scoring function is designed to integrate the embedding of timestamps and entities by restricting the semantic distance of all quadruple in the embedding space. In order to achieve explainable reasoning, the present invention designs an action selection network. The network guides the agent to select the best action from the candidate actions in each reasoning step, and finally generates the prediction results and reasoning path. In order to timely evaluate the actions selected by the agent and optimize the action selection network, the present invention also designs a fine-grained adaptive reward mechanism. The mechanism generates fine-grained adaptive rewards by analyzing the logic and semantics of the exploration path.

[0074] In order to retain the useful features of entities, this paper first uses a graph attention network to aggregate entity e i The features of the neighbor entities are used to update their embeddings Where m represents the dimension of entity embedding. The embedding of relation r is achieved through a linear transformation matrix Combine the embeddings of its main entity and target entity. In order to process the temporal displacement information, the present invention designs a temporal displacement encoder, which learns the representation of temporal displacement through a multi-layer perceptron. For a query question (e s ,r q ,? ,t q ) and a historical fact (e i ,r i+1 ,e i+1 ,t i+1 ), their timing shift Δt = t q -t i+1 The representation of is as follows:

[0075] Ψ(Δt)=σ(W t Δt+b)

[0076] Where σ is a Sigmoid activation function, and is a matrix with trainable parameters. The present invention also designs a distance scoring function. This function integrates temporal information and entity information by minimizing the Euclidean distance of all quadruple in the embedding space and the numerical order of temporal displacement:

[0077]

[0078] Where G(s i )=x i ⊙Ψ(Δt)+rx j ⊙Ψ(Δt), ⊙ represents the Hadamard product; X+ represents the set of true quadruple in the knowledge graph, X - By replacing X + The pseudo quadruple obtained by the main entity or target entity; T refers to the number of temporal shifts; ||·|| represents the L2 vector norm. Finally, the timestamp t i+1 e i The characterization is by e i The representation of and the representation of the time series shift are multiplied to obtain:

[0079] Based on entity embedding containing time information This paper proposes a fine-grained adaptive reward-based reasoner to achieve explainable reasoning. Specifically, the reasoner first designs an action selection network to generate explainable reasoning paths P and prediction results E. o The reasoner then designs a fine-grained adaptive reward mechanism to optimize the action selection network by maximizing the reward function.

[0080] The goal of temporal knowledge graph reasoning is to use an agent to query the main entity e s Start exploring outward and reach a new entity until the number of exploration steps reaches k. This process finally generates the prediction result e o and the detailed reasoning path h k The reasoner generates exploration paths of increasing length at each exploration step. Specifically, when the agent moves from e s Initially, it records its current state s at each exploration step l l =(r l ,t l ,e l ,e s ,t q ,r q ), where r l ,t l and e l represents the relationship, timestamp, and entity selected by the agent in step s. l It has multiple actions to choose from to reach the new state s l+1 These actions are represented as the candidate action set A l ={(e l ,r′,e′,t′),t q -t v ≤t l ≤t′<t q}, including e l The outgoing edges and neighbor entities of l =(r′,t′,e′) represents an action, t vIndicates the maximum time interval between historical events and query questions.

[0081] A l Contains multiple actions, and the present invention designs an action selection network to select the best action a for the intelligent agent l Specifically, the network uses a multi-layer perceptron to encode the state information of the agent at the lth exploration step to obtain the representation of the entities and relations containing the query question. Their representation is expressed as follows:

[0082]

[0083]

[0084] in is a trainable parameter matrix; | | represents the concatenation of two vectors by column; ReLU represents a nonlinear activation function. The action selection network selects the action based on the current state s l With candidate action A l The correlation between them calculates the probability distribution of all actions:

[0085]

[0086] Among them, softmax is a normalization function used to calculate the probability of selecting each action; <·> represents the similarity function. In order to simplify the calculation, cosine similarity is used here; X′ and R′ refer to A l The agent usually chooses the action with the highest probability. Finally, when the number of exploration steps of the agent reaches k, the entity e that the agent stays on o is the predicted result of the query question. At the same time, the reasoning path h consists of a series of actions. k will also be presented.

[0087] In order to solve the problem of sparse rewards and auxiliary rewards, this paper designs a fine-grained adaptive reward mechanism to optimize the action selection network. Specifically, it uses a recursive model bidirectional long short-term memory network to fuse historical actions H l-1 and the current action a l information to represent the exploration path h of step l l :

[0088]

[0089] Among them, H l-1 is the exploration path h in the (l-1)th reasoning step l-1 Characterization, h l-1 It consists of a series of elements visited by the agent, namely (e s ,t q), (r1, t1, e1), …, (r l-1 ,t l-1 ,e l-1 ). Based on H l , the present invention uses a multi-layer perceptron as a feature extractor to mine the content information of the path and evaluate its logical rationality. The logical reward of the path is defined as:

[0090] R logic (a l )=F fc (H l )

[0091] where F fc (·) is a multi-layer perceptron.

[0092] Use misalignment elimination from h l Remove all entities except the subject entity and the target entity, and query the embedding r of the relationship q Should be close to h l The present invention designs a similarity function that generates semantic rewards by distinguishing the differences between them at the semantic level:

[0093]

[0094] where F smi is a similarity function. In order to simplify the calculation, cosine similarity is used. Finally, the logical reward and semantic reward of the lth reasoning step are combined to construct a fine-grained adaptive reward:

[0095] R adapt (a l )=R logic (a l )+R sem (a l )

[0096] The present invention fixes the length of the agent's search path to k, which means that the agent will generate k discrete states. Each action selected by the action selection network will be rewarded, and the expected value of the cumulative adaptive reward of all training samples is defined as:

[0097]

[0098] The temporal knowledge graph reasoning model is trained by maximizing the expected value of the cumulative adaptive reward.

[0099] The effectiveness of the present invention can be further illustrated by the following simulation experiment. It should be noted that the parameters used in the experiment do not affect the generality of the present invention.

[0100] CPU: Intel Core CPU i7-10700, RAM 16.00GB, GPU: RTX3080ti, operating system Ubuntu20.04LTS, simulation design language Python.

[0101] The simulation process uses the ICEWS14 dataset to test the TODEAR model proposed in this invention. The ICEWS14 dataset is a subset of the integrated conflict early warning system (ICEWS) that occurred in 2014. The dataset is divided into a training set, a validation set, and a test set in a ratio of 80%, 10%, and 10% according to the timestamp.

[0102] The present invention runs on the Pytorch framework and NVIDIA RTX3080ti machine throughout the experiment. The parameters of the TODEAR model are initialized using the Xavier initializer and optimized using Adam. The number of layers of the graph attention network is set to 3, and the dimension of the node embedding is 100. The number of layers of the bidirectional long short-term memory network is 2. The multilayer perceptron is a 4-layer perceptron. The present invention sets the learning rate of Adam to 0.001 and the length of the inference step k to 3. To ensure the stability of the experimental results, all experiments were repeated three times.

[0103] In order to demonstrate the reasoning ability of the present invention and the interpretability of the reasoning process, the present invention Figure 3 The inference path of the present invention for two query problems in the ICEWS14 dataset is shown in the table. The right side of the table shows the probability of occurrence of some actions calculated by the action selection network, where the action with the highest probability is the action taken in the current step. Therefore, the table can clearly track the multiple evidences on which the prediction results depend.

[0104] Figure 4 Partial visualization of the TODEAR model reasoning path on the ICEWS14 dataset. Probability refers to the probability of selecting each action.

[0105] Given a query (South Korea, make a visit, ?, 2014-11-24), the TODEAR model completes its reasoning through three steps of exploration. In the first step, TODEAR processes various historical facts related to South Korea, mainly positive relational predicates with other countries. Because the action of signing a formal agreement has the strongest relationship with the facts in the query, the agent chooses (South Korea, sign a formal agreement, Japan, 2014-09-17) with the highest probability. In the second step, the agent chooses (Japan, express cooperation intention, Australia, 2014-09-19), which means that the two countries have a cooperative relationship before the timestamp of the query. In contrast, participating in negotiations is a relational predicate with negative semantics, which indicates that the two countries prefer to resolve the problem through negotiations. In the third step, the agent chooses the positive action of providing economic assistance, which may be related to South Korea's visit behavior. Therefore, the answer to the query is Morocco. For the second query (Poland, discuss by phone, ?, 2014-12-05), the TODEAR model also completes the reasoning through three steps of exploration. In the first step, the agent chooses (Poland, visit, France, 2014-09-20) because the action of visiting is closer to the semantics of discussing over the phone. The expression is mainly about the willingness to take action, but the behavior has not occurred. Similarly, the agent chooses the more positive relational predicate of providing humanitarian aid in the second step. In the third step, the agent finds that the probability of selecting the current node is greater than the probability of selecting any other node, so it stays in place and chooses Sudan as the answer. Through case studies, the present invention observes that the TODEAR model can provide a specific reasoning path for each query question, making the reasoning results interpretable.

[0106] The present invention has been tested on the ICEWS14, ICEWS18, ICEWS5-15 and GDELT datasets. The experimental results are as follows: Figure 4 shown.

[0107] Figure 5 User prediction performance of fine-grained reinforcement learning-based user behavior inference model and reinforcement learning-based comparison methods on ICEWS14, ICEWS18, ICEWS5-15, and GDELT datasets. The best and second-best results are marked in bold and underlined, respectively, and "-" means that the original paper did not test it.

[0108] Compared with the reinforcement learning-based method, its Prec@1, Prec@5, Prec@10 and MRR indicators have been improved by at least 2.04%, 3.38%, 7.01% and 0.73%. The present invention performs particularly well on the ICEWS05-15 dataset, which has extremely fine time granularity and rich timestamps, and traditional reasoning methods often forget early knowledge. The present invention can fully capture the multivariate representation of the temporal knowledge graph through its designed distance scoring function, thereby adapting to frequently changing historical facts.

[0109] Through a fine-grained adaptive reward mechanism, the present invention can converge quickly during the training process, reducing the model training time. Compared with other reinforcement learning methods, the present invention has shorter running time and higher efficiency on ICEWS14, ICEWS18, ICEWS5-15 and GDELT datasets.

[0110] The present invention adopts mature technologies such as graph attention network and bidirectional long short-term memory network, which is easy to integrate and apply in existing systems. Through modular design, the present invention can be combined with other graph neural networks and reinforcement learning methods to enhance the overall performance of the system.

[0111] The time complexity of the present invention is low, and it is suitable for the reasoning task of large-scale time series knowledge graph. By optimizing the embedding model and reward mechanism, the consumption of computing resources is reduced and the reasoning efficiency is improved.

[0112] The present invention can predict the future trend of public sentiment, provide decision support for social network platforms, and help control and guide public opinion. The present invention can be used to build an efficient question-answering system, generate accurate answers through time constraints, entity matching, and graph neural networks, and improve user experience. In the legal field, the present invention can assist in identifying similar cases, recommend legal texts, and improve the intelligence level of legal services.

[0113] In a preferred embodiment of the present invention, the present invention demonstrates its reasoning ability and the interpretability of the reasoning process on the ICEWS14 dataset. Specifically, for each query question, the present invention can gradually display each step of the reasoning process and clearly present the logical chain of reasoning. In addition, the present invention can also display the probability of occurrence of each action in each reasoning step, which provides users with a more intuitive understanding. In this way, not only can the final reasoning result be seen, but also each link in the reasoning process can be deeply understood, providing a basis for the optimization of subsequent models.

[0114] In another preferred embodiment of the present invention, compared with existing reinforcement learning-based methods on ICEWS14, ICEWS18, ICEWS5-15 and GDELT datasets, the improvements of Prec@1, Prec@5, Prec@10 and MRR indicators of the present invention are at least 2.04%, 3.38%, 7.01% and 0.73%.

[0115] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A temporal knowledge graph reasoning method based on fine-grained reinforcement learning, It is characterized in that The method comprises the following steps: S1: Use a temporal displacement encoder to obtain an entity embedding set containing temporal information; S2: embedding the entity information in the entity set to obtain prediction results and reasoning paths using an action selection network; The step S2 includes the following sub-steps: S21: obtaining state information of the entity information in each exploration step, generating a candidate action set corresponding to the action in the exploration step, and selecting the best action from the candidate action set; S22: performing characterization processing on the state information of each exploration step to obtain the characterization of the entity information and relationship in the query question; S23: Calculating the correlation between the state information after the characterization process and the candidate action set to obtain the probability distribution of all actions, so that the action with the highest probability is selected; S24: Each exploration step executes the above steps S21 to S23, so that when the exploration step is reached, the prediction result and the reasoning path of each exploration step are obtained.

2. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 1, It is characterized in that The step S1 includes the following sub-steps: S11: using a temporal displacement encoder to obtain a temporal displacement representation of the query question and the historical information; S12: The time information and the entity information are fused by using the temporal displacement representation and the distance scoring function to obtain the entity embedding set.

3. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 1, It is characterized in that The inference method further comprises step S3: Each action of the action selection network obtains an adaptive reward, and the adaptive reward is obtained by summing a logical reward function and a semantic reward function.

4. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as described in any one of claims 1 to 3, It is characterized in that The timing displacement encoder is: Ψ(Δt)=σ(W t Δt+b) σ is a Sigmoid activation function, and is a matrix with trainable parameters, m is the matrix dimension, and the time series displacement Δt=t q -t i+1 , t q is the timestamp of the query question, t i+1 Timestamp for historical issues.

5. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 4, It is characterized in that The fusion of the time information and entity information is: Temporal information is fused with entity information by minimizing the Euclidean distance of all quadruple pairs in the embedding space and the numerical order of temporal shifts: Among them, i refers to the i-th entity e i , j refers to the jth entity e j ;X + represents the set of true quadruple in the knowledge graph, X - By replacing X + A false quadruple obtained from the main entity or target entity; Refers to X + The real quadruple obtained by sampling, Refers to X - The false quadruple obtained by sampling; Among them, G(s i )=x i ⊙Ψ(Δt)+rx j ⊙Ψ(Δt), ⊙ represents the Hadamard product; T refers to the number of time series shifts; ||·|| represents L 2 vector norm; timestamp t i+1 e i Characterization By e i Characterization of X i Multiplying with the representation of the time series shift gives:

6. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 1, It is characterized in that The step S22 is: in is a trainable parameter matrix, m is the matrix dimension; Refers to the timestamp t l Time Entity l The representation of Refers to the timestamp t q Time Entity q The representation of q Refers to the relationship q The representation of l Refers to the relationship l Representation; || represents the concatenation of two vectors by column; ReLU represents a nonlinear activation function.

7. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 6, It is characterized in that The step S23 is: The action selection network is based on the current state s l With candidate action A l The correlation between them calculates the probability distribution of all actions: Where softmax is a normalization function that calculates the probability of selecting each action; <·> represents the similarity function, and X′ and R′ refer to A l Embeddings of all actions in .

8. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 3, It is characterized in that The logic reward function in step S3 is: R logic (a l )=F fc (H l ), where F fc (·) is a multi-layer perceptron, where Among them, H l-1 is the exploration path h in the l-1th reasoning step l-1 representation.

9. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 8, It is characterized in that The semantic reward function in step S3 is: The adaptive reward is: R adapt (a l )=R logic (a l )+R sem (a l )。 10. The temporal knowledge graph reasoning method based on fine-grained reinforcement learning as claimed in claim 9, It is characterized in that The knowledge graph reasoning method trains the reasoning model to maximize the expected value of the cumulative adaptive reward, where the expected value of the adaptive reward is: Q train Refers to the set of four tuples used to train the model, a 1 ,...,a k ~π refers to action a 1 ,...,a k belongs to the action set π.

Citation Information

Cited By

  • Time knowledge graph question and answer method based on multi-granularity implicit tense perception

    CN119761502A