Two-stage reinforcement learning time sequence knowledge graph reasoning method based on historical and time knowledge

Abstract

The invention provides a time sequence knowledge graph reasoning method based on two-stage reinforcement learning of historical and time knowledge for a time sequence knowledge graph reasoning task. According to the method, firstly, the obtained historical entities are analyzed and processed, and then whether the predicted entities are related to the historical entities or not is further identified through optimized comparative learning, so that the reasoning process can pay more attention to useful entities. According to the method, historical data and time background knowledge are effectively integrated, and more comprehensive and accurate support is provided for time sequence reasoning. The method has the main advantages that the ability of understanding complex time sequence data is enhanced, the accuracy of a reasoning result is improved, and the method can adapt to a dynamically changing environment. The method is suitable for various application scenes needing sequential reasoning, such as an intelligent recommendation system, dynamic data analysis, trend prediction and the like, and has wide application prospects and market value.

Description

Technical Field

[0001] The present invention belongs to the technical field of temporal knowledge graph reasoning, and specifically is a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge. Background Art

[0002] Knowledge graphs have been widely used in natural language processing downstream tasks such as automatic question answering, dialogue or information retrieval due to their good knowledge storage and reasoning capabilities. Traditional knowledge graphs are usually static knowledge graphs, which describe facts in the form of RDF triples, generally expressed in the form of (s, r, o), where s represents the head entity, o represents the tail entity, and r represents the type of relationship between them. However, in reality, the relationship facts between entities often have a certain timeliness, and the facts will continue to change over time. Static knowledge graphs ignore the timeliness of entity relationship expression and cannot characterize the evolutionary relationship of dynamic facts. Moreover, the prediction results based on static knowledge graphs are usually not accurate enough and are limited in many tasks. In addition, time also plays a very important role in some complex prediction and deductive reasoning tasks. For this reason, Temporal Knowledge Graphs (TKGs) were proposed, which add time information to the fact representation and expand the triples of static knowledge graphs into quadruples, which are expressed in the form of (s, r, o, t), where t represents time information. The temporal knowledge graph dynamically represents facts through quadruple, accurately captures temporal semantic dependencies, expresses rich temporal semantic information, and realizes the prediction of future temporal dynamic facts, which has important application value.

[0003] Temporal Knowledge Graphs (TKG) reasoning introduces temporal information into knowledge representation and reasoning tasks, aiming to use existing historical information to infer unknown information. It can not only mine and complete missing information of dynamic events, but also predict future events, showing good application prospects in downstream tasks such as stock prediction, public opinion monitoring and transaction recommendation. However, although the existing temporal knowledge graphs are large in size, they still have problems of missing and incomplete knowledge. How to effectively use historical knowledge and temporal information in the knowledge base to infer unknown knowledge, supplement and expand knowledge graphs is still an urgent problem to be solved in temporal knowledge graph reasoning. Since temporal knowledge is only valid within a fixed time period, and new knowledge will continue to emerge over time, temporal knowledge graph reasoning is more challenging than traditional knowledge graph reasoning.

[0004] Reasoning methods for temporal knowledge graphs have attracted widespread attention in recent years. The TTransE model addresses the problem of embedding temporal information by adding time to the embedding of relations for reasoning, but does not explicitly capture entity-level temporal patterns, such as event periodicity. The RENET model decomposes the joint probability distribution of related historical events into a series of conditional probability distributions and captures certain long-term dependencies, but ignores the problem of temporal variability, resulting in inaccurate entity predictions. In response to the lack of interpretability of existing knowledge graph reasoning results, the xERTE model established the first interpretable temporal relevance attention prediction model. The model is based on a new temporal relevance attention mechanism and retains the causality of temporal multi-relational data. However, the capture of local semantic information features of entities in temporal knowledge graphs is not comprehensive enough, and the recognition of some important entities and the prediction of new entities need further in-depth research. CyGNet combines the copy mode and the generation mode, takes the historical vocabulary as the modulus, and predicts new facts in the entire entity vocabulary. However, on the one hand, it ignores the value of non-negative frequency information, and on the other hand, it does not consider the problem of temporal variability in historical development. The RE-GCN model models and learns the evolution process of a certain length of historical sequence, but ignores the time-varying problem in the temporal knowledge graph reasoning process. The CEN model uses a curriculum learning strategy to solve the evolutionary patterns of different lengths, but this method requires continuous cyclic training of the data set, which greatly reduces the time efficiency of model training. The DA-Net model first obtains repeated historical facts, and then uses an attention mechanism combined with frequency statistical information to solve the time-varying problem. However, the statistical process of historical repeated facts and the attention supervision process both take a lot of time and are therefore not time-effective. In addition, for quadruple historical information events, existing studies rarely analyze the importance weights of different relationship events in the same timestamp to the reasoning event, which leads to errors in the prediction results of new events.

[0005] In summary, the challenges of current temporal knowledge graph reasoning methods mainly include: (1) Representation of complex knowledge. In addition to learning structural information and semantic information, temporal knowledge graph reasoning also needs to model the temporal information in knowledge, consider the timeliness of knowledge, and combine the historical information of things for contextual semantic representation. How to effectively model the structural information and temporal information of historical events within the same timestamp and between different timestamps is an important challenge. (2) The importance of the historical subgraph to the target query. The multi-relational entities in the historical subgraph of each timestamp have different degrees of influence on the target entity, and some non-important entities may affect the neighborhood information. (3) Prediction of new events that have not occurred. The occurrence of some new events may be unrelated to historical events and have not occurred in historical events. It is necessary to identify whether the predicted entity is related to historical events and whether it is necessary to pay attention to historical entities. Summary of the invention

[0006] The current temporal knowledge graph reasoning ignores the contextual information in the temporal knowledge graph structure. Moreover, it does not consider the importance of different relationships within each timestamp to the prediction of future facts. In order to further comprehensively and deeply model the semantic relationship of historical facts with different relationships and the temporal logical relationship between facts, the purpose of this invention is to propose a two-stage reinforcement learning temporal knowledge graph reasoning method based on historical and temporal knowledge, which adopts two-stage context representation learning for similar events and heterogeneous events in the obtained historical subgraph temporal sequence. In the first stage, the entities with the same relationship and heterogeneous relationship in the historical subgraph are analyzed and processed, and the graph attention mechanism is used to obtain the correlation scores of entities with different relationships; in the second stage, the optimized contrastive learning method is used to further compare and learn similar facts and heterogeneous facts, obtain historical information that is truly relevant to the target query, and finally predict entities based on the two-stage learning results.

[0007] The present invention is achieved by adopting the following technical solutions:

[0008] A two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge includes the following steps:

[0009] Step 1: Divide and process the time series data set

[0010] The data set is collected to obtain a four-tuple data set with time format. Then the data set is divided into events in different time periods according to time and represented in the form of four-tuples, which are represented as (s, r, o, t), where s represents the head entity, o represents the tail entity, r represents the relationship between them, and t represents the time information.

[0011] Step 2: Build a two-stage reinforcement learning network model to achieve entity prediction by acquiring historical subgraphs, analyzing and processing historical entities, and optimizing comparative learning of similar and heterogeneous events.

[0012] 2.1. Obtain and construct historical subgraphs within the relevant time period

[0013] The relevant historical subgraph transforms the events and time context related to the query into a structured historical subgraph; since there is a certain temporal relationship between the nodes in the temporal knowledge graph, for the entity prediction query task q(s, r, ?, t n ) or (?, r, o, t n), according to the known head entity s or tail entity o, the historical subgraph of each timestamp is obtained, and the event relationship within each timestamp is multi-relational; here, the latest historical event of a specific time length is extracted to perform historical modeling on it; in order to predict the missing entities in q, the historical entities within each timestamp are obtained in the history and expressed as The relationship is represented by r i , further representing the entities and relationships of each timestamp.

[0014] 2.2 Analysis and processing of historical entities

[0015] The historical target entities and corresponding relationships within each timestamp obtained in the previous step are analyzed and processed; first, a multi-relation neighborhood aggregator is used to aggregate the entity information within the same timestamp, and then the historical information of each timestamp is input into the GRU encoder to learn the dynamic characteristics of event evolution, and then the graph attention network is used to assign attention weights to each relationship. Finally, the historical entities are divided into positively correlated entities and negatively correlated entities, and the corresponding scores are calculated.

[0016] 2.3. Optimized comparative learning of similar events and heterogeneous events

[0017] The optimized contrastive learning method is used to compare and learn the positive and negative correlation historical information obtained in the previous step to identify historical entities that are truly relevant and irrelevant to the predicted entity.

[0018] 2.4. Perform entity prediction, combine the probability obtained from the analysis and processing of relevant historical entities with the optimized comparative learning of similar and heterogeneous events to obtain the final entity probability distribution, and select the entity with the highest probability as the predicted entity.

[0019] Step 3: Train and analyze the model

[0020] First, the model parameters are initialized, and then the historical subgraph of each time period is traversed to calculate the representation of each historically related entity. Further processing is performed to obtain the positively related entity representation and the negatively related entity representation, and then the probability distribution is obtained through the Softmax function; then optimized contrast learning is performed, and the TransE encoder is used to optimize and compare the above positively related entities and negatively related entities to obtain the related entity representation and the corresponding loss function; finally, the probability distribution of the candidate entities is obtained.

[0021] The technical solution provided by the present invention has the following advantages compared with the prior art:

[0022] First, the present invention analyzes and processes the acquired historical entities, and then further identifies whether the predicted entity is related to the historical entity through optimized comparative learning, so that the model can pay more attention to useful entities. Experiments show that this method can more accurately capture the time dependency and important entities in the data, with an accuracy rate of 85.53%.

[0023] Second, the reasoning algorithm of the method of the present invention greatly improves the computing efficiency, saves computing costs, and solves the problems of slow speed and insufficient resource utilization. Compared with the previous time series knowledge graph reasoning method, the method proposed in the present invention can effectively process and analyze large-scale time series data, adapt to the growing data demand, and capture more useful entity information, improve the ability of data analysis and reasoning, support more intelligent and accurate applications, and provide a new technical reference for the field of time series knowledge reasoning.

[0024] The present invention is reasonably designed and effectively integrates historical data and time background knowledge, providing more comprehensive and accurate support for time series reasoning. The present invention is suitable for various application scenarios that require time series reasoning, such as intelligent recommendation systems, dynamic data analysis, trend prediction, etc., and has broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 Represents a two-stage reinforcement learning model based on historical and temporal background knowledge.

[0028] Figure 2 Represents a multi-relation graph attention network.

[0029] Figure 3 Represents the MRR indicator graph of the ablation experiment.

[0030] Figure 4 Represents the Hits@1 indicator graph of the ablation experiment.

[0031] Figure 5 Represents the Hits@3 indicator graph of the ablation experiment.

[0032] Figure 6Represents the Hits@10 indicator graph of the ablation experiment.

[0033] Figure 7 Represents a plot of batch training size parameter analysis.

[0034] Figure 8 Represents the Droupout value parameter analysis graph. DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0037] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] Aiming at the task of temporal knowledge graph reasoning, the present invention proposes a temporal knowledge graph reasoning method based on two-stage reinforcement learning of history and time knowledge. The method first analyzes and processes the acquired historical entities, and then further identifies whether the predicted entity is related to the historical entity through optimized comparative learning, so that the reasoning process can pay more attention to useful entities. The main advantages of the method of the present invention include: enhancing the ability to understand complex time series data, improving the accuracy of reasoning results, and being able to adapt to dynamically changing environments.

[0039] 1. A two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is as follows:

[0040] Step 1: Divide the time series data set

[0041] Before processing, the data set must be collected. By crawling the data on the website of the Comprehensive Crisis Warning System to obtain data within a period of time and extracting the corresponding structured event data set from the Global Event, Language and Tone Database, a four-tuple data set with time form is obtained. These data sets are all timely and representative. Then the data set is divided into events in different time periods according to time, and represented in the form of four-tuples, which are represented in the form of (s, r, o, t), where s represents the head entity, o represents the tail entity, r represents the relationship between them, and t represents the time information; and the training set, test set and validation set are constructed in a ratio of 8:1:1.

[0042] Step 2: Figure 1 As shown in the figure, a two-stage reinforcement learning network model is constructed to achieve entity prediction through historical subgraph acquisition, historical entity analysis and processing, and optimized comparative learning of similar and heterogeneous events. The details are as follows:

[0043] 2.1. Obtain and construct historical subgraphs within relevant time periods. Determine time window events within a certain time range to identify relevant historical events and entities; then extract relevant data and filter out historical nodes and edges related to the current query or task from the time series knowledge graph, such as past events, relationships, and attribute changes; the next step is to construct a subgraph. Based on the extracted information, a subgraph containing these historical data is constructed for in-depth analysis and reasoning.

[0044] The relevant historical subgraph converts the events and time context related to the query into a structured historical subgraph. Since there is a certain temporal relationship between the nodes in the temporal knowledge graph, for the entity prediction query task q(s, r, ?, t n ) or (?, r, o, t n ), according to the known head entity s or tail entity o, the historical subgraph of each timestamp is obtained, and the event relationship within each timestamp is multi-relational. Here, the latest historical events of a certain length of time are extracted for historical modeling. In order to predict the missing entities in q, the historical entities within each timestamp obtained in the history are represented as The relationship is represented by r i , and further represent the entities and relations of each timestamp. Then, the input historical entity representations and their corresponding relations are analyzed and processed.

[0045] 2.2. Analysis and processing of historical entities. Analyze and process the historical target entities and corresponding relationships in each timestamp obtained in the previous step. First, a multi-relation neighborhood aggregator is used to aggregate the entity information in the same timestamp. Secondly, the historical information of each timestamp is input into the GRU encoder to learn the dynamic characteristics of event evolution. Then, the graph attention network is used to assign attention weights to each relationship. Finally, the historical entities are divided into positively related entities and negatively related entities, and the corresponding scores are calculated.

[0046] a. Multi-relation neighborhood aggregator: Since the entities in each timestamp are multi-relational, the multi-relation neighborhood aggregator is first used to aggregate the neighborhood entity information features corresponding to the same time header entity, and further obtain the graph information representation of the target entity s at that moment. The specific representation is shown in formula (2):

[0047]

[0048] Among them, N trepresents the set of neighbor nodes of the target entity s at the time of relationship r and t, C s It represents the number of edges in the graph of the target entity s at a certain timestamp, and is used as a normalization factor here. o l 、h s l Respectively represent entity e o and e s The trainable embedding vector, l represents the number of aggregation layers, W r l , W o l is the learnable weight matrix, and σ(·) is the activation function RELU.

[0049] b. GRU component: As time goes by, according to the characteristics of time-varying, the entity information will be constantly updated and changed, and the corresponding frequency information will also change. Therefore, the GRU component is used to record the changes of the neighborhood entities to further enhance the learning ability. The specific expression is shown in formula (3):

[0050]

[0051] c. Graph Attention Network: Add the historical information obtained by GRU to the graph attention network to assign different weights to neighborhood entities of different relationships. Some important nodes will receive higher weights, thereby alleviating the impact of unimportant facts on neighborhood information. The attention calculation formula is shown in formula (4):

[0052]

[0053] in is a parameter vector; g represents the element multiplication sign; rk is the relationship between the target head entity and the tail entity.

[0054] d. Positive and negative correlation entity scores: The present invention divides historical entities into positively correlated entities and negatively correlated entities according to whether the relationship is the same as the query relationship. In order to calculate the correlation score of the historical entity, the frequency of each historical entity appearing in the history is first counted. The specific calculation formula is shown in formula (5):

[0055]

[0056] Among them, C t (s,r) represents the number of times the entity appears in history, Indicates the number of times an entity appears at different timestamps. Since the historical facts in each timestamp are multi-relational, entities with the same relationship as the query are given positive relevance scores, and entities with different relationships with the query are given negative relevance scores.

[0057] The specific definition of positively correlated entity score is shown in formula (6):

[0058]

[0059] Among them, tanh is the activation function, Represents the connection symbol, are trainable weights, It is a trainable bias. Adding a bias here can play a stabilizing role when dealing with missing entities of different events. This is very necessary. Then multiply the output of the linear layer by the E vector, and add the frequency C according to the previous frequency statistics. t (s,r) and entity attention scores g that are assigned different weights ijk , thereby giving related entities higher scores and obtaining more accurate attention.

[0060] The score of negatively related entities is defined as shown in formula (7):

[0061]

[0062] Similar to the definition of positive correlation score. Tanh is the activation function. Represents the connection symbol, are trainable weights, is the trainable bias. Then multiply the output of the linear layer by the E vector, and add the frequency C according to the statistical information of the previous frequency t (s,r) and entity attention scores g that are assigned different weights ijk , thereby assigning higher scores to related entities.

[0063] Finally, the positively correlated entity scores and negatively correlated entity scores are passed through the softmax function to obtain the probability of predicting the tail entity in the history set. The specific calculation formulas are shown in formula (8) and formula (9):

[0064]

[0065] P1 and P2 are the probabilities of positively correlated entities and negatively correlated entities, respectively, where the entity with the largest probability value is the most likely predicted entity.

[0066] 2.3. Optimized comparative learning of similar events and heterogeneous events. As time changes, new events may occur, which have never appeared in history or have appeared less frequently in history. This requires a more thorough understanding of historical context information. Not only should entities related to the target query be found from the positively correlated entity set in the previous step, but also entities related to the target query should be found from the negatively correlated entity set. In addition, existing models generally have data sparsity problems, resulting in poor learning effects. Therefore, the present invention adopts an optimized comparative learning method to compare and learn the positively correlated and negatively correlated historical information obtained in the previous step, and identify historical entities that are truly relevant and irrelevant to the predicted entity.

[0067] First, the TransE embedding method is used to embed the positive related entities and corresponding relationships, negative related entities and corresponding relationships of the previous module, as well as their respective frequencies in history, so that they can retain certain historical information. Here, the TransE knowledge representation embedding method is used to better model similar knowledge representations and improve the accuracy of reasoning tasks. Assume I q Embedding representation for query information:

[0068]

[0069] Here, the historical subgraph sequence associated with the candidate tail entity is defined as n is the maximum length of the sequence, and each subgraph is multi-relational. First, the query information is projected onto the plane through the TransE embedding method, and then in the process of comparative training, the positively related entity part is used as the positive sample, and the negatively related entity part is used as the negative sample. The historical events related to the candidate tail entity are added to enrich the positive and negative sample data.

[0070] Next, let the minbatch be M, and define the set with the same relationship as the query q as Q(q). By minimizing the contrast loss function, we can determine whether to focus on historical entities or new entities. The specific loss function is shown in formula (11):

[0071]

[0072] in, is the temperature parameter, which is set to 0.1 here. After enriching the data with contrast samples, Minimizing the loss function and effectively modeling the characteristics of similar samples can help better model the semantic associations between similar entities in knowledge representation learning, thereby improving the representation and reasoning capabilities of the model.

[0073] Finally, a binary classifier is used to output a scalar between 0 and 1. Here, I is set qWhen it is greater than or equal to 0.5, it means that the prediction is more inclined to positively correlated entities. q When it is less than 0.5, it means that negatively correlated entities should be paid attention to. A mask strategy is used to process the predicted entities. Vector represents that when it is related to positively correlated entities, Z t s,r (o) takes the value of 1, negatively correlated entities, Z t s,r (o) takes the value of 0. The same goes for the opposite.

[0074] 2.4. Perform entity prediction, combine the probability obtained from the analysis and processing of relevant historical entities with the optimized comparative learning of similar and heterogeneous events to obtain the final entity probability distribution, and select the one with the highest probability as the final entity prediction.

[0075] First, the positive and negative related entities of the analysis and processing module of related historical entities are embedded as head and tail entities through TransE. Secondly, the attribute features around the head and tail entities are added, and the inverse of these attribute features associated with the relationship is used as the weight to enhance the learning ability of the model. The probability of positive related entity P1 and the probability of negative related entity P2 are added and averaged to obtain the probability P t s,r Finally, P t s,r With Z t s,r (o) The vectors are combined and the entity with the highest probability is selected as the final prediction object through formula (12).

[0076] P(o|s,r,C t (s,r) )=P t s,r (o)·Z t s,r (o) (12).

[0077] Step 3: Train and analyze the model

[0078] First, initialize the network model parameters, then traverse the historical subgraphs of each time period, calculate the representation of each historically relevant entity, further process to obtain the positively relevant entity representation and the negatively relevant entity representation, and then use the Softmax function to obtain the probability distribution. Then perform optimized contrast learning, use the TransE encoder to optimize and compare the positively relevant entities and negatively relevant entities, and obtain the relevant entity representation and the corresponding loss function. Finally, calculate the final candidate entity probability distribution.

[0079] Specifically, the network model is optimized by adjusting parameters. The batch training size of the dataset can be adjusted to 1024, the embedding dimension of entities and relationships to 200, the learning rate to 0.001, the dropout to 0.5 to prevent overfitting, and the Adam optimizer is used for parameter optimization. The epoch size of the training set is set to 30, the epoch size of the test set is set to 20, and the epoch size of the validation set is set to 10; the above parameter settings are all for the optimal effect of the model. Finally, the model parameters for entity prediction are trained using the cross entropy loss function.

[0080]

[0081] 2. Datasets used by the inference algorithm of the present invention

[0082] To evaluate the reasoning method proposed in the present invention, four commonly used time series datasets are used: ICEWS, YAGO, WIKI and GDELT. ICEWS dataset: The Integrated Crisis Early Warning System (ICEWS) is a database containing events with specific timestamps, which are linked to many other entities through logical predicates. In this embodiment, the ICEWS14 and ICEWS18 datasets are used. Both datasets divide each timestamp into 24-hour intervals, where the ICEWS14 dataset collects events from January 1, 2014 to December 31, 2014, and the ICEWS18 dataset collects events from January 1, 2018 to December 31, 2018. YAGO dataset: Contains a link database that integrates data from three sources, including the Wikipedia classification system, WordNet vocabulary definitions and GeoNames, and is collected from 2013 to 2017. The WIKI dataset is a time series knowledge base extracted from the Wikipedia database, which collects data from 2008 to 2017. The time interval of the YAGO and WIKI datasets is 1 year. The GDELT dataset is taken from the Global Database of Events, Language, and Tone (GDELT), which records broadcast, print and online news in more than 100 languages. Compared with the other four datasets, it has more entities and relationships. The statistical information of various datasets is shown in Table 2.

[0083] Table 2 Experimental dataset

[0084]

[0085] 3. Introduction to the experimental evaluation indicators of the present invention

[0086] The evaluation indicators generally used for temporal knowledge graph reasoning are mean reciprocal rankings (MRR) and hit rate Hits@K of the top K results. This embodiment uses Hits@1, Hits@3, and Hits@10 as evaluation metrics. The specific calculation formula is shown in (13):

[0087]

[0088] Here S is a set of tuples, |S| is the number of tuples, and rank i It refers to the predicted ranking of the i-th tuple. The larger the index is, the better. That is, the higher the predicted ranking is, the larger the reciprocal is, and the larger the sum is.

[0089]

[0090] Return the top k results from the final recommendation list sorted by score. II(·) is an indicator function (if the condition is true, the function value is 1, otherwise it is 0). The larger the indicator, the better.

[0091] IV. Ablation Experiment of the Present Invention

[0092] In order to verify the importance of each module of the TSECNET model, an ablation experiment is carried out while keeping the experimental settings unchanged. Variants are created by adjusting different components of the model. Here, the ICEWS18 and YAGO datasets are selected for experimental performance research. The experimental results are shown in Table 3 below.

[0093] Table 3 Ablation experiment results

[0094]

[0095] In order to verify the effectiveness of the two-stage module, the two-stage modules are eliminated and verified respectively. As shown in Table 3, TSECNET-HEP means that only the first-stage representation learning part is retained and the optimization contrast learning part is removed. TSECNET-OCL means that only the second-stage optimization contrast learning part is retained and the first-stage representation learning part is removed. It can be seen from the table that TSECNET-HEP and TSECNET-OCL have improved in all indicators. However, it is worth noting that in the ICEWS18 dataset, the H@3 index of TSECNET-HEP is 4.09% lower than that of TSECNET, and TSECNET-OCL is 4.04% lower than that of TSECNET.

[0096] The fact shows that although the indicators of SECNET-HEP and TSECNET-OCL have been improved, the effect is not as high as that of TSECNET, which shows that the combination of the two-stage modules is crucial and the effect is more obvious. By adjusting the use of model components, it is concluded that the combination of the two-stage modules plays an important role in the data set. The specific comparison results are as follows Figure 3-Figure 6 As shown in the figure, through the analysis of the four indicators of MRR, Hits@1, Hits@3, and Hits@10, as the number of Epoch training increases, the indicator results of TSECNET-HEP, TSECNET-OCL, and TSECNET are constantly improving, and TSECNET has the best effect in this process. When the Epoch is 30, the result is the best, proving that the TSECNET model has the best effect. Here, the ICEWS18 dataset is selected as the result comparison analysis.

[0097] 4. Performance comparison between the method of the present invention and other mainstream algorithms.

[0098] In order to verify the experimental effect of this reasoning method, it is compared with 13 baseline models, which can be divided into two categories of models. One is the static knowledge graph reasoning method, including TransE, DistMult, CompIEX, R-GCN, ConvE; the other is the temporal knowledge graph reasoning method, including HyTE, TTransE, TeMp, RE-NET, RE-GCN, CyGNet, EvoKG, HIP, CEN. Tables 4 and 5 respectively give the entity prediction results of the above different models on five datasets. The experimental results show that the TSECNET model has been improved on all five datasets, among which the improvement of various indicators on the WIKI and YAGO datasets is particularly obvious. On the WIKI dataset, Hits@1 has the highest improvement over the baseline model, which is 13 percentage points higher. On the YAGO dataset, the Hits@1 data indicator has increased by 8 percentage points, which is the most obvious improvement. The improvement is small on the ICEWS18, ICEWS14 and GDELT datasets, because the ICEWS and GDELT datasets are event-based datasets, the relationship networks they contain are relatively complex, the data volume is huge, the processing is slow, and the probability of new events is high. The time granularity of the WIKI and YAGO datasets is years, there are fewer relationship types, and most of the relationships remain unchanged, and the proportion of repetitive events is high. When predicting entities, the factual relationships that can be relied on are relatively stable, which makes the model improvement effect obvious.

[0099] Table 4 Experimental results of the proposed method on ICEWS18, ICEWS14 and GDELT datasets

[0100]

[0101] Table 5 Experimental results of the proposed method in WIKI and YAGO datasets

[0102]

[0103] V. Analysis of specific application results

[0104] Here, through three queries about national cooperation, we can see that the model of the present invention has obtained relevant historical entities, and after processing, the entity with the highest probability is selected as the final prediction. The predicted results are consistent with the actual results in the future. This proves the accuracy of the model and illustrates the effectiveness and value of the model in event prediction.

[0105] 6. Parameter Sensitivity Analysis

[0106] In order to evaluate the sensitivity of the inference method of the present invention to parameters, an experimental comparison of the batch size and Dropout value is conducted on the YAGO dataset. The batch training size is set to {64, 128, 256, 512, 1024} and the Dropout value is set to {0.1, 0.3, 0.5, 0.7, 0.9} to analyze the sensitivity of these two parameters. The results are shown in Figure 2. Figure 7 , Figure 8 As shown in the figure, it can be seen that when the batch training size and Dropout value are (1024, 0.5) on the YAGO dataset, MRR is 85.53%, Hits@1 is 85.28%, Hits@3 is 85.54%, and Hits@10 is 85.84%, which is the best effect. This proves that the TSECNET model is sensitive to batch training size and Dropout.

[0107] In summary, the temporal knowledge graph reasoning method described in the present invention significantly improves the ability to reason and predict in a dynamic data environment. By systematically constructing and analyzing historical subgraphs, combining temporal information and knowledge graph technology, the present invention not only optimizes the reasoning process, but also enhances the accuracy and reliability of the results. This method shows great potential in various application scenarios, including but not limited to intelligent search, data analysis, and intelligent recommendation systems.

[0108] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge, characterized by: The steps include: Step 1: Divide and process the time series data set The data set is collected to obtain a four-tuple data set with time format. Then the data set is divided into events in different time periods according to time and represented in the form of four-tuples, which are represented as (s, r, o, t), where s represents the head entity, o represents the tail entity, r represents the relationship between them, and t represents the time information; Step 2: Build a two-stage reinforcement learning network model to achieve entity prediction by acquiring historical subgraphs, analyzing and processing historical entities, and optimizing comparative learning of similar and heterogeneous events.

2. According to claim 1, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: Step 2 is as follows: 2.

1. Obtain and construct historical subgraphs within the relevant time period The relevant historical subgraph transforms the events and time context related to the query into a structured historical subgraph; since there is a certain temporal relationship between the nodes in the temporal knowledge graph, for the entity prediction query task q(s, r, ?, t n ) or (?, r, o, t n ), according to the known head entity s or tail entity o, the historical subgraph of each timestamp is obtained, and the event relationship within each timestamp is multi-relational; here, the latest historical event of a specific time length is extracted to perform historical modeling on it; in order to predict the missing entities in q, the historical entities within each timestamp obtained in the history are expressed as The relationship is represented by r i , further representing the entities and relations of each timestamp; 2.2 Analysis and processing of historical entities Analyze and process the historical target entities and corresponding relations in each timestamp obtained in the previous step; first, use a multi-relation neighborhood aggregator to aggregate the entity information in the same timestamp, then input the historical information of each timestamp into the GRU encoder to learn the dynamic characteristics of event evolution, then assign attention weights to each relationship through the graph attention network, and finally divide the historical entities into positively related entities and negatively related entities, and calculate the corresponding scores; 2.

3. Optimized comparative learning of similar events and heterogeneous events Adopting the optimized contrastive learning method, the positive and negative correlation historical information obtained in the previous step are contrasted and learned to identify the historical entities that are truly relevant and irrelevant to the predicted entity; 2.

4. Perform entity prediction, combine the probability obtained from the analysis and processing of relevant historical entities with the optimized comparative learning of similar and heterogeneous events to obtain the final entity probability distribution, and select the one with the highest probability as the final entity prediction.

3. According to claim 2, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: Step 2.2 is as follows: a. Multi-relation neighborhood aggregator: Since the entities in each timestamp are multi-relational, the multi-relation neighborhood aggregator is first used to aggregate the neighborhood entity information features corresponding to the same time header entity, and further obtain the graph information representation of the target entity s at that moment, which is specifically expressed as formula (2): where N t represents the set of neighbor nodes of the target entity s at the time of relationship r and t, C s represents the number of edges in the graph of the target entity s at a certain timestamp; h o l 、h s l Respectively represent entity e o and e s The trainable embedding vector, l represents the number of aggregation layers, W r l , W o l is the learnable weight matrix, σ(·) is the activation function RELU; b. GRU component: As time goes by, according to the characteristics of time-varying, the entity information will be constantly updated and changed, and the corresponding frequency information will also change. The GRU component is used to record the changes of the neighborhood entities to further enhance the learning ability. The specific expression is as follows: c. Graph Attention Network: Add the historical information obtained by GRU to the graph attention network to assign different weights to neighborhood entities with different relationships; The attention calculation formula is as follows: in is the parameter vector; g represents the element multiplication sign; r k is the relationship between the target head entity and the tail entity; d. Positive and negative related entity scores: According to whether the relationship is the same as the query relationship, the historical entities are divided into positively related entities and negatively related entities, and the frequency of each historical entity appearing in the history is counted. The specific calculation formula is as follows: Among them, C t (s,r) represents the number of times the entity appears in history, Indicates the number of times an entity appears at different timestamps; entities with the same relationship as the query are given a positive correlation score, and entities with different relationships are given a negative correlation score; The definition of the positively correlated entity score is as follows: The score for negatively related entities is defined as formula (7): Among them, tanh is the activation function, Represents the connection symbol, are trainable weights, is the trainable bias; Finally, the positively correlated entity scores and negatively correlated entity scores are passed through the softmax function to obtain the probability of predicting the tail entity in the history set. The specific calculation formulas are as follows: P1 and P2 are the probabilities of positively correlated entities and negatively correlated entities, respectively, where the entity with the largest probability value is the most likely predicted entity.

4. According to claim 2, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized in that: Step 2.3 is as follows: First, the TransE embedding method is used to embed the positive related entities and corresponding relationships, negative related entities and corresponding relationships, and their respective frequencies in history in the previous module, so that they can retain certain historical information; let I q Embedding representation for query information: The historical subgraph sequence associated with the candidate tail entity is defined as n is the maximum length of the sequence, and each subgraph is multi-relational. First, the query information is projected onto the plane through the TransE embedding method. Then, in the process of comparative training, the positively related entity part is used as the positive sample, and the negatively related entity part is used as the negative sample. The historical events related to the candidate tail entity are added to enrich the positive and negative sample data. Next, we define as follows: let the minbatch be M, define the set with the same relationship as the query q as Q(q); determine whether to focus on historical entities or new entities by minimizing the contrast loss function; the specific loss function is expressed as formula (11): in, is the temperature parameter; after enriching the data with contrast samples, Minimize the loss function; Finally, a binary classifier is used to output a scalar between 0 and 1, setting I q When it is greater than or equal to 0.5, it means that the prediction is more inclined to positively correlated entities. q When it is less than 0.5, it means that negatively related entities should be paid attention to; the predicted entities are processed through a mask strategy, using Vector representation, when associated with positively correlated entities, Z t s,r The value of (o) is 1 when it is associated with a negatively correlated entity. t s,r The value of (o) is 0.

5. According to claim 2, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: Step 2.4 is as follows: First, the positively correlated entities and negatively correlated entities processed by the related historical entity analysis are embedded into the head and tail entities through TransE. Secondly, the attribute features around the head and tail entities are added, and the inverse of these attribute features associated with the relationship is used as the weight to enhance the learning ability of the model; the probability of positively correlated entities P1 and the probability of negatively correlated entities P2 are added and averaged to obtain the probability P t s,r Finally, P t s,r With Z t s,r (o) The vectors are combined and the entity with the highest probability is selected as the final prediction object through formula (12); P(o|s,r,C t (s,r) )=P t s,r (o)·Z t s,r (o) (12).

6. According to claim 1, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: In step 1, the dataset is divided into training set, test set and validation set in a ratio of 8:1:

1.

7. According to claim 6, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: Also includes: Step 3: Train and analyze the model First, the network model parameters are initialized, and then the historical subgraph of each time period is traversed and processed to calculate the representation of each historically related entity. Further processing is performed to obtain the positively related entity representation and the negatively related entity representation, and then the probability distribution is obtained through the Softmax function; then optimized comparative learning is performed, and the TransE encoder is used to optimize and compare the above positively related entities and negatively related entities to obtain the related entity representation and the corresponding loss function; finally, the probability distribution of the candidate entities is calculated.

8. According to claim 7, a two-stage reinforcement learning temporal knowledge graph reasoning method based on history and time knowledge is characterized by: Adjust parameters to achieve the best results for the network model; By adjusting the batch training size of the dataset to 1024, the embedding dimension of entities and relations to 200, the learning rate to 0.001, the dropout to 0.5 to prevent overfitting, and the Adam optimizer to optimize the parameters; the epoch size of the training set is set to 30, the epoch size of the test set is set to 20, and the epoch size of the validation set is set to 10; Finally, the model parameters for entity prediction are trained through the cross entropy loss function;

Images