A distributed attention-based temporal knowledge graph reasoning method

Abstract

The present invention relates to a distributed attention temporal knowledge graph reasoning method, which accurately expresses the time evolution characteristics and the structural dependency of the temporal subgraph by reorganizing the temporal knowledge graph in time series, and then extracts historical repeated facts and historical frequency information based on the sparse matrix storing the historical subgraph information, and gives the query facts the first layer of distributed attention to the historical repeated facts based on the attention mechanism, and then gives the second layer of attention rewards and penalties to the first layer of attention scores by capturing the latest changes in the historical frequency information, so that the attention is more adapted to the time-varying characteristics, and finally uses the scores of the two layers of attention to infer and predict future events. Compared with the traditional prediction method, the present invention gives learnable distributed attention on different historical timestamps, rather than obtaining a fixed embedding representation through an encoder, so that the model has better ability to solve time-varying problems.

Description

Technical Field

[0001] The present invention relates to the field of temporal knowledge graph technology, and in particular to a distributed attention temporal knowledge graph reasoning method, and a computational model based on distributed attention modeling of temporal knowledge subgraph sequences to solve time-varying problems. Background Art

[0002] The temporal knowledge graph adds a time dimension to the traditional knowledge graph. It consists of entities (nodes), relationships (edges) and timestamps, and is usually presented in the form of a knowledge quadruple (s, p, o, t), where s represents the subject entity, p represents the relationship, o represents the object entity, and t represents the time information. For example, (Golden State Warriors, championship, NBA, 2018) means that the Golden State Warriors won the NBA (National Basketball Association) championship in 2018. The time dimension greatly enriches the expressive power of the knowledge graph in real scenarios.

[0003] In recent years, the temporal knowledge graph has been vigorously developed, including its wide application in crisis warning, stock prediction and other fields. However, various problems have also emerged, such as the incompleteness of the knowledge graph itself, which lacks some valuable facts. In addition, the temporal knowledge graph is actually a sequence of knowledge subgraphs in time order, and each timestamp subgraph has independent entity, relationship and structural information. The problem of insufficient modeling of the temporal evolution pattern of the temporal knowledge graph has also greatly affected the accuracy of temporal knowledge graph reasoning. The temporal knowledge graph reasoning method based on distributed attention gives distributed attention to the historical information at different timestamps instead of learning to obtain the unique embedding representation of historical entities, thereby paying full attention to the distributed information on different temporal subgraphs.

[0004] Temporal knowledge graph reasoning is essentially about predicting missing facts at specific timestamps. In particular, if only information at historical timestamps is used, then the task of predicting future events is more realistic. Some static reasoning methods completely ignore the time dimension of temporal knowledge graphs, including embedding-based methods such as TransE, RotatE, etc.; reinforcement learning-based methods such as DeepPath, MINERVA, etc.; graph convolutional network-based methods such as R-GCN, Comp-GCN, etc.

[0005] The study of attention mechanism originated from cognitive psychology and neuroscience. The human eye focuses on the target position that needs attention through rapid scanning, pays more attention to the area, extracts the necessary detailed information, and pays less attention to other areas, suppressing information overload. This is an ability for humans to use limited resources to quickly filter out valuable information from a large amount of information. The so-called attention mechanism is a mechanism that focuses on local information. For example, a certain image area in an image will often change with the change of tasks.

[0006] The "attention mechanism" actually aims to apply human perception and attention behavior to machines, so that machines can learn to perceive the important and unimportant parts of the data. The attention mechanism of deep learning simulates this process. For the data input into the neural network, through learning, it learns to pay attention to the key information content in the input information, and then pays attention to or uses it in the subsequent prediction or reasoning process. The attention mechanism can be broadly understood as a vector of importance weights: using the input embedding vector (embedding), calculate the correlation between the embedding vector and other embedding vectors, and use the sum of their values ​​as the approximate value of the output.

[0007] CN110472068A discloses a big data processing method, device and medium based on heterogeneous distributed knowledge graph. The method includes: constructing a node table and a relationship table of a heterogeneous distributed knowledge graph according to the data structure of a heterogeneous distributed knowledge base; determining a graph computing scenario according to a graph computing request, determining the type and / or attribute of nodes required for the graph computing scenario, and the type and / or attribute of edges; extracting at least one computing node corresponding to the graph computing scenario from the node table and the relationship table; filtering out node data corresponding to at least one computing node from the heterogeneous distributed knowledge graph; performing data processing on the filtered node data to obtain a data processing result based on the heterogeneous distributed knowledge graph. This embodiment provides an efficient data processing method for heterogeneous distributed knowledge graphs based on node tables and relationship tables.

[0008] CN112395423A discloses a recursive time series knowledge graph completion method and device, wherein the method includes: obtaining the corresponding static knowledge graph according to the acquired time series knowledge graph, and obtaining updated features through embedding learning of the static knowledge graph and features; adopting a recursive method, taking the sub-knowledge graph of the first timestamp as the starting point, taking the sub-knowledge graph, features and embedding learning parameters of the current timestamp as the input of embedding learning, obtaining updated embedding learning parameters and features, and taking the updated embedding learning parameters and features as the embedding learning parameters and features of the sub-knowledge graph of the next adjacent timestamp, until traversing the sub-knowledge graph sequence of all timestamps; then, performing fact prediction on the sub-knowledge graph of each timestamp.

[0009] CN112364108A discloses a method for completing a temporal knowledge graph based on a spatiotemporal architecture, which is characterized in that it includes the following steps: dividing the temporal knowledge graph to be completed into several static knowledge sets according to the time annotation of its knowledge, and constructing several knowledge networks through the knowledge in each set to obtain several snapshots; constructing a multifaceted graph attention network, and inputting the snapshots into the multifaceted graph attention network to obtain the static embedding representation of the entity under each snapshot; constructing an adaptive temporal attention mechanism, and obtaining the final embedding representation of the entity according to the static embedding representation of the entity and using the adaptive temporal attention mechanism; calculating the confidence of the knowledge in the temporal knowledge graph to be completed through the final embedding representation of the entity, and predicting the missing content in the temporal knowledge graph to be completed through the confidence.

[0010] Some recent works focus on predicting future events in temporal knowledge graphs. For example, RE-NET models the occurrence of facts as a conditional probability distribution of history; CyGNet regards entities appearing in historical timestamps as abstract summaries of future facts; HIP network transmits historical information from the perspectives of time, structure, and repetition for prediction; xERTE generates query subgraphs with a certain number of hops by constructing reasoning graphs; CluSTeR and TITer both use reinforcement learning to determine evolutionary patterns in query paths; RE-GCN learns entity representations containing evolutionary information by modeling subgraph sequences of recent historical timestamps.

[0011] However, the above-mentioned temporal knowledge graph reasoning methods are limited by the encoder-decoder architecture, and the time-varying problems in the temporal knowledge graph reasoning process are ignored. These methods tend to learn to obtain fixed entity embedding representations, which not only cannot capture new historical information in a timely manner, but also compress the dynamic evolution of historical information into an unchanging low-dimensional vector, which will inevitably cause incomplete distributed representation information under different historical timestamps. CEN attempts to solve the time-varying problem in an online learning setting, but is still limited to constantly adjusting the fixed representation vector of the wired length, rather than adopting a distributed modeling strategy, which is bound to cause distributed information loss.

[0012] In addition, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making the present invention, but due to space limitations, not all details and contents are listed in detail. However, this does not mean that the present invention does not have the characteristics of these prior arts. On the contrary, the present invention already has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art to the background technology. Summary of the invention

[0013] In view of the shortcomings of the prior art, the present invention provides a temporal knowledge graph reasoning method, system, electronic device and storage medium, aiming to solve at least one or more technical problems existing in the prior art.

[0014] To achieve the above objectives, the present invention provides a distributed attention temporal knowledge graph reasoning method, comprising the following parts:

[0015] The temporal knowledge graph is time-serialized, and the distribution of historical timestamp subgraphs is stored in a sparse matrix to accurately express the structural dependencies of the temporal subgraph sequence.

[0016] The initial first layer of attention of the fact of predicting timestamp to the historical repeated facts is constructed using the attention mechanism, which is responsible for capturing the traditional invariant features in the historical information.

[0017] The second layer of attention is constructed based on the statistics of historical frequency information evolving with timestamps. The scores of the first layer of attention are adjusted, and attention rewards and penalties are given to historical repeated facts and non-repeated facts respectively. It is responsible for processing the time-varying characteristics of historical information.

[0018] A flexible parameter training strategy is adopted to initialize the embedding vectors of entities and relations, as well as learnable parameters such as query transformation matrix and key-value transformation matrix. A non-learning multiple mapping strategy is used to represent time information, find the optimal model and complete the reasoning and prediction task of the time series knowledge graph.

[0019] Preferably, in the present invention, the temporal knowledge graph is time-serialized, and the distribution of the historical timestamp subgraph is stored through a sparse matrix to accurately express the structural dependency of the temporal subgraph sequence, including the following functions:

[0020] Dividing the temporal knowledge graph into a sequence of knowledge subgraphs in chronological order enables the representation of the temporal knowledge graph to complete the dimensionality reduction from quadruple to triple, and helps the internal time dependency of the temporal knowledge graph.

[0021] The sparse matrix records the historical patterns of predicted events in different periods under similar scenarios, and converts the time consumption of historical search into quantitative space consumption.

[0022] Preferably, in the present invention, the attention mechanism is used to construct the initial first-layer attention of the fact of predicting the timestamp to the historical repeated facts, which is responsible for capturing the traditional invariant features in the historical information, and its learning process includes the following steps:

[0023] In order to process query facts in batches, it is necessary to perform mask filling on the historical repeated fact sequences of each query in the same batch. Specifically, the sequence that is less than the maximum length in the batch is filled with globally non-existent relation-entity pairs, and then a mask matrix is ​​generated through identification tags, which are not involved in the attention operation, thereby greatly reducing the computational complexity.

[0024] Calculate the multi-head attention of the query matrix (Q) on the key-value matrix (K, V) after mask filling. Specifically, the scaled dot product attention operation is used. The weight matrix is ​​obtained by dot product of the query matrix (Q) and the key matrix (K) and divided by the scaling factor, and then clicked with the value matrix (V) to obtain the value matrix representing the attention. Each dimensional vector in the value matrix is ​​the initial distributed attention for each historical repeated fact.

[0025] Deep semantic information is supplemented by a fully connected feedforward neural network with many hidden units. In addition, the outputs of the multi-head attention and feedforward neural networks are layer normalized and residually connected to prevent gradient disappearance during training and accelerate convergence.

[0026] Preferably, in the present invention, the second layer of attention is constructed based on the statistics of historical frequency information evolving with timestamps, the score of the first layer of attention is adjusted, and attention rewards and penalties are given to historical repeated facts and non-repeated facts respectively, which is responsible for processing the time-varying characteristics of historical information, and its specific contents may include:

[0027] By timely superimposing the frequency statistical information in the new historical information, and then expressing the recent development and changes of knowledge through the changes in the historical frequency information, the initial attention of the first layer of attention is adjusted.

[0028] Based on the frequency statistics after the new history is updated, an attention penalty is imposed on facts that have not appeared in the history. Specifically, a large negative value is added to the first-level attention score.

[0029] Based on the frequency statistics of the new history update, the facts that appeared in the history are given attention rewards. Specifically, the updated frequency is input into the Softmax function to obtain a positive value between 0 and 1, which is added to the first layer of attention scores.

[0030] Preferably, the present invention provides a distributed attention temporal knowledge graph reasoning system, comprising:

[0031] An arrangement unit, configured to reorganize the time series knowledge graph in time series in the order of timestamps, and store the distribution of historical timestamp subgraphs through a sparse matrix;

[0032] A processing unit configured to construct a fact of a prediction timestamp using an attention mechanism and to give an initial first layer of attention to historical repeated facts;

[0033] an adjustment unit configured to construct a second layer of attention based on statistics of historical frequency information evolving with timestamps, and adjust the score of the first layer of attention according to the latest changes in knowledge;

[0034] The training unit is configured to adopt a parameter training strategy and use a multi-classification task based on cross entropy loss for model training.

[0035] Preferably, the present invention provides an electronic device for temporal knowledge graph reasoning, comprising:

[0036] one or more processors;

[0037] a memory for storing one or more computer programs;

[0038] When the one or more computer programs are executed by one or more processors, the one or more processors implement the temporal knowledge graph reasoning method described in the present invention.

[0039] Preferably, the present invention provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute the temporal knowledge graph reasoning method as described in the present invention.

[0040] Preferably, in the present invention, a flexible parameter training strategy is used for representation learning. For the embedding of time information, a non-training multiple mapping relationship is used to improve the representation efficiency and reduce the training time. The embedding representation of entities and relationships is initialized by vectors, and error control is set to ensure the accuracy of embedding. In addition, it is also necessary to initialize the query transformation matrix, key value transformation matrix, linear transformation coefficient bias and other learnable parameters, and innovatively regard the representation learning process as a multi-classification task with the size of the entity set as the number of classifications. The cross entropy loss function and AMSGrad optimizer are used to learn the parameters of the multi-classification task. Finally, the optimal parameter values ​​of the model are estimated by observing the predictive effectiveness of the facts on the verification sample, and then the optimal model is obtained to improve the prediction accuracy of time series reasoning.

[0041] Preferably, the present invention relates to a temporal knowledge graph reasoning model based on distributed attention, based on which different attention can be distributedly given to historical information according to their importance, so that queries can selectively select appropriate historical information for reference according to the different roles played by different historical timestamps to complete more accurate predictions. Compared with the traditional future event prediction model based on the encoder-decoder architecture, learnable distributed attention is given to different historical timestamps, rather than obtaining a fixed embedding for simple representation through the encoder, so that the model has better ability to solve time-varying problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a structural principle diagram of a reasoning model based on a distributed attention temporal knowledge graph according to a preferred embodiment of the present invention;

[0044] Figure 2 It is a structural schematic diagram of a reasoning system based on a distributed attention temporal knowledge graph according to a preferred embodiment of the present invention;

[0045] Figure 3 It is a structural schematic diagram of an electronic device based on a distributed attention temporal knowledge graph according to a preferred embodiment of the present invention.

[0046] Reference numerals list

[0047] 1: Arrangement unit; 2: Processing unit; 3: Adjustment unit; 4: Training unit; 10: Electronic device; 101: Processor; 102: Memory; 103: Communication bus; 104: Communication interface; 105: Random access memory; 106: Cache memory; 107: Storage unit; 108: Program module; 109: Utility; 110: External device; 111: Display; 112: Network adapter. DETAILED DESCRIPTION

[0048] The following is a detailed description with reference to the accompanying drawings.

[0049] Example 1

[0050] The present invention provides a temporal knowledge graph reasoning model and reasoning method based on distributed attention, and the reasoning method may include the following steps:

[0051] The temporal knowledge graph is time-serialized and a sparse matrix is ​​used to store the temporal subgraph distribution.

[0052] The attention mechanism is used to give the query facts an initial first-layer attention to the historical repeated facts.

[0053] The second layer of attention is constructed based on the changes in historical frequency statistics, and the scores of the first layer of attention are adjusted according to the latest changes in knowledge.

[0054] A parameter training strategy is adopted, and a multi-classification task based on cross entropy loss is used for model training.

[0055] According to a preferred embodiment, Figure 1 As shown, the step of time-seriesizing the time series knowledge graph may specifically include: In , assuming that ε is a set of entities of size N, is a set of relations of size P, is a timestamp set of size T, then the method of time serialization is to divide it into a sequence of time-series subgraphs in the order of timestamps Each subgraph is a complete static knowledge graph. n ), the temporal knowledge graph reasoning task can be expressed as a historical subgraph sequence Complete the incomplete fact (s, p, ?, t n ) or (?, p, o, t n ), where ? represents the missing object entity or the missing subject entity respectively.

[0056] Furthermore, the step of using a sparse matrix method to store the distribution of the historical subgraph may specifically include: setting the matrix size to two-dimensional N*P×N, and recording the repeated and non-repeated distribution information of each query fact at each historical timestamp. Since a single fact has only a specific time range and may only occur once in a single timestamp, the distribution matrix of each timestamp is actually a sparse matrix with only a small number of 1s and a large number of 0s. Using a sparse matrix storage method can greatly improve space utilization.

[0057] Specifically, a high-dimensional matrix can be represented by three one-dimensional vectors, namely, the value vector records the values ​​of non-zero elements in the two-dimensional matrix; the horizontal coordinate vector records the horizontal coordinate positions of non-zero elements in the two-dimensional matrix in turn; the vertical coordinate vector records the vertical coordinate positions of non-zero elements in the two-dimensional matrix in turn. Furthermore, if (s, p, o) has appeared in the t-th historical timestamp, then in the t-th sparse matrix, the element corresponding to the horizontal coordinate s*p and the vertical coordinate o will be represented as 1, otherwise it will be 0. Based on the sparse matrix, historical repeated facts {(s, p, o0, t0),...,(s, p, o0, t1),...,(s, p, o1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t21, t22, t23, t24, t25, t26, t27, t28, t29, t30, t31, t32, t33, t34, t35, t36, t37, t38, t39, t39, t31, t31, t32, t33, t34, t35, t36, t37, t38, t39, i , t i ),...,(s,p,o n-1 , t n-1 )} and historical frequency information is an N-dimensional vector, each dimension of which represents the frequency of occurrence of the corresponding entity in history, {t i |0≤i≤n-1} represents the current query timestamp t n The set of all historical timestamps of i} represents a collection of historical repeated entities.

[0058] According to a preferred embodiment, Figure 1 As shown in the figure, the step of using the attention mechanism to make the query fact give the initial first-level attention to the historical repeated facts may specifically include: for a query (s, p, ?, t n ), replace s, p, o i and t n It is represented as the embedding representation of the corresponding query entity, historical repeated entity, relationship and timestamp, then it can be expressed by Q = W q [s, p] generates the query matrix Q, through K = W k [p,o i ],V=W v [p,o i ] Generate key-value matrices K and V, where W q , W k and W v are coefficient matrices.

[0059] Furthermore, based on the attention mechanism, the query fact is given a learnable first-layer initial attention to the historical repeated facts, which is expressed as:

[0060]

[0061] Among them, d k is a scaling factor to prevent the Softmax function from vanishing gradients. Multi-head attention is to assign multiple learnable W to the operation at the same time based on self-attention. q , W k and W v Parameter matrix, so that the model can learn multiple semantic effects from the perspective of multiple subspaces. The feedforward neural network uses a fully connected network FFN(x)=W1(RELU(W2x)) with a hidden layer of 2048, where x is the output of multi-head attention, W1 and W2 are coefficient matrices, and RELU is used as the activation function.

[0062] In particular, the outputs of multi-head attention and feedforward neural networks are processed by layer normalization and residual connection to accelerate the convergence rate. Specifically, layer normalization is to scale the vector content to between 0 and 1, while residual connection is to add the input vector content of the network to the output vector to preserve certain input features.

[0063] According to a preferred embodiment, Figure 1 As shown in FIG. 1 , the second layer of attention is constructed based on the change of historical frequency statistical information, and the step of adjusting the score of the first layer of attention according to the latest change of knowledge may specifically include: assuming that the output vector of the first layer of attention is y, then firstly, through linear transformation and inputting it into the hyperbolic tangent function, the score of the first layer of attention is regularized to between -1 and 1 s t =tanh(W t [y,t n ]+b t ), then the score interval here is 2.

[0064] Then, the second layer of attention gives attention penalty to facts that have not appeared in history according to the latest historical frequency information. Specifically, a large negative value is added to its score, defined as For facts that have appeared in history, the most recently updated historical frequency statistics (denoted as ) gives attention reward. Specifically, a positive value is given based on the frequency statistics information on the first-level attention score. Among them, the base value δ is 2, which is the output score range of the first layer of attention, so that both layers of attention can play a role. The final inference prediction score can be expressed as

[0065] According to a preferred embodiment, Figure 1 As shown in the figure, the steps of adopting a flexible parameter training strategy and innovatively adopting a multi-classification task based on cross entropy loss to train the model may specifically include: the model represents and embeds the time information with a sequential relationship according to the method of non-parametric multiple mapping, that is, for a sequential time series such as {2014 / 1 / 1, 2014 / 1 / 2, ..., 2014 / 12 / 30, 2014 / 12 / 31}, first randomly initialize the embedding vector of the first timestamp 2014 / 1 / 1 to e, then the low-dimensional vector embedding of the sequential time series obtained by non-parametric multiple mapping can be expressed as T = {e, 2e, 3e, 4e, ..., Te}, where T is the number of timestamps that can be divided in a specific scenario. Then the time embedding T does not participate in the training of the model, and then directly gives the time information a sequential dependency, which not only reduces the computational complexity of the model, but also facilitates the temporal information modeling of the knowledge graph.

[0066] In particular, cross entropy represents two probability distributions p and q, where p represents the true distribution and q represents the non-true distribution. In the same set of events, the non-true distribution q is used to represent the average number of bits required for an event to occur. Cross entropy is generally used as a loss function for multi-classification problems, and is usually used as a distance measure between the predicted value and the true label value.

[0067] Furthermore, we regard the reasoning completion of the time series knowledge graph as a multi-classification task. A multi-classification task is a classification learning task with more than 2 classifications. For example, for the query (s, p, ?, t n ), a suitable entity needs to be selected from the candidate entity set to answer (complete) the missing object entity. The number of categories is the entity set size N, and the final prediction score is a multi-hot vector with a dimension size of N. The model will select the fact with the highest score in the vector as the result of future event prediction: o = argmax o∈ε (p(o|s,p,t n )). The cross entropy loss function for multi-classification tasks can be expressed as in, Represents the t-th temporal subgraph G t The lth benchmark entity in (i.e. the correct result of the prediction), Expressed as is the probability of the jth entity (entity numbered j) in the entity set ε, and then continuously reduces the size of the global loss function on the verification sample, and finally finds the appropriate parameters corresponding to the optimal model.

[0068] Example 2

[0069] like Figure 2As shown, based on the above-mentioned distributed attention temporal knowledge graph reasoning model and reasoning method, the present invention provides a distributed attention temporal knowledge graph reasoning system.

[0070] Specifically, the temporal knowledge graph reasoning system of the present invention may include:

[0071] Arrangement unit 1 is configured to time-serialize the temporal knowledge graph and use a sparse matrix to store the temporal subgraph distribution.

[0072] Processing unit 2 is configured to use an attention mechanism to give the query fact an initial first-level attention to the historical repeated fact.

[0073] The adjustment unit 3 is configured to construct a second-level attention based on the changes in historical frequency statistical information, and adjust the score of the first-level attention according to the latest changes in knowledge.

[0074] The training unit 4 is configured to adopt a parameter training strategy and use a multi-classification task based on cross entropy loss for model training.

[0075] According to a preferred embodiment, in this embodiment, the arrangement unit 1 is configured to perform the following steps: In , assuming that ε is a set of entities of size N, is a set of relations of size P, is a timestamp set of size T, then the method of time serialization is to divide it into a sequence of time-series subgraphs in the order of timestamps Each subgraph is a complete static knowledge graph. n ), the temporal knowledge graph reasoning task can be expressed as a historical subgraph sequence Complete the incomplete fact (s, p, ?, t n ) or (?, p, o, t n ), where ? represents the missing object entity or the missing subject entity respectively.

[0076] Furthermore, the arrangement unit 1 can store the distribution of the historical subgraph based on the sparse matrix method. Specifically, it includes: setting the matrix size to two-dimensional N*P×N, and recording the repeated and non-repeated distribution information of each query fact at each historical timestamp. Since a single fact has only a specific time range and may only occur once in a single timestamp, the distribution matrix of each timestamp is actually a sparse matrix with only a small number of 1s and a large number of 0s. The use of sparse matrix storage method can greatly improve space utilization.

[0077] Specifically, a high-dimensional matrix can be represented by three one-dimensional vectors, namely, the value vector records the values ​​of non-zero elements in the two-dimensional matrix; the horizontal coordinate vector records the horizontal coordinate positions of non-zero elements in the two-dimensional matrix in turn; the vertical coordinate vector records the vertical coordinate positions of non-zero elements in the two-dimensional matrix in turn. Furthermore, if (s, p, o) has appeared in the t-th historical timestamp, then in the t-th sparse matrix, the horizontal coordinate is s*p, and the element corresponding to the vertical coordinate i will be represented as 1, otherwise it will be 0. Based on the sparse matrix, historical repeated facts {(s, p, o0, t0),...,(s, p, o0, t1),...,(s, p, o1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t21, t22, t23, t24, t25, t26, t27, t28, t29, t30, t31, t32, t33, t34, t35, t36, t37, t38, t39, t40, t39, t41, t42, t43, t44, t45, t46, t47, t48, t49, t5 i , t i ),...,(s,p,o n-1 , t n-1 )} and historical frequency information is an N-dimensional vector, each dimension of which represents the frequency of occurrence of the corresponding entity in history, {t i |0≤i≤n-1} represents the current query timestamp t n The set of all historical timestamps of i} represents a collection of historical repeated entities.

[0078] According to a preferred implementation, in this embodiment, the processing unit 2 is configured to perform the following steps: using the attention mechanism to make the query fact give the initial first-level attention to the historical repeated facts. Specifically, for a query (s, p, ?, t n ), replace s, p, o i and t n It is represented as the embedding representation of the corresponding query entity, historical repeated entity, relationship and timestamp, then it can be expressed by Q = W q [s, p] generates the query matrix Q, through K = W k [p, oi], V = W v [p,o i ] Generate key-value matrices K and V, where W q , W k and W v are coefficient matrices.

[0079] Furthermore, the processing unit 2 gives the query fact a learnable first-level initial attention to the historical repeated fact based on the attention mechanism, which can be expressed as:

[0080]

[0081] Among them, d k is a scaling factor to prevent the Softmax function from vanishing gradients. Multi-head attention is to assign multiple learnable W to the operation at the same time based on self-attention. q , W kand W v Parameter matrix, so that the model can learn multiple semantic effects from the perspective of multiple subspaces. The feedforward neural network uses a fully connected network FFN(x)=W1(RELU(W2x)) with a hidden layer of 2048, where x is the output of the multi-head attention, W1 and W2 are coefficient matrices, and the activation function uses RELU. In particular, the outputs of the multi-head attention and feedforward neural networks are both processed by layer normalization and residual connection to speed up the convergence rate. Specifically, layer normalization scales the vector content to between 0 and 1, while residual connection adds the input vector content of the network to the output vector to preserve certain input features.

[0082] According to a preferred implementation, in this embodiment, the adjustment unit 3 is configured to perform the following steps: construct the second layer of attention based on the change of historical frequency statistical information, and adjust the score of the first layer of attention according to the latest change of knowledge. Specifically, it may include: assuming that the output vector of the first layer of attention is y, then firstly, through linear transformation and inputting it into the hyperbolic tangent function, the score of the first layer of attention is regularized to between -1 and 1 s t =tanh(W t [y,t n ]+b t ), then the score interval here is 2. Then, the second level of attention gives attention penalty to facts that have not appeared in history based on the latest historical frequency information. Specifically, a large negative value is added to its score, defined as For facts that have appeared in history, the most recently updated historical frequency statistics (denoted as ) gives attention reward. Specifically, a positive value is given based on the frequency statistics information on the first-level attention score. Among them, the base value δ is 2, which is the output score range of the first layer of attention, so that both layers of attention can play a role. The final inference prediction score can be expressed as

[0083] According to a preferred embodiment, in this embodiment, the training unit 4 is configured to perform the following steps: adopt a flexible parameter training strategy, and adopt a multi-classification task based on cross entropy loss to train the model. Specifically, it may include: the model represents and embeds the time information with a sequential relationship according to the method of non-parametric multiple mapping, that is, for a sequential time series such as {2014 / 1 / 1, 2014 / 1 / 2, ..., 2014 / 12 / 30, 2014 / 12 / 31}, first randomly initialize the embedding vector of the first timestamp 2014 / 1 / 1 to e, then the low-dimensional vector embedding of the sequential time series obtained by non-parametric multiple mapping can be expressed as T = {e, 2e, 3e, 4e, ..., Te}, where T is the number of timestamps that can be divided in a specific scenario. Then the time embedding T does not participate in the training of the model, and then directly gives the time information a sequential dependency, which not only reduces the computational complexity of the model, but also facilitates the temporal information modeling of the knowledge graph.

[0084] In particular, cross entropy represents two probability distributions p and q, where p represents the true distribution and q represents the non-true distribution. In the same set of events, the non-true distribution q is used to represent the average number of bits required for an event to occur. Cross entropy is generally used as a loss function for multi-classification problems, and is usually used as a distance measure between the predicted value and the true label value.

[0085] Furthermore, we regard the reasoning completion of the temporal knowledge graph as a multi-classification task, which is a classification learning task with more than 2 classifications. For example, for the query (s, p, ?, t n ), a suitable entity needs to be selected from the candidate entity set to answer (complete) the missing object entity, then the number of categories is the entity set size N, and the final prediction score is a multi-hot vector with a dimension size of N. The model will select the fact with the highest score in the vector as the result of future event prediction: i = argmax o∈ε (p(o|s,p,t n )). The cross entropy loss function for multi-classification tasks can be expressed as in, Represents the t-th temporal subgraph G t The lth benchmark entity in (i.e. the correct result of the prediction), Expressed as is the probability of the jth entity (entity numbered j) in the entity set ε, and then continuously reduces the size of the global loss function on the verification sample, and finally finds the appropriate parameters corresponding to the optimal model.

[0086] It should be understood that the number and functions of the modules in this embodiment are only for ease of explanation and should not be regarded as bringing any limitation to the functions and scope of use of the embodiments of the present invention. In some other optional ways, a larger number of modules or units can be set according to specific subdivision steps to implement the various functions and / or methods described in this embodiment.

[0087] Example 3

[0088] Figure 3 An electronic device 10 is shown for implementing the distributed attention temporal knowledge graph reasoning method described in the above-mentioned embodiment 1. Figure 3 The electronic device 10 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0089] Specifically, Figure 3 As shown, the electronic device 10 is in the form of a general-purpose computer device, and the electronic device may include:

[0090] One or more processors 101 .

[0091] The memory 102 is used to store one or more computer programs.

[0092] The communication bus 103 is used to connect different system components (including the processor 101 and the memory 102).

[0093] According to a preferred embodiment, the processor 101 executes the various functions and / or methods described in the embodiments of the present invention by running one or more computer programs stored in the memory 102, especially implementing the distributed attention temporal knowledge graph reasoning method as described in the present invention.

[0094] According to a preferred embodiment, the electronic device 10 may include a variety of computer system readable media, which may be any available media that can be accessed by the electronic device 10, including volatile and non-volatile media, removable and non-removable media.

[0095] According to a preferred implementation, the processor 101 includes but is not limited to a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Control Unit), and a SOC (System on Chip).

[0096] According to a preferred embodiment, the memory 102 includes but is not limited to a computer system readable medium in the form of a volatile memory, or other removable / non-removable and non-volatile computer system storage medium. Figure 3As shown, the memory 102 is, for example, a random access memory (RAM) 105 and / or a cache memory 106 .

[0097] According to a preferred embodiment, the communication bus 103 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. Specifically, these architectures include but are not limited to an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0098] According to a preferred embodiment, the storage unit 107 can be used to read and write non-removable, non-volatile magnetic media. Further, a disk drive for reading and writing a removable non-volatile disk (such as a floppy disk) and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media) can be provided. Each drive can be connected to the communication bus 103 via one or more data medium interfaces.

[0099] According to a preferred embodiment, the memory 102 may include at least one program product. The program product has at least one set of program modules 108 or at least one utility 109. These program modules 108 may be configured in the memory 102 to execute the functions and / or methods described in various embodiments of the present invention.

[0100] According to a preferred embodiment, the electronic device 10 can be connected to at least one external device 110 (such as a keyboard, a display 111, etc.) through a communication interface 104, or any device that enables the electronic device 70 to communicate with at least one other computing device (such as a network card, a modem, etc.).

[0101] According to a preferred embodiment, the electronic device 10 can communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) via a network adapter 112 .

[0102] According to a preferred embodiment, the network adapter 112 communicates with other modules of the electronic device 10 via the communication bus 103. Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 10, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0103] Example 4

[0104] The present invention also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute a distributed attention temporal knowledge graph reasoning method described in Example 1.

[0105] According to a preferred embodiment, the computer storage medium of Example 4 of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium includes, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.

[0106] According to a preferred embodiment, more specific examples of computer-readable storage media include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, an apparatus, or a device.

[0107] According to a preferred embodiment, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] According to a preferred embodiment, the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable or RF, etc., or any suitable combination of the above.

[0109] According to a preferred embodiment, the computer program code for performing the operation of the embodiment of the present invention can be written in one or more programming languages ​​or a combination thereof, and the programming language includes an object-oriented programming language, such as python, Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0110] In order to clearly introduce the technical solution, a specific implementation method of a distributed attention temporal knowledge graph reasoning method provided by an embodiment of the present invention is given in combination with a specific application scenario, as follows: The occurrence of certain events in the real world often has a certain historical reproducibility. If a decision is made on the query "What disease was a resident diagnosed with in March 2022?", then historical information needs to be extracted first. For example, the resident was diagnosed with pneumonia in May 2020, accompanied by complications such as cough and diarrhea. Due to the immaturity of the treatment level, the resident's condition could not be improved. Subsequently, in December 2020, he was diagnosed with a serious complication, sepsis, and still had symptoms such as cough. However, with the improvement of people's pneumonia treatment level, the resident's condition improved, and then in October 2021, he was only diagnosed with mild symptoms such as cough. Then the extracted historical repeated facts can be mainly represented as {pneumonia, sepsis, cough}. In the historical frequency information, before the resident's condition improved (which can be represented as October 2021), the frequency statistics of pneumonia and cough were similar, but higher than the frequency information of sepsis. It can be found that repeated facts play different roles in different time stages, and thus different answers can be given to queries at different timestamps. Time-varying means that the emphasis of the query on historical information (or the role of historical information at different timestamps on the query) changes dynamically over time. For the query (a certain resident, diagnosed,?, March 2022), the first layer of attention first applies initial attention to the embedding of the historical repeated facts {pneumonia, sepsis, cough} through the entity embedding of the resident based on the attention mechanism. This layer of attention starts learning from more distant historical information and does not consider the differences between different historical timestamps. It is mainly to capture invariant historical features. For example, if the resident has been diagnosed with pneumonia and cough at the same time for a long time, then there is a complication relationship between the two diseases. The second layer of attention adjusts the first layer of attention based on the recent changes in historical frequency information to capture time-varying features. For example, the attention of the same query to historical repeated facts focuses on changes caused by improved medical standards. Specifically, after October 2021, the frequency statistics of mild symptoms of cough will gradually exceed pneumonia and significantly exceed the frequency information of sepsis over time. Based on the changing frequency statistics, a certain attention reward is applied to the cough entity, and its impact on the prediction of illness in March 2022 is highlighted. For entities that have not appeared in history, that is, diseases that the patient has no history of for a long time, a certain attention penalty is given, that is, they are not included in the final prediction. Based on the modeling of unchanging and time-varying historical features, the query "What disease was the resident diagnosed with in March 2022?" can eventually be predicted with the "cough" entity.

[0111] It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention specification and its drawings are illustrative and do not constitute a limitation on the claims. The scope of protection of the present invention is defined by the claims and their equivalents. The present invention specification contains multiple inventive concepts, such as "preferably", "according to a preferred embodiment" or "optionally", all of which indicate that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application based on each inventive concept.

Claims

1. A distributed attention temporal knowledge graph reasoning method, characterized in that: include: The time series knowledge graph is reorganized in time series in the order of timestamps, and the distribution of historical timestamp subgraphs is stored in a sparse matrix; Adopt the attention mechanism to construct the facts of the predicted timestamp and give the initial first layer of attention to the historical repeated facts; The second layer of attention is constructed based on the statistics of historical frequency information evolving with timestamps, and the score of the first layer of attention is adjusted according to the latest changes in knowledge; A parameter training strategy is adopted, and a multi-classification task based on cross entropy loss is used for model training; The method of using the attention mechanism to construct the fact of predicting the timestamp and giving the initial first-level attention to the historical repeated facts includes: Mask and fill the historical repeated fact sequences of each query in the same batch; Calculate the multi-head attention of the query matrix Q on the key-value matrix after mask padding; The deep semantic information is supplemented by a fully connected feedforward neural network containing multiple hidden units; Perform layer normalization and residual connection operations on the output of multi-head attention and feedforward neural networks; The second-level attention is constructed based on the statistics of the historical frequency information evolving with the timestamp, and the score of the first-level attention is adjusted according to the latest changes in knowledge, including: Overlay frequency statistics in new historical information; Represent the development and changes of recent knowledge according to the changes in historical frequency information to adjust the initial attention of the first layer of attention; Based on the frequency statistics after the new history is updated, attention penalties are imposed on facts that have not appeared in history; Based on the frequency statistics after the new history is updated, attention rewards are given to facts that have appeared in history.

2. The distributed attention temporal knowledge graph reasoning method according to claim 1 is characterized in that: The method of reorganizing the time series knowledge graph in the order of timestamps and storing the distribution of the historical timestamp subgraphs through a sparse matrix includes: Divide the temporal knowledge graph into a sequence of knowledge subgraphs in chronological order, so that the representation of the temporal knowledge graph can be reduced from quadruples to triples; The historical patterns of predicted events in different periods under similar scenarios are recorded through sparse matrices, and the time consumption of historical search is converted into quantitative space consumption.

3. The distributed attention temporal knowledge graph reasoning method according to claim 2 is characterized in that: The parameter training strategy and the model training using a multi-classification task based on cross entropy loss include: Initialize at least one or more learnable parameters of a query transformation matrix, a key-value transformation matrix, and a linear transformation coefficient bias; The reasoning completion of the temporal knowledge graph is regarded as a multi-classification task with the entity set size as the number of categories; The cross entropy loss function and AMSGrad optimizer are used to learn the parameters of the multi-classification task to determine the facts with the highest scores and use them as the results for predicting future events.

4. The distributed attention temporal knowledge graph reasoning method according to claim 1 is characterized in that: The mask filling of the historical repeated fact sequence of each query in the same batch includes: Sequences less than the maximum length of the batch are padded with globally non-appearing relation-entity pairs to generate a mask matrix by identifying tokens and exclude them from the attention operation.

5. The distributed attention temporal knowledge graph reasoning method according to claim 4 is characterized in that: Calculating the multi-head attention of the query matrix Q on the key-value matrix after mask filling includes: The scaled dot product attention operation is used to obtain the weight matrix by performing the dot product between the query matrix Q and the key matrix K and dividing it by the scaling factor. Perform a dot product with the value matrix V to obtain a value matrix representing attention, where each dimensional vector in the value matrix is ​​the initial distributed attention for each historical repeated fact.

6. A distributed attention temporal knowledge graph reasoning system, characterized in that: include: An arrangement unit (1) is configured to reorganize the time series knowledge graph in a time-series order and store the distribution of the historical time stamp subgraphs through a sparse matrix; A processing unit (2) is configured to construct a fact of a prediction timestamp using an attention mechanism and to give an initial first layer of attention to historical repeated facts; An adjustment unit (3), configured to construct a second layer of attention based on statistics of historical frequency information evolving with timestamps, and adjust the score of the first layer of attention according to the latest changes in knowledge; A training unit (4) is configured to adopt a parameter training strategy and to perform model training using a multi-classification task based on a cross entropy loss; The method of using the attention mechanism to construct the fact of predicting the timestamp and giving the initial first-level attention to the historical repeated facts includes: Mask and fill the historical repeated fact sequences of each query in the same batch; Calculate the multi-head attention of the query matrix Q on the key-value matrix after mask padding; The deep semantic information is supplemented by a fully connected feedforward neural network containing multiple hidden units; Perform layer normalization and residual connection operations on the output of multi-head attention and feedforward neural networks; The second-level attention is constructed based on the statistics of the historical frequency information evolving with the timestamp, and the score of the first-level attention is adjusted according to the latest changes in knowledge, including: Overlay frequency statistics in new historical information; Represent the development and changes of recent knowledge according to the changes in historical frequency information to adjust the initial attention of the first layer of attention; Based on the frequency statistics after the new history is updated, attention penalties are imposed on facts that have not appeared in history; Based on the frequency statistics after the new history is updated, attention rewards are given to facts that have appeared in history.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors (101); A memory (102) for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors (101), the one or more processors (101) implement the distributed attention temporal knowledge graph reasoning method as described in any one of claims 1 to 5.

8. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the distributed attention temporal knowledge graph reasoning method as described in any one of claims 1 to 5.

Images