Time sequence knowledge graph prediction method based on continuous features and large model

Abstract

The invention particularly relates to a time sequence mapping knowledge domain prediction method based on continuous features and a large model. The method comprises the following steps: acquiring mapping knowledge domain data; constructing a knowledge graph prediction model, wherein the knowledge graph prediction model comprises a historical coding module, a continuous time neural network module, a feature fusion module and a training loss module; the historical coding module outputs a historical entity matrix, a historical relationship matrix and a historical matrix based on the knowledge graph data; the continuous time neural network module obtains a time continuous entity matrix and a time continuous relation matrix according to the historical entity matrix and the historical relation matrix; the feature fusion module generates probabilities of all candidate entities based on the time continuous entity matrix and the time continuous relation matrix; and the training loss module calculates final loss and generates probabilities of all final candidate entities so as to predict the time sequence knowledge graph. According to the method, continuous time dynamic modeling and multi-scale feature fusion are realized, and the prediction precision and generalization ability of the time sequence knowledge graph are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge graphs, and specifically relates to a temporal knowledge graph prediction method based on continuous features and large models. Background Art

[0002] Temporal knowledge graph prediction is a dynamic reasoning technology that integrates the time dimension and the structure of the knowledge graph, aiming to predict the occurrence of future events or relationships by modeling entities, relationships, and their evolution laws in the continuous time domain. The current mainstream methods are based on the Transformer architecture, using its self-attention mechanism to capture long-range dependencies in time series, and realizing the dynamic evolution analysis of the graph through entity relationship modeling.

[0003] The prior art can complete basic temporal pattern recognition, such as event relevance inference based on discrete timestamps, and representation learning of static structural relationships between entities. The current technology has the following problems:

[0004] 1. Lack of continuity of dynamic features in the continuous time domain

[0005] Existing models approximate the continuous time domain through discrete time intervals (such as fixed time windows), resulting in the loss of fine-grained time information. For example, repeated events densely occurring on the continuous time axis (such as high-frequency trading or periodic monitoring data) cannot be accurately modeled, and only the existence of events can be judged while ignoring their occurrence frequency and time correlation, resulting in prediction results deviating from the true temporal evolution law.

[0006] 2. Insufficient ability to represent time heterogeneity

[0007] Existing methods are difficult to uniformly process multi-scale time events. For example, the heterogeneous time patterns of second-level high-frequency events (such as real-time sensor data) and cross-year low-frequency events (such as historical trend analysis) are not effectively modeled, resulting in the model's inability to capture the correlation between events at different time scales and limiting the full-scale representation ability of the graph evolution pattern.

[0008] 3. Defects in co-modeling of structural features and semantic features

[0009] Current technologies usually independently process the structural connections (such as entity topological relationships) and semantic associations (such as event context meanings) of the graph, lacking a deep fusion mechanism. For example, in a relationship scenario where the entity structure is tight but the semantic difference is significant (such as the high-frequency interaction and low semantic correlation of users in a social network), existing models are prone to generate embedding representation biases, resulting in a decline in prediction generalization ability.

[0010] The above-mentioned defects directly affect the accuracy and applicability of the prediction of temporal knowledge graphs. Discretized time modeling leads to the loss of details of high-frequency events, insufficient processing of time heterogeneity causes the failure of cross-scale event association, and the lack of collaborative modeling of structure and semantics restricts the accurate characterization of complex relationships. The purpose of this invention is to systematically solve these problems and improve the technical performance of temporal knowledge graphs in continuous-time domain modeling, multi-scale event analysis, and structure and semantics fusion. Summary of the Invention

[0011] In view of the above technical problems of the lack of continuity of dynamic features in the continuous-time domain, insufficient ability to represent time heterogeneity, and defects in collaborative modeling of structural features and semantic features, a prediction method for temporal knowledge graphs based on continuous features and large models is provided. This invention mainly introduces innovative technologies such as continuous-time Transformer, feature fusion module, and training loss module, significantly improving the model's ability in simulating dynamic features, processing time heterogeneity, fusing structural and semantic features, and enhancing interpretability and robustness, laying a foundation for the wide application of temporal knowledge graphs.

[0012] The technical means adopted in this invention are as follows:

[0013] A prediction method for temporal knowledge graphs based on continuous features and large models, the steps include:

[0014] S101. Obtain knowledge graph data, where the knowledge graph data includes entity matrix, relationship matrix, and time information;

[0015] S102. Construct a knowledge graph prediction model, where the knowledge graph prediction model includes a historical encoding module, a continuous-time neural network module, a feature fusion module, and a training loss module connected in sequence;

[0016] S103. The historical encoding module outputs a historical entity matrix, a historical relationship matrix, and a historical matrix based on the knowledge graph data;

[0017] S104. The continuous-time neural network module includes a multi-head attention layer and a continuous-time layer. The multi-head attention layer obtains the multi-head attention output in continuous time according to the historical entity matrix and the historical relationship matrix; the continuous-time layer obtains a time-continuous entity matrix and a time-continuous relationship matrix according to the multi-head attention output in continuous time, the historical entity matrix, and the historical relationship matrix;

[0018] S105. The feature fusion module generates the probabilities of all candidate entities based on the time-continuous entity matrix, the time-continuous relationship matrix, and the time information;

[0019] S106. The training loss module generates updated predicted probabilities by using a large model to evaluate the probabilities of all candidate entities based on the knowledge graph data, constructs a loss function, calculates the final loss, and optimizes the knowledge graph prediction model with the final loss to generate the probabilities of all candidate entities finally for predicting the temporal knowledge graph.

[0020] Further, the working process of the historical encoding module includes:

[0021] Mapping the entity matrix to a historical key matrix and a historical value matrix, and mapping the relationship matrix to a historical query matrix;

[0022] Calculating the hidden layer state of each layer according to the historical key matrix, historical value matrix and historical query matrix:

[0023]

[0024] where, Q hem is the historical query matrix, K hem is the historical key matrix, V hem is the historical value matrix, is the hidden layer state at time t of the (l - 1)th layer, is the hidden layer state at time t of the lth layer;

[0025] Stacking the hidden layer states of each layer in multiple layers to generate the hidden layer states of the last layer corresponding to all times, and constructing a historical matrix according to the hidden layer states of the last layer corresponding to all times:

[0026]

[0027] where, H is the historical matrix, is the hidden layer state at time t of the Nth layer;

[0028] Generating a historical entity matrix and a historical relationship matrix from the historical matrix through an update function:

[0029] E hem , R hem = Update(H)

[0030] where, E hem is the historical entity matrix, R hem is the historical relationship matrix, and H is the historical matrix.

[0031] Further, the working process of the multi - head attention layer includes:

[0032] Performing a linear transformation on the historical entity matrix to obtain a time - continuous key matrix, a time - continuous value matrix and a time - continuous query matrix;

[0033] Perform a weighted transformation on the time - continuous query matrix, time - continuous key matrix, and time - continuous value matrix to generate a query matrix, a key matrix, and a value matrix corresponding to each attention head:

[0034]

[0035] Among them, Q ctf is the time - continuous query matrix, K ctf is the time - continuous key matrix, V ctf is the time - continuous value matrix, is the weight matrix corresponding to the query matrix, is the weight matrix corresponding to the key matrix, is the weight matrix corresponding to the value matrix, is the query matrix corresponding to each attention head, is the key matrix corresponding to each attention head, is the value matrix corresponding to each attention head;

[0036] Unfold the query matrix corresponding to each attention head by time steps to obtain query vectors at each time step. According to the query vectors at each time step, use natural cubic splines for modeling to generate query vectors with continuous - time representation;

[0037] Unfold the key matrix and value matrix corresponding to each attention head by time steps to obtain the key matrix and value matrix at each time step. Use neural differential equations to perform continuous - time modeling on the key vectors and value vectors to generate key vectors with continuous - time representation and value vectors with continuous - time representation:

[0038]

[0039] Among them, k i (t) is the key vector with continuous - time representation, v i (t) is the value vector with continuous - time representation, t is the current time point, t i is the initial time point, θ k is the parameter of the neural differential equation for the key vector, θ v is the parameter of the neural differential equation for the value vector, k i (t i ) is the key vector of the i - th attention head at the initial time, v i (t i ) is the value vector of the i - th attention head at the initial time;

[0040] Calculate the aggregated representation of the key matrix and value matrix based on the query vector with continuous - time representation and the key vector with continuous - time representation:

[0041]

[0042] Among them, δ i (t) is the aggregated representation of the key matrix and the value matrix, and q i (t) is the query vector in continuous-time representation, and k i (t) is the key vector in continuous-time representation;

[0043] Integrate the value vector in continuous-time representation over the time interval to generate the expected value of the value matrix:

[0044]

[0045] Among them, is the expected value of the value matrix, t is the current time point, and t i is the initial time point, and v i (t) is the value vector in continuous-time representation;

[0046] According to the aggregation of the key matrix and the value matrix and the expected value of the value matrix, use the normalization method to obtain the output of each attention head:

[0047]

[0048] Among them, head i (t) is the output of each attention head, and δ i (t) is the aggregated representation of the key matrix and the value matrix, and d k is the dimension of each attention head, is the expected value of the value matrix, and M represents the number of attention heads in the multi-head attention mechanism;

[0049] Concatenate the features of all attention heads, and linearly transform the concatenated features of the attention heads through the weight matrix to generate the multi-head attention output in continuous time:

[0050] CT-MHA(Q ctf , K ctf , V ctf , t) = W o ·concat(head1(t), …, head M (t))

[0051] Among them, Q ctf is the query matrix, K ctf is the key matrix, V ctf is the value matrix, head i (t) is the output of each attention head, and W o is the first weight matrix.

[0052] Furthermore, the working process of the continuous-time layer includes:

[0053] Concatenate the historical entity matrix and the historical relationship matrix to generate an input matrix. Based on the input matrix and time information, use the continuous-time multi-head attention mechanism to generate the multi-head attention output:

[0054]

[0055] Among them, is the multi-head attention output, X l is the input matrix, t is the time information, and x l (t) is a closed continuous differentiable time function;

[0056] Use a feed-forward network to perform a non-linear transformation on the multi-head attention output to obtain the output of each layer of the transformer. The calculation formula for the output of each layer of the transformer is:

[0057]

[0058] Among them, z l (t) is the output of the l-th layer of the transformer, is the multi-head attention output;

[0059] Stack the outputs of each layer of the transformer multiple times to generate the output of the last layer of the transformer. Decompose the output of the last layer of the transformer into a time-continuous entity matrix and a time-continuous relationship matrix:

[0060]

[0061] Among them, is the output of the last layer of the transformer, E ctf is the time-continuous entity matrix, and R ctf is the time-continuous relationship matrix.

[0062] Furthermore, the working process of the fusion module includes:

[0063] Based on the time information, construct a time decoder. Through the time decoder, generate time encoding, periodic encoding, and aperiodic encoding. The calculation formula for the time encoding is:

[0064] T t = LN(TFormer(EL(t)))

[0065] Among them, t is the time information, T t is the time encoding,

[0066] The calculation formula for the periodic encoding is:

[0067]

[0068] Among them, T t is the time encoding, is a periodic encoding, Λ(·) is a periodic function implemented using a sine function, and α p is the first ratio parameter, and β p is the second ratio parameter,

[0069] The formula for the aperiodic encoding is:

[0070]

[0071] where T t is the time encoding, is the aperiodic encoding, and α np is the third ratio parameter, and β np is the fourth ratio parameter;

[0072] Calculate the probabilities of all candidate entities based on the time - continuous entity matrix, time - continuous relationship matrix, periodic encoding, and aperiodic encoding:

[0073]

[0074] where λ is the probability of all candidate entities, s ctf is the query entity embedding, s ctf ∈ time - continuous entity matrix, r ctf is the query relationship embedding, r ctf ∈ time - continuous relationship matrix, h t is the historical feature, h t ∈ historical matrix, is the aperiodic encoding, is the periodic encoding, o i is the candidate query entity embedding, o i ∈ time - continuous entity matrix, and W λ is the second weight matrix.

[0075] Furthermore, the training loss module uses a large - model to evaluate the probabilities of all candidate entities and generates updated predicted probabilities, including:

[0076] Extract the static knowledge graph corresponding to each moment according to the knowledge graph data;

[0077] Encode the static knowledge graph to generate a static structure entity matrix and a static structure relationship matrix:

[0078] E sta , R sta =RotatE(KG sta )

[0079] where E sta is the static structure entity matrix, and R stais the static structure relationship matrix, KG sta is the static knowledge graph;

[0080] According to the static structure entity matrix and the static structure relationship matrix, use the large model to evaluate and generate the evaluation result of the large model:

[0081]

[0082] Among them, is the evaluation result of the large model, is the static structure entity embedding, ∈ static structure entity matrix, is the static structure relationship embedding, ∈ static structure relationship matrix, is the candidate static structure entity embedding, ∈ static structure entity matrix, Prompt is the prompt embedding;

[0083] According to the evaluation result of the large model and the probabilities of all the candidate entities, update the probabilities of all the candidate entities to generate the updated predicted probabilities:

[0084]

[0085] Among them, is the updated predicted probability, is the evaluation result of the large model, λ is the probabilities of all the candidate entities, is the first hyperparameter.

[0086] Furthermore, the construction process of the loss function is as follows:

[0087] Based on the static knowledge graph, extract the vectorized head entity, head relationship and tail entity of the large model. According to the vectorized head entity, head relationship and tail entity of the large model, calculate the similarity predicted by the large model:

[0088]

[0089] Among them, μ is the similarity predicted by the large model, is the head entity, is the head relationship, is the tail entity;

[0090] According to the semantic similarity and the updated predicted probability, construct the loss function:

[0091]

[0092] Among them, is the loss function, λ true is the true value, $\hat{p}$ is the updated predicted probability, $\mu$ is the similarity predicted by the large model, and $\alpha$ is the second hyperparameter.

[0093] Compared with the prior art, the present invention has the following advantages:

[0094] 1. Through the Continuous-Time Multi-Head Attention mechanism (CT-MHA) and Neural Differential Equations, the present invention can accurately model dynamic features in the continuous time domain, capture fine-grained time information of high-frequency events, effectively solve the problem of missing continuity of dynamic features, enable the model to more accurately simulate the dynamic evolution law of the knowledge graph, and improve the accuracy of prediction results.

[0095] 2. Through the time encoder, the present invention uniformly processes high-frequency events and low-frequency events at the second level, solves the problem of insufficient time heterogeneity representation ability. The model can effectively capture the correlation between events at different time scales, support the evolution analysis of the full-scale time series knowledge graph, and significantly improve the generalization ability and applicability of the model.

[0096] 3. Through the Large Language Model (LLM) and the adapter mechanism, the present invention deeply integrates the structural features and semantic features of the knowledge graph, solves the problem of the defect in co-modeling of structural features and semantic features in the prior art. The model can more accurately understand the semantic differences in complex relationship scenarios, improve the prediction generalization ability, and provide more reliable support for practical applications.

[0097] For the above reasons, the present invention can be widely promoted in the fields of knowledge graphs and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0099] Figure 1 It is a flowchart of a method for predicting a time series knowledge graph based on continuous features and a large model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0100] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0101] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0102] The overall model of the present invention is divided into four parts: a historical encoding module, a continuous-time Transformer module, a feature fusion module, and a training loss module. First, the historical encoding module uses the Transformer architecture to extract structural features from snapshots at each time point. Through this step, the model can capture the static structural information of the knowledge graph at different time points, providing a basis for subsequent dynamic evolution simulation. Secondly, the continuous-time Transformer module combines differential equations with the Transformer to simulate the dynamic evolution of the knowledge graph in the continuous-time domain. By integrating differential equations into the Transformer, the model can capture continuous-time features and effectively represent dynamic changes. This mechanism ensures that the model can achieve smooth time transitions when simulating the evolution of the knowledge graph over time, thus solving the problem of maintaining the continuity of dynamic features. Subsequently, the feature fusion module uses the Transformer to encode time information to address the heterogeneity of time representation. By integrating the encoded time features into the decoding process, the model can take into account changes at different time scales, thus effectively capturing and utilizing time variations. This step improves the generalization ability of the model under different time representations, enabling it to more accurately process complex time information. The training loss module uses a fine-tuned language model to update the probability distribution and performs semantic verification using a black box language model. By adopting a collaborative method that fuses semantic and structural information, the module can perform predictive inference. During the loss calculation process, a semantic verification mechanism is introduced to ensure the consistency of feature integration and reduce structural and semantic mismatches. This step enhances the model's ability to fuse structural and semantic features, improving the prediction and inference accuracy of the model.

[0103] As Figure 1 shown, the present invention provides a temporal knowledge graph prediction method based on continuous features and large models. The steps are specifically as follows:

[0104] S101. Obtain knowledge graph data, where the knowledge graph data includes an entity matrix, a relationship matrix, and time information.

[0105] S102. Construct a knowledge graph prediction model, where the knowledge graph prediction model includes a historical encoding module, a continuous time neural network module, a feature fusion module, and a training loss module connected in sequence.

[0106] S103. The historical encoding module outputs a historical entity matrix, a historical relationship matrix, and a historical matrix based on the knowledge graph data.

[0107] Specifically, the historical encoding module aims to capture time-dependent relationships and generate historical feature representations for knowledge graph entities and relationships. This module effectively processes time information using a Transformer-based architecture. At each time step, the module takes the entities and relationships that appear at that specific timestamp as input.

[0108] Map the entity matrix to a key matrix and a value matrix, and map the relationship matrix to a query matrix. In the Transformer framework, an entity s ∈ E is assigned to the key matrix K hem and the value matrix V hem , and a relationship o ∈ ε is mapped to the query matrix Q hem . This design uses the attention mechanism of the Transformer to model the interaction between entities and relationships.

[0109] Calculate the hidden layer state of each layer based on the key matrix, the value matrix, and the query matrix:

[0110]

[0111] where Q hem is the query matrix, K hem is the key matrix, V hem is the value matrix, is the hidden layer state of the (l - 1)-th layer at time t, is the hidden layer state of the l-th layer at time t. In the formula, FFN(·) represents a feed-forward neural network, and MHA(·) represents multi-head attention. The historical encoding module consists of N hidden layers, and the output of the N layers is the final result.

[0112] Stack the hidden layer states of each layer multiple times to generate the hidden layer state of the last layer corresponding to all time steps. Based on the hidden layer state of the last layer corresponding to all time steps, construct the historical matrix:

[0113]

[0114] where H is the historical matrix, is the hidden layer state of the N-th layer at time t.

[0115] Generate a historical entity matrix and a historical relationship matrix from the historical matrix through an update function:

[0116] E hem , R hem = Update(H)

[0117] Among them, E hem is the historical entity matrix, R hem is the historical relationship matrix, and H is the historical matrix.

[0118] S104. The continuous-time neural network module includes a multi-head attention layer and a continuous-time layer. The multi-head attention layer obtains the multi-head attention output in continuous time based on the historical entity matrix and the historical relationship matrix; the continuous-time layer obtains a time-continuous entity matrix and a time-continuous relationship matrix based on the multi-head attention output in continuous time, the historical entity matrix, and the historical relationship matrix.

[0119] The continuous-time neural network module is a continuous-time transformer specifically designed to simulate the dynamic and continuous evolution of a temporal knowledge graph. By integrating differential equations into its architecture, CTFormer captures the temporal continuity of entity and relationship changes, effectively modeling the smooth transitions and interactions that occur over time. This design enables the module to encode the evolving nature of the temporal knowledge graph while preserving structural and temporal dependencies. The continuous-time neural network module mainly relies on the continuous-time Transformer, which is a variant of the Transformer based on differential equations. The core component of the continuous-time transformer is the continuous-time multi-head attention mechanism.

[0120] Continuous-time multi-head attention mechanism. The continuous-time multi-head attention mechanism is the core component of the continuous-time transformer and the foundation of the continuous-time neural network module. This mechanism extends the traditional multi-head attention by introducing neural differential equations to simulate temporal dynamics. By embedding differential equations into the attention mechanism, the continuous-time multi-head attention mechanism enables the model to capture the continuous evolution of entity and relationship representations over time. This design allows the attention weights to consider temporal continuity, ensuring that the learned representations reflect the smooth and dynamic changes within the temporal knowledge graph.

[0121] Specifically, the workflow of the multi-head attention layer is as follows:

[0122] Perform a linear transformation on the historical entity matrix to obtain a time-continuous key matrix, a time-continuous value matrix, and a time-continuous query matrix;

[0123] Perform a weighted transformation on the time-continuous query matrix, the time-continuous key matrix, and the time-continuous value matrix to generate the query matrix, key matrix, and value matrix corresponding to each attention head:

[0124]

[0125] Among them, Q ctf is a time - continuous query matrix, K ctf is a time - continuous key matrix, V ctf is a time - continuous value matrix, is the weight matrix corresponding to the query matrix, is the weight matrix corresponding to the key matrix, is the weight matrix corresponding to the value matrix, is the query matrix corresponding to each attention head, is the key matrix corresponding to each attention head, is the value matrix corresponding to each attention head;

[0126] Unroll the query matrix corresponding to each attention head by time step to obtain the query vector at each time step. According to the query vector at each time step, use natural cubic spline modeling to generate a query vector with continuous - time representation;

[0127] Unroll the key matrix and value matrix corresponding to each attention head by time step to obtain the key matrix and value matrix at each time step. Use neural differential equations to perform continuous - time modeling on the key vector and value vector to generate a key vector with continuous - time representation and a value vector with continuous - time representation:

[0128]

[0129] Among them, k i (t) is the key vector with continuous - time representation, v i (t) is the value vector with continuous - time representation, t is the current time point, t i is the initial time point, θ k is the parameter of the neural differential equation of the key vector, θ v is the parameter of the neural differential equation of the value vector, k i (t i ) is the key vector of the i - th attention head at the initial time, v i (t i ) is the value vector of the i - th attention head at the initial time.

[0130] Calculate the aggregated representation of the key matrix and value matrix based on the query vector with continuous - time representation and the key vector with continuous - time representation:

[0131]

[0132] Among them, δ i (t) is the aggregated representation of the key matrix and value matrix, q i (t) is the query vector with continuous - time representation, k i(t) is the key vector in continuous-time representation.

[0133] Integrate the value vector in continuous-time representation over a time interval to generate the expected value of the value matrix:

[0134]

[0135] where, is the expected value of the value matrix, t is the current time point, t i is the initial time point, v i (t) is the value vector in continuous-time representation.

[0136] According to the aggregation of the key matrix and the value matrix and the expected value of the value matrix, use a normalization method to obtain the output of each attention head:

[0137]

[0138] where, head i (t) is the output of each attention head, δ i (t) is the aggregated representation of the key matrix and the value matrix, d k is the dimension of each attention head, is the expected value of the value matrix, M represents the number of attention heads in the multi-head attention mechanism.

[0139] Concatenate the features of all attention heads, and linearly transform the concatenated features of the attention heads through a weight matrix to generate the multi-head attention output in continuous time:

[0140] CT-MHA(Q ctf ,K ctf ,V ctf ,t) = W o ·concat(head1(t),…,head M (t)

[0141] where, Q ctf is the query matrix, K ctf is the key matrix, V ctf is the value matrix, head i (t) is the output of each attention head, W o is the first weight matrix.

[0142] The workflow of the continuous-time layer is as follows:

[0143] Concatenate the historical entity matrix and the historical relationship matrix to generate an input matrix. According to the input matrix and time information, use the continuous-time multi-head attention mechanism to generate the multi-head attention output:

[0144]

[0145] Among them, is the output of multi-head attention, X l is the input matrix, t is the time information, x l (t) is a closed continuous differentiable time function.

[0146] A feed-forward network is used to perform a non-linear transformation on the multi-head attention output to obtain the output of each layer of the transformer. The calculation formula for the output of each layer of the transformer is:

[0147]

[0148] Among them, z l (t) is the output of the l-th layer of the transformer, is the output of multi-head attention.

[0149] The outputs of each layer of the transformer are stacked in multiple layers to generate the output of the last layer of the transformer. The output of the last layer of the transformer is decomposed into a time-continuous entity matrix and a time-continuous relationship matrix:

[0150]

[0151] Among them, is the output of the last layer of the transformer, E ctf is the time-continuous entity matrix, R ctf is the time-continuous relationship matrix.

[0152] S105. The feature fusion module generates the probabilities of all candidate entities based on the time-continuous entity matrix, the time-continuous relationship matrix, and the time information.

[0153] The feature fusion module aims to integrate the time features and task-specific features and decode them to generate representations. By combining the time features, this module effectively alleviates the problem of inaccurate prediction caused by time heterogeneity. The feature fusion module consists of two key components: a time encoder that captures time information and a feature fusion that combines the time features and task-specific features to produce a unified representation.

[0154] Time encoder. The time encoding process addresses the challenge of time heterogeneity arising from different dimensions of time representation. The transformation-based encoding mechanism captures the relationships between different time dimensions and effectively processes complex and diverse time representations. In addition, sinusoidal function encoding is introduced to simulate the periodicity of time. This combination allows the model to encode time information in a way that can capture both the complex interdependencies between time dimensions and the inherent periodicity of time data.

[0155] The time decoder is as follows:

[0156] Based on the time information, a time decoder is constructed. Through the time decoder, time encoding, periodic encoding, and aperiodic encoding are generated. The calculation formula for time encoding is:

[0157] T t = LN(TFormer(EL(t)))

[0158] where t is the time information, and T t is the time encoding.

[0159] The calculation formula for periodic encoding is:

[0160]

[0161] where T t is the time encoding, is the periodic encoding, Λ(·) is a periodic function implemented using the sine function, α p is the first ratio parameter, and β p is the second ratio parameter.

[0162] The calculation formula for aperiodic encoding is:

[0163]

[0164] where T t is the time encoding, is the aperiodic encoding, α np is the third ratio parameter, and β np is the fourth ratio parameter.

[0165] The feature fusion steps are as follows:

[0166] The feature fusion step combines the time encoding information with the representations of entities and relationships. This fusion enables the model to jointly consider time, structure, and context information. Additionally, this module scores the candidate entities based on relevance and predicts the final result by leveraging the fused representations.

[0167] Based on the time-continuous entity matrix, time-continuous relationship matrix, periodic encoding, and aperiodic encoding, calculate the probabilities of all candidate entities:

[0168]

[0169] where λ is the probability of all candidate entities, s ctf is the query entity embedding, s ctf ∈ time-continuous entity matrix, r ctf is the query relationship embedding, r ctf ∈ time-continuous relationship matrix, h t is the historical feature, h t ∈ historical matrix, is non-periodic encoding, is periodic encoding, o i is the candidate query entity embedding, o i ∈ the time-continuous entity matrix, W λ is the second weight matrix, f(·) represents the Softplus activation function, concat(·) represents the concatenation operation, and ∪ represents the operation of concatenating the predicted probabilities of all candidate entities into a single set.

[0170] S106. The training loss module uses the large model to evaluate the probabilities of all candidate entities, generates updated predicted probabilities, constructs a loss function, calculates the final loss, and uses the final loss to optimize the knowledge graph prediction model to generate the probabilities of all candidate entities finally for predicting the temporal knowledge graph.

[0171] The training loss module calculates the loss of the model by aligning the structural features with semantic consistency to ensure effective optimization. This module guides the model to learn representations that accurately reflect the structural dependencies between entities while maintaining semantic consistency. This process ultimately enhances the prediction performance and generalization ability of the model. This module includes two key parts: a reordering strategy that fuses structural and semantic information and a semantic verification mechanism based on the large model. Reordering strategy based on the collaboration of semantics and structure. A reordering strategy that collaborates semantics and structure is designed to enhance the alignment of structural and semantic features in the temporal knowledge graph. This strategy first encodes the structural information of the knowledge graph, which is constructed from all the data in the temporal knowledge graph, and uses the RotatE model to generate structural encoding embeddings. This embedding is encoded using an adapter and input into the large model to evaluate the alignment between the candidate entity and the correct entity. If the candidate entity is correct, this strategy adjusts the prediction by increasing the probability of the entity's appearance.

[0172] Specifically, the working process of the training loss module is as follows:

[0173] According to the knowledge graph data, extract the static knowledge graph corresponding to each moment (the static knowledge graph is the knowledge graph summarized from the knowledge graphs at each moment in the temporal knowledge graph);

[0174] Encode the static knowledge graph to generate a static structural entity matrix and a static structural relationship matrix:

[0175] E sta ,R sta =RotatE(KG sta )

[0176] where, E sta is the static structural entity matrix, R sta is the static structural relationship matrix, KG stais a static knowledge graph, and RotatE(·) represents the RotatE model.

[0177] Embedding E sta and R sta Capture the static structural features of the model and process them through an adapter. These embeddings are combined with prompt tokens and inputs into a large language model for fine-tuning. The large model is used to evaluate the effectiveness of each candidate set and generate corresponding results for each set:

[0178]

[0179] where, is the evaluation result of the large model, is the static structural entity embedding, ∈ static structural entity matrix, is the static structural relation embedding, ∈ static structural relation matrix, is the candidate static structural entity embedding, ∈ static structural entity matrix, Prompt is the prompt embedding, LLM1(·) represents a large language model that can be fine-tuned, which predicts values and outputs True or False, Adapter(·) represents an adapter function for further processing, concat(·) represents the concatenation operation, and ∪ represents the operation of concatenating the prediction probabilities of all candidate entities into a single set.

[0180] According to the evaluation result of the large model and the probabilities of all candidate entities, update the probabilities of all candidate entities to generate the updated prediction probabilities:

[0181]

[0182] where, is the updated prediction probability, is the evaluation result of the large model, where each element in the set is a boolean value, λ is the probability of all candidate entities, is the first hyperparameter, sampling technology is used to select the updated candidate entities, and the number of samples is determined by the hyperparameter, mean(·) represents calculating the average value, is defined as a composite function that updates the values in B according to the prediction of A. Specifically, if the prediction of A returns True, the value of value will be added to the corresponding index element in A.

[0183] Large model-based semantic verification mechanism. The semantic verification mechanism uses a large model as a black box system to improve the accuracy of entity prediction. This mechanism first uses the large model to assign probability scores to candidate entities, indicating their likelihood of being correct. During training, the mechanism includes a loss function containing the Kullback-Keibler (KL) divergence. This divergence measures the difference between the predicted probability distribution of the black box large model and the predicted probability distribution generated by the previous module in the framework. By minimizing this loss, the mechanism ensures that the output of the early module is more closely integrated with the semantic reasoning ability of the large model.

[0184] Load the GPT2 model using the transformers library developed by Hugging Face, and then input the head entity, relation, and tail entity into the model to obtain the vector representations of the head entity, relation, and tail entity.

[0185] Based on the static knowledge graph, extract the vectorized head entity, head relation, and tail entity of the large model. According to the vectorized head entity, head relation, and tail entity of the large model, calculate the similarity predicted by the large model:

[0186]

[0187] where μ is the similarity predicted by the large model, is the head entity, is the head relation, is the tail entity. Inspired by SimRE, the large model used in this method is a black box model and has not undergone fine-tuning. concat(·) represents the concatenation operation, and SI(·) represents the similarity calculation.

[0188] Finally, assign a similarity score to each candidate entity, and represent the result vector as μ. According to the semantic similarity and the updated predicted probability, construct the loss function:

[0189]

[0190] where, is the loss function, λ true is the true value, is the updated predicted probability, μ is the similarity predicted by the large model, and α is the second hyperparameter used as a scaling factor.

[0191] In the present invention, S101 to S106 are executed sequentially.

[0192] The effective effects of the present invention are mainly reflected in the following aspects:

[0193] 1. Significantly improve the prediction performance of the temporal knowledge graph

[0194] The present invention performs excellently on multiple benchmark datasets. On the ICEWS14 dataset, the mean reciprocal rank of the present invention is 0.516, the hit rate for ten candidate answers is 0.679, the hit rate for three candidate answers is 0.481, and the hit rate for one candidate answer is 0.372. Using the continuous-time Transformer network, on the ICEWS14 dataset, the mean reciprocal rank performance is improved by 1.5%. Using the Transformer time representation method, on the ICEWS14 dataset, the mean reciprocal rank performance is increased by 0.3%. Using the structure and semantics cooperation mechanism based on large models, on the ICEWS14 dataset, the mean reciprocal rank performance is increased by 0.6%. Using large model semantic verification, on the ICEWS14 dataset, the mean reciprocal rank performance is increased by 1.1%. On the ICEWS05-15 dataset, the mean reciprocal rank of the present invention is 0.551, the hit rate for ten candidate answers is 0.792, the hit rate for three candidate answers is 0.651, and the hit rate for one candidate answer is 0.435. Using the continuous-time Transformer network, on the ICEWS05-15 dataset, the mean reciprocal rank performance is improved by 0.9%. Using the Transformer time representation method, on the ICEWS05-15 dataset, the mean reciprocal rank performance is increased by 0.7%. Using the structure and semantics cooperation mechanism based on large models, on the ICEWS05-15 dataset, the mean reciprocal rank performance is increased by 0.1%. Using large model semantic verification, on the ICEWS05-15 dataset, the mean reciprocal rank performance is increased by 1.3%. On the ICEWS18 dataset, the mean reciprocal rank of the present invention is 0.362, the hit rate for ten candidate answers is 0.268, the hit rate for three candidate answers is 0.474, and the hit rate for one candidate answer is 0.577. Using the continuous-time Transformer network, on the ICEWS18 dataset, using the Transformer time representation method, on the ICEWS18 dataset, the mean reciprocal rank performance is increased by 0.2%. Using the structure and semantics cooperation mechanism based on large models, on the ICEWS18 dataset, the mean reciprocal rank performance is increased by 0.1%. Using large model semantic verification, on the ICEWS18 dataset, the mean reciprocal rank performance is increased by 0.4%.

[0195] 2. Significantly improve the accuracy of dynamic feature simulation: By introducing a continuous-time neural network module, the present invention successfully combines differential equations with Transformer, enabling more accurate simulation of the dynamic evolution of the knowledge graph in the continuous-time domain. This improvement not only enhances the model's sensitivity to time changes but also ensures the continuity of dynamic features, providing a powerful tool for understanding the dynamic behavior of complex systems.

[0196] 3. Effectively address the heterogeneity of time representation: In response to the diversity of time representation, the present invention designs a feature fusion module to encode time information using a Transformer. This step enables the model to flexibly process knowledge graphs of different time scales, effectively capture and utilize time variations, thereby improving the generalization ability and adaptability of the model.

[0197] 4. Enhance the ability to fuse structural and semantic features: Through the semantic verification mechanism in the training loss module and the re-ranking strategy based on semantics and structure, the present invention significantly improves the model's ability to fuse structural and semantic features. This improvement not only enhances the model's understanding ability of complex relationships but also reduces the problems of structural and semantic mismatches, providing a guarantee for more accurate prediction and reasoning.

[0198] 5. Improve the interpretability and robustness of the model: By introducing a large language model for semantic verification and probability distribution update, the present invention not only improves the prediction accuracy of the model but also enhances its interpretability. At the same time, due to the powerful semantic understanding ability of the large model, the model shows higher robustness when facing noisy data or outliers, making it more reliable in practical applications.

[0199] In summary, through the introduction of innovative technologies such as continuous-time Transformer, feature fusion module, and training loss module, the present invention significantly improves the model's ability in simulating dynamic features, handling time heterogeneity, fusing structural and semantic features, and improving interpretability and robustness, laying a foundation for the wide application of temporal knowledge graphs.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A temporal knowledge graph prediction method based on continuous features and large models, characterized in that the steps Including: S101. Obtain knowledge graph data, where the knowledge graph data includes an entity matrix, a relationship matrix, and time information; S102. Construct a knowledge graph prediction model, where the knowledge graph prediction model includes a historical encoding module, a continuous-time neural network module, a feature fusion module, and a training loss module connected in sequence; S103. The historical encoding module outputs a historical entity matrix, a historical relationship matrix, and a historical matrix based on the knowledge graph data; S104. The continuous-time neural network module includes a multi-head attention layer and a continuous-time layer. The multi-head attention layer obtains a multi-head attention output in continuous time based on the historical entity matrix and the historical relationship matrix; The continuous-time layer obtains a time-continuous entity matrix and a time-continuous relationship matrix based on the multi-head attention output in continuous time, the historical entity matrix, and the historical relationship matrix; S105. The feature fusion module generates the probabilities of all candidate entities based on the time-continuous entity matrix, the time-continuous relationship matrix, and the time information; S106. The training loss module evaluates the probabilities of all candidate entities using a large model based on the knowledge graph data, generates updated prediction probabilities, constructs a loss function, calculates the final loss, optimizes the knowledge graph prediction model using the final loss, and generates the probabilities of all candidate entities finally to predict a temporal knowledge graph.

2. The temporal knowledge graph prediction method based on continuous features and large models according to claim 1, wherein The working process of the historical encoding module includes: Map the entity matrix to a historical key matrix and a historical value matrix, and map the relationship matrix to a historical query matrix; Calculate the hidden layer state of each layer according to the historical key matrix, the historical value matrix, and the historical query matrix; Among them, Q hem is the historical query matrix, K hem is the historical key matrix, V hem is the historical value matrix, is the hidden layer state of the (l - 1)-th layer at time t, is the hidden layer state of the l-th layer at time t; Stack the hidden layer states of each layer in multiple layers to generate the hidden layer state of the last layer corresponding to all time points. Construct a historical matrix according to the hidden layer state of the last layer corresponding to all time points; where H is the historical matrix, is the hidden layer state of the Nth layer at time t; Generate a historical entity matrix and a historical relationship matrix by passing the historical matrix through an update function; E hem ,R hem = Update(H) Among them, E hem is the historical entity matrix, R hem is the historical relationship matrix, and H is the historical matrix.

3. The method for predicting a temporal knowledge graph based on continuous features and a large model according to claim 1, wherein The working process of the multi-head attention layer includes: Perform a linear transformation on the historical entity matrix to obtain a time-continuous key matrix, a time-continuous value matrix, and a time-continuous query matrix; Perform a weighted transformation on the time-continuous query matrix, the time-continuous key matrix, and the time-continuous value matrix to generate a query matrix, a key matrix, and a value matrix corresponding to each attention head; Among them, Q ctf is the time - continuous query matrix, K ctf is the time - continuous key matrix, V ctf is the time - continuous value matrix, is the weight matrix corresponding to the query matrix, is the weight matrix corresponding to the key matrix, is the weight matrix corresponding to the value matrix, is the query matrix corresponding to each attention head, is the key matrix corresponding to each attention head, is the value matrix corresponding to each attention head; Expand the query matrix corresponding to each attention head by time steps to obtain a query vector for each time step. According to the query vector for each time step, use natural cubic splines to model and generate a query vector with continuous-time representation; Expand the key matrix and value matrix corresponding to each attention head by time steps to obtain the key matrix and value matrix for each time step. Use neural differential equations to perform continuous-time modeling on the key vector and value vector to generate a key vector with continuous-time representation and a value vector with continuous-time representation; where k i (t) is the key vector in continuous-time representation, v i (t) is the value vector in continuous-time representation, t is the current time point, t i is the initial time point, θ k is the parameter of the neural differential equation for the key vector, θ v is the parameter of the neural differential equation for the value vector, k i (t i ) is the key vector of the i-th attention head at the initial time, v i (t i ) is the value vector of the i-th attention head at the initial time; Calculate the aggregated representation of the key matrix and the value matrix according to the query vector with continuous-time representation and the key vector with continuous-time representation; where, δ i (t) is the aggregated representation of the key matrix and the value matrix, q i (t) is the query vector in continuous-time representation, k i (t) is the key vector in continuous-time representation; Integrate the value vector with continuous-time representation over a time interval to generate the expected value of the value matrix; Among them, is the expected value of the value matrix, t is the current time point, and t i is the initial time point, and v i (t) is the value vector represented in continuous time; According to the aggregation of the key matrix and the value matrix and the expected value of the value matrix, a normalization method is adopted to obtain the output of each attention head: where, head i (t) is the output of each attention head, and δ i (t) is the aggregated representation of the key matrix and the value matrix, and d k is the dimension of each attention head, is the expected value of the value matrix, and M represents the number of attention heads in the multi-head attention mechanism; Concatenate the features of all attention heads, and linearly transform the concatenated features of the attention heads through a weight matrix to generate the multi-head attention output in continuous time: CT-MHA(Q ctf ,K ctf ,V ctf ,t) = W o ·concat(head1(T),…,head M (t)) Among them, Q ctf is the query matrix, K ctf is the key matrix, V ctf is the value matrix, head i (T) is the output of each attention head, w o is the first weight matrix.

4. The method for predicting a temporal knowledge graph based on continuous features and a large model according to claim 1, wherein The working process of the continuous time layer includes: Concatenate the historical entity matrix and the historical relationship matrix to generate an input matrix. According to the input matrix and time information, adopt the continuous time multi-head attention mechanism to generate the multi-head attention output: Among them, is the output of multi-head attention, X l is the input matrix, t is the time information, z l (t) is a closed continuous differentiable time function; Use a feed-forward network to perform a non-linear transformation on the multi-head attention output to obtain the output of the transformer for each layer. The calculation formula for the output of the transformer for each layer is: Among them, z l (t) is the output of the transformer in the l-th layer, which is the output of the multi-head attention; Stack the outputs of the transformers for each layer in multiple layers to generate the output of the last layer of the transformer. Decompose the output of the last layer of the transformer into a time-continuous entity matrix and a time-continuous relationship matrix: Among them, is the output of the last-layer transformer, and E ctf is a time-continuous entity matrix, and R ctf is a time-continuous relationship matrix.

5. The method for predicting a temporal knowledge graph based on continuous features and a large model according to claim 1, wherein The working process of the fusion module includes: Based on time information, construct a time decoder. Through the time decoder, generate time encoding, periodic encoding, and aperiodic encoding. The calculation formula for the time encoding is: T t = LN(TFormer(EL(t))) where t is the time information, and T t is the time code The calculation formula for the periodic encoding is: Among them, T t is the time code, is the period code, Λ(·) is a periodic function implemented using a sine function, and α p is the first ratio parameter, and β p is the second ratio parameter. The calculation formula for the aperiodic encoding is: Among them, T t is the time code, is the aperiodic code, α np is the third ratio parameter, β np is the fourth ratio parameter; According to the time-continuous entity matrix, the time-continuous relationship matrix, the periodic encoding, and the aperiodic encoding, calculate the probabilities of all candidate entities: Among them, λ is the probability of all candidate entities, s ctf is the query entity embedding, s ctf ∈ the time - continuous entity matrix, r ctf is the query relation embedding, r ctf ∈ the time - continuous relation matrix, h t is the historical feature, h t ∈ the historical matrix, is the aperiodic encoding, is the periodic encoding, o i is the candidate query entity embedding, o i ∈ the time - continuous entity matrix, W λ is the second weight matrix.

6. The method for predicting a temporal knowledge graph based on continuous features and a large model according to claim 1, wherein The training loss module uses a large model to evaluate the probabilities of all candidate entities and generates updated predicted probabilities, including: Extract the static knowledge graph corresponding to each moment according to the knowledge graph data; Encode the static knowledge graph to generate a static structure entity matrix and a static structure relationship matrix: E sta ,R sta = RotatE(KG sta ) Among them, E sta is a static structure entity matrix, R sta is a static structure relationship matrix, and KG sta is a static knowledge graph; According to the static structure entity matrix and the static structure relationship matrix, use a large model to evaluate and generate the evaluation result of the large model: Among them, is the evaluation result of the large model, is the static structure entity embedding, ∈ static structure entity matrix, is the static structure relationship embedding, ∈ static structure relationship matrix, is the candidate static structure entity embedding, ∈ static structure entity matrix, Prompt is the prompt embedding; According to the evaluation result of the large model and the probabilities of all candidate entities, update the probabilities of all candidate entities to generate updated predicted probabilities: Among them, is the updated predicted probability, is the evaluation result of the large model, and λ is the probability of all candidate entities, is the first hyperparameter.

7. The method for predicting a temporal knowledge graph based on continuous features and a large model according to claim 1 or 6, wherein The construction process of the loss function is: Based on the static knowledge graph, extract the vectorized head entity, head relationship, and tail entity of the large model. According to the vectorized head entity, head relationship, and tail entity of the large model, calculate the similarity predicted by the large model: where μ is the similarity predicted by the large model, is the head entity, is the head relation, is the tail entity; Construct a loss function according to the semantic similarity and the updated predicted probabilities: Among them, is the loss function, λ true is the true value, is the updated predicted probability, μ is the similarity predicted by the large model, and α is the second hyperparameter.

Images