Time sequence knowledge graph reasoning method based on diffusion process

Abstract

The invention relates to the technical field of time sequence knowledge graph reasoning, in particular to a time sequence knowledge graph reasoning method based on a diffusion process. The method belongs to the technical field of knowledge graph reasoning. In order to solve the problem that uncertain information in rare / unseen future events is difficult to capture, the method comprises the following steps of: firstly, converting an object, a relationship and a time interval sequence related to a subject entity into uniform continuous representation by adopting a sequence representation learning module; gaussian noise is introduced strategically to enhance future event representation so as to simulate uncertainty of future events, and potential participants in the events are reconstructed by adopting a conditional denoising module based on a Transform model in consideration of time-varying characteristics of a time sequence knowledge graph structure. Then, Gaussian noise is strategically introduced to enhance the process, and the uncertainty of future events is simulated. In addition, uncertainty regularization loss is introduced in the method, historical events and rare / unknown events can be distinguished, and therefore the prediction definition of the model is improved, and the tendency of overfitting is reduced.

Description

Technical Field

[0001] The present invention relates to the field of information technology and provides a diffusion process-based temporal knowledge graph reasoning method. Background Art

[0002] Existing research usually generates deterministic representations of entities and relations by aggregating adjacent structural information and temporal information in temporal knowledge graph reasoning. These deterministic representations are then applied to scoring functions (such as ConvTransE) to evaluate the likelihood of potential future facts (events). Despite significant progress in temporal knowledge graph reasoning technology, these deterministic methods still have inherent flaws in dealing with the dynamics and uncertainty in future events, as shown in the following:

[0003] Inadequate uncertainty handling: Existing methods mainly use deterministic modeling, which makes it difficult to effectively capture and handle the uncertainty in future events, resulting in low prediction accuracy in rare or unseen event scenarios.

[0004] Overfitting risk: Since the model tends to rely more on frequently occurring historical events during training, existing methods are prone to overfitting these frequent events, thereby reducing the model's generalization ability to rare or unseen events.

[0005] Insufficient reasoning flexibility: Traditional reasoning methods are based on fixed structures and time representations, lacking adaptability to changes in time and relational structures, resulting in insufficient reasoning capabilities when dealing with time-varying structures.

[0006] The above defects are mainly due to the following reasons:

[0007] Lack of modeling of dynamics and uncertainty: Existing methods mostly use deterministic modeling, which ignores the inherent uncertainty and dynamics of future events and cannot effectively deal with complex and changing future scenarios.

[0008] Fixed representation limitation: Traditional methods reason through fixed entity and relationship representations and cannot adapt to complex changes in future events, resulting in insufficient prediction accuracy.

[0009] Bias towards historical data: Models often rely on a large amount of historical data, which causes them to be more inclined to predict events that have occurred frequently in history and ignore rare or unseen events.

[0010] In order to solve the above problems, this paper proposes a novel method, which redefines time series knowledge graph reasoning as a sequence prediction task, and uses the diffusion process and random sequence denoising method to effectively manage the dynamics and uncertainty in future events. Specifically, the main innovations of this invention include:

[0011] Introducing diffusion process and random sequence denoising methods: By converting time series knowledge graph reasoning into a sequence prediction task and adopting diffusion process and random sequence denoising methods, the present invention can process time series and structural information more flexibly, adapt to changes in future events, and improve the ability to model uncertainty and dynamics.

[0012] Uncertainty Regularization Loss: We introduce an uncertainty regularization loss to mitigate the risk of prediction bias, ensuring that the model can balance frequent historical events with rare or unseen events when predicting, thereby improving the accuracy and generalization of predictions.

[0013] Transformer-based conditional denoising module: By utilizing the Transformer-based conditional denoising module, the present invention can accurately reconstruct potential participants in future events and enhance the flexibility and accuracy of the model in dealing with time and structural changes.

[0014] In the process of overcoming these defects, the present invention solves the following technical problems:

[0015] Sequence representation learning: How to effectively transform sequences of objects, relations, and time intervals related to the subject entity into a unified continuous representation, laying a solid foundation for the subsequent denoising and reconstruction processes.

[0016] Noise introduction and management: How to strategically introduce Gaussian noise in the process of simulating the uncertainty of future events and ensure that the model can effectively manage and utilize this noise for reasoning.

[0017] Denoising and reconstruction: Develop a Transformer-based conditional denoising module that can accurately reconstruct potential participants in an event and ensure the accuracy of the model when dealing with time-varying structures.

[0018] Design of regularization loss: Design an effective uncertainty regularization loss to ensure that the model can correctly distinguish historical events from rare / unknown events, reduce overfitting, and improve the predictive performance of the model.

[0019] Through the above method, the present invention effectively overcomes the shortcomings of the existing technology in uncertainty processing, overfitting risk and reasoning flexibility, and significantly improves the prediction accuracy and generalization ability of time series knowledge graph reasoning tasks. Summary of the invention

[0020] To solve the problems existing in the prior art, the present invention provides a time series knowledge graph reasoning method based on diffusion process, which introduces Gaussian diffusion process into time series knowledge graph reasoning and explicitly manages the dynamics and uncertainty of future events through random sequence denoising process. In addition, the present invention also introduces an uncertainty regularization loss to mitigate the risk of prediction bias and ensure that the model does not overly favor frequently occurring historical scenarios and ignore rare or unseen events.

[0021] To achieve the above object, the present invention provides the following technical solutions:

[0022] The present invention provides a time series knowledge graph reasoning method based on a diffusion process, and the specific steps of the knowledge graph reasoning method are as follows:

[0023] Step S10: input data and use the sequence representation learning module to convert the sequence of objects, relations and time intervals related to the subject into a unified continuous representation;

[0024] Step S20: forward diffusion process, gradually introducing random Gaussian noise into the target object to obtain latent variables;

[0025] Step S30: reverse denoising process, using the temporal and relational embedding representations of facts as conditional information, and using the Transformer framework to reconstruct the object sequence from the noise to obtain the final result;

[0026] Step S40: Train the model and optimize the parameters according to the prediction loss and the introduced uncertainty regularization loss.

[0027] In the above scheme, the specific steps of step S10 are as follows:

[0028] Step S101, based on the entity embedding matrix The historical object sequence O 0:n And the target object o spliced ​​to get o i ∈O 0:n+1 = {O 0:n ,o}, and convert it into its corresponding embedding vector Where d is the number of entity types, and the final embedding representation of the object sequence is as follows:

[0029]

[0030] in Represents the embedding representation of the i-th entity in the sequence; Representation of objects in historical events, Represents the representation of the target object in the historical event, h is the hidden dimension size, [;] represents the connection operation of the first dimension, n represents an index variable, O0:n Represents the set of object objects from 0th to n-1th in the object object sequence;

[0031] Step S102: Based on the relationship embedding matrix E r , splicing historical event relations R 0:n And the target event relationship r, and project it into the continuous space:

[0032] r′=[E r (R 0:n );E r (r)]=[E r (r0); E r (r1);…;E r (r)]

[0033] R 0:n Represents the set of relations from 0th to n-1th in the relation sequence; in [E r (r0); E r (r1);…;E r (r)], r is the relationship of the target event, and the preceding r0 and r1 are the relationships of historical events.

[0034] Step S103, by calculating the time between each event and the query event 0:n time interval, and through Embed it to encode the time information:

[0035] t′=[E Δt (t); E Δt (t-1); ...; E Δt (0)];

[0036] Where E Δt represents a matrix that converts time intervals into embedding vectors;

[0037] In the above scheme, the specific steps of step S20 are as follows:

[0038] Get object sequence e 0 After the embedding representation, gradually move to the target object Gaussian noise is introduced, so the forward process is modeled as a Markov chain of Gaussian transitions:

[0039]

[0040] in The noise introduction strategy adopted by the diffusion process, the diffusion process step range m∈{1,2,…,M}, M represents the maximum number of steps in the forward process, and linear noise scheduling is used to adjust The ratio of added noise introduced:

[0041]

[0042] in represents the proportion of noise to be added at each step, the hyperparameter δ∈[0,1] controls the noise scale, and the two hyperparameters α min <α max ∈(0,1) represents the upper and lower limits of added noise.

[0043] In the above scheme, the specific steps of step S30 are as follows:

[0044] Step S301, introduce the encoded relationship sequence r' and time interval t' as condition information into the denoising process, as shown below:

[0045]

[0046] in represents the object sequence obtained after m-step noise addition operation. In the first step of the reverse process, Set to e 0 To ensure the stability of training, Set to non-trainable Gaussian noise, and use the Transformer encoder architecture as f θ To calculate the mean Specifically expressed as:

[0047]

[0048] Where r′ represents the embedding representation of the relation sequence obtained in step S102, and t′ represents the embedding representation of the time interval obtained in step S103. Represents the embedded representation of the object predicted after the Transformer encoder, introducing the step mapping function Emb step (*) to embed hidden representations under different noise levels;

[0049] Step S302: Since noise is only introduced to the target object in step S20, the target object representation output by the previous step of denoising is concatenated with the historical object information:

[0050]

[0051] in represents the embedding representation of the target object predicted after the first denoising step, e 0:n-1 Represents the embedded representation of the object in the historical event, Indicates that the second denoising step requires the input of the embedded representation of the target object;

[0052] Step S303, repeat the process of S301-S302 until the embedded representation of the target object predicted after the mth denoising step is obtained

[0053] In the above solution, the specific steps of step S40 are as follows:

[0054] Step S401: Usually, the diffusion model is trained using mean square error loss to quantify the difference between the original representation and the reconstructed representation. However, mean square error loss has the risk of unstable optimization in discrete space. Therefore, the distance between the quantized vectors is firstly quantified by dot product operation to estimate the probability distribution of the target object type:

[0055]

[0056] in, Indicates from The representation of the target object obtained in the output, (*) T is the matrix transpose operation, and “” represents the inner product operation. At the same time, to ensure an effective conditional generation process, the following reconstruction loss function is used:

[0057]

[0058] Among them, g i represents the one-hot encoding of the i-th real object entity, y i is the predicted probability.

[0059] In step S402, in order to prevent the model from overfitting to historical high-frequency events, an uncertainty-aware regularization loss is introduced, the purpose of which is to establish a clear boundary between unseen and observed facts. Specifically, first, a multi-layer perceptron is used to calculate the confidence score based on y as the supervisory signal for unseen and historical events:

[0060]

[0061] in Represents the confidence score, effectively distinguishing the probability of whether an event is a previously observed one. Binary vector is the historical occurrence of the event before time t, MLP represents multi-layer perceptron, Relu represents rectified linear unit, and y represents the predicted probability obtained in step S401;

[0062] Let P seen is the confidence score set corresponding to the observed facts, P non is the confidence score set corresponding to the unseen facts. Then, based on P seen and P non , minimize the following loss function:

[0063]

[0064] Where τ is the temperature coefficient used to adjust the smoothness of the output probability distribution, and u represents the value from P seen The confidence score samples sampled from P non The above loss function encourages the model to assign higher confidence scores to features indicating observed events and lower scores to features indicating unseen facts.

[0065] The beneficial effects of the present invention are:

[0066] 1. The present invention solves the problem of insufficient flexibility of the model in the prior art when processing time series and structural information by redefining time series knowledge graph reasoning as a technical means of sequence prediction tasks (step 1), thereby achieving the effect of improving the model's reasoning ability in a dynamic environment and processing complex time series structures.

[0067] 2. The present invention solves the problem that it is difficult to deal with the dynamics and uncertainty of future events in the prior art by introducing the diffusion process and random sequence denoising method (steps 2-3), thereby achieving the effect of effectively simulating the changes of future events and improving the accuracy of prediction.

[0068] 3. The present invention solves the problem of prediction bias and overfitting in the prior art model by adopting the technical means of uncertainty regularization loss (step 402), thereby achieving the effect of reducing the risk of prediction bias and balancing the relationship between historical events and rare or unseen facts. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The figure is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] See also Figure 1 , the present invention provides a technical solution:

[0072] For the time series knowledge graph in the field of political events, the present invention provides the following technical solution: a time series knowledge graph reasoning method based on the diffusion process. The specific steps of the time series knowledge graph reasoning method are as follows:

[0073] Step 1: Input historical political events and use the sequence representation learning module to convert the sequence of objects, relations, and time intervals related to the subject (person A) into a unified continuous representation;

[0074] Step S101, based on the entity embedding matrix The historical object sequence O 0:n = {someone B, organization, position} and the target object o = {someone C} to get o i ∈O 0:n+1 = {O 0:n ,o}, and convert it into its corresponding embedding vector Where d is the number of entity types, and the final embedding representation of the object sequence is as follows:

[0075]

[0076] in Represents the embedding representation of the i-th entity in the sequence; Representation of objects in historical events (person B, organization and position), represents the representation of the target object in the historical event (someone C), h is the hidden dimension size, [;] represents the connection operation of the first dimension, n represents an index variable, O 0:n Represents the set of object objects from 0th to n-1th in the object object sequence;

[0077] Step S102: Based on the relationship embedding matrix E r , splicing historical event relations R 0:n = {debate, speech, visit} and the target event relation r = {debate}, projecting it into the continuous space:

[0078] r′=[E r (R 0:n );E r (r)]=[E r (r0); E r (r1);…;E r (r)]

[0079] R 0:n Represents the set of relations from 0th to n-1th in the relation sequence;

[0080] Steps 1-3, by calculating the correlation between each event and the query event in O 0:n = the time interval in {2, 2, 1}, and through Embed it to encode the time information:

[0081] t=[E Δt (t); EΔt (t-1); ...; E Δt (0)].

[0082] Step 2: Forward diffusion process. Gradually introduce random Gaussian noise into the target object to obtain latent variables;

[0083] Get object sequence e 0 After the embedding representation, gradually move towards the embedding of the target object Gaussian noise is introduced. Therefore, the forward process is modeled as a Markov chain of Gaussian transitions:

[0084]

[0085] The diffusion process step range is m∈{1,2,…,M}, where M represents the maximum number of steps in the forward process. Linear noise scheduling is used to adjust The ratio of added noise introduced:

[0086]

[0087] in represents the proportion of noise to be added at each step, the hyperparameter δ∈[0,1] controls the noise scale, and the two hyperparameters α min <α max ∈(0,1) represents the upper and lower limits of added noise.

[0088] Step 3: Reverse denoising process. Using the temporal and relational embedding representations of facts as conditional information, the Transformer framework is used to reconstruct the object sequence from the noise.

[0089] Step 3-1, introduce the encoded relationship sequence r′ and time interval t′ as conditional information into the denoising process, as shown below:

[0090]

[0091] in represents the object sequence obtained after m-step noise addition operation. In the first step of the reverse process, Set to e 0 To ensure the stability of training, Set to non-trainable Gaussian noise, and use the Transformer encoder architecture as f θ To calculate the mean Specifically expressed as:

[0092]

[0093] Where r′ represents the embedding representation of the relation sequence obtained in step S102, and t′ represents the embedding representation of the time interval obtained in step S103. Represents the embedded representation of the object predicted after the Transformer encoder, introducing the step mapping function Emb step (*) to embed hidden representations under different noise levels;

[0094] Step 3-2, concatenate the target object representation output by the previous denoising step with the historical condition information and input it into the same Transformer encoder again:

[0095]

[0096] in represents the embedding representation of the target object predicted after the first denoising step, e 0:n-1 Represents the embedded representation of the object in the historical event, Indicates that the second denoising step requires the input of the embedded representation of the target object;

[0097] Step 4: Train the model and optimize the parameters based on the prediction loss and the introduced uncertainty regularization loss;

[0098] Step 4-1, usually the diffusion model is trained using mean square error loss to quantify the difference between the original representation and the reconstructed representation. However, mean square error loss has the risk of unstable optimization in discrete space. Therefore, the distance between the vectors is first quantified by the dot product operation to estimate the probability distribution of the target object type:

[0099]

[0100] in, Indicates from The representation of the target object obtained in the output, (*) T is the matrix transpose operation, and “” represents the inner product operation. At the same time, to ensure an effective conditional generation process, the following reconstruction loss function is used:

[0101]

[0102] Among them, g i represents the one-hot encoding of the i-th real object entity, y i is the predicted probability.

[0103] Step 4-2, in order to prevent the model from overfitting to high-frequency historical events (such as a debate between person A and person B), introduces an uncertainty-aware regularization loss, which aims to establish a clear boundary between unseen and observed facts. Specifically, first, a multi-layer perceptron is used to calculate the confidence score based on y as a supervisory signal for unseen and historical events:

[0104]

[0105] in Represents the confidence score, effectively distinguishing the probability of whether an event is a previously observed one. Binary vector is the historical occurrence of the event before time t, MLP represents multi-layer perceptron, Relu represents rectified linear unit, and y represents the predicted probability obtained in step S401;

[0106] Let P seen is the confidence score set corresponding to the observed facts, P non is the confidence score set corresponding to the unseen facts. Then, based on P seen and P non , minimize the following loss function:

[0107]

[0108] Where τ is the temperature coefficient used to adjust the smoothness of the output probability distribution, and u represents the value from P seen The confidence score samples sampled from P non The above loss function encourages the model to assign higher confidence scores to features indicating observed events and lower scores to features indicating unseen facts.

[0109] The nested named entity recognition method based on span boundary perception includes a sequence representation learning module, a forward diffusion method, a reverse denoising method, and an uncertainty regularization method, specifically:

[0110] Sequence representation learning module: Convert the sequence of objects, relations and time intervals related to the subject into a unified continuous representation;

[0111] Forward diffusion process. Gradually introduce random Gaussian noise into the target object to obtain latent variables;

[0112] Reverse denoising process: Using the temporal and relational embedding representations of facts as conditional information, the Transformer framework is used to reconstruct the object sequence from the noise;

[0113] The uncertainty regularization loss helps train the model and optimize its parameters.

[0114] The method and system of the present invention first use a sequence representation learning module to convert the objects, relationships and time interval sequences related to the subject entity into a unified continuous representation. The representation of future events is enhanced by strategically introducing Gaussian noise to simulate the uncertainty of future events, and considering the time-varying nature of the temporal knowledge graph structure, a conditional denoising module based on the Transformer model is used to reconstruct potential participants in the event. Then, this process is enhanced by strategically introducing Gaussian noise to simulate the uncertainty of future events. In addition, the method also introduces uncertainty regularization loss, which is conducive to distinguishing historical events from rare / unknown events, thereby improving the prediction clarity of the model and reducing the tendency of overfitting.

[0115] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A temporal knowledge graph reasoning method based on diffusion process, It is characterized in that The specific steps of the knowledge graph reasoning method are as follows: Step S10: input data and use the sequence representation learning module to convert the sequence of objects, relations and time intervals related to the subject into a unified continuous representation; Step S20: forward diffusion process, gradually introducing random Gaussian noise into the target object to obtain latent variables; Step S30: reverse denoising process, using the temporal and relational embedding representations of facts as conditional information, and using the Transformer framework to reconstruct the object sequence from the noise to obtain the final result; Step S40: Train the model and optimize the parameters according to the prediction loss and the introduced uncertainty regularization loss.

2. According to the diffusion process-based temporal knowledge graph reasoning method of claim 1, Features: The specific steps of step S10 are as follows: Step S101, based on the entity embedding matrix The historical object sequence O 0：n And the target object o spliced ​​to get o i ∈O 0：m+1 = {O 0：m，o } and convert it into its corresponding embedding vector Where d is the number of entity types, and the final embedding representation of the object sequence is as follows: in Represents the embedding representation of the i-th entity in the sequence; Representation of objects in historical events, represents the representation of the target object in the historical event, h is the hidden dimension size, [;] represents the connection operation of the first dimension, n represents an index variable, O 0：n Represents the set of object objects from 0th to n-1th in the object object sequence; Step S102: Based on the relationship embedding matrix E r , splicing historical event relations R 0：n And the target event relationship r, and project it into the continuous space: r′=[E r (R 0：n );E r (r)]=[E r (r 0 );E r (r 1 );...;E r (r)] R 0：n Represents the set of relations from 0th to n-1th in the relation sequence: Step S103, by calculating the time between each event and the query event 0：n time interval, and through Embed it to encode the time information: t′=[E Δt (t);E Δt (t-1);…;E Δt (0)]; Where E Δt Represents a matrix that converts time intervals into embedding vectors.

3. According to the diffusion process-based temporal knowledge graph reasoning method of claim 2, Features: The specific steps of step S20 are as follows: Get object sequence e 0 After the embedding representation, gradually move to the target object Gaussian noise is introduced, so the forward process is modeled as a Markov chain of Gaussian transitions: in The noise introduction strategy adopted by the diffusion process, the diffusion process step range m∈{1, 2, …, M}, M represents the maximum number of steps in the forward process, and linear noise scheduling is used to adjust The ratio of added noise introduced: in represents the proportion of noise to be added at each step, the hyperparameter δ∈[0,1] controls the noise scale, and the two hyperparameters α min <α max ∈(0, 1) represents the upper and lower limits of added noise.

4. According to the diffusion process-based temporal knowledge graph reasoning method of claim 3, Features: The specific steps of step S30 are as follows: Step S301, introduce the encoded relationship sequence r' and time interval t' as condition information into the denoising process, as shown below: in represents the object sequence obtained after m-step noise addition operation. In the first step of the reverse process, Set to e 0 To ensure the stability of training, Set to non-trainable Gaussian noise, and use the Transformer encoder architecture as f θ To calculate the mean Specifically expressed as: Where r′ represents the embedding representation of the relation sequence obtained in step S102, and t represents the embedding representation of the time interval obtained in step S103. Represents the embedded representation of the object predicted after the Transformer encoder, introducing the step mapping function Emb step (*) to embed hidden representations under different noise levels; Step S302: Since noise is only introduced to the target object in step S20, the target object representation output by the previous step of denoising is concatenated with the historical object information: in represents the embedding representation of the target object predicted after the first denoising step, e 0：n-1 Represents the embedded representation of the object in the historical event, Indicates that the second denoising step requires the input of the embedded representation of the target object; Step S303, repeat the process of S301-S302 until the embedded representation of the target object predicted after the mth denoising step is obtained 5. According to the diffusion process-based temporal knowledge graph reasoning method of claim 4, Features: The specific steps of step S40 are as follows: Step S401, usually the diffusion model is trained using mean square error loss to quantify the difference between the original representation and the reconstructed representation. However, mean square error loss has the risk of unstable optimization in discrete space. Therefore, the distance between the quantized vectors is firstly quantified by dot product operation to estimate the probability distribution of the target object type: in, Indicates from The representation of the target object obtained in the output, (*) T is the matrix transpose operation, "·" represents the inner product operation, and to ensure an effective conditional generation process, the following reconstruction loss function is used: Among them, g i represents the one-hot encoding of the j-th real object entity, y i is the predicted probability distribution. In step S402, in order to prevent the model from overfitting to historical high-frequency events, an uncertainty-aware regularization loss is introduced, the purpose of which is to establish a clear boundary between unseen and observed facts. Specifically, first, a multi-layer perceptron is used to calculate the confidence score based on y as the supervisory signal for unseen and historical events: in Represents the confidence score, effectively distinguishing the probability of whether an event is a previously observed one. Binary vector is the historical occurrence of the event before time t, MLP represents multi-layer perceptron, Relu represents rectified linear unit, and y represents the predicted probability obtained in step S401; Let P seen is the confidence score set corresponding to the observed facts, P non is the confidence score set corresponding to the unseen facts. Then, based on P seen and P non , minimize the following loss function: Where τ is the temperature coefficient used to adjust the smoothness of the output probability distribution, and u represents the value from P seen The confidence score samples sampled from P non The above loss function encourages the model to assign higher confidence scores to features indicating observed events and lower scores to features indicating unseen facts.

Images