Gradient decomposition-based continuous embedding method for increasing knowledge graph
By dividing the knowledge graph into snapshot subgraphs and converting it into constraint optimization problems, incremental training is adopted based on gradient decomposition, which solves the efficiency and effectiveness of entity and relationship embedding updates in the knowledge graph, and achieves powerful continuous embedding ability.
Patent Information
- Application Number
- CN202411569650.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-02-14
AI Technical Summary
It is difficult for the prior art to efficiently update entities and relationship embeddings in the knowledge graph, especially in the case of growing knowledge graphs, how to effectively merge the information brought by new triples and distinguish important information for preserving old triples.
Using a gradient decomposition method, the timing knowledge graph is divided into snapshot subgraphs, and the overall loss of the embedded model is divided into two different losses on the existing triplet and incremental triplet, which is converted into constraint optimization problems, and constructed A continuous embedding algorithm suitable for embedding models for incremental training.
A knowledge graph embedding model that balances learning efficiency and update effectiveness is realized, with strong continuous embedding capabilities, and can effectively learn and update entity and relationship embeddings in the knowledge graph.
Smart Images
Figure CN119443242A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of knowledge graphs and continuous learning technology, and in particular to a continuous embedding method of a growing knowledge graph based on gradient decomposition. Background Art
[0002] Knowledge graph (KG) embedding produces low-dimensional representations of KG entities and relations to facilitate downstream AI tasks such as knowledge retrieval, reasoning, and question answering. Classic KG embedding techniques, including TransE, TransD, and TransR, mainly focus on static KGs, where it is assumed that the elements and structures of the KG do not change over time.
[0003] However, in real-world scenarios, knowledge graphs continue to grow and evolve, that is, new entities and relations are constantly added to the knowledge graph. How to effectively learn the entity and relationship embeddings of the growing knowledge graph is of great significance for subsequent learning and prediction, and has attracted increasing attention in the field of knowledge graph embedding. The existing technologies can be roughly divided into two lines. One is to retrain the embedding model on the entire new dataset, which can capture the complete information of the KG, but the calculation is time-consuming. The other route is to update the embedding by fine-tuning the new triples. This method has a smaller amount of calculation, but often ignores the global impact caused by the arrival of new triples.
[0004] Therefore, we focus on continuous representation learning of growing knowledge graphs, aiming to effectively learn entity and relation embeddings as new facts continue to arrive. To achieve this goal, two problems need to be solved.
[0005] Question 1: How to efficiently obtain information from new triples? Since new triples bring new entities / relations and have implicit (or high-order) effects on the structure of the existing knowledge graph, an effective update strategy is needed to incorporate this information.
[0006] Question 2: How to distinguish and preserve the important information involved in old triples? Updating old entity / relation embeddings can easily affect many other embeddings of related entities / relationships, often leading to "catastrophic forgetting". Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention provides a method for continuously embedding a growing knowledge graph based on gradient decomposition. The embedding model of the present invention can adapt to the growing knowledge graph.
[0008] The technical solution of the present invention is: a method for continuously embedding a growing knowledge graph based on gradient decomposition, comprising the following steps:
[0009] S1) Split the time series knowledge graph into a series of snapshot subgraphs in chronological order to form a growing knowledge graph; and divide it into a training set and a test set;
[0010] S2), split the overall loss of the embedding model into two different losses on the existing triples and the incremental triples, and transform the continuous embedding task into a constrained optimization problem;
[0011] S3), constructing a continuous embedding algorithm suitable for the embedding model according to the constrained optimization problem;
[0012] S4), input each snapshot subgraph in the training set into the embedding model in sequence, and then use the continuous embedding algorithm for incremental training;
[0013] S5), input all snapshot subgraphs of the test set into the trained embedding model to obtain the final average accuracy of link prediction.
[0014] Preferably, in step S1), the triples in the temporal knowledge graph are sorted in chronological order, and the sorted triples are split into a series of snapshot subgraphs to form a continuously growing knowledge graph.
[0015] Preferably, in step S2), the overall loss of the snapshot at the tth moment of the embedding model It is expressed as:
[0016]
[0017] Among them, θ t represents the parameters of the embedding model at time t; represents the snapshot at time t; represents an incremental triple, Represents the snapshot at time t-1.
[0018] Preferably, in step S2), the snapshot at time t-1 Direct Impact Set and indirect impact set
[0019] Preferably, in step S2), the continuous embedding of the growing knowledge graph is converted into the following constrained optimization problem:
[0020]
[0021] The above optimization problem is solved by not increasing and Minimize under the constraint To approximate.
[0022] Preferably, in step S2), the constrained optimization problem is approximately:
[0023]
[0024] Preferably, in step S3), by measuring whether the triplet is well trained, it will directly affect the set Divided into working set and non-working set to effectively solve the optimization problem, given a triple<e′,r,e″> and damaged samples The scoring function of the embedded model can be used for scoring. The scoring function f(<e′,r,e″> ) is expressed as:
[0025]
[0026] Where h and n are the embedding vectors of entity e and relation r, respectively, and γ>0 is a hyperparameter. is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities; ′ t 、e ′ t ′ They represent the head entity and tail entity at time t respectively; h ′ 、h ″ Represent the embedding vectors of the head entity e′ and the tail entity e″ respectively; Respectively represent the head entity and tail entity The embedding vector of .
[0027] Preferably, in step S4), the embedding model is used in the incremental training process. The triples of are calculated using the marginal ranking loss function
[0028] Preferably, in step S4), for A triplet of , adding a regularization term to the marginal loss function.
[0029] The beneficial effects of the present invention are:
[0030] 1. The present invention transforms the loss function of the knowledge graph embedding model into a constrained optimization problem, providing a new method for promoting incremental representation learning of growing knowledge graphs;
[0031] 2. The present invention can strike a balance between learning efficiency and updating effectiveness; and has a strong continuous embedding capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1It is a schematic diagram of the process of the present invention;
[0033] Figure 2 Schematic diagram of the training method of the present invention. DETAILED DESCRIPTION
[0034] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:
[0035] like Figure 1 As shown, this embodiment provides a method for continuously embedding a growing knowledge graph based on gradient decomposition, comprising the following steps:
[0036] S1) Split the time series knowledge graph into a series of snapshot subgraphs in chronological order to form a growing knowledge graph; and divide it into a training set and a test set;
[0037] This embodiment sorts the triples in the time series knowledge graph in chronological order, and splits the sorted triples into a series of snapshot subgraphs to form a continuously growing knowledge graph, which is represented as follows:
[0038]
[0039] In the formula, ε t , denote the snapshot, entity set, and relationship set at time t, respectively, and and e′ t 、r t , e″ t Represent the head entity, relation and tail entity respectively.
[0040] S2), split the overall loss of the embedding model into two different losses on the existing triples and the incremental triples, and transform the continuous embedding task into a constrained optimization problem;
[0041] The overall loss of the embedding model at the t-th time snapshot It is expressed as:
[0042]
[0043] Among them, θ t represents the parameters of the embedding model at time t; represents the snapshot at time t; represents an incremental triple, Represents the snapshot at time t-1.
[0044] in, The loss through The loss and increment triplet of The sum of the losses is calculated.
[0045] In this embodiment, consider the incremental triple Entities and relations appearing in
[0046]
[0047] It can be divided into two subsets: direct impact set and indirect impact set in
[0048]
[0049] In this embodiment, the continuous embedding of the growing knowledge graph is converted into the following constrained optimization problem:
[0050]
[0051] The above optimization problem is solved by not increasing and Minimize under the constraint To approximate.
[0052] Among them, the loss and is initially minimized in the t-1 step; therefore, the above optimization problem can be solved by and Minimize under the constraint To approximate, in addition, due to The triples in The indirect impact, therefore the loss is often kept constant and can be ignored; therefore, the constrained optimization problem is approximately:
[0053]
[0054] S3), constructing a continuous embedding algorithm suitable for the embedding model according to the constrained optimization problem;
[0055] In this embodiment, by measuring whether the triples are well trained, it will directly affect the set Divide into working set and non-working set to effectively solve the optimization problem, given a triple<e′,r,e″> and damaged samples Scoring is performed using the score function of the embedding model, where the score function number f(<e′,r,e″> ) is expressed as:
[0056]
[0057] Where h and n are the embedding vectors of entity e and relation r, respectively, and γ>0 is a hyperparameter. is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities; e′ t , e″ t They represent the head entity and tail entity at time t respectively; h′ and h″ represent the embedding vectors of the head entity e′ and tail entity e″ respectively; Respectively represent the head entity and tail entity The embedding vector of .
[0058] According to the score function, the well-trained triple s =<e′,r,e″> The score f(s) of the trained triple is ≤ 0, while the score f(s) of the poorly trained triple is > 0; therefore, the following expression can be obtained:
[0059]
[0060] in, For the working set, The scores for the medium triplets indicate that they are not well trained; is a non-working set, which is a fraction indicating The triplets in have been well trained in the previous step;
[0061] For any given entity or relationship o, o in and The above can be calculated by the following formulas:
[0062]
[0063] in, and g o Respectively represent the gradient information of o on the working set, non-working set and incremental triples; and Respectively represent the working set, non-working set and incremental triples in parameter θ t Derivative on ; It means that o is in parameter θ t Derivative on ;
[0064] In order to achieve the optimization goal, we need to find the gradient direction z of o o , which minimizes loss, while maintaining The loss on does not increase, so the following minimization problem is introduced to find z o ;
[0065]
[0066] Among them, the superscript represents transpose;
[0067] Among them, the minimization objective So that the gradient direction z o Close to g o , thereby ensuring the minimization loss;
[0068] Two constraints make z o is the minimum working set loss and maintain the non-working set The above optimization problem is solved by using the KKT condition, and the solution is as follows:
[0069]
[0070] Where β≥0,
[0071] S4), input each snapshot subgraph in the training set into the embedding model in sequence, and then use the continuous embedding algorithm for incremental training;
[0072] In this embodiment, the embedding model is used in the incremental training process. The triples of are calculated using the marginal ranking loss function Right now:
[0073]
[0074] Where, [x] + represents the positive part of x; γ>0 is a marginal hyperparameter, is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities;
[0075] for The triplet of The parameter variation on is important to avoid loss oscillation on this subset, so a regularization term is added to the marginal loss function, namely:
[0076]
[0077] In the formula, [x] + represents the positive part of x, γ>0 is a marginal hyperparameter, λ is a hyperparameter, is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities.
[0078] S5), input all snapshot subgraphs of the test set into the trained embedding model to obtain the final average accuracy of link prediction.
[0079] In order to illustrate the advantages of the algorithm proposed by the method of this embodiment, as shown in Table 1, a quantitative scoring table of the average link prediction results of all KG snapshots of the method of this embodiment and some existing algorithms on the ICEWS14s-CL dataset is given. The ICEWS14s-CL dataset is widely used in the field of incremental learning of knowledge graphs. As can be seen from Table 1, the Ours method of this embodiment obtains the best MRR, Hits@3 and Hits@10 values, indicating that the Ours method of this embodiment obtains the best incremental learning effect.
[0080] Table 1 Quantitative analysis of the proposed method and existing algorithms on the ICEWS14s-CL dataset
[0081]
[0082]
[0083] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.
Claims
1. A continuous embedding method for growing knowledge graph based on gradient decomposition, characterized in that: The steps include: S1) Split the time series knowledge graph into a series of snapshot subgraphs in chronological order to form a growing knowledge graph; and divide it into a training set and a test set; S2), split the overall loss of the embedding model into two different losses on the existing triples and the incremental triples, and transform the continuous embedding task into a constrained optimization problem; S3), constructing a continuous embedding algorithm suitable for the embedding model according to the constrained optimization problem; S4), input each snapshot subgraph in the training set into the embedding model in sequence, and then use the continuous embedding algorithm for incremental training; S5), input all snapshot subgraphs of the test set into the trained embedding model to obtain the final average accuracy of link prediction.
2. According to claim 1, a method for continuously embedding a growing knowledge graph based on gradient decomposition is characterized in that: In step S1), the triples in the time series knowledge graph are sorted in chronological order, and the sorted triples are split into a series of snapshot subgraphs to form a continuously growing knowledge graph, wherein the continuously growing knowledge graph is represented as: In the formula, denote the snapshot, entity set, and relationship set at time t, respectively, and and e′ t 、r t , e″ t Represent the head entity, relation and tail entity respectively.
3. The continuous embedding method of a growing knowledge graph based on gradient decomposition according to claim 1 is characterized in that: In step S2), the overall loss of the embedding model at the t-th time snapshot is It is expressed as: Among them, θ t represents the parameters of the embedding model at time t; represents the snapshot at time t; represents an incremental triple, Represents the snapshot at time t-1.
4. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 3, characterized in that: In step S2), the snapshot at time t It is expressed as:
5. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 4, characterized in that: In step S2), the snapshot at time t-1 Direct Impact Set and indirect impact set Among them, the direct impact 6. The method for continuous embedding of a growing knowledge graph based on gradient decomposition according to claim 5, characterized in that: In step S2), the continuous embedding of the growing knowledge graph is converted into the following constrained optimization problem: The above optimization problem is solved by not increasing and Minimize under the constraint To approximate.
7. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 6, characterized in that: In step S2), the constrained optimization problem is approximately:
8. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 7, characterized in that: In step S3), by measuring whether the triples are well trained, it will directly affect the set Divided into working set and non-working set to effectively solve the optimization problem, given a triple<e′,r,e″> and damaged samples Scoring is performed through the scoring function of the embedded model, the scoring function f(<e′,r,e″> ) is expressed as: Where h and n are the embedding vectors of entity e and relation r, respectively, and γ>0 is a hyperparameter. is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities; e′ t , e″ t They represent the head entity and tail entity at time t respectively; h ′ 、h ″ Represent the embedding vectors of the head entity e′ and the tail entity e″ respectively; Respectively represent the head entity and tail entity The embedding vector of .
9. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 8, characterized in that: In step S4), the embedding model is trained in the incremental training process. The triples of are calculated using the marginal ranking loss function Where, [x] + represents the positive part of x; γ>0 is a marginal hyperparameter, is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities.
10. The method for continuously embedding a growing knowledge graph based on gradient decomposition according to claim 9, characterized in that: In step S4), for The triple of adds a regularization term to the marginal loss function, namely: In the formula, [x] + represents the positive part of x, γ>0 is a marginal hyperparameter, λ is a hyperparameter, is the set of corrupted triplets consisting of triplets whose heads or tails are replaced by random entities.