Robust continuous evolution method suitable for knowledge graph embedding

Abstract

The invention discloses a robust continuous evolution method suitable for knowledge graph embedding, and belongs to the field of knowledge graph embedding. The method specifically comprises the following steps: firstly, inputting a knowledge graph data set of a to-be-tested task, and randomly dividing a training set and a test set; then, obtaining head entity embedding eh of each data in the training set, and adding noise; obtaining a loss function of the RotatE model by combining a noiseless scene, obtaining a gradient of each parameter of the RotatE model through back propagation loss, and updating a head entity embedding eh and a relation embedding er of the model by adopting a random descent gradient algorithm until the number of iterations is reached; finally, candidate entity scores are calculated through embedding of the optimal model, and the model with the highest score is selected to be applied to a to-be-tested task. According to the method, the resistance of the existing knowledge graph embedding model to noise and abnormal data is remarkably enhanced, and the generalization ability of the knowledge graph embedding model and the stability and reliability in practical application are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph embedding, and in particular to a robust continuous evolution method suitable for knowledge graph embedding. Background Art

[0002] Knowledge graphs usually store knowledge in the form of multi-relation directed graphs. Each node in the graph represents an entity, the edge represents the relationship between two entities, and the direction of the edge represents the directionality of the relationship. Each fact in the knowledge graph is generally stored in the form of a triple of (head entity, relationship, tail entity). For example, (China, capital, Shanghai) is a triple.

[0003] With the rapid development of knowledge engineering and knowledge graphs, knowledge graph embedding technology has become a hot topic in academia and industry. Currently, knowledge graph embedding models mainly focus on link prediction tasks, that is, evaluating the possible confidence of a triple (head entity, relationship, tail entity) in the knowledge graph based on known entities and relationships.

[0004] The knowledge graph embedding model uses a scoring function to quantify the confidence of the triples. According to the design of the scoring function, the knowledge graph embedding model can be divided into the following three categories: geometric model, tensor decomposition model and neural network-based model. These models can capture the structure and semantic information of the knowledge graph to a certain extent by learning low-dimensional vector representations of entities and relations, thereby achieving good performance in link prediction tasks.

[0005] Although existing knowledge graph embedding models perform well on link prediction tasks, they still face challenges in noisy environments:

[0006] First, most models are sensitive to noise, and even a small amount of noise can seriously affect the performance of the model;

[0007] Secondly, when the knowledge graph is updated, the model needs to be retrained to adapt to the new data, which is time-consuming and resource-intensive. Summary of the invention

[0008] In order to solve the problem of poor performance of existing knowledge graph embedding models in multi-hop reasoning tasks, the present invention proposes a robust continuous evolution method suitable for knowledge graph embedding, which improves its stable robustness and generalization ability in noisy environments.

[0009] The specific steps are as follows:

[0010] Step 1: Input the knowledge graph dataset of the task to be tested and randomly divide it into training set, validation set and test set;

[0011] Initialize the unified query format and unify the queries in the link prediction task into tail entity queries by introducing inverse relations;

[0012] Step 2: Get the head entity embedding e of each data in the training set h , and add noise;

[0013] A progressive Gaussian noise strategy is used to add noise to the head entity embedding e h Thus, we can obtain the perturbation entity embedding e′ h .

[0014] Perturbed entity embedding indexed by the i-th dimension Calculated as:

[0015] ∈ is the head entity embedding e h Introducing Gaussian distribution Noise;

[0016] The progressive Gaussian noise strategy means that the introduced Gaussian noise intensity coefficient is gradually enhanced during the training process, and the enhancement speed is first slow, then accelerated, and finally slowed down again.

[0017] Assume that the noise intensity coefficient starts from 0 and eventually becomes At the Tth training step, the noise intensity coefficient is:

[0018]

[0019] Where T is the current step number, T max is the expected number of training steps.

[0020] Step 3, combining the scenes with and without noise, to obtain the loss function of the RotatE model;

[0021] Specifically:

[0022] 1) In the noise-free scenario, the original loss function is

[0023] γ is the set distance threshold hyperparameter, σ is the sigmoid function, and its expression is e t is the embedding of the tail candidate entity t, is the embedding of the tail entity predicted by the RotatE model when the head entity embedding is free of noise perturbations; the calculation formula is:

[0024]

[0025] stands for Hardman product; e r Relational embedding for data;

[0026] 2) In noisy scenarios, the stability loss is

[0027] in and They represent the probability that the predicted triple (h, r, t) is true without and with disturbance. h ,e r ,e t ), the perturbed vector is expressed as (e′ h ,e r ,e t );

[0028] 3) Finally, the joint loss function is calculated by the original model loss and the noise-based stability loss:

[0029]

[0030] Where λ is a hyperparameter.

[0031] Step 4: Obtain the gradient of each parameter of the RotatE model by back-propagating the loss, and use the stochastic descent gradient algorithm to update the model’s e h and e r , until the number of iterations is reached.

[0032] During the update process, the validation set is used as input in the form of (h, r, t), which represents the tail entity query.

[0033] Step 5: Calculate the candidate entity scores by embedding the optimal model, and select the model with the highest score to be applied to the task to be tested.

[0034] The formula for calculating the candidate entity score is:

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] 1) A robust continuous evolution method for knowledge graph embedding. Through a noise-based robust training method, a progressive Gaussian noise strategy, and a stability loss function, the existing knowledge graph embedding model's resistance to noise and abnormal data is significantly enhanced, the generalization ability of the knowledge graph embedding model is effectively improved, and the stability and reliability of the knowledge graph embedding model in practical applications are improved;

[0037] 2) A robust continuous evolution method for knowledge graph embedding. The training method can adapt to different knowledge graph embedding models and task requirements. Through adaptive noise selection and joint loss function design, it achieves comprehensive optimization of link prediction and multi-hop reasoning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a robust continuous evolution method applicable to knowledge graph embedding according to the present invention;

[0039] Figure 2 A schematic diagram of the present invention for calculating the loss function of the RotatE model in combination with scenes with and without noise; DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with the accompanying drawings and implementation examples so that those skilled in the art can better understand the present invention.

[0041] The present invention discloses a robust continuous evolution method suitable for knowledge graph embedding, which introduces noise during the training process to enhance the resistance of the existing knowledge graph embedding model to noise and abnormal data, improve the robustness and generalization performance of the existing knowledge graph embedding model, enhance its reasoning ability, and achieve the effect of robust continuous evolution.

[0042] like Figure 1 As shown, the specific steps are as follows:

[0043] Step 1: Input the knowledge graph dataset that needs to perform multi-hop reasoning tasks, randomly select multiple batches of data from the given knowledge graph data, and reasonably divide them into training set, validation set and test set;

[0044] Unify the query format and unify the queries in the link prediction task into tail entity queries by introducing inverse relations. h ;

[0045] Step 2, taking the training set as input, adopting the RotatE model embedded in the knowledge graph to obtain the output in the form of candidate entity scores through the input in the form of tail entity query, and using the noise-based training method in robust training to train the parameters of the RotatE model, and using the neural network optimizer to minimize the value of the loss function during the training process, its parameters are continuously updated.

[0046] Step 3: During the update process, the validation set is used as input, the scores of candidate entities are obtained through the model, and the indicators are calculated to select the best performing knowledge graph embedding model parameters.

[0047] The following uses the dataset FB15k-237 and the knowledge graph embedding model RotatE as examples to introduce the process of training the knowledge graph embedding model using the robust training method in the present invention.

[0048] Step 1: Input the knowledge graph dataset of the task to be tested and divide it into training set, validation set and test set;

[0049] Step 2: Obtain embedding representations of entities and relations;

[0050] Step 2.1, obtain the head entity embedding and relation embedding of the training set data.

[0051] The entity embeddings e are obtained by taking the head entity and relation r in the input of the form of tail entity query, respectively. h and relation embedding r .

[0052] Step 2.2, obtain the perturbed head entity embedding.

[0053] Since the queries are unified into the tail entity query, we only need to add noise to the embedding of the head entity. h The entity embedding e of the head entity h h Introducing Gaussian distribution The noise ∈ is obtained to obtain the perturbation entity embedding e′ h Formally, the process of introducing noise is as follows: Where i is the dimension index.

[0054] In particular, the noise is introduced using a progressive Gaussian noise strategy, that is, Gaussian noise of different intensities is introduced at different stages of the model training process, which is called the cosine growth strategy. Specifically, the introduced Gaussian noise intensity coefficient gradually increases during the training process, and the speed of this increase is slow at first, then accelerates, and finally slows down again. Assuming that the noise intensity coefficient starts from 0 and eventually becomes

[0055] At the tth training step, the noise intensity coefficient Where T max is the expected number of training steps.

[0056] Step 3, training model parameters;

[0057] Step 3.1, reasoning in a noise-free scenario, the RotatE model uses the initial head entity embedding and relation embedding to calculate the formula Get the embedding of the tail entity predicted by the model when the head entity embedding is not perturbed by noise In order to ensure that the performance of the model in the original link prediction task is not affected, this example retains the original loss of the knowledge graph representation learning model and uses the original model loss function Calculate the original loss

[0058] Step 3.2, reasoning in noisy scenarios, the RotatE model uses the perturbed head entity embedding and relation embedding to calculate the formula Get the embedding of the tail entity predicted by the model when the head entity embedding is perturbed by noise

[0059] Based on the original loss function, this method adopts stability loss It is expected that the model can produce similar inference results in the presence of noise as in the absence of noise. Given the original sample x and the perturbed sample x′, the output is f(x) and f(x′), and the stability loss defined by the current noise-based stability training method is as follows: where D(·,·) denotes an arbitrary function that measures the difference between two distributions.

[0060] On this basis, the stability loss of the knowledge graph embedding model under noise is defined:

[0061]

[0062] In the case where the connection prediction is regarded as a binary classification, for the triple (h, r, t) to be predicted, this example shows that the model is in the original vector (e h ,e r ,e t ) is judged as a real instance. The perturbed vector is expressed as (e′ h ,e r ,e t ), the probability of judging it as a real instance is recorded as And use cross entropy as a function D(·,·) to measure the difference between two probability distributions;

[0063] Specifically, the stability loss function is defined as follows: in and They represent the probability that the triple is true without and with disturbance respectively.

[0064] Step 3.3, calculate the joint loss function by using the original model loss and the noise-based stability loss. The calculation process is: Where λ is a hyperparameter.

[0065] The optimization objective of the model is defined as a joint loss function of the original model loss and the noise-based stability loss.

[0066] Step 4: Based on the loss obtained in step 3, the gradients of each model parameter are obtained through back propagation, and the model parameters are updated using the stochastic descent gradient algorithm.

[0067] Repeat steps 2 to 4 and save the optimal model parameters according to the hit@k index obtained by the model in the test set until the target number of iterations is reached.

[0068] The meaning of hit@k is that among the k predicted entities with the highest scores obtained by the model based on the query, at least one is correct, which is called a hit. The calculation formula is hit@k = number of hits / total number of queries.

[0069] Example:

[0070] The effectiveness of the method described in the present invention is verified by comparative experiments on the FB15k-237 and YAGO3-10 datasets, which are public knowledge graph datasets consisting of information retrieved from Freebase and Wiki corpora, respectively. All these datasets consist of training sets, validation sets, and test sets. Table 1 lists the statistics of the datasets, as follows:

[0071] Table 1

[0072] Dataset entity relation Number of triplets FB15k-237 14,505 237 1,179,040 YAGO3-10 123,182 37 592,213

Claims

1. A robust continuous evolution method for knowledge graph embedding, characterized in that: The specific steps are as follows: Step 1: Input the knowledge graph dataset of the task to be tested and randomly divide it into training set, validation set and test set; Initialize the unified query format and unify the queries in the link prediction task into tail entity queries by introducing inverse relations; Step 2: Get the head entity embedding e of each data in the training set h , and add noise; A progressive Gaussian noise strategy is used to add noise to the head entity embedding e h Thus, we can obtain the perturbation entity embedding e′ h ; The perturbed entity embedding e′ indexed by the i-th dimension h (i) Calculated as: ∈ is the head entity embedding e h Introducing Gaussian distribution Noise; Step 3, combining the scenes with and without noise, to obtain the loss function of the RotatE model; Specifically: 1) In the noise-free scenario, the original loss function is γ is the set distance threshold hyperparameter, σ is the sigmoid function, and its expression is e t is the embedding of the tail candidate entity t, is the embedding of the tail entity predicted by the RotatE model when the head entity embedding is not perturbed by noise; 2) In noisy scenarios, the stability loss is in and They represent the probability that the predicted triple (h, r, t) is true without disturbance and with disturbance respectively; the original vector of the triple (e h ,e r ,e t ), the perturbed vector is expressed as (e′ h ,e r ,e t ); e r Relational embedding for data; 3) Finally, the joint loss function is calculated by the original model loss and the noise-based stability loss: Where λ is a hyperparameter; Step 4: Obtain the gradient of each parameter of the RotatE model by back-propagating the loss, and use the stochastic descent gradient algorithm to update the model’s e h and e r , until the number of iterations is reached; During the update process, the validation set is used as input in the form of a triple (h, r, t), which represents the tail entity query; Step 5: Calculate the candidate entity scores by embedding the optimal model, and select the model with the highest score to be applied to the task to be tested; The formula for calculating the candidate entity score is: score(h,r,t)=-||e h οe r -e t ||.

2. A robust continuous evolution method for knowledge graph embedding as claimed in claim 1, characterized in that: The progressive Gaussian noise strategy means that the introduced Gaussian noise intensity coefficient is gradually enhanced during the training process, and the enhancement speed is first slow, then accelerated, and finally slowed down again; Assume that the noise intensity coefficient starts from 0 and eventually becomes At the Tth training step, the noise intensity coefficient is: Where T is the current step number, T max is the expected number of training steps.

3. A robust continuous evolution method for knowledge graph embedding as claimed in claim 1, characterized in that: Said The calculation formula is: ° represents the Haldmann product.

Images