A learning method for temporal knowledge graph representation based on quaternion rotation

Abstract

The present invention relates to a learning method for temporal knowledge graph representation based on quaternion rotation, comprising: S1, given a quaternion (h, r, t, τ), wherein h represents a head entity, r represents a relationship, t represents a tail entity, and τ represents a timestamp, the entity and time information are integrated and rotated in the quaternion space to complete the construction of the model; S2, in order to measure the effectiveness of the quaternion and measure the similarity between vectors based on the angle between the vectors, a scoring function is set to score the sample, and a loss function, parameter regularization and time smoothing constraints are set to obtain an objective function; S3, the performance of the model is evaluated by link prediction performance on the temporal knowledge graph. The present invention models the dynamic evolution characteristics of entities over time as a rotation transformation in the quaternion space, can effectively express the complex relationship model in the temporal knowledge graph, and can effectively improve the performance of the original model by adding parameter regularization terms and temporal smoothing constraints.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a temporal knowledge graph representation learning method based on quaternion rotation. Background Art

[0002] Knowledge graph is one of the basic technologies in the field of artificial intelligence and is widely used in various knowledge-driven tasks such as intelligent question-answering systems, recommendation systems, and search engines. Although different knowledge graphs contain a large amount of structured knowledge, they still have the problem of data sparsity. Therefore, the research on completion algorithms for knowledge graphs has always been one of the hot topics. However, existing research mainly focuses on static knowledge graphs, and less attention is paid to temporal knowledge graphs. But in fact, different events or behaviors will cause changes in entities and relationships, so the completion of temporal knowledge graphs has become an increasingly important issue. However, due to the instability and sparsity of data, as well as complex temporal dependencies, temporal knowledge graph completion is a challenging problem. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a temporal knowledge graph representation learning method based on quaternion rotation, so as to solve the problem that the existing knowledge graph completion is mainly aimed at static knowledge graphs, ignoring the characteristics of the dynamic evolution of fact triples, and is difficult to be directly applied in temporal knowledge graphs.

[0004] The purpose of the present invention is achieved through the following technical solution: a temporal knowledge graph representation learning method based on quaternion rotation, the method comprising:

[0005] S1. Given a quaternion (h, r, t, τ), where h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp, the entity and time information are fused and rotated in the quaternion space to complete the model construction;

[0006] S2. In order to measure the effectiveness of the quadruple and measure the similarity between vectors based on the angle between the vectors, a scoring function is set to score the samples, and a loss function, parameter regularization and time smoothing constraints are set to obtain the objective function;

[0007] S3. Evaluate model performance by link prediction performance on the temporal knowledge graph.

[0008] Given a quaternion (h, r, t, τ), where h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp, the entity and time information are fused and rotated in the quaternion space to complete the construction of the model, specifically including:

[0009] S11. Set the quaternion to q=q r +q i i+q j j+q k k, is an imaginary unit and satisfies ijk = i 2 =j 2 =k 2 =-1, ij = k, ji = -k, jk = i, kj = -i, ki = j, ik = -j, given a quaternion (h, r, t, τ), the quaternion of each element is expressed as:

[0010]

[0011] in, h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp;

[0012] S12, fuse entity and time information to capture the evolution of entities over time;

[0013] S13. After obtaining the time-related entity representation after the fusion of time and entity, the relationship is also represented as a rotation operation in the quaternion space, and the time-related head entity is infinitely rotated to the time-related tail entity through the relationship, so as to complete the establishment of the model.

[0014] The method of fusing entity and time information to capture the entity's evolution over time includes: using the Hamiltonian product in the quaternion space to model the entity's temporal evolution as a rotation operation in the quaternion space, that is, transforming an entity that is originally independent of time into an entity that is related to time, thereby utilizing time information.

[0015] The objective function obtained by setting the loss function, parameter regularization and time smoothing constraint includes:

[0016] Based on the scoring function Setting the loss function Train them;

[0017] Regularize the loss function parameters to get

[0018] Add a temporal smoothness constraint to the loss function

[0019] Finally, the objective function is obtained

[0020] The link prediction performance evaluation model performance on the time series knowledge graph includes:

[0021] Construct a candidate quadruple (h, r, t′, τ) or (h′, r, t, τ) by replacing the first real part or the last real part of the quadruple;

[0022] The scoring function is used to rank the candidate quadruple and the quadruple in the test set according to their scores, and MRR and Hit@k are used as evaluation indicators.

[0023] The present invention has the following advantages: a temporal knowledge graph representation learning method based on quaternion rotation, which models the dynamic evolution characteristics of entities over time as rotation transformations in quaternion space, can effectively express complex relational models in temporal knowledge graphs, and can effectively improve the performance of the original model by adding parameter regularization terms and temporal smoothing constraint terms. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided below in conjunction with the drawings is not intended to limit the scope of protection of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application. The present invention is further described below in conjunction with the drawings.

[0026] The present invention provides a temporal knowledge graph representation learning method based on quaternion rotation, which can effectively capture the dynamic evolution characteristics of entities over time and is directly applicable to temporal knowledge graphs. Compared with complex numbers, quaternions enable neural networks to encode potential interdependencies and internal dependencies between multi-dimensional input features to obtain better representation capabilities, and thus can more effectively fuse temporal information. In addition, some existing temporal knowledge graph embedding models are difficult to effectively learn some complex relational models (such as symmetric, antisymmetric, inverse relations, etc.), and the present invention can effectively express these complex relations.

[0027] The triples in the traditional knowledge graph of the present invention are expanded into quadruples (h, r, t, τ) containing timestamps, where h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp. Given a temporal knowledge graph, a vector representation and a scoring function are learned for the entity, relationship and timestamp respectively, and the score of the valid quadruplet is higher than that of the invalid quadruplet.

[0028] A Quaternion is a hypercomplex number containing one real part and three imaginary parts. It can be regarded as an extension of a regular complex number. The quaternion is defined as follows:

[0029] q=q r +q i i+q j j+q k k

[0030] in, is the imaginary unit and satisfies:

[0031] ijk=i 2 =j 2 =k 2 =-1, ij=k, ji=-k, jk=i, kj=-i, ki=j, ik=-j

[0032] Correspondingly, an n-dimensional quaternion vector Defined as: q = q r +q i i+q i j+q k k, where

[0033] Among them, the operation rules of quaternions are defined as follows:

[0034] Conjugation: Quaternions The conjugate q * Represented as q * =q r -q i iq j jq k k;

[0035] Addition: Two Quaternions and The addition is expressed as q+p=(q r +p r )+(q i +p i )i+(q j +p j )j+(q k +p k )k;

[0036] Scalar Multiplication: Scalar Lambda and Quaternions The multiplication of is expressed as λq=λq r +λq i i+λq j j+λq k k;

[0037] Norm: Quaternion The norm ||q|| is expressed as

[0038] Unit quaternion: The unit quaternion can be obtained by using the definition of norm Representation

[0039] Similarly, the unit quaternion vector is defined as

[0040] Hamiltonian Product: Two Quaternions and The Hamiltonian product Also called quaternion product, it is expressed as follows:

[0041]

[0042] Similarly, the Hamiltonian product of two quaternion vectors is expressed as follows:

[0043]

[0044] in, Represents the product of corresponding elements.

[0045] Similarly, the inner product of two quaternion vectors is expressed as follows:

[0046] q·p=q r T p r +q i T p i +q j T p j +q k T p k

[0047] The present invention uses ε to represent an entity set. Represents a set of relations, Representing a time set, a temporal knowledge graph is a set of fact quadruplets (h, r, t, τ), using Represents a set of real four-tuples, using The next step is to learn an efficient quaternion representation for entities, relations, and timestamps. The detailed process of building the model is as follows:

[0048] Given a quaternion (h, r, t, τ), the quaternion representations of its elements are:

[0049] q h =a h +b h i+c h j+d h k

[0050] q r =a r +b r i+c r j+d r k

[0051] q t =a t +b t i+c t j+d t k

[0052] q τ =a τ +b τ i+c τ j+d τ k

[0053] in,

[0054] In order to effectively capture the evolution of entities over time, the present invention fuses entity and time information. The specific fusion process is: using the Hamiltonian product in the quaternion space, the temporal evolution of the entity is modeled as a rotation operation in the quaternion space, that is, the original time-independent entity is transformed into a time-dependent entity, so as to utilize the time information:

[0055]

[0056]

[0057] in, represents the normalized result, Here, the time quaternion is normalized, which has a significant impact on the result. From a geometric point of view, the head entity q h and tail entity q t As a point in the quaternion space, the unit time quaternion is regarded as a rotation operation, such as Figure 1 shown.

[0058] After obtaining the time-related entity representation after the fusion of time and entity, the relationship is also represented as a rotation operation in the quaternion space, and the time-related head entity is rotated to the time-related tail entity as much as possible through the relationship to complete the model establishment, such as Figure 1 As shown. In order to make the rotation effective, the relationship also needs to be normalized, that is, to satisfy

[0059] Finally, in order for the training to proceed effectively, the loss function based on the scoring function is as follows:

[0060]

[0061] In order to be consistent with the norms of other regularization terms below, the following parameter regularization is adopted:

[0062]

[0063] In order to make the model learn similar representations for adjacent timestamps, this paper adds a time smoothing constraint to the loss function:

[0064] Therefore, the final objective function is a combination of the above three items, as follows:

[0065]

[0066] The objective function is used for iterative training; finally, the present invention evaluates the model performance through the link prediction performance on the temporal knowledge graph.

[0067] The present invention uses the most common ICEWS14, ICEWS05-15 and GDELT as the data sets for the experiments of the present invention. The task is to predict the fact of a missing head entity or tail entity. Specifically, the task is to answer queries of the form (h, r, ?, τ) or (?, r, t, τ). In order to answer these two queries, the present invention needs to construct the corresponding candidate quadruple (h, r, t′, τ) or (h′, r, t, τ) by replacing the head entity or the tail entity. Similar to TransE, the present invention also uses filtering settings. Through the scoring function According to the scores of the candidate quads and the quads in the test set, they are ranked, and the higher the score, the higher the ranking. Accordingly, the present invention uses MRR and Hit@k as evaluation indicators, where MRR is the reciprocal of the average ranking of the test quads, and the larger the value, the better the algorithm performance; Hit@k is the percentage of the test quads ranked in the top k, and the larger the value, the better the algorithm performance. The commonly used values ​​of k are 1, 3, and 10.

[0068] One of the outstanding features of the present invention is that it can effectively express some complex relationship patterns in the time series knowledge graph. For example, for symmetric relationships, the imaginary part of the relationship quaternion can be set to 0 to express the symmetric relationship; for antisymmetric relationships, for general quaternion representation, the present invention naturally has this ability, that is, the imaginary part of the relationship quaternion is not 0; for inverse relationships, a pair of inverse relationships can be represented by a pair of conjugate quaternions.

[0069] In order to verify the effectiveness of the algorithm, the present invention tests the performance of the model in the link prediction task. The purpose of this task is to complete the quadruple with missing head entity or tail entity, that is, (h, r, ?, τ) or (h, r, ?, τ). For a quadruple (h, r, t, τ) in a test set, first construct a candidate quadruple by replacing the head entity or tail entity, and then calculate the scores of the test quadruple (h, r, t, τ) and the candidate quadruple, and rank them according to the scores.

[0070] In order to measure the pros and cons of the algorithm from different perspectives, this paper uses the evaluation indicators MRR and Hit@k to measure the performance of the algorithm. MRR represents the reciprocal of the average ranking, and the larger the value, the better the model performance; Hit@k represents the proportion of the top k rankings, and the larger the value, the better the model performance. This paper takes the datasets ICEWS14, ICEWS05-15 and GDELT as examples. The statistical information of these three datasets is shown in Table 1:

[0071] Table 1 Statistics of different data sets

[0072]

[0073] The present invention uses the PyTorch framework and a GeForce RTX 2080 graphics card for model training. For hyperparameters, the present invention uses grid search to find the best experimental parameters. For λ1 and λ2, the present invention limits them to {0.1, 0.01, 0.001}, the embedding dimension is limited to {100, 200, 300, 500}, the ratio of negative samples to positive samples is limited to {5, 10, 20}, the batch size is fixed to 512, and the learning rate is fixed to 0.1. For the two subset datasets of ICEWS, the number of training times is set to 5000, and for GDELT, the number of training times is set to 2000.

[0074] The present invention uses the static knowledge graph completion model and the temporal knowledge graph completion model as the baseline model, wherein the static knowledge graph completion model directly ignores the time information in the quadruple. For each tested quadruple, the present invention adopts a filtering setting, that is, if the candidate quadruple obtained by replacing the head entity or the tail entity appears in the training set, the validation set or the test set, the present invention will filter it out.

[0075] Table 2 shows the performance of the proposed model and the baseline model on the ICEWS14 dataset. As can be seen from the table, the proposed model achieves state-of-the-art results in almost all indicators. Compared with the latest model TeRo, MRR is improved by 0.6, Hit@1 is improved by 0.4, Hit@3 is improved by 1.5, and Hit@10 is improved by 0.4.

[0076] Table 2 Comparative experimental results on the ICEWS14 dataset

[0077]

[0078]

[0079] Table 3 shows the performance of the proposed model and the baseline model on the ICEWS05-15 dataset. As can be seen from the table, compared with TeRo, MRR is improved by 1.6, Hit@1 is improved by 1.4, Hit@3 is improved by 1.1, and Hit@10 is improved by 1.3. It can be clearly seen that the proposed model performs better on the ICEWS05-15 dataset, which may be because the dataset is larger in scale and has more entities, relationships, and timestamps.

[0080] Table 3 Comparative experimental results on the ICEWS05-15 dataset

[0081]

[0082]

[0083] Table 4 shows the performance of the proposed model and the baseline model on the GDELT dataset. As can be seen from the table, compared with DE-SimplE, the proposed model has improved in both Hit@1 and Hit@3, with an increase of 0.4 respectively.

[0084] Table 4 Comparative experimental results on the GDELT dataset

[0085]

[0086]

[0087] Based on the above analysis, the model of the present invention performs excellently on these three data sets. It can effectively model the quadruple in the temporal knowledge graph and effectively capture the temporal dependency characteristics of entities, thereby effectively completing the sparse temporal knowledge graph.

[0088] The present invention has better performance in complex space modeling than in real space modeling, and rotation operations have better performance than translation operations. The quaternion network has good performance in fields such as deep learning. The quaternion space naturally has the characteristics of rotation and is more suitable for rotation operations in vector space. Therefore, the present invention models the dynamic evolution characteristics of entities over time as rotation transformations in quaternion space. In addition, the model of the present invention can also effectively express complex relationship models in time-series knowledge graphs. Studies have shown that in the data sets ICEWS14, ICEWS05-15 and GDELT, many evaluation indicators are improved compared to existing algorithms.

[0089] The existing invention 3DRTE uses time information by fusing the relationship with the time information. The fusing method is to treat the relationship and the timestamp as a sequence and use the Attention mechanism to obtain the output, thereby obtaining the relationship that integrates the time information. On this basis, the relationship is used as a rotation transformation from the head entity to the tail entity. The present invention uses time information by fusing the entity with the time information. The fusing method is to represent both the entity and time with quaternions and use time as a rotation transformation to obtain the entity representation at a specific moment. On this basis, the relationship is also used as a rotation transformation from the head entity to the tail entity. The present invention and the existing invention 3DRTE are fundamentally different in the way they use time information.

[0090] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A learning method for temporal knowledge graph representation based on quaternion rotation, characterized by: The method comprises: S1. Given a quaternion (h, r, t, τ), where h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp, the entity and time information are fused and rotated in the quaternion space to complete the model construction; S2. In order to measure the effectiveness of the quadruple and measure the similarity between vectors based on the angle between the vectors, a scoring function is set to score the samples, and a loss function, parameter regularization and time smoothing constraints are set to obtain the objective function; S3, evaluate the model performance by link prediction performance on the temporal knowledge graph; Given a quaternion (h, r, t, τ), where h represents the head entity, r represents the relationship, t represents the tail entity, and τ represents the timestamp, the entity and time information are fused and rotated in the quaternion space to complete the model construction, specifically including: S11. Set the quaternion to , is an imaginary unit and satisfies , given a quaternion (h, r, t, τ), the quaternion of each element is represented as: ； in, ; S12, fuse entity and time information to capture the evolution of entities over time; S13. After obtaining the time-related entity representation after the fusion of time and entity, the relationship is also represented as a rotation operation in the quaternion space, and the time-related head entity is infinitely rotated to the time-related tail entity through the relationship, so as to complete the establishment of the model.

2. According to the method for learning time series knowledge graph representation based on quaternion rotation according to claim 1, it is characterized by: The method of fusing entity and time information to capture the entity's evolution over time includes: using the Hamiltonian product in the quaternion space to model the entity's temporal evolution as a rotation operation in the quaternion space, that is, transforming an entity that is originally independent of time into an entity that is related to time, thereby utilizing time information.

3. According to the method for learning time series knowledge graph representation based on quaternion rotation according to claim 1, it is characterized by: The objective function obtained by setting the loss function, parameter regularization and time smoothing constraint includes: Based on the scoring function Setting the loss function Train them; Regularize the loss function parameters to get ; Add a temporal smoothness constraint to the loss function ; Finally, the objective function is obtained .

4. According to the method for learning time series knowledge graph representation based on quaternion rotation according to claim 1, it is characterized by: The link prediction performance evaluation model performance on the time series knowledge graph includes: Construct a candidate quadruple by replacing the first or last real part of the quadruple or ; The scoring function is used to rank the candidate quadruple and the quadruple in the test set according to their scores, and MRR and Hit@k are used as evaluation indicators.

Images