Knowledge graph embedding method for attention mechanism and negative sampling skill

Abstract

The invention provides a knowledge graph embedding method for an attention mechanism and a negative sampling skill, and the specific implementation method comprises the steps: firstly enabling structured data of a knowledge graph to form a triple data set, and carrying out the random division of the data set; secondly, constructing an attention mechanism and an embedded model TransE-Adapt, introducing the attention mechanism into the head entity, the relationship and the tail entity, and calculating an adaptive negative sampling probability matrix B to optimize and adjust the feature extraction capability of the model and quickly and accurately obtain the features of knowledge; then defining a loss function, completing training through gradient descent of an adaptive learning rate and a back propagation algorithm, and obtaining stable model weight parameters; finally, the trained model is loaded, multi-dimensional embedding of the head entity, the relation and the tail entity of the knowledge graph can be obtained, the multi-dimensional embedding serves as input data of a downstream model, and multiple tasks can be carried out by building a downstream deep learning model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and knowledge graph technology, and specifically to a knowledge graph embedding and method; Background Art

[0002] Knowledge graph embedding is a technology that maps the entity and relationship data in the knowledge graph to a low-dimensional vector space in various ways. Knowledge graph is a structured data used to represent entities, relationships, and relationships between entities. However, the high dimensionality and sparsity of knowledge graphs limit the effect of their direct application. For this reason, knowledge graph embedding technology came into being. It represents the entities and relationships in the graph as low-dimensional vectors, thereby facilitating their application in machine learning and deep learning models. Reference 1 "Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data [J]. Advances in neural information processing systems, 2013, 26. ", the TransE model proposed is one of the most representative models in the field of knowledge graph embedding representation. It assumes that a relation can be regarded as a translation from the head entity to the tail entity, that is, for a triple (head, relation, tail), it should satisfy head + relation ≈ tail. The gradient descent algorithm is used to train and adjust the vector representation of entities and relations, so that the prediction results of the model are closer to the real triple. The TransE model proposes an important idea of ​​knowledge graph embedding representation and pioneered the Trans series of knowledge graph embedding methods; Reference 2 "Ji, Guoliang, et al. "Knowledge graph embedding via dynamic mapping matrix." Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers).2015. "Based on the basic method of TransE, multiple mapping matrices are introduced to better embed the input entities and relations, proving that improving the input embedding method can improve the model performance; by optimizing more advanced and intelligent input embedding methods, the embedding representation ability of knowledge graphs is improved, which plays an important role in the application of knowledge graphs and knowledge mining and reasoning; Summary of the invention

[0003] Based on the above situation, the present invention provides a knowledge graph embedding method combining attention mechanism and negative sampling technique. Step 1: data preprocessing. The data source adopts knowledge graph structured graph data. First, the "head entity-relationship-tail entity" triple representation of the knowledge graph is obtained and processed into a label form. The specific steps are as follows:

[0004] (1) Define the knowledge graph G; G consists of a head entity, a relationship, and a tail entity triple; G has i unique entities and j unique relationships, i is not greater than the total number of entity data, and j is not greater than the total number of relationship data; use entity i Represents the i-th non-repeated entity, relation j Indicates the jth non-repeating relationship, and the entity list is sorted and obtained as L entity , represents the entity set, and the relationship list is L relation , represents a set of relations;

[0005] L entity =[entity0,entity1,entity2,...entity i ]

[0006] L relation =[relation0,relation1,relation2,...relation j ]

[0007] L entity , L relation Elements are used as keys and numeric labels as values ​​to construct a mapping dictionary between entities and relationships;

[0008] Dic entity ={entity0:0,entity1:1,entity2:2,...entity i :i}

[0009] Dic relation ={relation0:0,relation1:1,relation2:2,...relation j :j}

[0010] (2) According to Dick entity 、Dic relation , will I n Perform key-value mapping to convert entities and relationships in triples into digital labels, and then randomly divide the knowledge graph G into training set I train , validation set I validation and test set I test ;

[0011] Step 2: Build a knowledge graph embedding representation model. The specific steps are as follows:

[0012] Build the TransE-Adapt model. The embedding representation dimension of the model is m, ranging from 1 to 256; the number of attention mechanism heads is n, ranging from 1 to 12, and can be divided by m; the relationship weight matrix is ​​represented by A, and the adaptive negative sampling weight matrix is ​​represented by B. The specific steps are as follows:

[0013] Using the data obtained in step 1 train , the training set triples have y pairs, the input embedding operation is E(·), and the embedded representation tensor E is obtained mb , the dimensions are 3×y×m,

[0014] E mb =E(I train )

[0015] E mb [u] is E mb The element index in the triple is represented by the u-th element in the triple, where u ranges from 0 to 2. The embedding of the head entity is represented by E heads , the embedding of the relation is represented as E relation , the embedding of the tail entity is represented as E tails , with dimensions of y×m,

[0016] E heads =E mb [0]

[0017] E relation =E mb [1]

[0018] E tails =E mb [2]

[0019] Q is the initial matrix of the attention mechanism, with a dimension of y×m; W K and W V is the transfer matrix for attention calculation, with a dimension of m×m; K and V are matrices after Q is linearly transformed, with a dimension of y×m,

[0020] K=Q·W K

[0021] V=Q·W V

[0022] The attention mechanism layer operation is A tt (·) operation formula, the embedding representation of the knowledge graph is defined as E out , the dimensions are 3×y×m,

[0023]

[0024] E out =A tt (E mb )

[0025] Use E true represents the embedding representation of the tail entity positive sample, with dimensions y×m.

[0026] E true =E tails

[0027] Use E false Represents the embedding representation of the tail entity negative sample, with a dimension of y×m, E false The row elements in come from E true The row elements of E true The row elements of corresponding rows in are all different.

[0028] Define the optimization objective of TransE-Adapt, where ε is positive and its size is close to 0, the initial relationship weight matrix is ​​A, and the dimension of A is m×m.

[0029] |A tt (E heads )+A tt (E relation ·A)-A tt (E true )|<ε

[0030] Calculate d and d′ respectively,

[0031] d=A tt (E heads )+A tt (E relation ·A)-A tt (E true )

[0032] d′=A tt (E heads )+A tt (E relation ·A)-A tt (E false )

[0033] E mb There are y middle-tail entities, with a total of k categories, k≤y, and both y and k are positive integers, C kh It represents the number of occurrences of the h-th tail entity among all k-th tail entities, and the probability p of the h-th entity in random replacement extraction. kh ,

[0034]

[0035] Construct a weight matrix P with a dimension of 1×k, P h Represents the hth element of the matrix, defining the probability of the hth tail entity being drawn in a random negative sample with replacement.

[0036]

[0037] After normalizing the matrix P, we get the adaptive negative sampling weight matrix B, which has a dimension of 1×k. Each element in the matrix corresponds to the random replacement extraction probability of the tail entity, ‖P h ‖ is P h The modular operation of

[0038]

[0039] Define the loss function, σ represents the distance between positive sampling and negative sampling, σ>0

[0040] L=∑(entity,relation∈G)[σ+dd′]

[0041] Step 3: Train the model through the back propagation algorithm and optimize the model using the gradient descent algorithm with adaptive learning rate. The optimization goal is to minimize the function loss L. The parameter μ is positive and its size is close to 0. After multiple rounds of training, the loss of the model gradually decreases, and L increases with the number of training rounds, satisfying

[0042] |L-σ|<μ

[0043] The model is considered to have converged, the current model parameters are recorded, and the model training is terminated; the model training weights are saved, and the TransE-Adapt model parameter learning is completed. The parameters of the model embedding layer are the embedding representation vectors of the triples in G;

[0044] Step 4: Load the TransE-Adapt model, use the knowledge graph processing method in step 1, and follow the model operation process in step 2 to obtain the triple embedding representation E of the knowledge graph. out , as the input data of the downstream model, by building a deep learning model, it can carry out various tasks such as knowledge linking and knowledge classification.

[0045] The innovative features of the present invention are as follows:

[0046] (1) A knowledge graph embedding method combined with attention mechanism is proposed, and an innovative knowledge graph embedding model TransE-Adapt is designed. This method introduces attention mechanism into the knowledge graph embedding representation task. Different from the traditional knowledge graph embedding method, this method proposes the initialization method of Q, K, and V parameters in the attention mechanism in the knowledge graph embedding task and the calculation method of attention transfer matrix. By simulating the link relationship of "head entity-relationship-tail entity", the structured data characteristics of the knowledge graph are fully utilized, and the optimization target of the additive model combined with the attention mechanism is designed to obtain high-quality knowledge graph embedding representation.

[0047] (2) During the model training process, an adaptive negative sampling weight matrix B is designed. The traditional random sampling method has the disadvantages of random sampling and uncontrollable model training effect. The adaptive negative sampling weight is calculated through an innovative algorithm. According to the data distribution characteristics of the data set, the tail entity samples with fewer occurrences are given a larger negative sampling probability, so that the model can be trained and fitted under stronger noise conditions, thereby increasing the model's noise resistance and generalization capabilities. When the model encounters unfamiliar knowledge entities, it can obtain better embedded representation and more accurate knowledge classification and reasoning capabilities.

[0048] Beneficial effects:

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) The present invention proposes a method for integrating the attention mechanism into the knowledge graph embedding representation model to improve the model's feature extraction capability;

[0051] (2) The adaptive negative sampling weight matrix B construction method proposed in the present invention can automatically construct a negative sampling probability matrix according to the data distribution of the data set, so that the tail entities with fewer and higher quality are more likely to be negatively sampled by the model, thus optimizing the model training strategy of the traditional method;

[0052] (3) The proposed TransE-Adapt deep learning model optimizes the noise resistance and generalization capabilities of the TransE model, which can be better applied to downstream tasks; BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1It is a specific implementation flow chart of the present invention, and the main process is as follows: first, perform data preprocessing, convert the data of the knowledge graph into "entity-relationship-entity" triple data, build a one-to-one correspondence between entities, relationships and digital labels, and then randomly divide them into training set, test set and verification set according to the index; then build a TransE-Adapt model, use the attention mechanism to perform secondary embedding of entities and relationships, and improve the embedding representation ability of entities and relationships. Then, analyze and process the data distribution of the tail entities, and construct an adaptive negative sampling weight matrix B. Since it represents the probability of each type of tail entity being extracted in negative sampling, entities with fewer occurrences will have a greater probability of being extracted. Then, a loss function for comparing positive sampling and negative sampling is constructed, the positive sampling loss is minimized, and the distance between positive sampling and negative sampling is maximized. The loss is calculated and the model is back-propagated and optimized. When the loss function no longer decreases, the model training is stopped and the model parameters are saved; finally, the model is loaded, and the embedded representation of the knowledge graph obtained can be used for various tasks such as knowledge linking and knowledge classification;

[0054] Figure 2 It is the TransE-Adapt model structure proposed in the present invention. In the training set, the data to be embedded E0, E1, etc. are entities, and R0 and R1 are relationships. The input data is embedded once through the entity relationship tensor embedding layer; then the attention layer initializes the Q, K, and V matrices, calculates the attention weight of the embedding once, and obtains the entity embedding representation and the relationship embedding representation; followed by the relationship weight linear transformation layer and the adaptive negative sampling matrix, which linearly transform each type of relationship weight to specialize the embedding representation ability of the relationship, obtain the negative sampling probability by calculating the adaptive negative sampling weight matrix B, construct positive sampling and negative sampling samples, and calculate the loss of the model according to the set loss function; DETAILED DESCRIPTION

[0055] In order to facilitate the understanding of the purpose and technical solution of the present invention, a detailed description is given by embedding the knowledge graph of the EMU advanced maintenance process into an implementation example, and the implementation specifically includes the following steps:

[0056] Step 1: Analyze the knowledge graph dataset and obtain unique digital labels for entities and data in the knowledge graph. The specific steps are as follows:

[0057] (1) The data set uses the EMU maintenance process knowledge graph and enumerates the triple relationship pairs therein. There is a triple relationship of the form [air spring, use, inside micrometer]. This structured data indicates that the inside micrometer is required for air spring maintenance. The data triple representation of the EMU advanced maintenance process knowledge graph is obtained, and the entities and relationships are added to the program list.

[0058] L entity =[air spring, motor, bogie,...]

[0059] L relation =[use, belong to, include, ...]

[0060] Then L entity , L relation The element is the key, and the numeric label is used as the value to construct the mapping dictionary between entities and relationships according to the OneHot encoding method;

[0061] Dic entity ={air spring:0,motor:1,bogie:2,...torque wrench:463}

[0062] Dic relation ={Use: 0, Belong to: 1, Include: 2, Maintenance Standard: 3, Maintenance Station: 4}

[0063] (3) Randomly divide G into training set, validation set and test set as the input of the model. entity 、Dic relation , the training set I train Labeled, training set format is as follows:

[0064] I train =[[0,1,3],[1,0,4]....[86,4,463]]

[0065] Step 2: Build a knowledge graph embedding representation model. The specific steps are as follows:

[0066] Build the TransE-Adapt model, with the embedding representation dimension m set to 128, the number of attention mechanism heads n set to 8, and a total of 1052 pairs of triplets;

[0067] Next, use the data obtained in step 1 train , using the TransE-Adapt model for embedding representation,

[0068] E mb =E(I train )

[0069] The embedding of the head entity is denoted as E heads , the embedding of the relation is represented as E relation , the embedding of the tail entity is represented as E tails ,

[0070] E heads =E mb [0]

[0071] E relation =E mb [1]

[0072] Etails =E mb [2]

[0073] Relation weight matrix,

[0074] A0, A1, ...A j ∈A

[0075] Introduce the attention mechanism and perform attention operation on the input to improve the model feature extraction capability. The attention layer operation is represented by A tt (·),

[0076]

[0077] Q is the initial matrix of the attention mechanism, with a dimension of 1052×128; W K and W V The transfer matrix for attention calculation is 128×128 in dimension. K and V are matrices after linear transformation of Q, with a dimension of 1052×128.

[0078] K=Q·W K

[0079] V=Q·W V

[0080] The embedding representation of the knowledge graph is defined as E out ,

[0081] E out =A tt (E mb )

[0082] Use E false Represents the embedding representation of the tail entity negative sample, with a dimension of 1052×128, E false The row elements in come from E true The row elements of E true The row elements of corresponding rows in are all different.

[0083] Define the optimization objective of TransE-Adapt, where ε is positive and its size is close to 0, the initial relationship weight matrix is ​​A, and the dimension of A is 128×128.

[0084] |A tt (E heads )+A tt (E relation ·A)-A tt (E true )|<ε

[0085] Calculate d and d′ respectively,

[0086] d=A tt (Eheads )+A tt (E relation ·A)-A tt (E true )

[0087] d′=A tt (E heads )+A tt (E relation ·A)-A tt (E false )

[0088] E mb There are 1052 middle-tail entities, totaling 189 categories, so k is 189. Then when randomly replacing negative samples, the probability of extracting the hth category is,

[0089]

[0090] Among them C kh Indicates the number of occurrences of the h-th type of tail entity among the 189 types of tail entities.

[0091] Define a 1×189 weight matrix P, where each column element represents the probability of the h-th tail entity being drawn in a random replacement negative sample.

[0092]

[0093] After normalizing the matrix P, we get the adaptive negative sampling weight matrix B, which has a dimension of 1×k. Each element in the matrix corresponds to the random replacement extraction probability of the tail entity, ‖P h ‖ is P h The modular operation of

[0094]

[0095] Define the loss function of TransE-Adapt, σ is 1,

[0096] L = ∑[1+dd′]

[0097] TransE-Adapt is trained through back-propagation and uses a gradient descent algorithm with an adaptive learning rate for model optimization. The Adam optimizer is selected as the optimizer, the initial learning rate is set to 0.001, the number of learning rate decay rounds is set to 50, and the decay rate gamma value is set to 0.6 for back-propagation training. After 300 rounds of training, the loss function no longer decreases, and the model training is stopped.

[0098] Step 3: Save the training weights of the model as TransE-Adapt.pth, obtain the embedding representation of the triples in the knowledge graph, and complete the TransE-Adapt model parameter learning;

[0099] Step 4: Load the saved model file TransE-Adapt.pth, use the knowledge graph processing method in step 1, and follow the model operation process in step 2 to obtain the triple embedding representation E of the knowledge graph G. out ,As the input data of the downstream model, by building deep learning models, such as linear connection models, various tasks such as knowledge linking,,knowledge classification, etc. can be carried out.

Claims

1. A knowledge graph embedding method with attention mechanism and negative sampling technique, comprising the following steps: Step 1: Data preprocessing. The data source uses structured graph data of the knowledge graph. First, obtain the "head entity-relationship-tail entity" triple representation of the knowledge graph and process it into a label form. The specific steps are as follows: (1) Define the knowledge graph G; G consists of a head entity, a relationship, and a tail entity triple; G has i unique entities and j unique relationships, i is not greater than the total number of entity data, and j is not greater than the total number of relationship data; use entity i Represents the i-th non-repeated entity, relation j Indicates the jth non-repeating relationship, and the entity list is sorted and obtained as L entity , represents the entity set, and the relationship list is L relation , represents a set of relations; L entity =[entity0,entity1,entity2,...entity i ] L relation =[relation0,relation1,relation2,...relation j ] L entity , L relation Elements are used as keys and numeric labels as values ​​to construct a mapping dictionary between entities and relationships; Dic entity ={entity0:0,entity1:1,entity2:2,...entity i :i} Dic relation ={relation0:0,relation1:1,relation2:2,...relation j :j} (2) According to Dick entity 、Dic relation , will I n Perform key-value mapping to convert entities and relationships in triples into digital labels, and then randomly divide the knowledge graph G into training set I train , validation set I validation and test set I test ; Step 2: Build a knowledge graph embedding representation model. The specific steps are as follows: Build the TransE-Adapt model. The embedding representation dimension of the model is m, ranging from 1 to 256; the number of attention mechanism heads is n, ranging from 1 to 12, and can be divided by m; the relationship weight matrix is ​​represented by A, and the adaptive negative sampling weight matrix is ​​represented by B. The specific steps are as follows: Using the data obtained in step 1 train , the training set triples have y pairs, the input embedding operation is E(·), and the embedded representation tensor E is obtained mb , the dimensions are 3×y×m, E mb =E(I train ) E mb [u] is E mb The element index in the triple is represented by the u-th element in the triple, where u ranges from 0 to 2. The embedding of the head entity is represented by E heads , the embedding of the relation is represented as E relation , the embedding of the tail entity is represented as E tails , with dimensions of y×m, AND heads =And mb [0] AND relation =And mb [1] AND tails =And mb [2] Q is the initial matrix of the attention mechanism, with a dimension of y×m; W K and W V is the transfer matrix for attention calculation, with a dimension of m×m; K and V are matrices after Q is linearly transformed, with a dimension of y×m, K=Q·W K V=Q·W V The attention mechanism layer operation is A tt (·) operation formula, the embedding representation of the knowledge graph is defined as E out , the dimensions are 3×y×m, AND out =A tt (AND mb ) Use E true represents the embedding representation of the tail entity positive sample, with dimensions y×m. AND true =And tails Use E false Represents the embedding representation of the tail entity negative sample, with a dimension of y×m, E false The row elements in come from E true The row elements of E true The row elements of corresponding rows in are all different. Define the optimization objective of TransE-Adapt, where ε is positive and its size is close to 0, the initial relationship weight matrix is ​​A, and the dimension of A is m×m. |A tt (AND heads )+A tt (AND relation ·A)-A tt (AND true )|<ε Calculate d and d′ respectively, d=A tt (AND heads )+A tt (AND relation ·A)-A tt (AND true ) d′=A tt (AND heads )+A tt (AND relation ·A)-A tt (AND false ) E mb There are y middle-tail entities, with a total of k categories, k≤y, and both y and k are positive integers, C kh It represents the number of occurrences of the h-th tail entity among all k-th tail entities, and the probability p of the h-th entity in random replacement extraction. kh , Construct a weight matrix P with a dimension of 1×k, P h Represents the h-th element of the matrix, defining the probability of the h-th tail entity being drawn in a random negative sample with replacement. After normalizing the matrix P, we get the adaptive negative sampling weight matrix B, which has a dimension of 1×k. Each element in the matrix corresponds to the random replacement extraction probability of the tail entity, ‖P h ‖ is P h The modular operation of Define the loss function, σ represents the distance between positive sampling and negative sampling, σ>0 L=∑(entity,relation∈G)[σ+dd′] Step 3: Train the model through the back propagation algorithm and optimize the model using the gradient descent algorithm with adaptive learning rate. The optimization goal is to minimize the function loss L. The parameter μ is positive and its size is close to 0. After multiple rounds of training, the loss of the model gradually decreases, and L increases with the number of training rounds, satisfying |L-σ|<μ The model is considered to have converged, the current model parameters are recorded, and the model training is terminated; the model training weights are saved, and the TransE-Adapt model parameter learning is completed. The parameters of the model embedding layer are the embedding representation vectors of the triples in G; Step 4: Load the TransE-Adapt model, use the knowledge graph processing method in step 1, and follow the model operation process in step 2 to obtain the triple embedding representation E of the knowledge graph. out , as the input data of the downstream model, by building a deep learning model, it can carry out various tasks such as knowledge linking and knowledge classification.

Images