Transformer-based large model knowledge graph representation method

Abstract

The application discloses a large model knowledge graph representation method based on a Transformer, and comprises the following steps: (1) randomly sampling a subgraph containing a central triple from a knowledge graph to construct a mask subgraph sequence; (2) extracting embedding representation of nodes in the mask subgraph sequence by using the Transformer, wherein a multi-dimensional power of an adjacency matrix is used as structure information during extraction, and an encoding vector of the structure information is added to an attention mechanism of the Transformer to obtain embedding representation of a mask node; (3) performing word sense prediction on the embedding representation of the mask node by using a classifier to obtain a word sense prediction result; (4) constructing a loss function and optimizing model parameters; and (5) performing completion of the knowledge graph by using the model with optimized parameters. The method can fully capture structure information and context semantic information in the knowledge graph.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of knowledge graph and machine learning, and particularly relates to a large model knowledge graph representation method based on Transformer. BACKGROUND

[0002] Transformer models have achieved remarkable performance in many fields, including natural language processing, computer vision, and graph mining. However, when it comes to the field of knowledge graph representation, models based on translation distance (such as TransE, TransH, etc. embed relationship knowledge into vector space, and then optimize the target object by using a predefined score function on these vectors) are dominant, but it is very difficult for current models to encode all information into a vector. Several works are currently using graph context information and graph neural networks (GNN) or attention-based models to learn entity relationship representations. However, these methods are still limited in the expression ability of shallow network architecture.

[0003] The knowledge graph completion task is an important application in knowledge graph representation. Specifically, for the knowledge graph completion task: given a relation graph G = (E, R), the graph consists of a node set E = {e1, e2, e3, …} and a relation set R = {r1, r2, r3, …}. The structure of the graph can be represented by an adjacency matrix A ∈ {0, 1} |V|×|V| , where V represents the number of nodes in the graph. If there is a path between two adjacent nodes, the value of the element A[i, j] in the adjacency matrix is 1, otherwise it is 0. In the knowledge graph completion task, a triple is represented as where is the head entity, is the relationship, is the tail entity. Let ν mask represent the missing part in the triple. For simplicity, consider the link prediction task of the tail entity . The task of knowledge graph completion is to learn a mapping where the label is the missing tail entity The prediction of the relationship can also be done in the same way, replacing with the symbol ν mask .

[0004] When a Transformer model is used to learn a knowledge graph, the Transformer model is deep enough, but it does not have a performance matching the graph domain. Existing methods either use very complex score function functions such as traditional TransE and RotatE, for example, the RotatE model disclosed in the existing document Rotate: Knowledge graph embedding by relational rotation in complex space, and for example, the TransE model disclosed in the existing document Translating Embeddings for Modeling Multi-relational Data, or rely on external knowledge of text, and fail to fully capture the internal semantic information and structural information of the knowledge graph. SUMMARY

[0005] In view of the above, the purpose of the present application is to provide a Transformer-based large model knowledge graph representation method, which can fully capture the structural information and contextual semantic information in the knowledge graph. And use the mask method as the knowledge graph representation learning, improve the accuracy of the knowledge graph representation learning, and further improve the ability of the knowledge graph completion.

[0006] To achieve the above invention purpose, the Transformer-based large model knowledge graph representation method provided by the embodiment comprises the following steps:

[0007] Step 1, randomly sampling a subgraph containing a center triple from a knowledge graph, taking the entity nodes and the relationships between the entity nodes in the subgraph as context nodes, serializing the center triple and the corresponding context nodes to obtain a subgraph sequence, and constructing an adjacency matrix of the subgraph sequence, and masking any node in the center triple in the subgraph sequence to obtain a masked subgraph sequence;

[0008] Step 2, using a Transformer to extract the embedding representation of the nodes in the masked subgraph sequence, and when extracting, using the multi-dimensional power of the adjacency matrix as the structural information, and adding the encoding vector of the structural information to the attention mechanism of the Transformer to obtain the embedding representation of the masked nodes;

[0009] Step 3, using a classifier to perform word sense prediction on the embedding representation of the masked nodes to obtain a word sense prediction result;

[0010] Step 4, construct a supervised loss based on the word sense prediction result and the word sense true value of the candidate node, and construct a consistency regularization loss based on positive and negative sample sampling, and optimize the parameters of the Transformer and the classifier based on the supervised loss and the consistency regularization loss;

[0011] Step 5, for the knowledge graph to be completed, the word sense prediction result of the missing node is predicted by using the parameters of the optimized Transformer and the classifier, and the knowledge graph is completed by using the word sense prediction result.

[0012] Preferably, the subgraph sequence is represented as Wherein, represents the center triple, respectively represent the head entity node, the relationship node between the head entity and the tail entity, and the tail entity node, v1, v2, v3, …, v n are context nodes, and the center triple After masking any node in the center triple, the masked subgraph sequence x mask is obtained. mask Assuming that the masked node is the tail entity node, the masked subgraph sequence x mask is represented as:

[0013]

[0014] Preferably, when using the Transformer to extract the embedding representation of the nodes in the masked subgraph sequence, the attention mechanism is used to calculate the attention weight, and the encoding vector of the structural information is added in the following way:

[0015]

[0016]

[0017] Wherein, α ij represents the attention weight calculated according to the i-th hidden vector and the j-th hidden vector h j , W Q and W k represent the weight matrix of linear transformation in the Q channel and the K channel of the attention mechanism, d represents the dimension of the feature vector, φ(i,j) represents the encoding vector value of the structural information, and f structure (·) represents a linear encoding operation, respectively represent the first power, the second power, …, the power of the normalized adjacency matrix , and m is an adjustable parameter with a value of 1-5.

[0018] Preferably, the constructed supervised loss is represented as:

[0019]

[0020] Among them, the character c represents the candidate node set, which includes entity nodes and relationship nodes, and the character s represents a specific sampling subgraph in the knowledge graph. represents the sampled mask sub-image sequence, represents the true value label of the candidate node, Represents the meaning prediction results of the mask subgraph sequence for the candidate nodes.

[0021] Preferably, the constructed consistency regularization loss is expressed as:

[0022]

[0023] Among them, c t and c t-1 is the embedding representation of the mask node in the input positive sample in the two epochs before and after, c j represents the embedding representation of the sampled negative sample mask node, sim(·) represents the cosine similarity, τ is a hyperparameter, and the negative sample is a mask subgraph sequence that refers to other center triplets different from the positive sample.

[0024] Preferably, the mask sub-image sequence needs to be vector initialized before being input into the Transformer. The specific process is expressed as follows:

[0025] h′=h+z semantic

[0026] Among them, h represents the random initialization of a single word in the sequence, z semantic It represents the semantic embedding vector of a single word using the pre-trained language model, and h′ represents the final vector initialization of a single word.

[0027] Preferably, in step 5, the knowledge graph to be completed is received, the subgraph containing the triples to be completed is extracted from the knowledge graph, and the subgraph sequence and the corresponding adjacency matrix are constructed. Then, the embedded representation of the missing nodes is extracted using a parameter-optimized Transformer, and the classifier is used to perform word meaning prediction on the embedded representation of the missing nodes to obtain the word meaning prediction result, and the missing content in the triples is completed based on the word meaning prediction result.

[0028] Preferably, when completing the knowledge graph, construct a sequence of k subgraphs containing missing triples and the corresponding adjacency matrix, and predict the embedding representations of the corresponding k missing nodes, then merge the multiple embedding representations to obtain a merged embedding representation, and use the classifier to perform word meaning prediction on the merged embedding representation to obtain a word meaning prediction result; wherein, the merged embedding representation h k represents the embedding representation of the missing nodes in the k-th subgraph sequence.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. It can effectively utilize the structural features of the graph, no longer use the traditional score function-based approach, and use the Transformer model to solve the knowledge graph completion task.

[0031] 2. The effect of adopting this structural feature is better than TransE. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 This is a flowchart of the Transformer-based large model knowledge graph representation method provided in the embodiment;

[0034] Figure 2 It is a structural diagram of a large model based on Transformer provided in an embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0036] like Figure 1 As shown, the Transformer-based large model knowledge graph representation method provided in the embodiment includes the following steps:

[0037] Step 1: Construct a mask sub-image sequence as sample data.

[0038] In the embodiment, the knowledge graph is often very large and the entire knowledge graph cannot be directly put into the Transformer. To alleviate this limitation, Triplet2Seq is used to construct triple subgraphs, and the triple subgraphs are used as sequence input to encode local information structure. The sampled subgraph containing the central triple is represented as in, represents the central triplet, Represent the head entity node, the relationship node between the head entity and the tail entity, and the tail entity node, ν1, ν2, ν3,…, ν nThey are all context nodes, indicating context information context, and the non-triplet entity nodes and the relationships between entity nodes in the subgraph are all considered as context nodes, such as Figure 2 As shown in the figure, entity nodes are represented by large circles and relationship nodes are represented by small circles. During sampling, given a predefined maximum number of nodes D, a triplet subgraph is constructed by sampling M neighboring nodes around the central triplet. This has the following beneficial effects:

[0039] (1) Transformer cannot obtain the attention of the entire knowledge graph because the computational complexity is N represents the length of the sequence. Therefore, it is beneficial to fully utilize effective sampling strategies on a large relational knowledge graph;

[0040] (2) The knowledge graph construction task is based on triples, so each triple is selected as the central node, and then the surrounding nodes are randomly sampled as context. This strategy makes the structured triples more effective in training and enhances the consistency between the training task and downstream tasks;

[0041] (3) The relationships in knowledge graphs are usually dense and rich. Since the sequence length of the Transformer is limited, the problem during training is that it is impossible to sample the entire graph of each triple. A dynamic sampling strategy is needed to capture the entire local structural information. The samples around the same central triplet may be different in each epoch of training.

[0042] After obtaining the triple subgraph sequence, we need to randomly mask the input specific tokens and then predict these masked tokens. The central triplet Mask any node in the random masking process, and randomly mask the token in the center triplet, specifically and relationship nodes Will be randomly masked to obtain the mask sub-image sequence x mask , assuming that the masked node is the tail entity node, then the masked subgraph sequence x mask Expressed as:

[0043]

[0044] Due to the unique structure of the triplet graph, only one “masked” token is randomly replaced each time to better integrate contextual information.

[0045] In the embodiment, an adjacency matrix A is constructed based on the triple subgraph, and the normalized adjacency matrix A can be obtained by normalizing the adjacency matrix A. Should The neighbor adjacency matrix power obtained by raising it to the power of m Maintain the spatial relationship between node pairs and use it to represent the number of paths between node pairs with a step length of m.

[0046] Step 2: Use Transformer to extract the embedded representation of the nodes in the masked subgraph sequence.

[0047] In the embodiment, the masked subgraph sequence needs to be initialized with a vector before being input into the Transformer. The embodiment uses the semantic encoding as input and uses the semantic encoding as an additional signal for the graph transformer. Specifically, a semantic encoding is developed that assigns a pre-trained semantic embedding vector to each node. Since the semantic encoding is applied to each node, it only needs to be added to the node features as input, expressed as:

[0048] h′=h+z semantic

[0049] Among them, h represents the random initialization of a single word in the sequence, z semantic It represents the semantic embedding vector of a single word using the pre-trained language model, and h′ represents the final vector initialization of a single word.

[0050] By using semantic embedding vectors in the input, the attention mechanism can capture the semantic importance signals in the query and key. Therefore, the Transformer can simultaneously capture the semantic relevance and node importance in the attention mechanism.

[0051] In the embodiment, when using Transformer to extract the embedded representation of the node, the multi-dimensional power of the adjacency matrix is ​​used as the structural information, and the encoding vector of the structural information is added to the attention mechanism of the Transformer to obtain the embedded representation of the masked node. Specifically, in the attention mechanism, the input sequence vector is linearly transformed through the three channels Q (Query), K (Key), and V (Value) to obtain the corresponding three matrices, and then the encoding vector of the structural information is added in the following way:

[0052]

[0053]

[0054] Among them, α ij Represents the hidden vector h according to the i-th hidden vector and the j-th hidden vector h j Calculate the attention weight, W Q and W krepresents the weight matrix of the linear transformation in the Q channel and K channel in the attention mechanism, d represents the feature vector dimension, φ(i,j) represents the encoding vector value of the structural information, and f structure (·) represents the linear encoding operation, Represents the normalized adjacency matrix The first power, second power, ..., power of , m is an adjustable parameter with a value of 1-5.

[0055] The α ij After softmax, the embedding vector of the mask node is obtained by matrix multiplication (MatMul) with the matrix of the V channel.

[0056] Step 3: Use the classifier to perform word meaning prediction on the embedded representation of the mask node to obtain the word meaning prediction result.

[0057] In an embodiment, the classifier may use MLP to implement semantic prediction classification.

[0058] In step 4, a supervision loss is constructed based on the word meaning prediction result and the true value of the word meaning of the candidate node. At the same time, a consistency regularization loss is constructed based on positive and negative sample sampling, and the parameters of the Transformer and classifier are optimized based on the supervision loss consistency regularization loss.

[0059] In the embodiment, given the sampled masked node sequence The goal is to predict the masked token, that is So the Transformer's Loss is defined as follows:

[0060]

[0061] Among them, the character c represents the candidate node set, which includes entity nodes and relationship nodes, and the character s represents a specific sampling subgraph in the knowledge graph. represents the sampled mask sub-image sequence, represents the true value label of the candidate node, Represents the meaning prediction results of the mask subgraph sequence for the candidate nodes.

[0062] In this embodiment, in order to overcome the problem of inconsistency in sampling subgraphs of the same triplet, a context comparison strategy is introduced during dynamic sampling. Given the hidden layer representation h of the subgraph sequence in an epoch, the goal is to minimize the difference between different contexts of the same triplet. For details, see the consistency regularization loss L contextual , expressed as:

[0063]

[0064] Among them, c tand c t-1 is the embedding representation of the mask node in the input positive sample in the two epochs before and after, c j represents the embedding representation of the sampled negative sample mask node, sim(·) represents the cosine similarity, τ is a hyperparameter, and the negative sample is a mask subgraph sequence that refers to other center triplets different from the positive sample.

[0065] Based on the above supervision loss and consistency regularization loss, the total loss L constructed overall for:

[0066] L overall =L MKM +λL contextual

[0067] Where λ represents the adjustment ratio. Using the total loss L overall Optimize the parameters of the Transformer and classifier to reduce the difference between triplets with the same center and increase the difference between triplets with different centers.

[0068] Step 5: For the knowledge graph to be completed, use the parameters of the optimized Transformer and classifier to predict the meaning prediction results of the missing nodes, and use the meaning prediction results to complete the knowledge graph.

[0069] Masked knowledge modeling is a general strategy that can enhance the context encoding process. However, when performing connection prediction, the challenge lies in predicting the unknown structure of the mask token, which may lead to serious label leakage problems. In order to bridge the gap between training and testing scenarios and ensure fair comparison, the context of the predicted entity will be deleted. Specifically, a knowledge graph to be completed is received, and the triples to be completed are extracted from the knowledge graph, and a sequence of triples to be completed is constructed, which is not a subgraph containing the triples to be completed. That is, the context information in the subgraph is deleted, and only the triples to be completed are retained. At the same time, after constructing the adjacency matrix of the triples to be completed, the parameter-optimized Transformer is used to extract the embedded representation of the missing node, and the classifier is used to predict the word meaning of the embedded representation of the missing node to obtain the word meaning prediction result. The missing content in the triple is completed based on the word meaning prediction result.

[0070] In the embodiment, in order to increase the accuracy of the prediction, when completing the knowledge graph, k subgraph sequences containing missing triples and corresponding adjacency matrices are constructed, and the embedding representations of the corresponding k missing nodes are predicted. Then, the multiple embedding representations are merged to obtain a merged embedding representation, and the classifier is used to perform word meaning prediction on the merged embedding representation to obtain a word meaning prediction result; wherein, the merged embedding representation h k represents the embedding representation of the missing nodes in the k-th subgraph sequence.

[0071] The Transformer-based large-model knowledge graph representation method provided in the above embodiment no longer uses a complex score function compared to the traditional translation distance-based model, making the model simple and direct. In addition, it does not require the introduction of additional node information and can effectively capture the semantic information and structural information within the knowledge graph, thereby improving the accuracy of completion.

[0072] The above embodiment provides a Transformer-based large-model knowledge graph representation method for completing missing parts of the knowledge graph, which is specifically applied to the e-commerce field. In the e-commerce knowledge graph, entities represent users, products, etc., and relationships represent the purchasing relationship between users and products, the acquaintance relationship between users, and the product dimension relationship and category dimension relationship between users and products. Among them, the product dimension relationship includes the similar product relationship and the product description relationship, and the category dimension relationship includes the hierarchical relationship of the category to which the product belongs. Steps 1 to 5 in the Transformer-based large-model knowledge graph representation method are used to predict new users or products in the e-commerce knowledge graph, and use the new users or products to complete the e-commerce knowledge graph.

[0073] The Transformer-based large-model knowledge graph representation method provided in the above embodiment can also be used to complete the protein knowledge graph in the biomedical field. In the protein knowledge graph, entities represent proteins and protein functions, and relationships represent the connection between proteins and protein functions, indicating whether a protein has the corresponding protein function and the interaction between proteins. Steps 1 to 5 of the Transformer-based large-model knowledge graph representation method are used to predict new proteins or protein functions, and use these new proteins or protein functions to complete the e-commerce knowledge graph.

[0074] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large model knowledge graph representation method based on Transformer, characterized by: The following steps are involved: Step 1: Randomly sample a subgraph containing a central triple from the knowledge graph, take the non-triplet entity nodes and the relationships between entity nodes in the subgraph as context nodes, serialize the central triples and the corresponding context nodes to obtain a subgraph sequence, and construct an adjacency matrix for the subgraph sequence. Mask any node in the central triple in the subgraph sequence to obtain a masked subgraph sequence. Step 2: Use Transformer to extract the embedded representation of the nodes in the masked subgraph sequence. During the extraction, the multi-dimensional power of the adjacency matrix is ​​used as the structural information, and the encoding vector of the structural information is added to the attention mechanism of the Transformer to obtain the embedded representation of the masked node; Step 3: Use the classifier to predict the meaning of the embedded representation of the mask node to obtain the meaning prediction result; Step 4: Build a supervised loss based on the word meaning prediction results and the true meaning value of the candidate node. At the same time, build a consistency regularization loss based on positive and negative sample sampling. Optimize the parameters of the Transformer and classifier based on the supervised loss consistency regularization loss. Step 5: For the knowledge graph to be completed, use the parameters of the optimized Transformer and classifier to predict the meaning prediction results of the missing nodes, and use the meaning prediction results to complete the knowledge graph.

2. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: The subgraph sequence is represented as in, represents the central triplet, Represent the head entity node, the relationship node between the head entity and the tail entity, and the tail entity node, ν1, ν2, ν3,…, ν n are all context nodes, and the central triplet After masking any node in , we get the mask subgraph sequence x mask , assuming that the masked node is the tail entity node, then the masked subgraph sequence x mask Expressed as:

3. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: When using Transformer to extract the embedded representation of nodes in the masked subgraph sequence, the encoding vector of the structural information is added in the following way when the attention mechanism is used to calculate the attention weight: Among them, α ij Represents the hidden vector h according to the i-th hidden vector and the j-th hidden vector h j Calculate the attention weight, W Q and W k represents the weight matrix of the linear transformation in the Q channel and K channel in the attention mechanism, d represents the dimension of the feature vector, φ(i,j) represents the encoding vector value of the structural information, and f structure (·) represents the linear encoding operation, Represents the normalized adjacency matrix The first power, second power, ..., power of , m is an adjustable parameter with a value of 1-5.

4. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: The constructed supervision loss is expressed as: Among them, the character c represents the candidate node set, which includes entity nodes and relationship nodes, and the character s represents a specific sampling subgraph in the knowledge graph. represents the sampled mask sub-image sequence, represents the true value label of the candidate node, Represents the meaning prediction results of the mask subgraph sequence for the candidate nodes.

5. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: The constructed consistency regularization loss is expressed as: Among them, c t and c t-1 is the embedding representation of the mask node in the input positive sample in the two epochs before and after, c j represents the embedding representation of the sampled negative sample mask node, sim(·) represents the cosine similarity, τ is a hyperparameter, and the negative sample is a mask subgraph sequence that refers to other center triplets different from the positive sample.

6. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: The mask sub-image sequence needs to be vector initialized before being input into the Transformer. The specific process is expressed as follows: Among them, h represents the random initialization of a single word in the sequence, It represents the semantic embedding vector of a single word using the pre-trained language model, and h′ represents the final vector initialization of a single word.

7. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: In step 5, the knowledge graph to be completed is received, the triples to be completed are extracted from the knowledge graph, and the sequence of triples to be completed and the corresponding adjacency matrix are constructed. Then, the embedded representation of the missing nodes is extracted using the parameter-optimized Transformer, and the classifier is used to predict the word meaning of the embedded representation of the missing nodes to obtain the word meaning prediction result. The missing content in the triples is completed based on the word meaning prediction result.

8. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: When completing the knowledge graph, a sequence of k subgraphs containing missing triples and the corresponding adjacency matrix are constructed, and the embedding representations of the corresponding k missing nodes are predicted. Then, multiple embedding representations are merged to obtain a merged embedding representation. The classifier is used to predict the meaning of the merged embedding representation to obtain the word meaning prediction result. Among them, the merged embedding representation h k represents the embedding representation of the missing nodes in the k-th subgraph sequence.

9. The Transformer-based large model knowledge graph representation method according to claim 1 is characterized in that: The large-model knowledge graph representation method is used to complete the e-commerce knowledge graph. In the e-commerce knowledge graph, entities represent users, products, etc., and relationships represent the purchase relationship between users and products, the acquaintance relationship between users, and the product dimension relationship and category dimension relationship between users and products. Among them, the product dimension relationship includes the relationship between similar products and the relationship between product descriptions, and the category dimension relationship includes the hierarchical relationship of the categories to which the products belong. The large-model knowledge graph representation method uses steps 1-5 to predict new users or products in the e-commerce knowledge graph, and uses the new users or products to complete the e-commerce knowledge graph.

10. The Transformer-based large model knowledge graph representation method according to claim 1, characterized in that: The large model knowledge graph representation method is used to complete the protein knowledge graph. In the protein knowledge graph, entities represent proteins and protein functions, and relationships represent the connection between proteins and protein functions, indicating whether proteins have corresponding protein functions. Relationships also represent the interactions between proteins. The large-model knowledge graph representation method uses steps 1-5 to predict new proteins or protein functions in the protein knowledge graph, and uses the new proteins or protein functions to complete the protein knowledge graph.

Images