A common sense knowledge graph completion method based on graph convolution network and knowledge distillation

Abstract

The application designs a common sense knowledge graph completion method based on a graph convolution network and knowledge distillation, and belongs to the common sense knowledge graph completion field; a common sense knowledge graph completion model based on a graph convolution network and knowledge distillation is designed, the model is specifically a Teacher-Student framework and is divided into a teacher model and a student model; wherein a parallel model structure is designed for the teacher model, and better results are output without adding similar edges; for the student model, knowledge in the teacher model is extracted into the student model through knowledge distillation, and the student model is used to optimize the results of the teacher model; the model generalization effect is better, and finally, better indexes are obtained.

Description

Technical Field

[0001] This invention belongs to the field of common sense knowledge graph completion, specifically involving a common sense knowledge graph completion method based on graph convolutional networks and knowledge distillation. Background Technology

[0002] Since its initial introduction in Google's search engine in 2012, the concept of knowledge graphs has gained immense popularity, leading to the development of knowledge fusion frameworks for constructing large-scale knowledge graphs. The key idea is to embed components of the knowledge graph, including transforming entities and relations into a continuous vector space, thereby simplifying operations while preserving the original structure of the knowledge graph. These entity and relation embeddings can then be applied to various tasks, such as knowledge graph completion, relation extraction, entity classification, and entity parsing.

[0003] Knowledge graphs (KGs) play a crucial role in many tasks of Natural Language Processing (NLP), but they are generally constructed manually or learned directly through representation learning, leaving them far from perfect. The objective world contains countless entities, and the human subjective world contains an even greater number of uncountable concepts. Furthermore, the complex relationships between these entities and concepts lead most knowledge graphs to face the dilemma of incomplete knowledge. Therefore, KG completion (KGC) becomes key to solving this problem.

[0004] Common sense knowledge is a collection of background information that people want to understand, assume, and use appropriately. It's knowledge shared within specific cultural or age contexts. This common sense can be acquired through learning or experience. Common sense knowledge graph (CKG) completion is a sub-direction of knowledge graphs (KGC). Entities in a common sense knowledge graph consist of free-format text, such as ConceptNet, making it larger and sparser than a standard knowledge graph. This also makes it more challenging than KGC.

[0005] Completion models based on the encoder-decode framework alleviate data sparsity by using a pre-trained BERT model and adding similar edges, and utilize graph convolutional networks (GCNs) to encode structured information, achieving good results. For example, the completion model (InitiveE) improves upon this by dynamically adding similar edges instead of statically adding them, also achieving good results.

[0006] Existing methods for completing commonsense knowledge graphs alleviate sparsity by adding similar edges, but this typically increases computational complexity. Encode-Decode frameworks use Graph Convolutional Networks (RGCNs) for structural encoding. While RGCN design parameters consider the influence of edges, they are calculated using static parameters, resulting in limited flexibility. Combining BERT embeddings with GCN embeddings may lead to insufficient knowledge inclusion in BERT embeddings. Adding similar edges can mitigate data sparsity, but changes to the model structure may render the impact of similar edges insignificant. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention designs a common sense knowledge graph completion method based on graph convolutional networks and knowledge distillation. This method can reduce the complexity of the proposed model and achieve better performance in common sense knowledge graph completion without adding similar edges.

[0008] A commonsense knowledge graph completion method based on graph convolutional networks and knowledge distillation specifically includes the following steps:

[0009] Step 1: First, use the BERT language model, take the existing knowledge graph as training data, and continue to train BERT according to the BERT masking MLM method, so that the BERT model can be adapted to the knowledge completion task.

[0010] Step 2: Input the head entity into the trained BERT model to obtain the text encoding of the head entity within the knowledge graph; then use this encoding as the initial encoding for training the Relational Graph Gate Attention Network (RGGAT) to prevent random initialization from affecting the final result.

[0011] The representation of the head entity is composed of a fusion of structural representation and text representation;

[0012] The graph convolutional model RGGAT improves upon the traditional Relational Graph Attention Network (RGAT) convolutional model by adding a gating layer during message passing to update graph nodes. This gating layer generates a weight to control the attention of each head, making the entire RGGAT model more expressive and obtaining better graph structure information. In addition, residual layers are added between adjacent graph convolutional layers to prevent the loss of information from the underlying network during training.

[0013] Step 3: The input to the graph convolutional network RGGAT requires not only entity encoding but also relation encoding. Random initialization is adopted, and the RGGAT model will iteratively learn the corresponding encoding as it is trained.

[0014] Step 4: After obtaining the head entity code and relation code, concatenate them and input them into the decoding layer to fuse them. After fusion, input them into the classification layer for classification prediction to predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters.

[0015] The decoding layer enables full interaction between entity encoding and relation encoding, and the ConvTransE convolutional model is selected as the decoding layer structure. In addition, it is modified to a parallel structure. Finally, the decoding layer fuses the input text representation and structured representation, and then inputs them into the classification layer of the output module to perform classification prediction and predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters.

[0016] Step 5: Calculate the loss between the predicted and actual results, and set the corresponding hyperparameters for iterative training;

[0017] Step 6: Using the overall model formed in steps 1-5 above as the teacher model, the corresponding predicted score distribution is obtained; the teacher model is designed as a parallel model structure, which outputs better results without adding similar edges; the teacher model is specifically encoded by a multi-head attention graph convolution module controlled by a gated network, the encoded information is fused by parallel ConvTransE, and the output module outputs the results;

[0018] Step 7: Finally, based on the idea of ​​knowledge distillation, using the predicted score distribution data obtained in Step 6 and the open training data of the common sense knowledge graph, a BERT model is trained as a student model. The final student model is used to reorder the output results of the teacher model to obtain the prediction results of the common sense tail entities. The student model extracts the prediction information from the teacher model through knowledge distillation and uses the student model to optimize the results of the teacher model. The student model has a BERT network structure, and after training, it is used to reorder the output results of the teacher model.

[0019] Beneficial technical effects of the present invention:

[0020] This invention proposes a novel graph convolutional architecture, RGGAT. Compared to the traditional RGAT model, it introduces a sub-convolutional gating layer to control the attention weights of each multi-head when aggregating neighbor node information, resulting in richer model representation. The decoding structure is designed as a parallel structure instead of the traditional serial structure, eliminating the need for the "similar edge" operation required in traditional models, which significantly increases training speed. The parallel structure achieves the same performance metrics without the need for "similar edges," and the training speed is greatly reduced. This invention also introduces knowledge distillation into the knowledge completion model, i.e., the teacher model. By distilling knowledge, the prediction results of the teacher model are introduced into the student model, resulting in better generalization performance and ultimately superior performance metrics. Attached Figure Description

[0021] Figure 1 This invention provides a flowchart of a common sense knowledge graph completion method based on graph convolutional networks and knowledge distillation. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0023] A commonsense knowledge graph completion method based on graph convolutional networks and knowledge distillation is shown in the appendix. Figure 1 As shown, the specific steps include:

[0024] Step 1: First, use the BERT model, take the existing knowledge graph as training data, and continue to train BERT according to the BERT masking MLM method, so that the BERT model can be adapted to the knowledge completion task.

[0025] Step 2: Input the entities into the trained BERT model to obtain the text encoding of the entities within the knowledge graph; then use this encoding as the initial encoding for training the graph convolutional network RGGAT to prevent random initialization from affecting the final result;

[0026] The representation of the entity is generally composed of a fusion of structural representation and textual representation. Common structural representations typically employ graph convolution, such as RGAT, for training and encoding. To obtain better graph structure information, this invention proposes a novel graph convolution model, RGGAT, which improves upon the traditional RGAT convolution model. During message passing and graph node updates, a gating layer is added. This gating layer generates a weight to control the attention of each head, thus enriching the expressive power of the entire RGGAT model and obtaining better graph structure information. Furthermore, residual layers are added between adjacent graph convolutional layers to prevent the loss of information from the underlying network during training.

[0027] Step 3: The input of the graph convolutional network RGGAT requires not only the encoding of entities, but also the encoding of relations. Considering that there are very few types of relations, random initialization is adopted. Subsequently, the RGGAT model will learn the corresponding encodings iteratively as it is trained.

[0028] Step 4: After obtaining the entity code and relation code, concatenate them and input them into the decoding layer to fuse them. After fusion, input them into the classification layer for classification prediction to predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters.

[0029] The decoding layer enables full interaction between entity encoding and relation encoding, and the ConvTransE model is selected as the decoding layer structure. In addition, the traditional decoding structure is generally a serial structure, and modifying it to a parallel structure will have additional effects. When training the model, it is not necessary to add detailed edges to expand the extra data, and it can achieve the same effect as the state-of-the-art model.

[0030] Furthermore, the drawback of parallel structures is that an extra forward propagation is required. However, through experiments, it was found that parallel input is not required, yet the same prediction results can be obtained. Through this modification, only some additional computation was added, but the training time of the model was significantly reduced, and the final result did not deteriorate.

[0031] Finally, the decoding layer fully integrates the input text representation and structured representation, and then inputs them into the classification layer of the output module to perform classification prediction and predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters.

[0032] Step 5: Calculate the loss between the predicted and actual results, and set the corresponding training hyperparameters, such as learning rate, number of iterations, learning strategy, etc., and perform iterative training;

[0033] Step 6: Using the overall model formed in steps 1-5 above as the teacher model, the corresponding predicted score distribution is obtained; for the specific teacher model part, the structure encoding is obtained by the multi-head attention map convolution module controlled by the gated network, the encoded information is fused by the parallel ConvTransE, and the output module outputs the result.

[0034] Step 7: Finally, based on the idea of ​​knowledge distillation, using the predicted score distribution data obtained in Step 6 and the open training data of the commonsense knowledge graph, a 6-layer BERT model is trained as the student model. This process is called distillation, which involves distilling the knowledge from the teacher model into the student model. The final student model is used to re-rank the output results of the teacher model to obtain the commonsense tail entity prediction results. The hits@10 is 1.82% higher than the optimal result, and other metrics are also improved. The student model extracts the predicted information from the teacher model through knowledge distillation and uses the student model to optimize the teacher model's results. The student model has a BERT network structure, and after training, it is used to re-rank the output results of the teacher model. This method outperforms the state-of-the-art baseline in ConceptNet.

[0035] This common sense knowledge graph completion model is mainly used for common sense knowledge graph completion. It uses the CN-100K common sense domain comparison dataset and the existing best common sense completion model InductiveE for comparative experiments. As can be seen from the experimental results in Table 1, the model has a good improvement in all indicators of Distillation H-RGGAT.

[0036] Table 1 compares the experimental results;

[0037]

[0038]

Claims

1. A commonsense knowledge graph completion method based on graph convolutional networks and knowledge distillation, characterized in that, Specifically, the following steps are included: Step 1: First, use the BERT language model, take the existing knowledge graph as training data, and continue to train BERT according to the BERT masking MLM method, so that the BERT model can be adapted to the knowledge completion task. Step 2: Input the head entity into the trained BERT model to obtain the text encoding of the head entity within the knowledge graph; then use this encoding as the initial encoding for training the Relational Graph Gate Attention Network (RGGAT) to prevent random initialization from affecting the final result. Among them, the graph convolutional model RGGAT improves upon the traditional Relational Graph Attention Network (RGAT) convolutional model by adding a gating layer when updating graph nodes through message passing. This gating layer generates a weight to control the attention of each head. In addition, residual layers are added between adjacent graph convolutional layers to prevent the loss of structured information from the underlying network during training. Step 3: The input to the graph convolutional network RGGAT requires not only entity encoding but also relation encoding. Random initialization is adopted, and the RGGAT model will iteratively learn the corresponding encoding as it is trained. Step 4: After obtaining the head entity code and relation code, concatenate them and input them into the decoding layer to fuse them. After fusion, input them into the classification layer for classification prediction to predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters. The decoding layer enables full interaction between entity encoding and relation encoding, and uses the ConvTransE convolutional model as the decoding layer structure. In addition, it is modified to a parallel structure. Finally, the decoding layer fuses the input text representation and structured representation, and then inputs them to the classification layer of the output module to perform classification prediction and predict the tail entity. Since the decoding layer adopts a parallel structure, there are two prediction results. Therefore, the two prediction results are fused by using hyperparameters. Step 5: Calculate the loss between the predicted and actual results, and set the corresponding hyperparameters for iterative training; Step 6: Using the overall model formed in steps 1-5 above as the teacher model, obtain the corresponding predicted score distribution; Step 7: Finally, based on the idea of ​​knowledge distillation, using the predicted score distribution data obtained in Step 6 and the open training data of the common sense knowledge graph, a BERT model is trained as a student model. The final student model is used to reorder the output results of the teacher model to obtain the prediction results of the common sense tail entity. The student model is a BERT network structure. After the student model is trained, it is used to reorder the output results of the teacher model.

2. The common sense knowledge graph completion method based on graph convolutional networks and knowledge distillation according to claim 1, characterized in that, The representation of the head entity is composed of a fusion of structural representation and textual representation.

3. The common sense knowledge graph completion method based on graph convolutional networks and knowledge distillation according to claim 1, characterized in that, The teacher model is designed as a parallel model structure without adding similar edges. Specifically, the structure encoding is obtained by a multi-head attention graph convolution module controlled by a gated network, and the encoded information is fused by a parallel ConvTransE module. The output module outputs the result.

4. The commonsense knowledge graph completion method based on graph convolutional networks and knowledge distillation according to claim 1, characterized in that, The student model extracts the predicted information from the teacher model into the student model through knowledge distillation, and uses the student model to optimize the results of the teacher model.

Images