Knowledge graph completion method, system and device based on transformer and progressive distillation

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a knowledge graph completion method, system and device based on Transfomer and progressive distillation. The scheme comprises: I. obtaining sample data set for model training composed of a large number of triple vector; II. performing heavy mask processing on the sample data set, which is used for pre-training of the language model based on Transfomer; III. taking the pre-trained voice model as a teacher model, and taking the simplified teacher model as a student model to construct a self-supervised learning framework; IV. performing progressive distillation on the teacher model and the student model; in each round of the progressive distillation process, the mask rate of the sample data is reduced, and the student model is taken as the teacher model, and the student model is simplified to be used as the student model in the next round. Finally, the model meeting the accuracy and light weight is saved for subsequent knowledge graph completion tasks of the user. The application solves the problem that the existing language model parameters are too large and cannot meet the light weight application requirements.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a knowledge graph completion method based on Transformer and progressive distillation, a lightweight language model training system based on progressive distillation, and a lightweight knowledge graph completion device. Background Technology

[0002] Knowledge graph creation is a fundamental function of artificial intelligence. A knowledge graph is a graph-structured knowledge base, typically composed of triples (h, r, t), where h is the head entity, t is the tail entity, and r is the relation. Popular knowledge graphs include YAGO, Freebase, Wikidata, etc. Knowledge graphs have proven well-suited for various downstream tasks, such as intelligent question answering, recommendation systems, and information extraction. Despite their proven effectiveness, knowledge graphs still face the problem of missing information as real-world data increases. Therefore, knowledge graph completion tasks are crucial for the automated construction of knowledge graphs.

[0003] Existing knowledge graph completion tasks are generally divided into two categories: structure-based and description-based methods. Structure-based methods utilize the topological and triplet structure information of the knowledge graph (KG) to represent the feature vectors of entity relations. Description-based methods, on the other hand, learn representations by incorporating semantic descriptions of entities and relations based on pre-trained language models. Clearly, with the rise of pre-trained language models, description-based methods have gradually become dominant. By using semantic descriptions of entities and relations as auxiliary information, they deeply mine the latent knowledge within the pre-trained language model. Description-based methods not only address the limitations of structure-based methods in handling inductive KGC tasks but also achieve significant performance improvements on transductive KGC tasks.

[0004] However, while description-based methods improve performance, they also bring problems such as large number of model parameters and high computational cost. Existing high-performance network models require server-level data processing equipment and cannot be deployed on a wide range of low-computing-power terminal devices (such as smartphones, smartwatches, tablets, etc.). This restricts their application in downstream tasks such as real-time recommendation systems. Summary of the Invention

[0005] To address the problem that existing language models used for knowledge graph completion tasks have excessively large parameters, failing to meet the requirements for lightweight applications, this invention provides a knowledge graph completion method, system, and device based on Transformer and progressive distillation.

[0006] This invention is achieved using the following technical solution:

[0007] A knowledge graph completion method based on Transformer and progressive distillation is proposed. This method constructs a lightweight and high-precision language model and utilizes it to complete the knowledge graph completion task. The knowledge graph completion method includes the following steps:

[0008] S1: Obtain massive amounts of text data for constructing a knowledge graph, and extract statements from the text data that contain head entities, relations, tail entities, and descriptive information of the head and tail entities.

[0009] S2: Convert each statement into a corresponding triple vector according to a preset format, thereby obtaining a sample dataset containing a large number of triple vectors.

[0010] S3: Mask the triple vectors in the sample dataset according to the maximum masking rate, and use them to pre-train a high-performance speech model to obtain the required teacher model.

[0011] S4: Simplify the teacher model, construct a self-supervised learning framework, and then train it on the sample dataset using a progressive distillation strategy to obtain a simplified model that meets the required accuracy and lightweight requirements. The process is as follows:

[0012] S41: In the initial stage, the speech model is used as the teacher model, and the teacher model in this round is simplified and used as the student model.

[0013] S42: Construct a self-supervised learning framework based on teacher and student models; in the self-supervised learning framework, the pre-trained teacher model is used to generate labels to guide the training of the student model.

[0014] S43: Design a masking module in front of the self-supervised learning framework to mask the input triple vectors according to a preset mask rate.

[0015] S44: Preset loss function for the training phase, which includes label loss, mask generation feature loss, and score loss.

[0016] S45: Obtain the sample dataset, input it into the masking module and perform random masking according to the preset masking rate, and then synchronously input the processed data into the teacher model and student model.

[0017] S46: Calculate the loss function value for this round of training, determine whether the accuracy requirement has been met, and make the following decision:

[0018] (1) If so, save the model parameters of the student model at the current stage and use it as a high-level model that can be further simplified.

[0019] (2) If not, save the model parameters of the teacher model at the current stage and output them as the final simplified model.

[0020] S47: Determine whether the advanced model acquired in this round meets the lightweight requirements, and make the following decision:

[0021] (1) If so, the advanced model is output as the final simplified model.

[0022] (2) If not, the high-level model is further simplified, and the resulting low-level model is used as the student model for the next round. The student model of the current round is used as the teacher model for the next round. The masking rate of the masking model is reduced by a preset margin.

[0023] Finally, return to step S45 and continue training the student model using a progressive distillation method.

[0024] S5: Input the triple vector to be completed into the simplified model obtained in the previous step. The simplified model outputs the complete triple vector after semantic completion according to the knowledge graph established in the training phase, and uses it for subsequent applications.

[0025] As a further improvement of the present invention, in step S1, the head entity and the tail entity represent two objects involved in a complete statement, the relationship represents the association between the two objects, and the description information is supplementary information for the head entity and the tail entity.

[0026] As a further improvement of the present invention, in step S2, the standard format of the triple vector is: [head entity][head entity description][relation][tail entity][tail entity description]. A triple vector that simultaneously includes [head entity], [relation], and [tail entity] is a complete vector, while a triple vector missing any one of [head entity], [relation], or [tail entity] is a missing vector to be completed.

[0027] As a further improvement of the present invention, in step S3, the speech model is trained using a BERT base model containing 12 layers; the BERT base model consists of a 12-layer Transformer encoder, 12 attention heads, 768 hidden sizes and 110M parameters.

[0028] As a further improvement of the present invention, in step S4, in the progressive distillation strategy, the maximum masking rate of the masking module is set to 20%, and is gradually reduced by 50% in each round of self-distillation training. Simultaneously, the student model that passed the previous round of training is used as the teacher model for the next round, and the layers of the student model that passed the training are simplified by three layers and used as the student model for the next round.

[0029] As a further improvement of the present invention, in step S44, the loss function set during the training phase... The expression is as follows:

[0030]

[0031] In the above formula, α and β are the balance coefficients of the loss, respectively; Indicates label loss; This represents the feature loss generated by the mask; This indicates a score loss.

[0032] Among them, score loss The calculation formula is as follows:

[0033]

[0034] In the above formula, score S The score represents the output score of the student model. T This represents the score output of the teacher model; MSE represents the function for calculating the mean squared error.

[0035] As a further improvement of the present invention, tag loss The calculation method is as follows:

[0036] (1) Calculate the scores of the head entity relation vector and tail entity vector output by the student model using cosine similarity, as follows:

[0037]

[0038] In the above formula, e hr Represents the head entity relation vector, e hr = [Head Entity][Head Entity Description][Relationship]; e t Represents the tail entity vector, e t = [Tail Entity][Tail Entity Description];

[0039] (2) Use the cross-entropy function to calculate the label loss of the score from the label and the scoring function.

[0040]

[0041] Here, CrossEntropy represents the cross-entropy function, and L is the label output by the teacher model.

[0042] As a further improvement of the present invention, the mask generation feature loss The calculation method is as follows:

[0043] (1) Perform a masking operation on the triplet vectors input to the teacher model and the student model:

[0044]

[0045] Here, Mask represents a masking operation at a random location. Represents the student feature vector. It is the teacher's feature vector; λ represents the masking rate; and These represent the student feature vector and the teacher feature vector before masking, respectively.

[0046] (2) The feature loss is generated by generating the mask between the teacher model and the student model using the following formula.

[0047]

[0048] This invention also includes a lightweight language model training system based on a progressive distillation method. This system uses the aforementioned knowledge graph completion method based on Transformer and progressive distillation to construct a lightweight language model capable of completing text content association information based on a pre-generated knowledge graph. The lightweight language model training system includes: a text extraction unit, a triple generation unit, a mask feature generation unit, a teacher model creation unit, a model simplification unit, and a progressive distillation unit.

[0049] The text extraction unit is used to acquire massive amounts of text data related to the pre-knowledge graph and extract a large number of standardized statements from the text data, including head entities, relations, tail entities, and descriptive information of head entities and tail entities.

[0050] The triple generation unit is used to convert each statement into a corresponding triple vector according to a preset format, thereby obtaining a sample dataset containing a large number of triple vectors.

[0051] The mask feature generation unit is used to randomly mask the input triple vector according to a preset mask rate to obtain mask features with missing information; the mask features with different mask rates are used as inputs to the teacher model and student model at different stages of the progressive distillation process.

[0052] The teacher model creation unit is used to pre-train a ransformer-based neural network using the acquired complete sample dataset, thereby obtaining a large-scale, high-performance speech model, which serves as the initial teacher model in the progressive distillation process.

[0053] The model simplification element is used to simplify the teacher model in each round of the progressive distillation process to obtain the student model required in the next round of training.

[0054] The progressive distillation unit is used to create a self-supervised learning framework using a teacher model and a student model. The teacher model generates the labels needed for training the student model. During training, knowledge transfer occurs between the teacher and student models regarding mask features, global triplet information, and labels. The mask generation feature loss, score loss, and label loss are calculated between the teacher and student models. In each round of training, the mask feature generation unit gradually reduces the mask ratio, generating more comprehensive mask features. Simultaneously, in each round of the self-supervised learning framework, the student model from the previous round is used as the teacher model for the next round, and the simplified teacher model is used as the student model for the next round. Ultimately, a high-precision and lightweight student model is trained and used as the required language model.

[0055] The present invention also includes a lightweight knowledge graph completion device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates the language model finally constructed in the aforementioned lightweight language model training system based on the progressive distillation method; and uses the knowledge graph in the language model to complete the features of missing triplet vectors.

[0056] The technical solution provided by this invention has the following beneficial effects:

[0057] This invention uses triples and their entity text descriptions as input features, employing the original BERT-based language model. A progressive distillation strategy is used to simplify the original speech model. This progressive distillation strategy consists of two stages: pre-distillation and progressive distillation. In the pre-distillation stage, a mask-generated feature distillation method is used to improve baseline performance and serve as a teacher model to guide higher-level student models. In the progressive distillation stage, a multi-grade student model is designed, distilling each grade level during the process. Knowledge transfer is performed based on mask-generated feature information and global triple information. Mask-generated feature distillation masks part of the teacher model's input data, prompting the teacher model to generate rich representation information to guide the student model. This achieves knowledge transfer from the teacher model to the student model. Finally, two hyperparameters are used to balance the label loss, mask-generated feature loss, and score loss.

[0058] This invention is very helpful for knowledge graph completion tasks. The model in the pre-distillation stage can significantly improve the accuracy of knowledge graph completion technology without increasing the model load, while the model in the progressive stage can greatly reduce the model parameters while maintaining a certain level of performance. Attached Figure Description

[0059] Figure 1This is a schematic diagram illustrating the principle of the knowledge graph completion method based on Transformer and progressive distillation provided in Embodiment 1 of the present invention.

[0060] Figure 2 This is a flowchart of the steps in the knowledge graph completion method based on Transformer and progressive distillation provided in Embodiment 1 of the present invention.

[0061] Figure 3 This is a flowchart of the progressive distillation training strategy in Embodiment 1 of the present invention.

[0062] Figure 4 This is a schematic diagram illustrating the principle of the self-supervised learning framework constructed in Embodiment 1 of the present invention.

[0063] Figure 5 This is a system architecture diagram of a lightweight language model training system based on the progressive distillation method provided in Embodiment 1 of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0065] Example 1

[0066] This embodiment provides a knowledge graph completion method based on Transformer and progressive distillation, which is used to construct a lightweight and high-precision language model and utilize it to complete the knowledge graph completion task. The knowledge graph completion method based on Transformer and progressive distillation in this embodiment includes two stages: pre-distillation and progressive distillation, such as... Figure 1 As shown, the main idea of ​​the solution is as follows:

[0067] First, the triplet feature vectors derived from entity text descriptions are used as input features of the network model. Then, a sample dataset consisting of massive text data is used to pre-distill the high-precision language model based on BERT-base. In the pre-distillation stage, the triplet vectors in the sample dataset are masked according to the maximum masking rate.

[0068] Next, in the progressive distillation stage, a mask-generated feature distillation method is used to improve the performance of the baseline and serve as the teacher model to guide the advanced student models. In this progressive distillation stage, the teacher model is simplified to obtain multiple student models of different sizes, which can be referred to as advanced student models, intermediate student models, and basic student models. Then, in each training process, the original sample data is used to generate corresponding round sample datasets by gradually decreasing the mask ratio, and these datasets are used to complete knowledge transfer between the teacher and student models. In each round of progressive distillation, the student model from the previous stage either serves as the teacher model for the next stage, while the network size of the student models is progressively reduced, and the mask ratio of the sample datasets used for training is also progressively decreased.

[0069] Ultimately, after multiple rounds of progressive distillation, the accuracy of knowledge graph completion technology can be significantly improved without increasing the model load. The simplified model obtained in the progressive stage can greatly reduce the model parameters while maintaining a certain predictive performance.

[0070] Specifically, such as Figure 2 As shown, the knowledge graph completion method provided in this embodiment includes the following steps:

[0071] S1: Obtain massive amounts of text data for constructing the knowledge graph, and extract statements from the text data that contain head entities, relations, tail entities, and descriptive information for the head and tail entities. The head and tail entities represent two objects involved in a complete statement, the relations represent the association between the two objects, and the descriptive information is supplementary information for the head and tail entities.

[0072] S2: Convert each statement into a corresponding triple vector according to a preset format, thus obtaining a sample dataset containing a large number of triple vectors. The standard format of the triple vector is: [head entity][head entity description][relation][tail entity][tail entity description]. Among them, a triple vector that includes [head entity], [relation], and [tail entity] is a complete vector, and a triple vector that is missing any one of [head entity], [relation], or [tail entity] is a missing vector to be completed.

[0073] S3: Mask the triple vectors in the sample dataset according to the maximum masking rate, and use them to pre-train a high-performance speech model to obtain the required teacher model.

[0074] The high-precision speech model obtained in the current step is mainly used as the initial teacher model required for the subsequent progressive distillation stage. The stronger the performance of the teacher model, the higher the accuracy of the student model obtained in the subsequent knowledge transfer stage can be guaranteed.

[0075] In this embodiment, the speech model is trained using a BERT base (Bidirectional Encoder Representations from Transformers) model containing 12 layers. The BERT base model consists of 12 Transformer encoder layers, 12 attention heads, 768 hidden values, and 110M parameters. The BERT architecture comprises multiple stacked Transformer encoders. Each Transformer encoder encapsulates two sub-layers: a self-attention layer and a feedforward layer.

[0076] S4: Simplify the teacher model, construct a self-supervised learning framework, and then train a simplified model on the sample dataset using a progressive distillation strategy to obtain a model that meets the required accuracy and lightweight requirements. For example... Figure 3 As shown, the training process for progressive distillation is as follows:

[0077] S41: In the initial stage, the speech model is used as the teacher model, and the teacher model in this round is simplified and used as the student model.

[0078] S42: Construct a self-supervised learning framework based on teacher and student models; in this framework, a pre-trained teacher model is used to generate labels to guide the training of the student model. For example... Figure 4 In the progressive distillation strategy, the student model that passed the previous round of training is used as the teacher model for the next round, and the layers of the student model that passed the previous round of training are simplified by three layers and used as the student model for the next round.

[0079] S43: Design a masking module in front of the self-supervised learning framework to mask the input triple vectors according to a preset mask rate.

[0080] S44: Preset loss functions for the training phase, including label loss, mask generation feature loss, and score loss. Loss functions set during the training phase. The expression is as follows:

[0081]

[0082] In the above formula, α and β are the balance coefficients of the loss, respectively; Indicates label loss; This represents the feature loss generated by the mask; This indicates a score loss.

[0083] Among them, score loss The calculation formula is as follows:

[0084]

[0085] In the above formula, score S The score represents the output score of the student model. T This represents the score output of the teacher model; MSE represents the function for calculating the mean squared error.

[0086] Among them, label loss The calculation method is as follows:

[0087] (1) Calculate the scores of the head entity relation vector and tail entity vector output by the student model using cosine similarity, as follows:

[0088]

[0089] In the above formula, e hr Represents the head entity relation vector, e hr = [Head Entity][Head Entity Description][Relationship]; e t Represents the tail entity vector, e t = [Tail Entity][Tail Entity Description];

[0090] (2) Use the cross-entropy function to calculate the label loss of the score from the label and the scoring function.

[0091]

[0092] Here, CrossEntropy represents the cross-entropy function, and L is the label output by the teacher model.

[0093] Mask generation feature loss The calculation method is as follows:

[0094] (1) Perform a masking operation on the triplet vectors input to the teacher model and the student model:

[0095]

[0096] Here, Mask represents a masking operation at a random location. Represents the student feature vector. It is the teacher's feature vector; λ represents the masking rate; and These represent the student feature vector and the teacher feature vector before masking, respectively.

[0097] (2) The feature loss is generated by generating the mask between the teacher model and the student model using the following formula.

[0098]

[0099] S45: Obtain the sample dataset, input it into the masking module and perform random masking according to the preset masking rate, and then synchronously input the processed data into the teacher model and student model.

[0100] In the progressive distillation strategy, the maximum masking rate of the masking module is set to 20%, and the initial masking rate of the masking module is the maximum masking rate. Then, the masking rate is decreased sequentially in 50% increments during each round of self-distillation training. For example, in the second round of training, the masking rate is set to 10%, in the third round to 5%, and so on. The masking rate in the final round is set to 0.

[0101] It is particularly important to emphasize that in the solution of this invention, the mask length corresponding to the minimum masking rate should be greater than the maximum length of the relation vector. That is, in the random masking process, the relation vector should be avoided as much as possible, and only a part of the head entity, head entity description, tail entity or tail entity description should be masked.

[0102] S46: Calculate the loss function value for this round of training, determine whether the accuracy requirement has been met, and make the following decision:

[0103] (1) If so, save the model parameters of the student model at the current stage and use it as a high-level model that can be further simplified.

[0104] (2) If not, save the model parameters of the teacher model at the current stage and output them as the final simplified model.

[0105] S47: Determine whether the advanced model acquired in this round meets the lightweight requirements, and make the following decision:

[0106] (1) If so, the advanced model will be output as the final simplified model.

[0107] (2) If not, further simplify the high-level model and use the resulting low-level model as the student model for the next round, and use the current round's student model as the teacher model for the next round. Reduce the masking rate of the masking model by a preset margin, and finally return to step S45 to continue training the student model using a progressive distillation method.

[0108] S5: Input the triple vector to be completed into the simplified model obtained in the previous step. The simplified model outputs the complete triple vector after semantic completion according to the knowledge graph established in the training phase, and uses it for subsequent applications.

[0109] Example 2

[0110] This embodiment provides a lightweight language model training system based on the progressive distillation method. It employs a knowledge graph completion method based on Transformer and progressive distillation, as described in Embodiment 1, to construct a lightweight language model capable of completing text content with associated information based on a pre-generated knowledge graph. Figure 5 As shown, the lightweight language model training system includes: a text extraction unit, a triple generation unit, a mask feature generation unit, a teacher model creation unit, a model simplification unit, and an asymptotic distillation unit.

[0111] The text extraction unit is used to acquire massive amounts of text data related to the pre-knowledge graph and extract a large number of standardized statements from the text data, including head entities, relations, tail entities, and descriptive information of head entities and tail entities.

[0112] The triple generation unit is used to convert each statement into a corresponding triple vector according to a preset format, thereby obtaining a sample dataset containing a large number of triple vectors.

[0113] The mask feature generation unit is used to randomly mask the input triple vector according to a preset mask rate to obtain mask features with missing information; the mask features with different mask rates are used as inputs to the teacher model and student model at different stages of the progressive distillation process.

[0114] The teacher model creation unit is used to pre-train a Transformer-based neural network using the acquired complete sample dataset, thereby obtaining a large-scale, high-performance speech model, which serves as the initial teacher model in the progressive distillation process.

[0115] The model simplification element is used to simplify the teacher model in each round of the progressive distillation process to obtain the student model required in the next round of training.

[0116] The progressive distillation unit is used to create a self-supervised learning framework using a teacher model and a student model. The teacher model generates the labels needed for training the student model. During training, knowledge transfer occurs between the teacher and student models regarding mask features, global triplet information, and labels. The mask generation feature loss, score loss, and label loss are calculated between the teacher and student models. In each round of training, the mask feature generation unit gradually reduces the mask ratio, generating more comprehensive mask features. Simultaneously, in each round of the self-supervised learning framework, the student model from the previous round is used as the teacher model for the next round, and the simplified teacher model is used as the student model for the next round. Ultimately, a high-precision and lightweight student model is trained and used as the required language model.

[0117] Example 3

[0118] This embodiment provides a lightweight knowledge graph completion device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates the language model finally constructed in the lightweight language model training system based on the progressive distillation method of Embodiment 2; and uses the knowledge graph in the language model to complete the features of missing triplet vectors.

[0119] The knowledge graph completion device provided in this embodiment is essentially a computer device for data processing and instruction generation, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer device provided in this embodiment can be a smart terminal, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. The computer device in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus.

[0120] In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as the hard disk or RAM of the computer device.

[0121] In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0122] In some embodiments, a processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device. In this embodiment, the processor is used to run program code stored in memory or process data.

[0123] Simulation Experiment

[0124] To verify the performance of the knowledge graph completion method based on Transformer and progressive distillation provided in this invention, this experiment selected the BERT base model as the original high-performance language model and conducted experiments on the public datasets WN18RR and FB15K237.

[0125] The experiment consisted of one round of pre-distillation and four rounds of progressive distillation. In the pre-distillation phase, the mask ratio of the dataset was set to 20%, resulting in a 12-layer teacher model. In the progressive distillation phase, the mask ratios were set to 20%, 10%, 5%, and 0, respectively, resulting in 12, 9, 6, and 3-layer student models. This experiment also used baseline networks of different sizes trained directly on the corresponding datasets as control groups for comparison with the proposed method (PMD).

[0126] To quantitatively evaluate the knowledge graph completion results, MRR (Mean Reciprocal Rank), the average inverse rank of test triples, and Hits@k (k belongs to {1,3,10}), the proportion of correct entities ranking in the top K, were used as evaluation metrics. The final simulation experimental data are shown in Table 1.

[0127] Table 1: Model simplification effect of the present invention on WN18RR and FB15K237

[0128]

[0129] The table above shows the test results of the BERT base model trained using the progressive distillation process of this invention on the WN18RR and FB15K237 datasets. "Baseline" represents the original network, and "PMD" represents the result after progressive distillation. Analysis of the data in the table shows that the model using progressive distillation exhibits a significant performance improvement compared to the baseline system. This is because the teacher model suffers from limited representation information. The progressive distillation method of this invention, by combining a mask-based feature distillation strategy and fine-grained reduction of model parameters, transfers knowledge from the teacher model to the student model, enabling the three-layer model to learn useful knowledge from the teacher model. This significantly reduces the number of parameters while maintaining a certain level of performance.

[0130] Furthermore, these results demonstrate the effectiveness of the proposed solution. The experimental results actually prove that, based on the model in the pre-distillation stage, the present invention, after progressive distillation, can achieve performance exceeding that of the teacher model without increasing the model load.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A knowledge graph completion method based on Transformer and progressive distillation, characterized in that, It is used to construct a lightweight and high-precision language model and to complete the knowledge graph completion task; the knowledge graph completion method includes the following steps: S1: Obtain massive amounts of text data for building knowledge graphs, and extract statements from the text data that contain head entities, relations, tail entities, and descriptive information of head entities and tail entities; S2: Convert each statement into a corresponding triple vector according to a preset format, thereby obtaining a sample dataset containing a large number of triple vectors; S3: Mask the triplet vectors in the sample dataset according to the maximum masking rate, and use them to pre-train a high-performance speech model to obtain the required teacher model; S4: Simplify the teacher model, construct a self-supervised learning framework, and then train a simplified model on the sample dataset using a progressive distillation strategy to obtain a model that meets the required accuracy and lightweight requirements. The process is as follows: S41: In the initial stage, the teacher model of this round is simplified and used as the student model; S42: Construct a self-supervised learning framework based on a teacher model and a student model; in the self-supervised learning framework, a pre-trained teacher model is used to generate labels to guide the training of the student model; S43: Design a masking module in front of the self-supervised learning framework to mask the input triple vector according to a preset masking rate; S44: Preset loss function for the training phase, the loss function including label loss, mask generation feature loss and score loss; S45: Obtain the sample dataset, input it into the masking module for random masking according to the preset masking rate, and synchronously input the processed data into the teacher model and student model; S46: Calculate the loss function value for this round of training, determine whether the accuracy requirement has been met, and make the following decision: (1) If so, save the model parameters of the student model at the current stage and use it as a high-level model that can be further simplified; (2) If not, save the model parameters of the teacher model at the current stage and output them as the final simplified model; S47: Determine whether the advanced model acquired in this round meets the lightweight requirements, and make the following decision: (1) If so, the advanced model is output as the final simplified model; (2) If not, the high-level model is further simplified, and the resulting low-level model is used as the student model for the next round. The student model of the current round is used as the teacher model for the next round, and the masking rate of the masking module is reduced by a preset amount. Finally, return to step S45 and continue to train the student model by using a progressive distillation method. S5: Input the triple vector to be completed into the simplified model obtained in the previous step. The simplified model outputs the complete triple vector after semantic completion according to the knowledge graph established in the training phase, and uses it for subsequent applications.

2. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 1, characterized in that: In step S1, the head entity and tail entity represent two objects involved in a complete statement, and the relationship represents the association between the two objects; the description information is supplementary information for the head entity and tail entity.

3. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 1, characterized in that: In step S2, the standard format of the triple vector is: [head entity][head entity description][relation][tail entity][tail entity description]; wherein, a triple vector that includes [head entity], [relation], and [tail entity] is a complete vector, and a triple vector that is missing any one of [head entity], [relation], or [tail entity] is a missing vector to be completed.

4. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 1, characterized in that: In step S3, the language model is trained using a BERT base model containing 12 layers.

5. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 1, characterized in that: In step S4, in the progressive distillation strategy, the maximum masking rate of the masking module is set to 20%, and it is gradually reduced by 50% in each round of self-distillation training. At the same time, the student model that passed the previous round of training is used as the teacher model for the next round, and the layer of the student model that passed the training is simplified by three layers and used as the student model for the next round.

6. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 5, characterized in that, In step S44, the loss function is set during the training phase. The expression is as follows: In the above formula, and These are the loss balance coefficients; Indicates label loss; This represents the feature loss generated by the mask; Indicates score loss; Wherein, the score loss The calculation formula is as follows: In the above formula, This represents the score output of the student model. This represents the score output of the teacher model.

7. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 6, characterized in that: The label loss The calculation method is as follows: (1) Calculate the scores of the head entity relation vector and tail entity vector output by the student model using cosine similarity. The formula is as follows: In the above formula, Represents the head entity relationship vector. =[Header Entity][Header Entity Description][Relationship]; Represents the tail entity vector. =[Tail Entity][Tail Entity Description]; (2) Use the cross-entropy function to calculate the label loss of the score of the label and the scoring function. : in, Represents the cross-entropy function. L These are the labels output by the teacher model.

8. The knowledge graph completion method based on Transformer and progressive distillation as described in claim 7, characterized in that: The mask generation feature loss The calculation method is as follows: (1) Perform a masking operation on the triplet vectors input to the teacher model and the student model: Here, Mask represents a masking operation at a random location. Represents the student feature vector. It is the teacher's feature vector; Indicates the mask rate; and These represent the student feature vector and the teacher feature vector before masking, respectively. (2) The feature loss is generated by generating the mask between the teacher model and the student model using the following formula. : In the above formula, MSE represents the function for calculating the mean squared error.

9. A lightweight training system for language models based on a progressive distillation method, characterized in that, It is used to construct a lightweight language model capable of completing text content based on pre-generated knowledge graphs using the knowledge graph completion method based on Transformer and progressive distillation as described in any one of claims 1-8; the lightweight language model training system includes: The text extraction unit is used to acquire massive amounts of text data related to the pre-knowledge graph and extract a large number of normalized statements from the text data, which contain head entities, relations, tail entities, and descriptive information of head entities and tail entities. The triplet generation unit is used to convert each statement into a corresponding triplet vector according to a preset format, thereby obtaining a sample dataset containing a large number of triplet vectors. The mask feature generation unit is used to randomly mask the input triple vector according to a preset mask rate to obtain mask features with missing information; the mask features with different mask rates are used as inputs to the teacher model and student model at different stages of the progressive distillation process; The teacher model creation unit is used to pre-train a ransformer-based neural network using the acquired complete sample dataset to obtain a large-scale, high-performance speech model, which serves as the initial teacher model in the progressive distillation process. The model simplification element is used to simplify the teacher model in each round of the progressive distillation process to obtain the student model required in the next round of training. The progressive distillation unit is used to create a self-supervised learning framework using a teacher model and a student model. The teacher model generates the labels needed for training the student model. During training, knowledge transfer occurs between the teacher and student models, focusing on mask features, global triplet information, and labels. The unit also calculates the mask generation feature loss, score loss, and label loss between the teacher and student models. In each round of training, the mask feature generation unit gradually reduces the mask ratio, generating more comprehensive mask features. Simultaneously, in each round of the self-supervised learning framework, the student model from the previous round is used as the teacher model for the next round, and the simplified teacher model is used as the student model for the next round. Ultimately, a high-precision and lightweight student model is trained and used as the desired language model.

10. A lightweight knowledge graph completion device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it creates the language model finally constructed in the language model lightweight training system based on the progressive distillation method as described in claim 9; and uses the knowledge graph in the language model to complete the features of the missing triple vectors.

Images