A zero-shot knowledge graph completion method based on pre-trained language model

Abstract

The present invention discloses a zero-shot knowledge graph completion method based on a pre-trained language model, which specifically includes the following steps: 1. knowledge graph data preprocessing, sampling a zero-shot knowledge graph completion dataset from the knowledge graph, and further obtaining description sentences of relationships and triple text sentences consisting of entity and relationship text information in all triples of relationships; 2. fine-tuning using the pre-trained language model Sentence-BERT to learn vector representations of relationship description sentences and triple text sentences; 3. verification using a zero-shot knowledge graph validation set to adjust to optimal network parameters; 4. testing using a zero-shot knowledge graph test set to statistically analyze the model effect; this scheme fully utilizes the capabilities of the pre-trained language model and can achieve the same effect as the method that requires additional use of neighbor entity information.

Description

Technical Field

[0001] The present invention relates to a zero-shot knowledge graph completion method, and specifically to a zero-shot knowledge graph completion method based on a pre-trained language model. Background Art

[0002] A knowledge graph is a collection of triples, each of which consists of a head entity h, a relation r, and a tail entity t. Existing knowledge graph datasets include Freebase, Yago, and WordNet. Knowledge graphs are widely used in applications such as question answering, information retrieval, recommendation systems, and natural language processing. Although knowledge graphs contain a large number of entities and relations, they are still incomplete. To complete knowledge graphs, many studies map entities and relations into a latent space and then use a scoring function to score triples to distinguish between positive and negative samples. However, for unseen relations, these relations usually do not have enough training examples, and previous methods cannot solve this problem well. Therefore, the knowledge graph completion task requires a model that can recognize unknown relations without seeing any related facts.

[0003] 1. Small Sample Knowledge Graph Completion Method

[0004] The goal of knowledge graph completion is to complete missing triples in a knowledge graph. Few-shot knowledge graph completion is the task of completing a knowledge graph using a small number of relevant examples. Zero-shot knowledge graph completion also falls under the category of few-shot knowledge graph completion. The Gmatching model is the first study of few-shot relation learning tasks in knowledge graphs. This algorithm first builds a matching network to learn entity embeddings and one-hop neighbor information, and then identifies unknown relations by calculating the matching score between the query triple and a given instance. The MetaR model introduces metadata to address the problem of few-shot knowledge graph completion. By designing relational and gradient meta-elements, it preserves prior knowledge and improves model fitting speed. The CogKR model builds a cognitive graph by aggregating evidence from multiple paths and inferring facts through these paths. These methods perform well when the unknown relations have relevant training examples, but without any training examples, they cannot predict unseen relations.

[0005] 2. Zero-Shot Knowledge Graph Completion Method

[0006] Existing zero-shot knowledge graph completion methods are all based on the idea of zero-shot learning. The mainstream method of zero-shot learning is to transform the traditional zero-shot learning problem into a supervised learning problem. Among them, the generative model-based method uses a generative adversarial network to synthesize training data and generate features of unseen classes based on the semantic description of unseen classes.

[0007] The ZSGAN model is the first zero-shot relationship learning study in knowledge graphs. It uses a generative adversarial network to generate relational embeddings for unseen relationships. Relational description sentences are generated using the TF-IDF algorithm to generate sentence features, while entity pair features are extracted from supporting candidate entity pairs and neighboring entity pairs using a feature extraction algorithm. The SEGAN model uses a pre-trained structural encoder to generate a structured representation of the relationship, which in turn guides the generator to generate a relationship description sentence representation that integrates the structured embedding of the knowledge graph to address the discrepancy between the textual description of the relationship and the knowledge graph embedding. Furthermore, an entity encoder combining an attention mechanism and convolution operations is used to capture the global features of entities and the interactions between entity pairs to enhance the features of entity pairs. All of these methods require additional neighboring entity information to solve the zero-shot knowledge graph completion problem.

[0008] Research has shown that in the case of small-shot learning, pre-trained language models have accumulated a large amount of background knowledge and language representation capabilities through the pre-training phase, and can better understand and infer the meaning of the input text. However, the pre-trained language model approach has not been used in the field of zero-shot knowledge graph completion. Following the idea of zero-shot learning, external knowledge is used to predict unknown relationships, transforming the zero-shot knowledge graph completion problem into a supervised learning problem. During the training phase, vector representations of relationship description sentences and triple text sentences in the same vector space are learned. During the testing phase, the nearest point search method in the semantic vector space is used to obtain the candidate triple text sentence that is closest to the relationship description sentence vector. In other words, the model assigns a higher similarity to the triple text sentence containing the correct entity among many candidate entities. Summary of the Invention

[0009] In order to fill the gap in the field of zero-shot knowledge graph completion based on pre-trained language models, the present invention proposes a zero-shot knowledge graph completion method based on a pre-trained language model. The main feature of this method is that it uses the text information of entities and relationships and the description information of relationships through a pre-trained language model to solve the zero-shot knowledge graph completion problem. This method regards the relationship description sentences in the knowledge graph and the triple text sentences corresponding to the relationship as similar text sequences as positive samples, and regards the relationship description sentences and other candidate entity triple text sentences as dissimilar text sequences as negative samples. The pre-trained Sentence-BERT language model is used to calculate the similarity between unknown relationship description sentences and candidate triple text sentences, thereby sorting and obtaining predicted entities.

[0010] To implement the above method, the technical solution adopted by the present invention is: a zero-shot knowledge graph completion method based on a pre-trained language model, the specific steps are as follows:

[0011] Step 1: First, preprocess the knowledge graphs of different fields into eight files. The processed files include a knowledge graph relationship training set, a knowledge graph relationship verification set, a knowledge graph relationship test set, a knowledge graph triple file, an entity text file, a relationship text file, a relationship description text file, and a relationship candidate entity file.

[0012] Step 2: Construct corresponding text files for the knowledge graph relationship training set, knowledge graph relationship verification set, and knowledge graph relationship test set obtained in step 1. Use the relationship candidate entity file obtained in step 1 to construct candidate triples for the triples corresponding to the relationships contained in the knowledge graph relationship training set, verification set, and test set. Traverse the knowledge graph triple files obtained in step 1 to ensure that the candidate triples have not appeared in the knowledge graph. Finally, construct the relationship description sentences and candidate triple text sentences contained in the knowledge graph relationship training set, verification set, and test set into corresponding text sentence pairs.

[0013] Step 3: Input the text sentence pairs of the knowledge graph relationship training set obtained in step 2 into the Sentence-BERT model, directly input the sentence pairs into the Tokenizer layer, use BertTokenizer to divide the text sentences into several subwords, obtain Input Embedding, send it to the BertModel model, send the output word vector to the pooling layer, use the average pooling strategy, obtain the sentence vector with a vector dimension of 384, perform L2 normalization, and output the normalized sentence vector.

[0014] Step 4: Calculate the cosine distance between the sentence vectors of the positive sample sentence pair and the negative sample sentence pair output by the model in step 3. Select the sentence pairs for training in each batch according to the strategy of selecting relatively difficult sentence pair labels. Calculate the loss through the loss function and use the loss of each batch data for iterative optimization.

[0015] Step 5: Use the knowledge graph validation set to verify and adjust to the optimal network parameters.

[0016] Step 6: Test the knowledge graph test set, collect test results, and evaluate the model using MRR, HIT1, HIT5, and HIT10 evaluation indicators.

[0017] The preprocessing in step 1 is as follows:

[0018] We sampled the relationships and their triplets from the knowledge graph dataset and divided the relationships in the NELL knowledge graph into training, validation, and test sets in a ratio of 139:10:32. Each set contains the divided relationships and their corresponding triplets. We also counted the triplets of all relationships in the training, validation, and test sets and saved them in the knowledge graph triple files.

[0019] Obtain the textual statements of entities and relationships in the dataset and the description statements of each relationship through the knowledge graph information, and save them in entity text files, relationship text files, and relationship description text files in sequence;

[0020] The knowledge graph triple file is used to count the tail entity types of all triples under each relationship, and the entities under all tail entity types in the triples corresponding to the relationship are taken as candidate entities and saved in the relationship candidate entity file.

[0021] The text sentence pairs corresponding to the dataset constructed in step 2 are as follows:

[0022] For each triple of all relations in the training set, candidate entities are sampled from the relation candidate entity file. The number of candidate entities sampled for each triple is 100. These candidate entities are grouped into candidate triples with the same head entity and relation. The knowledge graph triple file is traversed to ensure that the candidate triples do not appear in the knowledge graph.

[0023] For each true triple and candidate triple under each relation, according to the positions of head_entity_text, relation_text, and tail_entity_text in the sentence template "relation_textdescribe the relation of head_entity_text and tail_entity_text", the corresponding content is obtained from the entity text file and the relationship text file, and filled into the template to form a candidate triple text sentence. The relationship description sentence and the true triple text sentence are used as positive sample sentence pairs, and the relationship description sentence and the candidate triple text sentence are used as negative sample sentence pairs.

[0024] Each triple of all relations in the validation set is sampled from the relation candidate entity file, with the number of candidate entities sampled for each triple being 50. Then, positive and negative sample text sentence pairs are constructed according to the training set operation process.

[0025] For each triple of all relations in the test set, all candidate entities of each triple are sampled from the relation candidate entity file, and then positive and negative sample text sentence pairs are constructed according to the training set operation process.

[0026] Among them, in step 3, the Sentence-BERT model is input to obtain the sentence vector. The specific operation is as follows: the text sentence pair of the input model is passed through the Tokenizer layer, and the WordPiece algorithm of BertTokenizer is used to segment the words, and further obtain word_embedding, position_embedding and token_type_embedding, which respectively represent the word embedding corresponding to the subword, the position embedding of each word in the sentence, and the paragraph embedding of the sentence where each word is located. The three constitute the Input Embedding of the model, which is sent to the BertModel model, and passes through the BertEmbeddings layer, the BertEncoder layer composed of 6 layers of BertLayer, and the BertPooler layer in sequence. Finally, a 384-dimensional word vector is output, and the average pooling strategy is used to output a 384-dimensional sentence vector. L2 normalization is used to obtain two sentence vectors.

[0027] The strategy for determining the cosine distance and the label selection for the relatively difficult sentence pair in step 4 is as follows: Calculate the cosine distance between the two sentence vectors output by the model. The value range is [0, 2], representing the semantic similarity between two sentences of different lengths. The larger the cosine distance, the lower the semantic similarity between the two sentences, and the smaller the cosine distance, the higher the semantic similarity between the two sentences. The cosine distance is:

[0028]

[0029] Set the cosine distances of the positive and negative sample sentence pairs to poss and negs respectively. In the same batch, there are multiple elements in poss and negs. The strategy for selecting training samples is:

[0030] pos_pairs=poss if(poss>min(negs)) (2)

[0031] neg_pairs=negs if(negs<max(poss)) (3)

[0032] The selection strategy is to select samples whose cosine distance is greater than the minimum cosine distance among the negative sample sentence pairs for the positive sample sentence pairs poss, to ensure that the cosine distance of the positive sample sentence pairs is relatively large, so that the model can learn the positive sample sentence pairs pos_pairs with relatively low semantic similarity; for the negative sample sentence pairs negs, select samples whose cosine distance is less than the maximum cosine distance among the positive sample sentence pairs, to ensure that the cosine distance of the negative sample sentence pairs is relatively small, so that the model can learn the negative sample sentence pairs neg_pairs with relatively high semantic similarity.

[0033] The loss function in step 4 is as follows: in the same batch, sentence pairs with relatively difficult label determination are selected as training sentence pairs. The positive sample sentence pairs are selected with relatively large cosine distances, and the negative sample sentence pairs are selected with relatively small cosine distances. The loss function of the positive and negative sample pairs is:

[0034]

[0035] Where m represents the number of elements in pos_pairs, n represents the number of elements in neg_pairs, γ is a fixed margin, and ReLu() is the ReLu activation function, which is expressed as follows:

[0036] ReLu(x)=max(0,x) (5)

[0037] In the data preprocessing stage, this scheme first divides the relations and their triples to obtain a zero-sample knowledge graph dataset. Secondly, the text information of entities and relations is obtained through the knowledge graph information. The entity type of each entity is obtained using the knowledge graph information. The tail entities of all triples under the relationship are traversed, the tail entity types are counted, and the entities of these tail entity types are used as candidate entities of the relationship. Finally, the real triples and the sampled candidate triples are constructed into triple text sentences, and the natural language template is used to convert the triples into more natural and fluent sentences. The natural language template uses "relation_text describe the relation ofhead_entity_textand tail_entity_text." After filling in the corresponding entity and relationship information, a natural and fluent triple text sentence is constructed, such as "concept agricultural product cut into geometric shape describe the relationof concept agricultural product mango and concept geometric shape cubes."

[0038] This solution uses the all-MiniLM-L6-v2 model, a language model pre-trained with Sentence-BERT. This model is pre-trained on over 1 billion training pairs and is designed as a general-purpose model. Although the all-mpnet-base-v2 model provides the best quality, the all-MiniLM-L6-v2 model is five times faster than the former, but still provides good performance.

[0039] This solution uses OnlineContrastiveLoss as the loss function. The model input is a sentence pair consisting of a relationship description sentence and a triple text sentence corresponding to the relationship, but there is no corresponding similarity. Therefore, the ContrastiveLoss series is used. The input of this series is two text sentences and a label. If the label is equal to 1, the model will reduce the distance between the two text sentences. If the label is equal to 0, the model will increase the distance between the two text sentences. The OnlineContrastiveLoss loss function selects the sentence pairs for training in each batch according to the strategy of relatively difficult sentence pair label determination.

[0040] The present invention introduces a pre-trained language model into the field of zero-shot knowledge graph completion, trains and fine-tunes the pre-trained language model, and achieves a huge improvement compared to the direct prediction of the original pre-trained language model. It can achieve the effect of a zero-shot knowledge graph generation model that requires additional use of neighbor entity information, filling the gap in the field of zero-shot knowledge graph completion based on pre-trained language models. The present invention uses a pre-trained language model to regard relationship description sentences and triple text sentences as similar text sequences, and converts the zero-shot knowledge graph completion problem into a supervised learning problem. The present invention uses the text information of the knowledge graph in the model, which can learn the potential information of entities and relationships compared to the existing technology, and is used to improve the completion performance of the model. It also introduces the idea of comparative learning, based on the OnlineContrastiveLoss loss function, to shorten the semantic distance between semantically similar text sentences, enhance the semantic features of triple text sentences, so that zero-shot knowledge graph completion is not limited to the structural information of the original knowledge graph, effectively utilizes the text information in the knowledge graph, improves the zero-shot knowledge graph completion effect, and can achieve the same effect as the method that requires additional use of neighbor entity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall flow chart of the present invention;

[0042] Figure 2 Schematic diagram of the Sentence-BERT model;

[0043] Figure 3 Schematic diagram of the strategy for selecting training samples within a batch. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0045] The zero-shot knowledge graph completion method based on the pre-training model proposed in this invention includes a training part and a testing part. Figure 1As shown in the figure, the network needs to be trained and fine-tuned first. After the training is completed, the relationship description sentences in the test set and all candidate triple text sentences of all triples under the relationship are input into the model to obtain the vector representation of the relationship description sentences and the candidate triple text sentences, calculate the similarity and sort them, and the ranking of the candidate triple text sentences obtained by sorting is the ranking of the candidate entities, and the test results of the model are obtained by statistics.

[0046] Example: Taking the NELL-ZS dataset as an example, zero-sample knowledge graph completion is performed on it to illustrate the steps of the present invention in detail.

[0047] Experimental conditions: A computer with an Intel(R) processor (3.2GHz) and 124GB of random access memory (RAM), an Ubuntu 14.04 64-bit operating system, and an NVIDIA GTX1080ti (12GB) graphics card was selected for network training. The software environment was the deep learning framework PyTorch 1.10.1.

[0048] Experimental Subjects: The training, validation, and test sets are all from the NELL-ZS dataset. These three sets are strictly separated, with a ratio of 139:10:32 between the training, validation, and test sets. This ensures that the intersection of the relationships among the training, validation, and test sets is empty. NELL-ZS is a subset of the knowledge graph NELL. ZS indicates that this dataset is a zero-shot dataset. It contains 65,567 entities and 188,392 triples.

[0049] The specific structure of the zero-shot knowledge graph completion model based on the pre-trained language model is shown in Table 1:

[0050] Table 1 Network structure of zero-shot knowledge graph completion model based on pre-trained language model

[0051]

[0052] (1) Training phase

[0053] Step 1: Construct the dataset. The training set used in this tutorial contains 139 relations, each of which has multiple triples. The total number of triples in the training set is 181,053. To construct triple text sentences, we use the relevant information in the knowledge graph to obtain the text information of all entities and relations appearing in the dataset. We also use the names of entities in the NELL dataset to obtain the entity types. For example, the entity name "concept:date:sept__29" indicates that the entity belongs to the date type. After obtaining the entity type, we count all entities of the type of the tail entity in all triples under each relation as candidate entities for that relation.

[0054] For each triple under each relation, the relation description statement and the triple text statement constructed according to the template are used as positive samples of the model input, and their labels are set to 1.

[0055] For each triple under each relationship, the tail entity is replaced by the entity in the relationship candidate entity to form 100 candidate triples. In the process of constructing the candidate triples, it is necessary to ensure that the 100 generated candidate triples do not appear in the knowledge graph. Finally, the relationship description sentence and the candidate triple text sentence constructed according to the template are used as negative samples of the model input, and their labels are set to 0.

[0056] There are 10 relations in the validation set, including 1856 triplets. The same construction method as the training set is used, but only 50 candidate triplets are generated for each triple under each relation.

[0057] There are 32 relations in the test set, including 5483 triples. The construction method is the same as that of the training set, but for each triple under each relation, all candidate entities under the relation are generated into candidate triples.

[0058] Step 2: Fine-tune the pre-trained language model. Following the process of step 1, the training set is processed into multiple text sentence pairs and their labels, and then fed into the Sentence-BERT model, as shown in the following example: Figure 2 As shown in the figure, for each text sentence pair, it is input into the two parameter-sharing BERT models. All the word vectors of the BERT model output sentence are passed to the pooling layer for average pooling, that is, the average of all word vectors is calculated to obtain the sentence vector representation v and ν of each sentence. The dimension is 384. The sentence vector is L2 normalized to obtain the final sentence vector. The cosine similarity is calculated for these two vectors:

[0059]

[0060] Considering that the relationship description sentence and the candidate triple text sentence may have a certain degree of semantic similarity, in order to allow the model to learn more useful information, this description uses OnlineContrastiveLoss as the loss function. First, the cosine distance of the text sentence pair is calculated:

[0061] dist(υ,v)=1-cosθ (7)

[0062] The cosine distance ranges from [0, 2]. The larger the cosine distance, the lower the similarity between the sentence pairs. When a pair of text sentences differs significantly in length but has similar content, their Euclidean distance in feature space is typically large if word frequency or word vectors are used as features. However, using the cosine distance, the angle between them may be small. The smaller the cosine distance, the lower the semantic similarity between the text sentence pairs.

[0063] In the same batch, the cosine distance of the text sentence pairs is calculated according to the above steps, and the sentences are divided into positive sample sentence pairs and negative sample sentence pairs according to whether the label is 1 or 0, such as Figure 3 As shown in Figure 2, there are three situations in the sample selection strategy for positive and negative sample sentence pairs in the same batch:

[0064] In the first case, both positive and negative sentence pairs retain the shaded samples for training. For positive sentence pairs, retain the samples whose cosine distance is greater than the minimum cosine distance for the negative sentence pair; for negative sentence pairs, retain the samples whose cosine distance is less than the maximum cosine distance for the positive sentence pair. In other words, select positive and negative sentence pairs that are difficult to distinguish for training.

[0065] In the second case, all samples of both positive and negative sentence pairs are retained as training samples. The cosine values of all samples of the positive sentence pair are greater than the minimum cosine distance of the negative sentence pair, and all samples of the negative sentence pair are less than the maximum cosine distance of the positive sentence pair.

[0066] In the third case, all positive and negative sentence pairs do not need to be used as training samples. The cosine distances of all positive sentence pairs are less than the minimum cosine distance of the negative sentence pairs, and the cosine distances of all negative sentence pairs are greater than the maximum cosine distance of the positive sentence pairs, indicating that the model has correctly distinguished the positive and negative sentence pairs.

[0067] Taking a batch as an example, the cosine distance of the positive and negative sample sentence pairs is obtained after passing:

[0068] poss=dist7=[0.5857,0.5599,0.8024,0.5711,0.5025,0.5329,0.5237]

[0069] negs=dist 57 =[0.5936,0.7566,0.8807,0.5753,......,0.3910,0.4813]

[0070] The above positive and negative sample sentence pairs and negative sample sentence pairs are processed as follows Figure 3The processing shown in the figure shows that the maximum cosine distance between the positive and negative sample sentences is 0.8024, and the minimum cosine distance between the negative sample sentences is 0.3910, resulting in the following positive and negative sample sentence pairs:

[0071] pos_pairs=dist7=[0.5857,0.5599,0.8024,0.5711,0.5025,0.5329,0.5237]

[0072] neg_pairs=dist 53 =[0.5936,0.7566,0.5753,0.6906,......,0.3910,0.4813]

[0073] After determining the training samples, the cosine distance between positive and negative samples is used to define the loss function as follows, where γ is a fixed margin:

[0074]

[0075] Step 3: Construct the network. Stack the different layers of the network as shown in Table 1, using the network architecture of the Sentence-BERT model. During training, use AdamW optimization, set the initial learning rate to 0.00001, the batch size to 64, a fixed margin γ to 0.5, and the warmup_steps to 5485906, which is 1% of the total training data. Finally, train for three epochs.

[0076] (2) Testing phase

[0077] After training, the model is saved and used to encode the unseen relationship description sentences and all candidate triple text sentences in all triples under each relationship. The vector representation of the two sentences in the text sentence pair is obtained, the cosine distance between the vectors is calculated, and they are sorted. The ranking of the candidate triple text sentences obtained by sorting represents the ranking of the candidate entities, thereby calculating the MRR, HIT1, HIT5, and HIT10 evaluation indicators.

[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A zero-shot knowledge graph completion method based on a pre-trained language model, characterized in that: The specific steps are as follows: Step 1: First, preprocess the knowledge graphs of different fields into eight files. The processed files include a knowledge graph relationship training set, a knowledge graph relationship verification set, a knowledge graph relationship test set, a knowledge graph triple file, an entity text file, a relationship text file, a relationship description text file, and a relationship candidate entity file. Step 2: Construct corresponding text files for the knowledge graph relationship training set, knowledge graph relationship verification set, and knowledge graph relationship test set obtained in step 1. Use the relationship candidate entity file obtained in step 1 to construct candidate triples for the triples corresponding to the relationships contained in the knowledge graph relationship training set, verification set, and test set. Traverse the knowledge graph triple files obtained in step 1 to ensure that the candidate triples have not appeared in the knowledge graph. Finally, construct the relationship description sentences and candidate triple text sentences contained in the knowledge graph relationship training set, verification set, and test set into corresponding text sentence pairs. Step 3: Input the text sentence pairs of the knowledge graph relationship training set obtained in step 2 into the Sentence-BERT model, directly input the sentence pairs into the Tokenizer layer, use BertTokenizer to divide the text sentences into several subwords, obtain Input Embedding, send it to the BertModel model, send the output word vector to the pooling layer, use the average pooling strategy, obtain the sentence vector with a vector dimension of 384, perform L2 normalization, and output the normalized sentence vector. Step 4: Calculate the cosine distance between the sentence vectors of the positive sample sentence pair and the negative sample sentence pair output by the model in step 3. Select the sentence pairs for training in each batch according to the strategy of selecting relatively difficult sentence pair labels. Calculate the loss through the loss function and use the loss of each batch data for iterative optimization. Step 5: Use the knowledge graph validation set to verify and adjust to the optimal network parameters. Step 6: Test the knowledge graph test set, collect test results, and evaluate the model using MRR, HIT1, HIT5, and HIT10 evaluation indicators.

2. A zero-shot knowledge graph completion method based on a pre-trained language model according to claim 1, characterized in that: The preprocessing in step 1 is as follows: We sampled the relationships and their triplets from the knowledge graph dataset and divided the relationships in the NELL knowledge graph into training, validation, and test sets in a ratio of 139:10:

32. Each set contains the divided relationships and their corresponding triplets. We also counted the triplets of all relationships in the training, validation, and test sets and saved them in the knowledge graph triple files. Obtain the textual statements of entities and relationships in the dataset and the description statements of each relationship through the knowledge graph information, and save them in entity text files, relationship text files, and relationship description text files in sequence; The knowledge graph triple file is used to count the tail entity types of all triples under each relationship, and the entities under all tail entity types in the triples corresponding to the relationship are taken as candidate entities and saved in the relationship candidate entity file.

3. The zero-shot knowledge graph completion method based on a pre-trained language model according to claim 1, characterized in that: The text sentence pairs corresponding to the dataset are constructed in step 2, as follows: For each triple of all relations in the training set, candidate entities are sampled from the relation candidate entity file. The number of candidate entities sampled for each triple is 100. These candidate entities are grouped into candidate triples with the same head entity and relation. The knowledge graph triple file is traversed to ensure that the candidate triples do not appear in the knowledge graph. For each true triple and candidate triple under each relation, according to the positions of head_entity_text, relation_text, and tail_entity_text in the sentence template "relation_textdescribe the relation of head_entity_text and tail_entity_text", the corresponding content is obtained from the entity text file and the relationship text file, and filled into the template to form a candidate triple text sentence. The relationship description sentence and the true triple text sentence are used as positive sample sentence pairs, and the relationship description sentence and the candidate triple text sentence are used as negative sample sentence pairs. Each triple of all relations in the validation set is sampled from the relation candidate entity file, with the number of candidate entities sampled for each triple being 50. Then, positive and negative sample text sentence pairs are constructed according to the training set operation process. For each triple of all relations in the test set, all candidate entities of each triple are sampled from the relation candidate entity file, and then positive and negative sample text sentence pairs are constructed according to the training set operation process.

4. The zero-shot knowledge graph completion method based on a pre-trained language model according to claim 1, characterized in that: In step 3, the Sentence-BERT model is input to obtain the sentence vector. The specific operation is as follows: the text sentence pair of the input model is passed through the Tokenizer layer, and the word is segmented by the WordPiece algorithm of BertTokenizer, and word_embedding, position_embedding and token_type_embedding are further obtained, which respectively represent the word embedding corresponding to the subword, the position embedding of each word in the sentence, and the paragraph embedding of the sentence where each word is located. The three constitute the InputEmbedding of the model, which is sent to the BertModel model, and passes through the BertEmbeddings layer, the BertEncoder layer composed of 6 layers of BertLayer, and the BertPooler layer in turn, and finally outputs a 384-dimensional word vector. The average pooling strategy is used to output a 384-dimensional sentence vector, and L2 normalization is used to obtain two sentence vectors.

5. The zero-shot knowledge graph completion method based on a pre-trained language model according to claim 1, characterized in that: The strategy for determining the cosine distance and the label selection for the relatively difficult sentence pair in step 4 is as follows: Calculate the cosine distance between the two sentence vectors output by the model. The value range is [0, 2], representing the semantic similarity between two sentences of different lengths. The larger the cosine distance, the lower the semantic similarity between the two sentences, and the smaller the cosine distance, the higher the semantic similarity between the two sentences. The cosine distance is: Set the cosine distances of the positive and negative sample sentence pairs to poss and negs respectively. In the same batch, there are multiple elements in poss and negs. The strategy for selecting training samples is: pos_pairs=poss if(poss>min(negs)) (2) neg_pairs=negs if(negs<max(poss)) (3) The selection strategy is to select samples whose cosine distance is greater than the minimum cosine distance among the negative sample sentence pairs for the positive sample sentence pairs poss, to ensure that the cosine distance of the positive sample sentence pairs is relatively large, so that the model can learn the positive sample sentence pairs pos_pairs with relatively low semantic similarity; for the negative sample sentence pairs negs, select samples whose cosine distance is less than the maximum cosine distance among the positive sample sentence pairs, to ensure that the cosine distance of the negative sample sentence pairs is relatively small, so that the model can learn the negative sample sentence pairs neg_pairs with relatively high semantic similarity.

6. The zero-shot knowledge graph completion method based on a pre-trained language model according to claim 1, characterized in that: The loss function in step 4 is as follows: in the same batch, sentence pairs with relatively difficult label determination are selected as training sentence pairs. The positive sample sentence pairs are selected with relatively large cosine distances, and the negative sample sentence pairs are selected with relatively small cosine distances. The loss function of the positive and negative sample pairs is: Where m represents the number of elements in pos_pairs, n represents the number of elements in neg_pairs, γ is a fixed margin, and ReLu() is the ReLu activation function, which is expressed as follows: ReLu(x)=max(0,x) (5).

Images