A Zero-Shot Knowledge Graph Completion Method Based on Ontology Adapter

Abstract

The present invention discloses a zero-shot knowledge graph completion method based on an ontology adapter, which includes: obtaining ontology knowledge related to the zero-shot knowledge graph completion task, where the ontology knowledge includes entity type information, concept hierarchy relationship information, relationship type constraint information, and combined logic constraint information of relationships; designing ontology adapters for each type of ontology knowledge based on a neural network, and after adding the ontology adapters to a pre-trained language model, using each type of ontology knowledge to train the pre-trained language model with the added ontology adapter corresponding to the ontology knowledge type to inject each type of ontology knowledge, so as to obtain a pre-trained language model with introduced ontology knowledge; using the pre-trained language model with introduced ontology knowledge for downstream zero-shot knowledge graph completion tasks. This method is based on a pre-trained language model enhanced with ontology knowledge, and better solves the problem of completing knowledge graphs under zero-shot conditions.

Description

Technical Field

[0001] The present invention relates to the field of zero-shot knowledge graph completion, and particularly to a zero-shot knowledge graph completion method based on an ontology adapter. Background Art

[0002] In recent years, relying on its powerful knowledge representation and reasoning capabilities, knowledge graphs have played an important role in artificial intelligence applications such as search engines, personal assistants, intelligent question answering, recommendation systems, data fusion, etc. Although knowledge graphs contain a large number of entities and relationship facts, many existing knowledge graphs are still incomplete. Based on this, the task of knowledge graph completion (also known as link prediction) is proposed to complete the missing relationship facts (i.e., triples) in the graph. With the development of deep learning technology, many research works on knowledge graph completion mainly focus on the related technologies of knowledge graph representation learning, which represents the entities and relationships in the graph in a low-dimensional vector space, and based on the vectors of these entities and relationships, effectively and efficiently performs the knowledge graph completion task through some statistical calculation methods.

[0003] However, most of the existing representation learning methods are based on the closed-world assumption, that is, the number of entities and relationships contained in the knowledge graph is fixed, and the model can only learn the representations of known entities and relationships and predict and complete their missing triples. For newly added entities or relationships, these representation learning methods need to be retrained to obtain the representations of the new entities or new relationships. Considering that many knowledge graphs have the characteristic of fast evolution speed, that is, new entities or new relationships emerge continuously, it is unrealistic to continuously retrain the model and collect labeled data (i.e., related triples) for these new entities / relationships. Therefore, the zero-shot knowledge graph completion task (Zero-shot Knowledge Graph Completion, ZSKGC) is proposed to handle the prediction problems of these new entities or new relationships without retraining the model.

[0004] Such methods usually utilize the external information of new entities or new relationships, such as text description information, to obtain the representations of these new entities or new relationships, and based on the information in the text domain, make up for the problem of insufficient structured triple information in the knowledge graph. Based on this, a class of existing research works introduces pre-trained language models such as ELMo, GPT, BERT, etc., and with the help of the powerful text encoding ability, context semantic capture ability of words and the rich language background knowledge contained in the pre-trained language model, better utilizes the text description information of entities and relationships to help with zero-shot knowledge graph completion. At the same time, the text-based representation enables the zero-shot knowledge graph completion method based on the pre-trained language model to handle both new entities and new relationships that appear during prediction, compared with the previous zero-shot methods that can only handle new entities (or new relationships) while requiring the relationships (or entities) to be known.

[0005] However, the entity and relationship information contained in the text description is relatively limited and usually contains noise. People hope to introduce external information with richer semantics to improve the ability of knowledge graph completion under zero-shot conditions. Summary of the Invention

[0006] In view of the above, the purpose of the present invention is to provide a zero-shot knowledge graph completion method based on an ontology adapter. By injecting the ontology knowledge of the knowledge graph into the pre-trained language model through the ontology adapter, the pre-trained language model enhanced by ontology knowledge can better solve the problem of knowledge graph completion under zero-shot conditions.

[0007] To achieve the above object of the invention, the present invention provides the following technical solutions:

[0008] A zero-shot knowledge graph completion method based on an ontology adapter, comprising the following steps:

[0009] Obtain ontology knowledge related to the zero-shot knowledge graph completion task, where the ontology knowledge includes entity type information, concept hierarchy relationship information, relationship type constraint information, and relationship combination logic constraint information;

[0010] Design an ontology adapter for each type of ontology knowledge based on a neural network. After adding the ontology adapter to the pre-trained language model, use each type of ontology knowledge to train the pre-trained language model with the ontology adapter added corresponding to the ontology knowledge type to inject each type of ontology knowledge and obtain a pre-trained language model incorporating ontology knowledge;

[0011] Use the pre-trained language model incorporating ontology knowledge for downstream zero-shot knowledge graph completion tasks.

[0012] In one embodiment, the ontology adapter includes multiple adapter layers, and each adapter layer includes at least 2 full self-attention layers and at least 2 mapping layers.

[0013] In one embodiment, the pre-trained language model is a pre-trained language model containing full self-attention layers; insert the adapter layer of the ontology adapter after each full self-attention layer of the pre-trained language model, and at the same time, the adapter layers are connected to each other, and the output of the last adapter layer is the output of the ontology adapter.

[0014] In one embodiment, when the category of ontology knowledge is entity type information, design an entity type adapter to inject entity type information;

[0015] During task training, convert the entity type information into natural language corpus, and input the converted natural language corpus into a pre-trained language model with an entity type adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the entity type adapter through a masked language model to inject the entity type information into the entity type adapter, obtaining a pre-trained language model with entity type information introduced.

[0016] In one embodiment, when the category of ontology knowledge is concept hierarchy relationship information, to inject the concept hierarchy relationship information, design a concept hierarchy adapter; perform task training in the following two ways:

[0017] Method 1: Use a sentence template to convert the concept hierarchy relationship information into corpus, and input the converted corpus into a pre-trained language model with a concept hierarchy adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the concept hierarchy adapter through a masked language model to inject the concept hierarchy relationship information into the concept hierarchy adapter, obtaining a pre-trained language model with concept hierarchy relationship information introduced; and / or,

[0018] Method 2: Represent each concept as a sentence, and concepts with hierarchical relationships jointly form a document with context before and after, and convert it into corpus in this way. Input the converted corpus into a pre-trained language model with a concept hierarchy adapter added, fix the original parameters of the pre-trained language model, and optimize the parameters of the concept hierarchy adapter through a lower sentence prediction method to inject the concept hierarchy relationship information into the concept hierarchy adapter, obtaining a pre-trained language model with concept hierarchy relationship information introduced.

[0019] In one embodiment, when the category of ontology knowledge is relationship type constraint information, to inject the relationship type constraint information, design a relationship type constraint adapter;

[0020] During task training, convert the relationship type constraint information with graph structure features into corpus, and input the converted corpus into a pre-trained language model with a relationship type constraint adapter added, and optimize the parameters of the relationship type constraint adapter through a masked language model and a structure information recovery method to inject the relationship type constraint information into the relationship type constraint adapter, obtaining a pre-trained language model with relationship type constraint information introduced.

[0021] In one embodiment, the conversion of the relationship type constraint information with graph structure features into corpus includes:

[0022] For the relationship type constraint information with graph structure features, extract the one-hop subgraph around each node in the graph structure, and generate the adjacency matrix and node position matrix of each one-hop subgraph as the corpus.

[0023] In one embodiment, when the category of the ontology knowledge is the combined logic constraint information of relationships, in order to inject the combined logic constraint information of relationships, a combined logic adapter is designed, and the following two methods are used for task training:

[0024] Method 1: For the combined logic constraint information formed by multiple relationships, use a sentence template to convert the combined logic constraint information into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the combined logic adapter through a masked language model to inject the combined logic constraint information of relationships into the combined logic adapter, and obtain a pre-trained language model with the combined logic constraint information of relationships introduced; and / or,

[0025] Method 2: For the combined logic constraint information formed by multiple relationships, splice and represent the relationships participating in the combination as one sentence, and represent the semantic relationship formed by the aforementioned relationship combination as another sentence. The two sentences form a document with a context relationship before and after, and thus are converted into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the combined logic adapter through a lower sentence prediction method to inject the combined logic constraint information of relationships into the combined logic adapter, and obtain a pre-trained language model with the combined logic constraint information of relationships introduced.

[0026] Wherein, the sentence template is the remaining relationship combination splicing result obtained after splicing multiple relationships.

[0027] In one embodiment, the use of the pre-trained language model introducing ontology knowledge for the downstream zero-shot knowledge graph completion task includes:

[0028] After splicing the test triples into sentences according to their text descriptions, input them into the pre-trained language model introducing ontology knowledge, and use the original parameters of the pre-trained language model to encode to obtain a first representation containing language knowledge, and at the same time use various ontology adapters to encode to obtain a second representation containing various ontology knowledge;

[0029] Splice the first representation and the second representation as the final representation of the test triples and input them into a classifier, and use the classifier to predict whether the test triples are valid. The valid test triples are used for the completion of the zero-shot knowledge graph.

[0030] Compared with the prior art, the beneficial effects of the present invention at least include:

[0031] (1) Different from the application scenarios of knowledge injection in existing pre-trained language models, the present invention injects a richer type of ontology knowledge into the pre-trained language model based on an ontology adapter, including entity type information, concept hierarchy relationship information, relationship type constraint information, and combined logic constraints of relationships, enabling the pre-trained language model to provide not only language knowledge for downstream zero-shot knowledge graph completion tasks but also richer ontology knowledge of the knowledge graph for downstream tasks, thereby improving the ability of zero-shot knowledge graph completion based on the pre-trained language model.

[0032] (2) Different from existing zero-shot knowledge graph completion methods that can only handle new entities or new relationships that emerge during testing, while requiring relationships or entities to be known, the method provided by the present invention can be used to handle scenarios where new entities and new relationships appear simultaneously during testing, with a broader application scenario. Brief Description of the Drawings

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1 Schematic flowchart of the zero-shot knowledge graph completion method based on an ontology adapter provided for the embodiment;

[0035] Figure 2 Schematic structural diagram of the pre-trained language model with an external ontology adapter provided for the embodiment;

[0036] Figure 3 Schematic diagram of concept hierarchy relationship information in the NELL-ZS dataset provided for the embodiment;

[0037] Figure 4 Schematic diagram of relationship type constraint information in the NELL-ZS dataset provided for the embodiment;

[0038] Figure 5 Schematic diagram of the corpus process of relationship type constraint information in the NELL-ZS dataset provided for the embodiment;

[0039] Figure 6 Schematic diagram of the ontology adapter-enhanced pre-trained language model for downstream knowledge graph completion tasks provided for the embodiment. Detailed Embodiments

[0040] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0041] The present invention is inspired by the abstract ontology layer of the knowledge graph, that is, the knowledge graph usually contains an abstract ontology layer, which summarizes the term axioms and meta-information related to entities and relationships in the knowledge graph. The term axioms define information such as entity type information, the hierarchical structure of entity types and relationships, the type information of the head and tail entities associated with relationships, and the existential quantifier of entity types; the meta-information defines the text definitions, supplementary information, and descriptive information of entity categories and relationships. These information can largely summarize the characteristics of entities and relationships in the knowledge graph and the semantic information they contain, and can be used to model the semantic connections between known entities / relationships and new entities / relationships, bringing richer external information to entities / relationships. Inspired by this, the embodiment provides a zero-shot knowledge graph completion method based on an ontology adapter.

[0042] The zero-shot knowledge graph completion method based on an ontology adapter provided by the embodiment can be used for any knowledge graph with the above ontology knowledge and entity and relationship text description information. Figure 1 FIG. is a schematic flowchart of the zero-shot knowledge graph completion method based on an ontology adapter provided by the embodiment. As Figure 1 shown, the zero-shot knowledge graph completion method based on an ontology adapter provided by the embodiment includes the following steps:

[0043] Step 1, obtain ontology knowledge related to the zero-shot knowledge graph completion task.

[0044] The ontology knowledge obtained by the embodiment is external information other than text that can help the zero-shot knowledge graph completion task. The specific ontology knowledge includes entity type information, concept hierarchical relationship information, relationship type constraint information, and relationship combination logic constraint information.

[0045] Among them, the entity type information refers to the type information of entities. The concept hierarchical relationship information includes entity type hierarchical information and relationship hierarchical information. The relationship type constraint information is the type information of the head entity and the tail entity associated through the relationship. The relationship combination logic constraint information means that some relationships can be obtained by combining other relationships. For example, the relationship brotherOf and the relationship parentOf can be combined to obtain the relationship uncleOf. These ontology knowledge can help guide the judgment of whether a triple holds to achieve the completion of the zero-shot knowledge graph.

[0046] Step 2: Design an ontology adapter for each type of ontology knowledge based on the neural network. After adding the ontology adapter to the pre-trained language model, train the ontology adapter to inject each type of ontology knowledge, and obtain a pre-trained language model incorporating ontology knowledge.

[0047] In the embodiment, for each type of ontology knowledge, a specific ontology adapter is designed, and a pre-training task specific to the ontology knowledge is designed to optimize the parameters of the ontology adapter so as to inject the ontology knowledge into the pre-trained language model. Among them, the specific pre-training task related to the ontology knowledge is mainly reflected in the conversion process of the corpus and is achieved by controlling the training method. In this way, the pre-trained language model incorporating ontology knowledge obtained will contain richer ontology knowledge and be better used for the downstream zero-shot knowledge graph completion task.

[0048] In the embodiment, the pre-trained language model is a pre-trained language model including a full self-attention layer (i.e., Transformer layer). The ontology adapter includes multiple adapter layers, and each adapter layer includes at least 2 full self-attention layers and at least 2 mapping layers. Preferably, each adapter layer contains at least 2 full self-attention layers and 2 mapping layers. As Figure 2 shown, when adding the ontology adapter to the pre-trained language model, insert the adapter layer of the ontology adapter after each full self-attention layer of the pre-trained language model, and the adapter layers are connected to each other. The output of the last adapter layer is the output of the ontology adapter. When performing the pre-training task specific to the ontology knowledge, the output of the input statement under the original parameters of the pre-trained language model is concatenated with the output of the ontology adapter and then the target task is carried out. In this way, under the guidance of different pre-training tasks, different types of adapters corresponding to different types of ontology knowledge are pre-trained using different types of ontology knowledge to inject different types of ontology knowledge.

[0049] In the embodiment, when the type of ontology knowledge is entity type information, to inject the entity type information, an entity type adapter is designed, and the entity type adapter adopts the above ontology adapter structure.

[0050] During task training, convert the entity type information into natural language corpus, and input the converted natural language corpus into the pre-trained language model with the entity type adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the entity type adapter through the Masked Language Model (MLM) to inject the entity type information into the entity type adapter, and obtain a pre-trained language model incorporating entity type information.

[0051] The entity type information can be texturized in the form of sentence templates, that is, transformed into natural language corpus. For example, for entity a and its type A, using the sentence template, the text sentence "The type of entity a is A" is obtained. A large number of text sentences form a corpus containing entity and its type information.

[0052] MLM is used to randomly mask words in text sentences and encode sentence features by predicting the masked words based on the information in the context around the words. That is, the parameters of the entity type adapter are optimized according to the task of predicting the masked words using the information in the context around the words, so as to inject the entity type information into the entity type adapter.

[0053] In the embodiment, when the category of the ontology knowledge is concept hierarchical relationship information, in order to inject the concept hierarchical relationship information, a concept hierarchy adapter is designed, and the concept hierarchy adapter adopts the above ontology adapter structure. The following two methods are used for task training:

[0054] Method 1: Use the sentence template to transform the concept hierarchical relationship information into a corpus. The transformed corpus is input into the pre-trained language model with the concept hierarchy adapter added. The original parameters of the pre-trained language model are fixed, and the parameters of the concept hierarchy adapter are optimized through the masked language model, so as to inject the concept hierarchical relationship information into the concept hierarchy adapter, and obtain a pre-trained language model with the concept hierarchical relationship information introduced.

[0055] Method 2: Each concept is represented as a sentence, and concepts with hierarchical relationships jointly form a document with context before and after, and thus transformed into a corpus. The transformed corpus is input into the pre-trained language model with the concept hierarchy adapter added. The original parameters of the pre-trained language model are fixed, and the parameters of the concept hierarchy adapter are optimized through the lower sentence prediction method, so as to inject the concept hierarchical relationship information into the concept hierarchy adapter, and obtain a pre-trained language model with the concept hierarchical relationship information introduced.

[0056] In the embodiment, different methods are used to transform the hierarchical relationship between concepts into a corpus that can reflect the hierarchical relationship constraints. When the transformation method is different, the corresponding pre-training tasks and pre-training methods are also different. When using the sentence template to texturize the concepts (entity types or relationships) with hierarchical relationships, for example, for two concepts A and B with hierarchical relationships, using the sentence template, text sentences such as "A is the parent concept of B" are obtained. A large number of text sentences form a corpus that can reflect the hierarchical relationship constraints. For this type of corpus transformation method, the parameters of the concept hierarchy adapter are optimized using MLM, that is, the parameters of the concept hierarchy adapter are optimized according to the task of predicting the masked words using the information in the context around the words, so as to inject the concept hierarchical relationship information into the concept hierarchy adapter.

[0057] When each concept is represented as a sentence, multiple concepts with a hierarchical relationship together form a document with context. For such a corpus transformation method, the corresponding pre-training task is to predict the lower-level sentence based on the upper-level sentence, that is, to optimize the parameters of the concept hierarchy adapter by using the method of Next Sentence Prediction, so as to inject the concept hierarchy relationship information into the concept hierarchy adapter.

[0058] In the embodiment, when the category of the ontology knowledge is the relationship type constraint information, in order to inject the relationship type constraint information, a relationship type constraint adapter is designed, and the relationship type constraint adapter adopts the above ontology adapter structure.

[0059] During task training, the relationship type constraint information with graph structure features is transformed into a corpus, and the transformed corpus is input into a pre-trained language model with a relationship type constraint adapter added, and the parameters of the relationship type constraint adapter are optimized by the masked language model and the Structure Restoration method, so as to inject the relationship type constraint information into the relationship type constraint adapter, and a pre-trained language model with the relationship type constraint information introduced is obtained.

[0060] For the relationship type constraint information, the set training task is to predict the masked word according to the information of the context around the word, and at the same time restore the structure information. By performing this training task, the parameters of the relationship type constraint adapter are optimized, so as to inject the relationship type constraint information into the relationship type constraint adapter.

[0061] In the embodiment, the transformation of the relationship type constraint information with graph structure features into a corpus includes:

[0062] For the relationship type constraint information with graph structure features, extract the one-hop subgraph around each node in the graph structure, and generate the adjacency matrix and node position matrix of each one-hop subgraph as the corpus.

[0063] Since the relationship type constraint information has a graph structure, and the nodes in the graph structure represent entity types, therefore, when performing corpus transformation, for a certain node in the graph structure, first extract the one-hop subgraph around the node, and convert the nodes and relationship edges in the one-hop subgraph into a node sequence in a certain order and number them, and then generate the adjacency matrix and node position matrix according to the node sequence and the corresponding numbers. Among them, the adjacency matrix retains the subgraph structure of the one-hop subgraph, and the node position matrix retains the connection information of the nodes and relationship edges. The transformed node sequence is injected into the adapter network as text for encoding. While training with the masked language model, the connection information between nodes is restored according to the node position matrix of the encoded node representation, that is, to predict whether the three nodes in each row of the node position matrix form a triple.

[0064] In the embodiment, when the category of the ontology knowledge is the combined logic constraint information of relationships, in order to inject the combined logic constraint information of relationships, a combined logic adapter is designed, and the combined logic adapter adopts the above ontology adapter structure. The following two methods are used for task training:

[0065] Method 1: For the combined logic constraint information formed by multiple relationships, use a sentence template to convert the combined logic constraint information into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the combined logic adapter through the masked language model, so as to inject the combined logic constraint information of relationships into the combined logic adapter, and obtain a pre-trained language model with the combined logic constraint information of relationships introduced; and / or,

[0066] Method 2: For the combined logic constraint information formed by multiple relationships, use the relationships with the required logic as the relationships participating in the combination, combine and splice them to represent a sentence, and use the semantic relationship formed by the above-mentioned relationship combination to represent another sentence. For example, if A and B are husband and wife, and B is the father of C, then it can be inferred that A is the mother of C; the two sentences form a document with a context relationship before and after, and are thus converted into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added. Fix the original parameters of the pre-trained language model, and optimize the parameters of the combined logic adapter through the lower sentence prediction method, so as to inject the combined logic constraint information of relationships into the combined logic adapter, and obtain a pre-trained language model with the combined logic constraint information of relationships introduced.

[0067] In the embodiment, different methods are used to convert the hierarchical relationship between concepts into corpus that can reflect the hierarchical relationship constraint. When the conversion methods are different, the corresponding pre-trained tasks and pre-trained methods are also different. For the combined logic constraint information formed by multiple relationships, use a sentence template to convert the combined logic constraint information into corpus, where the sentence template is the result of combining and splicing the remaining relationship combinations after combining multiple relationships. For example, for three relationships r1, r2, and r3 with a combined relationship, use the sentence template to obtain text sentences such as "After combining relationships r1 and r2, relationship r3 can be obtained". A large number of text sentences form corpus that can reflect the knowledge of relationship combination constraints. The pre-trained task corresponding to this conversion method is to predict the masked word according to the information of the context around the word, that is, use MLM to optimize the parameters of the combined logic adapter, so as to inject the combined logic constraint information of relationships into the combined logic adapter.

[0068] For the combinatorial logic constraint information formed by multiple relationships, combine and splice the relationships participating in the combination to represent a sentence, and represent the semantic relationship formed by the aforementioned relationship combination as another sentence. The two sentences form a document with a context relationship before and after, and are thus transformed into corpus. For example, for three relationships r1, r2, and r3 with a combinatorial relationship, combine and splice relationships r1 and r2 and represent them as a sentence, while relationship r3 is represented as another sentence. The combinatorial relationship constraints of relationships r1, r2, and r3 together form a document with a context relationship before and after, and are thus transformed into corpus. The pre-training task corresponding to this transformation method is to predict the lower sentence based on the upper sentence, that is, use the lower sentence prediction method to optimize the parameters of the combinatorial logic adapter to inject the combinatorial logic constraint information of the relationships into the combinatorial logic adapter. Among them, the lower sentence prediction method is the same as the masked language model (MLM), that is, input two sentences and predict whether there is a logical dependence relationship before and after between these two sentences.

[0069] Step 3, use the pre-trained language model incorporating ontology knowledge for the downstream zero-shot knowledge graph completion task.

[0070] In the embodiment, using the pre-trained language model incorporating ontology knowledge for the downstream zero-shot knowledge graph completion task includes:

[0071] After splicing the test triples into sentences according to their text descriptions, input them into the pre-trained language model incorporating ontology knowledge, and use the original parameters of the pre-trained language model to encode to obtain a first representation containing language knowledge, and at the same time use various ontology adapters to encode to obtain a second representation containing various ontology knowledge;

[0072] Splice the first representation and the second representation as the final representation of the test triples and input them into the classifier, and use the classifier to predict whether the test triples hold. The valid test triples are used for the completion of the zero-shot knowledge graph.

[0073] As Figure 6 shown, the input sentence formed by splicing the test triples according to their text descriptions is input into the pre-trained language model incorporating ontology knowledge, and the original parameters of the pre-trained language model are used to encode to obtain a first representation containing language knowledge, s p , and at the same time, the representations s type , s hie , s cons , s comp , are encoded using the entity type adapter, concept hierarchy adapter, relationship type constraint adapter, and combinatorial logic adapter to form a second representation S = {s type , s hie , s cons , s comp}, and the first representation s pAfter being concatenated with the second representation S, it is input into a binary classifier to predict whether the triple holds.

[0074] The above zero-shot knowledge graph completion method based on the ontology adapter, in the mode of the adapter, while retaining the language background knowledge in the pre-trained language model, facilitates the introduction of more diverse types of knowledge. On this basis, the designed ontology adapter brings the ontology knowledge specific to the knowledge graph to the pre-trained language model, thus bringing more external information to the zero-shot knowledge graph completion task based on the pre-trained language model to improve the ability of knowledge graph completion under zero-shot conditions.

[0075] Currently, many existing large-scale knowledge graphs have the problem of incompleteness, such as Wikidata, DBpedia, and the e-commerce knowledge graph in the vertical domain (whose graph scale can reach hundreds of billions), etc. To solve such problems, knowledge graph embedding technology is usually used for graph completion, that is, to complete any missing one of the knowledge graph triples (head entity, relation, tail entity). With the continuous evolution and expansion of such knowledge graphs, for the newly added entities or relations in the graph, considering their scale, it is very difficult to re-train the knowledge embedding model for knowledge graph completion work. Therefore, the zero-shot work for knowledge graphs, especially large-scale knowledge graphs, urgently needs to be developed. The above method provided by the embodiment can be well applied to the e-commerce knowledge graph for knowledge graph completion.

[0076] To better illustrate the effect of the zero-shot knowledge graph completion method based on the ontology adapter provided by the above embodiment, this embodiment takes the knowledge graph NELL (Never-Ending Language Learner) and its zero-shot dataset NELL-ZS, and the pre-trained language model BERT as examples for illustration.

[0077] In the zero-shot knowledge graph completion method, first, relevant ontology knowledge is extracted from the ontology file / project of NELL. The open ontology project of NELL is stored in a csv file in the form of RDF triples. The data of this csv file consists of three columns, corresponding to the head entity, relation (also called predicate), and tail entity of the triple respectively. The type information of the entity, the hierarchical information of the entity type, and the hierarchical information of the relation can be extracted through the predicate "generalizations". The head entity type constraint information of the relation can be extracted through the predicate "domain", and the tail entity type constraint information can be extracted through the predicate "range". The hierarchical structure of the extracted entity type information is as Figure 3 shown in (a) in Figure 3 and the hierarchical structure of the concept hierarchical relationship information is as Figure 4The structure diagram combination pattern formed based on the extracted relationship type constraint information Search for relationship groups in the graph that satisfy the above constraints, that is, for the relationship r2 with entity type B as the head entity, when the tail entity is C, there are relationships r3 and r1, where the head entity type of r1 is A and the tail entity type is B, and the head entity type of r3 is A and the tail entity type is C. The extracted ontology knowledge will be subsequently converted into corpus for the training of the ontology adapter.

[0078] Then, insert an adapter layer composed of 2 full self-attention layers and 2 mapping layers after each Transformer layer of BERT, as Figure 2 shown. Based on this network structure, different types of ontology knowledge are injected respectively based on different pre-trained corpora and pre-trained tasks.

[0079] Corpus conversion of entity type information. For example, for the entity atlanta_hartsfield and its type information airport, a sentence such as "The type of entity atlanta_hartsfield is airport" can be obtained using a sentence template.

[0080] Corpus conversion of concept hierarchy relationship information can be achieved in two ways. For example, for Figure 3 the entity type hierarchy relationship shown in (a) below, using the sentence template method, the hierarchical constraint between the entity types building and hotel can be represented by a text sentence such as "Building is the parent type of hotel". And the sentence- and document-based method treats each entity type as a sentence.

[0081] Corpus conversion of relationship type constraint information. For example, for Figure 4 the structure diagram formed by the relationship type constraint information shown below, for the entity type node, its one-hop subgraph around it can be converted into the single-relationship graph shown in (a) below when considering the connection relationship as a node. Based on this single-relationship graph, node sequences and number information can be obtained as shown in (b) below. At the same time, an adjacency matrix and a node position matrix that retain the single-relationship graph structure are generated, as shown in (c) below and (d) in 5 respectively. Figure 5 In (a) below, based on this single-relationship graph, node sequences and number information can be obtained as shown in (b) below. At the same time, an adjacency matrix and a node position matrix that retain the single-relationship graph structure are generated, as shown in (c) below and (d) in 5 respectively. Figure 5 In (b) below, at the same time, an adjacency matrix and a node position matrix that retain the single-relationship graph structure are generated, as shown in (c) below and (d) in 5 respectively. Figure 5 In (c) below and (d) in 5 respectively.

[0082] Corpus conversion of the combined logical constraint information of relationships can be achieved in two ways. For example, for the relationships parentOf, brotherOf, and uncleOf with combined constraints Using the sentence template, text sentences such as "After combining the relationships brotherOf and parentOf, the relationship uncleOf can be obtained" can be obtained. For the sentence- and document-based method, the relationships brotherOf and parentOf can be first combined and spliced into a sentence, while the relationship uncleOf is separately represented as a sentence. There are context constraints between these two sentences.

[0083] Finally, for the triple (h, r, t) in the knowledge graph, it is spliced into a sentence according to its text description and connected by the character [SEP]. The text description of the relationship can be obtained through the predicate "description" in its ontology file, and the text description of the entity is the name information of the entity. Subsequently, the obtained triple sentence is input into the pre-trained language model enhanced by the ontology adapter to obtain two representations s p and S = {s type , s hie , s cons , s comp}. The above sentence representations s p and S are spliced to obtain the final sentence representation: s = [s p ; s type ; s hie ; s cons ; s comp . At the same time, it is input into the fully connected layer to train a binary classifier to judge whether the currently input triple holds. The specific process is as Figure 6 shown.

[0084] During testing, for the test triple containing new entities and new relationships, it is input into the pre-trained language model enhanced by the ontology adapter in the same way to obtain the sentence representation, and the binary classifier is used to judge whether the current triple holds. If it holds, the completion of a knowledge graph relationship fact is completed.

[0085] The above specific embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A zero-shot knowledge graph completion method based on an ontology adapter, comprising the following steps: Obtain ontology knowledge related to the zero-shot knowledge graph completion task, where the ontology knowledge includes entity type information, concept hierarchy relationship information, relationship type constraint information, or combined logical constraint information of relationships; Design an ontology adapter for each type of ontology knowledge based on a neural network, and add the ontology adapter to a pre-trained language model. Among them, the ontology adapter includes multiple adapter layers, and the pre-trained language model is a pre-trained language model containing a full self-attention layer; insert the adapter layer of the ontology adapter after each full self-attention layer of the pre-trained language model, and at the same time, the adapter layers are interconnected, and the output of the last adapter layer is the output of the ontology adapter; Use each type of ontology knowledge to train the pre-trained language model with the added ontology adapter corresponding to the ontology knowledge type to inject each type of ontology knowledge, and obtain a pre-trained language model incorporating ontology knowledge, including: When the category of ontology knowledge is entity type information, design an entity type adapter for injecting entity type information. During task training, convert the entity type information into natural language corpus, and input the converted natural language corpus into the pre-trained language model with the added entity type adapter. Fix the original parameters of the pre-trained language model, and optimize the parameters of the entity type adapter through a masked language model to inject the entity type information into the entity type adapter, and obtain a pre-trained language model incorporating entity type information; When the category of ontology knowledge is concept hierarchy relationship information, design a concept hierarchy adapter for injecting concept hierarchy relationship information; perform task training in the following two ways: Method 1: Use a sentence template to convert the concept hierarchy relationship information into corpus, and input the converted corpus into the pre-trained language model with the added concept hierarchy adapter. Fix the original parameters of the pre-trained language model, and optimize the parameters of the concept hierarchy adapter through a masked language model to inject the concept hierarchy relationship information into the concept hierarchy adapter, and obtain a pre-trained language model incorporating concept hierarchy relationship information; and / or, Method 2: Represent each concept as a sentence, and concepts with hierarchical relationships jointly form a document with context before and after, and convert it into corpus. Input the converted corpus into the pre-trained language model with the added concept hierarchy adapter. Fix the original parameters of the pre-trained language model, and optimize the parameters of the concept hierarchy adapter through a lower sentence prediction method to inject the concept hierarchy relationship information into the concept hierarchy adapter, and obtain a pre-trained language model incorporating concept hierarchy relationship information; When the category of the ontology knowledge is relationship type constraint information, to inject the relationship type constraint information, a relationship type constraint adapter is designed. During task training, the relationship type constraint information with graph structure features is converted into corpus. The converted corpus is input into a pre-trained language model with a relationship type constraint adapter added, and the parameters of the relationship type constraint adapter are optimized through a masked language model and a structure information recovery method to inject the relationship type constraint information into the relationship type constraint adapter, obtaining a pre-trained language model with the relationship type constraint information introduced; When the category of the ontology knowledge is the combined logic constraint information of relationships, to inject the combined logic constraint information of relationships, a combined logic adapter is designed, and the task training is carried out in the following two ways: Way 1: For the combined logic constraint information formed by multiple relationships, a sentence template is used to convert the combined logic constraint information into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added, the original parameters of the pre-trained language model are fixed, and the parameters of the combined logic adapter are optimized through a masked language model to inject the combined logic constraint information of relationships into the combined logic adapter, obtaining a pre-trained language model with the combined logic constraint information of relationships introduced; and / or, Way 2: For the combined logic constraint information formed by multiple relationships, the relationships participating in the combination are combined and spliced to represent a sentence, and the semantic relationship formed by the aforementioned relationship combination is represented by another sentence. The two sentences form a document with a context relationship before and after, and are thus converted into corpus. The converted corpus is input into a pre-trained language model with a combined logic adapter added, the original parameters of the pre-trained language model are fixed, and the parameters of the combined logic adapter are optimized through a lower sentence prediction method to inject the combined logic constraint information of relationships into the combined logic adapter, obtaining a pre-trained language model with the combined logic constraint information of relationships introduced; The pre-trained language model with the ontology knowledge introduced is used for the downstream zero-shot knowledge graph completion task.

2. The zero-shot knowledge graph completion method based on an ontology adapter according to claim 1, wherein Each adapter layer includes at least 2 full self-attention layers and at least 2 mapping layers.

3. The zero-shot knowledge graph completion method based on an ontology adapter according to claim 1, characterized in that The conversion of the relationship type constraint information with graph structure features into corpus includes: For the relationship type constraint information with graph structure features, the one-hop subgraph around each node in the graph structure is extracted, and the adjacency matrix and node position matrix of each one-hop subgraph are generated as the corpus.

4. The zero-shot knowledge graph completion method based on an ontology adapter according to claim 1, wherein The sentence template is the result of splicing the remaining relationship combinations after splicing multiple relationship combinations.

5. The zero-shot knowledge graph completion method based on an ontology adapter according to claim 1, wherein The use of the pre-trained language model with the ontology knowledge introduced for the downstream zero-shot knowledge graph completion task includes: After the test triple is spliced into a sentence according to its text description, it is input into the pre-trained language model with the ontology knowledge introduced. The first representation containing language knowledge is encoded using the original parameters of the pre-trained language model, and at the same time, the second representation containing various ontology knowledge is encoded using various ontology adapters; The first representation and the second representation are spliced as the final representation of the test triple and input into a classifier, and the classifier is used to predict whether the test triple holds. The test triples that hold are used for the completion of the zero-shot knowledge graph.

Images