Construction method of large-scale knowledge graph pre-training model and reasoning platform

Abstract

A method for constructing a large-scale knowledge graph pre-training model and a reasoning platform belongs to the technical field of knowledge graphs of computers, and comprises the following steps: adding prompt information into an entity and relationship sequence, so that a model captures richer information and more accurate entity and relationship representation; a structure adjustment function and a rule adjustment function are provided, the scores of the candidate entities are adjusted, and the knowledge graph completion precision is improved; experiments prove that the method is remarkable in effectiveness and superiority on a biomedical knowledge graph completion task, and the completion of the biomedical knowledge graph can be realized with higher precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer knowledge graphs, and specifically relates to a method for constructing a large-scale knowledge graph pre-training model and reasoning platform. Background Art

[0002] The biomedical knowledge graph is a complex relational graph that integrates various knowledge in the biomedical field. It contains biomedical related entities and relationships, such as entities such as proteins, drugs, and diseases, and relationships such as treatment, inhibition, and promotion. Relevant researchers can obtain rich information through reasoning and analysis of graph structures, so as to better understand complex biomedical phenomena and provide support for related fields. Large-scale biomedical knowledge graphs are increasingly widely used in the biomedical field. These applications are of great significance for promoting biomedical research and improving the effectiveness of disease diagnosis and treatment. The reasoning ability of knowledge graphs is becoming increasingly important in the biomedical field, especially in drug repositioning, drug-target interactions, and biomedical knowledge reasoning. These applications are built on high-quality biomedical knowledge graphs. However, many knowledge graphs are not yet perfect, which poses a challenge to the accuracy and effectiveness of their applications. If we want to obtain larger-scale and higher-quality biomedical knowledge graphs, we need to complete the existing biomedical knowledge graphs.

[0003] The completion technology of large-scale biomedical knowledge graphs is one of the current research hotspots in the fields of bioinformatics and artificial intelligence. The core goal of this technology is to fill the gaps in the knowledge graph through algorithms and computational models, correct erroneous information, and thus improve the overall quality and reliability of the graph. Existing knowledge graph completion methods include graph structure-based methods and language model-based methods.

[0004] There are currently a large number of graph-based methods. For example, the TransE model regards relations as translations between entities. For a triple (head entity, relation, tail entity), the model attempts to learn an embedding so that the embedding of the head entity plus the embedding of the relation is close to the embedding of the tail entity. DistMult represents entities and relations as vectors and uses vector dot products to model the relationship between entities. ConvE uses convolutional neural networks to capture local patterns between entities and relations, represents the embeddings of entities and relations as 2D matrices, and learns their combined representations through convolution operations. In addition, there are methods such as TransH, TransR, and CapsE, but the above methods cannot handle the diversity and complexity of entities and relations, have insufficient filtering capabilities for erroneous results, and do not consider semantic information.

[0005] Large-scale pre-trained language models, such as BERT, have achieved remarkable results in the field of natural language processing. These models are trained on large-scale texts, learn semantics and context information in the text, and provide support for downstream tasks. Since the model has learned a lot of information, users can achieve excellent performance in various natural language processing related tasks by directly using it or fine-tuning it. Language model-based methods introduce these large-scale pre-trained language models into knowledge graph completion. For example, the KG-BERT model and the MTLKGC model are early works that apply the BERT model to knowledge graph completion tasks. KG-BERT processes knowledge graphs in the same way as processing text sequences, obtains triple representations through large-scale pre-trained language models, and then performs binary or multi-classification on the representation to perform graph completion and relationship prediction and other related work, providing new ideas for knowledge graph completion. However, since it is an early work, KG-BERT does not take into account the rich domain information of the entities in the knowledge graph itself, resulting in the inability to effectively distinguish entities with similar word composition but low correlation when using language models to obtain embeddings. In addition, the structural information in the knowledge graph cannot be effectively used.

[0006] MTLKGC alleviates the problem of distinguishing similar entities to a certain extent by setting three tasks (link prediction, relationship prediction, and relevance ranking), but it still does not fundamentally solve the problem in KG-BERT. Summary of the invention

[0007] The purpose of the present invention is to provide a method for constructing a large-scale knowledge graph pre-training model and reasoning platform to address the problems that previous work has been unable to handle the diversity and complexity of entities and relationships, ignored the semantic information of entities, lost the structural information of knowledge graphs, and had unsatisfactory completion effects on large-scale knowledge graphs.

[0008] A method for constructing a large-scale knowledge graph pre-training model and reasoning platform of the present invention comprises the following steps:

[0009] 1.1 The encoder-decoder architecture is adopted. The encoder uses the semantic information and prompt information of the entity to generate embedding vectors for the entity and the relationship for subsequent use by the decoder; the decoder uses the entity and relationship representations generated by the encoder to complete the task of completing the biomedical knowledge graph; the graph-sequence conversion module with prompt information - PKG2S, the pre-trained large-scale language model encoding module based on pseudo-twin neural network - PSN-PLLME, the contrastive learning module, the information extraction module, the structure and rule-based score adjustment module and the biomedical knowledge graph completion module are added;

[0010] 1.2 In the PKG2S module, the graph data is first converted into a text sequence; for each entity, a path of length N is randomly selected from its neighboring nodes to obtain the entity sequence It is expressed as:

[0011]

[0012] Where: e i is the central entity, i.e., the entity whose embedding representation you want to obtain; j 、e k Yes i Neighbor entities;

[0013] Add the relevant domain knowledge of biomedical entities and relations to the entity sequence to obtain the entity description sequence It is expressed as follows:

[0014]

[0015] in: It is entity e i The prompt information is entity description, relationship definition, or contextual information and domain knowledge related to entities and relationships;

[0016] 1.3 In the PSN-PLLME module, the pseudo-twin neural network structure is used to put the head entity description sequence and the relationship into a BERT, and the tail entity description sequence is put into another BERT to obtain the representation of each token. The above two BERT models have the same structure but do not share parameters. Average pooling and L2 normalization are used to obtain the embedding representation hr of the head entity with the relationship and the embedding representation t of the tail entity. The reason for choosing to generate the embedding representation of each entity as the head entity under each relationship and its representation when it is a tail entity alone is that this can significantly reduce the cost of the model in actual usage scenarios;

[0017] 1.4 The key components of the contrastive learning module include (1) positive sample pairs: two or more views generated from the same instance after enhancement or conversion; (2) negative samples: samples from different instances. In contrastive learning, the selection of negative samples is very important because they help the model learn the features that distinguish different instances; (3) similarity measurement: contrastive learning requires a metric to measure the similarity between samples. Commonly used metrics include cosine similarity, Euclidean distance or inner product; (4) loss function: the loss function of contrastive learning is usually based on InfoNCE loss or its variants, which aims to maximize the similarity between positive sample pairs and minimize the similarity between negative samples. The negative sample generation method in the SimKGC model is adopted, including: (1) in-batch negative sampling: using entities in the same batch during training as the basis for generating negative samples; (2) pre-batch negative sampling: using entities in the first n batches during training as the basis for generating negative samples; (3) self-negative sampling. Sampling): For the tail entity prediction (h, r, ?), the text-based method tends to assign high scores to the head entity h due to the high text overlap; using the head entity h as the self-negative sample, the mathematical expression of the loss function is:

[0018]

[0019] Where: Φ(h,r,t) is the cosine similarity between the head entity h and the tail entity t under the relationship r, calculated by the following formula:

[0020]

[0021] Where: |N| is the number of negative samples of the triple of head entity h and relation r; t i is the tail entity of the i-th sample; γ is a constant that can adjust the importance of the original triple in the biomedical knowledge graph;

[0022] 1.5 In the score adjustment module based on structure and rules, in order to make the results of knowledge graph completion more reasonable and accurate, the present invention needs to introduce the structural information of the original graph to assist the completion results, that is, the score adjustment of candidate entities based on structure, specifically: for biomedical knowledge graph completion, if two entities are close in the graph, that is, they can reach each other through fewer edges, they are considered to be highly related. If the distance between two entities in the knowledge graph is within k hops, the similarity score of the two entities is increased, and its mathematical expression is:

[0023]

[0024] Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r; distance(h,t) is the distance between entity h and entity t in the knowledge graph. If the distance between them is less than or equal to k, their similarity scores are increased by α. If the distance between them is greater than k, their similarity scores are not adjusted.

[0025] The knowledge generated by graph completion based on entity similarity may be wrong in terms of rules. If the platform imposes certain restrictions on the rules, its performance can be improved. Therefore, for knowledge that is wrong in terms of rules, the platform needs to filter out entities that do not conform to the rules, that is, adjust the scores of candidate entities based on the rules. For candidate entities that conform to the rules, they are considered True and the original similarity score is maintained. Otherwise, they are considered False and their similarity score with the corresponding head entity is set to 0, as shown in the following formula:

[0026]

[0027] Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r;

[0028] 1.6 In the biomedical knowledge graph completion module, for the completion task (e i ,r,? ), select the entity with the head e i The tail entity e with the maximum similarity score under relation r j As a result of completion;

[0029] 1.7 The information extraction module extracts valuable information from the knowledge graph, including structural information and inference rules. Structural information refers to the entities and relationships represented by nodes and edges in the knowledge graph, as well as the distance between entities in the knowledge graph. This information helps the platform understand the relationship between entities, thereby making more accurate predictions in the completion process, because knowledge graphs often show spatial locality, that is, the relationship between nodes that are close is often closer than the relationship between nodes that are far away. In the biomedical knowledge graph, structural information can help the platform understand the relationship between diseases and drugs. If a disease is close to a drug, then the drug may have a closer relationship with the disease. On the contrary, if a drug is far away from a target, then the possibility that the drug is related to the target is much smaller. Inference rules are rules extracted from the knowledge graph to describe the constraints between entities and relationships, eliminate unreasonable predictions of the platform in the completion process, and improve the accuracy of the completion results. For the completion task <drug A, treatment,?>, then? must be a disease, otherwise the completion item is completely meaningless.

[0030] The present invention adds prompt information to entity and relationship sequences, so that the model can capture richer information and more accurate entity and relationship representations; proposes structure adjustment functions and rule adjustment functions to adjust the scores of candidate entities and improve the accuracy of knowledge graph completion; experiments have proved that the present invention is significantly effective and superior in the task of biomedical knowledge graph completion, and can achieve biomedical knowledge graph completion with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of a method for building a large-scale knowledge graph pre-training model and reasoning platform. DETAILED DESCRIPTION

[0032] The present invention is described below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, a method for constructing a large-scale knowledge graph pre-training model and reasoning platform of the present invention comprises the following steps:

[0034] 1.1 The encoder-decoder architecture is adopted. The encoder uses the semantic information and prompt information of the entity to generate embedding vectors for the entity and the relationship for subsequent use by the decoder; the decoder uses the entity and relationship representations generated by the encoder to complete the task of completing the biomedical knowledge graph; the present invention includes six modules: a graph-sequence conversion module with added prompt information - PKG2S, a pre-trained large-scale language model encoding module based on pseudo-twin neural network - PSN-PLLME, a contrastive learning module, an information extraction module, a structure- and rule-based score adjustment module, and a biomedical knowledge graph completion module;

[0035] 1.2 In the PKG2S module, the graph data is first converted into a text sequence; for each entity, a path of length N is randomly selected from its neighboring nodes to obtain the entity sequence It is expressed as:

[0036]

[0037] Where: e i is the central entity, i.e., the entity whose embedding representation you want to obtain; j 、e k Yes i Neighbor entities;

[0038] Add the relevant domain knowledge of biomedical entities and relations to the entity sequence to obtain the entity description sequence It is expressed as follows:

[0039]

[0040] Where: dei It is entity e i The prompt information is entity description, relationship definition, or contextual information and domain knowledge related to entities and relationships;

[0041] 1.3 In the PSN-PLLME module, the pseudo-twin neural network structure is used to put the head entity description sequence and the relationship into a BERT, and the tail entity description sequence is put into another BERT to obtain the representation of each token. The above two BERT models have the same structure but do not share parameters; average pooling and L2 normalization are used to obtain the embedding representation hr of the head entity with the relationship and the embedding representation t of the tail entity respectively;

[0042] 1.4 The key components of the contrastive learning module include (1) positive sample pairs: two or more views generated from the same instance after enhancement or conversion; (2) negative samples: samples from different instances; (3) similarity measurement: contrastive learning requires a metric to measure the similarity between samples. Commonly used metrics include cosine similarity, Euclidean distance or inner product; (4) loss function: the loss function of contrastive learning is usually based on InfoNCE loss or its variants; the negative sample generation method in the SimKGC model is adopted, including: (1) intra-batch negative sampling: using entities in the same batch during training as the basis for generating negative samples; (2) pre-batch negative sampling: using entities in the first n batches during training as the basis for generating negative samples; (3) self-negative sampling: for tail entity prediction (h, r, ?), text-based methods tend to assign high scores to the head entity h; using the head entity h as the self-negative sample, the mathematical expression of the loss function is:

[0043]

[0044] Where: Φ(h,r,t) is the cosine similarity between the head entity h and the tail entity t under the relationship r, calculated by the following formula:

[0045]

[0046] Where: |N| is the number of negative samples of the triple of head entity h and relation r; t i is the tail entity of the i-th sample; γ is a constant;

[0047] 1.5 In the structure- and rule-based score adjustment module, the candidate entity scores are adjusted. The structure-based adjustment is as follows: For biomedical knowledge graph completion, if two entities are close in the graph, that is, they can reach each other through fewer edges, they are considered to be highly related. If the distance between two entities in the knowledge graph is within k hops, the similarity score of the two entities is increased. The mathematical expression is:

[0048]

[0049] Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r; distance(h,t) is the distance between entity h and entity t in the knowledge graph. If the distance between them is less than or equal to k, their similarity scores are increased by α. If the distance between them is greater than k, their similarity scores are not adjusted.

[0050] The rule-based adjustment is as follows: for knowledge that is incorrect in the rules, the platform needs to filter out entities that do not conform to the rules, that is, adjust the score of candidate entities based on the rules. For candidate entities that conform to the rules, they are regarded as True and the original similarity score is maintained. Otherwise, they are regarded as False and the similarity score with the corresponding head entity is set to 0, as shown in the following formula:

[0051]

[0052] Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r;

[0053] 1.6 In the biomedical knowledge graph completion module, for the completion task (e i ,r,? ), select the entity with the head e i The tail entity e with the maximum similarity score under relation r j As a result of completion;

[0054] 1.7 The information extraction module extracts valuable information from the knowledge graph, including structural information and inference rules. Structural information refers to the entities and relationships represented by nodes and edges in the knowledge graph, and the distance between entities in the knowledge graph. Inference rules are rules extracted from the knowledge graph to describe the constraints between entities and relationships, eliminate unreasonable predictions of the platform during the completion process, and improve the accuracy of the completion results. For the completion task <drug A, treatment, ?>, then ? must be a disease, otherwise the completion item is completely meaningless.

Claims

1. A method for constructing a large-scale knowledge graph pre-training model and reasoning platform, characterized in that The following steps are involved: 1.1 The encoder-decoder architecture is adopted. The encoder uses the semantic information and prompt information of the entity to generate embedding vectors for the entity and the relationship for subsequent use by the decoder; the decoder uses the entity and relationship representations generated by the encoder to complete the task of completing the biomedical knowledge graph; the graph-sequence conversion module with prompt information - PKG2S, the pre-trained large-scale language model encoding module based on pseudo-twin neural network - PSN-PLLME, the contrastive learning module, the information extraction module, the structure and rule-based score adjustment module and the biomedical knowledge graph completion module are added; 1.2 In the PKG2S module, the graph data is first converted into a text sequence; for each entity, a path of length N is randomly selected from its neighboring nodes to obtain the entity sequence It is expressed as: Where: e i is the central entity, i.e., the entity whose embedding representation you want to obtain; j 、e k Yes i Neighbor entities; Add the relevant domain knowledge of biomedical entities and relations to the entity sequence to obtain the entity description sequence It is expressed as follows: in: It is entity e i The prompt information is entity description, relationship definition, or contextual information and domain knowledge related to entities and relationships; 1.3 In the PSN-PLLME module, the pseudo-twin neural network structure is used to put the head entity description sequence and the relationship into a BERT, and the tail entity description sequence is put into another BERT to obtain the representation of each token. The above two BERT models have the same structure but do not share parameters; average pooling and L2 normalization are used to obtain the embedding representation hr of the head entity with the relationship and the embedding representation t of the tail entity respectively; 1.4 The key components of the contrastive learning module include (1) positive sample pairs: two or more views generated from the same instance after enhancement or conversion; (2) negative samples: samples from different instances; (3) similarity measurement: contrastive learning requires a metric to measure the similarity between samples. Commonly used metrics include cosine similarity, Euclidean distance or inner product; (4) loss function: the loss function of contrastive learning is usually based on InfoNCE loss or its variants; the negative sample generation method in the SimKGC model is adopted, including: (1) intra-batch negative sampling: using entities in the same batch during training as the basis for generating negative samples; (2) pre-batch negative sampling: using entities in the first n batches during training as the basis for generating negative samples; (3) self-negative sampling: for tail entity prediction (h, r, ?), text-based methods tend to assign high scores to the head entity h; using the head entity h as the self-negative sample, the mathematical expression of the loss function is: Where: Φ(h,r,t) is the cosine similarity between the head entity h and the tail entity t under the relationship r, calculated by the following formula: Where: |N| is the number of negative samples of the triple of head entity h and relation r; t i is the tail entity of the i-th sample; γ is a constant; 1.5 In the structure- and rule-based score adjustment module, the candidate entity scores are adjusted. The structure-based adjustment is as follows: For biomedical knowledge graph completion, if two entities are close in the graph, that is, they can reach each other through fewer edges, they are considered to be highly related. If the distance between two entities in the knowledge graph is within k hops, the similarity score of the two entities is increased. The mathematical expression is: Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r; distance(h,t) is the distance between entity h and entity t in the knowledge graph. If the distance between them is less than or equal to k, their similarity scores are increased by α. If the distance between them is greater than k, their similarity scores are not adjusted. The rule-based adjustment is as follows: for knowledge that is incorrect in the rules, the platform needs to filter out entities that do not conform to the rules, that is, adjust the score of candidate entities based on the rules. For candidate entities that conform to the rules, they are regarded as True and the original similarity score is maintained. Otherwise, they are regarded as False and the similarity score with the corresponding head entity is set to 0, as shown in the following formula: Where: Φ(h,r,t) is the similarity score between the head entity h and the tail entity t under the relation r; 1.6 In the biomedical knowledge graph completion module, for the completion task (e i ,r,? ), select the entity with the head e i The tail entity e with the maximum similarity score under relation r j As a result of completion; 1.7 The information extraction module extracts valuable information from the knowledge graph, including structural information and inference rules. Structural information refers to the entities and relationships represented by nodes and edges in the knowledge graph, and the distance between entities in the knowledge graph. Inference rules are rules extracted from the knowledge graph to describe the constraints between entities and relationships, eliminate unreasonable predictions of the platform during the completion process, and improve the accuracy of the completion results. For the completion task <drug A, treatment, ?>, then ? must be a disease, otherwise the completion item is completely meaningless.

Images