Method for realizing combined extraction of entities and relationships in knowledge graph

Abstract

The invention belongs to the field of natural language processing and knowledge graph construction, and particularly relates to a method for realizing combined extraction of entities and relationships in a knowledge graph, which comprises the following steps of: 1) data preprocessing: collecting text data containing entities and relationships in each field, and labeling the entities and the relationships; 2) feature extraction: establishing a Bi-LSTM model, and performing feature extraction on the text data; performing word segmentation on the extracted text data, converting the extracted text data into word vectors, and taking the word vectors as input of a Bi-LSTM model; the Bi-LSTM model captures context information in the text data; 3) through CRF sequence labeling, outputting a final sequence labeling result for identifying entities and relationships, and ensuring that isolated entity labels do not appear; 4) TransE model training: on the basis of a sequence labeling result, further extracting a relationship between entities in the sequence labeling result; 5) constructing a relation graph between entities; and meanwhile, reasoning is carried out by utilizing entities and relationships in the knowledge graph so as to find new knowledge and relationships.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing and knowledge graph construction, and specifically is a method for realizing the joint extraction of entities and relationships in a knowledge graph. Background Art

[0002] Joint extraction of entities and relations in knowledge graphs is a complex and evolving field that combines multiple technologies such as natural language processing, machine learning, and deep learning. In text, various entities need to be identified first, which is usually achieved through named entity recognition technology, which can identify entities with specific meanings from text. After identifying the entities, the relationships between these entities need to be further extracted. In recent years, deep learning technology, especially neural network models, has played an important role in the joint extraction of entities and relations in knowledge graphs. These models can automatically learn features in text to more accurately identify entities and extract relations, and, through pre-trained models, provide powerful language representation capabilities for the construction of knowledge graphs. By fine-tuning on these pre-trained models, the accuracy of entity recognition and relationship extraction can be significantly improved. Traditionally, entity recognition and relationship extraction are regarded as two independent tasks and are processed in a pipeline manner. However, this approach suffers from the problem of error propagation, that is, errors in entity recognition will affect the accuracy of relationship extraction. Summary of the invention

[0003] The purpose of this invention is to provide a method for jointly extracting entities and relations in knowledge graphs by combining TransE and Bi-LSTM-CRF joint learning, treating entity recognition and relation extraction as a whole task. This method can make full use of the interactive information between entities and relations and improve the accuracy and efficiency of extraction.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for realizing the joint extraction of entities and relationships in a knowledge graph, comprising the following steps:

[0005] 1) Data preprocessing: Collect text data containing entities and relationships in various fields from text readers and storage devices such as external hard drives and solid-state drives, and annotate the text data with entities and relationships, including entity types and relationship types;

[0006] 2) Feature extraction: Establish a Bi-LSTM model to extract features from text data; convert the extracted text data into word vectors as input to the Bi-LSTM model; the Bi-LSTM model captures contextual information in text data;

[0007] 3) Through CRF sequence labeling: By receiving the output of Bi-LSTM, outputting the hidden state sequence corresponding to the input sequence, the hidden state sequence contains the context information of the input sequence, and processing the hidden state sequence, and outputting the final sequence labeling result for identifying entities and relations to ensure that no isolated entity labels appear;

[0008] 4) TransE model training: Based on the sequence annotation results, further extract the relationship between entities in the sequence annotation results;

[0009] 5) Knowledge graph construction: Link the extracted entities with existing entities in the knowledge graph to eliminate ambiguity and ensure data consistency. Then, based on the extracted relationships and vector representations between entities, build a relationship graph between entities. Finally, integrate knowledge from different sources to eliminate redundancy and conflict and form a complete and accurate knowledge graph. At the same time, use the entities and relationships in the knowledge graph for reasoning to discover new knowledge and relationships.

[0010] The Bi-LSTM model includes: an input layer, a hidden layer, and an output layer;

[0011] The input layer of the Bi-LSTM model is used to receive raw data and pass it to the hidden layer for processing, represented by a single node X1;

[0012] The hidden layer of the Bi-LSTM model is used to perform weighting, summing, and activation operations on the signals transmitted from the input layer to extract data features;

[0013] The output layer of the Bi-LSTM model is from Y1 to Y n It is composed of multiple nodes, representing the final output of the neural network. The output layer calculates and outputs the final result based on the feature signal transmitted from the hidden layer.

[0014] The step 2) is specifically:

[0015] 2-1) Input the extracted text data into the word vector layer for word segmentation processing, and segment the text according to words or other basic language units;

[0016] 2-2) Build a word vector table, that is, map all segmented words to a unique integer index for subsequent word vector representation;

[0017] 2-3) By querying the word vector table, the word vectors of the words corresponding to the sentence sequence are obtained as the input sequence of the Bi-LSTM layer;

[0018] 2-4) The multiple parameter matrices of the Bi-LSTM layer are randomly initialized. The input sequence obtained by the word vector layer is input into the Bi-LSTM layer, that is, it is simultaneously input into the forward LSTM layer and the backward LSTM layer for model training, and the outputs of the LSTM in the two directions are concatenated as the output of the hidden layer.

[0019] The step 3) is specifically as follows: the CRF layer is responsible for labeling the sequence of sentences, using the parameter matrix to label each position, and solving the optimal path based on the Viterbi algorithm, that is:

[0020] 3-1) For the observation sequence O1n=O1, O2,...,On, the corresponding state sequence S1n=S1, S2,...,Sn has the conditional probability of existence, that is:

[0021]

[0022] Among them, Zo represents the normalization factor of the state sequence, fj represents the characteristic function that changes with the state, and λj represents the global characteristic weight;

[0023] 3-2) Obtain the optimal feature weights through training, and use the Viterbi algorithm to obtain a globally optimal labeling sequence.

[0024] The TransE model training is as follows: entities and relationships are represented as vectors in a low-dimensional vector space through the TransE model, the vector representations of the entities and relationships are initialized, and the vectors are continuously optimized, including the following steps:

[0025] 4-1) Each entity and relationship is represented by a vector of fixed dimension, that is, each entity and relationship is represented by a vector of dimension d;

[0026] 4-2) Triples in the knowledge graph, namely: (entity vector, relationship vector, tail entity vector)

[0027] The vector form of the triple in the knowledge graph, i.e.: h+r=t;

[0028] Among them, h, r, and t represent the vector representations of the head entity, relation, and tail entity, respectively.

[0029] 4-3) The TransE model is used to make the head entity vector plus the relationship vector in the triple equal to the tail entity vector, and the margin-based triple loss function is used to learn the vector representation of entities and relationships;

[0030] 4-4) TransE optimizes the loss function through the stochastic gradient descent algorithm and continuously adjusts the vector representation of entities and relationships, so that the model can better fit the triple data in the knowledge graph;

[0031] 4-5) Iterative Optimization: The TransE model iteratively optimizes the vector representations of entities and relations, and continuously updates the vector representations so that the model gradually learns the semantic relationship between entities and relations, so that the model learns accurate vector representations of entities and relations.

[0032] The construction of the knowledge graph includes the following steps:

[0033] 5-1) Entity linking: Linking entities extracted from text with existing entities in the knowledge graph. Using entity linking technology, we can identify entities in the text and their corresponding entities in the knowledge graph.

[0034] 5-2) Constructing an entity relationship graph: Based on the extracted relationships and vector representations between entities, a relationship graph between entities is constructed. The entity relationship graph is constructed by calculating the similarity between entities or the distance between relationship vectors. Entities and relationships are represented as nodes and edges in the relationship graph.

[0035] 5-3) Knowledge graph reasoning: Use the GAT graph neural network model to represent the graph structure in the knowledge graph, so as to infer missing relationships or associations between entities.

[0036] The entity linking is matched based on entity name, context information, and entity attributes to ensure that the extracted entities are correctly associated with entities in the knowledge graph.

[0037] In step 5), the entities and relationships in the knowledge graph are used for reasoning to discover new knowledge and relationships, specifically:

[0038] Through the knowledge graph reasoning algorithm, we can discover hidden associations between entities and predict possible new relationships between entities, thereby enriching the content of the knowledge graph.

[0039] The present invention has the following beneficial effects and advantages:

[0040] 1. The present invention provides a method for combining TransE with Bi-LSTM-CRF joint learning, which can treat entity recognition and relationship extraction as a whole task. Compared with traditional methods, this method can make full use of the interactive information between entities and relationships and improve the accuracy and efficiency of extraction.

[0041] 2. The present invention can effectively solve the problem of error transmission and the accuracy of relationship extraction when processing two independent tasks of entity recognition and relationship extraction in a pipeline manner by combining TransE with Bi-LSTM-CRF. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1Flowchart of the method for realizing joint extraction of entities and relationships in knowledge graphs in the present invention;

[0043] Figure 2 Schematic diagram of the Bi-LSTM model layer structure of the present invention;

[0044] Figure 3 A schematic diagram of the structure of the Bi-LSTM-CRF model of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0047] like Figure 1 As shown, it is a flow chart of the method for realizing the joint extraction of entities and relationships in the knowledge graph of the present invention. The method for realizing the joint extraction of entities and relationships in the knowledge graph of the present invention comprises the following steps:

[0048] 1) Data preprocessing: Collect text data containing entities and relationships in various fields from text readers and storage devices such as external hard drives and solid-state drives, and annotate the text data with entities and relationships, including entity types and relationship types;

[0049] 2) Feature extraction: Establish a Bi-LSTM model to extract features from text data; convert the extracted text data into word vectors as the input of the Bi-LSTM model; the Bi-LSTM model captures the contextual information in the text data, whether forward or backward. ;

[0050] Step 2), specifically:

[0051] 2-1) Input the extracted text data into the word vector layer for word segmentation processing, and segment the text according to words or other basic language units;

[0052] 2-2) Build a word vector table, that is, map all segmented words to a unique integer index for subsequent word vector representation;

[0053] 2-3) By querying the word vector table, the word vectors of the words corresponding to the sentence sequence are obtained as the input sequence of the Bi-LSTM layer;

[0054] 2-4) The multiple parameter matrices of the Bi-LSTM layer are randomly initialized. The input sequence obtained by the word vector layer is input into the Bi-LSTM layer, that is, it is simultaneously input into the forward LSTM layer and the backward LSTM layer for model training, and the outputs of the LSTM in the two directions are concatenated as the output of the hidden layer.

[0055] 3) Through CRF sequence labeling: By receiving the output of Bi-LSTM, outputting the hidden state sequence corresponding to the input sequence, the hidden state sequence contains the context information of the input sequence, and processing the hidden state sequence, and outputting the final sequence labeling result for identifying entities and relations to ensure that no isolated entity labels appear;

[0056] 4) TransE model training: Based on the sequence annotation results, further extract the relationship between entities in the sequence annotation results;

[0057] TransE model training is as follows: entities and relationships are represented as vectors in a low-dimensional vector space through the TransE model, the vector representations of entities and relationships are initialized, and the vectors are continuously optimized, including the following steps:

[0058] 4-1) Each entity and relationship is represented by a vector of fixed dimension, that is, each entity and relationship is represented by a vector of dimension d;

[0059] 4-2) Triples in the knowledge graph, namely: (entity vector, relationship vector, tail entity vector)

[0060] The vector form of the triple in the knowledge graph, i.e.: h+r=t;

[0061] Among them, h, r, and t represent the vector representations of the head entity, relation, and tail entity, respectively.

[0062] 4-3) The TransE model is used to make the head entity vector plus the relationship vector in the triple equal to the tail entity vector, and the margin-based triple loss function is used to learn the vector representation of entities and relationships;

[0063] 4-4) TransE optimizes the loss function through the stochastic gradient descent algorithm and continuously adjusts the vector representation of entities and relationships, so that the model can better fit the triple data in the knowledge graph;

[0064] 4-5) Iterative Optimization: The TransE model iteratively optimizes the vector representations of entities and relations, and continuously updates the vector representations so that the model gradually learns the semantic relationship between entities and relations, so that the model learns accurate vector representations of entities and relations.

[0065] 5) Knowledge graph construction: Link the extracted entities with existing entities in the knowledge graph to eliminate ambiguity and ensure data consistency. Build a relationship graph between entities based on the extracted relationships and vector representations between entities. At the same time, use the entities and relationships in the knowledge graph for reasoning to discover new knowledge and relationships.

[0066] 5-1) Entity linking: Linking entities extracted from text with existing entities in the knowledge graph. Using entity linking technology, we can identify entities in the text and their corresponding entities in the knowledge graph.

[0067] 5-2) Constructing an entity relationship graph: Based on the extracted relationships and vector representations between entities, a relationship graph between entities is constructed. The entity relationship graph is constructed by calculating the similarity between entities or the distance between relationship vectors. Entities and relationships are represented as nodes and edges in the relationship graph.

[0068] 5-3) Knowledge graph reasoning: Use the GAT graph neural network model to represent the graph structure in the knowledge graph, so as to infer missing relationships or associations between entities.

[0069] The entity linking is matched based on entity name, context information, and entity attributes to ensure that the extracted entities are correctly associated with entities in the knowledge graph.

[0070] like Figure 2 , which is a schematic diagram of the layer structure of the Bi-LSTM model of the present invention. The Bi-LSTM model of the present invention includes: an input layer, a hidden layer, and an output layer;

[0071] The input layer of the Bi-LSTM model is used to receive raw data and pass it to the hidden layer for processing, represented by a single node X1;

[0072] The hidden layer of the Bi-LSTM model is used to perform weighting, summing, and activation operations on the signals transmitted from the input layer to extract data features;

[0073] The output layer of the Bi-LSTM model is from Y1 to Y n It is composed of multiple nodes, representing the final output of the neural network. The output layer calculates and outputs the final result based on the feature signal transmitted from the hidden layer.

[0074] like Figure 3As shown in the figure, it is a schematic diagram of the structure of the Bi-LSTM-CRF model of the present invention. CRF is a model that takes a given random variable as input and seeks the conditional probability distribution of the output random variable. It is a continuously optimized maximum entropy model that uses a global normalization method to avoid data bias in sequence labeling by relying on feature weights that change with the state and global optimal feature weights. It receives the output of Bi-LSTM and outputs the final sequence labeling result to identify entities and relationships, ensuring that no isolated entity labels appear.

[0075] In step 3) of the method of the present invention, the CRF layer is responsible for the sequence annotation of the sentence, annotating each position using the parameter matrix, and solving the optimal path based on the Viterbi algorithm, that is:

[0076] 3-1) For the observation sequence O1n=O1, O2,...,On, the corresponding state sequence S1n=S1, S2,...,Sn has the conditional probability of existence, that is:

[0077]

[0078] Among them, Zo represents the normalization factor of the state sequence, fj represents the characteristic function that changes with the state, and λj represents the global characteristic weight;

[0079] 3-2) Obtain the optimal feature weights through training, and use the Viterbi algorithm to obtain a globally optimal labeling sequence.

[0080] In step 5), the entities and relationships in the knowledge graph are used for reasoning to discover new knowledge and relationships, specifically:

[0081] Through the knowledge graph reasoning algorithm, we can discover hidden associations between entities and predict possible new relationships between entities, thereby enriching the content of the knowledge graph.

[0082] In the above model, for LSTM related experiments, the TensorFlow deep learning framework is adopted, and for the traditional CRF model, CRF++0.58 is used for experiments.

[0083] In summary, the present invention provides a method of combining TransE and Bi-LSTM-CRF for joint learning, which can treat entity recognition and relationship extraction as a whole task for processing. By combining TransE with Bi-LSTM-CRF, the problem of error transmission and the accuracy of relationship extraction when processing two independent tasks of entity recognition and relationship extraction in a pipeline manner can be effectively solved. Adding TransE can make full use of the interactive information between entities and relationships and improve the accuracy and efficiency of extraction.

[0084] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0085] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for joint extraction of entities and relationships in a knowledge graph, characterized in that: The following steps are involved: 1) Data preprocessing: Collect text data containing entities and relationships in various fields from text readers and storage devices such as external hard drives and solid-state drives, and annotate the text data with entities and relationships, including entity types and relationship types; 2) Feature extraction: Establish a Bi-LSTM model to extract features from text data; convert the extracted text data into word vectors as input to the Bi-LSTM model; the Bi-LSTM model captures contextual information in text data; 3) Through CRF sequence labeling: By receiving the output of Bi-LSTM, outputting the hidden state sequence corresponding to the input sequence, the hidden state sequence contains the context information of the input sequence, and processing the hidden state sequence, and outputting the final sequence labeling result for identifying entities and relations to ensure that no isolated entity labels appear; 4) TransE model training: Based on the sequence annotation results, further extract the relationship between entities in the sequence annotation results; 5) Knowledge graph construction: Link the extracted entities with existing entities in the knowledge graph to eliminate ambiguity and ensure data consistency. Build a relationship graph between entities based on the extracted relationships and vector representations between entities. At the same time, use the entities and relationships in the knowledge graph for reasoning to discover new knowledge and relationships.

2. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: The Bi-LSTM model includes: an input layer, a hidden layer, and an output layer; The input layer of the Bi-LSTM model is used to receive raw data and pass it to the hidden layer for processing, represented by a single node X1; The hidden layer of the Bi-LSTM model is used to perform weighting, summing, and activation operations on the signals transmitted from the input layer to extract data features; The output layer of the Bi-LSTM model is from Y1 to Y n It is composed of multiple nodes, representing the final output of the neural network. The output layer calculates and outputs the final result based on the feature signal transmitted from the hidden layer.

3. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: The step 2) is specifically: 2-1) Input the extracted text data into the word vector layer for word segmentation processing, and segment the text according to words or other basic language units; 2-2) Build a word vector table, that is, map all segmented words to a unique integer index for subsequent word vector representation; 2-3) By querying the word vector table, the word vectors of the words corresponding to the sentence sequence are obtained as the input sequence of the Bi-LSTM layer; 2-4) The multiple parameter matrices of the Bi-LSTM layer are randomly initialized. The input sequence obtained by the word vector layer is input into the Bi-LSTM layer, that is, it is simultaneously input into the forward LSTM layer and the backward LSTM layer for model training, and the outputs of the LSTM in the two directions are concatenated as the output of the hidden layer.

4. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: The step 3) is specifically as follows: the CRF layer is responsible for labeling the sequence of sentences, using the parameter matrix to label each position, and solving the optimal path based on the Viterbi algorithm, that is: 3-1) For the observation sequence O1n=O1, O2,...,On, the corresponding state sequence S1n=S1, S2,...,Sn has the conditional probability of existence, that is: Among them, Zo represents the normalization factor of the state sequence, fj represents the characteristic function that changes with the state, and λj represents the global characteristic weight; 3-2) Obtain the optimal feature weights through training, and use the Viterbi algorithm to obtain a globally optimal labeling sequence.

5. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: The TransE model training is as follows: entities and relationships are represented as vectors in a low-dimensional vector space through the TransE model, the vector representations of the entities and relationships are initialized, and the vectors are continuously optimized, including the following steps: 4-1) Each entity and relationship is represented by a vector of fixed dimension, that is, each entity and relationship is represented by a vector of dimension d; 4-2) Triples in the knowledge graph, namely: (entity vector, relationship vector, tail entity vector); The vector form of the triple in the knowledge graph, i.e.: h+r=t; Among them, h, r, and t represent the vector representations of the head entity, relation, and tail entity, respectively. 4-3) The TransE model is used to make the head entity vector plus the relationship vector in the triple equal to the tail entity vector, and the margin-based triple loss function is used to learn the vector representation of entities and relationships; 4-4) TransE optimizes the loss function through the stochastic gradient descent algorithm and continuously adjusts the vector representation of entities and relationships, so that the model can better fit the triple data in the knowledge graph; 4-5) Iterative Optimization: The TransE model iteratively optimizes the vector representations of entities and relations, and continuously updates the vector representations so that the model gradually learns the semantic relationship between entities and relations, so that the model learns accurate vector representations of entities and relations.

6. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: The construction of the knowledge graph includes the following steps: 5-1) Entity linking: Linking entities extracted from text with existing entities in the knowledge graph. Using entity linking technology, we can identify entities in the text and their corresponding entities in the knowledge graph. 5-2) Constructing an entity relationship graph: Based on the extracted relationships and vector representations between entities, a relationship graph between entities is constructed. The entity relationship graph is constructed by calculating the similarity between entities or the distance between relationship vectors. Entities and relationships are represented as nodes and edges in the relationship graph. 5-3) Knowledge graph reasoning: Use the GAT graph neural network model to represent the graph structure in the knowledge graph, so as to infer missing relationships or associations between entities.

7. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 6, characterized in that: The entity linking is matched based on entity name, context information, and entity attributes to ensure that the extracted entities are correctly associated with entities in the knowledge graph.

8. A method for realizing joint extraction of entities and relationships in a knowledge graph according to claim 1, characterized in that: In step 5), the entities and relationships in the knowledge graph are used for reasoning to discover new knowledge and relationships, specifically: Through the knowledge graph reasoning algorithm, we can discover hidden associations between entities and predict possible new relationships between entities, thereby enriching the content of the knowledge graph.

Images