Knowledge extraction algorithm supporting automatic construction of knowledge graph and based on RoBERTa-wwm, Bi-LSTM, CRF and attention mechanism

Abstract

The invention provides a knowledge extraction algorithm supporting automatic construction of a knowledge graph, efficient and accurate knowledge extraction is realized by combining RoBERTa-wwm, Bi-LSTM, CRF and an attention mechanism, and the technical key points are as follows: 1, text preprocessing: using a RoBERTa-wwm model to generate high-quality word vectors, and capturing text deep semantics; 2, sequence modeling: inputting word vectors into a Bi-LSTM network to carry out bidirectional sequence modeling, and enhancing the representation capability of context information; and 3, sequence decoding: decoding Bi-LSTM output by using a CRF model, and accurately identifying an entity. And 4, feature extraction and integration: extracting a feature vector from the output of RoBERTa-wwm by applying an Attention attention mechanism, and integrating a relationship and an entity feature. And 5, constructing a knowledge graph: applying the extracted entity and relationship information to automatic construction of the knowledge graph, and realizing storage and application of knowledge. According to the method, by fusing an advanced natural language processing technology, the automation level and accuracy of knowledge extraction are remarkably improved, and the method is suitable for knowledge graph construction requirements of various text data.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and relates to automated knowledge graph construction technology, specifically a knowledge extraction algorithm based on RoBERTa-wwm, Bi-LSTM, CRF and attention mechanisms. Background Technology

[0002] Knowledge graphs, as an important method of knowledge representation and storage, can effectively organize structured and semi-structured information into graphical structures, facilitating knowledge retrieval and reasoning. However, traditional knowledge graph construction methods often rely heavily on manual intervention, resulting in low efficiency and insufficient accuracy. With the advancement of natural language processing technology, automated knowledge extraction techniques have gradually become key to solving these problems.

[0003] In existing technologies, deep learning-based knowledge extraction methods have shown good results in entity recognition and relation extraction, but they still face challenges when processing complex text, such as insufficient context modeling and inaccurate information focusing. Therefore, improving the automation level and accuracy of knowledge extraction remains an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a knowledge extraction algorithm that supports the automated construction of knowledge graphs. This algorithm combines RoBERTa-wwm, Bi-LSTM, CRF and attention mechanisms to improve the accuracy and automation of knowledge extraction.

[0005] The technical solution of the present invention includes:

[0006] 1. Preprocessing text based on the RoBERTa-wwm model:

[0007] The RoBERTa-wwm model is used to segment the input text and generate corresponding word vector representations. As a pre-trained language model, the RoBERTa-wwm model can provide rich contextual information and semantic representations.

[0008] 2. Use a Bi-LSTM network to model word vector sequences:

[0009] The generated word vectors are input into a Bi-LSTM network for sequence modeling. The Bi-LSTM network can capture the bidirectional dependencies between words, thus representing the contextual information in the text more accurately.

[0010] 3. Sequence decoding using Conditional Random Fields (CRF):

[0011] The output of the Bi-LSTM network is used as the input to the CRF model, and constraints are added to decode the sequence. The CRF model can reduce the problem of incoherent entity labels in the output sequence and ensure the effectiveness of entity prediction results.

[0012] 4. Apply the Attention mechanism to extract feature vectors from the RoBERTa-wwm output:

[0013] The encoded vector sequence output by the RoBERTa-wwm model is reused, and the Attention mechanism is used to extract feature vectors with contextual information, integrating the extracted relationship and entity features.

[0014] 5. Construct a knowledge graph:

[0015] The entity and relation information obtained through the above steps is converted into "subject-relation-object" triples for constructing a knowledge graph. The construction of the knowledge graph includes named entity recognition, relation extraction, and knowledge storage and representation.

[0016] Beneficial effects

[0017] The knowledge extraction algorithm of this invention combines an advanced RoBERTa-wwm pre-trained model, a Bi-LSTM network, a CRF model, and an attention mechanism, which can effectively improve the automation level and accuracy of knowledge extraction. Through deep modeling and information weighting of text, this algorithm can better capture entities and relationships in text, support efficient knowledge graph construction, and thus meet the needs of practical application scenarios. Attached Figure Description

[0018] Figure 1 : Framework diagram of the knowledge extraction algorithm of this invention.

[0019] Figure 2 Flowchart of preprocessing input text statements for the RoBERTa-wwm model.

[0020] Figure 3 : A schematic diagram of a named entity recognition scheme.

[0021] Figure 4 Architecture diagram of a named entity recognition model based on RoBERTa-wwm, Bi-LSTM and CRF. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The specific implementation is not limited to the following description and may be adjusted according to actual circumstances.

[0023] Example 1: As Figure 2As shown in the flowchart, the RoBERTa-wwm model is applied to preprocess a piece of ordinary text to generate word vector representations. Then, the word vectors are input into a Bi-LSTM network for context modeling, followed by entity decoding and prediction using a CRF. Next, an Attention mechanism is used to extract feature vectors with contextual information, integrating relational and entity features, and concatenating the subject and object representations obtained from the acquired features. Finally, the relationships between entities are classified, and then predicted and converted into "subject-relationship-object" triples conforming to the graph pattern.

[0024] Example 2: As Figure 4 As shown, the algorithm of this invention is applied to military publication text data in the military field. By using RoBERTa-wwm, Bi-LSTM and CRF, and incorporating an Attention mechanism, the publication text is processed to extract entities in sentences and the relationships between entities in order to construct a military knowledge graph.

Claims

1. A knowledge extraction algorithm that supports automated construction of knowledge graphs, characterized by: (a) The text is preprocessed based on the RoBERTa-wwm model to extract word vector representations from the text; (b) Use a Bi-LSTM network to perform sequence modeling on word vectors to capture contextual information between words; (c) Use Conditional Random Field (CRF) to decode the optimal sequence, predict the sequence labeling results, and identify entity information; (d) Apply the Attention mechanism to extract feature vectors with contextual information, and integrate the extracted relationship and entity features; (e) The entity and relation information obtained through the above steps are converted into "subject-relation-object" triples for constructing a knowledge graph.

2. The algorithm according to claim 1, characterized in that, The RoBERTa-wwm model is used to process the input text through word segmentation and to encode the text using pre-trained word vectors.

3. The algorithm according to claim 1, characterized in that, Bi-LSTM networks employ bidirectional long short-term memory networks to bidirectionally model the input word vector sequence, thereby capturing the bidirectional dependencies of words in the context.

4. The algorithm according to claim 1, characterized in that, The CRF model is used to add constraints to decode the sequence output by the Bi-LSTM network and predict the sequence labeling results, thereby ensuring the effectiveness of the entity prediction results.

5. The algorithm according to claim 1, characterized in that, The Attention mechanism is used to extract contextually relevant feature vectors from the encoded vector sequence output by the RoBERTa-wwm model, integrating the extracted relational and entity features.

6. The algorithm according to claim 1, characterized in that, The construction of the knowledge graph includes named entity recognition, relation extraction, and knowledge storage and representation.

Images