Standard short text entity relation extraction method based on RoBERTa and convolutional neural network

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a standard short text entity relation extraction method based on RoBERTa and a convolutional neural network. The application uses a RoBERTa pre-training language model to perform upstream coding on a standard short text, uses a multi-layer bidirectional Transformer (Trm) network in the RoBERTa pre-training language model to perform information mining, and then a downstream network uses structural, semantic and other characteristics of the short text to extract entities and relations therebetween. Since the deep learning technology is used to extract entities and relations therebetween, the method has good generality and transfer learning capability, and also has good extraction effect on some standard short text templates that have not been learned.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks. Background Technology

[0002] Currently, with the development of artificial intelligence technology and its integration into the field of natural language processing, the field is ushering in a new wave of intelligence. Entity relation extraction, as one of the focuses of natural language processing, still requires the use of deep learning technology to improve the accuracy of entity relation extraction in standard short texts. At the same time, using deep learning technology can effectively save the cost of manually reconstructing extraction rules.

[0003] The current technology has the following shortcomings:

[0004] 1. The format of short texts is diverse. Currently, most extraction methods require manual formulation of corresponding extraction rules based on the characteristics of the short texts, which consumes a lot of manual labor costs.

[0005] 2. Standardized short texts have high information density. Compared with other texts, the number of characters that can be used as tags for learning between entities in short texts is relatively small. Therefore, entity extraction from standardized short texts is more difficult.

[0006] 3. The types of entities in standard short texts are diverse, and mainstream rule-based extraction methods cannot accurately distinguish the types of entities being identified, resulting in poor actual extraction results.

[0007] 4. Rule-based short text entity relation extraction methods have poor versatility. They need to be extracted from data with relatively fixed text formats that conform to their extraction rules. At the same time, rule-based extraction methods do not have the ability to transfer learning. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] The technical problem to be solved by this invention is: how to solve the problems of high error rate in entity type identification, poor effect in entity relationship extraction, high labor cost and poor versatility of rule-based entity relationship extraction methods for short texts, so as to improve the accuracy and versatility of entity relationship extraction methods for short texts.

[0010] (II) Technical Solution

[0011] To address the aforementioned technical problems, this invention provides a method for extracting entity relations from standardized short texts based on RoBERTa and convolutional neural networks. To better solve the problem of entity relation extraction from standardized short texts, this invention's method comprehensively considers factors such as the semantics, deep semantics, and inherent logic of the text's structure. The method includes the following steps:

[0012] Step 1: Cluster the features of the training data samples of the standardized short text, and label all training data samples with categories K. TS =1, 2, 3…N;

[0013] Step 2: Divide the extracted named entity types. There are a total of M named entity types in the training dataset. Label the named entities for all training samples in the training dataset for each of the N data sample types.

[0014] Step 3: Set the text encoding parameters of the RoBERTa pre-trained language model. The encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. Based on the encoding dictionary, it converts the characters in the standard short text into corresponding character codes. Through a bidirectional multi-layer TRM network, it mines the deep features of the standard short text layer by layer. After being encoded by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained.

[0015] Step 4: The mined feature encoding tensor is fitted twice using a two-layer linear fully connected network. The fitting results are then classified in parallel according to the entity type to effectively identify different types of entities and output the prediction results in the form of a numerical sequence.

[0016] Step 5: Determine whether there is a relationship between the identified entities, and obtain the entity text that shows a relationship between two entities;

[0017] Step 6: Output the entity text corresponding to the character position and the entity type determined; and output the entity text that has a relationship between the two entities.

[0018] In step 1, clustering is performed based on the data training sample features of the existing standardized short text training dataset. The sentence structure semantic features of the standardized short text are used as the basis for clustering the data training samples, and all data training samples in the sorted and classified N training datasets are labeled with categories.

[0019] In step 2, the extracted named entity types are divided according to the extraction target of the standard short text. There are a total of M named entity types in the training dataset. Named entities are labeled one by one for all data training samples in the training dataset of N data sample types. The labeling method is to label each character based on the constituent bits (Token) of the sentence.

[0020] In step 2, the constituent parts of the sentence are segmented by the encoder of the RoBERTa pre-trained language model, and the constituent parts T of the sentence are... i Perform character-by-character annotation, T i =0, 1, 2…M; where category 0 represents the constituent bits of named entities that do not need to be recognized; the named entity category labels of the training data samples are as follows: Figure 1 As shown.

[0021] In step 3, the text encoding parameters of the RoBERTa pre-trained language model are set. The maximum encoding amount of the encoder of the RoBERTa pre-trained language model is set according to the longest text character count in the training dataset. The maximum encoding amount is 20 characters larger than the longest text, so as to improve the generality of the model for training samples with encoding lengths exceeding those in the training dataset in application scenarios.

[0022] In step 3, the encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. All sample data in the completed training dataset have the same length, so that the subsequent pre-trained model can learn from the data training samples in batches, thereby improving the performance and training efficiency of the entity relation extraction model.

[0023] In step 3, the text in the standard short text is converted into corresponding character codes based on the encoding dictionary, and the attention feature value of "padding position" is set to 0, while the attention value of "text character position" is set to 1. The RoBERTa pre-trained model mines the deep features of the standard short text layer by layer through a bidirectional multi-layer TRM network, thereby better analyzing the features of entity constituent positions and entity types. After encoding by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained. This workflow is as follows: Figure 2 As shown.

[0024] In step 4, the parallel binary classification entity extraction network is a downstream task model for entity extraction. This network first fits the mined feature encoding tensor through a first-layer linear fully connected network. Then, it uses the Doupout layer of the first-layer linear fully connected network to randomly delete some neurons. The random deletion rate of the Doupout layer is 30%, that is, only 70% of the original fitted features are retained each time and sent to the second-layer linear fully connected network for secondary fitting. Finally, the output of the second-layer linear fully connected network is sent to the parallel binary classifier.

[0025] The parallel binary classifier is designed based on entity types. Each entity type corresponds to a binary classification neural network, which is a linear fully connected layer with two output neurons. These neurons determine whether a character belongs to a particular entity type: 1 if it does, and 0 otherwise. The binary classification network is trained for each of the different entity types in the text, reducing information entropy in the multi-class network and thus improving entity recognition accuracy. Furthermore, overlapping entity types may exist among the entities to be extracted; the parallel classifier effectively identifies multiple entity types.

[0026] In step 5, the convolutional entity relation extraction neural network is used to further determine whether there is a relationship between the extracted entities. The input of the convolutional entity relation extraction neural network consists of three parts: the feature encoding tensor of the standard short text, the feature encoding of entity 1, and the feature encoding of entity 2. The feature size of the standard short text is Q×S, where Q is the character length in the standard short text and S is the encoding feature length of a character in the standard short text. The semantic feature tensor of the standard short text is extracted by the convolutional neural network (CNN), and the dimension of the extracted semantic feature tensor is 1×S.

[0027] The feature encoding tensors of entity 1 and entity 2 are summed and averaged according to their character bits, and the 1-dimensional tensor of the summation is used as the entity feature tensor of the two entities. The entity feature tensors of the two entities are then concatenated with the semantic feature tensor. The fused feature of the concatenation is used as the input of the fully connected network. The output of the fully connected network is the prediction result of whether there is a relationship between the two entities. If there is a relationship, the output is "1"; if there is no relationship, the output is "0".

[0028] In step 6, the numerical sequence results predicted by the network model are converted into standard text feature content. First, based on the numerical sequence prediction results output by the parallel binary entity extraction network in step 4, the entity text corresponding to the character position and the determined entity type are output. Second, the entity texts of the two entities that have a relationship in step 5 are output. The output result is a set in the format "[[entity1 text, entity1 type], relationship type, [entity2 text, entity2 type]]".

[0029] (III) Beneficial Effects

[0030] To address the high-density information and diverse entity types inherent in standardized short text extraction, this invention employs a RoBERTa pre-trained language model for upstream encoding. Information mining is then performed using a multi-layer bidirectional Transformer (Trm) network within the RoBERTa pre-trained language model. Subsequently, the downstream network utilizes the structural and semantic features of the short text to extract entities and their relationships. Because deep learning techniques are used for entity and relationship extraction, this method exhibits good generality and transfer learning capabilities, demonstrating good extraction performance even for standard short text templates that have not undergone learning.

[0031] 1. Key points of the present invention compared with the prior art

[0032] (1) The semantic and structural features of standard short texts are mined by using the multi-layer parallel TRM structure in the RoBERTa pre-trained language model. The convergence of downstream models in the entity relation extraction model is accelerated by using the pre-trained language model, and the amount of data required to train the model is reduced.

[0033] (2) The traditional horizontal multi-class classifier is changed to multiple parallel vertical binary classifiers. Each binary classifier only performs the identification of a single type of entity, thereby reducing the information entropy of the classifier and improving the accuracy of entity type identification. At the same time, this parallel design can also solve the problem of overlapping entity types.

[0034] (3) This invention utilizes fusion features to improve the accuracy of entity relationship extraction. It further analyzes the deep semantic features of short texts through convolutional neural networks and fuses them with the feature tensors of two entities. Through comprehensive reasoning based on fusion features, it predicts the two-point relationship between the two entities.

[0035] (4) The entity relationship extraction model based on the RoBERTa pre-trained language model and convolutional neural network has good versatility. It still has a good extraction effect on short text styles that are not in the training dataset, saving the cost of secondary manual annotation.

[0036] (5) The entity relationship extraction model constructed by deep learning in this invention has strong transfer learning ability. For some standard short texts with poor extraction effect, such data can be added to the training dataset and the model parameters can be fine-tuned to improve the extraction effect of such standard short texts.

[0037] 2. Effects of the present invention:

[0038] This invention proposes a method for extracting entity relationships from standardized short texts based on RoBERTa and convolutional neural networks. It employs a RoBERTa pre-trained language model to encode the text, deeply mining its semantic and structural features. This reduces the amount of training data while further improving the entity extraction performance of downstream models. The downstream task extraction model, namely the parallel binary classification entity extraction network, incorporates a parallel binary classifier to improve the accuracy of entity type identification and address the issue of overlapping entity types. After entity identification, a convolutional neural network is used to extract the semantic feature tensor of the standardized short text. The fusion features of entity features and text semantics are then used to predict and infer relationships between different entities. The predicted numerical sequence results from the entity extraction model are then compared with the original text, and each element is translated into a textual relationship result. This entity relationship extraction model improves the accuracy of entity relationship identification and possesses good generality. Furthermore, it allows for transfer learning for new domain data, further expanding the model's application scenarios. Attached Figure Description

[0039] Figure 1 A diagram illustrating the naming entity type annotation for sample data.

[0040] Figure 2 A schematic diagram of the encoding network structure for the RoBERTa pre-trained model.

[0041] Figure 3 This is a schematic diagram of a parallel binary entity extraction network structure.

[0042] Figure 4 Workflow diagram for extracting entity relations from a convolutional neural network. Detailed Implementation

[0043] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0044] To address the aforementioned technical problems, this invention provides a method for extracting entity relations from standardized short texts based on RoBERTa and convolutional neural networks. To better solve the problem of entity relation extraction from standardized short texts, this invention's method comprehensively considers factors such as the semantics, deep semantics, and inherent logic of the text's structure. The method includes the following steps:

[0045] Step 1: Cluster the features of the training data samples of the standardized short text, and label all training data samples with categories K. TS =1, 2, 3…N;

[0046] Step 2: Divide the extracted named entity types. There are a total of M named entity types in the training dataset. Label the named entities for all training samples in the training dataset for each of the N data sample types.

[0047] Step 3: Set the text encoding parameters of the RoBERTa pre-trained language model. The encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. Based on the encoding dictionary, it converts the characters in the standard short text into corresponding character codes. Through a bidirectional multi-layer TRM network, it mines the deep features of the standard short text layer by layer. After being encoded by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained.

[0048] Step 4: The mined feature encoding tensor is fitted twice using a two-layer linear fully connected network. The fitting results are then classified in parallel according to the entity type to effectively identify different types of entities and output the prediction results in the form of a numerical sequence.

[0049] Step 5: Determine whether there is a relationship between the identified entities, and obtain the entity text that shows a relationship between two entities;

[0050] Step 6: Output the entity text corresponding to the character position and the entity type determined; and output the entity text that has a relationship between the two entities.

[0051] In step 1, clustering is performed based on the data training sample features of the existing standardized short text training dataset. The sentence structure semantic features of the standardized short text are used as the basis for clustering the data training samples, and all data training samples in the sorted and classified N training datasets are labeled with categories.

[0052] In step 2, the extracted named entity types are divided according to the extraction target of the standard short text. There are a total of M named entity types in the training dataset. Named entities are labeled one by one for all data training samples in the training dataset of N data sample types. The labeling method is to label each character based on the constituent bits (Token) of the sentence.

[0053] In step 2, the constituent parts of the sentence are segmented by the encoder of the RoBERTa pre-trained language model, and the constituent parts T of the sentence are... i Perform character-by-character annotation, T i =0, 1, 2…M; where category 0 represents the constituent bits of named entities that do not need to be recognized; the named entity category labels of the training data samples are as follows: Figure 1 As shown.

[0054] In step 3, the text encoding parameters of the RoBERTa pre-trained language model are set. The maximum encoding amount of the encoder of the RoBERTa pre-trained language model is set according to the longest text character count in the training dataset. The maximum encoding amount is 20 characters larger than the longest text, so as to improve the generality of the model for training samples with encoding lengths exceeding those in the training dataset in application scenarios.

[0055] In step 3, the encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. All sample data in the completed training dataset have the same length, so that the subsequent pre-trained model can learn from the data training samples in batches, thereby improving the performance and training efficiency of the entity relation extraction model.

[0056] In step 3, the text in the standard short text is converted into corresponding character codes based on the encoding dictionary, and the attention feature value of "padding position" is set to 0, while the attention value of "text character position" is set to 1. The RoBERTa pre-trained model mines the deep features of the standard short text layer by layer through a bidirectional multi-layer TRM network, thereby better analyzing the features of entity constituent positions and entity types. After encoding by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained. This workflow is as follows: Figure 2 As shown.

[0057] In step 4, the parallel binary classification entity extraction network is a downstream task model for entity extraction. This network first fits the mined feature encoding tensor through a first-layer linear fully connected network. Then, it uses the Doupout layer of the first-layer linear fully connected network to randomly delete some neurons. The random deletion rate of the Doupout layer is 30%, that is, only 70% of the original fitted features are retained each time and sent to the second-layer linear fully connected network for secondary fitting. Finally, the output of the second-layer linear fully connected network is sent to the parallel binary classifier.

[0058] The parallel binary classifier is designed based on entity types. Each entity type corresponds to a binary classification neural network, which is a linear fully connected layer with two output neurons. These neurons determine whether a character belongs to a particular entity type: 1 if it does, and 0 otherwise. The binary classification network is trained for each of the different entity types in the text, reducing information entropy in the multi-class network and thus improving entity recognition accuracy. Furthermore, overlapping entity types may exist among the entities to be extracted; the parallel classifier effectively identifies multiple entity types.

[0059] In step 5, the convolutional entity relation extraction neural network is used to further determine whether there is a relationship between the extracted entities. The input of the convolutional entity relation extraction neural network consists of three parts: the feature encoding tensor of the standard short text, the feature encoding of entity 1, and the feature encoding of entity 2. The feature size of the standard short text is Q×S, where Q is the character length in the standard short text and S is the encoding feature length of a character in the standard short text. The semantic feature tensor of the standard short text is extracted by the convolutional neural network (CNN), and the dimension of the extracted semantic feature tensor is 1×S.

[0060] The feature encoding tensors of entity 1 and entity 2 are summed and averaged according to their character bits, and the 1-dimensional tensor of the summation is used as the entity feature tensor of the two entities. The entity feature tensors of the two entities are then concatenated with the semantic feature tensor. The fused feature of the concatenation is used as the input of the fully connected network. The output of the fully connected network is the prediction result of whether there is a relationship between the two entities. If there is a relationship, the output is "1"; if there is no relationship, the output is "0".

[0061] In step 6, the numerical sequence results predicted by the network model are converted into standard text feature content. First, based on the numerical sequence prediction results output by the parallel binary entity extraction network in step 4, the entity text corresponding to the character position and the determined entity type are output. Second, the entity texts of the two entities that have a relationship in step 5 are output. The output result is a set in the format "[[entity1 text, entity1 type], relationship type, [entity2 text, entity2 type]]".

[0062] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks, characterized in that, The method includes the following steps: Step 1: Cluster the features of the training data samples of the standardized short text, and label all training data samples with categories K. TS =1, 2, 3…N; Step 2: Divide the extracted named entity types. There are a total of M named entity types in the training dataset. Label the named entities for all training samples in the training dataset for each of the N data sample types. Step 3: Set the text encoding parameters of the RoBERTa pre-trained language model. The encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. Based on the encoding dictionary, it converts the characters in the standard short text into corresponding character codes. Through a bidirectional multi-layer TRM network, it mines the deep features of the standard short text layer by layer. After being encoded by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained. Step 4: The mined feature encoding tensor is fitted twice using a two-layer linear fully connected network. The fitting results are then classified in parallel according to the entity type to effectively identify different types of entities and output the prediction results in the form of a numerical sequence. Step 5: Determine whether there is a relationship between the identified entities, and obtain the entity text that shows a relationship between two entities; Step 6: Output the entity text corresponding to the character position and the entity type determined; and output the entity text that has a relationship between the two entities.

2. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 1, characterized in that, In step 1, clustering is performed based on the data training sample features of the existing standardized short text training dataset. The sentence structure semantic features of the standardized short text are used as the basis for clustering the data training samples, and all data training samples in the sorted and classified N training datasets are labeled with categories.

3. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 2, characterized in that, In step 2, the extracted named entity types are divided according to the extraction target of the standard short text. There are a total of M named entity types in the training dataset. Named entities are labeled one by one for all data training samples in the training dataset of N data sample types. The labeling method is to label each part of the sentence.

4. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 3, characterized in that, In step 2, the constituent parts of the sentence are segmented by the encoder of the RoBERTa pre-trained language model, and the constituent parts T of the sentence are... i Perform character-by-character annotation, T i =0, 1, 2…M; where type 0 represents the constituent bits of named entities that do not need to be identified.

5. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 4, characterized in that, In step 3, the text encoding parameters of the RoBERTa pre-trained language model are set. The maximum encoding amount of the encoder of the RoBERTa pre-trained language model is set according to the longest text character count in the training dataset. The maximum encoding amount is increased by 20 characters based on the longest text, so as to improve the generality of the model for training samples with encoding lengths exceeding those in the training dataset in application scenarios.

6. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 5, characterized in that, In step 3, the encoder of the RoBERTa pre-trained language model completes the standard short text according to the set maximum encoding amount. All sample data in the completed training dataset have the same length, so that the subsequent pre-trained model can learn from the data training samples in batches, thereby improving the performance and training efficiency of the entity relation extraction model.

7. The method for extracting entity relations from standard short texts based on RoBERTa and convolutional neural networks as described in claim 6, characterized in that, In step 3, the text in the standard short text is converted into the corresponding character encoding based on the encoding dictionary, and the attention feature value of "padding position" is set to 0, and the attention value of "text character position" is set to 1. The RoBERTa pre-trained model mines the deep features of the standard short text layer by layer through a bidirectional multi-layer TRM network, so as to better analyze the features of entity constituent positions and entity types. After encoding by the RoBERTa pre-trained model, the feature encoding tensor of the standard short text is obtained.

8. The method for extracting entity relationships from standard short texts based on RoBERTa and convolutional neural networks as described in claim 7, characterized in that, In step 4, the parallel binary classification entity extraction network is the downstream task model of entity extraction. This network first fits the mined feature encoding tensor through the first layer of the linear fully connected network. Then, it uses the Dropout layer of the first layer of the linear fully connected network to randomly delete some neurons. The random deletion rate of the Dropout layer is 30%, that is, only 70% of the original fitted features are retained each time and sent to the second layer of the linear fully connected network for secondary fitting. Finally, the output of the second layer of the linear fully connected network is sent to the parallel binary classifier. The parallel binary classifier is designed according to the entity type. Each entity type has a corresponding binary classification neural network. The binary classification neural network for each entity is a linear fully connected layer with two output neurons. The output neurons determine whether the character belongs to this entity type. If the character belongs to this entity type, the output is 1; otherwise, the output is 0. The binary classification network is trained for each of the different entity types in the text, thereby reducing the information entropy in the multi-class network and improving the accuracy of entity recognition. Furthermore, there may be overlapping categories among the entities to be extracted, and multiple categories of entities can be effectively identified through parallel classifiers.

9. The method for extracting entity relations from standard short texts based on RoBERTa and convolutional neural networks as described in claim 8, characterized in that, In step 5, the convolutional entity relation extraction neural network is used to further determine whether there is a relationship between the extracted entities. The input of the convolutional entity relation extraction neural network consists of three parts, including the feature encoding tensor of the standard short text, the feature encoding of entity 1, and the feature encoding of entity 2. The standard short text feature size is Q×S, where Q is the character length in the standard short text and S is the encoding feature length of a character in the standard short text. The semantic feature tensor of the standard short text is extracted through a convolutional neural network, and the extracted semantic feature tensor has a dimension of 1×S. The feature encoding tensors of entity 1 and entity 2 are summed and averaged according to their character bits, and the 1-dimensional tensor of the summation is used as the entity feature tensor of the two entities. The entity feature tensors of the two entities are then concatenated with the semantic feature tensor. The concatenated fused features are used as the input of the fully connected network. The output of the fully connected network is the prediction result of whether there is a relationship between the two entities. If there is a relationship, the output is "1"; if there is no relationship, the output is "0".

10. The method for extracting entity relations from standard short texts based on RoBERTa and convolutional neural networks as described in claim 9, characterized in that, In step 6, the numerical sequence results predicted by the network model are converted into standard text feature content. First, based on the numerical sequence prediction results output by the parallel binary entity extraction network in step 4, the entity text corresponding to the character position and the determined entity type are output. Second, the entity texts of the two entities that have a relationship in step 5 are output. The output result is a set in the format "[[entity1 text, entity1 type], relationship type, [entity2 text, entity2 type]]".

Images