EBAC fish text joint extraction method integrated with ResNeXt

Abstract

This invention belongs to the field of joint extraction of text entity relationships and relates to a ResNeXt-integrated EBAC fish text joint extraction method. This method uses fish science popularization text as input into a deep learning model for joint entity relationship extraction. The deep learning model includes: an ERNIE pre-training model that encodes the fish science popularization text to obtain corresponding word vectors; a BiLSTM model that captures dependencies in word vectors to obtain corresponding hidden state vectors; an Attention model that assigns weights to the hidden state vectors to generate corresponding semantic vectors. The Attention model incorporates ResNeXt concepts to improve extraction performance without increasing model complexity; and a CRF decoder that decodes the semantic vectors to obtain canonical labels, completing joint entity relationship extraction. This invention addresses the problems of weak semantics between entities, semantic dilution over long sequences, and long-distance dependencies, achieving excellent extraction performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of text entity relationship joint extraction, and specifically relates to an EBAC (Enhanced Representation from kNowledge IntEgration+Attention mechanism+Bi-directional Long Short-Term Memory+Conditional Random Field, EBAC) fish popular science text entity relationship joint extraction method based on the ResNeXt idea. Background Art

[0002] In recent years, the task of managing fish science information has become increasingly complex, resulting in a large amount of unstructured fish-related texts containing redundant information. Currently, entity relationship extraction methods are mainly used to obtain useful feature information from these massive texts.

[0003] Miwa et al. proposed an end-to-end model based on BiLSTM and Tree-LSTM structures to extract entities and relations simultaneously. However, the process of obtaining triples is still a pipeline method. That is, when all entities are identified and sent to the relation classification module in pairs to predict the relationship between them, some entity pairs that do not have a relationship are also input into the module, resulting in information redundancy. In addition, when detecting entities, the long-distance dependency between entity labels is ignored (Makoto Miwa and Mohit Bansal. End-to-end Relation Extraction using LSTMs on Sequences and Tree Structures. [J]. CoRR, 2016, abs / 1601.00770.); Zheng et al. proposed to pass the input sentence through a common Embedding layer and BiLSTM model to solve the long-distance dependency problem, and then use an LSTM for entity recognition and a CNN for relation extraction (Suncong Zheng, et al. Joint entity and relation extraction based on a hybrid neural network). network[J].Neurocomputing,2016,257:59-66.); Zheng et al. proposed a tagging mechanism to solve the information redundancy problem, which is to convert the joint extraction problem into a sequence tagging problem. However, when the labels are merged into entity-relationship triples, the relationship overlaps due to the nearest combination (Suncong Zheng,et al.Joint extraction of entities and relations based on anovel tagging scheme[J].ar Xiv preprint ar Xiv,2017.1706(1706).); Wang et al. proposed to convert the joint extraction task into a directed graph problem by designing a directed graph mechanism, and used a transfer-based parsing framework to solve the overlapping problem of relationships between an entity and multiple entities (Shaolei Wang,Yue Zhang,Wanxiang Che,et al.Joint extraction of entities and relations based on anovel graph scheme[J].2018.). However, compared with the above-mentioned general fields, the existing chapter-level fish science texts not only have overlapping problems, but also have a large number of problems with weak semantics between entities. Therefore, a knowledge enhancement pre-training model based on transfer learning is used to solve this problem.

[0004] The application of transfer learning in natural language processing tasks is usually reflected through pre-training models. The knowledge-enhanced pre-training model ERNIE models words, entities, and entity relationships through massive data. In response to the problem that BERT has difficulty learning complete semantic representation when processing Chinese, Sun et al. proposed the ERNIE model to directly model prior semantic knowledge units, thereby enhancing the semantic representation ability of the model (Sun Y, Wang S, Li Y, et al. ERNIE: Enhanced Representation through Knowledge Integration[J]. 2019.); Niu Yuting et al. integrated the ERNIE pre-training word model with the DPC-NN model for Chinese text classification, and improved the ERNIE-DPCNN model. The advantages of the ERNIE model, such as its ability to capture long-distance information, improved the model's accuracy and reduced costs (Niu Yuting, Chen Boqi, Chen Bin. Chinese text classification based on the improved ERNIE-DPCNN model[J]. Journal of Jiangsu Normal University (Natural Science Edition), 2021, 39(01): 47-52.).

[0005] Although there are a large number of general-purpose entity relationship extraction research methods to deal with text overlap, they cannot effectively solve the problems of weak semantics between a large number of entities in chapter-level fish science texts, long-sequence semantic dilution, and long-distance dependencies. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides an EBAC fish text joint extraction method integrated with ResNeXt.

[0007] The purpose of the present invention is to provide a EBAC fish text joint extraction method integrating ResNeXt, comprising:

[0008] The fish science text is input into the deep learning model for entity relationship joint extraction;

[0009] The deep learning model includes:

[0010] The ERNIE pre-trained model is used to encode the fish science text to obtain corresponding word vectors;

[0011] A BiLSTM model is used to capture the dependencies in the word vectors and obtain the corresponding hidden state vectors;

[0012] The Attention model is used to assign weights to the hidden state vector and generate corresponding semantic vectors. The Attention model integrates the ResNeXt concept to improve the extraction effect without increasing the model complexity.

[0013] The CRF decoder is used to decode the semantic vector to obtain corresponding canonical labels and determine entity relationships according to the canonical labels.

[0014] Preferably, the above-mentioned EBAC fish text joint extraction method integrating ResNeXt needs to train the BiLSTM model before extracting the entity relationship, and the training method is:

[0015] Obtain fish science popularization texts and establish a fish science popularization text database;

[0016] Using some fish science popularization texts in the fish science popularization text library as a training set;

[0017] Annotating the fish science texts in the training set;

[0018] Input the fish science text in the training set into the ERNIE pre-training model to obtain the corresponding word vector;

[0019] Input the word vector into the BiLSTM model to obtain the corresponding hidden state vector;

[0020] Input the hidden state vector into the Attention model that integrates the ResNeXt concept to obtain the corresponding semantic vector;

[0021] Decode the semantic vector using a CRF decoder to obtain corresponding canonical labels and entity relationships;

[0022] The deep learning model is adjusted according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set to complete the training of the deep learning model.

[0023] Preferably, the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, after using part of the fish science popularization texts in the fish science popularization text library as a training set, further includes:

[0024] Using some fish science popularization texts in the fish science popularization text library as a test set;

[0025] After adjusting the deep learning model according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set, the method further includes:

[0026] The adjusted deep learning model is tested using the test set.

[0027] Preferably, the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, after obtaining the fish science popularization text, further includes:

[0028] The fish science text is cleaned.

[0029] Preferably, the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, the method for obtaining and cleaning the fish popular science text is as follows:

[0030] The fish science information from multiple encyclopedia websites was obtained by Python crawler, and then integrated and cleaned to remove noise to obtain fish science text.

[0031] Preferably, in the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, the fish science information includes fish level classification, nicknames, main distribution locations, common habitats, preferred prey objects, and origins of first-generation fish species.

[0032] Preferably, in the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, the ratio of the number of fish science texts in the training set and the test set is 7:3.

[0033] Preferably, the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt annotates the fish science text as follows:

[0034] According to the characteristics of fish science popularization texts, annotation tools are used and the "BIOS" method is used for annotation.

[0035] Preferably, the above-mentioned EBAC fish text joint extraction method integrated with ResNeXt, the method for obtaining the hidden state vector is as follows:

[0036] The BiLSTM model is used to merge the results of the forward hidden layer vector and the reverse hidden layer vector processing and use them as the hidden state vector of the current character.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. To address the issues of weak semantics between entities, long-distance dependencies, and long-sequence semantic dilution in chapter-level fish science popularization text, this paper proposes a joint fish text extraction method integrating EBAC with ResNeXt. This method effectively improves the extraction of chapter-level fish science popularization text. Compared with existing entity relationship extraction methods, this method achieves the best results on fish science popularization text, achieving good results in precision, recall, and F1-score. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the process of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand and implement the technical solution of the present invention, the present invention is further described below with reference to specific embodiments and drawings.

[0041] The present invention is a fusion of ResNeXt EBAC fish text joint extraction method, which is to send the annotated fish popular science text into the deep learning model for entity relationship joint extraction. Figure 1 The steps shown are as follows:

[0042] Step 1: Crawl, integrate, and clean approximately 220,000 characters from multiple encyclopedia websites to build a fish science text library, covering popular science information on various marine fish (hereinafter referred to as fish), including their classification, nicknames, main distribution locations, habitats, preferred prey, and origins of first-generation fish species.

[0043] Specifically, a large amount of fish science information was obtained from encyclopedia websites such as Wanwei Encyclopedia, Baidu Encyclopedia, and 360 Encyclopedia through Python crawler, and manual cleaning was performed to remove noise that has nothing to do with fish science information. The obtained text was used to construct a fish science text library for the joint extraction of entity relationships.

[0044] Step 2: Divide the fish science text library into a training set and a test set, with the ratio of fish science texts between the two being 7:3. Based on the characteristics of the fish science texts, the annotation tool "BIOS" was used to annotate them, where "NAME" represents the subject category and "OTH" is the collective name for all object relationship categories. This method has 21 annotation types: 3 for subjects, 17 for objects, and 1 for non-entities.

[0045] Based on the needs of today's fish science industry and the feature analysis of the entire text, seven highly relevant entity relationships were selected for extraction tasks, including Latin scientific name entity relationship, alias entity relationship, hyponym entity relationship, predation type entity relationship, habitat entity relationship, distribution location entity relationship and model origin entity relationship.

[0046] Traditional annotation schemes use the "BIO" notation to indicate the position of each character within an entity. B indicates that the character is the first character in the entity, I indicates that the character is not the first character in the entity, and O indicates a non-entity character. Based on the characteristics of fish science knowledge, a "BIOS" tag category definition is proposed. This adds an S tag to the "BIO" position tag to indicate that the entity is a single character. NAME represents the subject category, and OTH is a general term for all object relationship categories, represented by the letters XM, BM, SX, BS, QX, FB, and CD, respectively, for a total of 21 annotation forms.

[0047] Step 3: Use the ERNIE pre-trained model, based on transfer learning, to encode the acquired fish science text and convert characters into word vectors. This model, having learned a vast amount of prior semantic knowledge, identifies and masks phrases and entities to improve its ability to learn semantic representations between entities. This enhances the model's semantic expressiveness and addresses the problem of weak semantics between entities.

[0048] Step 4: Input the word vector obtained by ERNIE into the BiLSTM model. The BiLSTM model is composed of a forward LSTM and a backward LSTM.

[0049] The LSTM model solves the problem of vanishing or exploding gradients caused by a large number of consecutive multiplications during the gradient propagation process in the RNN model. It consists of four parts: a forget gate, an input gate, an internal memory unit, and an output gate. The model can learn which information to remember and forget through training, thereby effectively capturing long-range dependencies. The state calculation of each LSTM unit is expressed as:

[0050] f t =σ(W f ·[h t-1 ,x t ]+b f ) (1)

[0051] i t =σ(W i ·[h t-1 ,x t ]+b i ) (2)

[0052]

[0053]

[0054] O t =σ(W o [h t-1 ,x t ]+b o ) (5)

[0055] h t =o t *tanh(C t ) (6)

[0056] Where f, i, and O are the forget gate, input gate, and output gate in the LSTM model respectively; C t 、 C t-1 represents the internal memory unit; x is the input at time t; W is the weight matrix; b is the bias; σ is the sigmoid activation function; h t is the output at time t. However, LSTM cannot encode information from the back to the front, so BiLSTM comes into being.

[0057] The BiLSTM model is used to merge the results of the forward hidden layer vector and the reverse hidden layer vector processing and use them as the hidden state vector of the current character. The processing process is expressed as:

[0058] v t =[h t :h t ′] (7)

[0059] V={v 1, …,v t, …,v T} (8)

[0060] where h t and h t ′ represents the forward hidden layer vector and the reverse hidden layer vector respectively; V represents the combination of hidden layer vectors at all times, which is transmitted to the Attention model as the output of the BiLSTM model; t represents time.

[0061] Step 5: The hidden state vector obtained by the BiLSTM model is passed to the Attention model. The Attention model can assign weights to information and generate continuously changing semantic vectors to enable the model to focus on key information and suppress useless information. Each word output includes an additional attention vector. The attention weight is determined by the hidden state at the previous moment and the hidden state of each encoder input. The calculation process is expressed as:

[0062] M=tanh(H) (9)

[0063] α=Softmax(ω T M) (10)

[0064] r′=Hα T (11)

[0065] r=tanh(r′) (12)

[0066] Where M is the result of H connected to an activation function of tanh, H is the set of outputs of the upper hidden layer, that is, H = V, where H∈Rdw×T, dw is the dimension of the word vector, T is the length of the sentence; ω is the trained parameter vector, ω T Transpose it; α is the weight matrix, similarly, α T is its transpose; r′ is the distributed representation of the sentence; and r is the output of the layer.

[0067] In addition, the Attention model incorporates the ResNeXt concept derived from the incomplete network and stacks the Attention mechanism in parallel. The best effect is achieved when the parallel hyperparameter is 3, and overfitting will occur if the parallel hyperparameter exceeds 3 dimensions. Each branch uses a serial multi-layer attention mechanism, which improves the accuracy of the model without significantly increasing the number of parameters. At the same time, due to the same topological structure, this strategy reduces the selection of hyperparameters. Therefore, the Attention model improves the extraction effect without increasing the complexity of the model.

[0068] Step 6: Finally, the CRF decoder is introduced for decoding to obtain the standardized canonical labels. The entity relationship is determined based on the canonical labels. Finally, the comparison results of the obtained entity relationship and the canonical labels are standardized and output.

[0069] It should be noted that before applying the deep learning model, the deep learning model needs to be trained. The training method is:

[0070] Obtain fish science popularization texts and establish a fish science popularization text database;

[0071] Using some fish science popularization texts in the fish science popularization text library as a training set;

[0072] Annotating the fish science texts in the training set;

[0073] Input the fish science text in the training set into the ERNIE pre-training model to obtain the corresponding word vector;

[0074] Input the word vector into the BiLSTM model to obtain the corresponding hidden state vector;

[0075] Input the hidden state vector into the Attention model that integrates the ResNeXt concept to obtain the corresponding semantic vector;

[0076] Decoding the semantic vector using a CRF decoder to obtain corresponding canonical labels, and determining entity relationships based on the canonical labels;

[0077] The deep learning model is adjusted according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set to complete the training of the deep learning model.

[0078] After using some of the fish science popularization texts in the fish science popularization text library as a training set, the following is also included:

[0079] Using some fish science popularization texts in the fish science popularization text library as a test set;

[0080] After adjusting the deep learning model according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set, the method further includes:

[0081] The adjusted deep learning model is tested using the test set.

[0082] The entity relationship extraction effect data of the embodiment of the present invention (i.e., the comparison results of entity relationships and standard labels) are shown in Table 1. Table 1 shows the extraction effect data of 7 specific entity relationships and the overall extraction effect data from top to bottom.

[0083] Table 1 Entity relationship extraction effect data

[0084]

[0085] Table 2 shows performance comparisons between the embodiments of the present invention and other models. This includes a comparative analysis of the impact of the attention mechanism with the ERNIE+BiLSTM+CRF model, and a comparative analysis of the impact of pre-training with the BiLSTM+Atttention+CRF model. EBC and BAC refer to the ERNIE+BiLSTM+CRF and BiLSTM+Atttention+CRF module structures, respectively. Furthermore, this experiment also conducted an external module performance study using the recently proposed ERNIE model, a classic pre-trained model in this research field, BERT, to examine whether ERNIE is more suitable for this application. The proposed model is more accurate and effectively improves extraction performance.

[0086] Table 2 Performance comparison data between different models

[0087]

[0088] Table 3 shows a performance comparison between the implementation of the present invention and a model without transfer learning. Based on the learning methods of the pre-trained model, we investigated the effectiveness of domain relearning using transfer learning, and compared the results with no relearning. The results demonstrate that the model constructed using the implementation of the present invention can effectively extract entity relationships from text, significantly improving the F1 score for entity relationship triple extraction.

[0089] Table 3. Performance comparison of models without transfer learning

[0090]

[0091] It should be noted that when numerical ranges are mentioned in the present invention, it should be understood that both endpoints of each numerical range and any value between the two endpoints may be selected. Since the steps and methods used are the same as those in the embodiments, in order to avoid redundancy, the present invention describes preferred embodiments. Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they understand the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0092] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. The EBAC fish text joint extraction method integrated with ResNeXt is characterized by: include: The fish science text is input into the deep learning model for entity relationship joint extraction; The deep learning model includes: The ERNIE pre-trained model is used to encode the fish science text and obtain corresponding word vectors; A BiLSTM model is used to capture the dependencies in the word vectors and obtain the corresponding hidden state vectors; The Attention model is used to assign weights to the hidden state vector and generate corresponding semantic vectors. The Attention model integrates the ResNeXt concept to improve the extraction effect without increasing the model complexity. A CRF decoder is used to decode the semantic vector to obtain corresponding canonical labels and determine entity relationships based on the canonical labels; Before extracting the entity relationship, the BiLSTM model needs to be trained. The training method is: Obtain fish science popularization texts and establish a fish science popularization text database; Using some fish science popularization texts in the fish science popularization text library as a training set; Annotating the fish science texts in the training set; Input the fish science text in the training set into the ERNIE pre-training model to obtain the corresponding word vector; The character vector is input into the BiLSTM model to obtain the corresponding hidden state vector. The hidden state vector is obtained as follows: the BiLSTM model is used to merge the results of the forward hidden layer vector and the reverse hidden layer vector processing and use it as the hidden state vector of the current character. The processing process is expressed as follows: v t =[h t :h t '] V={v 1, …,v t, …,v T } Among them, h t and h t ' respectively represent the forward hidden layer vector and the reverse hidden layer vector; V represents the combination of hidden layer vectors at all times, which is transmitted to the Attention model as the output of the BiLSTM model; t represents time; Input the hidden state vector into the Attention model that integrates the ResNeXt concept to obtain the corresponding semantic vector; Decode the semantic vector using a CRF decoder to obtain corresponding canonical labels and entity relationships; The deep learning model is adjusted according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set to complete the training of the deep learning model.

2. The EBAC fish text joint extraction method integrating ResNeXt according to claim 1 is characterized in that, After using some of the fish science popularization texts in the fish science popularization text library as a training set, the following is also included: Using some fish science popularization texts in the fish science popularization text library as a test set; After adjusting the deep learning model according to the error between the entity relationship obtained by decoding the CRF decoder and the entity relationship annotated by the fish science text in the training set, the method further includes: The adjusted deep learning model is tested using the test set.

3. The EBAC fish text joint extraction method integrating ResNeXt according to claim 1 is characterized in that, After obtaining the fish science text, it also includes: The fish science text is cleaned.

4. The EBAC fish text joint extraction method integrating ResNeXt according to claim 3 is characterized in that, The method for obtaining and cleaning the fish science popularization text is as follows: By using Python crawlers, we obtain fish science information from multiple encyclopedia websites, integrate and clean them, remove noise, and obtain fish science text.

5. The EBAC fish text joint extraction method integrating ResNeXt according to claim 4 is characterized in that, The fish science information includes fish level classification, nicknames, main distribution locations, common habitats, preferred prey, and the origin of first-generation fish species.

6. The EBAC fish text joint extraction method integrating ResNeXt according to claim 2 is characterized in that, The ratio of fish science texts in the training set and the test set is 7:

3.

7. The EBAC fish text joint extraction method integrating ResNeXt according to claim 1 is characterized in that, The fish science text is annotated as follows: According to the characteristics of fish popular science texts, annotation tools were used and the "BIOS" method was used for annotation.

Images