A nested entity recognition method for water conservancy text

Abstract

The present invention discloses a nested entity recognition method for water conservancy texts. First, a basic dictionary is formed based on existing water conservancy industry standards and third-party word libraries, and a new word discovery algorithm is used to mine new words to expand the basic dictionary; secondly, in the word embedding process, a domain dictionary is used to integrate the information of the vocabulary into the character representation to enhance the semantic representation; then, in the encoding process, a bidirectional long short-term memory network is used for feature extraction; finally, in the decoding process, an outside-to-inside decoding method is proposed. When the model predicts an entity, it will automatically search for internal entities until there is no entity inside. The present invention combines water conservancy field knowledge and achieves good results in identifying nested entities in water conservancy texts.

Description

Technical Field

[0001] The invention belongs to nested entity recognition, and specifically relates to a nested entity recognition method for water conservancy texts. Background Art

[0002] Entity relationship extraction is one of the key tasks in the field of natural language processing. It can automatically extract structured knowledge from unstructured text and store the knowledge in a knowledge base in this format so that the computer can understand it, thus playing a vital supporting role in downstream applications such as knowledge graph construction, intelligent question-answering systems, and information retrieval. Entity relationship extraction includes two key tasks, namely named entity recognition (NER) and relation extraction (RE). The named entity recognition task initially defines all entity types, then extracts all entities from the text, and assigns type labels corresponding to the entities. Common entity type labels include human names (LOC), place names (LOC), and organizations (ORG). The relation extraction task is based on the named entity recognition task and assigns pre-defined relationship types to entity pairs according to the semantics of a given sentence.

[0003] With the in-depth integration of advanced technologies from all walks of life such as big data, cloud computing, and the Internet of Things with application scenarios in the water conservancy field, the development of water conservancy digitalization and informatization has been promoted. Under such continuous development, the requirements for underlying data support are becoming higher and higher. From the initial single-source data that can meet data requirements, to the current multi-source heterogeneous data drive, the data style and data scale are constantly expanding, from the initial data support to knowledge support. Therefore, how to automatically extract the knowledge in water conservancy text data is particularly important.

[0004] Moreover, frontier applications in water conservancy cannot do without the support of knowledge graphs, and entities are the basis for the construction of water conservancy knowledge graphs. At present, self-supervised models are mostly used for named entity recognition, and self-supervised models require a large amount of labeled data. However, the data in the field of water conservancy is relatively small, and it is difficult to achieve good results by directly migrating existing models. Entities in the field of water conservancy have certain domain specificity, and another entity may be nested inside an entity, or even a multi-layer nested structure. Therefore, it is necessary to design a nested entity recognition method for water conservancy text. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a nested entity recognition method for water conservancy texts, which achieves good results in identifying nested entities in water conservancy texts.

[0006] Technical solution: The present invention provides a nested entity recognition method for water conservancy text, comprising the following steps:

[0007] (1) A basic dictionary is formed based on existing water conservancy industry standards and third-party word libraries, and a new word discovery algorithm is used to mine new words to expand the basic dictionary;

[0008] (2) In the word embedding process, domain dictionaries are used to incorporate vocabulary information into character representation and enhance semantic representation;

[0009] (3) During the encoding process, a bidirectional long short-term memory network is used for feature extraction;

[0010] (4) During the decoding process, an outside-to-inside decoding method is proposed. When an entity is predicted, it will automatically search for internal entities until there are no entities inside.

[0011] Furthermore, the step (1) comprises the following steps:

[0012] (11) Use information entropy and coagulation to mine new words, locate the position where the words in the dictionary appear in the original corpus, calculate information entropy to judge the richness of the adjacent words of the word, and if the information entropy is less than 20, concatenate the word with the adjacent words to form a new word;

[0013] (12) When a new word is formed, the word's solidification degree is calculated to determine whether to add the word to the dictionary. When the solidification degree is greater than 1.25, the word is added to the dictionary and duplicates are removed from the dictionary.

[0014] Furthermore, the step (2) comprises the following steps:

[0015] (21) Incorporate lexical information into character representation, retain all possible segmentation results obtained from the dictionary for each character, and construct a {B, M, E, S} word set for each character in the sentence;

[0016] (22) Merge the words in each word set, compress them into a vector of fixed dimension, and use existing tools for word embedding.

[0017] Furthermore, the step (4) comprises the following steps:

[0018] (41) Select CRF for basic decoding, prepare a CRF for each entity type, and find each possible entity in an iterative manner from outside to inside;

[0019] (42) Use multiple CRFs to calculate the CRF score of the entire sentence, the globally optimal label sequence, obtain all external entities, and add the entities to the entity set; if the sequence output by the encoding layer is H = h1,h2,h3,…,h n,h i represents the feature vector of the i-th character, Represents the entity label sequence when the feature vector sequence is H and the entity type is k. Calculate The CRF score calculation formula is as follows:

[0020]

[0021] in, Represents the label To Tags If the transfer score To Tags There is no correlation. on the contrary, and Represents the weight matrix and bias vector;

[0022] (43) A new span is formed with the starting position of the entity, and the CRF is used again to calculate the score to obtain the label sequence of the internal entity, and the internal entity is added to the entity set;

[0023] (44) Repeat step (43) until no new entities appear.

[0024] Furthermore, the step (21) comprises the following steps:

[0025] (211) calculating a word set {B}, where {B} represents the set of all words starting with the character, and all words in the set belong to the dictionary;

[0026] (212) calculating a word set {M}, where {M} represents the set of all words with the character as the middle character, and all words in the set belong to the dictionary;

[0027] (213) calculating a word set {E}, where {E} represents the set of all words ending with the character, and all words in the set belong to the dictionary;

[0028] (214) Calculate the word set {S}, where {S} represents the character itself;

[0029] (215) When a word set does not contain any words that meet the requirements, it is marked as "Null";

[0030] (216) Concatenate the four word sets corresponding to each character.

[0031] Furthermore, the step (22) comprises the following steps:

[0032] (221) The frequency of occurrence of a word in the corpus is used as a weight to form a vector of fixed dimension;

[0033] (222) Use pre-trained ELMO for word embedding.

[0034] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: Aiming at the particularity of nested entity recognition tasks in the field of water conservancy, the present invention uses a new word discovery method based on information entropy and solidification to expand the dictionary in the field of water conservancy; uses the dictionary to construct the {B, M, E, S} word set, integrates the vocabulary information into the representation of characters, enhances the semantic representation, and uses the pre-trained ELMO for word embedding; in the encoding process, a bidirectional long short-term memory network is used for feature extraction, and in the decoding process, in order to be able to identify nested entities, a conditional random field is prepared for each entity type, and when decoding a sentence, multiple CRFs are first used to calculate the globally optimal label sequence of the sentence, and a new span is formed with external entities, and then the possible internal entities are automatically and iteratively explored, effectively solving the problem of nested entity recognition in the field of water conservancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0037] The present invention proposes a nested entity recognition method for water conservancy texts. In view of the particularity of the nested entity recognition task in the field of water conservancy, by using the constructed domain dictionary, a {B, M, E, S} word set is constructed for the input text, the vocabulary information is integrated into the representation of the word, the semantic representation is enhanced, and the pre-trained ELMO is used for word embedding. In the encoding process, a bidirectional long short-term memory network is used for feature extraction. In the decoding process, in order to be able to identify the nested entities, a conditional random field is prepared for each entity type. When decoding the sentence, multiple CRFs are first used to calculate the globally optimal label sequence of the sentence, and a new span is formed with external entities, and then the possible internal entities are automatically explored iteratively. Figure 1 As shown, the specific steps include:

[0038] S1: Form a basic dictionary from existing water conservancy industry standards, third-party thesaurus and other sources, and use a new word discovery algorithm to mine new words to expand the dictionary.

[0039] Relevant terms were collected from the water conservancy industry standards "Common Terms in Water Conservancy Informatization (SLZ 376-22007)", "Technical Terms in Water Conservancy and Hydropower Engineering (SL 26-22012)", "Water Conservancy Project Name Code (SZDB / Z 2-2005)", and "Water Conservancy Official Document Thesaurus (SL / Z347-2006)", and expanded using third-party thesaurus of Baidu and Sogou input methods to form a basic dictionary.

[0040] Use information entropy and solidification to mine new words, locate the words in the dictionary where they appear in the original expectation, calculate information entropy to judge the richness of the adjacent words of the word, and if the information entropy is less than 20, then concatenate the word with the adjacent words to form a new word.

[0041] When a new word is formed, the word's coagulation degree is calculated to determine whether to add the word to the dictionary. When the coagulation degree is greater than 1.25, the word is added to the dictionary. The dictionary is deduplicated.

[0042] S2: In the word embedding process, domain dictionaries are used to incorporate vocabulary information into the representation of characters and enhance semantic representation.

[0043] Incorporate vocabulary information into character representation, retain all possible segmentation results obtained from the dictionary for each character, and construct a {B, M, E, S} word set for each character in the sentence.

[0044] Calculate the word set {B}, {B} represents the set of all words that use this character as the starting character, and all words in the set belong to the dictionary. Calculate the word set {M}, {M} represents the set of all words that use this character as the middle character, and all words in the set belong to the dictionary. Calculate the word set {E}, {E} represents the set of all words that use this character as the ending character, and all words in the set belong to the dictionary. Calculate the word set {S}, {S} represents the character itself. When a word set does not have any words that meet the requirements, it is marked as "Null". Concatenate the four word sets corresponding to each character.

[0045] When there is a sentence "Shenzhen River is located at the mouth of the Pearl River.", and the dictionary contains the words "Pearl River", "Pearl River Mouth", and "Jiangkou". At this time, the {B,M,E,S} word set corresponding to "Pearl" is {{Pearl River, Pearl River Mouth}, {Null}, {Null}, {Pearl}}, and the {B,M,E,S} word set corresponding to "Jiang" is {{Jiangkou}, {Pearl River Mouth}, {Pearl River}, {Jiang}}.

[0046] Merge the words in each vocabulary, compress them into a vector of fixed dimension, and use existing tools for word embedding.

[0047] Because the length of words in each word set is different and the number of elements is different, in order to focus on more important words in the process of compressing the vector, the frequency of words in the corpus is used as the weight to form a vector of fixed dimension. The pre-trained ELMO is used for word embedding.

[0048] Calculate the frequency of the words "Pearl River", "Pearl River Mouth", and "Jiangkou". The frequency of the word is used as the weight to calculate the vector representation. And use the Embeddings from Language Models (ELMO) to convert it into a word vector, and concatenate all the character word vectors to form the word vector matrix of the entire sentence.

[0049] S3: During the encoding process, a bidirectional long short-term memory network is used for feature extraction.

[0050] The word embedding layer outputs the word vector matrix, and uses the bidirectional long short-term memory network to extract features. The forward long short-term memory network calculates the encoding information from left to right and obtains the forward hidden features. The backward long short-term memory network calculates the encoding information from right to left and obtains the backward hidden features. Finally, the forward and backward hidden features are concatenated to form the output of the encoding layer.

[0051] S4: During the decoding process, a decoding method from outside to inside is designed. When the model predicts an entity, it will automatically search for internal entities until there are no entities inside. The specific steps include:

[0052] 1) The decoding layer assigns a specified label to each position of each feature output by the encoding layer, and there are certain constraints before and after the labels, that is, the label of the current character will be associated with the labels of the adjacent characters. Therefore, Conditional Random Fields (CRF) is used for decoding.

[0053] 2) Considering that each position of a nested entity will correspond to multiple labels, a CRF is prepared for each entity type. Eight types of CRFs are prepared to find each possible entity in an iterative manner from outside to inside.

[0054] 3) Use multiple CRFs to calculate the CRF score of the entire sentence, the globally optimal label sequence, obtain all external entities, and add the entities to the entity set; if the sequence output by the encoding layer is H = h1,h2,h3,…,h n ,h i represents the feature vector of the i-th character, Represents the entity label sequence when the feature vector sequence is H and the entity type is k. Calculate The CRF score calculation formula is as follows:

[0055]

[0056] in, Represents the label To Tags If the transfer score To Tags There is no correlation. on the contrary, and Represents the weight matrix and bias vector. When there is a sentence "Shenzhen River is located at the mouth of the Pearl River.", after the first iteration, the external entities <Shenzhen River, place name> and <Pearl River Estuary, place name> are extracted. At this time, the entity set is {Shenzhen River, Pearl River Estuary}.

[0057] 4) Use the entity start position to form a new span, re-use CRF to calculate the score, and then get the label sequence of the internal entity. Add the internal entity to the entity set. Use the entity <Shenzhen River, place name> as the new span, re-calculate the CRF score, and get the entity <Shenzhen, place name>.

[0058] 5) Repeat step 4) until no new entity appears, and the decoding process ends.

[0059] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The industry description is only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A nested entity recognition method for water conservancy text, characterized in that: The following steps are involved: (1) A basic dictionary is formed based on existing water conservancy industry standards and third-party word libraries, and a new word discovery algorithm is used to mine new words to expand the basic dictionary; (2) In the word embedding process, domain dictionaries are used to incorporate vocabulary information into character representation to enhance semantic representation; (3) During the encoding process, a bidirectional long short-term memory network is used for feature extraction; (4) During the decoding process, an outside-to-inside decoding method is proposed. When an entity is predicted, it will automatically search for internal entities until there are no entities inside. The step (1) comprises the following steps: (11) Use information entropy and coagulation to mine new words, locate the position where the words in the dictionary appear in the original corpus, calculate information entropy to judge the richness of the adjacent words of the word, and if the information entropy is less than 20, concatenate the word with the adjacent words to form a new word; (12) When a new word is formed, the word's solidification degree is calculated to determine whether to add the word to the dictionary. When the solidification degree is greater than 1.25, the word is added to the dictionary and the dictionary is deduplicated; The step (4) comprises the following steps: (41) Select CRF for basic decoding, prepare a CRF for each entity type, and find each possible entity in an iterative manner from outside to inside; (42) Use multiple CRFs to calculate the CRF score of the entire sentence, the globally optimal label sequence, obtain all external entities, and add the entities to the entity set; if the sequence output by the encoding layer is H = h1,h2,h3,…,h n ,h i represents the feature vector of the i-th character, Represents the entity label sequence when the feature vector sequence is H and the entity type is k. Calculate The CRF score calculation formula is as follows: in, Represents the label To Tags If the transfer score To Tags There is no correlation. on the contrary, and Represents the weight matrix and bias vector; (43) A new span is formed with the entity starting position, and the CRF is used again to calculate the score to obtain the label sequence of the internal entity, and the internal entity is added to the entity set; (44) Repeat step (43) until no new entities appear.

2. A nested entity recognition method for water conservancy text according to claim 1, characterized in that: The step (2) comprises the following steps: (21) Incorporate lexical information into character representation, retain all possible segmentation results obtained from the dictionary for each character, and construct a {B, M, E, S} word set for each character in the sentence; (22) Merge the words in each word set, compress them into a vector of fixed dimension, and use existing tools for word embedding.

3. A nested entity recognition method for water conservancy text according to claim 2, characterized in that: The step (21) comprises the following steps: (211) calculating a word set {B}, where {B} represents the set of all words starting with the character, and all words in the set belong to the dictionary; (212) calculating a word set {M}, where {M} represents the set of all words with the character as the middle character, and all words in the set belong to the dictionary; (213) calculating a word set {E}, where {E} represents the set of all words ending with the character, and all words in the set belong to the dictionary; (214) Calculate the word set {S}, where {S} represents the character itself; (215) When a word set does not contain any words that meet the requirements, it is marked as "Null"; (216) Concatenate the four word sets corresponding to each character.

4. A nested entity recognition method for water conservancy text according to claim 2, characterized in that: The step (22) comprises the following steps: (221) The frequency of occurrence of a word in the corpus is used as a weight to form a vector of fixed dimension; (222) Use pre-trained ELMO for word embedding.

Images