Named entity recognition model used in field of electric power operation and maintenance

Abstract

A named entity recognition model used in the field of electric power operation inspection comprises a coding layer, a graph convolution network layer, a span presentation layer and a decoding layer, the input of the coding layer is used for inputting statements, the output end of the coding layer is connected with the input end of the graph convolution network layer, the output end of the graph convolution network layer is connected with the input end of the span presentation layer, and the output end of the span presentation layer is connected with the input end of the decoding layer. The output end of the span presentation layer is connected with the input end of the decoding layer, and the output end of the decoding layer outputs electronic components; compared with a traditional named entity recognition model, the technical scheme provided by the model introduces a BiGRU to help the model obtain more detailed text information, the model can understand a text structure more clearly by adding a rotation position code, and the entity recognition performance of the model in the field of electric power operation inspection is improved.

Description

Technical Field

[0001] The present invention relates to the field of power system operation and maintenance technology, and in particular to a named entity recognition model and method based on BiGRU and graph convolutional network GCN. Background Art

[0002] Named entity recognition (NER) is a basic task in the field of natural language processing (NLP). Its goal is to extract named entities with practical value from unstructured raw text. These named entities can be names of people, places, and organizations. NER has a wide range of applications in information extraction, text mining, and text understanding, and provides an important foundation for the automated processing of large-scale unstructured power operation and inspection text data.

[0003] Traditional dictionary- and rule-based methods perform well in some specific fields and small-scale data. Yuan Jindou et al. studied dictionary- and rule-based named entity recognition. They accurately extracted entities related to power safety from unstructured texts through three methods: domain entity dictionary, word formation feature character rule matching, and part-of-speech combination feature rule matching. They achieved excellent performance in small-scale test experiments. However, its scalability is poor and it is difficult to adapt to the diverse and complex text information in the power field. Deep learning models have become the mainstream method in the field of named entity recognition with their powerful feature extraction capabilities. In particular, models such as BiLSTM-CRF capture contextual information through bidirectional long short-term memory networks and improve the accuracy of entity recognition by combining conditional random fields. They have been widely used in NER tasks in the power industry. In recent years, methods based on pre-trained models have brought new breakthroughs to NER tasks. Models such as BERT and ERNIE can better understand contextual semantics and provide more generalized word vector representations for downstream tasks by pre-training on large-scale data. This significantly improves the application effect of NER in the field of power operation and inspection. In addition, the introduction of graph neural networks (GCNs) also provides a new perspective for named entity recognition. GCN is able to model the relationships between entities and performs well especially when processing complex unstructured data.

[0004] To solve the problem of identifying valuable structured information in unstructured power text, this paper proposes a named entity recognition method based on BiGRU and graph convolutional network. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problem that the existing power operation and inspection technology has low accuracy in named entity recognition tasks for unstructured power texts and cannot meet the needs of power operation and inspection, and proposes a named entity recognition technology based on graph convolutional network GCN and bidirectional gated recurrent unit BiGRU.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A named entity recognition model for the field of electric power operation and inspection, the model comprises an encoding layer, a graph convolutional network layer, a span representation layer, and a decoding layer, wherein the input of the encoding layer is used to input sentences, the output end of the encoding layer is connected to the input end of the graph convolutional network layer, the output end of the graph convolutional network layer is connected to the input end of the span representation layer, the output end of the span representation layer is connected to the input end of the decoding layer, and the output end of the decoding layer outputs the type of electronic components.

[0008] The internal structure of the encoding layer includes the pre-trained language model BERT and the bidirectional gated recurrent unit BiGRU;

[0009] The input of BERT is used to input sentences, the output of BERT is connected to the input of BiGRU, and the output of BiGRU outputs sentence representation.

[0010] The internal structure of the graph convolutional network layer includes an attention guidance layer, a dense connection layer, and a linear connection combination layer;

[0011] The input of the attention guidance layer is used for sentence representation, the output of the attention guidance layer is connected to the input of the dense connection layer, the output of the dense connection layer is connected to the input of the linear connection layer, and the output of the linear connection layer is used to output the sentence representation that integrates grammatical information.

[0012] The input of the span representation layer is used for sentence representation, and the output of the span representation layer outputs the span representation.

[0013] The internal structure of the decoding layer includes a multi-layer perceptron MLP;

[0014] The input of the MLP is used for span representation, and the output of the MLP outputs the electronic component type.

[0015] When the model is working, the following steps are adopted:

[0016] Step 1: Preprocess the data.

[0017] Step 2: Input the preprocessed data into BERT to obtain sentence representation;

[0018] Step 3: Send the sentence representation into BiGRU for deeper feature extraction;

[0019] Step 4: Send the sentence representation into the graph convolutional network to obtain the grammatical information of the power text;

[0020] Step 5: Enumerate the obtained sentence representations containing grammatical information to obtain span representations;

[0021] Step 6: Send the span representation to MLP for relationship analysis and prediction, determine the entity type, and obtain the type of power component;

[0022] Through the above steps, named entity recognition is performed on power operation and inspection data.

[0023] Step 1 specifically includes the following steps:

[0024] Step 1-1: Preprocess the acquired data set to remove special characters in the text; use the word segmentation tool Stanford CoreNLP to segment the text data and obtain the part of speech corresponding to the word in the segmentation result;

[0025] Step 2 specifically includes the following sub-steps:

[0026] Step 2-1: Input the preprocessed data into the pre-trained language model BERT to convert the text into text encoding;

[0027] Step 3 specifically includes the following sub-steps:

[0028] Step 3-1: Send the acquired text encoding to the BiGRU layer, obtain the forward GRU and reverse GRU to extract features from the text information, and obtain the bidirectional semantic relationship. The process formula is as follows:

[0029]

[0030] Among them, w t represents the word embedding encoding vector input at time t, Refers to the input vector of the forward GRU at time t-1, Refers to the input vector of the reverse GRU at time t-1, z represents the forward direction, and f represents the reverse direction. represents the representation vector of the forward GRU at time t, represents the representation vector of the reverse GRU at time t, H represents the deeper semantic information at a certain moment obtained through BiGRU network coding, and finally the final result H = {h1,h2,,,h N}, h N refers to the representation vector of the Nth text, GRU refers to the GRU network, Represents addition of elements.

[0031] Step 4 specifically includes the following sub-steps:

[0032] Step 4-1: Send the acquired semantic information to the attention guidance layer, use rotation coding to obtain the relative position information of the nodes and model the dependency relationship between the nodes. The specific calculation formula is as follows:

[0033]

[0034] Among them, W t Q and W t K is the weight of the query and key value for the tth head, is the vector representation of the t-th head, is the adjacency matrix updated by the tth head, d head is the vector dimension and softmax is the activation function.

[0035] The adjacency matrix is ​​updated by projecting the sentence representation of the t-th head into the query and key value of the rotational position encoding using the weight. The calculation formula of the rotation encoding is as follows:

[0036]

[0037] in represents the angle of clockwise rotation of iθ1, and represents the angle of counterclockwise rotation of jθ1, and the multiplication of the two becomes the angle of counterclockwise rotation of (jθ1-iθ1). By utilizing the characteristics of coordinate rotation, the position representation only has the rotation difference information, and is ultimately only related to (iθ1-jθ1). q' and k' represent the query and key value of the initial attention mechanism, and q and k are the query and key values ​​finally obtained for the rotation position encoding;

[0038] Step 4-2: Send the updated adjacency matrix to the dense connection layer. The calculation formula is as follows:

[0039]

[0040] Among them, W t and b t are the weights and biases of the t-th head, is the adjacency matrix updated by the t-th head, H t-1 is the vector representation of t-1 heads, is the updated vector representation of the t-th head.

[0041] Step 4-3: Send the obtained vector representation to the linear connection layer, and use the linear combination layer to merge the output of each head, that is, in is the weight, is the vector representation of the Nth head, It is the final output of the graph convolutional network.

[0042] Step 5 specifically includes the following sub-steps:

[0043] Step 5-1: Enumerate the obtained text representation to get si,j =[h i ,h j ,ω], where h i ,h j is the span representation of span i,j, ω is a 20-dimensional embedding layer used to represent the length of the span, and s i,j represents the span representation of span i,j.

[0044] Step 6 specifically includes the following sub-steps:

[0045] Step 6-1: Perform entity fragment recognition on the obtained span representation, and use MLP to distinguish whether the span is an entity fragment and its entity type. The calculation formula is as follows:

[0046] p = softmax(MLP(s i,j ))

[0047] Among them, p represents the possibility of entity categories, such as equipment, department, and none (i.e., not an entity fragment), and s i,j It represents the span from position i to j, MLP represents the multi-layer perceptron module, and softmax represents the activation function.

[0048] Compared with the prior art, the present invention has the following technical effects:

[0049] 1) Compared with the traditional named entity recognition model BiLSTM-CRF, the present invention obtains a more fine-grained vector representation of the text by adding a bidirectional gated recurrent unit BiGRU, while effectively capturing the bidirectional semantic representation of the text, capturing the temporal relationship and the mutual influence between multiple variables, and improving the model's understanding of the text;

[0050] 2) The present invention adds rotational position coding to the graph convolutional network and uses rotational coding to obtain the relative position information of the nodes, so that the model can more effectively capture the relative position relationship between the nodes in the graph structure, help the model better understand the global structure of the nodes in the graph and better model the dependency relationship between the nodes, and improve the performance of the model in identifying entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0052] Figure 1 It is a schematic diagram of the model structure of the present invention;

[0053] Figure 2 Schematic diagram of the internal structure of the graph convolutional network. DETAILED DESCRIPTION

[0054] A named entity recognition model for the field of electric power operation and inspection, the model structure includes four parts: an encoding layer 1, a graph convolutional network layer 2, a span representation layer 3, and a decoding layer 4, wherein the input of the encoding layer 1 is used to input sentences, the output end of the encoding layer 1 is connected to the input end of the graph convolutional network layer 2, the output end of the graph convolutional network layer 2 is connected to the input end of the span representation layer 3, the output end of the span representation layer 3 is connected to the input end of the decoding layer 4, and the output end of the decoding layer 4 outputs the type of electronic components.

[0055] The internal structure of the encoding layer 1 includes the pre-trained language model BERT and the bidirectional gated recurrent unit BiGRU;

[0056] The input of BERT is used to input sentences, the output of BERT is connected to the input of BiGRU, and the output of BiGRU outputs sentence representations;

[0057] The internal structure of the graph convolutional network layer 2 includes an attention guidance layer 201, a dense connection layer 202, and a linear connection combination layer 203;

[0058] The input of the attention guidance layer 201 is used for sentence representation, the output end of the attention guidance layer 201 is connected to the input end of the dense connection layer 202, the output end of the dense connection layer 202 is connected to the input end of the linear connection layer 203, and the linear connection layer 203 obtains the sentence representation of the output end output fusion grammatical information;

[0059] The input of the span representation layer 3 is used for sentence representation, and the output of the span representation layer 3 outputs the span representation.

[0060] The internal structure of the decoding layer 4 includes a multi-layer perceptron MLP, wherein the input of the MLP is used for span representation, and the output of the MLP outputs the type of electronic component.

[0061] The above model works by taking the following steps:

[0062] The following steps are involved:

[0063] Step 1: Preprocess the data.

[0064] Step 2: Input the preprocessed data into BERT to obtain sentence representation;

[0065] Step 3: Send the sentence representation into BiGRU for deeper feature extraction;

[0066] Step 4: Send the sentence representation into the graph convolutional network to obtain the grammatical information of the power text;

[0067] Step 5: Enumerate the obtained sentence representations containing grammatical information to obtain span representations;

[0068] Step 6: Send the span representation to MLP for relationship analysis and prediction, determine the entity type, and obtain the power component;

[0069] Through the above steps, named entity recognition is performed on power operation and inspection data.

[0070] Step 1 specifically includes the following steps:

[0071] Step 1-1: Preprocess the acquired data set to remove special characters in the text; use the word segmentation tool Stanford CoreNLP to segment the text data and obtain the part of speech corresponding to the words in the word segmentation results.

[0072] Step 2 specifically includes the following sub-steps:

[0073] Step 2-1: Input the preprocessed data into the pre-trained language model BERT to convert the text into text encoding.

[0074] Step 3 specifically includes the following sub-steps:

[0075] Step 3-1: Send the acquired text encoding to the BiGRU layer, obtain the forward GRU and reverse GRU to extract features from the text information, and obtain the bidirectional semantic relationship. The process formula is as follows:

[0076]

[0077]

[0078] Among them, w t represents the word embedding encoding vector input at time t, Refers to the input vector of the forward GRU at time t-1, Refers to the input vector of the reverse GRU at time t-1, z represents the forward direction, and f represents the reverse direction. represents the representation vector of the forward GRU at time t, represents the representation vector of the reverse GRU at time t, H represents the deeper semantic information at a certain moment obtained through BiGRU network coding, and finally the final result H = {h1,h2,,,h N}, h N refers to the representation vector of the Nth text, GRU refers to the GRU network, Represents addition of elements.

[0079] Step 4 specifically includes the following sub-steps:

[0080] Step 4-1: Send the acquired semantic information to the attention guidance layer, use rotation coding to obtain the relative position information of the nodes and model the dependency relationship between the nodes. The specific calculation formula is as follows:

[0081]

[0082] Among them, W t Q and W t K is the weight of the query and key value for the tth head, is the vector representation of the t-th head, is the adjacency matrix updated by the tth head, d head is the vector dimension and softmax is the activation function.

[0083] The adjacency matrix is ​​updated by projecting the sentence representation of the t-th head into the query and key value of the rotational position encoding using the weight. The calculation formula of the rotation encoding is as follows:

[0084]

[0085] in represents the angle of clockwise rotation of iθ1, and represents the angle of counterclockwise rotation of jθ1, and the multiplication of the two becomes the angle of counterclockwise rotation of (jθ1-iθ1). By utilizing the characteristics of coordinate rotation, the position representation only has the rotation difference information, and is ultimately only related to (iθ1-jθ1). q' and k' represent the query and key value of the initial attention mechanism, and q and k are the query and key values ​​finally obtained for the rotation position encoding;

[0086] Step 4-2: Send the updated adjacency matrix to the dense connection layer. The calculation formula is as follows:

[0087]

[0088] Among them, W t and b t are the weights and biases of the t-th head, is the adjacency matrix updated by the t-th head, H t-1 is the vector representation of t-1 heads, is the updated vector representation of the t-th head.

[0089] Step 4-3: Send the obtained vector representation to the linear connection layer, and use the linear combination layer to merge the output of each head, that is, in is the weight, is the final output of the graph convolutional network. It is the final output of the graph convolutional network.

[0090] Step 5 specifically includes the following sub-steps:

[0091] Step 5-1: Enumerate the obtained text representation to get s i,j =[h i ,h j ,ω], where h i ,h j is the span representation of span i,j, ω is a 20-dimensional embedding layer used to represent the length of the span, and s i,j represents the span representation of span i,j.

[0092] Step 6 specifically includes the following sub-steps:

[0093] Step 6-1: Perform entity fragment recognition on the obtained span representation, and use MLP to distinguish whether the span is an entity fragment and its entity type. The calculation formula is as follows:

[0094] p = softmax(MLP(s i,j ))

[0095] Among them, p represents the possibility of entity categories, such as equipment, department, and none (i.e., not an entity fragment), and s i,j It represents the span from position i to j, MLP represents the multi-layer perceptron module, and softmax represents the activation function.

[0096] Example:

[0097] On the power grid operation and inspection record dataset, the performance of the proposed model method is compared with the following baseline models;

[0098]

[0099] The above table lists the results of the model of the present invention and other baseline models on all data sets. It can be observed that the model of this paper achieved an F1 value of 88.646% in the named entity recognition task for text in the field of power operation and inspection, and the accuracy and F1 value of the model are better than those of the other three baseline models. Compared with the traditional Bert+CRF method, the F1 value of this model has been improved by 1.223%. Compared with the W2NER model with better performance, the model proposed in this paper has an F1 value improvement of 0.284%, and its accuracy value has also been greatly improved.

Claims

1. A named entity recognition model for the field of power operation and inspection, characterized in that: The model comprises an encoding layer (1), a graph convolutional network layer (2), a span representation layer (3), and a decoding layer (4), wherein the input of the encoding layer (1) is used to input a sentence, the output end of the encoding layer (1) is connected to the input end of the graph convolutional network layer (2), the output end of the graph convolutional network layer (2) is connected to the input end of the span representation layer (3), the output end of the span representation layer (3) is connected to the input end of the decoding layer (4), and the output end of the decoding layer (4) outputs the type of electronic components.

2. The model according to claim 1, characterized in that The internal structure of the encoding layer (1) includes a pre-trained language model BERT (101) and a bidirectional gated recurrent unit BiGRU (102); The input of BERT (101) is used to input a sentence, the output of BERT (101) is connected to the input of BiGRU (102), and the output of BiGRU (102) outputs a sentence representation.

3. The model according to claim 1, characterized in that The internal structure of the graph convolutional network layer (2) includes an attention guidance layer (201), a dense connection layer (202), and a linear connection combination layer (203); The input of the attention guidance layer (201) is used for sentence representation, the output of the attention guidance layer (201) is connected to the input of the dense connection layer (202), the output of the dense connection layer (202) is connected to the input of the linear connection layer (203), and the output of the linear connection layer (203) is used to output the sentence representation that integrates grammatical information.

4. The model according to claim 1, characterized in that The input of the span representation layer (3) is used for sentence representation, and the output of the span representation layer (3) outputs the span representation.

5. The model according to claim 1, characterized in that The internal structure of the decoding layer (4) includes a multi-layer perceptron MLP; The input of the MLP is used for span representation, and the output of the MLP outputs the electronic component type.

6. The model according to any one of claims 1 to 5, characterized in that When the model is working, the following steps are adopted: Step 1: Preprocess the data. Step 2: Input the preprocessed data into BERT to obtain sentence representation; Step 3: Send the sentence representation into BiGRU for deeper feature extraction; Step 4: Send the sentence representation into the graph convolutional network to obtain the grammatical information of the power text; Step 5: Enumerate the obtained sentence representations containing grammatical information to obtain span representations; Step 6: Send the span representation to MLP for relationship analysis and prediction, determine the entity type, and obtain the type of power component; Through the above steps, named entity recognition is performed on power operation and inspection data.

7. The model according to claim 6, characterized in that Step 1 specifically includes the following steps: Step 1-1: Preprocess the acquired data set to remove special characters in the text; use the word segmentation tool Stanford CoreNLP to segment the text data and obtain the part of speech corresponding to the word in the segmentation result; Step 2 specifically includes the following sub-steps: Step 2-1: Input the preprocessed data into the pre-trained language model BERT to convert the text into text encoding; Step 3 specifically includes the following sub-steps: Step 3-1: Send the acquired text encoding to the BiGRU layer, obtain the forward GRU and reverse GRU to extract features from the text information, and obtain the bidirectional semantic relationship. The process formula is as follows: Among them, w t represents the word embedding encoding vector input at time t, Refers to the input vector of the forward GRU at time t-1, Refers to the input vector of the reverse GRU at time t-1, z represents the forward direction, and f represents the reverse direction. represents the representation vector of the forward GRU at time t, represents the representation vector of the reverse GRU at time t, H represents the deeper semantic information at a certain moment obtained through BiGRU network coding, and finally the final result H = {h1,h2,,,h N }, h N refers to the representation vector of the Nth text, GRU refers to the GRU network, Represents addition of elements.

8. The model according to claim 6, characterized in that Step 4 specifically includes the following sub-steps: Step 4-1: Send the acquired semantic information to the attention guidance layer, use rotation encoding to obtain the relative position information of the nodes and model the dependency relationship between the nodes. The specific calculation formula is as follows: Among them, W t Q and W t K is the weight of the query and key value for the tth head, is the vector representation of the t-th head, is the adjacency matrix updated by the tth head, d head is the vector dimension, softmax is the activation function; The adjacency matrix is ​​updated by projecting the sentence representation of the t-th head into the query and key value of the rotational position encoding using the weight. The calculation formula of the rotation encoding is as follows: in represents the angle of clockwise rotation of iθ1, and represents the angle of counterclockwise rotation of jθ1, and the product of the two becomes the angle of counterclockwise rotation of (jθ1-iθ1); by utilizing the characteristics of coordinate rotation, the position representation only has the rotation difference information, and is ultimately only related to (iθ1-jθ1). q' and k' represent the query and key value of the initial attention mechanism, and q and k are the query and key values ​​finally obtained for rotation position encoding; Step 4-2: Send the updated adjacency matrix to the dense connection layer. The calculation formula is as follows: Among them, W t and b t are the weights and biases of the t-th head, is the adjacency matrix updated by the t-th head, H t-1 is the vector representation of t-1 heads, is the updated vector representation of the t-th head; Step 4-3: Send the obtained vector representation to the linear connection layer, and use the linear combination layer to merge the output of each head, that is, in is the weight, is the vector representation of the Nth head, It is the final output of the graph convolutional network; Step 5 specifically includes the following sub-steps: Step 5-1: Enumerate the obtained text representation to get s i,j =[h i ,h j ,ω], where h i ,h j is the span representation of span i,j, ω is a 20-dimensional embedding layer used to represent the length of the span, and s i,j represents the span representation of span i,j.

9. The model according to claim 6, characterized in that Step 6 specifically includes the following sub-steps: Step 6-1: Perform entity fragment recognition on the obtained span representation, and use MLP to distinguish whether the span is an entity fragment and its entity type. The calculation formula is as follows: p=softmax(MLP(s i,j )) Among them, p represents the possibility of entity categories, such as equipment, department, and none (i.e., not an entity fragment), and s i,j It represents the span from position i to j, MLP represents the multi-layer perceptron module, and softmax represents the activation function.

Images