Electric power information message automatic extraction method based on deep learning technology

Abstract

The invention provides an electric power information message automatic extraction method based on a deep learning technology, relates to the field of natural language processing, and aims to solve the technical problems of high calculation cost and low universality and accuracy of a large amount of unstructured fault text data in an electric power system, and provides the following scheme: S1, inputting an electric power fault text; s2, preprocessing the power fault text to obtain a basic unit in a discrete form; s3, based on a text classification module, performing text classification on the basic units, and outputting a classification result; s4, based on a named entity recognition module, performing named entity recognition on the basic unit, and outputting an entity recognition result; the method has a good application prospect in the field of automatic extraction of electric power information messages.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for automatically extracting electric power information messages based on deep learning technology. Background Art

[0002] my country's power system is currently the largest power grid in the world. In the long-term production, operation and informatization process, a large amount of power text data containing important information has been accumulated, such as Figure 4 The fault records generated by the power units during the maintenance of various electrical equipment in the power system are shown in the figure. These fault records store important information such as fault problem description, fault cause analysis, fault classification, fault equipment and maintenance records, and are an important support for realizing the informatization and intelligence of maintenance work. However, these fault texts are not only huge in number but also have unstructured characteristics, which brings great difficulties to manual analysis of fault texts. Therefore, the research on the analysis and extraction technology of power system text data is of great significance. In recent years, data mining and deep learning technologies are continuously penetrating various industrial fields, and the power system has also entered the artificial intelligence stage with the support of these technologies. The automatic extraction technology for power information messages belongs to the category of NLP tasks, specifically text classification and named entity recognition tasks. Chinese text classification and named entity recognition have always been an important and difficult technology, especially when applied in various professional fields, it needs to be closely combined with professional knowledge in various fields.

[0003] Text classification is a classic problem in NLP. The purpose of the classification task is to assign labels to units such as sentences, paragraphs, and documents. Classifying fault text information in the power system according to the fault type can assist power maintenance personnel in quickly locating the fault type and quickly handling the fault according to classic fault cases. Classic machine learning-based text classification methods include naive Bayes, support vector machine, hidden Markov model (HMM), etc. These models all follow the following two steps: the first step is to extract features from the text; the second step is to input the extracted features into the classifier for prediction. However, the features extracted by the traditional model are pre-defined and difficult to generalize to new tasks. Therefore, methods based on deep learning gradually replace traditional machine learning methods.

[0004] Convolutional neural networks (CNN) and recurrent neural networks (RNN) have been widely used in the field of NLP. For example, TextCNN, an unbiased model of CNN, has achieved remarkable results in text classification tasks such as sentiment analysis and topic classification; DPCNN, a deep pyramid convolutional neural network, improves computational accuracy by increasing network depth. Among the many variants of RNN, long short-term memory (LSTM) is the most popular architecture, such as the Tree-LSTM model, which generalizes LSTM to a tree-structured network type to learn rich semantic representations. However, the common defect of RNN and CNN is that the computational cost increases with the length of the sentence.

[0005] Named Entity Recognition (NER) is a core NLP technology used to identify entities such as people, locations, organizations, and dates in text. Automatic extraction of named entities for power fault text information can quickly locate important information such as fault location, fault time, and faulty equipment, facilitating maintenance work. NER methods mainly include rule-based, statistical machine learning, and deep learning methods. First, rule-based methods involve ontology dictionaries and semantic frameworks to extract defective components and attributes for further research. However, due to the poor generalization of rule-based methods, it is difficult to adapt to complex language texts. Although machine learning solves this problem, machine learning-based methods are highly dependent on the quality of the training data set, and the versatility and accuracy of the algorithm need to be improved. Summary of the invention

[0006] In order to solve the technical problems of high computation cost, low versatility and low accuracy of a large amount of unstructured fault text data in the power system, the present invention provides the following solutions:

[0007] A method for automatically extracting electric power information messages based on deep learning technology, the method comprising the following steps:

[0008] S1. Input power failure text;

[0009] S2. Preprocessing the power fault text to obtain basic units in discrete form;

[0010] S3. Based on a text classification module, perform text classification on the basic unit and output a classification result, wherein the text classification module includes a BERT model and a SENet module;

[0011] S4, based on a named entity recognition module, performing named entity recognition on the basic unit and outputting an entity recognition result, wherein the named entity recognition module includes a BERT model, a Bi-LSTM model and a CRF model;

[0012] S5. Integrate the classification result and the entity recognition result to obtain a final power information message extraction result.

[0013] Furthermore, the core network of the BERT model consists of multiple Transformer encoder stacks.

[0014] Furthermore, the SENet is an attention module with compression and excitation operations.

[0015] Further, the S3 comprises the following steps:

[0016] S31, encoding the basic unit using the BERT model to obtain text feature representation;

[0017] S32, inputting the text feature representation into the SENet module to obtain an enhanced feature representation;

[0018] S33. Classify the enhanced feature representation through a classification layer to obtain the classification result.

[0019] Furthermore, the Bi-LSTM model is a bidirectional long short-term memory network model.

[0020] Furthermore, the CRF model is a probabilistic graphical model.

[0021] Further, the S4 comprises the following steps:

[0022] S41, encoding the basic unit using the BERT model to obtain text feature representation;

[0023] S42, inputting the text feature representation into the Bi-LSTM model, performing forward and reverse feature learning, and obtaining a bidirectional feature representation;

[0024] S43: Use the CRF model to perform sequence labeling on the bidirectional feature representation to obtain the entity recognition result.

[0025] The beneficial effects of the present invention are:

[0026] 1. Since the BERT model uses a bidirectional Transformer encoder, it allows parallel operations, which greatly reduces the computational cost.

[0027] 2. The text classification module based on the BERT model and SENet module can accurately classify power fault texts, helping users quickly understand the content and category of fault texts, thereby guiding subsequent processing and decision-making.

[0028] 3. The named entity recognition module using the BERT model, Bi-LSTM model and CRF model can identify key entity information in the power failure text, such as equipment, location, time, etc., providing important support for subsequent analysis and processing.

[0029] 4. By integrating the classification results and entity recognition results, the extraction results of the power information message are finally obtained, enabling users to quickly obtain key information and conduct further analysis and respond to fault events or problems, thereby improving the efficiency of information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a method for automatic extraction of power information messages based on deep learning technology;

[0031] Figure 2 Schematic diagram of text classification module;

[0032] Figure 3 It is a schematic diagram of the named entity recognition module;

[0033] Figure 4 Generate process for power fault text;

[0034] Figure 5 This is a schematic diagram of the Transformer model structure;

[0035] Figure 6 Schematic diagram of the BERT model;

[0036] Figure 7 Schematic diagram of BERT input vector;

[0037] Figure 8 This is a schematic diagram of the SENet module of the present invention;

[0038] Fig. 9 This is a schematic diagram of the Bi-LSTM model of the present invention;

[0039] Fig.10 This is a schematic diagram of the CRF model of the present invention;

[0040] Fig.11 This is the training and verification accuracy change chart of the text classification module (BERT-SENet);

[0041] Fig.12 This is a graph showing the changes in various evaluation indicators during the training process of the named entity recognition module (BERT-Bi-LSTM-CRF). DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] Embodiment 1, a method for automatically extracting electric power information messages based on deep learning technology, the method comprising the following steps:

[0044] S1. Input power failure text;

[0045] S2. Preprocessing the power fault text to obtain basic units in discrete form;

[0046] S3. Based on a text classification module, perform text classification on the basic unit and output a classification result, wherein the text classification module includes a BERT model and a SENet module;

[0047] S4, based on a named entity recognition module, performing named entity recognition on the basic unit and outputting an entity recognition result, wherein the named entity recognition module includes a BERT model, a Bi-LSTM model and a CRF model;

[0048] S5. Integrate the classification result and the entity recognition result to obtain a final power information message extraction result.

[0049] Specifically, the method for automatically extracting electric power information messages based on deep learning technology described in the present invention has the following overall process: Figure 1 shown.

[0050] Before classification and named entity extraction, the input power fault text sequence is preprocessed by word segmentation, sentence segmentation, sequence padding or truncation, and mask generation to make the text sequence meet the requirements of BERT input sequence. In order to make all input sequences have the same length, shorter sequences need to be padded. The padded part does not have features, so this part will be marked by the mask as not needing attention; if the sequence length exceeds the maximum length allowed by the BERT model, the sequence needs to be truncated to adapt to the model.

[0051] After the input text data is processed to meet the classification and named entity recognition tasks respectively, the text classification and named entity recognition tasks are performed respectively, and the fault classification and named entity recognition structure are output. Finally, the results of the two subtasks are integrated to obtain the final result of the hybrid power information message extraction model.

[0052] The specific process of the text classification module is as follows Figure 2 As shown in Figure 3 as shown

[0053] The training data of the BERT-Bi-LSTM-CRF network used by the named entity recognition module often adopts the BIOES annotation method, that is, each word segmentation is labeled with one of [B, I, O, E, S]. B indicates that the word segmentation is the first character of a named entity, I indicates that it is the middle character of the entity, E indicates that it is the last character of the entity, O indicates that the word segmentation is not an entity and is often used to mark irrelevant characters, and S indicates that the word segmentation itself is an entity. Taking "Qingshan Electric Power Plant" as an example, the annotation sequence "BIIIE" should be obtained according to the BIOES annotation method for each character segmentation.

[0054] Taking "Qingshan Electric Power Plant" as an example as the input text sequence of the network, then this text sequence is segmented into five characters: "qing", "shan", "dian", "re", "chang". According to the corpus table, the foregoing characters are mapped into vector form and input into the BERT layer; the Bi-LSTM layer receives the output vector of BERT, splices it, and then sends it into the forward and reverse LSTMs for encoding, using Bi-LSTM to capture context features and output the label probability corresponding to a single character; since the character labels output by the softmax function used in Bi-LSTM are independent of each other and there is a problem of unreasonable sequence, the CRF layer is applied to solve this problem, which can correct the output result of the Bi-LSTM layer and finally output the reasonable sequence with the highest probability.

[0055] Furthermore, the core network of the BERT model is composed of multiple stacked Transformer encoders.

[0056] Specifically, BERT stands for Bidirectional Encoder Representation from Transformers, which is implemented by Devlin et al. based on the improvement of the Transformer encoder. The core of the Transformer model is the attention mechanism, which is used to select important features of the input sequence and suppress irrelevant features, enabling the Transformer model to more effectively capture the dependencies in the input sequence. The Transformer model is an encoder-decoder architecture, as Figure 5 as shown

[0057] The text used as the input of Transformer needs to be pre-processed by word segmentation, that is, the text sequence is divided into discrete forms such as phrases or words as the basic unit of a model input (i.e., token). The token sequence after word segmentation is mapped to a word embedding vector (i.e., Input Embedding, IE) through an embedding matrix. The embedding matrix can be pre-trained using algorithms such as Word2Vec and GloVe. Transformer uses sine and cosine position encoding to obtain the position embedding vector (i.e., Positional Embedding, PE) of the word in the sequence. Specifically, for a given sequence, let t represent the position, then is the PE corresponding to position t, d is the dimension of the vector, k is the position of the word in the sequence, It can be calculated by formula (1-2).

[0058]

[0059] Among them, ω k Scaling factor to control the range of input values ​​and ensure uniqueness of position codes.

[0060] The frequencies of position embedding vectors of different dimensions are different. In order to ensure that IE and PE can be added, the dimension of PE is generally made equal to the dimension of IE. By adding IE and PE, the input vector of the Transformer model can be obtained.

[0061] The Transformer encoder consists of multiple encoding layers connected in series, which in turn consist of a multi-head attention layer (Multi-Head Attention), a feedforward neural network layer (FeedForward), and a residual normalization layer (Add&Norm). The multi-head attention layer is based on the self-attention mechanism, which is different from the traditional attention mechanism in that it shifts the focus from the sequence alignment of the source sequence and the target sequence to the relationship between tokens in the source sequence. The weights transmitted in the self-attention mechanism change dynamically with the change of words.

[0062] The self-attention mechanism consists of the following two steps:

[0063] In the first step, the two-dimensional vector matrix composed of text or the output matrix of the previous layer is input and recorded as X = [x1, x2, ..., x N ], through linear transformation, we get the query matrix, key matrix and value matrix, which are denoted as Q, K and V respectively. The calculation formula is as follows

[0064] Q=W Q X

[0065] K=W k X

[0066] V=W V X (3)

[0067] Among them, W Q , W K and W V is a linear transformation matrix, which is continuously updated during back propagation.

[0068] In the second step, after obtaining the matrices Q, K, and V, the output of the self-attention mechanism can be calculated, as shown in formula (4):

[0069]

[0070] Among them, d k is the number of columns of Q and K.

[0071] The multi-head attention layer uses multiple self-attention heads to process the input sequence based on the attention mechanism. The multi-head model can obtain the ability to focus on different channels at different abstraction levels, thereby capturing the relationship between different tokens in the sequence. The output of the multi-head attention layer is shown in formula (5)

[0072] MultiHeadAttention(Q,K,V)=Concat[h1,h2,...,h i ,...,h n ]W o (5)

[0073] In the formula, Concat means concatenation, h i represents the i-th head, n is the number of heads, W o It is the linear transformation parameter matrix that integrates the outputs of different attention heads.

[0074] The residual normalization layer is composed of residual connection and layer normalization. The calculation of this layer is shown in the following formula

[0075] Add&Norm(X)=LayerNorm(X+MultiHeadAttention(X)) (6)

[0076] Add&Norm(X)=LayerNorm(X+FeedForward(X)) (7)

[0077] The residual connection between the input matrix X and the multi-head attention output MultiHeadAttention(X) can improve the model's ability to capture text features at different levels. LayerNorm is a layer normalization operation, as shown in formula (8):

[0078]

[0079] Where x is the input variable, E[x] is the mean of the normal distribution, Var[x] is the variance of the normal distribution, ε is a very small number to prevent the denominator from being zero, and γ and β are trainable parameters. Layer normalization ensures that the distribution of data remains consistent by normalizing all features in each vector. This processing method can maintain the time series relationship between different vectors, thereby preventing data from being concentrated in the saturation area due to the action of the activation function, and effectively alleviating the problem of gradient disappearance.

[0080] The feedforward neural network layer is a two-layer fully connected network. The first layer uses Relu as the activation function, and the second layer does not use an activation function. The operation of this layer is as follows

[0081] FeedForward(X)=max(0,XW1+b1)W2+b2 (9)

[0082] Where X is the input vector, W1, W2 are trainable weights, and b1, b2 are bias vectors.

[0083] The multi-head attention layer and the normalization layer, the feedforward neural network and the normalization layer are connected with residual connections respectively, and then the two are connected in series to obtain an encoding layer. Multiple encoding layers in series constitute the encoder of the Transformer.

[0084] The BERT model consists of an input layer and an intermediate layer. Its overall structure is as follows: Figure 6 BERT uses a stack of Transformer encoder-side networks. The self-attention mechanism in the encoder uses the tokens of its context when encoding a token. "Using the context at the same time" means BERT's bidirectional performance, which is different from Bi-LSTM, which inputs text in reverse order to achieve bidirectional representation.

[0085] The input vector of BERT consists of the sum of the position vector, character vector, and field vector, such as Figure 7 As shown. The processing of position vectors in BERT is different from the sine-cosine position encoding used in Transformer. Instead, it uses an embedding matrix to generate the position vector of each token in the input sequence. The character vector is generated by a subvector table. BERT converts each token into a 768-dimensional vector, and the two special tokens [CLS] and [SEP] are inserted at the beginning and end of the input text before word segmentation. Field vectors are used to distinguish sentences. For example, the vector corresponding to the first sentence is assigned all 0s, and the vector corresponding to the second sentence is assigned all 1s.

[0086] After adding the above three vectors, we can get the initial input vector of BERT, which needs to pass through a layer normalization and a random dropout layer. Layer normalization plays a role in alleviating the vanishing gradient, and random dropout plays a role in preventing overfitting.

[0087] After obtaining the input vector of BERT, it is input to the middle layer to extract text features. The BERT middle layer uses a multi-layer stacked Transformer encoder to model the contextual features and semantic features in the input sequence, and finally generates a dense vector corresponding to each token. The BERT-Base-Chinese model selected by the present invention has 12 stacked layers, the input vector dimension is 768, the dimension of the hidden layer is also 768, and the activation function of the model is Gaussian linear unit (GELU), as shown in formula (10).

[0088]

[0089] Among them, x is the input of the activation function, and tanh is the hyperbolic tangent function.

[0090] After loading the pre-trained BERT model, you can implement downstream tasks by fine-tuning BERT for different NLP tasks.

[0091] Furthermore, the SENet is an attention module with compression and excitation operations.

[0092] Specifically, the SENet module enhances the representation ability of convolutional neural networks by introducing a new structural unit "Squeeze-and-Excitation" (SE) block. It is to improve the representation ability of convolutional neural networks (CNNs) by explicitly modeling the dependencies between convolutional feature channels, thereby significantly improving network performance with almost no increase in computational cost. The two-step core operations of the SENet module are compression (Squeeze) and excitation (Excitation), as described. The SENet module is as follows Figure 8 shown.

[0093] SENet first aggregates the spatial dimensions (height H and width W) of the input feature map through a global average pooling operation to generate a channel descriptor for each channel. This step effectively compresses the global spatial information into a channel vector, capturing the global distribution of channel feature responses. This global information is crucial for the subsequent recalibration process.

[0094] After the compression step, an activation mechanism is applied, which is essentially a self-gating mechanism consisting of two fully connected (FC) layers and a non-linear activation function (usually sigmoid). The first FC layer reduces the dimensionality of the channel descriptor, applies a ReLU non-linear activation, and then the second FC layer projects it back to the original channel dimension. This process models the non-linear interactions between channels and produces a set of channel weights.

[0095] The output of the excitation operation is used to recalibrate the original input feature map. Each channel of the input feature map is scaled by the corresponding scalar in the excitation output. This step selectively emphasizes informative features while suppressing useless features, allowing the model to focus on the most relevant features for the task.

[0096] SENet was first proposed to improve the performance of deep neural networks in the field of computer vision (CV) for image classification, object detection, and instance segmentation tasks. The present invention creatively applies it to natural language processing (NLP). However, the data structure transmitted in the CV field network is different from that in the NLP field network. In order to combine the BERT-Base-Chinese model used in the present invention with the SENet module, some adjustments need to be made to the model architecture.

[0097] The original SENet module is designed to process convolutional feature maps with spatial dimensions, while the output of the BERT classification task is generally a two-dimensional tensor with dimensions [batch_size, hidden_size], where batch_size is the batch size of the input sample, and hidden_size is the hidden layer dimension of the BERT model (for example, the BERT-Base-Chinese model used in the present invention is 768). Therefore, it is necessary to increase the two dimensions of the BERT classification output from a two-dimensional tensor to a four-dimensional tensor to meet the input requirements of SENet. After the adjusted BERT output is sent to the SENet module to obtain the output, the SENet output needs to be restored to the original two-dimensional tensor shape to facilitate the subsequent fully connected layer classification processing.

[0098] Further, the S3 comprises the following steps:

[0099] S31, encoding the basic unit using the BERT model to obtain text feature representation;

[0100] S32, inputting the text feature representation into the SENet module to obtain an enhanced feature representation;

[0101] S33. Classify the enhanced feature representation through a classification layer to obtain the classification result.

[0102] Furthermore, the Bi-LSTM model is a bidirectional long short-term memory network model.

[0103] Specifically, the bidirectional long short-term memory network model (Bi-LSTM) is composed of two independent LSTM models. In the Bi-LSTM model, the input text sequence is input in the two LSTM models in forward and reverse order for feature learning, and then the output vectors of the two LSTM networks are spliced ​​as the final feature vector. The Bi-LSTM model of the present invention is as follows Fig. 9 shown.

[0104] Let i, f, and o represent the input gate, forget gate, and output gate of LSTM respectively. Assume that the format features and content features of the text are represented as: lo 、f ct , whose dimensions are d0 and d1 respectively, then the feature vector of the t-th text unit can be expressed as in

[0105] The samples captured for each training can be expressed as Where n is the number of samples captured for word training. Let h t is the output of the hidden layer in the current paragraph t, and the output at the previous moment is h t-1 , then the value of the forget gate f can be calculated by formula (11-13) t , the value of the output gate i t and the candidate value c' of the current state of the memory cell t

[0106] f t =σ(w f x t +w f h t-1 )+b f (11)

[0107] i t =σ(w i x t +w i h t-1 )+b i (12)

[0108] c' t =tanh(w c x t +w c h t-1 )+b c (13)

[0109] Among them, σ is the activation function Sigmoid; w is the weight matrix of the network; b is the bias vector of the network.

[0110] The current state value c can be calculated by formula (13): t , where c t-1 is the state at the previous moment, c' t is the candidate value of the current state, i t is the input of the current state, f t is the value that needs to be forgotten.

[0111] The value of the output gate o is calculated by formula (14) t ,

[0112] o t =σ(w o x t +w o h t-1 )+b o (14)

[0113] Among them, w o is the weight matrix of the output gate network, b o Bias vector for the output gate network.

[0114] The hidden state h of Bi-LSTM at the current moment is calculated by formula (15): t ,h t That is, the feature vector obtained by concatenating the t-th text unit in the text after it is processed by LSTM in two opposite directions, forward and reverse.

[0115] h t =o t ×tanh(c t ) (15)

[0116] Furthermore, the CRF model is a probabilistic graphical model.

[0117] Specifically, the conditional random field (CRF) belongs to the probabilistic graphical model and is a special form of Markov random field. CRF is a model commonly used to solve sequence labeling problems. Specifically, it constructs a Markov random field of the output sequence based on the input sequence to solve the sequence labeling problem. The CRF model described in the present invention is as follows Fig.10 shown.

[0118] Assuming that the input sequence of CRF is X and the output sequence is Y, under the condition that the input sequence X takes the value of x and the output sequence takes the value of y, the linear chain conditional random field P(Y|X) has the following form:

[0119]

[0120] In the formula, Z(x) is used to normalize

[0121] Z(x)=∑y exp(∑ i,k λ k t k (y i-1 ,y i ,x,i)+∑ i,l μ l s l (y i ,x,i)) (17)

[0122] s l =(y i ,x,i),l=1,2,...L (18)

[0123] t k (y i-1 ,y i ,x,i), k=1,2,...K (19)

[0124] In formula (16), i represents the position of the current node in the sequence; t k is the local feature function of the Y node context and is only related to the previous node, K is the total number of local feature functions of the node; s l is the characteristic function on the Y node and is only related to the current node, L is the number of current nodes; λ k and μ l are and the corresponding weights respectively.

[0125] Further, the S4 comprises the following steps:

[0126] S41, encoding the basic unit using the BERT model to obtain text feature representation;

[0127] S42, inputting the text feature representation into the Bi-LSTM model, performing forward and reverse feature learning, and obtaining a bidirectional feature representation;

[0128] S43: Use the CRF model to perform sequence labeling on the bidirectional feature representation to obtain the entity recognition result.

[0129] Embodiment 2:

[0130] Before analyzing the comparison results of the hybrid power information message automatic extraction model proposed in the present invention with the classic algorithm in text classification and named entity extraction tasks, the commonly used evaluation indicators in the NLP field are introduced. The evaluation indicators are described in combination with the binary classification confusion matrix shown in Table 1.

[0131] Table 1 Confusion matrix

[0132]

[0133] In Table 1, TP refers to the number of both actual and predicted positive examples; TN refers to the number of both actual and predicted negative examples; FN refers to the number of actual positive examples that are predicted as negative examples; FP refers to the number of actual negative examples that are predicted as positive examples.

[0134] Next, we will give an overview of the commonly used evaluation indicators in the NLP field:

[0135] Accuracy:

[0136] The accuracy value is the proportion of correctly predicted samples in the sample set, which is the most commonly used evaluation indicator, as shown in formula (20):

[0137]

[0138] Precision

[0139] The precision rate refers to the prediction result, that is, the proportion of true positive examples (TP) in the samples actually predicted as positive examples, as shown in formula (21):

[0140]

[0141] Recall

[0142] The recall rate refers to the proportion of samples that are correctly predicted in the actual positive sample set, as shown in formula (22):

[0143]

[0144] F1-score

[0145] F1-score is the weighted average of recall and precision, and is a comprehensive measure of the two, as shown in formula (23):

[0146]

[0147] The data set of the BERT-SENet text classification network proposed in the present invention comes from the Chinese text classification data set (THUCNews) launched by the Natural Language Processing Laboratory of Tsinghua University. THUCNews is generated by filtering and screening the historical data of Sina News RSS subscription channel from 2005 to 2011, and contains 740,000 news documents (2.19GB), all in UTF-8 plain text format. Based on the original Sina News classification system, THUCNews re-integrates and divides 14 candidate classification categories: finance, lottery, real estate, stocks, home, education, technology, society, fashion, current affairs, sports, constellations, games, and entertainment.

[0148] This specific embodiment extracts 200,000 news headlines in ten categories from the THUCNews dataset as a dataset, with 20,000 texts in each category. The ten categories are finance, real estate, stocks, education, technology, society, current affairs, sports, games, and entertainment. From these 200,000 texts, 180,000 texts are randomly extracted as a training set, 10,000 texts as a validation set, and 10,000 texts as a test set.

[0149] On the above dataset, the training accuracy and verification accuracy changes during the training of the text classification module (BERT-SENet) are as follows: Fig.11 As shown in the figure, as the training step size increases, the training accuracy and verification accuracy increase rapidly and finally tend to be flat.

[0150] The test results of the text classification module (BERT-SENet) proposed in the present invention are compared and evaluated with classic text classification models, such as TextCNN, BiLSTM, DPCNN, Transformer, and Transformer-based BERT model. Based on the evaluation indicators mentioned above, the comparison results are shown in Table 2. It can be seen that the BERT-SENet model proposed in the present invention has improved in all four evaluation indicators, and it can be concluded that the BERT-SENet model has a good improvement in text classification tasks.

[0151] Table 2 Comparison results between BERT-SENet model and classic model test

[0152]

[0153] The named entity recognition module (BERT-Bi-LSTM-CRF) used in the present invention tests the precision, recall and changes during the training process. Fig.12 As shown in the figure, with the increase of training step length, the above evaluation indicators increase rapidly and finally tend to be flat.

[0154] The test results of the BERT-Bi-LSTM-CRF network are compared with those of the BERT, BERT-Bi-LSTM and BERT-CRF networks, and the comparison results are shown in Table 3 based on the aforementioned evaluation indicators. As can be seen from the table, the BERT-Bi-LSTM-CRF network not only has a good improvement on BERT in the named entity recognition task, but also outperforms the BERT-Bi-LSTM and BERT-CRF networks.

[0155] Table 3 Comparison of named entity recognition results of various networks

[0156]

[0157] This embodiment has the following beneficial effects:

[0158] The hybrid electric power information message automatic extraction model proposed in the present invention, by combining advanced natural language processing technology, effectively realizes the automatic classification of electric power fault text and the extraction of key information. The text classification subtask of the model uses the BERT-SENet network, while the named entity recognition task uses the BERT-Bi-LSTM-CRF network, both of which show excellent performance in the corresponding tasks. Experimental comparison shows that the hybrid model surpasses the existing classical algorithms in evaluation indicators such as accuracy, precision, and recall. This research not only improves the efficiency of power system fault handling, but also provides new ideas and methods for the automatic analysis of text data in similar fields.

Claims

1. A method for automatically extracting power information messages based on deep learning technology, characterized in that: The method comprises the following steps: S1. Input power failure text; S2. Preprocessing the power fault text to obtain basic units in discrete form; S3. Based on a text classification module, perform text classification on the basic unit and output a classification result, wherein the text classification module includes a BERT model and a SENet module; S4, based on a named entity recognition module, performing named entity recognition on the basic unit and outputting an entity recognition result, wherein the named entity recognition module includes a BERT model, a Bi-LSTM model and a CRF model; S5. Integrate the classification result and the entity recognition result to obtain a final power information message extraction result.

2. According to claim 1, a method for automatically extracting electric power information messages based on deep learning technology is characterized in that: The core network of the BERT model consists of multiple Transformer encoder stacks.

3. According to the method for automatic extraction of electric power information messages based on deep learning technology in claim 1, it is characterized in that: The SENet is an attention module with compression and excitation operations.

4. The method for automatically extracting electric power information messages based on deep learning technology according to claim 1 is characterized in that: The S3 comprises the following steps: S31, encoding the basic unit using the BERT model to obtain text feature representation; S32, inputting the text feature representation into the SENet module to obtain an enhanced feature representation; S33. Classify the enhanced feature representation through a classification layer to obtain the classification result.

5. The method for automatically extracting electric power information messages based on deep learning technology according to claim 1 is characterized in that: The Bi-LSTM model is a bidirectional long short-term memory network model.

6. The method for automatically extracting electric power information messages based on deep learning technology according to claim 1, characterized in that: The CRF model is a probabilistic graphical model.

7. The method for automatically extracting electric power information messages based on deep learning technology according to claim 1 is characterized in that: The S4 comprises the following steps: S41, encoding the basic unit using the BERT model to obtain text feature representation; S42, inputting the text feature representation into the Bi-LSTM model, performing forward and reverse feature learning, and obtaining a bidirectional feature representation; S43: Use the CRF model to perform sequence labeling on the bidirectional feature representation to obtain the entity recognition result.

Images