Entity information extraction method for power defect text based on improved Transformer encoder

Abstract

The present invention relates to the intersection of artificial intelligence and power systems, and is a method for extracting entity information from power defect text based on an improved Transformer encoder. This method introduces a pre-trained language model, a dictionary, a fine-tuned TENER model, and a conditional random field model to build a CWG-TENER model. Using annotated power system secondary equipment defect text, the model is optimized for training and test selection to obtain a power equipment defect text information extraction model. The power equipment defect text to be extracted is input into the power equipment defect text information extraction model to obtain the extracted information. The present invention can be used to extract entity information involved in power system secondary equipment defect text, and can provide auxiliary decision-making when power system secondary equipment fails.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence and power systems, and in particular relates to a method for extracting entity information from power defect text based on an improved Transformer encoder. Background Art

[0002] The informatization of the power sector has led to an explosive growth in data related to power systems. Defect texts for power equipment record a vast amount of production process information, highlighting the importance of mining this valuable information for the development of the power industry. However, current power defect texts lack efficient structured management, and irregular text entry hinders information utilization. Due to insufficient utilization of defect text information, the same defect often recurs in different regions. If a defect occurs during operation, maintenance personnel cannot rely solely on their own experience to promptly determine the exact cause of the defect. This can lead to a series of cascading failures due to the lack of timely and appropriate treatment of the critical defect. Furthermore, with the continuous development of artificial intelligence (AI), the application of AI technology to the power industry is an inevitable requirement for its development. Natural language processing technology has been successfully applied in power systems, such as in the power Internet of Things (IoT) and intelligent power search engines. Named Entity Recognition (NER), a fundamental task of natural language processing, enables more efficient utilization of power defect texts due to its powerful information extraction and classification capabilities.

[0003] In recent years, the application of named entity recognition in machine translation, question-answering systems, and other fields has demonstrated its advantages in entity recognition, including the utilization of centrally stored information in power systems. Many researchers have attempted to use NER to address cross-regional access to unstructured text in power systems, assist decision-making, and build intelligent diagnostic platforms, with promising results.

[0004] However, the various recurrent neural networks (RNNs) currently used in NER tasks for power text data lack sufficient utilization of long-range context and parallel capabilities, limiting their application in scenarios where large amounts of defect text are accumulating in power systems and high accuracy is required. Therefore, many researchers have attempted to use existing neural network models based on concepts such as self-attention mechanisms and position embedding to build a model that can improve the problems of the bidirectional long short-term memory (BiLSTM) network, a typical recurrent neural network used for NER. Specifically, foreign researchers proposed a Transformer encoder that uses a fully connected self-attention structure to simulate long-range context. This encoder has begun to be applied to various NLP tasks and has achieved impressive results. However, research and results on the problem of information extraction from power defect text have not yet been carried out. Summary of the Invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides an entity information extraction method for power defect text based on an improved Transformer encoder. By introducing a pre-trained language model on the basis of the original Transformer-based NER model to convert text data into word vectors, and introducing a dictionary obtained by word segmentation of a large amount of corpus to fuse word information on the basis of character information, and updating the network in a graph manner to better fuse characters, words and global information, the power defect text entity information can be extracted more accurately.

[0006] The present invention can be achieved by adopting the following technical solutions:

[0007] A method for extracting entity information from power defect text based on an improved Transformer encoder, the method comprising:

[0008] S1. Import the defect record data of the secondary equipment in the power system and mark the data text;

[0009] S2. Introduce the pre-trained model, dictionary, fine-tune the TENER model and conditional random field model, build the CWG-TENER model, and use the annotated data text to optimize and train the CWG-TENER model to obtain the power equipment defect text information extraction model;

[0010] S3. Input the power equipment defect text to be extracted into the power equipment defect text information extraction model to obtain the extracted information.

[0011] Specifically, step S2 includes:

[0012] S21. Introduce a pre-trained model and a dictionary to extract character vectors of the data text and word vectors of dictionary words, where the dictionary is obtained based on word segmentation of a plurality of original corpora;

[0013] S22. The extracted character vectors form a character vector set C. The data text is matched with words in the dictionary, and the word vectors corresponding to the matched words form a word vector set W:

[0014] S23, build word graph CWG model;

[0015] S24. Replace the CRF layer of the Transformer model with a fully connected layer so that the output dimension is the same as the word vector dimension, and obtain the fine-tuned TENER model;

[0016] S25. Use the character vector set C and the word vector set W as the input of the fine-tuning TENER model to obtain the initial value C of the output node feature vector0 and the initial value of the edge eigenvector W 0 , the initial value of the node eigenvector C 0 and the initial value of the edge eigenvector W 0 Replace the nodes and edges of the CWG model respectively, and define the initial value of the CWG model global variable as g 0 ;

[0017] S26, perform aggregation calculations on the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model respectively to obtain the character vector after the first aggregation Word vector and the global vector

[0018] S27, character vector Word vector and the global vector Replace the nodes of the CWG model, the edges of the CWG model and the global variables of the CWG model;

[0019] S28. Update the character vector and word vector by fine-tuning the TENER model, and calculate the updated output of the global vector using the LSTM network state update formula;

[0020] S29. Replace the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model with the updated character vectors, word vectors, and global vectors, respectively, and aggregate the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model;

[0021] S210, looping steps S28 to S29 for T times to obtain a final set of character feature vectors;

[0022] S211, inputting the final character feature vector set into the conditional random field model CRF, and calculating and outputting the optimal label sequence;

[0023] S212. Optimize the model parameters using the Adam optimizer based on the optimal label sequence, perform cyclic training for a predetermined number of times, and obtain a power equipment defect text information extraction model.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] The present invention provides an entity information extraction method for power defect text based on an improved Transformer encoder. By introducing a pre-trained language model, a dictionary, a fine-tuned TENER model and a conditional random field model, a CWG-TENER model is built. The model is optimized for training and test selection using annotated power system secondary equipment defect text to obtain a power equipment defect text information extraction model. The model can be used to extract entity information involved in power system secondary equipment defect text, more effectively extracting the required entity information from the power system secondary equipment defect text, so as to subsequently build a knowledge graph and provide auxiliary decision-making when power system secondary equipment fails. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0027] Figure 1 This is a flowchart of the key information extraction process of the defect record text of the power secondary equipment in the embodiment of the present invention;

[0028] Figure 2 is a labeled example diagram of an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of the structure of the CWG model in the present invention in an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the CWG-TENER model structure and operation flow in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is obvious that the embodiments described are only some embodiments of the present invention, not all embodiments, and the implementation of the present invention is not limited to these. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] Example 1:

[0033] This embodiment is based on the problem of extracting "defect phenomenon" information from the text of functional defects of secondary equipment in the power system. It constructs a "word graph" model for the text of functional defects of secondary equipment in the power system, and uses an encoder suitable for NER improved based on Transformer to aggregate and update the graph neural network. Finally, it uses the conditional random field model to output the label sequence for the text of functional defects of secondary equipment in the power system, and extracts the "defect phenomenon" information according to the label sequence.

[0034] like Figure 1 As shown, this embodiment provides a method for extracting entity information from power defect text based on an improved Transformer encoder, which specifically includes the following steps:

[0035] S1. Import the data text of the secondary equipment defect record of the power system and annotate the data text. The data text annotation result example is Figure 2 shown.

[0036] Taking the extracted phrase "defect phenomenon" as an example, the first letter of the phrase representing the defect phenomenon in the text is marked as "B", the remaining characters in the phrase are marked as "I", and the characters in the text that are not related to "defect phenomenon" are marked as "O".

[0037] by Figure 2 Take the Chinese text "The protection device operates abnormally." as an example. The phrase representing the "defect phenomenon" is "device operates abnormally", so "install" is marked as "B", "install operation abnormally" is marked as "I", and "protection" and "." are not related to the "defect phenomenon" and are marked as "O".

[0038] S2: Introduce the pre-trained model, dictionary, fine-tune the TENER model and conditional random field model, build the CWG-TENER model, use the data text annotated in S1 to optimize and train the model, and obtain the power equipment defect text information extraction model.

[0039] S21: Introduce a pre-trained model and dictionary to extract character vectors of data text and word vectors of dictionary words. The dictionary is obtained based on word segmentation of a large amount of original corpus. The pre-trained model is any one of the following: BERT model, BERT-wwm model, and ERNIE model.

[0040] BERT (Bidirectional Encoder Representations from Transformers) is a Transformer-based bidirectional encoder released by Google in 2018. It is "the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus." A pre-trained BERT model requires only an additional output layer for fine-tuning to generate models capable of handling a wide range of natural language processing tasks.

[0041] BERT-wwm (Whole Word Masking) is an upgraded version of BERT released by Google in 2019. It primarily changes the training sample generation strategy during the pre-training phase. It replaces the original word-piece masking method with whole-word masking. For Chinese applications, if one character is masked, all other characters belonging to the same word are also masked.

[0042] ERNIE (Enhanced Representation through Knowledge Integration) is a BERT-based optimization model released by Baidu in 2019. It mainly improves the masking mechanism and consists of three types of masks: basic-level masking (word piece), phrase-level masking (WWM style), and entity-level masking.

[0043] BERT / BERT-wwm uses Wikipedia data for training and performs better on formal text. ERNIE, on the other hand, uses additional online data from Baidu Tieba and Zhidao, making it more effective for informal text (e.g., Weibo). To process Traditional Chinese data, BERT or BERT-wwm is recommended, as Traditional Chinese is rarely present in ERNIE's vocabulary.

[0044] S22: The extracted character vectors constitute a character vector set C, the data text is matched with the words in the dictionary, and the word vectors corresponding to the matched words constitute a word vector set W.

[0045] The character vector set C is:

[0046] C=[c1,c2,...,c m ]

[0047] Among them, c1, c2, ..., c m is the character vector extracted from the data text by the pre-trained model, and m is the total number of characters in the text.

[0048] Attach Figure 2 Take the sequence "Protection device is operating abnormally." as an example. There are 9 characters in total, m = 9, and the dimension of the character vector is d model , C obtained from this sequence is a d model ×m matrix.

[0049] The word vector set W is:

[0050]

[0051] in, is the word vector corresponding to the matched word, and its dimension is the same as the character vector, b i 、e i are the first and last characters of the word corresponding to the i-th word vector, and n is the total number of words matched by the data text in the dictionary.

[0052] The specific definition of a matching word is: if a word in the dictionary contains any character in the data text, then the word is a matching word.

[0053] Attach Figure 2 Take the sequence "Protection device operating abnormally." as an example, there are 5 words "protection", "device", "operation", "abnormal", and "protection device", n=5, and W obtained from this sequence is a d model ×n matrix.

[0054] S23: Build a Character-Word Graph (CWG) model. The CWG model is a directed graph consisting of data text information, where the character vector c i Nodes that make up the graph, word vectors Composed from the character b j The corresponding node points to the character e j The edge corresponding to the node.

[0055] By the attached Figure 2 The CWG model constructed by the sequence is as shown in the attached Figure 3 shown.

[0056] Next, the CWG model undergoes an "update → aggregate → update → ... → aggregate" cycle to extract text features. Specifically, the following steps are performed: character vectors C (nodes) and word vectors W (edges) are updated and the initial value of the global vector g is calculated → character vectors C (nodes), word vectors W (edges), and global vector g are aggregated → character vectors C (nodes), word vectors W (edges), and global vector g are updated → ... → character vectors C (nodes), word vectors W (edges), and global vector g are aggregated. This process is described in detail below.

[0057] S24: Introduce a fine-tuned TENER model to perform the "update" operation on the CWG model. The TENER model is an improved Transformer model based on the named entity recognition task. The specific fine-tuning method is to replace the model's CRF layer with a fully connected layer to make the output dimension the same as the word vector dimension, thus obtaining a fine-tuned TENER model.

[0058] As in this embodiment, when the node is updated in the subsequent steps, the output obtained by the attention mechanism is The formula of the fully connected layer is:

[0059] C t+1 =U Linear C t+1 ′+B Linear

[0060] Among them, U Linear 、B Linear are all trainable parameters in the fully connected layer, Therefore, the character vector C obtained after completing a round of update t+1 Dimensions and C t Same, that is

[0061] S25: Use the character vector set C and word vector set W obtained in S22 as the input of the fine-tuning model TENER to obtain the output C 0 and W 0 , that is, the character vector and word vector obtained after the first round of "update" operation, are used as the initial value of the feature vector of the node and edge, replacing the node and edge of the CWG model in S23. At the same time, the initial value of the global variable of the CWG model is defined as g 0 =average(C,W);

[0062] S26: Aggregate the eigenvectors of the CWG character nodes, edge eigenvectors, and global variables respectively to obtain the character vector after the first aggregation. Word vector and the global vector The specific method is:

[0063] The aggregation formula of the node is:

[0064]

[0065] Here, i represents the i-th character, and t represents the t-th round of update. is the feature vector of the character node before the t-th round of aggregation, is the character node feature vector after aggregation, for The predecessor node feature vector of for The incoming edge eigenvector of Indicates concatenation of two vectors, and MultiAtt() indicates aggregation using multi-head attention.

[0066] The edge aggregation formula is:

[0067]

[0068] in, is the characteristic vector of the edge from node b to node e before the tth round of aggregation, is the feature vector of the edge after aggregation, For the edge w b，e The set of feature vectors corresponding to all characters that match the word.

[0069] The calculation formula for global variables is:

[0070]

[0071]

[0072]

[0073] in, is a set of feature vectors corresponding to all characters in the input text sequence, The word vectors corresponding to all matched words form a word vector set, g t is the global vector before the t-th round of aggregation, is the global vector after incorporating character vector information during the t-th round of aggregation, is the global vector after incorporating word vector information during the t-th round of aggregation, is the final global vector obtained from the t-th round of aggregation.

[0074] S27: Character vector Word vector and the global vector The feature vectors of the character nodes, the edge feature vectors and the global feature vector of the CWG model are replaced.

[0075] S28: Update the character vectors and word vectors by fine-tuning the TENER model, and calculate the updated output of the global vector using the LSTM network state update formula.

[0076] S281, perform t rounds of aggregation based on the feature vector of the character node, and use the output of t rounds of aggregation as the input of the fine-tuned TENER model to obtain a new character vector. structure As the input of the fine-tuned TENER model, update, The specific formula is:

[0077]

[0078] Here, i represents the i-th character, and t represents the t-th round of update. is the feature vector of the character node before the t-th round of aggregation, is the character node feature vector after the t-th round of aggregation, is the global vector after round t aggregation, Input for the constructed fine-tuned TENER model.

[0079] The Input the fine-tuned TENER model to update and get the output C t+1 , specifically:

[0080]

[0081] Among them, FTTENER c () represents the fine-tuned TENER model for character vectors.

[0082] The fine-tuned TENER model includes an attention mechanism and position encoding, and the attention mechanism includes a single-head attention mechanism and a multi-head attention mechanism.

[0083] Relative to The position code is:

[0084]

[0085]

[0086]

[0087] Among them, i and j represent the i-th character and the j-th character respectively, p i，j for Relative to The relative position encoding vector of the elements, (2k), (2k+1) is the index of the element in a word vector, d input is the input dimension of the FTTENER model, R ij For the end Relative to Position encoding.

[0088] In this embodiment, when the single-head attention mechanism is used in fine-tuning the TENER model, the specific formula is as follows.

[0089] Input is the

[0090]

[0091] where d model is the dimension of the character vector, and m is the total number of characters in the text. The input consists of three learnable matrices Projected into different spaces, the output of the attention mechanism can be obtained by the following equation:

[0092]

[0093]

[0094] Among them, Q is the query vector in the attention mechanism, K is the key vector in the attention mechanism, V is the value vector in the attention mechanism, Q y is the query vector of the y-th text character, K z is the key vector of the zth text character, represents the transpose of the vector, z is the character number of the yth text character, A y，z Represents the attention value of the y-th text character to the z-th text character. The output of the single-head attention mechanism is shown in S24:

[0095]

[0096] According to the position encoding formula, d input =5d model , d model is the dimension of the character encoding vector, then Relative to The position code is:

[0097]

[0098]

[0099]

[0100] When fine-tuning the multi-head attention mechanism in the TENER model to improve self-attention ability, set n groups of mapping matrices Its output equation is as follows:

[0101]

[0102]

[0103]

[0104] Among them, n is the number of heads, and the superscript h is the head index, that is, Q(h) , K (h) 、V (h) They are the query vector, key vector and value vector of the hth head respectively. is the learnable matrix corresponding to the above three vectors, is the query vector of the hth head of the yth text character, is the transpose of the key vector of the hth head of the zth text character, is the attention value of the y-th text character to the z-th text character in the h-th head, head (h) is the output of the h-th head in the multi-head attention mechanism. is a learnable parameter, then the output C t+1 'for:

[0105] C t+1 ′=W o [head (1) ;...;head (n) ]

[0106] At this time, the dimension d input =5d model , Relative to The position coding is the same as that of single head.

[0107] S282, perform t rounds of aggregation based on the edge feature vectors, and use the output of t rounds of aggregation as the input of the fine-tuned TENER model to obtain the new word vector. Specifically, the new word vector is obtained from S26. structure As the input of the fine-tuned TENER model, the specific formula is:

[0108]

[0109] Here, i represents the i-th edge and t represents the t-th round of update. is the eigenvector of the front edge of the t-th round aggregation, is the eigenvector after the t-th round of aggregation, is the global vector after round t aggregation, Input for the constructed fine-tuned TENER model.

[0110] The Input the fine-tuned TENER model to update and get the output W t+1 :

[0111]

[0112] Among them, FTTENER w() represents the fine-tuned TENER model for word vectors.

[0113] The position encoding of the input in the FTTENER model described in S282 is slightly different from that in the model described in S281, specifically:

[0114]

[0115]

[0116]

[0117]

[0118] in, represents the distance between the first character of the i-th word and the first character of the j-th word, Represents the distance between the first character of the i-th word and the last character of the j-th word, and so on.

[0119]

[0120]

[0121] Among them, p pos are the four relative position encoding vectors of words, pos are (2k) is the index of an element in a word vector, d input is the input dimension of the FTTENER model.

[0122] In this embodiment, as shown in the attached Figure 3 As shown, The rest of the relative position codes are similar. d model is the encoding dimension of the word vector.

[0123]

[0124] Among them, U r is a trainable parameter representing a linear layer, so that the final R ij The dimension of is the same as the input vector dimension. In this embodiment, R ij Encode the position of the final i-th word vector relative to the j-th word vector.

[0125] S283, calculate and update global variables through LSTM network state update formula Get g t+1 , LSTM network is long short-term memory network, which is often used to process NLP problems. The update method of refers to the update method of the status value, and the calculation formula is:

[0126]

[0127]

[0128]

[0129] g t+1 =f t+1 ⊙g t +i t+1 ⊙u t+1

[0130] Among them, U, V and b are trainable parameters. Refers to i and f respectively, that is, formula 1 actually contains two formulas, u t +1 、i t+1 、f t+1 、 Intermediate variables are introduced to clarify the formula.

[0131] S29: Substituting the updated character vectors, word vectors, and global vectors for the nodes, edges, and global variables of the CWG model, respectively, and aggregating the nodes, edges, and global variables of the CWG model;

[0132] S210: looping step S28 to step S29 for T times to obtain a final set of character feature vectors.

[0133] S211: Input the character vector set corresponding to the final node into the conditional random field model CRF to calculate the optimal label sequence. The specific calculation formula includes:

[0134]

[0135]

[0136] Among them, i represents the i-th node; is the transpose of the final eigenvector of the i-th node; l i is the label of the i-th node; and For label l i-1 and l i The trainable parameters of The intermediate variables introduced to clearly express the formula represent and l i-1 、l i 、 The calculation formula related to these three variables and The meaning of is the same; is the best label sequence, represents the label of the i-th node in the best label sequence, The meaning of is the same; Y(s) is the set of all labels in the current situation s; Indicates that the best label sequence under the current situation s is probability.

[0137] For the training process, the loss function is:

[0138]

[0139] Where N is the total number of tag sequences contained in Y(s).

[0140] In this embodiment, if the Figure 2 The dataset where the sequence shown is located is the training set, and the label dictionary is defined:

[0141] tag2label={B:0,I:1,O:2}

[0142] Then for this sequence, A random combination of three labels, a total of 3 9 There are 3 possible values ​​in Y(s) 9 elements.

[0143] For the testing and decoding process, the optimal label sequence y is found by * :

[0144] y * =argmax y∈Y(s) p(y|s)

[0145] Among them, p(y|s) represents the probability of any label sequence y under the current situation s, and the best label sequence y * The labeling result corresponding to each input character.

[0146] In this embodiment, if the Figure 2 The sequence shown is a test set. Define the label dictionary, and the best result y is completely correct. * =2,2,0,1,1,1,1,1,2.

[0147] S212: Sequence y obtained according to S28 * ,The Adam optimizer is used to optimize the model parameters, and after a certain number of cyclic training, a CWG-TENER model with better effect for extracting text information of power equipment defects is obtained.

[0148] In this embodiment, the model effect is judged based on the performance of the model on the test set, specifically based on three commonly used indicators of named entity recognition tasks: precision, recall rate and F1 value.

[0149] S3. Input the power equipment defect text to be extracted into the power equipment defect text information extraction model to obtain the extracted information.

[0150] The information to be extracted may include: information such as defect phenomena, defect causes, and solutions involved in the defect record text of the power secondary equipment.

[0151] The extracted information can be used to construct a subsequent knowledge graph to query solutions when power system secondary equipment fails, providing support for decision-making. This model can obtain more complete and accurate entity information than existing models, making the resulting support decision-making system more practical.

[0152] In this embodiment, the target extraction information is defect phenomenon information, and the overall model architecture is as shown in the attached Figure 4 shown.

[0153] First, the data text is fed into the pre-trained language model BERT / BERT-wwm / ERNIE. The characters in the data text are converted into character vectors, and the words in the dictionary are converted into word vectors of the same dimension using the pre-trained language model. The TENER model performs initial feature extraction on the character and word vectors, obtaining the initial values ​​for these vectors, which then enter the "aggregate → update → aggregate →..." loop. The initial value of the global vector is also calculated. The character vectors, word vectors, and global vector are then passed through the "aggregate" layer to obtain the aggregated output. The character and word vectors are then positionally encoded and fed into an N-headed Transformer layer, the "update" layer, to obtain the updated output. The updated output of the global vector is also calculated. Finally, the character vectors, word vectors, and global vector are fed into a linear layer to obtain an output of the same dimension as the initial output. This output is then fed into the "aggregate" layer again, and this cycle repeats T times. After the final aggregation operation, the final character feature vector is fed into the CRF layer to obtain the final label output.

[0154] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. The entity information extraction method for power defect text based on the improved Transformer encoder is characterized by: The following steps are involved: S1. Import the defect record data of the secondary equipment in the power system and mark the data text; S2. Introduce the pre-trained model, dictionary, fine-tune the TENER model and conditional random field model, build the CWG-TENER model, and use the annotated data text to optimize and train the CWG-TENER model to obtain the power equipment defect text information extraction model; The step S2 comprises: S21. Introduce a pre-trained model and a dictionary to extract character vectors of the data text and word vectors of dictionary words, where the dictionary is obtained based on word segmentation of a plurality of original corpora; S22, the extracted character vectors constitute a character vector set C, the data text is matched with words in the dictionary, and the word vectors corresponding to the matched words constitute a word vector set W; S23, build word graph CWG model; S24. Replace the CRF layer of the Transformer model with a fully connected layer so that the output dimension is the same as the word vector dimension, and obtain the fine-tuned TENER model; S25. Use the character vector set C and the word vector set W as the input of the fine-tuning TENER model to obtain the initial value C of the output node feature vector 0 and the initial value of the edge eigenvector W 0 , the initial value of the node eigenvector C 0 and the initial value of the edge eigenvector W 0 Replace the nodes and edges of the CWG model respectively, and define the initial value of the CWG model global variable as g 0 ; S26, perform aggregation calculations on the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model respectively to obtain the character vector after the first aggregation Word vector and the global vector S27, character vector Word vector and the global vector Replace the nodes of the CWG model, the edges of the CWG model and the global variables of the CWG model; S28. Update the character vector and word vector by fine-tuning the TENER model, and calculate the updated output of the global vector using the LSTM network state update formula; S29. Replace the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model with the updated character vectors, word vectors, and global vectors, respectively, and aggregate the nodes of the CWG model, the edges of the CWG model, and the global variables of the CWG model; S210, looping steps S28 to S29 for T times to obtain a final set of character feature vectors; S211, inputting the final character feature vector set into the conditional random field model CRF, and calculating and outputting the optimal label sequence; S212, optimizing the model parameters using the Adam optimizer based on the optimal label sequence, and looping the training for a predetermined number of times to obtain a power equipment defect text information extraction model; S3. Input the power equipment defect text to be extracted into the power equipment defect text information extraction model to obtain the extracted information.

2. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: The marking of the data text includes: The first letter of the phrase representing the defect phenomenon in the data text is marked as "B", the remaining characters in the phrase are marked as "I", and the characters in the text that do not represent the defect phenomenon are marked as "O".

3. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: The pre-trained model is any one of the BERT model, the BERT-wwm model and the ERNIE model.

4. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: The CWG model is a directed graph consisting of data text information, where character vectors constitute the nodes of the graph and word vectors Composed from the character b j The corresponding node points to the character e j The edge corresponding to the node.

5. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: The step S26 performs aggregation calculations on the nodes of the CWG model, the edges of the CWG model and the global variables of the CWG model respectively, wherein: The aggregation formula for the nodes of the CWG model is: Among them, i represents the i-th character, t represents the t-th round of update, is the feature vector of the character node before the t-th round of aggregation, is the character node feature vector after aggregation, for The predecessor node feature vector of for The incoming edge eigenvector of Indicates concatenation of two vectors, MultiAtt() indicates aggregation using multi-head attention; The aggregation formula for the edges of the CWG model is: in, is the characteristic vector of the edge from node b to node e before the tth round of aggregation, is the feature vector of the edge after aggregation, For the edge w b,e A set of feature vectors corresponding to all characters that match the word; The calculation formula for the global variables of the CWG model is: in, is a set of feature vectors corresponding to all characters in the input text sequence, The word vectors corresponding to all matched words form a word vector set, g t is the global vector before the t-th round of aggregation, is the global vector after incorporating character vector information during the t-th round of aggregation, is the global vector after incorporating word vector information during the t-th round of aggregation, is the final global vector obtained from the t-th round of aggregation.

6. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 5 is characterized in that: The step S28 includes: S281, perform t rounds of aggregation based on the feature vectors of the character nodes, add the output of the t rounds of aggregation to the position code as the input of the N-head fine-tuning TENER model, and obtain an updated character vector; S282, perform t rounds of aggregation based on the edge feature vectors, add the output of the t rounds of aggregation to the positional encoding as the input of the N-head fine-tuned TENER model, and obtain the updated word vector; S283, calculate and update global variables through LSTM network state update formula Get g t+1 , the calculation formula is: g t+1 =f t+1 ⊙g t +i t+1 ⊙u t+1 ; Among them, U, V and b are trainable parameters. Refers to i and f, u respectively t+1 、i t+1 、f t+1 、 is an intermediate variable.

7. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: In step S211, the final character feature vector set is input into the conditional random field model CRF to calculate and output the optimal label sequence, wherein the calculation formula used includes: Among them, i represents the i-th node; is the transpose of the final eigenvector of the i-th node; l i is the label of the i-th node, and For label l i-1 and l i The trainable parameters of The intermediate variables introduced to clearly express the formula represent and l i-1 、l i 、 The calculation formula related to these three variables and The meaning of is the same; is the best label sequence, represents the label of the i-th node in the best label sequence, The meaning of is the same; Y(s) is the set of all labels in the current situation s, Indicates that the best label sequence under the current situation s is probability.

8. The entity information extraction method for power defect text based on the improved Transformer encoder according to claim 1 is characterized in that: The information to be extracted in step S3 includes: defect phenomenon information, defect cause information, and solution information involved in the defect record text of the power secondary equipment.

Images