Substation alarm event identification method based on pre-training model and data enhancement

Abstract

The application discloses a substation alarm event identification method based on a pre-training model and data enhancement, and belongs to the field of substation intelligent operation and maintenance. The identification method carries out data cleaning through an uneven alarm information data preprocessing method and an improved alarm information data enhancement method, pre-trains a model by using a general corpus, fine-tunes a classifier by introducing a dynamic scaling cross-entropy loss, and obtains a final substation alarm event according to an identification result. The method disclosed by the application effectively detects the fault type of the substation alarm event under the condition of sample imbalance by extracting long-distance text features in the alarm information, provides a more accurate identification method for substation intelligent operation and maintenance, and meets the needs of field operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance of substations, specifically to a method for identifying substation alarm events based on pre-trained models and data augmentation. Background Technology

[0002] With the continuous development of the power grid, the number of substations is increasing daily, and the corresponding alarm information data is growing explosively. The large volume of initial alarm notifications makes it difficult for maintenance personnel to quickly determine the true type of fault and take timely action. At the same time, the operational characteristics of the power grid require a transformation of the existing dispatching model, enhancing the intelligence level of dispatching incident handling, and improving incident handling efficiency to ensure the safe and stable operation of the power grid.

[0003] In recent years, with the continuous development of machine learning, substation fault diagnosis methods combining natural language processing technology have gradually emerged, shifting the identification of alarm events from model-driven to data-driven. However, traditional machine learning models cannot effectively represent the semantics inherent in natural language, thus requiring high-quality labeled data. Even after representation, they can only be used for simple judgments of the importance of accidents, but cannot determine the accurate category of alarm events, thus becoming a bottleneck in substation alarm event identification. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a substation alarm event identification method based on pre-trained models and data augmentation.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A substation alarm event identification method based on pre-trained models and data augmentation includes the following steps:

[0007] Collect real alarm event data from substations.

[0008] Data preprocessing is performed on unevenly distributed real alarm event data, sampling and text enhancement are performed to obtain an alarm information training sample library.

[0009] The model is pre-trained using a general corpus, extracting text features from both the word and sentence levels to perform semantic representation, training the model to fit the target, and enabling direct application in the recognition of alarm events.

[0010] Based on the alarm information training sample library, the Focal Loss loss function is introduced as the loss function of the classifier after the fitted pre-trained model, and the alarm event is accurately identified by dynamically scaling the loss.

[0011] Optionally, the actual alarm event data includes alarm information and its corresponding fault type, occurrence time, alarm text, and alarm event code during the actual operation and maintenance of the substation.

[0012] Optionally, the text enhancement includes synonym replacement, random insertion, random swapping, and random deletion of the alarm information text.

[0013] Optionally, the text enhancement includes the following steps: in the alarm information dataset In the training dataset, there are a total of n alarm events, where U and y are the text content and label of the alarm event, respectively, and i is the sequential index of the alarm event.

[0014] When the total number of alarm event categories is q, for any alarm message with any label y∈{1,…,q}, choose any of the following steps for processing:

[0015] Step 1) Randomly select m non-stop word lexical units from the alarm information, and randomly select one of their synonyms to replace them;

[0016] Step 2) Randomly select a word from the stop word library that has no actual semantic meaning, and randomly insert it into a certain position in the alarm message, repeating m times;

[0017] Step 3) Randomly select two words from the alarm message and swap their positions, repeating m times;

[0018] Step 4) Randomly delete a word from the alarm message with probability p, repeating m times;

[0019] A hyperparameter β is introduced, and the number of repetitions is combined with the number of lexical units by setting m = βl; in particular, p = β during the random deletion process; a stop word library is selected as the random insertion word library, so that the inserted lexical units contain less of their own semantics and no longer have additional semantic interference; a portion of samples are randomly selected from the generated multiple texts to supplement the dataset for model training.

[0020] Optionally, the model is the BERT model.

[0021] Optionally, in the BERT model, a multi-head self-attention mechanism is used to dynamically represent the meaning of lexical units, projecting the query and key-value pairs into multiple vector spaces to obtain text representations under different attention weights, enabling the model to focus on the correct fault-device association; the corresponding input values ​​are mapped to the output through the query and a set of key-value pairs. Q, K i V i These represent the query, key-value pair, and value pair, respectively, where i is the corresponding index and d... kThis represents the dimension of the query vector. After obtaining multiple text representations, multi-head attention merges and reduces the dimensionality of the results to serve as the final representation of the term.

[0022] Optionally, during the training of the BERT model, positional embedding is added to the text vector representation, using trigonometric functions as the positional embedding function. Based on parity, the positional representation of the alarm term in the alarm statement is as follows: pos represents the sentence to which the word belongs, i represents the index of the word, and d represents the index of the word. model Dimensions of word embedding

[0023] Optionally, the BERT model can be pre-trained using masked language and sentence prediction methods.

[0024] Optionally, Focal Loss can be used to fine-tune the alarm event recognition model, and its calculation expression is as follows:

[0025]

[0026] N is the number of samples, q is the number of classes, c is the class index, and y is the number of classes. ic For the corresponding category, p ic Let α be the probability that sample i belongs to category c, α be the weight coefficient that is inversely proportional to the probability distribution of the sample, and γ be the scaling factor.

[0027] A computer-readable storage medium storing instructions that, when executed, enable the aforementioned substation alarm event identification method.

[0028] The beneficial effects of this invention are:

[0029] 1. This invention proposes an improved data augmentation-based alarm information preprocessing method. This data augmentation method effectively alleviates the problem that alarm information constitutes a small proportion of normal operation information during actual maintenance. By utilizing synonym and stop word libraries, the information of the relatively small alarm information types is expanded, significantly improving the performance of alarm fault type identification in small sample scenarios.

[0030] 2. The prediction method of this invention proposes an alarm information feature extraction process based on a pre-trained model. The feature extraction and vector representation model is trained on a general domain corpus. By masking language and sentence prediction tasks, information at the word and sentence levels is learned simultaneously. Based on multi-head attention, the model focuses on the correct alarm event-device association as much as possible.

[0031] 3. The prediction method of this invention introduces the Focal Loss function to design a fine-tuning process for the alarm event recognition model. The output of a specific neuron is taken as the representation of the alarm event. The dynamic scaling of the loss allows the information provided by alarm types with a small number of samples to play a normal role in event recognition, which significantly improves the accuracy of categories with a small proportion. Attached Figure Description

[0032] The invention will now be further described with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart of the substation alarm event identification method of the present invention for identifying substation alarm events in a scenario with uneven sample distribution;

[0034] Figure 2 This is a flowchart of the improved alarm text enhancement method of the present invention;

[0035] Figure 3 This invention relates to the substation alarm event identification method's attention to different word units when extracting alarm text features;

[0036] Figure 4 This is the confusion matrix of the identification results of the substation alarm event identification method of the present invention on real substation alarm information. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In some embodiments of the present invention, a substation alarm event identification method based on a pre-trained model and data augmentation is disclosed, comprising the following steps:

[0039] Step 1: Collect real alarm event data from substations where the sample size is insufficient and the frequency of occurrence varies significantly. The collected sample data includes data types such as occurrence time, substation name, alarm device, interval, and alarm text. In the data cleaning stage, firstly, symbols and spaces are removed based on the mapping of specific expressions. Secondly, word segmentation is performed at the granular level according to the power industry dictionary and the Chinese dictionary. Finally, mixed representations that do not belong to the physical quantity dimension are removed as noise words to reduce the dimensionality of the input features.

[0040] Specifically, the aforementioned real alarm event data with a shortage of samples and a significant difference in frequency can be real alarm event data with a sample size of less than 7.5% and an extreme difference in sample size between different events exceeding 46 times.

[0041] Step two involves augmenting a small sample of real alarm information data, proposing an improved alarm information data augmentation method, and sampling for text augmentation. Due to the specialized and unique nature of power operation and maintenance terminology, this paper incorporates a deprecated thesaurus into the data augmentation process. In the alarm information dataset... In the training dataset, there are n alarm events, where U and y are the text content and label of the alarm event, respectively, and i is the sequential index of the alarm event. When the total number of alarm event categories is q, for any alarm message with label y∈{1,…,q}, choose one of the following methods to operate:

[0042] 1) Synonym replacement: Randomly select m non-stop words from the alarm information, and randomly select one of their synonyms to replace them.

[0043] 2) Random insertion: Randomly select a word from the stop word library without actual semantic meaning and randomly insert it into a certain position in the alarm message, repeating m times.

[0044] 3) Random swap: Randomly select two words in the alarm message and swap their positions, repeating m times.

[0045] 4) Random deletion: Randomly delete a word in the alarm message with probability p, repeating m times.

[0046] When the length *l* varies significantly, since there are multiple alarm messages representing a specific alarm event, using the same number of enhancements *m* can lead to excessively high or low randomness in some enhanced samples. Therefore, a hyperparameter β is introduced, and the number of repetitions is combined with the number of tokens by setting *m = βl*. Specifically, during random deletion, *p = β*. By using a stop word library as the random insertion library, the inserted tokens contain less of their own semantics, eliminating additional semantic interference and accelerating model fitting. Finally, a portion of samples is randomly selected from the generated multiple texts to supplement the dataset for model training.

[0047] Step 3: Pre-train the model using a general corpus, extracting long-distance text features at both the lexical and sentence levels for semantic representation, and directly applying this to downstream alarm recognition tasks. The long-distance text can be text with a distance of 10-480 characters. A multi-head attention mechanism is used to identify the dependence of alarm information classification results on different device conditions or analog values ​​in different scenarios. By exploring various fault coupling relationships, the model's ability to capture correct fault information is enhanced. By adding lexical positional information to the vectorized representation of the text, the alarm information text is represented jointly at the lexical, positional, and sentence dimensions. The model learns lexical-level information through a masked language task, replacing 15% of the lexical units in a sentence with the [MASK] label and using an unsupervised method to predict the word bidirectionally. It also learns sentence-level information through a sentence prediction task, where given a sentence pair (A, B), sentence B is either the next sentence of sentence A or a sentence randomly selected from the corpus. The model learns sentence-level knowledge by predicting the relationships between sentences. Sentence pairs are marked with [CLS] at the beginning, sentences are separated by [SEP], and blank word positions are filled with [PAD]. The loss function of the pre-trained model is shown in equations (1)-(3):

[0048]

[0049]

[0050] L(θ,θ1,θ2) = L1(θ,θ1)+L2(θ,θ2) (3)

[0051] In equations (1)-(3) above, L1 is the loss function of the masked language model, L2 is the loss function of the sentence prediction model, θ is the hyperparameter of the model encoder, θ1 is the output layer parameter of the masked language model encoder, and θ2 is the parameter of the classifier connected to the sentence prediction model encoder. The set of words for [MASK] masking is M, the corpus dictionary is F, and the sentence set is N. l and U i It represents the corresponding word and sentence.

[0052] Step 4: Fine-tune the alarm event recognition model based on the training sample library obtained in Step 2. After pre-training, for the alarm information classification task, a classification model suitable for text data needs to be designed after the pre-trained model, and supervised training is performed using the alarm information training set pre-processed above. During the pre-training process, the [CLS] marker at the beginning of the sentence has no initial information, and the corresponding output after training is not affected by the semantics of the word itself. Semantic representation is completed at the sentence level, which can be used as the feature vector of the alarm information as a whole for classification. The output corresponding to [CLS] is pooled, passed through a feedforward neural network, and then input into a Softmax model with the same dimension as the alarm information category to obtain the predicted alarm event classification result.

[0053] Multi-class classification tasks typically use cross-entropy as the loss function, as shown in Equation (4):

[0054]

[0055] N is the number of samples, q is the number of classes, c is the class index, and y is the number of classes. ic For the corresponding category, p ic Let be the probability that sample i belongs to category c.

[0056] In formula (4), p ic or (1-p) ic The value can be used to characterize the confidence level of the classifier in the result; the larger the value, the easier the sample is to be classified. Focal Loss introduces a scaling factor γ on the basis of cross-entropy to reduce the loss of easily classified samples. At the same time, it uses the inverse ratio of the sample distribution probability as the weight coefficient α, so that the contribution of a small number of samples to the model is increased. The formula of Focal Loss is shown in Equation (5):

[0057]

[0058] The introduction of weighting coefficient α and difficulty scaling factor γ significantly improves the accuracy of classes with a smaller proportion of the sample. By combining with data augmentation methods, more accurate identification and classification can be achieved on the more common problem of uneven sample distribution.

[0059] In some specific embodiments of the present invention, alarm data from a substation between January and December 2020 were selected as the data source for the implementation case, and the method described in the above embodiments was implemented. Through manual annotation by operation and maintenance experts, a total of 1161 alarm events were identified, containing 3308 alarm messages. Based on the alarm subject, the alarm events were classified into 22 fault types according to actual production categories, as shown in Table 1.

[0060] Table 1 Alarm Event Types

[0061]

[0062] Figure 2 A flowchart illustrates the process of cleaning alarm information from a real sample database. It shows that the alarm information has been processed and transformed into vectors containing key features before being input into the model. First, after extracting the original alarm information from the real sample database, initial text cleaning is performed based on mappings of special expressions and removal of symbols and spaces. Second, word segmentation is performed using a power industry dictionary and a Chinese dictionary. Third, mixed representations that do not belong to physical quantities and substation names are treated as noise words and removed. Next, the training dataset is expanded, with the selected augmented sample entries ranging from 1 to 5 depending on the alarm event type. Finally, an index dictionary is constructed based on the current word capacity, and the alarm text is vectorized using word index representations. Text vectors belonging to the same alarm event are used as input to the model.

[0063] To verify the model's performance, commonly used machine learning methods were used as comparative examples, including word2vec word vectors, LSTM, and Fast Text, which are classic text representation and feature extraction models. The comparison architecture settings are shown in Table 2.

[0064] Table 2 Comparative Experiment Setup

[0065]

[0066] Based on the recognition results of the test set, the reliability of the recognition model is evaluated using macroscopic accuracy, precision, recall, and F-1 score. The calculation formulas are shown in equations (6)-(8):

[0067]

[0068]

[0069]

[0070] TP, FP, and FN represent the positive samples predicted as positive, the negative samples predicted as positive, and the positive samples predicted as negative, respectively, and q represents the total number of sample categories.

[0071] The recognition accuracy and comparison results obtained using the method disclosed in this application are shown in Table 3.

[0072] Table 3 Comparison of Experimental Results

[0073]

[0074]

[0075] The evaluation results show that the pre-trained BERT model performs excellently on the alarm information classification task. Compared with the base model W2V-LSTM, BERT achieves a 9 percentage point improvement in accuracy and a 23.2% improvement in F1 score. Both data augmentation and Focal Loss have a positive effect on improving classification performance, but data augmentation is more significant. Introducing Focal Loss alone improves the model's accuracy by 2%, but decreases the F1 score. Introducing data augmentation, however, improves the model's accuracy and F1 score by 6% and 3%, respectively. This indicates that when the training set has a small number of samples, introducing more samples through data augmentation is more effective in improving the recognition performance than improving the performance of difficult-to-classify samples. The method in this application represents alarm text using a pre-trained model, effectively learning text features at different levels. When the number of alarm information samples is small and the distribution is imbalanced, the addition of data augmentation and improved loss function effectively improves the classification accuracy of the small proportion of samples compared to the base model, thereby improving the generalization ability of the entire model to distinguish the true alarm event categories. By learning from alarm message text, this method can maintain a classification accuracy of over 91% for all types of alarm events with a total classification accuracy of 96.09% even with limited training data. Figure 4 The confusion matrix of the proposed method in a real-world scenario is shown, demonstrating that the proposed method has good performance in the field of alarm event recognition.

[0076] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0077] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A substation alarm event identification method based on pre-training model and data enhancement, characterized in that, The method comprises the following steps: collecting real alarm event data in a transformer substation; performing data preprocessing on the unevenly distributed real alarm event data, sampling and text enhancement to obtain an alarm information training sample library; pre-training the model using a general corpus, extracting text features from the word and sentence levels to perform semantic representation, training the model to fit, and enabling direct application in alarm event recognition; based on the alarm information training sample library, introducing a Focal Loss loss function as the loss function of the classifier after completing the pre-trained model fitting, and completing alarm event recognition according to the dynamic scaling of the loss; the text enhancement includes synonym replacement, random insertion, random exchange and random deletion of the alarm information text; The text enhancement includes the following steps: in the alarm information dataset D train = {(U i , y i )}n i=1, the training dataset contains n alarm events, U and y are the text content and label of the alarm event respectively, and i is the sequence index corresponding to the alarm event; when the total number of alarm event categories is q, for any alarm information in a label y∈{1,…,q}, the following steps are selected for processing: Step 1) randomly select m non-stop word tokens from the alarm information, and randomly select one from their synonyms to replace; Step 2) randomly select a word token from the stop word library without actual semantics, and randomly insert it into a certain position in the alarm information, repeat m times; Step 3) randomly select two word tokens in the alarm information and exchange their positions, repeat m times; Step 4) randomly delete a word token in the alarm information with a probability p, repeat m times; introduce a hyperparameter β, combine the number of repetitions with the number of tokens by setting m=βl, l is the length of the token; in the random deletion process, p=β; select the stop word library as the random insertion word library, so that the inserted token contains less semantic itself and no additional semantic interference; randomly select some samples from the generated multiple texts and supplement them into the data set for model training.

2. The substation alarm event identification method based on pre-trained model and data augmentation according to claim 1, characterized in that, The real alarm event data includes alarm information and its corresponding fault type, occurrence time, alarm text and alarm event code in the actual operation process of the transformer substation.

3. The substation alarm event identification method based on pre-trained model and data augmentation according to claim 1, characterized in that, The model is a BERT model.

4. The substation alarm event identification method based on pre-trained model and data augmentation according to claim 3, characterized in that, In the BERT model, The dynamic representation of the meaning of the word is obtained by using the multi-head self-attention mechanism, the query and the key value are projected into multiple vector spaces, the text representation under different attention weights is obtained, and the model pays attention to the correct fault-equipment association; the corresponding input value is mapped to the output by the query and a group of key values, wherein Q, K i , V i represent the query, the key value and the value respectively, i is the corresponding index, d k is the dimension of the query vector; after obtaining multiple text representations, the multi-head attention combines and reduces the dimension of the results as the final representation of the word.

5. The substation alarm event identification method based on pre-trained model and data augmentation according to claim 3, characterized in that, In the training of the BERT model, position embedding is added in the text vector representation, and a trigonometric function is used as a function of the position embedding. According to the odd and even positions, the position of the alarm word in the alarm sentence is represented as pos represents the sentence to which the word belongs, i represents the index corresponding to the word, d model is the dimension of the word embedding.

6. The substation alarm event identification method based on pre-trained model and data augmentation of claim 3, wherein, the BERT model is pre-trained using the mask language and sentence prediction method.

7. The transformer substation alarm event recognition method based on pre-trained model and data enhancement according to claim 1, characterized in that, Focal Loss is used for alarm event recognition model fine-tuning, and its calculation expression is as follows: N is the number of samples, q is the number of classes, c is the class index, y ic is the corresponding class, p ic is the probability that sample i belongs to class c, the weight coefficient a is inversely proportional to the distribution probability of the sample, and g is the scaling factor.

8. A computer readable storage medium storing instructions, wherein, The instructions can implement the transformer substation alarm event recognition method of any one of claims 1-7 when executed.

Images