Refined fitting method and equipment for identifying transformer faults based on text mining

Abstract

The present invention discloses a method and device for identifying transformer faults by fine-tuning fitting based on text mining, and belongs to the technical field of transformer fault identification. The method for identifying transformer faults by fine-tuning fitting based on text mining of the present invention, during data preprocessing, segments the transformer fault text data, and performs data enhancement processing at the same time, which can effectively avoid semantic information loss on the one hand, and provide as many features as possible to the main model in a limited sample volume on the other hand, so that the fault description text with a small number of samples in the transformer fault type accounts for a larger proportion when the model calculates the fitting loss value, avoiding overfitting and enhancing the model recognition effect; linear dimensionality reduction processing is performed through the fully connected layer to obtain a global semantic vector; the Canopy+K-Means model is used to cluster all calculation results, identify the specific fault location and fault type of the transformer fault text data, and improve the overall semantic understanding ability, and the scheme is scientific, reasonable and feasible.

Description

Technical Field

[0001] The invention relates to a method and device for identifying transformer faults through refined fitting based on text mining, and belongs to the technical field of transformer fault identification. Background Art

[0002] Power transformers are one of the indispensable equipment for the normal operation of power systems. After a transformer fails, the inspectors record and judge the situation in text form according to the on-site situation to form a fault analysis report. Due to the uneven practical experience, knowledge reserves and professional capabilities of inspectors, not only are there different description forms for the same fault phenomenon, but it is often difficult to accurately judge the cause of the fault, resulting in misjudgment. Therefore, the method of relying on manual judgment is not efficient. With the development of natural language processing technology in the field of artificial intelligence, automatic fault identification based on power text information mining can be realized.

[0003] Existing equipment for power text information mining is mainly based on TextCNN, BiLSTM, BERT and traditional machine learning methods. The classification methods used in existing equipment mainly include logistic regression classifier, Bayesian classifier, support vector machine, etc. These devices can realize machine recognition and fault classification of short texts with a large number of samples and a single number of fault types. However, when recognizing and classifying long transformer fault texts with small samples and an uncertain number of fault types, overfitting is prone to occur, resulting in extremely low fault recognition accuracy.

[0004] Furthermore, existing equipment cannot provide patrol personnel with high-accuracy fault information in real time, and cannot meet the actual application requirements of power sites. Summary of the invention

[0005] In view of the defects of the prior art, the first purpose of the present invention is to provide a method for segmenting transformer fault text data during data preprocessing and performing data enhancement processing at the same time, which can effectively avoid the loss of semantic information on the one hand, and provide as many features as possible to the main model in a limited sample size on the other hand; then the enhanced text data is vectorized; then the initial parameters of text feature extraction are set using the language representation model BERT pre-training unit; all calculation results are clustered using the Canopy+K-Means model to identify the specific fault location and fault type of the transformer fault text data, thereby effectively improving the quality of semantic feature extraction and enhancing the overall semantic understanding ability. The scheme is scientific and reasonable, and a practical method for identifying transformer faults based on text mining and refined fitting is proposed.

[0006] In view of the defects of the prior art, the second purpose of the present invention is to provide a device that can provide real-time feedback of high-accuracy fault information to inspection personnel, and can realize real-time processing and analysis of transformer fault cases. It has a high-accuracy recognition effect on a small number of long texts of fault descriptions containing multiple fault types, alleviates personnel requirements, and greatly reduces labor costs. It can identify transformer faults through refined fitting based on text mining.

[0007] To achieve one of the above purposes, the first technical solution of the present invention is:

[0008] A refined fitting method for identifying transformer faults based on text mining.

[0009] The following steps are involved:

[0010] Step 1: Segment the sentences with excessive words in the transformer fault text data and perform enhancement processing to obtain enhanced text data, which is used to provide as many features as possible to the main model within a limited sample size;

[0011] Step 2: Perform initial vectorization on the enhanced file data in the first step to obtain an initial vectorized text matrix, which is used to convert it into a vector form that can be understood and calculated by the computer.

[0012] Step 3: Encode the text matrix initially vectorized in step 2 using residual connection and layer normalization to obtain a sentence vector;

[0013] Step 4: Build the language representation model BERT to perform linear dimensionality reduction on the sentence vector in the third step to obtain the probabilities of various fault labels; and dynamically adjust the recognition loss value of the language representation model BERT by adding weights;

[0014] Step 5: Perform cluster analysis on the probabilities of various fault labels in step 4 using the Canopy+K-Means model;

[0015] Extract the type label corresponding to the transformer fault text data;

[0016] Step 6: Based on the type label in step 5, identify the specific fault location and fault type described by the transformer fault text data.

[0017] After continuous exploration and experimentation, the present invention segments the transformer fault text data during data preprocessing and performs data enhancement processing at the same time. On the one hand, it can effectively avoid the loss of semantic information, and on the other hand, it can provide as many features as possible to the main model in a limited sample size; then the processed enhanced text data is input and vectorized; then the language representation model BERT pre-training unit is used to set the initial parameters of text feature extraction; the Canopy+K-Means model is used to cluster all the calculation results to identify the specific fault location and fault type of the transformer fault text data, thereby effectively improving the quality of semantic feature extraction and enhancing the overall semantic understanding ability. The scheme is scientific, reasonable and feasible.

[0018] Furthermore, the present invention provides a method for identifying transformer faults through refined fitting based on text mining, which improves the recognition accuracy of long texts of transformer faults including multiple fault types and meets the needs of actual on-site applications; and can realize efficient processing and analysis of transformer fault cases, and has a high-accuracy recognition effect on a small number of long texts of fault descriptions containing multiple fault types, thereby alleviating personnel requirements and greatly reducing labor costs.

[0019] As the preferred technical measures:

[0020] In the first step, the transformer fault text data includes a fault description text and a fault type, which are regularly filled in the same table column;

[0021] Since information will be lost when the fault description text exceeds the maximum text length, in the description text of the same fault type, when the sentence text length exceeds the maximum text length, the original transformer fault text is segmented with punctuation marks, which includes the following contents;

[0022] The maximum length of each sentence text is L characters, and the minimum length of each sentence text is M;

[0023] When the sentence length is less than M, several sentences are concatenated to meet the minimum character length;

[0024] When the sentence length is in the interval [M, L], the sentence is not concatenated;

[0025] The concatenated sentences are sorted by length, and sentences of similar length are put into the same training batch to avoid the difference in sentence length affecting the training time of the same batch.

[0026] At the same time, the fault description text provides more feature vectors to the main model after data enhancement processing.

[0027] As the preferred technical measures:

[0028] In the second step, vectorization processing includes the following:

[0029] The single character in the transformer fault text data is taken as the smallest unit token, and then the initial text vector matrix Xe,0 is obtained through several embedding layers;

[0030] The plurality of embedding layers include at least a tag embedding layer, a fragment embedding layer, and a position embedding layer.

[0031] As the preferred technical measures:

[0032] The markup embedding layer includes the following:

[0033] The transformer mechanism is used to randomly initialize the single characters in the transformer fault text data, convert each character into a fixed-dimensional vector, and then perform token-id conversion. The corresponding vector representation is obtained through id mapping, and then vectors [CLS] and [SEP] are added at the beginning and end of each text respectively.

[0034] The fragment embedding layer consists of the following:

[0035] Use vector [SEP] to classify whether two texts are semantically similar, represented by at least two vectors. The index number of the first sentence vector is A, and the indexes of all tags belonging to this sentence are marked as A; the index number of the second sentence vector is B, and the indexes of all tags belonging to this sentence are marked as B;

[0036] The vector of the fragment embedding layer is obtained by comparing the index number with the fragment embedding table;

[0037] The position embedding layer is a vector comparison lookup table, which includes the following contents:

[0038] When the number of characters in a text is less than the maximum text length, it needs to be padded. The padded position is meaningless and does not participate in subsequent calculations.

[0039] Initial text vector matrix X e,0 It is obtained by summing the vector representations of several embedding layers and passed to the language representation model BERT.

[0040] As the preferred technical measures:

[0041] In the third step, the sentence vector is processed in parallel through 12 coding layers, and the vector y is obtained by identifying the vector [CLS] at the beginning of the sentence. [CLS] ; Residual connections and layer normalization within each encoding layer are used to solve the problem of feature extraction saturation;

[0042] The process of obtaining the sentence vector is as follows;

[0043] The text matrix is ​​X e，0 , which is obtained by comparing it with the weight matrix W of the pre-trained model Q , W K , W V Multiply to get the query vector matrix Q, key vector matrix K and value vector matrix V;

[0044] Multiply Q and K to get the weight table of each word and other words;

[0045] In order to prevent the score from increasing as the vector dimension increases, divide the product of Q and K by Then the logistic regression model Softmax is used for normalization. After normalization, it is multiplied with V to give weights to each character. Finally, the multiplied result vectors are added to obtain a new sentence vector. The specific calculation formula is as follows:

[0046] X e，i ×W Q =Q

[0047] X e，i ×W K =K

[0048] X e，i ×W V =V

[0049]

[0050] Q.shape = [m, d k ]

[0051] K T .shape=[d k , m]

[0052] V.shape = [m, d v ]

[0053] A=QK T , shape = [m, m]

[0054]

[0055] B=softmax(A, dim=0), shape=[m, m]

[0056] out = B*V, shape = [m, d v ]

[0057] Where d is the vector dimension;

[0058] d Q，sh is the vector dimension corresponding to each character in the query vector matrix Q;

[0059] d K，sh is the vector dimension corresponding to each character in the key vector matrix K;

[0060] d V，sh The vector dimension corresponding to each character in the value vector matrix V matrix;

[0061] m is the total number of words in each text;

[0062] The above calculation formula is the way that the single-head attention mechanism obtains the global semantic fusion vector of a single character. The vector dimension corresponding to each character in the Q, K, V matrix is ​​equal to the vector dimension d under the multi-head attention mechanism. mh Divided by the number of self-attention mechanisms n, n is 12 in the model of the present invention, d mh is 768, and the calculation formula is as follows:

[0063]

[0064] Among them, sh stands for single-head abbreviation, and mh stands for multi-head abbreviation;

[0065] The multi-head attention calculation method is used to fuse the key semantic information and obtain a new sentence vector. Then, the new sentence vector y is transformed into [CLS] For subsequent work;

[0066]

[0067] MultiHead(Q,K,V)=Concat(head1,...,headh)W 0 .

[0068] As the preferred technical measures:

[0069] In the fourth step, the dimension of the sentence vector y[CLS] finally output by the encoding layer of the language representation model BERT is d mh , which is 768-dimensional and contains only semantic vector information. It is then processed by linear dimensionality reduction in the fully connected layer of the language representation model BERT to obtain the probability of various fault labels;

[0070] Use the logistic regression function Softmax to perform probability normalization on y[CLS] to obtain a one-dimensional vector y out , the value corresponding to this vector represents the probability of the corresponding fault type label;

[0071] The label corresponding to the maximum probability value is selected as the predicted transformer fault type, and its calculation formula is as follows:

[0072] y out=Softmax(W out ·y [CLS] +b out )

[0073] Where W out and b out is the weight matrix and bias vector of the fully connected layer.

[0074] The method for dynamically adjusting the language representation model BERT recognition loss value is as follows:

[0075] The number of iterations is one of the hyperparameters that mainly determines the performance of the model. If the number of iterations is too large, the model will be underfitted and the model training speed will be very slow. If the number of iterations is too small, the model will be overfitted. Due to the small number of transformer fault texts, the initial number of iterations is set to 50. When the model accuracy cannot be significantly improved in consecutive batches, the number of iterations is tried in sequence of 40, 30, 20, 10, and 5. During the debugging process, the model has the best fitting effect when the number of iterations is 10.

[0076] However, transformer faults are diverse, including common faults and occasional faults, so the number of transformer fault instances is unevenly distributed. The feature vector that can be provided by each type of transformer fault text is proportional to the number of texts of this fault type. If the cross entropy method commonly used in the field of power systems is directly used to calculate the loss value, that is, the weight of each fault text is the same, then overfitting is likely to occur when identifying fault types with a small number of samples, and the recognition effect is not ideal. In order to further improve the model fitting effect, the present invention has made adjustments to the loss value calculation method.

[0077] Weights are added on the basis of the cross entropy loss function to give higher weights to categories with less data, so that the network model pays attention to small sample categories and performs fine fitting on all types of fault description texts. The calculation formula is as follows:

[0078]

[0079] Where L j represents the model loss value of the jth training batch, N represents the number of transformer fault texts in a training batch; p i Represents the recognition accuracy of the i-th fault sample label in a training batch.

[0080] Through debugging, the device of the present invention has the best effect when N=2, β=2; by adding weights to dynamically adjust the recognition loss value of the model, the fault description text with a small number of samples in the transformer fault type accounts for a larger proportion when the model calculates the fitting loss value, avoiding overfitting and enhancing the model recognition effect.

[0081] As the preferred technical measures:

[0082] In the fifth step, the Canopy+K-Means model includes a Canopy unit and a K-Means unit;

[0083] The Canopy unit is used to calculate the label probability value y out Perform preliminary clustering to obtain K value and K initial cluster centers; the K-Means unit is used to calculate the label probability value y out Perform fine cluster analysis;

[0084] The cluster analysis method is as follows:

[0085] All label probability values ​​y out It is stored in the computer in the form of a set list, and a distance threshold L is selected. L can be calculated by the following formula:

[0086]

[0087] Where D represents the Euclidean distance operation of the point, and N represents the total number of transformer fault texts;

[0088] From the label probability value y out Randomly select a point P1(n1, n2) from the set and regard it as the first type of fault. Continue to randomly select points P2, P3, P4, ... P from the set. N , calculate the distance from the newly taken point to all points belonging to the first type of fault type. If the distance is less than L, delete the point from the set list. If the distance is greater than L, extract the point from the set list as a new fault type. Repeat the calculation until the points in the set are empty, the preliminary clustering is completed, and the K value and K initial clustering centers are obtained. The present invention calculates K to be 15.

[0089] Then, perform K-means unit operation, calculate the distances from all points to the K initial cluster centers, classify each point into a category with the initial cluster center that is closest to it, and then calculate the average value of each cluster as the new cluster center. Repeat the iteration until the position of the cluster center no longer changes, and finally extract the type label corresponding to the transformer fault text data.

[0090] As the preferred technical measures:

[0091] It also includes the performance evaluation of transformer fault identification equipment, which specifically includes the following:

[0092] The transformer fault text dataset is randomly divided into a training set and a test set in a ratio of 8:2. The training set is used for equipment training, and the test set is used for equipment performance evaluation.

[0093] The performance evaluation index is the F1 value, and the calculation formula of the F1 value is as follows:

[0094]

[0095]

[0096]

[0097] To achieve one of the above purposes, the second technical solution of the present invention is:

[0098] Refined fitting based on text mining to identify transformer faulty equipment,

[0099] Apply the above-mentioned text mining-based refined fitting method to identify transformer faults;

[0100] It includes a keyboard, a display screen, a processor, an input transformer fault text unit, a main storage unit, an external storage unit, a power supply module, a refined fitting transformer fault identification module based on text mining, and an output transformer fault location and type unit;

[0101] The refined fitting transformer fault identification module based on text mining includes a text vectorization unit, a language representation model BERT encoding unit, a transformer fault label probability calculation unit, and a Canopy+K-Means clustering analysis unit, all of which are encapsulated and stored in the main storage unit;

[0102] The inspection personnel input the transformer fault text data into the transformer fault text unit via the keyboard, or read the transformer fault text data from the external storage unit;

[0103] The main storage unit stores and reads the transformer fault text data transmitted to the device and transmits it to the refined fitting transformer fault identification module based on text mining for processing and analysis. The result data analyzed by the refined fitting transformer fault identification module is then returned to the main storage unit. Finally, the identification result is output through the output transformer fault location and type unit.

[0104] After continuous exploration and experiments, the present invention is provided with a keyboard, a display screen, a processor, an input transformer fault text unit, a main storage unit, an external storage unit, a power supply module, a refined fitting identification module for transformer faults based on text mining, and an output transformer fault location and type unit. It can provide real-time feedback of high-accuracy fault information to inspection personnel, and can realize real-time processing and analysis of transformer fault cases. It has a high-accuracy recognition effect on a small number of long texts of fault descriptions containing multiple fault types, alleviates personnel requirements, and greatly reduces labor costs.

[0105] To achieve one of the above purposes, the third technical solution of the present invention is:

[0106] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for identifying transformer faults through refined fitting based on text mining.

[0107] Compared with the prior art, the present invention has the following beneficial effects:

[0108] After continuous exploration and experimentation, the present invention segments the transformer fault text data during data preprocessing and performs data enhancement processing at the same time. On the one hand, it can effectively avoid the loss of semantic information, and on the other hand, it can provide as many features as possible to the main model in a limited sample size; then the processed enhanced text data is input and vectorized; then the language representation model BERT pre-training unit is used to set the initial parameters for text feature extraction, and then the language representation model BERT encoding layer is entered. By continuously iterating the text data, the key weight parameters of the language representation model BERT model can be improved, and the recognition loss values ​​of texts of different fault types can be dynamically adjusted. Weights can be used to obtain the optimal global feature vector, so that the fault description texts with a small number of samples in the transformer fault type account for a larger proportion when the model calculates the fitting loss value, avoiding overfitting and enhancing the model recognition effect; then the vector is input into the fully connected layer for linear dimensionality reduction to obtain the global semantic vector; finally, the global semantic vector is normalized to obtain the label probability value of each fault text, and the Canopy+K-Means model is used to cluster all the calculation results to identify the specific fault location and fault type of the transformer fault text data, thereby effectively improving the quality of semantic feature extraction and enhancing the overall semantic understanding ability. The solution is scientific, reasonable and feasible.

[0109] Furthermore, after continuous exploration and experimentation, the present invention is provided with a keyboard, a display screen, a processor, an input transformer fault text unit, a main storage unit, an external storage unit, a power supply module, a refined fitting identification module for transformer faults based on text mining, and an output transformer fault location and type unit. It can provide real-time feedback of high-accuracy fault information to inspection personnel, and can realize real-time processing and analysis of transformer fault cases. It has a high-accuracy recognition effect on a small number of long texts of fault descriptions containing multiple fault types, which alleviates personnel requirements and greatly reduces labor costs.

[0110] Furthermore, the present invention provides a transformer fault identification device suitable for long text, multiple fault types, and small samples for the transformer fault identification problem, which has the following advantages:

[0111] 1) The calculation method of the dynamic adjustment model loss value proposed in the present invention avoids the overfitting problem caused by the small amount of data, and has a better fitting effect than the traditional cross entropy loss value calculation method. By increasing the loss value weight of small sample data and reducing the loss value weight of data with a large sample size and easy to identify, N is taken as 2 in the loss function calculation formula, and the best clustering effect is achieved when it is taken as 2, which realizes the refined fitting of the training samples and effectively improves the recognition effect of the equipment on transformer fault text with uneven data distribution and multiple fault types.

[0112] 2) The device provided by the present invention realizes high-accuracy machine recognition of long text containing fault descriptions of multiple fault types. Compared with the currently commonly used TextCNN, BiLSTM, single BERT and traditional machine learning methods, its F1 value can be increased by 4% to 50%, which greatly improves the efficiency of transformer fault identification and reduces labor costs.

[0113] 3) The device of the present invention uses Canopy+K-Means to perform clustering processing on the calculation results of the probability of transformer fault type labels. When K is 15, the best clustering effect is achieved, which can avoid the output of irrelevant recognition results.

[0114] 4) The device of the present invention utilizes transformer fault text to continuously and iteratively improve key parameters such as the query vector matrix, key vector matrix, and value vector matrix in the Chinese BERT pre-training model, which can improve the clustering task effect downstream of the model.

[0115] 5) The device of the present invention uses segmentation and splicing sentences to perform data enhancement processing on the transformer fault text, so that the main model can learn more feature vectors in a limited amount of data.

[0116] 6) The device provided by the present invention can provide real-time feedback of high-accuracy fault identification information to inspection personnel, meeting the practical application requirements of power sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 A training and evaluation flow chart of the refined fitting method for identifying transformer faults of the present invention;

[0118] Figure 2 Generate X for the single transformer fault description text of the present invention e,0 Flowchart of vector matrix;

[0119] Figure 3 is a diagram of a coding layer of BERT of the present invention;

[0120] Figure 4 This is a flow chart of the Canopy+K-Means model clustering analysis of the present invention;

[0121] Figure 5A hardware configuration diagram of a device for fine-tuning and identifying transformer faults according to the present invention;

[0122] Figure 6 This is a block diagram of the transformer fault identification module based on refined fitting of text mining in the present invention. DETAILED DESCRIPTION

[0123] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0124] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0125] A specific embodiment of the method for identifying transformer faults by refined fitting based on text mining in the present invention:

[0126] A refined fitting method for identifying transformer faults based on text mining.

[0127] The following steps are involved:

[0128] Step 1: Segment the sentences with excessive word count in the existing transformer fault text data and perform enhancement processing to obtain enhanced text data, which is used to provide as many features as possible to the main model within a limited sample size;

[0129] Step 2: Perform initial vectorization on the enhanced file data in the first step to obtain an initial vectorized text matrix, which is used to convert it into a vector form that can be understood and calculated by the computer.

[0130] Step 3: Encode the text matrix initially vectorized in step 2 using residual connection and layer normalization to obtain a sentence vector;

[0131] Step 4: Build the language representation model BERT to perform linear dimensionality reduction on the sentence vector in the third step to obtain the probabilities of various fault labels; and dynamically adjust the recognition loss value of the language representation model BERT by adding weights;

[0132] Step 5: Perform cluster analysis on the probabilities of various fault labels in step 4 using the Canopy+K-Means model;

[0133] The Canopy+K-Means model includes Canopy units and K-Means units;

[0134] First, use the Canopy unit to perform preliminary clustering on the label probability value yout to obtain the K value and K initial clustering centers; then use the K-Means unit to perform fine clustering analysis on the label probability value yout;

[0135] Finally, the type label corresponding to the transformer fault text data is extracted;

[0136] Step 6: Based on the type label in step 5, identify the specific fault location and fault type described by the transformer fault text data.

[0137] like Figure 1-Figure 4 As shown, a best specific embodiment of the method for identifying transformer faults by refined fitting based on text mining of the present invention is as follows:

[0138] The present invention combines the BERT model and the Canopy+K-Means algorithm to propose a transformer fault refined fitting identification method including multiple fault types.

[0139] A refined fitting method for identifying transformer faults based on text mining includes the following contents:

[0140] First, the original transformer fault analysis report is input to automatically extract the fault phenomenon description and fault cause in the transformer fault report; during data preprocessing, the transformer fault text is enhanced to provide the main model with as many features as possible within a limited sample size; then the processed text data is input and the data is vectorized; the BERT pre-trained model is used to set the initial parameters for text feature extraction, and then the BERT encoding layer is entered. By continuously iterating the text data, the key weight parameters of the BERT model can be improved, and the recognition loss value weights of texts of different fault types can be dynamically adjusted to obtain the optimal global feature vector; the vector is then input into the fully connected layer for linear dimensionality reduction to obtain a global semantic vector; finally, the global semantic vector is normalized to obtain the label probability value of each fault text, and Canopy+K-Means is used to cluster all the calculation results to identify the specific fault location and fault type of the transformer fault description text.

[0141] Furthermore, the refined fitting method for identifying transformer faults based on text mining specifically includes the following steps:

[0142] Step 1: Enhance the existing transformer fault text data

[0143] In the existing transformer fault report, the fault description text and fault type are regularly filled in the same table column. The maximum text length that BERT can process is 512 characters. When the input fault description text exceeds the maximum text length, information loss will occur. Therefore, it is set that in the description text of the same fault type, when the sentence text length exceeds the maximum text length, the original transformer fault text is segmented by punctuation marks. It is stipulated that the maximum length of each sentence is 150 characters, and the minimum character length is 12. When the sentence length is less than 12, two sentences or three sentences need to be spliced ​​to meet the minimum character length. When the sentence length is in the interval [12, 150], the sentence is not spliced. The spliced ​​sentences are sorted according to the length, and sentences of similar length are put into the same training batch to avoid the influence of different sentence lengths on the training time of the same batch. After data enhancement, the transformer fault text provides more feature vectors to the main model.

[0144] Step 2: Initial vectorization of transformer fault text

[0145] The present invention inputs the above-mentioned transformer fault text data after data enhancement processing, firstly performs vectorization processing on the transformer fault text data, and converts it into a vector form that can be understood and calculated by the computer. The single character in the transformer fault text data is used as the minimum unit token, and then the initial text vector matrix X is obtained through three embedding layers. e,0. The first is the Token embedding layer, which uses the random initialization of the transformer mechanism to convert each character into a fixed-dimensional vector, and then performs a token-id conversion. After the id mapping, the corresponding vector representation is obtained, and then [CLS] and [SEP] vectors are added at the beginning and end of each text respectively. This layer can solve the problem of polysemy. In BERT, each character is represented as a 768-dimensional vector. The second is the Segment embedding layer, whose purpose is to use [SEP] to classify whether two texts are semantically similar. It is represented by only two vectors. The index number of the first sentence vector is 0, and the indexes of all tokens belonging to this sentence are marked as 0. On the contrary, the index number of the last sentence vector is 1, and the indexes of all tokens belonging to this sentence are marked as 1. The segment embedded vector is obtained by comparing the index number with the segment embedding table. The third one is the Position embedding layer. In BERT, the Position embedding layer is used to extract the sequential features of the text sequence. The maximum text length that BERT can process is 512. When the number of words in the text is less than the maximum text length, it needs to be padded. The padded position is meaningless and does not participate in subsequent calculations. This embedding layer is equivalent to a vector comparison query table of size (512, 768). For example, when inputting "transformer" and "main transformer", since "transformer" and "main" are both the first words in the input sequence, they have the same Position embedding. Finally, the vector representations of the three embedding layers are summed to obtain the initial text vector matrix X e，0 , which is also the input representation passed to the encoding layer of the BERT pre-trained model.

[0146] Step 3: Encode the initial vectorized text matrix

[0147] The present invention completes the initial text vector matrix X of vector expression e，0 As the input of the BERT pre-trained model, the word vector is encoded in parallel through 12 encoding layers, and y is obtained by identifying the [CLS] vector at the beginning of the sentence. [CLS] Each encoding layer uses residual connection processing, and obtains a new vector matrix X through summation, layer normalization, and feedforward neural network. e，1 , used to solve the problem of model feature extraction saturation, see Figure 3 The specific calculation process is described as follows.

[0148] The transformer fault description text is initially vectorized to obtain the word embedding matrix X e，0 , through the weight matrix W of the pre-trained model Q , W K , W VMultiply the query vector matrix Q, the key vector matrix K and the value vector matrix V. Multiply Q and K to get the weight table of each word and other words. In order to prevent the score from increasing with the increase of vector dimension, divide the product of Q and K by Then use Softmax to normalize by row, multiply it with V after normalization to assign weights to each character, and finally add the multiplication result vectors to get the new sentence vector.

[0149] X e，i ×W Q =Q

[0150] X e，i ×W K =K

[0151] X e，i ×W V =V

[0152]

[0153] Q.shape = [m, d k ]

[0154] K T .shape=[d k , m]

[0155] V.shape = [m, d v ]

[0156] A=QK T , shape = [m, m]

[0157]

[0158] B=softmax(A, dim=0), shape=[m, m]

[0159] out = B*V, shape = [m, d v ]

[0160] The above calculation formula is the way to obtain the global semantic fusion vector of a single character by a single-head attention mechanism. The vector dimension d corresponding to each character in the Q, K, V matrices is equal to the vector dimension under the multi-head attention mechanism divided by the number of self-attention mechanisms n. In the model of the present invention, n is 12, and d mh is 768, m is the total number of words in each text, sh represents the abbreviation of single-head, and mh represents the abbreviation of multi-head.

[0161]

[0162] The present invention uses a multi-head attention calculation method to fuse key semantic information and obtain a new sentence vector, and then identifies the global semantic vector [CLS] at the beginning of the sentence to convert the new sentence vector y [CLS] For subsequent work.

[0163]

[0164] MultiHead(Q,K,V)=Concat(head1,...,hcad h )W 0

[0165] Step 4: Transformer fault type label probability calculation

[0166] The sentence vector y finally output by the BERT encoding layer in the present invention is [CLS] The dimension is d mh , is 768-dimensional and contains only semantic vector information. It also needs to be processed by the fully connected layer in BERT for linear dimensionality reduction to obtain the probability of various fault labels. [CLS] Perform probability normalization to obtain a one-dimensional vector y out , the value corresponding to the vector represents the probability of the corresponding fault type label. The present invention selects the label corresponding to the maximum probability value as the predicted transformer fault type. The calculation formula is as follows:

[0167] y out =Softmax(W out ·y [CLS] +b out )

[0168] Where W out and b out is the weight matrix and bias vector of the fully connected layer.

[0169] Step 5: Dynamically adjust the model recognition loss value

[0170] The number of iterations is one of the hyperparameters that mainly determines the performance of the model. If the number of iterations is too large, the model will be underfitted and the model training speed will be very slow. If the number of iterations is too small, the model will be overfitted. Due to the small number of transformer fault texts, the present invention sets the initial number of iterations to 50. When the model accuracy cannot be significantly improved in consecutive batches, the number of iterations is tried in sequence of 40, 30, 20, 10, and 5. During the debugging process, the model has the best fitting effect when the number of iterations is 10.

[0171] However, transformer faults are diverse, including common faults and occasional faults, so the number of transformer fault instances is unevenly distributed. The feature vector that can be provided by each type of transformer fault text is proportional to the number of texts of this fault type. If the cross entropy method commonly used in the field of power systems is directly used to calculate the loss value, that is, the weight of each fault text is the same, then when identifying fault types with a small number of samples, overfitting is likely to occur, and the recognition effect is not ideal. In order to further improve the model fitting effect, the present invention has made adjustments to the loss value calculation method. Weights are added on the basis of the cross entropy loss function, and higher weights are given to categories with less data, so that the network model pays attention to small sample categories. In this way, all types of fault description texts can be fine-tuned for fitting, and the calculation formula is as follows:

[0172]

[0173] Where L j represents the model loss value of the jth training batch, N represents the number of transformer fault texts in a training batch; p i Represents the recognition accuracy of the i-th fault sample label in a training batch.

[0174] Through debugging, the device of the present invention has the best effect when N = 2, β = 2. By adding weights to dynamically adjust the recognition loss value of the model, the fault description text with a small number of samples in the transformer fault type accounts for a larger proportion when the model calculates the fitting loss value, avoiding overfitting and enhancing the model recognition effect.

[0175] Step 6: Canopy+K-Means cluster analysis

[0176] The probability of each type of transformer fault description text corresponding to the type label is similar. The present invention takes advantage of the speed of the Canopy algorithm and first out The label probability values ​​are preliminarily clustered to obtain the K value and K initial cluster centers. Then the K-Means algorithm is used to cluster y out The label probability values ​​are subjected to fine cluster analysis. The specific operation is as follows.

[0177] All y out The label probability value is stored in the computer in the form of a list set. A distance threshold L is selected, and L can be calculated by the following formula:

[0178]

[0179] Where D represents the Euclidean distance operation of the point, and N represents the total number of transformer fault texts.

[0180] From y outA point P1(n1, n2) is randomly selected from the set of label probability values ​​and regarded as the first type of fault. Then, points P2, P3, P4, ... P are randomly selected from the set. N , calculate the distance from the newly taken point to all points belonging to the first type of fault type. If the distance is less than L, delete the point from the list set. If the distance is greater than L, extract the point from the list set as a new fault type. Repeat the calculation until the points in the set are empty, the preliminary clustering is completed, and the K value and K initial cluster centers are obtained. The present invention calculates K to be 15.

[0181] Then, a K-means operation is performed to calculate the distances from all points to the K initial cluster centers. Each point is classified into one category with the initial cluster center that is closest to it. The average value of each cluster is calculated as the new cluster center. The operation is repeated until the position of the cluster center no longer changes. Finally, the type label corresponding to the transformer fault description text is extracted.

[0182] Step 7: Transformer Fault Identification Equipment Performance Evaluation

[0183] The transformer fault text data set is randomly divided into a training set and a test set in a ratio of 8:2. The training set is used for equipment training, and the test set is used for equipment performance evaluation. The present invention uses the F1 value as the performance evaluation index of the equipment, and the F1 value of the equipment can be calculated using the following formula.

[0184]

[0185]

[0186]

[0187] like Figure 5 As shown, a specific embodiment of the present invention for identifying transformer fault equipment by refined fitting based on text mining is as follows:

[0188] Based on the above method, the present invention forms a set of corresponding transformer fault equipment, the hardware configuration of which includes a keyboard, a 9.7-inch 1080P display screen, a Zynq UltraScale+MPsoc processor, an input transformer fault text unit, a main storage unit, an external storage unit, a power supply module, a refined fitting transformer fault identification module based on text mining, and an output transformer fault location and type unit, and a display device. The refined fitting transformer fault identification module based on text mining includes a text vectorization unit, a BERT encoding unit, a transformer fault label probability calculation unit, and a Canopy+K-Means clustering analysis unit, all of which are encapsulated and stored in the main storage unit.

[0189] The overall operation process of the equipment is that the inspection personnel input the transformer fault text unit through the keyboard or read the transformer fault text data into the equipment through the external storage unit. The main storage unit stores the text data transmitted into the equipment and reads it to the refined fitting transformer fault identification module based on text mining for processing and analysis. The result data analyzed by the module is then returned to the main storage unit. Finally, the identification result is output through the output transformer fault location and type unit and displayed on the display device.

[0190] like Figure 6 As shown, a specific embodiment of the transformer fault identification module by fine fitting of the present invention is as follows:

[0191] The text mining-based refined fitting transformer fault identification module includes a transformer fault text initial vectorization unit, a transformer fault text vector matrix encoding unit, a transformer fault type label probability calculation unit, and a Canopy+K-Means clustering analysis unit, all of which are encapsulated and stored in the main storage unit.

[0192] The refined fitting transformer fault identification device based on text mining provided by the present invention is an identification device for transformer fault text features, which effectively considers the characteristics of long fault text, multiple fault types, multiple ambiguities, and difficult segmentation, and can solve the problem of loss of key information such as word order and position information caused by the long distance between characters, and distinguish the specific fault type. Finally, the transformer fault identification device can obtain a very high F1 value. In the actual working environment of the power site, the relevant power personnel may misjudge the type of transformer fault due to lack of experience. The device provided by the present invention has important reference significance for the on-site personnel to handle the same type of transformer faults and realize the prediction of the transformer fault type. For details, please refer to Table 1.

[0193] Table 1

[0194]

[0195]

[0196]

[0197] A computer medium embodiment using the method of the present invention:

[0198] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for identifying transformer faults through refined fitting based on text mining.

[0199] Explanation of terms in this application:

[0200] Machine Learning

[0201] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0202] Electricity Text

[0203] A large amount of text data is generated in power operation rules, operation and maintenance records, dispatch operation records, marketing files, customer service, and fault repair.

[0204] Language representation model BERT

[0205] The full name is Bidirectional Encoder Representation from Transformers, which refers to the bidirectional encoding representation based on transformer. It is a pre-training model proposed by Google AI Research Institute in October 2018.

[0206] LSTM

[0207] The full name is Long short term memory, which refers to the long short-term memory artificial neural network. It is a neural network specially designed to solve the long-term dependency problem existing in the general Recurrent Neural Network.

[0208] BiLSTM

[0209] BiLSTM stands for Bidirectional Long Short Term Memory, which refers to a bidirectional long short-term memory artificial neural network with an attention mechanism. It is suitable for scenarios with higher complexity and can better capture long-term dependencies in text.

[0210] TextCNN

[0211] The full name is Text Convolutional Neural Networks, which refers to text convolutional neural networks. It is a convolutional neural network proposed by Yoon Kim in 2014 for text classification.

[0212] Overfitting

[0213] Refers to making assumptions overly strict in order to obtain consistent assumptions.

[0214] Logistic Regression Classifier

[0215] The process of fitting some existing data points with a straight line is called regression. The main idea of ​​the logistic regression classifier is to establish a regression formula for the classification boundary based on the existing data and classify it accordingly.

[0216] Bayesian Classifier

[0217] The posterior probability of an object is calculated using the Bayesian formula, that is, the probability that the object belongs to a certain class, and the class with the largest posterior probability is selected as the class to which the object belongs.

[0218] Support Vector Machine

[0219] Support vector machine is a binary classification model. Its purpose is to find a hyperplane to segment samples. The principle of segmentation is to maximize the interval, which is ultimately converted into a convex quadratic programming problem to solve.

[0220] Text vectorization

[0221] Representing text as a series of vectors that can express the semantics of the text is an important way to represent text.

[0222] Pre-trained models

[0223] A set of weight values ​​that yields relatively good results at the end of training a task model can be widely used in other models as basic parameter values.

[0224] [CLS] and [SEP]

[0225] [CLS] and [SEP] stand for classification and separator, respectively. They are vectors that serve as markers in the BERT language representation model. [CLS] is placed at the beginning of a sentence, and [SEP] is placed at the end of a sentence.

[0226] coding

[0227] Convert natural language sequences into mathematical expressions.

[0228] Clustering

[0229] The process of grouping a collection of physical or abstract objects into clusters of similar objects is called clustering.

[0230] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0231] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A refined fitting method for identifying transformer faults based on text mining, It is characterized in that The following steps are involved: Step 1: Segment the sentences with excessive words in the transformer fault text data and perform enhancement processing to obtain enhanced text data; Step 2: Perform initial vectorization on the enhanced file data in the first step to obtain the initial vectorized text matrix. Step 3: Encode the text matrix initially vectorized in step 2 using residual connection and layer normalization to obtain a sentence vector; The sentence vector is processed in parallel through 12 encoding layers, and the vector y is obtained by identifying the vector [CLS] at the beginning of the sentence. [CLS] ; Residual connections and layer normalization within each encoding layer are used to solve the problem of feature extraction saturation; The process of obtaining the sentence vector is as follows; The text matrix is ​​X e,0 , which is obtained by comparing it with the weight matrix W of the pre-trained model Q , W K , W V Multiply to get the query vector matrix Q, key vector matrix K and value vector matrix V; Multiply Q and K to get the weight table of each word and other words; Divide the product of Q and K by Then the logistic regression model Softmax is used for normalization. After normalization, it is multiplied by V to give weights to each character. Finally, the multiplied result vectors are added to obtain a new sentence vector. The specific calculation formula is as follows: X e，i ×W Q =Q X e，i ×W K =K X e，i ×W V =V Q.shape=[m,d k ] K T .shape=[d k ,m] V.shape=[m,d v ] A=QK T ,shape=[m,m] B=softmax(A,dim=0),shape=[m,m] out=B*V,shape=[m,d v ] Where d is the vector dimension; d Q,sh is the vector dimension corresponding to each character in the query vector matrix Q; d K,sh is the vector dimension corresponding to each character in the key vector matrix K; d V,sh The vector dimension corresponding to each character in the value vector matrix V matrix; m is the total number of words in each text; The vector dimension corresponding to each character in the Q, K, V matrices is equal to the vector dimension d under the multi-head attention mechanism mh Divided by the number of self-attention mechanisms n, the calculation formula is as follows: Among them, sh stands for single-head abbreviation, and mh stands for multi-head abbreviation; The multi-head attention calculation method is used to fuse the key semantic information and obtain a new sentence vector. Then, the new sentence vector y is transformed into [CLS] For subsequent work; Step 4: Build the language representation model BERT to perform linear dimensionality reduction on the sentence vector in the third step to obtain the probabilities of various fault labels; and dynamically adjust the recognition loss value of the language representation model BERT by adding weights; The dimension of the sentence vector y[CLS] finally output by the encoding layer of the language representation model BERT is d mh , which is 768-dimensional and contains only semantic vector information. It is then processed by linear dimensionality reduction in the fully connected layer of the language representation model BERT to obtain the probability of various fault labels; Use the logistic regression function Softmax to perform probability normalization on y[CLS] to obtain a one-dimensional vector y out , the value corresponding to this vector represents the probability of the corresponding fault type label; The label corresponding to the maximum probability value is selected as the predicted transformer fault type, and its calculation formula is as follows: y out =Softmax(W out ·y [CLS] +b out ) Where W out and b out is the weight matrix and bias vector of the fully connected layer; The method for dynamically adjusting the language representation model BERT recognition loss value is as follows: Weights are added on the basis of the cross entropy loss function to give higher weights to categories with less data, so that the network model pays attention to small sample categories and performs fine fitting on all types of fault description texts. The calculation formula is as follows: Where L j represents the model loss value of the jth training batch, N represents the number of transformer fault texts in a training batch; p i Represents the recognition accuracy of the i-th fault sample label in a training batch; Step 5: Perform cluster analysis on the probabilities of various fault labels in step 4 using the Canopy+K-Means model; Extract the type label corresponding to the transformer fault text data; In the fifth step, the Canopy+K-Means model includes a Canopy unit and a K-Means unit; The Canopy unit is used to calculate the label probability value y out Perform preliminary clustering to obtain the K value and K initial cluster centers; The K-Means unit is used to calculate the label probability value y out Perform fine cluster analysis; The cluster analysis method is as follows: All label probability values ​​y out It is stored in the computer in the form of a set list, and a distance threshold L is selected. L can be calculated by the following formula: Where D represents the Euclidean distance operation of the point, and N represents the total number of transformer fault texts; From the label probability value y out Randomly select a point P1(n1, n2) from the set and regard it as the first type of fault. Continue to randomly select points P2, P3, P4, ... P from the set. N , calculate the distance from the newly taken point to all points belonging to the first type of fault type. If the distance is less than L, delete the point from the set list. If the distance is greater than L, extract the point from the set list as a new fault type. Repeat the calculation until the points in the set are empty, the preliminary clustering is completed, and the K value and K initial cluster centers are obtained. Then, perform K-means unit operation to calculate the distances from all points to the K initial cluster centers, classify each point into one category with the initial cluster center that is closest to it, and then calculate the average value of each cluster as the new cluster center. Repeat the iteration until the position of the cluster center no longer changes, and finally extract the type label corresponding to the transformer fault text data. Step 6: Based on the type label in step 5, identify the specific fault location and fault type described by the transformer fault text data.

2. The method for identifying transformer faults by refined fitting based on text mining according to claim 1, characterized in that: In the first step, the transformer fault text data includes a fault description text and a fault type, which are regularly filled in the same table column; The segmentation method includes the following: The maximum length of each sentence text is L characters, and the minimum length of each sentence text is M; When the sentence length is less than M, several sentences are concatenated; When the sentence length is in the interval [M, L], the sentence is not concatenated; The concatenated sentences are sorted by length, and sentences of similar length are put into the same training batch.

3. The method for identifying transformer faults by refined fitting based on text mining according to claim 1, characterized in that: In the second step, vectorization processing includes the following: The single character in the transformer fault text data is used as the smallest unit token, and then the initial text vector matrix X is obtained through several embedding layers. e,0 ; The plurality of embedding layers include at least a tag embedding layer, a fragment embedding layer, and a position embedding layer.

4. The method for identifying transformer faults by refined fitting based on text mining according to claim 3 is characterized in that: The markup embedding layer includes the following: The transformer mechanism is used to randomly initialize the single characters in the transformer fault text data, convert each character into a fixed-dimensional vector, and then perform token-id conversion. The corresponding vector representation is obtained through id mapping, and then vector [CLS] and vector [SEP] are added at the beginning and end of each text respectively. The fragment embedding layer consists of the following: Use vectors [SEP] to classify whether two texts are semantically similar, represented by at least two vectors. The index number of the first sentence vector is A, and the indexes of all tags belonging to this sentence are marked as A; The index number of the second sentence vector is B, and the indexes of all tags belonging to this sentence are marked with B; The vector of the fragment embedding layer is obtained by comparing the index number with the fragment embedding table; The vector comparison lookup table of the position embedding layer includes the following contents: When the number of characters in a text is less than the maximum text length, it needs to be padded. The padded position is meaningless and does not participate in subsequent calculations. Initial text vector matrix X e,0 It is obtained by summing the vector representations of several embedding layers and passed to the language representation model BERT.

5. The method for identifying transformer faults by refined fitting based on text mining according to claim 1, characterized in that: It also includes the performance evaluation of transformer fault identification equipment, which specifically includes the following: The transformer fault text dataset is randomly divided into a training set and a test set in a ratio of 8:

2. The training set is used for equipment training, and the test set is used for equipment performance evaluation. The performance evaluation index is the F1 value, and the calculation formula of the F1 value is as follows:

6. Refined fitting based on text mining to identify transformer faulty equipment, characterized in that: Apply the refined fitting method for identifying transformer faults based on text mining as described in any one of claims 1 to 5; It includes a keyboard, a display screen, a processor, an input transformer fault text unit, a main storage unit, an external storage unit, a power supply module, a refined fitting transformer fault identification module based on text mining, and an output transformer fault location and type unit; The refined fitting transformer fault identification module based on text mining includes a text vectorization unit, a language representation model BERT encoding unit, a transformer fault label probability calculation unit, and a Canopy+K-Means clustering analysis unit, all of which are encapsulated and stored in the main storage unit; The inspection personnel input the transformer fault text data into the transformer fault text unit via the keyboard, or read the transformer fault text data from the external storage unit; The main storage unit stores and reads the transformer fault text data transmitted to the device and transmits it to the refined fitting transformer fault identification module based on text mining for processing and analysis. The result data analyzed by the refined fitting transformer fault identification module is then returned to the main storage unit. Finally, the identification result is output through the output transformer fault location and type unit.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method for identifying transformer faults by refined fitting based on text mining as described in any one of claims 1 to 5 is implemented.

Images