A power system anomaly monitoring method based on intelligent mining of log features
Through intelligent mining methods based on log features, DBSCAN clustering and natural language processing technology, the automated analysis of abnormal monitoring of power system is realized, the problem of inefficient screening is solved, the abnormal detection performance and audit efficiency are improved, and the Real-time abnormal monitoring and rapid problem positioning capabilities.
Patent Information
- Application Number
- CN202111297960.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-04
AI Technical Summary
The monitoring of abnormalities in existing power systems mainly relies on manual screening, which is inefficient and cannot detect problems in real time, and may lead to serious consequences.
Using a method based on intelligent mining of log features, exception logs are filtered through DBSCAN clustering algorithm and Pearson correlation coefficients, and log information is automatically analyzed in combination with natural language processing technology to achieve abnormal detection without manual intervention.
It improves abnormal detection performance, improves the degree and efficiency of audit automation, saves manpower and material resources, and can accurately identify power system abnormalities in multiple time spans and multiple scenarios, providing real-time monitoring and rapid problem positioning.
Smart Images

Figure CN114048870B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data information processing, and in particular to a method for monitoring abnormalities in a power system based on intelligent mining of log features. Background Art
[0002] With the construction and development of smart grids, the application of power system automation is becoming more and more extensive. At present, power automation functions are mainly used in production inspection and regulation of power enterprises, system and component protection, and information transmission automation. These links will generate massive log data in the power system during operation. The log data usually records the real-time or periodic data of the status of each component during equipment operation, and records the error information of the equipment at the time when the power system fails. Due to the large number of equipment and instruments in the power system, the problems that may arise are complex and diverse. At the same time, due to the requirements of the power system for stability and rapid response, it has strict requirements on the real-time processing. Therefore, how to find problems in time from massive power logs and find out the causes of the problems has become a headache for power companies.
[0003] At present, power system abnormality monitoring mainly relies on manual screening, and a team of power experts with relevant experience promptly and effectively investigates the cause of the problem to ensure the normal operation of the power system. However, this method is time-consuming and labor-intensive, and inefficient. Manual work cannot guarantee that abnormalities will be discovered at any time within 24 hours, and sometimes negligence may lead to some serious consequences. Therefore, the inefficient method of manually checking logs to discover problems in the traditional power system is destined to be eliminated. In recent years, with the development of artificial intelligence technology, more and more companies are applying artificial intelligence technology to real business scenarios. Based on this background, we can consider applying artificial intelligence technology to abnormal monitoring of power systems. Summary of the invention
[0004] In view of the nonlinear, high-dimensional, and time-strongly coupled characteristics of power big data, the present invention intelligently mines and analyzes the log features of the power system, extracts relevant indicators that reflect the quality of power operation and maintenance, monitors whether the platform is abnormal during operation, and summarizes and analyzes the abnormal log information text, providing a basis for power system operation and maintenance personnel. The present invention proposes a power system abnormality monitoring method based on intelligent mining of log features, and improves the algorithm based on the characteristics of business scenarios and the defects of the original algorithm, so that the whole process does not require manual intervention, automatically analyzes log information, and improves the abnormality detection performance. The proposed method can effectively realize abnormal power detection, improve the automation degree and efficiency of power inspection, save manpower and material resources, and has a high application prospect.
[0005] In order to achieve the above object, the technical solution of the present invention is as follows:
[0006] A method for monitoring abnormality of a power system based on intelligent mining of log features comprises the following steps:
[0007] Step S1, collecting log data in multiple time spans and multiple scenarios and performing preprocessing to form log data samples;
[0008] Step S2, performing cluster analysis on the log data sample based on the DBSCAN clustering algorithm to obtain clustering results, and using the Pearson correlation coefficient to screen the clustering results to obtain abnormal logs in the log data sample;
[0009] Step S3, using natural language processing technology to parse the exception log text, and feeding back the parsed exception log summary to relevant staff.
[0010] Preferably, the step S1 specifically comprises the following steps:
[0011] Step S101, selecting power log data of the past T years from the platform, where T is an integer not less than 1;
[0012] Step S102, based on the seasonal data screening, according to the four seasons of spring, summer, autumn and winter, two months of electricity log data are collected in each season; based on the time period screening, the electricity log data of different time periods in the morning and evening are collected; based on the holidays screening, the electricity log data of holidays and large-scale events are collected; based on the regional differences between cities and counties, the electricity log data of urban areas and rural towns are collected; based on the regional functional characteristics, the electricity log data of industrial areas, residential areas and commercial areas are collected;
[0013] Step S103, shuffle and integrate the log data collected in step S102 to create a final log data sample.
[0014] Preferably, the step S2 specifically comprises the following steps:
[0015] Step S201, calculating the Euclidean distance between each data point in the log data sample;
[0016] Step S202, sorting the obtained Euclidean distance results in ascending order to construct a distance matrix;
[0017] Step S203, calculate the matrix average value in columns to obtain the average distance vector, randomly select an unlabeled log sample point in the cluster, and use the value in the average distance vector of the unlabeled log sample point as the candidate value of the ε-nearest neighbor; use each generated candidate value to calculate the number of data points included in the ε-nearest neighbor, take the average of the number of data points in each neighborhood, and establish a candidate set of the ε-nearest neighbor and the density threshold Minpts;
[0018] Step S204, performing DBSCAN clustering using the candidate set parameters, and calculating the silhouette coefficient of the clustering result;
[0019] Step S205, repeating steps S201 to S204, evaluating the silhouette coefficients of clustering results under different candidate parameters, and determining the optimal clustering result;
[0020] Step S206: Use the Pearson correlation coefficient to screen the clustering results to obtain abnormal logs in the log data sample.
[0021] Preferably, the preprocessing includes noise data removal, duplicate data processing, numerical missing value processing, categorical feature encoding, numerical standardization and log text data processing.
[0022] Preferably, the categorical feature encoding includes a one-hot encoding method or a frequency encoding method; and the standardization adopts a min-max standardization method.
[0023] Preferably, the step S3 specifically comprises the following steps:
[0024] Step S301, extracting all words in the error log in the log data sample and constructing an abnormal keyword feature set;
[0025] Step S302, calculate the information gain value IG of the abnormal keyword, and the specific calculation formula is:
[0026]
[0027] in, Represents y i The probability of the category text appearing in the data set, that is, y i The number of category texts divided by the total number of texts, y i The category text is information log text or error log text; Indicates that the data set contains keyword feature items x i The probability of a text containing feature item x i The number of texts divided by the total number of texts; x i Indicates that it does not contain the keyword feature item x i The text of p(y i |x i ) indicates that the text contains the keyword feature item x i When belongs to y i The conditional probability of the category, that is, the keyword feature item x i and belongs to category y i The number of texts divided by the number of texts containing the keyword feature x i The number of texts;
[0028] Step S303: Integrate the calculated information gain value of the abnormal keyword into the TF-IDF algorithm to obtain an improved TF-IDF calculation formula:
[0029]
[0030]
[0031] If abnormal keywords exist in the log, the improved weight calculation formula ω is used new , calculate the weight value of the feature item, otherwise it is calculated according to the traditional weight calculation formula ω, where t represents the feature item in the text, A represents the error log keyword set, B represents the power log text, ω represents the weight value of the feature item t in the text d, Indicates the information gain value of abnormal keywords;
[0032] Step S304: sort all abnormal keywords according to the feature item weight values calculated in step S303, select several abnormal keyword sets with the highest weight values to form an abnormal log summary, and send it to relevant staff.
[0033] Based on the above technical solution, the beneficial effects of the present invention are:
[0034] 1) Based on the business scenario characteristics of the power system, the present invention collects log samples in multiple time spans and multiple situations to create a data set with moderate data volume, full scenario coverage, and obvious features, so that the data samples are more in line with the actual scenario. Based on this feature sample, the model generalization performance is stronger;
[0035] 2) Based on the characteristics of power log data, the present invention uses machine learning cluster analysis to find outliers to detect the occurrence of abnormal conditions, and further improves the original algorithm so that the algorithm automatically mines the optimal parameters, optimizes the model according to the evaluation feedback mechanism, calculates the correlation to determine the abnormal log, and realizes fully automatic analysis of the data;
[0036] 3) In addition to real-time monitoring of abnormal situations, the platform uses natural language processing technology to intelligently feedback key information of abnormal log results (abnormal log summary), and improves the original algorithm by introducing error log information gain to adjust the word frequency weight. This information can help power operation and maintenance personnel to timely understand the basic situation of the current equipment operation, lock the error point more quickly, and improve maintenance efficiency;
[0037] 4) The abnormal monitoring and analysis method based on data mining of the present invention has high recognition accuracy, low computational complexity, adopts an unsupervised artificial intelligence algorithm, does not require manual data annotation, has strong portability, and can be widely used in multiple fields such as medical treatment, logistics management, and industrial production. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0039] Figure 1 is a flow chart of a method for monitoring abnormality of a power system based on intelligent mining of log features in an embodiment;
[0040] Figure 2 It is a flow chart of creating log data samples in a method for monitoring abnormality of a power system based on intelligent mining of log features in an embodiment;
[0041] Figure 3 The diagram is a basic conceptual diagram of a DBSCAN algorithm in a method for monitoring abnormalities in a power system based on intelligent mining of log features in an embodiment. DETAILED DESCRIPTION
[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0043] like Figure 1 As shown, this embodiment proposes a method for monitoring power system anomalies based on intelligent mining of log features, which can timely and effectively detect abnormal behaviors in system events, provide early warnings for possible abnormal situations, and provide feedback on abnormal log summaries. This method can automatically parse logs based on different log data sources, and discover abnormal behaviors reflected in logs during system operation through statistical machine learning anomaly detection methods, thereby realizing intelligent monitoring. Based on the power system business scenario, this technical solution is specifically described as follows:
[0044] 1. Log data sample collection
[0045] Based on the actual scenario, data is collected from the power system data warehouse according to the existing power log data to generate data samples. The logs of the power system are mainly divided into error logs and information logs. The error log mainly records the error information during the operation of the equipment, and the information log records the equipment status regularly by setting triggers or time intervals. However, in actual applications, the error log data is often very small, with only a few pieces of data, and sometimes there is not even a single error message, which may lead to a lack of negative samples in abnormal analysis. At the same time, the amount of data recorded by the power platform is large, and how to collect data that can reflect the operating characteristics of the system is also crucial.
[0046] Typical power system logs generally include the following attributes: log occurrence date, event occurrence time, log source system, event duration, operator, business object, etc. From these, we can filter out some attributes related to system fault data and conduct focused correlation analysis. See the table below for specific attributes:
[0047] Serial number Property Name 1 Log unique ID 2 IP address 3 Device Name 4 Log Level 5 Log content 6 Power load 7 Power Consumption 8 Status Name 9 Reasons for change 10 Exception information 11 Log creation time 12 Operator 13 …
[0048] The usual method for constructing a data set is to use data from the most recent month or months as sample data, but the sample data obtained by such a construction method has a single focus and cannot cover the entire power system operation cycle. If data with a longer time span is collected, the sample size will be too large, and subsequent analysis will be extremely time-consuming and inefficient. In view of the above situation, the present invention designs a log data collection method suitable for power platform scenarios, which collects samples in multiple time spans and multiple scenarios. The specific method is: (1) First, select log data from the platform for the past year; (2) Filter data based on seasons, because electricity consumption in different seasons will be different, and summer is often the peak period of electricity consumption, so two months of data are selected in each of the four seasons of spring, summer, autumn, and winter; (3) Filter data based on time periods, because electricity consumption at night is often greater than that during the day, so data is selected in different time periods, such as 9 am to 12 pm and 2 pm to 5 pm. 10:00-21:00 at night, and 0:00-3:00 in the morning; (4) Filter data based on holidays, and collect electricity logs during holidays and major events, such as the Spring Festival, National Day, and Mid-Autumn Festival; (5) Filter data based on differences in cities, counties, and districts. There are differences in electricity consumption between urban areas and rural towns, so data can be collected by region; (6) Filter data based on regions. The electricity consumption in industrial areas, residential areas, and central business areas will be different, so log data from industrial areas, residential areas, and commercial areas will be collected; (7) Shuffle and integrate the data collected in the above steps to create the final data set. The sample generation process is attached. Figure 2 .
[0049] (II) Log preprocessing
[0050] The original sample data stored in the power platform is complex and has many features. It is not suitable to be directly used as the subsequent model training data. Some data may be irrelevant to the subsequent analysis operations or contain interference items. Therefore, it is necessary to preprocess the data before building the model. Preprocessing mainly includes:
[0051] (1) Noise data removal
[0052] In the sample data, the data stored in the database of the power system may be associated with other information, so it often includes log ID, creation time, IP address, equipment name, special symbols, department, maintenance personnel and other feature data. These feature items are not related to abnormal monitoring, and these attributes cannot describe the distribution law of the sample itself. If these features are retained, it will not only greatly increase the workload of text processing, but also have a direct impact on a series of subsequent operations, thereby reducing the efficiency of text processing. Therefore, removing these noise data is the first task before abnormal analysis.
[0053] (2) Duplicate data processing
[0054] There will be duplicate values in the process of data collection. Duplicate values may be caused by repeated startup of the platform program or problems in the storage stage. The merge method is used to merge equal records into one record by judging whether the attribute values between records are equal.
[0055] (3) Handling of missing values
[0056] Process the null value data in the data record. The default value may be caused by the platform not collecting performance data normally. If the distance of the sample attribute is measurable, the average value of the valid value of the attribute is used to interpolate the missing value; if the distance is not measurable, the mode of the valid value of the attribute is used to interpolate the missing value. Multiple interpolation can also be used. Multiple interpolation assumes that the value to be interpolated is random. In practice, it is usually estimated that the value to be interpolated is added with different noises to form multiple groups of optional interpolation values. According to some selection criteria, the most appropriate interpolation value is selected.
[0057] (4) Categorical feature encoding
[0058] Some of the input features are unordered categorical variables and should be converted into matrices through One-Hot Encoding. For example, the current state feature has three states, so One-Hot Encoding is used for encoding.
[0059] If the number of categories is less than 10, one-hot encoding is used; if the number of categories is more than 10, frequency encoding is used.
[0060] (5) Numerical standardization
[0061] After the features are digitized, they are normalized. This solution uses the min-max normalization method to normalize the original data x 1 , x 2 ...x n Perform a linear transformation so that the result y i Falling to the interval [0, 1], the conversion function is as follows:
[0062]
[0063] (6) Log text data processing
[0064] In addition to numerical data, power logs often also record text data, such as the basic situation or configuration information of each device in the current operating period of the system, or the current status when a platform fails. The addition of this information can help power maintenance personnel to find out the cause of the problem more quickly, so this type of data needs to be processed separately. The specific processing steps are: first, the text is segmented into Chinese words, and then the stop word list is loaded to remove unimportant entries in the segmentation results and retain important entries. Unlike the usual removal of English content when deleting stop words, since the English information in the system platform log often contains some important information, only Chinese stop words, punctuation or special symbols are deleted.
[0065] (III) Log Clustering Mining
[0066] After completing the preprocessing of the power log, we can get a data sample set containing numerical and text data. Now we start to use numerical data for cluster mining and use clustering algorithms to predict possible abnormal information. Most of the data set is information logs, and there are also a small number of error logs. These real error logs need to be paid special attention to.
[0067] (1) DBSCAN algorithm and its improvement
[0068] Given that the dimension of the power log data set is relatively low and the clusters cannot be determined, this solution uses the DBSCAN clustering algorithm to mine abnormal data in the power log. The DBSCAN algorithm is a classic density-based clustering algorithm that can identify clusters of any number and shape in a noisy data set. The advantage of the algorithm is that it does not need to specify the number of clusters, can cluster dense data sets of any shape, and the clustering results do not depend on the traversal order of sample points. The algorithm is mainly based on the following concepts:
[0069] 1. ε-neighborhood: The area within the ε radius of an object p is called the ε-neighborhood of object p;
[0070] 2. Density: the number of data points contained in the ε-neighborhood;
[0071] 3. Core point: Set X i For any point in the set, when the number of points in its ε-neighborhood exceeds the density threshold Minpts, the point X i As the core point;
[0072] 4. Direct density reachability: It can determine whether two points belong to the same cluster. If the following two conditions are met, point p2 is said to be directly density reachable from point p1. (a) The two points are close enough to satisfy distance(pi, p2) < ε; (b) There are enough points in the ε-neighborhood of p1 to satisfy, that is, p1 is a core point;
[0073] 5. Density reachable: In a set D, if there exists an object chain p 1 , p 2 ,…,p n , let p 1 =q,p i It is from p (i-1) If p is directly density-reachable with respect to ε and Minpts, then object p is density-reachable from object q with respect to ε and Minpts;
[0074] 6. Density connectability: Assume that there are points O, A and B in the sample set D. When points O and A satisfy density reachability, and points O and B also satisfy density reachability, then points A and B satisfy density connectability.
[0075] Based on the above concepts, the basic idea of the DBSCAN algorithm is to determine whether each point in the entire clustered data set belongs to the same class according to whether the number of sample points within the threshold radius ε is greater than the density threshold Minpts. If the current point p is a core point, then other points in its ε neighborhood are also added to the set, and the cluster where the current data point is located is continuously expanded until all data points that meet the current cluster are traversed to form a complete cluster. If point p is a boundary point, that is, the number of other points in its ε neighborhood is less than the threshold Minpts, then skip this point and continue to select the next point in the data set for processing. The remaining points in the data set are selected as noise (abnormal) points. The query process under a two-dimensional data set is as follows: Figure 3 shown.
[0076] However, the drawback of this algorithm is that it requires users to manually set the two parameters, ε-nearest neighbor and density threshold Minpts, based on experience, which results in the clustering effect depending on the user's experience. Usually, users need to constantly try to obtain appropriate ε-nearest neighbor and Minpts parameter values, which greatly increases the difficulty of operation. In addition, since it is necessary to regularly perform clustering analysis on power log data of different time spans, the parameter values are also different, further increasing the difficulty of practical application. In addition, the business purpose of this scenario is to detect possible abnormal logs. The original algorithm failed to effectively determine whether the data is abnormal, which may cause data misjudgment.
[0077] In order to improve the clustering effect and the efficiency of abnormal monitoring, three improvements were made based on the original algorithm in combination with the characteristics of power log data. The improvements include: adaptive parameter adjustment, introduction of a new evaluation mechanism as a clustering evaluation indicator and abnormal data detection. The specific instructions are as follows:
[0078] 1. Adaptive parameter adjustment
[0079] The algorithm updates the parameters by calculating the distance between the log feature vectors. The specific improvement method is to first calculate the Euclidean distance L between each power log i and other log data in the data set. Dis , forming a distance vector. The Euclidean distance reflects the close connection between the logs. The larger the distance, the closer the connection.
[0080] L i =[L Dis (i, 1), L Dis (i, 2),…,L Dis (i, N)]
[0081] For this vector L i Arrange in ascending order according to the degree of closeness, and get the vector L s,i . Then follow the steps above to create a distance matrix of all the logs in the dataset:
[0082] L=[L s,1 , L s,2 , …, L s,N ] T
[0083] Calculate the matrix average in columns to get the average distance vector This vector reflects the closeness between a log and any data in the data set. The N values in the average vector are used as candidate values of the ε-nearest neighbor to form the candidate set X of the parameter ε-nearest neighbor. ε , as shown below:
[0084]
[0085] Utilizing X ε For each candidate value generated in the above example, the number of data points in its ε-nearest neighborhood is calculated. The amount of data contained in the neighborhood of each log sample point is different, so the number of data points in each neighborhood is averaged to establish the candidate set X of the parameter density threshold Minpts. Minpts .
[0086] Finally, the candidate set X ε , X Minpts By inputting N groups of parameters into the DBSCAN algorithm, the number of clusters corresponding to each group of parameters can be obtained. Combined with the newly introduced clustering result evaluation index, the optimal number of clusters for the power log data set can be determined. This method does not require manual parameter adjustment. The distance vector between logs obtained by calculation can well reflect the distribution characteristics of the data set itself, making the clustering process more focused on the data itself. After using this method to determine the algorithm parameters, the clustering results are closer to the distribution characteristics of the data, and the recognition results are more in line with the real business scenario.
[0087] 2. Clustering evaluation indicators
[0088] This application uses the silhouette coefficient as the clustering result evaluation index. The calculation formula of the silhouette coefficient SC of all sample points is as follows:
[0089]
[0090] Where N is the number of sample points; a i is the average value of the dissimilarity between vector i and other points in the same cluster; b i It is the minimum value of the average dissimilarity between vector i and other clusters. The value of SC is between -1 and 1. The larger the value, the better the clustering effect.
[0091] 3. Abnormal data detection
[0092] Through the above-mentioned improved method, although the optimal number of clusters has been determined, not every piece of noise data in the clustering result corresponding to the optimal parameters is power abnormality data, and misjudgment may occur. Therefore, the judgment method of noise log data is further improved, and the final abnormal data is determined by checking the data correlation. The specific steps are as follows:
[0093] 1) After obtaining the optimal parameters, DBSCAN is used for clustering. At this time, there is a noisy log data set D in the clustering result Noise And the log set D of each cluster 1 , D 2 , …, D K .
[0094] 2) Calculate the cluster center point of each cluster in the clustering results, and take the log data closest to the cluster center as the core data.
[0095] 3) Calculate D Noise The Pearson correlation coefficient between each noise log and each core log data in is used to measure the degree of correlation between two variables. The larger the correlation coefficient, the stronger the correlation between the two variables, and vice versa. For two samples X = {x 1 , x 2 , …, x n},Y={y 1 ,y 2 , …, y n}, the Pearson correlation coefficient formula is as follows:
[0096]
[0097] in, are the means of X and Y respectively, ρ(X,Y)∈[-1,1].
[0098] 4) Compare the sizes of the Pearson coefficients. If the correlation between the noise point and the cluster center data is low, it is determined to be abnormal log data; otherwise, the noise is divided into the cluster with the largest correlation coefficient.
[0099] 5) Traverse D through step 4) Noise , and finally get the abnormal log data set D outlier .
[0100] (2) Error log analysis
[0101] The data set contains information logs when the power equipment is operating normally, and also contains a small amount of error logs when the equipment fails. The clustering of error logs requires special treatment during cluster analysis. Since the DBSCAN algorithm is based on density distribution clustering, data points with similar features are divided into the same cluster. Therefore, data points near error logs are similar and may be outliers.
[0102] Based on this situation, this scheme retrieves the distribution of all error logs from the clustering results, takes the K points closest to the error log data point in Euclidean distance, and these K data points belong to the same cluster as the error log point. The value of K is the ratio of the number of error logs in the data set to the total number of samples, but such a value may result in too many abnormal points, so only samples with a wider distribution of error log data points are retrieved.
[0103] To sum up, the improved DBSCAN algorithm process is as follows:
[0104] Input: Dataset D
[0105] Output: ε-nearest neighbor and Minpts optimal parameters, abnormal log data D outlier
[0106] 1. Calculate the Euclidean distance between each data point in the data sample;
[0107] 2. Sort the distance results in ascending order and establish a distance matrix;
[0108] 3. Calculate the mean of each column to obtain the parameter candidate set of ε-nearest neighbor and Minpts;
[0109] 4. Perform DBSCAN clustering using the candidate set parameters and calculate the silhouette coefficient of the result;
[0110] 5. Repeat the above steps to evaluate the silhouette coefficient of the clustering results under different candidate parameters and determine the optimal clustering effect;
[0111] 6. Based on the optimal clustering analysis results, determine possible abnormal logs.
[0112] (IV) Intelligent analysis and feedback of abnormal logs
[0113] By training the log clustering model, the power platform can predict whether there may be abnormal conditions in this log when the latest log data is input. However, simply feeding back the prediction results cannot allow equipment managers to clearly understand the current operation status of the platform. Relevant staff are still confused about the abnormal situation and do not know where to start. It may take a lot of time to query the log details. In order to solve this problem, an abnormal log parsing module is designed, which uses natural language processing technology to extract key information from the log text to help equipment maintenance personnel accurately locate the source of the abnormality.
[0114] (1) TF-IDF algorithm and its improvement
[0115] The TF-IDF algorithm is used to extract key information from abnormal logs. TF-IDF is a commonly used weight value calculation method, which is widely used in the fields of text mining and information extraction to evaluate the importance of a certain term to the entire document set or a document in the corpus. The term frequency TF indicates the frequency of the term in the document; the inverse document frequency IDF reflects the importance of the term in the document data set. The main calculation formula is shown below:
[0116]
[0117]
[0118] ω=TF×IDF
[0119] TF values usually need to be normalized, usually the frequency m of feature item t appearing in text d dt Divide by the total number of words M in the text d d , to prevent it from being biased towards long files. IDF is the total number of files N divided by the number of files n containing feature item t. t (Adding 0.1 is to avoid the situation where the denominator is 0), and then taking the logarithm of the quotient is obtained. ω represents the weight value of feature item t in text d.
[0120] The shortcoming of the TF-IDF algorithm is that it cannot well characterize the distribution of feature words between categories. It only considers the word frequency factor when calculating, and the information extracted is mostly content with high frequency. However, in the power log, most of the logs are information output when the equipment is running smoothly, and there are very few error logs, but the content of the error log is very important. Therefore, the traditional TF-IDF algorithm will only extract the key information when the equipment is normal, and the error information will be ignored because it appears less frequently.
[0121] In view of this situation, the traditional TF-IDF algorithm is improved by taking the error log content in the data sample as an important text feature item and incorporating this feature item into the original weight calculation formula to enhance the algorithm's ability to distinguish erroneous data. The specific process is as follows:
[0122] For all the error logs in the preprocessed data set, all the words are extracted. These words contain a lot of text-specific content and have good abnormality distinction capabilities, so they are constructed into an abnormal keyword feature set.
[0123] For all the information logs in the preprocessed data set, the information gain value is calculated by counting whether the keywords obtained in the previous step appear in the logs. The information gain value is used to measure how much information the feature can bring for category distinction. The larger the information gain value, the stronger the feature item's ability to distinguish abnormal information categories. The specific calculation formula is:
[0124]
[0125] in, Represents y i The probability of the category text appearing in the data set, that is, y i The number of category texts divided by the total number of texts, y i The category text is information log text or error log text; Indicates that the data set contains keyword feature items x i The probability of a text containing feature item x i The number of texts divided by the total number of texts; x i Indicates that it does not contain the keyword feature item x i The text of p(y i |x i ) indicates that the text contains the keyword feature item x i When belongs to y i The conditional probability of the category, that is, the keyword feature item x i and belongs to category y i The number of texts divided by the number of texts containing the keyword feature x i The number of texts.
[0126] When calculating TF-IDF, combine it with the result of the previous step to improve the weight calculation method of word frequency and obtain the improved TF-IDF calculation formula:
[0127]
[0128]
[0129] Identify keywords related to error messages in the log. If the keywords exist in the log, use the improved weight calculation formula ωnew , calculate the feature item weight value, otherwise it is calculated according to the traditional weight calculation formula ω. The text feature item weight assignment is as follows: t represents the feature item in the text, A represents the error log keyword set, B represents the power log text, ω represents the weight value of feature item t in text d, Indicates the information gain value of abnormal keywords.
[0130] In summary, the improved TF-IDF algorithm can be used to obtain the importance weight of each word in the abnormal log, and finally the Top-K words with higher TF-IDF values are taken as the results and fed back to the power system operation and maintenance personnel. The value of K can be determined by the user, and finally the key information in the clustering abnormal results can be successfully obtained.
[0131] The above is only a preferred implementation of a method for monitoring abnormalities in a power system based on intelligent mining of log features disclosed in this embodiment, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.
Claims
1. A power system anomaly monitoring method based on intelligent mining of log features, It is characterized in that The steps include: Step S1, collecting log data in multiple time spans and multiple scenarios and performing preprocessing to form log data samples; Step S2, performing cluster analysis on the log data sample based on the DBSCAN clustering algorithm to obtain clustering results, and using the Pearson correlation coefficient to screen the clustering results to obtain abnormal logs in the log data sample; Step S3, using natural language processing technology to parse the abnormal log text, and feeding back the parsed abnormal log summary to relevant staff, specifically includes the following steps: Step S301, extracting all words in the error log in the log data sample and constructing an abnormal keyword feature set; Step S302, calculate the information gain value IG of the abnormal keyword, and the specific calculation formula is: in, Represents y i The probability of the category text appearing in the data set, that is, y i The number of category texts divided by the total number of texts, y i The category text is information log text or error log text; Indicates that the data set contains keyword feature items x i The probability of a text containing feature item x i The number of texts divided by the total number of texts; x i Indicates that it does not contain the keyword feature item x i The text of p(y i |x i ) indicates that the text contains the keyword feature item x i When belongs to y i The conditional probability of the category, that is, the keyword feature item x i and belongs to category y i The number of texts divided by the number of texts containing the keyword feature x i The number of texts; Step S303: Integrate the calculated information gain value of the abnormal keyword into the TF-IDF algorithm to obtain an improved TF-IDF calculation formula: If abnormal keywords exist in the log, the improved weight calculation formula ω is used new , calculate the weight value of the feature item, otherwise it is calculated according to the traditional weight calculation formula ω, where t represents the feature item in the text, A represents the error log keyword set, B represents the power log text, ω represents the weight value of the feature item t in the text d, Indicates the information gain value of abnormal keywords; Step S304: sort all abnormal keywords according to the feature item weight values calculated in step S303, select several abnormal keyword sets with the highest weight values to form an abnormal log summary, and send it to relevant staff.
2. According to claim 1, a method for monitoring abnormalities in a power system based on intelligent mining of log features, It is characterized in that The step S1 specifically includes the following steps: Step S101, selecting power log data of the past T years from the platform, where T is an integer not less than 1; Step S102, based on the seasonal data screening, according to the four seasons of spring, summer, autumn and winter, two months of electricity log data are collected in each season; based on the time period screening, the electricity log data of different time periods in the morning and evening are collected; based on the holidays screening, the electricity log data of holidays and large-scale events are collected; based on the regional differences between cities and counties, the electricity log data of urban areas and rural towns are collected; based on the regional functional characteristics, the electricity log data of industrial areas, residential areas and commercial areas are collected; Step S103, disrupting and integrating the log data collected in step S102 to create a final log data sample.
3. According to claim 1, a method for monitoring abnormality of a power system based on intelligent mining of log features, It is characterized in that The step S2 specifically includes the following steps: Step S201, calculating the Euclidean distance between each data point in the log data sample; Step S202, sorting the obtained Euclidean distance results in ascending order to construct a distance matrix; Step S203, calculate the matrix average value in columns to obtain the average distance vector, randomly select an unlabeled log sample point in the cluster, and use the value in the average distance vector of the unlabeled log sample point as the candidate value of the ε-nearest neighbor; use each generated candidate value to calculate the number of data points included in the ε-nearest neighbor, take the average of the number of data points in each neighborhood, and establish a candidate set of the ε-nearest neighbor and the density threshold Minpts; Step S204, performing DBSCAN clustering using the candidate set parameters, and calculating the silhouette coefficient of the clustering result; Step S205, repeating steps S201 to S204, evaluating the silhouette coefficients of clustering results under different candidate parameters, and determining the optimal clustering result; Step S206: Use the Pearson correlation coefficient to screen the clustering results to obtain abnormal logs in the log data sample.
4. A method for monitoring abnormality of a power system based on intelligent mining of log features according to claim 1 or 3, It is characterized in that The preprocessing includes noise data removal, duplicate data processing, numerical missing value processing, categorical feature encoding, numerical standardization and log text data processing.
5. According to claim 4, a method for monitoring abnormality of a power system based on intelligent mining of log features, It is characterized in that The categorical feature encoding includes a one-hot encoding method or a frequency encoding method; the standardization adopts a min-max standardization method.
Citation Information
Cited By
A method for detecting log anomalies in a power dispatch automation system
CN120723588B
