A method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering

The data set is balanced by the nearest neighbor search clustering and SMOTE algorithm, combined with the random forest model, and the problem of low accuracy of abnormal detection in the power scheduling monitoring data is solved, and efficient abnormal detection of complex distribution modes is achieved.

CN114722947BActive Publication Date: 2024-12-06BEIJING UNIV OF POSTS & TELECOMM
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210383775.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-12
Publication Date
2024-12-06
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

Due to the large amount of data, many feature dimensions, complex distribution patterns and unbalanced number of abnormal samples, the existing abnormal detection methods have low accuracy, especially when abnormal samples form multiple small clusters, classification tasks are more difficult.

Method used

Using a method based on the search clustering of nearest neighbors, we can find the neighbors of anomalies by calculating the Euclidean distance between samples, identify the clusters of abnormal samples and filter the noise, and use the SMOTE algorithm to generate new samples in each cluster to balance the data set. Finally, we use Random forest model performs anomaly detection.

Benefits of technology

It improves the accuracy of abnormal detection of power scheduling monitoring data, effectively solves data imbalance and small cluster problems, and improves the accuracy and safety of power grid status monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114722947B_ABST
    Figure CN114722947B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention proposes a method for detecting anomalies in power dispatch monitoring data based on nearest neighbor search clustering, including: inputting power dispatch monitoring historical data with positive abnormal labels as a training data set, finding k nearest neighbors of abnormal samples in the feature space by calculating the Euclidean distance between samples; determining whether the sample is noise or belongs to a certain abnormal sample cluster through the nearest neighbor labels of the abnormal sample, and iteratively searching in the sample neighbors in sequence until no more abnormal samples belonging to the cluster are found; filtering the noise of the clustered data and calculating the number of abnormal samples to be generated in each cluster, and synthesizing new samples in each cluster using SMOTE linear interpolation to balance the data set; using the balanced data set to train a random forest model to detect abnormal samples in power dispatch monitoring data. The technical solution provided by the embodiment of the present invention can improve the accuracy of anomaly detection in power dispatch monitoring data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method for detecting anomaly of electric power dispatching monitoring data, and in particular to a method for detecting anomaly of electric power dispatching monitoring data based on nearest neighbor search clustering. [Background technology]

[0002] With the development of society and the expansion of the scale of power systems, the construction of smart grids is also developing rapidly. The smart grid system aims to solve various problems in the entire process from power generation facilities to power transmission and transformation to end users by actively utilizing communication and information technology. With the advancement of power grid intelligence, further requirements are put forward for the efficiency, safety and stability of power grid dispatching. The power dispatching automation system realizes the "horizontal integration and vertical connection" of power grid dispatching, ensures the safe, stable, economical and environmentally friendly operation of the power grid, and plays a very important role in the power generation, transmission, transformation and distribution links of the power system.

[0003] Although the power dispatching automation system can provide a large amount of power grid operation data and analysis results, which makes it convenient for power dispatching system operators to accurately assess the power grid status, it has a massive amount of data with many feature dimensions and complex relationships between them. As the command center of the power grid, once the dispatching system's business is abnormal, it will have a great impact on the stable operation of the power grid. Therefore, in order to ensure the safe and reliable operation of the smart grid, the anomaly detection technology of power dispatching monitoring data has become increasingly important.

[0004] The power dispatch monitoring data has the characteristics of large data volume, numerous feature dimensions, and complex and diverse distribution patterns. At present, most of them are detected by setting thresholds with the help of expert experience. This method is not flexible and has limited accuracy. Therefore, it is necessary to introduce machine learning methods to achieve more accurate anomaly detection. However, the dispatch automation system is in a stable operation state most of the time, and there are few abnormal situations. Therefore, the overall data sample has the problem of imbalance in the number of positive abnormal samples. In addition, due to the various reasons for the occurrence of abnormal situations, the patterns of abnormal samples in the data space are also different, that is, there is a problem that abnormal samples form multiple small clusters. When the number of data samples is extremely unbalanced, the trained classifier will be biased towards normal samples, resulting in a decrease in the detection accuracy of abnormal samples. At the same time, if the abnormal samples form multiple small clusters, the small cluster problem encountered in the category imbalance classification will make the classification task more difficult. Therefore, a supervised anomaly detection algorithm that can rebalance the data and solve the small cluster problem is proposed to improve the detection accuracy of abnormalities, which is of great significance for strengthening power grid status monitoring and ensuring power grid security. [Summary of the invention]

[0005] In view of this, the present invention proposes a method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering to improve the performance of anomaly detection in power dispatching monitoring data.

[0006] The present invention proposes a method for detecting anomaly in power dispatch monitoring data based on nearest neighbor search clustering, comprising the following steps:

[0007] (1) The historical data of power dispatch monitoring with positive anomaly labels are used as the training data set input, and the k nearest neighbors of the abnormal samples in the feature space are found by calculating the Euclidean distance between samples. Specifically,

[0008] The real-time resource occupancy data of the processes related to the power dispatching system business collected by the power dispatching monitoring system is used as the training data set input, denoted as X = {x 1 ,x 2 ,x 3 ,...,x i ,...,x N}, where x i ∈R u , i=1,2,...,N, R represents a real number set, u=6 represents the dimension of the data, including process CPU occupancy, memory occupancy, disk IO, network IO, number of threads, number of network connections; the total number of samples in the input data set X is N; the positive anomaly label of the training data set X is input, recorded as y={y 1 ,y 2 ,y 3 ,...,y i ,...,y N}, which corresponds one-to-one to the samples in X, y i ∈{0,1}, where y i =0 represents x i is a normal sample, y i =1 represents x i is an abnormal sample, i=1,2,...,N;

[0009] y i =0 for all x i Denote the normal sample set X of the training set nor , y i =1 for all x i Denote the abnormal sample set X of the training set abn , calculate X abn The i-th sample x in abn_i The Euclidean distance d(x abn_i ,x j ), x j ∈X, and take the distance x abn_iThe most recent k samples, denoted as x abn_i The nearest neighbor set kNN(x abn_i );

[0010] (2) Determine whether the sample is noise or belongs to a certain abnormal sample cluster through the neighbor labels of the abnormal sample, and iterate the search in the sample's neighbors until no more abnormal samples belonging to the cluster are found. Specifically:

[0011] Traverse X abn For all abnormal samples in , for each abnormal sample x abn_i First, determine whether it has been marked as belonging to an abnormal cluster. If x abn_i If x is marked, skip the sample. abn_i If it has not been marked, then get x abn_i The nearest neighbor set kNN(x abn_i ); if kNN(x abn_i ) all belong to the normal sample set X nor ,Right now x a ∈X nor , then x abn_i Labeled as noise sample set X noise In the sample, if kNN(x abn_i ) contains abnormal samples, that is, x b ∈X abn , then x abn_i With kNN(x abn_i ) are marked as the nth cluster C n samples, where n = 1, 2, 3, ...;

[0012] Then, for kNN(x abn_i ) all abnormal samples x abn_j , get its nearest neighbor kNN(x abn_j ), if there are unlabeled abnormal samples, they will also be marked as the nth cluster C n samples and continue to perform k-nearest neighbor search until all found abnormal samples have been labeled;

[0013] So far, for x abn_i The recursive marking ends, let n = n + 1, and continue to traverse the abnormal samples until all abnormal samples are marked as belonging to a certain abnormal cluster or to the noise sample set X noise So far, let the final value of n be M;

[0014] (3) Filter the noise of the clustered data and calculate the number of abnormal samples that need to be generated in each cluster. Based on this, use SMOTE linear interpolation to synthesize new samples in each cluster to balance the data set. Specifically:

[0015] For X abn The abnormal sample x in abn_i , if it belongs to the noise sample set X noise , then it is removed from X abn Delete it and get the denoised abnormal sample set X′ abn =X abn -X noise ; Then, calculate each abnormal cluster C m The number of abnormal samples that need to be generated m =num Cm ×(IR-1), where m = 1, 2, ..., M, num Cm The abnormal cluster C m The number of samples, IR is the imbalance rate of the entire data set, that is, |X nor | represents the normal sample set X nor The number of samples, |X′ abn | represents the denoised abnormal sample set X′ abn The number of samples;

[0016] Use the SMOTE algorithm to generate new abnormal samples in each cluster. Specifically, for an abnormal cluster C m , randomly select one of the abnormal samples x Cm_i ∈C m , in x Cm_i The nearest neighbor set kNN(x Cm_i ) randomly selects a sample x s ∈kNN(x Cm_i ), generate a new sample x by linear interpolation syn =x Cm_i +w×(x s -x Cm_i ), where w is a random parameter between [0,1]; all generated samples x syn Denoted as the generated sample set X syn , and compare it with the normal sample set X nor And the denoised abnormal sample set X′ abn Combined, we get the balanced training data set X′=X nor ∪X′ abn ∪X syn ;

[0017] (4) Use the balanced data set to train a random forest model to detect abnormal samples in the power dispatch monitoring data. Specifically:

[0018] Build a random forest model model, which contains the following parameters: the number of base classifiers n_estimators, the basis for determining the split point criterion, the maximum depth max_depth, the minimum number of samples per split min_samples_split, and the minimum number of samples for leaf nodes min_samples_leaf;

[0019] Use the balanced data set X′ to train it and obtain the trained random forest model model; based on the anomaly detection model model, for a sample point x to be tested test Perform anomaly detection and convert x test Input the model, and the model will output its label y test , if y test =0, then x test is a normal sample, if y test =1, then x test is an abnormal sample; based on this, anomaly detection of power dispatch monitoring data is realized.

[0020] In step (1) of the above method, the value of k is 3ln(N) retained to an integer, where N is the total number of samples in the input data set X.

[0021] In step (4) of the above method, the parameters of the random forest model model are: the number of base classifiers n_estimators = 100, the criterion for determining the split point is the gini coefficient, the maximum depth max_depth = 8, the minimum number of samples for each split min_samples_split = 2, and the minimum number of samples for leaf nodes min_samples_leaf = 1.

[0022] The power dispatch monitoring data anomaly detection method improves the anomaly detection accuracy of the power dispatch monitoring data.

[0023] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0024] In the technical solution implemented by the present invention, a k-nearest neighbor search algorithm is designed to adaptively identify different small clusters in abnormal samples, and then the small cluster information is used to filter noise and calculate the number of samples that need to be balanced in each cluster, and the data is balanced through a sample generation method. Finally, the balanced data set is used to train a random forest model and detect anomalies, thereby improving the accuracy of anomaly detection in power dispatch monitoring data.

Brief Description of the Drawings

[0025] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 It is a schematic diagram of the framework flow of a method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering proposed by the present invention;

[0027] Figure 2 is a schematic diagram of performing iterative nearest neighbor search for an abnormal sample;

[0028] Figure 3 It is a schematic diagram of the input data and output results of the algorithm of the present invention. [Specific implementation method]

[0029] In order to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0030] It should be clear that the described embodiments of the invention are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments of the invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the invention.

[0031] The present invention provides a method for detecting anomalies in power dispatch monitoring data based on nearest neighbor search clustering. In order to meet the anomaly detection requirements of power dispatch monitoring data, the present invention designs a nearest neighbor search clustering algorithm to identify small clusters of abnormal samples and noise, uses linear interpolation to generate samples to balance the data set, and uses the balanced data set to train a random forest model to detect anomalies in power dispatch monitoring data.

[0032] Figure 1 The present invention is a schematic diagram of a framework flow of a method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering. The method comprises the following steps:

[0033] Step 101: input the historical data of power dispatch monitoring with positive abnormal labels as a training data set, and find the k nearest neighbors of the abnormal samples in the feature space by calculating the Euclidean distance between the samples, specifically:

[0034] The real-time resource occupancy data of the processes related to the power dispatching system business collected by the power dispatching monitoring system is used as the training data set input, denoted as X = {x 1 ,x 2 ,x 3 ,...,xi ,...,x N}, where x i ∈R u , i=1,2,...,N, R represents a real number set, u=6 represents the dimension of the data, including process CPU occupancy, memory occupancy, disk IO, network IO, number of threads, number of network connections; the total number of samples in the input data set X is N; the positive anomaly label of the training data set X is input, recorded as y={y 1 ,y 2 ,y 3 ,...,y i ,...,y N}, which corresponds one-to-one to the samples in X, y i ∈{0,1}, where y i =0 represents x i is a normal sample, y i =1 represents x i is an abnormal sample, i=1,2,...,N.

[0035] y i =0 for all x i Denote the normal sample set X of the training set nor , y i =1 for all x i Denote the abnormal sample set X of the training set abn , calculate X abn The i-th sample x in abn_i The Euclidean distance d(x abn_i ,x j ), x j ∈X, and take the distance x abn_i The most recent k samples, denoted as x abn_i The nearest neighbor set kNN(x abn_i ), where the value of k is 3ln(N), and N is the total number of samples in the input data set X.

[0036] Step 102, determine whether the sample is noise or belongs to a certain abnormal sample cluster through the neighbor labels of the abnormal sample, and iterate and search in the sample neighbors in sequence until no more abnormal samples belonging to the cluster are found, specifically:

[0037] Traverse X abn For all abnormal samples in , for each abnormal sample x abn_i First, determine whether it has been marked as belonging to an abnormal cluster. If x abn_i If x is marked, skip the sample. abn_i If it has not been marked, then get x abn_iThe nearest neighbor set kNN(x abn_i ); if kNN(x abn_i ) all belong to the normal sample set X nor ,Right now x a ∈X nor , then x abn_i Labeled as noise sample set X noise In the sample, if kNN(x abn_i ) contains abnormal samples, that is, x b ∈X abn , then x abn_i With kNN(x abn_i ) are marked as the nth cluster C n samples, where n = 1, 2, 3, ...; Figure 2 Shows that for sample x abn_i Schematic diagram of the iterative nearest neighbor search process.

[0038] Then, for kNN(x abn_i ) all abnormal samples x abn_j , get its nearest neighbor kNN(x abn_j ), if there are unlabeled abnormal samples, they will also be marked as the nth cluster C n samples and continue to perform k-nearest neighbor search until all found abnormal samples have been labeled.

[0039] So far, for x abn_i The recursive marking ends, let n = n + 1, and continue to traverse the abnormal samples until all abnormal samples are marked as belonging to a certain abnormal cluster or to the noise sample set X noise So far, let the final value of n be M.

[0040] Step 103, filter the noise of the clustered data and calculate the number of abnormal samples that need to be generated in each cluster, and then use SMOTE linear interpolation to synthesize new samples in each cluster to balance the data set, specifically:

[0041] For X abn The abnormal sample x in abn_i , if it belongs to the noise sample set X noise , then it is removed from X abn Delete it and get the denoised abnormal sample set X′ abn =X abn -X noise ; Then, calculate each abnormal cluster C m The number of abnormal samples that need to be generated m =num Cm×(IR-1), where m = 1, 2, ..., M, num Cm The abnormal cluster C m The number of samples, IR is the imbalance rate of the entire data set, that is, |X nor | represents the normal sample set X nor The number of samples, |X′ abn | represents the denoised abnormal sample set X′ abn The number of samples.

[0042] Use the SMOTE algorithm to generate new abnormal samples in each cluster. Specifically, for an abnormal cluster C m , randomly select one of the abnormal samples x Cm_i ∈C m , in x Cm_i The nearest neighbor set kNN(x Cm_i ) randomly selects a sample x s ∈kNN(x Cm_i ), generate a new sample x by linear interpolation syn =x Cm_i +w×(x s -x Cm_i ), where w is a random parameter between [0,1]; all generated samples x syn Denoted as the generated sample set X syn , and compare it with the normal sample set X nor And the denoised abnormal sample set X′ abn Combined, we get the balanced training data set X′=X nor ∪X′ abn ∪X syn .

[0043] Step 104, using the balanced data set to train a random forest model to detect abnormal samples in the power dispatch monitoring data, specifically:

[0044] A random forest model model is established, which contains the following parameters: the number of base classifiers n_estimators, the basis for determining the split point criterion, the maximum depth max_depth, the minimum number of samples per split min_samples_split, and the minimum number of samples for leaf nodes min_samples_leaf.

[0045] Use the balanced data set X′ to train it and obtain the trained random forest model model; based on the anomaly detection model model, for a sample point x to be tested test Perform anomaly detection and convert x testInput the model, and the model will output its label y test , if y test =0, then x test is a normal sample, if y test =1, then x test is an abnormal sample; based on this, anomaly detection of power dispatch monitoring data is realized.

[0046] Figure 3 It is a schematic diagram of the input data and output results of the algorithm of the present invention. The input of the algorithm of the present invention is the real-time resource occupancy data of the processes related to the power dispatching system business collected by the power dispatching monitoring system, including the process CPU occupancy rate, memory occupancy rate, disk IO, network IO, number of threads, and number of network connections; the output of the algorithm of the present invention is the category label issued by the method model for each sample, 1 for abnormality and 0 for normal.

[0047] Algorithm 1 is the pseudo code of the clustering algorithm based on k-nearest neighbor search:

[0048]

[0049]

[0050] Algorithm 2 is the pseudo code of the sample generation method based on nearest neighbor search clustering:

[0051]

[0052] For the specific embodiment, 15 public data sets are used for testing. The data sets come from various fields, and all features are standardized after maximum and minimum scaling. The specific information of the data set is shown in Table 1. In order to ensure the reliability of the experimental results, this experiment will adopt a 5-fold cross-validation method, that is, the data set is divided into 5 folds, 4 folds are taken as training sets to train the classifier, and the remaining 1 fold is used as a test set to test the classification effect of the classifier. After repeating 5 times, the average value is taken as the experimental result of this 5-fold cross-validation method. Then 10 such 5-fold cross-validation methods are performed, and their average value is taken as the final experimental result.

[0053] Table 1 Datasets used in specific examples

[0054] Dataset Sample size Number of features Number of abnormal samples Normal sample size Imbalance rate glass1 214 9 76 138 1.82 wisconsin 683 9 239 444 1.86 pima 768 8 268 500 1.87 glass 214 9 70 144 2.06 haberman 306 3 81 225 2.78 vehicle2 846 18 218 628 2.88 ecoli1 336 7 77 259 3.36 new-thyroid1 215 5 35 180 5.14 yeast3 1484 8 163 1321 8.1 yeast-0-5-6-7-9 528 8 51 477 9.35 vowel0 988 13 90 898 9.98 yeast-1vs7 459 7 30 429 14.3 yeast-2_vs_8 482 8 20 462 23.1 yeast5 1484 8 44 1440 32.73 ecoli-0-1-3-7vs2-6 281 7 7 247 39.14

[0055] To verify the effectiveness of the proposed algorithm, the comparison methods selected in this experiment include 10 classic or time-effective sample generation methods that are currently more mainstream: SMOTE, Borderline-SMOTE1, Borderline-SMOTE2, Kmeans-SMOTE, Safe_Level_SMOTE, SMOTE_OUT, G-SMOTE, SMOTE_D, SOMO, and ADASYN. The proposed method is represented by SC-SMOTE. In order to be fair and unified, the neighbor parameter k selected by all methods for interpolation is the same, which is 3ln(N) retained to an integer, where N is the total number of samples in the input data set X. All methods are combined with random forests after balancing the data set.

[0056] In order to take into account the classification accuracy and recall rate of normal samples and abnormal samples, this experiment will use F-measure and G-mean as evaluation indicators. Specifically, for the binary anomaly detection problem, the confusion matrix represents the correctly classified samples and the incorrectly classified samples, as shown in Table 2:

[0057] Table 2 Confusion matrix for binary classification problem

[0058] Samples predicted to be positive Samples predicted as negative True positive samples TP FN True negative samples FP TN

[0059] In the case of class imbalance, abnormal samples are considered as positive samples and normal samples are considered as negative samples. In binary classification problems, many performance metrics can be calculated using the confusion matrix. Among the classic classifier performance metrics, recall, also known as sensitivity, represents the proportion of correctly classified samples in the positive class. Therefore, the definition of recall is as follows:

[0060]

[0061] In contrast, accuracy, also known as confidence, represents the number of samples that are actually positive among all samples predicted to be positive. Therefore, the definition of accuracy is as follows:

[0062]

[0063] F-measure combines recall and precision to obtain an effective metric to measure classification performance. It is the harmonic mean of recall and precision. The definition of F-measure is as follows:

[0064]

[0065] In order to consider the recall rates of normal samples and abnormal samples at the same time, this chapter uses G-mean as the second evaluation indicator, which is the geometric mean of the recall rates of positive and negative classes. Therefore, the definition of G-mean is as follows:

[0066]

[0067] The F-measure results of the embodiments of the present invention and the comparative methods on the public data sets are shown in Table 3. It can be seen that the power dispatch monitoring data anomaly detection method based on nearest neighbor search clustering of the present invention has obtained an F-measure that exceeds other methods on most public data sets, and has obtained the highest average F-measure and the lowest average rank.

[0068] Table 3 F-measure results of the comparison methods on the public dataset

[0069]

[0070]

[0071] The G-mean results of the embodiments of the present invention and the comparative methods on the public data sets are shown in Table 3. It can be seen that the power dispatch monitoring data anomaly detection method based on nearest neighbor search clustering of the present invention has obtained a G-mean that exceeds other methods on most public data sets, and has obtained the highest average G-mean and the lowest average rank. Combining the results of Tables 3 and 4, it can be seen that the anomaly detection accuracy of the method of the embodiment of the present invention is high.

[0072] Table 4 G-mean results of the comparison methods on the public dataset

[0073]

[0074]

[0075] The embodiments of the present invention are also applied to three types of business anomalies of smart grid dispatching and control systems, which are data jump, application disconnection, and non-refresh of telemetry meters.

[0076] Tables 5 and 6 respectively show the F-measure and G-mean results of the embodiments of the present invention and other comparative methods on these three anomalies.

[0077] Table 5 Experimental results of business data set under F-measure indicator

[0078] Data jump Application disconnection Telemetry table not refreshing Baseline 0.9772 0.9902 0.9920 SMOTE 0.9729 0.9799 0.9753 B1-SMOTE 0.9783 0.9918 0.9877 B2-SMOTE 0.9778 0.9840 0.9760 Kmeans-SMOTE 0.9709 0.9943 0.9792 Safe_Level_SMOTE 0.9795 0.9767 0.9787 SMOTE_OUT 0.9818 0.9818 0.9817 G-SMOTE 0.9775 0.9819 0.9816 SMOTE_D 0.9888 0.9760 0.9791 SOMO 0.9874 0.9910 0.9893 ADASYN 0.9703 0.9825 0.9936 SC-SMOTE 0.9927 0.9954 0.9959

[0079] Table 6 Experimental results of business data set under G-mean indicator

[0080] Data jump Application disconnection Telemetry table not refreshing Baseline 0.9835 0.9779 0.9859 SMOTE 0.9771 0.9927 0.9914 B1-SMOTE 0.9864 0.9935 0.9850 B2-SMOTE 0.9839 0.9909 0.9813 Kmeans-SMOTE 0.9878 0.9791 0.9827 Safe_Level_SMOTE 0.9858 0.9849 0.9918 SMOTE_OUT 0.9705 0.9874 0.9808 G-SMOTE 0.9749 0.9931 0.9925 SMOTE_D 0.9892 0.9844 0.9767 SOMO 0.9866 0.9826 0.9846 ADASYN 0.9795 0.9932 0.9841 SC-SMOTE 0.9918 0.9976 0.9956

[0081] It can be seen from Table 5 and Table 6 that compared with all the selected comparison methods, the power dispatch monitoring data anomaly detection method based on nearest neighbor search clustering of the present invention has achieved better results. The comparison results on the anomalies of three actual power dispatch monitoring data combined with the comparison results on a large number of public data sets show that the present invention can effectively reduce the false alarm rate and improve the accuracy when dealing with power dispatch monitoring data anomaly detection with complex data distribution and small abnormal clusters.

[0082] In summary, the embodiments of the present invention have the following beneficial effects:

[0083] In the technical solution implemented by the present invention, the historical data of power dispatch monitoring with positive abnormal labels are input as the training data set, and the k nearest neighbors of the abnormal sample in the feature space are found by calculating the Euclidean distance between the samples; the nearest neighbor labels of the abnormal sample are used to determine whether the sample is noise or belongs to a certain abnormal sample cluster, and the sample neighbors are searched iteratively in sequence until no more abnormal samples belonging to the cluster are found; the noise is filtered out of the clustered data and the number of abnormal samples that need to be generated in each cluster is calculated, and new samples are synthesized in each cluster using SMOTE linear interpolation to balance the data set; the balanced data set is used to train the random forest model to detect abnormal samples in the power dispatch monitoring data. According to the technical solution provided by the embodiment of the present invention, when faced with data imbalance in the power dispatch monitoring data, and the data distribution is relatively complex, the distribution patterns are diverse, and the abnormal samples form small clusters, compared with other typical sample generation methods that balance the data set and then perform supervised anomaly detection methods, this method can improve the accuracy of anomaly detection in power dispatch monitoring data.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting anomaly in power dispatch monitoring data based on nearest neighbor search clustering, characterized in that: The steps include: (1) The historical data of power dispatch monitoring with positive anomaly labels are used as the training data set input, and the k nearest neighbors of the abnormal samples in the feature space are found by calculating the Euclidean distance between samples. Specifically, The real-time resource occupancy data of the processes related to the power dispatching system business collected by the power dispatching monitoring system is used as the training data set input, denoted as X = {x1, x2, x3, ..., x i ,...,x N }, where x i ∈R u , i=1,2,...,N, R represents a real number set, u=6 represents the dimension of the data, including process CPU usage, memory usage, disk IO, network IO, number of threads, and number of network connections; the total number of samples in the input data set X is N; the positive anomaly label of the training data set X is input, recorded as y={y1,y2,y3,...,y i ,...,y N }, which corresponds one-to-one to the samples in X, y i ∈{0,1}, where y i =0 represents x i is a normal sample, y i =1 represents x i is an abnormal sample, i=1,2,...,N; y i =0 for all x i Denote the normal sample set X of the training set nor , y i =1 for all x i Denote the abnormal sample set X of the training set abn , calculate X abn The i-th sample x in abn_i The Euclidean distance d(x abn_i ,x j ), x j ∈X, and take the distance x abn_i The most recent k samples, denoted as x abn_i The nearest neighbor set kNN(x abn_i ); (2) Determine whether the sample is noise or belongs to a certain abnormal sample cluster through the neighbor labels of the abnormal sample, and iterate the search in the sample's neighbors until no more abnormal samples belonging to the cluster are found. Specifically: Traverse X abn For all abnormal samples in , for each abnormal sample x abn_i First, determine whether it has been marked as belonging to an abnormal cluster. If x abn_i If x is marked, skip the sample. abn_i If it has not been marked, then get x abn_i The nearest neighbor set kNN(x abn_i ); if kNN(x abn_i ) all belong to the normal sample set X nor ,Right now Then x abn_i Labeled as noise sample set X noise In the sample, if kNN(x abn_i ) contains abnormal samples, that is, Then x abn_i With kNN(x abn_i ) are marked as the nth cluster C n samples, where n = 1, 2, 3, ...; Then, for kNN(x abn_i ) all abnormal samples x abn_j , get its nearest neighbor kNN(x abn_j ), if there are unlabeled abnormal samples, they will also be marked as the nth cluster C n samples and continue to perform k-nearest neighbor search until all found abnormal samples have been labeled; So far, for x abn_i The recursive marking ends, let n = n + 1, and continue to traverse the abnormal samples until all abnormal samples are marked as belonging to a certain abnormal cluster or to the noise sample set X noise So far, let the final value of n be M; (3) Filter the noise of the clustered data and calculate the number of abnormal samples that need to be generated in each cluster. Based on this, use SMOTE linear interpolation to synthesize new samples in each cluster to balance the data set. Specifically: For X abn The abnormal sample x in abn_i , if it belongs to the noise sample set X noise , then it is removed from X abn Delete it and get the denoised abnormal sample set X′ abn =X abn -X noise ; Then, calculate each abnormal cluster C m The number of abnormal samples that need to be generated m =num Cm ×(IR-1), where m = 1, 2, ..., M, num Cm The abnormal cluster C m The number of samples, IR is the imbalance rate of the entire data set, that is, |X nor | represents the normal sample set X nor The number of samples, |X′ abn | represents the denoised abnormal sample set X′ abn The number of samples; Use the SMOTE algorithm to generate new abnormal samples in each cluster. Specifically, for an abnormal cluster C m , randomly select one of the abnormal samples x Cm_i ∈C m , in x Cm_i The nearest neighbor set kNN(x Cm_i ) randomly selects a sample x s ∈kNN(x Cm_i ), generate a new sample x by linear interpolation syn =x Cm_i +w×(x s -x Cm_i ), where w is a random parameter between [0,1]; all generated samples x syn Denoted as the generated sample set X syn , and compare it with the normal sample set X nor And the denoised abnormal sample set X′ abn Combined, we get the balanced training data set X′=X nor ∪X′ abn ∪X syn ; (4) Use the balanced data set to train a random forest model to detect abnormal samples in the power dispatch monitoring data. Specifically: Build a random forest model model, which contains the following parameters: the number of base classifiers n_estimators, the basis for determining the split point criterion, the maximum depth max_depth, the minimum number of samples per split min_samples_split, and the minimum number of samples for leaf nodes min_samples_leaf; Use the balanced data set X′ to train it and obtain the trained random forest model model; based on the anomaly detection model model, for a sample point x to be tested test Perform anomaly detection and convert x test Input the model, and the model will output its label y test , if y test =0, then x test is a normal sample, if y test =1, then x test is an abnormal sample; based on this, anomaly detection of power dispatch monitoring data is realized.

2. The method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering according to claim 1 is characterized in that: In step (1), the value of k is 3ln(N) which is rounded to an integer, where N is the total number of samples in the input data set X.

3. The method for detecting anomaly in power dispatching monitoring data based on nearest neighbor search clustering according to claim 1 is characterized in that: In step (4), the parameters of the random forest model model are: the number of base classifiers n_estimators = 100, the criterion for determining the split point is the gini coefficient, the maximum depth max_depth = 8, the minimum number of samples for each split min_samples_split = 2, and the minimum number of samples for leaf nodes min_samples_leaf = 1.

Citation Information

Patent Citations

  • Power dispatching data exception detection method based on logarithm interval isolation

    CN112181706A

  • Aircraft lead-acid storage battery anomaly detection semi-supervised method based on isolation

    CN112329350A