Abnormal data evaluation and restoration method based on power system

Abstract

The invention relates to the technical field of power system data processing, and discloses a power system-based abnormal data evaluation and restoration method, which comprises the following steps of S1, data preprocessing; s2, establishing an abnormal data evaluation model; S2.1, identifying incomplete data by using a K-nearest neighbor algorithm; s2.2, identifying non-timely data by adopting a time sequence anomaly detection method based on a sliding window; s2.3, identifying inaccurate data through an isolated forest algorithm; s3, abnormal data is repaired; S3.1, a weighted mean value correction model is adopted to repair marked incomplete and inaccurate data; s3.2, repairing the delayed data by using the ARIMA (Autoregressive Integrated Moving Average) prediction model; and S4, fusion of correction results: forming comprehensive correction of different types of abnormal data by using a weighted ARIMA fusion correction model in combination with weighted mean correction and ARIMA prediction results. Through the multi-algorithm fusion strategy, limitation possibly brought by a single algorithm is avoided, and therefore the accuracy and pertinence of abnormal data identification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system data processing, and in particular to an abnormal data evaluation and repair method based on a power system. Background Art

[0002] In the power system, the quality of data directly affects the stability, prediction ability and decision-making efficiency of the system. With the increasing complexity and intelligence of the power system, massive amounts of power data are collected and transmitted in real time and used in key links such as load forecasting, dispatch optimization, and equipment monitoring. However, due to the complexity of the power data collection and transmission process, the emergence of abnormal data is inevitable. These abnormal data may appear due to various factors (such as sensor failure, communication delay, external environmental interference, etc.), which in turn affects the normal operation of the power system. Abnormal data in the power system can generally be divided into three categories: inaccurate data, incomplete data, and untimely data. These abnormal data not only affect the prediction accuracy of the power system, but may also lead to a series of problems such as wrong dispatching decisions, equipment fault detection errors, and reduced system stability.

[0003] Many existing methods rely on traditional statistical analysis methods, such as mean, variance analysis, interpolation, etc. Although these methods are simple and easy to use, they often fail to take into account the complex relationships and potential nonlinear characteristics between data when processing complex power data, and are easily interfered by noise. For various types of abnormal data (such as inaccurate, incomplete, untimely, etc.), traditional methods are usually unable to effectively distinguish and repair them, and the accuracy and pertinence of abnormal data identification are low, resulting in unsatisfactory repair effects. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an abnormal data evaluation and repair method based on a power system, which solves the problem that traditional methods are usually unable to effectively distinguish and repair, resulting in low accuracy and pertinence in abnormal data identification.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an abnormal data evaluation and repair method based on a power system comprises the following steps:

[0006] S1. Data preprocessing

[0007] S1.1. Perform missing value statistical analysis on the original power data, calculate the missing rate of each data point, and mark the data points whose missing rate exceeds the preset threshold as incomplete data;

[0008] S1.2. Standardize the features without missing values ​​and convert the feature values ​​into a standard normal distribution with a mean of 0 and a variance of 1.

[0009] S2. Abnormal data evaluation model

[0010] S2.1, using K-nearest neighbor algorithm to identify incomplete data;

[0011] S2.2, using a sliding window-based time series anomaly detection method to identify untimely data;

[0012] S2.3, Identify imprecise data through the isolation forest algorithm;

[0013] S3. Abnormal data repair

[0014] S3.1, using the weighted mean correction model to repair the incomplete and inaccurate data of the label;

[0015] S3.2, use the ARIMA forecasting model to repair untimely data;

[0016] S4. Correction result fusion

[0017] The weighted ARIMA fusion correction model is used to combine the results of weighted mean correction and ARIMA prediction to form a comprehensive correction for different types of abnormal data.

[0018] Preferably, in step S2.1, the K-nearest neighbor algorithm is used to calculate the weighted Euclidean distance between each data point and other data points to evaluate the similarity, wherein the weight of the missing feature is lower than the weight of the non-missing feature.

[0019] Preferably, the sliding window time series anomaly detection method in step S2.2 uses multiple fixed-size windows to segment the time series data, and detects untimely data in each window through the statistical characteristics of the mean and variance.

[0020] Preferably, the isolation forest algorithm in step S2.3 evaluates the degree of abnormality by generating an isolation tree and calculating the path length of each data point, and data points with shorter paths are marked as inaccurate data.

[0021] Preferably, the weighted mean correction model in step S3.1 assigns weights according to the distances of multiple data points adjacent to the abnormal data point, and the closer the distance, the higher the weight of the neighbor.

[0022] Preferably, the orders p, d, q of the ARIMA forecasting model in step S3.2 are selected according to the autocorrelation and partial autocorrelation of the time series to best predict the untimely data.

[0023] Preferably, the weighted ARIMA fusion correction model in step S4 combines the smoothness of the weighted mean correction model and the time series prediction ability of the ARIMA model.

[0024] Preferably, after step S2, the repaired data is evaluated for accuracy of abnormal data identification through a confusion matrix, and the confusion matrix is ​​used to measure the degree of match between the predicted results of the statistical model and the actual results.

[0025] Preferably, after the step S4, the correction result is visualized and analyzed through a scatter plot, wherein the comparison between the identification label and the actual label is used to observe the correction effect.

[0026] The present invention provides an abnormal data evaluation and repair method based on a power system, which has the following beneficial effects:

[0027] 1. The present invention combines multiple algorithms such as KNN algorithm, sliding window time series anomaly detection algorithm and isolation forest algorithm, respectively targeting three types of abnormal data: incomplete, untimely and inaccurate. This multi-algorithm fusion strategy avoids the limitations that may be brought by a single algorithm, thereby improving the accuracy and pertinence of abnormal data identification.

[0028] 2. The present invention combines multiple machine learning and statistical methods such as KNN, sliding window time series anomaly detection, isolation forest, etc. to detect different types of abnormal data from multiple angles. Through the fusion of multiple algorithms, the limitations that may be caused by a single algorithm are avoided, and the accuracy of identifying various abnormal data types such as incomplete, untimely, and inaccurate is improved. This multi-level evaluation and repair method greatly improves the adaptability and robustness of the model in complex power system data.

[0029] 3. The present invention combines the weighted mean correction model with the ARIMA time series prediction model, and the method can provide an automated repair solution for different types of abnormal data. The weighted mean correction corrects incomplete or inaccurate data by smoothing data fluctuations, while the ARIMA model repairs untimely data through time series prediction. The automated repair process can process a large amount of power data quickly and efficiently, reduce manual intervention, and improve data processing efficiency.

[0030] 4. The present invention uses the ARIMA model to repair untimely data, which can ensure that the repaired data maintains good time series consistency. By accurately modeling time series data, it is ensured that the repaired data can be consistent with other data in the power system, which helps to maintain the timeliness and consistency of the system, thereby improving the accuracy of load forecasting, scheduling decisions and real-time monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A method flow chart of the abnormal data evaluation and repair method based on the power system of the present invention;

[0032] Figure 2 The triple evaluation flow chart of abnormal data of the present invention is as follows;

[0033] Figure 3 It is the weighted ARIMA fusion correction flow chart of the present invention;

[0034] Figure 4 It is a scatter diagram of the comparison data between the actual label and the identification label of the present invention;

[0035] Figure 5 It is a scatter diagram of the data correction results of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Please see attached Figure 1 The embodiment of the present invention provides an abnormal data evaluation and repair method based on a power system, comprising the following steps:

[0038] S1. Data preprocessing

[0039] S1.1. Perform missing value statistical analysis on the original power data, calculate the missing rate of each data point, and mark the data points whose missing rate exceeds the preset threshold as incomplete data;

[0040] S1.2. Standardize the features without missing values ​​and convert the feature values ​​into a standard normal distribution with a mean of 0 and a variance of 1.

[0041] S2. Abnormal data evaluation model

[0042] S2.1, using K-nearest neighbor algorithm to identify incomplete data;

[0043] S2.2, using a sliding window-based time series anomaly detection method to identify untimely data;

[0044] S2.3, Identify imprecise data through the isolation forest algorithm;

[0045] S3. Abnormal data repair

[0046] S3.1, using the weighted mean correction model to repair the incomplete and inaccurate data of the label;

[0047] S3.2, use the ARIMA forecasting model to repair untimely data;

[0048] S4. Correction result fusion

[0049] The weighted ARIMA fusion correction model is used to combine the results of weighted mean correction and ARIMA prediction to form a comprehensive correction for different types of abnormal data.

[0050] Specifically, S1. Data preprocessing

[0051] S1.1. Statistical analysis of missing values: Before evaluating the raw data of the power system, the missing values ​​must be processed first. By performing missing rate statistics on each data point and each feature, the missing rate is calculated and marked. The missing rate refers to the proportion of missing features in each data point. Set a missing rate threshold (for example, 30%). If the missing rate of a data point exceeds the threshold, the data point will be marked as "incomplete data". If the missing rate is lower than the threshold, the data can be further processed.

[0052] S1.2, Feature Standardization: In the process of data preprocessing, in order to ensure that the algorithms in the subsequent steps can effectively process the data, it is necessary to standardize the non-missing features. The purpose of standardization is to convert the data values ​​of different features into data with a uniform scale. The standardization formula is as follows:

[0053]

[0054] where μ k and σ k are the mean and standard deviation of the kth feature respectively.

[0055] In step S2.1, the K-nearest neighbor algorithm is used to calculate the weighted Euclidean distance between each data point and other data points to evaluate the similarity, where the weight of missing features is lower than the weight of non-missing features.

[0056] Specifically, K-nearest neighbor algorithm (KNN) is a distance-based unsupervised learning method used to determine whether a data point is abnormal. In this method, KNN is used to identify incomplete data. Similarity is evaluated by calculating the distance between a data point and other data points. Specifically, the Euclidean distance between each data point and other data points is first calculated, ignoring the dimension of missing features. Then K neighbors with the smallest distance are selected, and the missing value of the data point is predicted based on the feature values ​​of these neighbors. Weight calculation can be achieved by weighted Euclidean distance, in which neighbors closer to the data point will be given a higher weight.

[0057] The sliding window time series anomaly detection method in step S2.2 uses multiple fixed-size windows to segment the time series data, and detects untimely data in each window through the statistical characteristics of the mean and variance.

[0058] Specifically, for time series data in power data, there is usually a delay or lag. In order to identify untimely data, an anomaly detection method based on a sliding window is adopted. This method divides the time series data into multiple windows of fixed size, each of which contains a certain number of time steps. The data in each window will be calculated through statistical analysis (such as mean, variance) to calculate its overall trend. According to the set Z-Score threshold, anomaly detection is performed on each data point in the window. If the Z-Score value of a data point exceeds the set threshold, the point is considered to be abnormal "untimely data".

[0059] The isolation forest algorithm in step S2.3 evaluates the degree of abnormality by generating an isolation tree and calculating the path length of each data point. Data points with shorter paths are marked as inaccurate data.

[0060] Specifically, Isolation Forest is an unsupervised learning method based on random tree generation, which is suitable for anomaly detection of high-dimensional data. In the present invention, Isolation Forest is used to identify imprecise data. By generating multiple random trees and partitioning the data space, the abnormality of each data point is evaluated. Data points with shorter paths are more easily isolated, indicating that these points are abnormal, that is, imprecise data. The abnormality of each data point is measured by the ease with which it is isolated, and points with shorter paths are considered abnormal.

[0061] The weighted mean correction model in step S3.1 assigns weights according to the distances of multiple data points adjacent to the abnormal data point, and the closer the distance, the higher the weight of the neighbor.

[0062] Specifically, for incomplete and inaccurate data, a weighted mean correction model is used to repair them. The repair method fills in missing values ​​or corrects outliers by calculating the weighted mean of neighboring data points. The weighted mean of each data point is determined by the characteristic value and distance of its neighboring data points. The closer the neighbor to the data point, the greater the weight.

[0063] The order p, d, q of the ARIMA forecasting model in step S3.2 is selected based on the autocorrelation and partial autocorrelation of the time series to best predict the untimely data.

[0064] Specifically, ARIMA time series forecasting: For untimely data, the ARIMA model is used to repair it. The ARIMA model is a model widely used for time series forecasting, which has three parts: autoregression (AR), difference (I) and moving average (MA). First, by analyzing the historical data, the appropriate model order (p, d, q) is selected, where p represents the autoregression term, d represents the number of differences, and q represents the moving average term. By fitting historical data, the ARIMA model can predict missing time series data, thereby supplementing the untimely data points.

[0065] The weighted ARIMA fusion correction model in step S4 combines the smoothness of the weighted mean correction model and the time series prediction ability of the ARIMA model.

[0066] Specifically, in the correction process, the advantages of weighted mean correction and ARIMA forecasting are combined. For incomplete and inaccurate data, the weighted mean correction method is used to smooth data fluctuations; for untimely data, the ARIMA forecasting model is used to fill in the missing data. Through this fusion method, the characteristics of the two models can be fully utilized to improve the accuracy and continuity of data repair. The fusion correction process can balance the contributions of the two correction methods by setting weight coefficients to ensure that the final result is both smooth and in line with the characteristics of the time series.

[0067] After step S2, the repaired data is evaluated for the accuracy of abnormal data identification through the confusion matrix, which is used to measure the matching degree between the predicted results of the statistical model and the actual results.

[0068] Specifically, the corrected data is evaluated through the confusion matrix. The confusion matrix is ​​used to compare the difference between the model prediction and the actual data, and calculate the accuracy, recall, precision and other indicators to evaluate the accuracy of the recognition process. For each type of abnormal data (incomplete, untimely, inaccurate data), the confusion matrix is ​​used to analyze the recognition effect of the model on these data.

[0069] After step S4, the correction results are visualized and analyzed through scatter plots, in which the comparison between the identified labels and the actual labels is used to observe the correction effect.

[0070] Specifically, the corrected data is compared with the original data, and a scatter plot is drawn for visual analysis. By observing the distribution of data before and after correction, the correction effect can be intuitively displayed. For example, normal data and corrected data should be consistent, and abnormal data should show a trend similar to normal data after correction.

[0071] The following is an introduction in conjunction with specific embodiments:

[0072] S1. Data preprocessing

[0073] Before evaluation, missing value processing is usually required to avoid the impact of missing values ​​on identification and prediction models. However, directly filling in missing values ​​will affect the identification effect of incomplete data. Therefore, we choose to perform missing value statistical analysis and marking. By calculating the missing rate for each data point, we preliminarily mark those points whose missing rate exceeds the set threshold for reference. Then, in order to ensure the consistent feature scale required by the subsequent algorithm, the non-missing features are standardized.

[0074] Missing value statistical analysis

[0075] The missing rate of each data point and feature is counted to determine the data with serious missing information, which helps KNN to identify incomplete data more accurately. i Each feature dimension x in i , k, count the missing rate of this point:

[0076]

[0077] According to the set threshold, data points with missing rates exceeding the threshold, i.e., data points with more missing data, are preliminarily marked as incomplete data for reference.

[0078] Feature Standardization

[0079] For the existing complete features, standardization or normalization can be performed to adapt to the scale requirements of algorithms such as KNN and isolation forest in the later identification process. The standardization formula is as follows:

[0080]

[0081] where μ k and σ k are the mean and standard deviation of the kth feature respectively.

[0082] S2. Abnormal data evaluation model

[0083] This paper builds a comprehensive evaluation model for abnormal data by combining a multi-stage processing flow to gradually filter and identify anomalies in the data. These algorithms can be applied to the input data in sequence, in the order of KNN to identify incomplete data, sliding window time series anomaly detection to identify untimely data, and isolation forest to identify inaccurate data. Each step filters the data in layers and identifies and marks the type. In this combined process, the output of each step can be used as the input of the next step to improve the efficiency and accuracy of the entire system. Since the identified abnormal data needs to be visualized, the abnormal data is temporarily only marked without filling in this stage. After the process is completed, each data point is labeled and output according to the incomplete, untimely, and inaccurate anomaly type. The comprehensive evaluation model is as follows Figure 2 shown.

[0084] Multi-stage processing flow principle

[0085] Multi-stage processing flows are often used in complex systems or data processing, where tasks are split into multiple independent but orderly steps. The tasks of each stage are carried out based on the output results of the previous stage, gradually approaching the final goal, so that each stage focuses on specific tasks, reduces complexity, and can more flexibly adjust the settings and parameters of each stage, ensuring that the operation of each stage is clear and modular, which is convenient for optimization, monitoring and troubleshooting.

[0086] A multi-stage process is usually represented as a directed graph, where nodes represent stages and edges represent dependencies between stages. Suppose a task flow consists of n stages, each stage S i Requires the previous stage S i-1 The output of is used as input:

[0087] s i =f i (S i-1 (x))(3)

[0088] Among them, f i Represents the processing function of the i-th stage, and x is the input data.

[0089] Phase 1 (KNN algorithm evaluates incomplete data)

[0090] K-nearest neighbor algorithm (KNN) is a simple and effective machine learning algorithm, mainly used for classification and regression tasks. It is based on the following assumption: similar input points usually have similar outputs. After inputting the data points in the feature space, KNN calculates the distance between the sample to be classified and the known samples, selects K nearest neighbor samples for classification, and finally outputs the classification label.

[0091] After the missing value statistical analysis in the preprocessing process, the KNN identification stage is entered to identify more incomplete data points through the integrity of neighboring points. i , ignoring the dimension of missing features, only calculating the Euclidean distance (or weighted distance) on the observed features, and finding the K nearest neighbor data points:

[0092]

[0093] Among them, ω k =0 indicates missing value weight, ω k =1 indicates normal value weight.

[0094] Based on the KNN algorithm and further marking incomplete data according to the similarity with neighbor data, if one or more data points x i If the missing rate is high or the difference with the adjacent data is large (that is, it is impossible to find enough neighbors), it is marked as incomplete data.

[0095] Phase 2 (Sliding window-based time series anomaly detection to evaluate untimely data)

[0096] The time series anomaly detection algorithm based on sliding windows is a commonly used detection method. It uses sliding windows to segment time series and analyze them to find abnormal points that are different from normal behavior. The basic idea is to detect whether the data in a window range deviates from the normal behavior pattern, so as to identify untimely data.

[0097] Sliding window is a segmentation method that divides time series data into multiple consecutive windows, each containing a certain number of time steps. Each time the statistical characteristics of the data in the window are calculated, the window moves forward one or more steps. Assume that the time series is divided into window size W and step size S, and the data in each window is represented as X i :

[0098] X i ={x iS+1 , x iS+2 , …, x iS+W} (5)

[0099] In each window, the temporal changes of data are captured by extracting time series features (such as mean, variance, trend, cycle, etc.). Calculate the mean μ in each window i and variance σ i :

[0100]

[0101]

[0102] For time series data Differential Trend Indicator T i It can be expressed as:

[0103] T i =x t -x t-1 (t=2,3,…,T) (8)

[0104] In the sliding window, the average difference value of the data points in the window is calculated to measure the overall trend of the window:

[0105]

[0106] Based on the extracted features, the time series anomaly detection algorithm, namely the Z-Score detection method, is used to identify and mark untimely data. For each data point x in the window t , calculate its Z-Score, and detect untimely data based on the set threshold:

[0107]

[0108] If | Z t |>Zthreshold , then this point is untimely data.

[0109] Phase 3 (Isolation Forest Algorithm Evaluates Imprecise Data)

[0110] Isolation Forest is an unsupervised machine learning algorithm for anomaly detection, centered around the core idea that outliers are easier to isolate. Compared to normal points, outliers are sparser in feature space and are therefore easier to isolate with a few segmentation steps. The Isolation Forest algorithm achieves this isolation by building multiple random trees and judging the degree of anomaly of each data point based on how easy it is to isolate it. In this stage, the Isolation Forest algorithm estimates the upper limit of the error and measures the error to identify and mark imprecise data.

[0111] Isolation forest generates isolated trees by recursively and randomly dividing the feature space, which is used to calculate the path length h(x) of the data point to evaluate the degree of abnormality. The expected value of the path length is calculated, and the average path length c(n) of the data set size n in the isolation forest is approximately:

[0112]

[0113] Here, γ≈0.577215 is the Euler constant.

[0114] For each data point, Isolation Forest estimates the upper limit of the error based on its path length h(x). Points with shorter paths are often more abnormal. Combined with the path length range of normal points, the upper limit of the error threshold E can be defined. threshold The degree of deviation is measured by calculating the anomaly score s(x, n) of each data point. Points exceeding the upper error threshold are inaccurate data and are marked.

[0115]

[0116] If s(x,n)>E threshold , then mark the point as inaccurate data.

[0117] S3. Abnormal data repair

[0118] The comprehensive evaluation model shown in the above S2 section completes the identification and classification of three types of abnormal data and enters the correction process. For inaccurate and incomplete data, the weighted mean correction model is used to gradually smooth abnormal fluctuations and fill missing values. Then, for untimely data, the ARIMA forecasting model is applied to correct the missing data for time series forecasting to ensure the temporal consistency of the data. Finally, the corrected data is integrated and output as a corrected complete data set. The weighted ARIMA fusion correction process is as follows: Figure 3 .

[0119] Weighted mean correction model

[0120] The weighted mean correction model is used to deal with inaccurate and incomplete data, and smooth or fill in abnormal data through weighted average. Assume that the time series data is There are inaccurate or incomplete observations, denoted by x t异常 For inaccurate or incomplete data points x t异常 , using the weighted mean of its neighboring k data points as the correction value, the weighted mean can be expressed as:

[0121]

[0122] Among them, ω i is the weight of the ith neighboring data point (generally it can be adjusted according to the degree of proximity, for example, the closer to the outlier, the greater the weight). Then select the weight. If the data fluctuation is small, you can use uniform weight and set all weights. Higher weights are given to data points that are closer in time to the outlier point. Where α is the attenuation factor, |it 异常 | represents the distance to the outlier point.

[0123] ARIMA forecasting model

[0124] The ARIMA forecasting model is used to process untimely data, that is, delayed or missing data in the time series. Assume that the untimely outlier point is x t异常 , the ARIMA model can be used to predict x by fitting the ARIMA model and using its prediction function and previous data t异常 The ARIMA model is expressed as ARIMA(p, d, q). For an original time series x t ,y t is a stationary series after difference, and the ARIMA model can be written as:

[0125]

[0126] The weighted mean focuses on the smoothness and continuity of data and is suitable for random anomalies, while ARIMA targets trend and cyclical data and is suitable for untimely data correction. The weighted ARIMA fusion correction model, which combines the smoothing characteristics of the weighted mean and the time series prediction ability of ARIMA, can more comprehensively correct different types of abnormal data and avoid the limitations of a single model. For imprecise data with large noise, the weight of the weighted mean can be appropriately increased; and for missing data with strong time dependence, the parameters of ARIMA can be adjusted to improve the prediction accuracy.

[0127] S4. Correction result fusion

[0128] Confusion matrix to evaluate recognition accuracy

[0129] The confusion matrix is ​​a tool for evaluating the performance of classification models, especially for binary classification problems, and can also be extended to multi-classification problems. It provides statistical information of four results by comparing the matching between the predicted results of the statistical model and the true labels: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). This information can help analyze the prediction accuracy, error rate, and deviation of the model:

[0130] TP: The model correctly predicts a sample as positive when it is actually positive.

[0131] FP: The model mistakenly predicts a sample that is actually negative as positive (also called "false positive" or "false alarm").

[0132] FN: The model mistakenly predicts samples that are actually positive as negative (also called "false negatives" or "missing reports").

[0133] TN: The model correctly predicts a sample as negative when it is actually negative.

[0134] The confusion matrix is ​​used to calculate the accuracy of recognition, that is, the proportion of samples predicted correctly.

[0135]

[0136] Relative error evaluation correction effect

[0137] Relative Error is a measure of the deviation between the predicted value or the corrected value and the true value relative to the true value. When evaluating the effect of data correction, the relative error can help determine how close the corrected data is to the standard value, thereby evaluating the quality of the correction. This article sets the correction as successful if the relative error does not exceed 0.6%.

[0138] Experimental results analysis

[0139] Scatter plot analysis of identification results

[0140] The scatter plot is drawn with the x-axis as the data label and the y-axis as the data value to achieve the purpose of visual analysis of the recognition results. For the presentation of the recognition results, this article uses the comparison between the recognition label and the actual label, and observes the color of the data point. If the two labels of a data point correspond, it will be displayed in black in the figure. If the two labels do not correspond, they will be other colors, such as Figure 4 .

[0141] As shown in the figure, there are 96 black points, which represent data points that have been successfully identified in four categories of data (normal data, inaccurate data, untimely data, and incomplete data). The rest are data points that have been incorrectly identified. There is a blue point in the figure, which represents data that is actually normal but identified as inaccurate data, a green point, which represents data that is actually normal but identified as untimely data, and two pink points, which represent data that is actually untimely but both identified as normal data. It can also be seen from the figure that incomplete data is easier to identify, and there are no wrong points.

[0142] From the identification results, we can get: TP = 44, FP = 2, FN = 2, TN = 52, from which we can calculate the identification accuracy:

[0143]

[0144] Scatter plot analysis of correction results

[0145] The scatter plot is drawn with the x-axis as the data label and the y-axis as the correction value to achieve the purpose of visual analysis of the correction results. For the presentation of the correction results, this article uses the correction value of the data to compare with the data requirements. If the value after correction meets the requirements, it means that the correction is successful, and it is black. If the value does not meet the requirements, it means that the correction is not very successful or even failed, and it is other colors, such as Figure 5 .

[0146] As shown in the figure, 97 points are black, which represent data points that have been successfully corrected. The rest are data points that have failed to be corrected, including two blue points, which are incomplete data before correction and whose values ​​after correction deviate from the normal data range, and one yellow point, which represents untimely data before correction and whose values ​​after correction deviate from the normal required data range. It can also be seen from the figure that incomplete data is prone to prediction results that do not meet the requirements during the prediction process, while the other two types of abnormal data are not prone to errors when repaired.

[0147] From the correction results, it can be obtained that when Y=229.5, X1=228.08, X2=228.04, X3=228.12, the relative errors of these three data are:

[0148]

[0149]

[0150]

[0151] It can be verified from the calculation results that the correction results corresponding to the three points of different colors in the figure are consistent with the unsuccessful results.

[0152] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An abnormal data evaluation and repair method based on a power system, characterized in that: The following steps are involved: S1. Data preprocessing S1.

1. Perform missing value statistical analysis on the original power data, calculate the missing rate of each data point, and mark the data points whose missing rate exceeds the preset threshold as incomplete data; S1.

2. Standardize the features without missing values ​​and convert the feature values ​​into a standard normal distribution with a mean of 0 and a variance of 1. S2. Abnormal data evaluation model S2.1, using K-nearest neighbor algorithm to identify incomplete data; S2.2, using a sliding window-based time series anomaly detection method to identify untimely data; S2.3, Identify imprecise data through the isolation forest algorithm; S3. Abnormal data repair S3.1, using the weighted mean correction model to repair the incomplete and inaccurate data of the label; S3.2, use the ARIMA forecasting model to repair untimely data; S4. Correction result fusion The weighted ARIMA fusion correction model is used to combine the results of weighted mean correction and ARIMA prediction to form a comprehensive correction for different types of abnormal data.

2. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: In step S2.1, the K-nearest neighbor algorithm is used to calculate the weighted Euclidean distance between each data point and other data points to evaluate the similarity, wherein the weight of the missing feature is lower than the weight of the non-missing feature.

3. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: The sliding window time series anomaly detection method in step S2.2 uses multiple fixed-size windows to segment the time series data, and detects untimely data in each window through the statistical characteristics of the mean and variance.

4. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: The isolation forest algorithm in step S2.3 generates an isolation tree and calculates the path length of each data point to evaluate its abnormality, and the data points with shorter paths are marked as inaccurate data.

5. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: The weighted mean correction model in step S3.1 assigns weights according to the distances of multiple data points adjacent to the abnormal data point, and the closer the distance, the higher the weight of the neighbor.

6. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: The orders p, d, q of the ARIMA forecasting model in step S3.2 are selected according to the autocorrelation and partial autocorrelation of the time series to best forecast the untimely data.

7. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: The weighted ARIMA fusion correction model in step S4 combines the smoothness of the weighted mean correction model and the time series prediction ability of the ARIMA model.

8. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: After step S2, the repaired data is evaluated for the accuracy of abnormal data identification through a confusion matrix, and the confusion matrix is ​​used to measure the matching degree between the predicted results of the statistical model and the actual results.

9. The abnormal data evaluation and repair method based on the power system according to claim 1 is characterized in that: After the step S4, the correction result is visualized and analyzed through a scatter plot, wherein the comparison between the identified label and the actual label is used to observe the correction effect.

Images