A method for outlier detection and cleaning of electricity consumption data based on DBSCAN and KNN algorithms

Abstract

This invention discloses a method for outlier detection and cleaning of electricity consumption data based on DBSCAN and KNN algorithms. The method includes: pre-filling missing values ​​in the electricity consumption data; performing outlier detection on the pre-filled data using both single-dimensional and multi-dimensional data based on an improved DBSCAN clustering algorithm to identify anomalous outlier values, and setting these outlier values ​​to null; then cleaning the electricity consumption data again using mathematical statistics and data mining rules; filling in missing values ​​using an improved KNN nearest neighbor algorithm; for the data after outlier correction, incorporating the influence of feature attributes into the KNN calculation process using principal component analysis to obtain the final estimated values; and finally evaluating the cleaned data using root mean square error. This invention achieves outlier detection and cleaning of user electricity consumption data, improves the quality of low-voltage user electricity consumption data, and ensures the reliability of subsequent research results on abnormal electricity consumption behavior of low-voltage users.

Description

Technical Field

[0001] This invention relates to the detection and processing of outliers in data, with a focus on methods for detecting outliers and filling missing values ​​in power data. Background Technology

[0002] With the continuous advancement of information technology construction in power grid companies, the amount of business data in ledgers has increased dramatically. Various data types, including relational data, text data, and real-time data, are experiencing massive growth, resulting in a large accumulation of business data. Due to the diverse types and large quantities of equipment involved in power data collection, inconsistent quality, equipment replacement, and errors and omissions caused by manual data entry, the quality of ledger data is generally low. The quality of this data directly impacts subsequent data analysis and modeling applications based on various business scenarios. Therefore, adopting automated and process-oriented methods for efficient and accurate data inspection is essential to improving data quality. Standardizing ledger data processing and replacing manual inspection with automated testing can improve the speed of data quality inspection in power companies and provide a solid data environment foundation for subsequent data analysis, mining, and model building. Summary of the Invention

[0003] The purpose of this invention is to provide a method for outlier detection and cleaning of low-voltage user electricity consumption data based on improved DBSCAN and KNN algorithms. By leveraging optimized data mining algorithms, it is possible to better achieve intelligent detection and identification of outliers and filling of missing values ​​in the data. After data cleaning, the data quality is improved.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] An outlier detection and cleaning method for electricity consumption data based on DBSCAN and KNN algorithms includes:

[0006] Pre-fill missing values ​​in electricity consumption data;

[0007] Outlier detection was performed on the pre-filled electricity consumption data for both single-dimensional data and multi-dimensional data based on the improved DBSCAN clustering algorithm to identify outlier values. Outlier values ​​were then set to null.

[0008] Then, mathematical statistics and data mining rules are used to clean the electricity consumption data again.

[0009] Missing values ​​are filled using an improved KNN nearest neighbor imputation algorithm: For data after outlier correction, principal component analysis is used to incorporate the influence of feature attributes into the KNN calculation process to obtain the final estimated value.

[0010] In a preferred embodiment of the present invention, the power consumption data is evaluated multiple times using the root mean square error, including but not limited to evaluating the power consumption data after detecting, removing, and filling missing values, using the root mean square error to evaluate it, and comparing the error between the processed power consumption data and the actual data.

[0011] In a preferred embodiment of the present invention, outlier detection of single-dimensional data based on statistical methods is performed by using one of the following techniques: descriptive statistical analysis, box plots, or Laida's rule, to detect outliers in single-dimensional electricity consumption data item by item.

[0012] In a preferred embodiment of the present invention, outlier detection of multidimensional data based on the improved DBSCAN clustering algorithm is performed by estimating the parameters of DBSCAN using kernel probability density: the Eps neighborhood radius and the MinPts density threshold, i.e., the minimum number of points within a cluster. The improved DBSCAN clustering algorithm is then used to detect outliers in the multidimensional dataset of user electricity current, voltage, electricity consumption, and power.

[0013] In a preferred embodiment of the present invention, the improved KNN imputation algorithm process is as follows:

[0014] Step 1: Construct the correlation coefficient matrix of the data.

[0015] First, the data matrix is ​​initialized to construct a complete data matrix X. m×n Here, m represents the number of data records, and n represents the data dimension. Missing data markers for outliers are excluded. The data is then standardized, and the covariance of different attribute dimensions is calculated to obtain an n-dimensional covariance matrix.

[0016] The formula for the covariance of two random variables is as follows:

[0017]

[0018] n-dimensional covariance matrix:

[0019]

[0020] Step 2: Calculate the estimated value using the KNN algorithm;

[0021] First, calculate the Euclidean distance for the complete dataset. Since missing values ​​exist in the dataset, we temporarily disregard the dimension corresponding to the missing values ​​when calculating the Euclidean distance. The magnitude of the distance determines the accuracy of subsequent KNN imputation. Let X... i To represent the i-th data record, construct the Euclidean distance matrix:

[0022]

[0023] Iterate through the original data starting from the first row and first column to find the missing value x. ij If the row number of the i-th row is i, then the distance matrix corresponding to the i-th row is the i-th row of dis. Then, determine the number K of nearest neighbors to use, and select the K smallest values ​​from the corresponding row of dis to form a vector:

[0024] d = (dj×1, dj×2, ..., dj×k)

[0025] Finally, the substitution value x0 is obtained through weighted calculation:

[0026]

[0027] X pj It is the value at the corresponding position of the nearest neighbor, and p is the number of columns in the distance matrix corresponding to the number of rows in the original data matrix;

[0028] Step 3: Calculate the dimensional correlation values ​​to obtain the final imputed value;

[0029] First, calculate the deviation (degree of deviation from the center) for each value, which is each value minus the mean of the statistics for that attribute:

[0030]

[0031] m0 is the number of non-missing values ​​in this attribute, x ij These are the corresponding statistical values;

[0032] Next, calculate the influence of the dimension corresponding to the missing value. Perform the same operation on the observations of the K nearest neighbors (excluding the missing value), then sum them and take the mean to obtain the influence values ​​of the attribute dimensions corresponding to the complete data and the missing value data.

[0033]

[0034] r is the number of non-missing data points in this row of a data item, and also the number of dimensions that can be statistically analyzed to influence missing values ​​under real-world conditions. Coeff is a i The corresponding covariance value;

[0035] Finally, the previously estimated replacement value x0 is added to the dimensionality correlation value to obtain the final missing value imputation value x″:

[0036] x″=x0+x′

[0037] In a preferred embodiment of the present invention, data pre-population using statistical rules and business rules includes:

[0038] By calculating and analyzing the statistical characteristics of the data—median, mean, and mode—missing data values ​​are pre-filled.

[0039] For the power archive data obtained by the middle platform, a data frame is generated, where each row represents a user ID and each column represents an attribute feature. For missing values ​​in the data, the median, mean, and mode are calculated according to the attribute of the corresponding column. If the data in the column has a normal symmetrical distribution, the mean is used to fill the gaps. If the data is skewed, the median is used to fill the gaps. If a data point appears frequently, the mode is used to fill the gaps.

[0040] Fill in the missing values ​​based on the actual electricity usage scenarios of residents;

[0041] Based on past user data, if a user has a low frequency of electricity use, low electricity demand, and low energy consumption, the missing values ​​for that user will be filled with 0.

[0042] In a preferred embodiment of the present invention, the outlier detection process for multidimensional data based on the improved DBSCAN clustering algorithm is as follows:

[0043] Step 1: Based on the distribution characteristics of the original data, estimate the reasonable range of Eps using kernel density, and then determine the candidate range of MinPts through expectation.

[0044] Step 2: Use the contour coefficient method to determine the optimal parameters from the candidate range;

[0045] Step 3: The reshaped DBSCAN algorithm is used to cluster multi-dimensional attribute data such as current, voltage, power consumption, and power to identify outlier noise points and set them to null.

[0046] In a preferred embodiment of the present invention, the root mean square error is used to evaluate the cleaned data, including:

[0047] After outlier detection, removal, and missing value imputation, an evaluation is performed using the root mean square error (RMSE) method. The error between the processed data and the actual data is compared. A random sample of existing accurate values ​​is selected, corrected, and imputed. The RMSE of the corrected values ​​is then compared to the accurate values. The specific formula is as follows:

[0048]

[0049] Where, x i The original value, x″ i To fill in the numerical values;

[0050] It uses the root mean square error (RMSE) as an indicator to evaluate the effectiveness of missing data imputation. The smaller the RMSE, the better the missing data imputation. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a method for detecting and cleaning outliers in low-voltage user electricity consumption data based on improved DBSCAN and KNN algorithms, provided as an example of this invention;

[0053] Figure 2 A flowchart for outlier detection in unidimensional data based on statistical methods, provided for the implementation of this invention;

[0054] Figure 3 A flowchart for multidimensional data outlier detection based on the improved DBSCAN clustering algorithm provided for the implementation of this invention;

[0055] Figure 4 A flowchart illustrating the missing value filling process based on the improved KNN nearest neighbor filling algorithm provided for the implementation of this invention;

[0056] Figure 5 A schematic diagram illustrating data cleaning for implementing the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] A method for outlier detection and cleaning of electricity consumption data based on DBSCAN and KNN algorithms, comprising:

[0059] Data pre-filling using statistical and business rules: For missing values ​​in the original power archive dataset, fill in the missing values ​​by calculating and analyzing the statistical characteristics in the data: median, mean, mode, and combining the actual electricity consumption scenarios of residents.

[0060] Outlier detection in unidimensional data based on statistical methods: Based on relevant statistical methods, descriptive statistical analysis, box plots, Laida's rule and other techniques are mainly used to detect outliers in unidimensional electricity consumption data item by item.

[0061] Outlier detection in multidimensional data based on the improved DBSCAN clustering algorithm: The parameters of DBSCAN, Eps (neighborhood radius) and MinPts (density threshold, i.e., the minimum number of points in a cluster), are estimated using kernel probability density. The improved DBSCAN clustering algorithm is then used to detect outliers in multidimensional datasets of user electricity consumption, including current, voltage, electricity consumption, and power.

[0062] Full-data cleaning method: Using mathematical statistics and data mining rules, dirty data is cleaned, including: correcting erroneous data, deleting duplicate data, and filling in missing data.

[0063] Missing values ​​are filled using an improved KNN nearest neighbor imputation algorithm: For data after excluding outliers, the covariance matrix generated during principal component analysis is used as the correlation of the overall electricity consumption characteristics. The influence of the electricity consumption characteristics is calculated from the deviations and corresponding correlations of the missing terms and the K nearest neighbors, and then incorporated into the KNN calculation process to obtain the final estimated value.

[0064] The root mean square error (RMSE) is used to evaluate the cleaned data: After outlier detection, removal, and missing value filling, the data is evaluated using the RMSE, and the error between the processed data and the actual data is compared.

[0065] Preferably, data pre-population using statistical rules and business rules includes:

[0066] By calculating and analyzing the statistical characteristics of the data—median, mean, and mode—missing data values ​​are pre-filled.

[0067] For the power data archives acquired by the middle platform, a data frame is generated, where each row represents a user ID and each column represents an attribute feature. For any missing values ​​in the data, the median, mean, and mode are calculated according to the attribute of the corresponding column. If the data in a column has a normal symmetrical distribution, the mean is used to fill the gaps; if the data is skewed, the median is used; if a data point appears frequently, the mode is used to fill the gaps.

[0068] Fill in the missing values ​​based on the actual electricity usage scenarios of residents;

[0069] Based on the user's past data, if the user has a low frequency of electricity use, low electricity demand, and low energy consumption, the missing values ​​for that user will be filled with 0.

[0070] Preferably, outlier detection for unidimensional data based on statistical methods includes:

[0071] ① We will start with descriptive statistical methods to analyze the data characteristics of 96 points of current, 96 points of voltage, and electricity consumption. From the perspective of statistical analysis, we will examine the basic descriptive indicators such as the mean, median, maximum, minimum, quantile, and standard deviation of the data to determine the difference between its value and the surrounding values. If the difference from other values ​​is large and the change is obvious, then it will be identified as an abnormal outlier.

[0072] ② Outliers in the 96-point current and voltage curve data were identified using the Raida criterion, also known as the 3σ criterion. The 3σ criterion assumes that a set of test data contains only random errors, calculates the standard deviation, and determines an interval based on a certain probability. Errors exceeding this interval are considered gross errors, not random errors, and data containing such errors should be removed. This discriminant principle and method are limited to processing normally or approximately normally distributed sample data. It relies on a sufficiently large number of measurements (samples > 10). Using this criterion to remove gross errors is unreliable when the number of measurements is small. Generally, if any data point exceeds three times the standard deviation σ, it is considered a statistical outlier. Outliers identified using the Raida criterion are left blank.

[0073] The 3δ rule is:

[0074] The probability that the object's value falls within the range (μ-σ, μ+σ) is 0.6827.

[0075] The probability that the object's value is distributed in (μ-2σ, μ+2σ) is 0.9545.

[0076] The probability that the object's value is distributed in (μ-3σ, μ+3σ) is 0.9973.

[0077] As the rule states, the values ​​in the dataset are almost entirely concentrated in the interval (μ-3σ, μ+3σ), with the probability of them falling outside this range being less than 0.3%. That is, if any data point exceeds three times the standard deviation σ, then these points are judged as statistical outliers or outliers.

[0078] Secondly, box plots are used to analyze outliers in the electricity consumption data. A box plot is a graphical representation of digital data formed by its quartiles. This is a very simple yet effective method for visualizing outliers. Consider the upper and lower whiskers as the boundaries of the data distribution. Any data point above the upper whisker or below the lower whisker can be considered an outlier or anomaly.

[0079] The concept of interquartile range (IQR) is used to construct box plots. IQR is a statistical concept that measures statistical dispersion and data variability by dividing a dataset into quartiles. The interquartile range is the difference between the third quartile and the first quartile (IQR = Q3 - Q1). In this context, outliers are defined as observations below the lower tentacles of the box plot (or Q1 - 1.5 x IQR) or above the upper tentacles (or Q3 + 1.5 x IQR).

[0080] ③ Next, we use box plots to perform outlier analysis on the electricity consumption data. A box plot is a graphical representation of digital data formed by its quartiles. This is a very simple yet effective method for visualizing outliers. Consider the upper and lower whiskers as the boundaries of the data distribution. IQR represents the interquartile range, which is the difference between the third quartile and the first quartile. The quartile is represented by Q. Generally, in statistics, we define Q3 + 1.5 * IQR as the upper whisker of the box plot and Q1 - 1.5 * IQR as the lower whisker. Any data point above the upper whisker or below the lower whisker can be considered an outlier or an anomaly. For the current, voltage, and electricity consumption data, we perform box plot validation analysis on each column of data, and treat the identified outliers as null values.

[0081] Preferably, outlier detection in multidimensional data based on the improved DBSCAN clustering algorithm includes: The main principle of DBSCAN algorithm for identifying outliers is as follows: The number of data points within a given radius (eps) in the dataset is not less than a certain threshold (Minpts). A point n is selected in the dataset in a certain order. If n is a core point, a neighborhood query is performed to obtain the neighborhood of n. If the neighboring points and n belong to the same class, these points are used as seed points for the next iteration. The class of seed points is expanded by continuously performing region queries until a complete cluster is found. Then, this process is repeated to continue searching for the remaining classes. Finally, the remaining points that do not belong to any class are determined to be noise points.

[0082] Suppose that the data X follows an independent distribution F, and x1, x2, ..., x3 are drawn from it. n Given n sample points and a probability density function f, the kernel density estimation formula is as follows:

[0083]

[0084] Where h>0 represents the bandwidth, is a smoothing parameter, and K(x) represents the kernel function. K(x) also satisfies the following condition:

[0085] K(x)≥0, ∫K(x)dx=1, ∫xK(x)dx=0,∫x 2 K(x)dx>0

[0086] The accuracy of kernel density estimation often depends on the choice of bandwidth; different bandwidths tend to lead to different fitting estimation results. Since a larger bandwidth value results in a smoother probability distribution curve, and a smaller bandwidth value results in a steeper probability distribution, the mean square integral error (MISE) function is chosen to determine the bandwidth value, as shown in the following formula:

[0087]

[0088] Under the weak assumptions, the asymptotic MISE is:

[0089]

[0090] Where R(K) and m2(k) are defined as:

[0091] R(K)=∫K(x) 2 dx,m2(K)=∫x 2 K(x)dx

[0092] To minimize the risk level, i.e., to minimize the mean square integral error, which is to minimize the AMISE, we take the partial derivative of AMISE and set it to zero:

[0093]

[0094]

[0095] In the above formula, m and R can be determined by the kernel function.

[0096] Since kernel density estimation cannot directly derive the optimal parameters for DBSCAN, it can only estimate a reasonable range for parameter selection. Therefore, the Euclidean distance between all samples of the power data archives acquired by the platform can be calculated based on their current, voltage, and electricity consumption, generating a distance matrix Dist. A density-distance curve is then plotted using kernel density estimation. From this curve, the candidate range for Eps can be predicted, and its expected value can be calculated. Based on the distance matrix Dist, the estimated range of MinPts in the given dataset is determined, as follows:

[0097]

[0098] Among them, P i This represents the number of samples contained within the Eps neighborhood of object i.

[0099] For the two reasonable intervals of Eps and MinPts obtained from kernel density estimation, the silhouette coefficient is then used to optimize the parameters within the interval to determine the final suitable parameters. The silhouette coefficient is a commonly used evaluation metric for cluster analysis, effectively utilizing information on cluster density and inter-cluster dispersion to reflect the clustering effect. The formula for calculating the silhouette coefficient is as follows:

[0100]

[0101] Among them, a i b represents the average distance from the i-th object to other objects in its cluster. i This represents the average distance from the i-th object to all objects in other clusters besides the i-th object. s(i)∈[-1,1], and the closer this value is to 1, the more reasonable the classification.

[0102] The finalized parameters are fed into the DBSCAN algorithm, and the electricity consumption data of each user profile is subjected to DBSCAN density clustering according to current, voltage and electricity consumption. After the clustering is completed, the remaining points that are far from the cluster center and are not classified into clusters are the outliers.

[0103] The main steps and processing flow of DBSCAN are as follows:

[0104] Input: Data set, neighborhood radius Eps, threshold number of data objects in the neighborhood MinPts;

[0105] Step 1: Select any data object point p from the dataset;

[0106] Step 2: If the selected data object point p is the core point for parameters Eps and MinPts, then find all data object points that are density-reachable from p and form a cluster;

[0107] Step 3: If the selected data object point p is an edge point, select another data object point;

[0108] Step 4: Repeat steps (2) and (3) until all points have been processed.

[0109] Output: Density-connected clusters.

[0110] Since the two parameters of DBSCAN need to be determined manually in advance and judged based on experience, the clustering effect is difficult to achieve as expected for unevenly distributed power data and complex situations, and the resulting error fluctuations are also large.

[0111] Based on this, we use the kernel density function to estimate the parameters of DBSCAN. We first estimate the reasonable range of parameters, and then use the silhouette coefficient to determine the optimal specific parameter values ​​in the range, which will serve as our final clustering.

[0112] Construct the kernel density function:

[0113] Suppose that the data X follows an independent distribution F, and x1, x2, ..., x3 are drawn from it. n Given n sample points, the probability density function is f, h>0 represents the bandwidth, is a smoothing parameter, and K(x) represents the kernel function with the following density:

[0114]

[0115] K(x) also satisfies the following condition:

[0116] K(x)≥0, ∫K(x)dx=1, ∫xK(x)dx=0,∫x 2 K(x)dx>0

[0117] Since the choice of bandwidth is difficult to determine, and different bandwidths often lead to significant differences in the fitting results, the mean square integral error (MISE) function is chosen to determine its bandwidth value, as shown in the following formula:

[0118]

[0119] Under the weak assumptions, the asymptotic MISE is:

[0120] MISE(h)=AMISE(h)+o(1 / (nh)+h 4 )

[0121]

[0122] Where R(K) and m2(k) are defined as:

[0123] R(K)=∫K(x) 2 dx,m2(K)=∫x 2 K(x)dx

[0124] To minimize the risk level, i.e., to minimize the mean square integral error, which is to minimize the AMISE, we take the partial derivative of AMISE and set it to zero:

[0125]

[0126]

[0127] In the above formula, m and R can be determined by the kernel function.

[0128] Once the bandwidth is determined, the kernel density function can be determined. The inter-sample distance Dist of the extracted sample dataset is calculated using the Euclidean distance formula. A kernel density estimation curve is plotted with distance on the horizontal axis and density on the vertical axis. Information on inter-cluster density and intra-cluster density can be obtained from the curve. In clustering, the distance with higher intra-cluster density is selected to determine the candidate range for Eps.

[0129] Based on the estimated range of Eps, the expected value method is used to calculate the reasonable range of MinPts in the given dataset using the distance matrix Dist. The formula is as follows:

[0130]

[0131] Among them, P i This represents the number of samples contained within the Eps neighborhood of object i.

[0132] For the two reasonable intervals of Eps and MinPts obtained from kernel density estimation, the silhouette coefficient is then used to optimize the parameters within the interval to determine the final suitable parameters. The silhouette coefficient is a commonly used evaluation metric for cluster analysis, effectively utilizing information on cluster density and inter-cluster dispersion to reflect the clustering effect. The formula for calculating the silhouette coefficient is as follows:

[0133]

[0134] Among them, a i b represents the average distance from the i-th object to other objects in its cluster. i This represents the average distance from the i-th object to all objects in other clusters besides the i-th object. s(i) ∈ [-1, 1], and a value closer to 1 indicates a more reasonable classification. The silhouette coefficients are compared, and the maximum value is selected to determine the optimal parameters for DBSCAN clustering.

[0135] The overall process for outlier detection in multidimensional data based on the improved DBSCAN clustering algorithm is as follows:

[0136] Step 1: Based on the distribution characteristics of the original data, estimate the reasonable range of Eps using kernel density, and then determine the candidate range of MinPts by expectation.

[0137] Step 2: Use the silhouette coefficient method to determine the optimal parameters from the candidate range.

[0138] Step 3: The reshaped DBSCAN algorithm is used to cluster multi-dimensional attribute data such as current, voltage, power consumption, and power to identify outlier noise points and set them to null.

[0139] Preferred methods for cleaning full-volume data include: using mathematical statistics and data mining rules to clean dirty data. Data cleaning is the process of simplifying a database to remove duplicate records and converting the remaining data into a standard, acceptable format.

[0140] The standard data cleaning model involves inputting data into a data cleaning processor, "cleaning" the data through a series of steps, and then outputting the cleaned data in the desired format. Data cleaning addresses issues such as missing values, out-of-bounds values, inconsistencies, and duplicate data by considering aspects such as accuracy, completeness, uniqueness, and validity. Data cleaning is generally application-specific and difficult to generalize into a unified method and procedure. The main solutions are as follows:

[0141] ① Methods for resolving incomplete data

[0142] In most cases, missing values ​​must be entered manually. Missing values ​​can be derived from this data source or other data sources, and can be replaced by the mean, maximum, minimum, or more complex probability estimates.

[0143] ② Error value monitoring and solutions

[0144] Statistical analysis methods can be used to identify potential errors or outliers, such as deviation analysis, identifying values ​​that do not conform to distributions or regression equations. Alternatively, simple rule bases (common sense rules, business-specific rules, etc.) can be used to check data, or constraints between different attributes and external data can be used to detect and clean data.

[0145] ③ Methods for monitoring and eliminating duplicate records

[0146] Records with identical attribute values ​​in the database are considered duplicate records. The equality of records is determined by comparing whether the attribute values ​​are equal, and equal records are merged into one record.

[0147] Preferably, the missing value filling based on the improved KNN nearest neighbor filling algorithm includes:

[0148] The KNN nearest neighbor imputation algorithm is simple and intuitive. Given a target data set containing missing items, it first calculates the Euclidean distance between the target data and each record in the input training dataset. For example, the Euclidean distance between x0 and other records in n-dimensional space can be expressed as... Select the k data records with the smallest Euclidean distance as the k nearest neighbors, and then calculate the weights of the k nearest neighbors of the target data. Finally, estimate the target data value for the missing items and fill them in.

[0149] The main steps of the KNN algorithm can be summarized as follows:

[0150] Step 1: Data initialization, constructing a complete value data matrix: (x1, x2, ..., x j ,…,x m ) T Where [X]r is the r-th attribute of the data, r≤n, and m is the number of samples;

[0151] Step 2: Calculate the Euclidean distance between the target data and the data records in the complete value data matrix, given a missing instance x. ir :

[0152]

[0153] Step 3: Select the k nearest neighbors with the smallest distance as the k nearest neighbors of the target data;

[0154] Step 4: Calculate the weights of the k nearest neighbors of the target data and fill in the missing data:

[0155]

[0156]

[0157] To improve the imputation effect of missing data and enhance the correlation of attributes, the covariance matrix generated during principal component analysis is used as the correlation of the overall electricity consumption characteristics for the data after excluding outliers. The influence of the electricity consumption characteristics is calculated from the deviations and corresponding correlations of the missing items and the K nearest neighbors, and then incorporated into the KNN calculation process to obtain the final estimated value.

[0158] The improved KNN imputation algorithm is as follows:

[0159] Step 1: Construct the correlation coefficient matrix of the data.

[0160] First, the data matrix is ​​initialized to construct a complete data matrix X. m×n Here, m represents the number of data records, and n represents the data dimension. Missing data markers for outliers are excluded. The data is then standardized, and the covariance of different attribute dimensions is calculated to obtain an n-dimensional covariance matrix.

[0161] The formula for the covariance of two random variables is as follows:

[0162]

[0163] n-dimensional covariance matrix:

[0164]

[0165] Step 2: Calculate the estimated value using the KNN algorithm.

[0166] First, calculate the Euclidean distance for the complete dataset. Since missing values ​​exist in the dataset, we temporarily ignore the dimension corresponding to the missing values ​​when calculating the Euclidean distance. The magnitude of the distance determines the accuracy of subsequent KNN imputation, denoted by X. i To represent the i-th data record, construct the Euclidean distance matrix:

[0167]

[0168] Iterate through the original data starting from the first row and first column to find the missing value x. ij If the row number of the i-th row is i, then the distance matrix corresponding to the i-th row is the i-th row of dis. Then, determine the number K of nearest neighbors to use, and select the K smallest values ​​from the corresponding row of dis to form a vector:

[0169] d = (dj×1, dj×2, ..., dj×k)

[0170] Finally, the substitution value x0 is obtained through weighted calculation:

[0171]

[0172] X pj is the value at the corresponding nearest neighbor position, and p is the number of columns in the distance matrix corresponding to the number of rows in the original data matrix.

[0173] Step 3: Calculate the dimensional correlation values ​​to obtain the final imputed values.

[0174] Combine the covariance matrix obtained earlier with the K-nearest neighbor data to form a K-nearest neighbor matrix, and calculate the deviation (degree of deviation from the center) of each value, which is each value minus the mean of the statistical values ​​under that attribute:

[0175]

[0176] m0 is the number of non-missing values ​​in this attribute, x ij That is the corresponding statistical value.

[0177] Next, calculate the influence of the dimension corresponding to the missing value. Perform the same operation on the observations of the K nearest neighbors (excluding the missing value), then sum them and take the mean to obtain the influence values ​​of the attribute dimensions corresponding to the complete data and the missing value data.

[0178]

[0179] r is the number of non-missing data points in this row of a data item, and also the number of dimensions that can be statistically analyzed to influence missing values ​​under real-world conditions. Coeff is a i The corresponding covariance value.

[0180] Finally, the previously estimated replacement value x0 is added to the dimensionality correlation value to obtain the final missing value imputation value x″:

[0181] x″=x0+x′

[0182] Preferably, the root mean square error is used to evaluate the cleaned and filled data: the effects of detecting outliers, setting blanks, and filling missing values ​​are evaluated, and the root mean square error is used to compare the error between the processed data and the real data.

[0183] A set of data is randomly selected for testing. First, a portion of the existing accurate values ​​in the original dataset is randomly extracted, corrected, and filled. The root mean square error of the corrected values ​​is compared with that of the accurate values. The specific formula is as follows:

[0184]

[0185] Where xi is the original value, x″ i To fill in the numerical values.

[0186] As defined by the formula, the smaller the root mean square error, the better the improved algorithm's effect on data cleaning and filling.

[0187] Example:

[0188] Please see Figure 1 The document presents a flowchart of a method for detecting and cleaning outliers in low-voltage user electricity consumption data based on improved DBSCAN and KNN algorithms, as provided in an embodiment of the present invention, including:

[0189] S101: Pre-populate data using statistical rules and business rules.

[0190] First, for the missing values ​​in the original power archive dataset, we fill in the missing values ​​by calculating and analyzing the statistical characteristics in the data: median, mean, mode, and combining them with the actual electricity consumption scenarios of residents.

[0191] S102: Outlier detection in unidimensional data based on statistical methods.

[0192] Based on relevant statistical methods, mainly descriptive statistical analysis, box plots, and Laida's rule, anomalies were detected in the pre-filled one-dimensional electricity consumption data, and the detected anomaly data were cleared.

[0193] S103: Outlier Detection in Multidimensional Data Based on Improved DBSCAN Clustering Algorithm.

[0194] For multidimensional data, the DBSCAN density clustering algorithm is used for outlier detection. However, this algorithm requires pre-determining two parameters, Eps and MinPts. Different parameter selections lead to different clustering patterns, affecting the error compared to the actual results. Therefore, kernel probability density is used to estimate the DBSCAN parameters: Eps (neighborhood radius) and MinPts (density threshold, i.e., the minimum number of points within a cluster). The improved DBSCAN clustering algorithm is then used to detect outliers in the multidimensional dataset of user electricity consumption, including current, voltage, electricity consumption, and power. Detected outliers are then cleared.

[0195] S104: Cleaning method based on full data.

[0196] Using mathematical statistics and data mining rules, dirty data is cleaned, including: correcting erroneous data, deleting duplicate data, and filling in missing data.

[0197] S105: Filling in missing values ​​based on an improved KNN nearest neighbor filling algorithm.

[0198] For the data after excluding outliers, the covariance matrix generated during principal component analysis is used as the correlation of the overall electricity consumption characteristics. The influence of the electricity consumption characteristics is calculated from the deviations of the missing terms and the K nearest neighbors and the corresponding correlations. This is then incorporated into the KNN calculation process to obtain the final estimated value.

[0199] S106: Use root mean square error to evaluate the cleaned and filled data.

[0200] After outlier detection, removal, and missing value filling, an evaluation is conducted using the root mean square error method to compare the error between the processed data and the actual data.

[0201] Please see Figure 2 It presents a flowchart of outlier detection for unidimensional data based on statistical methods provided in this invention, including:

[0202] S201: Use descriptive statistical methods to calculate basic descriptive indicators such as the mean, median, maximum, minimum, quantiles, and standard deviation of the data to identify outliers.

[0203] Statistical characteristic indicators were calculated for 96 points of current, 96 points of voltage, and power consumption data. The values ​​were compared, and outliers were detected by judging the magnitude of the fluctuations in the values.

[0204] S202: Based on the Raida criterion, also known as the 3σ criterion, identify the outliers in the 96-point current and voltage curve data.

[0205] According to the central theorem, when the amount of data is large, the distribution of the data tends to be normal. By using the Laida criterion, the current and voltage data outside the 3σ range can be identified as outliers.

[0206] S203: Use box plots to verify anomalies in electricity consumption data.

[0207] Data is represented by the interquartile range (IQR), a statistical concept that is the difference between the third quartile and the first quartile. It measures statistical dispersion and data variability by dividing the dataset into quartiles. Outliers are defined as observations that are below the lower tentacles of the box plot (or Q1 - 1.5 x IQR) or above the upper tentacles of the box plot (or Q3 + 1.5 x IQR).

[0208] Please see Figure 3 The present invention provides a flowchart of multidimensional data outlier detection based on the improved DBSCAN clustering algorithm.

[0209] S301: Based on the distribution characteristics of the original data, the reasonable range of Eps is estimated using kernel density, and then the candidate range of MinPts is determined by expectation.

[0210] Suppose that the data X follows an independent distribution F, and x1, x2, ..., x3 are drawn from it. n Given n sample points, the probability density function is f, h>0 represents the bandwidth, is a smoothing parameter, and K(x) represents the kernel function with the following density:

[0211]

[0212] The bandwidth is determined by selecting the mean square integral error (MISE) function, as shown in the following formula:

[0213]

[0214] Under the weak assumptions, the asymptotic MISE is:

[0215] MISE(h)=AMISE(h)+o(1 / (nh)+h 4 )

[0216]

[0217] Where R(K) and m2(k) are defined as:

[0218] R(K)=∫K(x) 2 dx,m2(K)=∫x 2 K(x)dx

[0219] To find the minimum mean square integral error, which is also to find the minimum AMISE, we take the partial derivative of AMISE and set it to zero:

[0220]

[0221]

[0222] Once the bandwidth is determined, the kernel density function can be determined. The inter-sample distance Dist of the extracted sample dataset is calculated using the Euclidean distance formula. A kernel density estimation curve is plotted with distance on the horizontal axis and density on the vertical axis. Information on inter-cluster density and intra-cluster density can be obtained from the curve. In clustering, the distance with higher intra-cluster density is selected to determine the candidate range for Eps.

[0223] Based on the estimated range of Eps, the expected value method is used to calculate the reasonable range of MinPts in the given dataset using the distance matrix Dist. The formula is as follows:

[0224]

[0225] Where Pi represents the number of samples contained in the Eps neighborhood of object i.

[0226] S302: Use the profile coefficient method to determine the optimal parameters from the candidate range.

[0227] For the two reasonable interval values ​​of Eps and MinPts obtained from kernel density estimation, the silhouette coefficient is then used to optimize the parameters within the interval to determine the final suitable specific parameters. The silhouette coefficient is constructed as follows:

[0228]

[0229] Where a i represents the cohesion of the sample points, j represents other sample points in the same class as sample i, and distance represents the distance between i and j. Therefore, a i A smaller value indicates a denser cluster. The closer the silhouette coefficient is to 1, the better the clustering effect of the algorithm. The parameter value with the largest silhouette coefficient is selected as the final DBSCAN clustering parameter.

[0230] S303: The reshaped DBSCAN algorithm is used to cluster multi-dimensional attribute data such as current, voltage, power consumption, and power to identify outlier noise points and set them to null.

[0231] With the determined DBSCAN parameters fixed, clustering is performed on multi-dimensional data such as current, voltage, electricity consumption, and power in the power archive. Outliers based on attributes such as current, voltage, electricity consumption, and power are detected and cleared to prepare for subsequent filling work.

[0232] Please see Figure 4 The present invention provides a flowchart of the missing value filling algorithm based on the improved KNN nearest neighbor filling algorithm provided in the present invention.

[0233] S401: Construct the correlation coefficient matrix of the data.

[0234] First, the data matrix is ​​initialized to construct a complete data matrix X. m×n Here, m represents the number of data records, and n represents the data dimension. Missing data markers for outliers are excluded. The data is then standardized, and the covariance of different attribute dimensions is calculated to obtain an n-dimensional covariance matrix.

[0235] The formula for the covariance of two random variables is as follows:

[0236]

[0237] n-dimensional covariance matrix:

[0238]

[0239] S402: Calculate the estimated value using the KNN algorithm.

[0240] First, calculate the Euclidean distance for the complete dataset. Since missing values ​​exist in the dataset, the dimension corresponding to the missing values ​​is temporarily ignored when calculating the Euclidean distance. The magnitude of the distance determines the accuracy of subsequent KNN imputation, denoted by X. i To represent the i-th data record, construct the Euclidean distance matrix:

[0241]

[0242] Starting from the first row and first column of the original data, iterate through the data to find the missing value xij. The row number of xij is i, and the distance matrix corresponding to the i-th row is the i-th row of dis. Then determine the number K of nearest neighbors to use, and select the K smallest values ​​from the corresponding rows of dis to form a vector.

[0243] d = (dj×1, dj×2, ..., dj×k)

[0244] Finally, the substitution value x0 is obtained through weighted calculation:

[0245]

[0246] Xpj is the value at the corresponding nearest neighbor position, and p is the number of columns in the distance matrix corresponding to the number of rows in the original data matrix.

[0247] S403: Form a K-nearest neighbor matrix and calculate its dimensional correlation value to obtain the final filling value.

[0248] Combine the covariance matrix obtained earlier with the K-nearest neighbor data to form a K-nearest neighbor matrix, and calculate the deviation (degree of deviation from the center) of each value, which is each value minus the mean of the statistical values ​​under that attribute:

[0249]

[0250] m0 is the number of non-missing values ​​in this attribute, and xij is the corresponding statistical value.

[0251] Next, calculate the influence of the dimension corresponding to the missing value. Perform the same operation on the observations of the K nearest neighbors (excluding the missing value), then sum them and take the mean to obtain the influence values ​​of the attribute dimensions corresponding to the complete data and the missing value data.

[0252]

[0253] r is the number of non-missing data points in this row of a data item, and also the number of dimensions that can be statistically analyzed to influence missing values ​​under real-world conditions. Coeff is a i The corresponding covariance value.

[0254] Finally, the previously estimated replacement value x0 is added to the dimensionality correlation value to obtain the final missing value imputation value x″:

[0255] x″=x0+x′

[0256] 1. Please refer to Figure 5 Cleaning methods based on full data include:

[0257] Data cleaning and denoising encompasses ensuring data consistency, completeness, compliance, and accuracy. Data cleaning is typically performed by computers rather than manually. The process involves cleaning and denoising the extracted data, filling in missing data, smoothing noisy data, identifying and removing outliers, and resolving inconsistencies. This improves the accuracy, completeness, and consistency of the data. High-quality user profiles and electricity consumption information are essential to guaranteeing the accuracy of identifying, analyzing, and mining results related to electricity theft.

[0258] Data cleaning uses techniques such as mathematical statistics, data mining, or predefined cleaning rules to transform dirty data into data that meets data quality requirements.

Claims

1. A method for outlier detection and cleaning of electricity consumption data based on DBSCAN and KNN algorithms, characterized in that, include: Pre-fill missing values ​​in electricity consumption data; Outlier detection was performed on the pre-filled electricity consumption data for both single-dimensional data and multi-dimensional data based on the improved DBSCAN clustering algorithm. The DBSCAN parameters were estimated using kernel probability density to determine the outlier values, and the outlier values ​​were set to null. Then, mathematical statistics and data mining rules are used to clean the electricity consumption data again. Missing value filling based on the improved KNN nearest neighbor filling algorithm: For the data after excluding outliers, principal component analysis is used to incorporate the influence of feature attributes into the KNN calculation process to obtain the final estimated value; The improved KNN imputation algorithm is as follows: Step 1: Construct the correlation coefficient matrix of the data: First, the data matrix is ​​initialized to construct a complete data matrix X. m×n m represents the number of data records, and n represents the data dimension. After removing missing data markers for outliers, the data is standardized, and the covariance of different attribute dimensions of the data is calculated to obtain an n-dimensional covariance matrix. The formula for the covariance of two random variables is as follows: ； n-dimensional covariance matrix: ； Step 2: Calculate the estimated value using the KNN algorithm: First, calculate the Euclidean distance for the complete dataset. Since missing values ​​exist in the dataset, we temporarily disregard the dimension corresponding to the missing values ​​when calculating the Euclidean distance. The magnitude of the distance determines the accuracy of subsequent KNN imputation. Let X... i To represent the i-th data record, construct the Euclidean distance matrix: ； Iterate through the original data starting from the first row and first column to find the missing value x. ij If the row number of the i-th row is i, then the distance matrix corresponding to the i-th row is the i-th row of dis. Then, determine the number K of the nearest neighbors to use, and select the K smallest values ​​from the corresponding row of dis to form a vector: d = (dj×1, dj×2, ..., dj×k); Finally, the substitution value x0 is obtained through weighted calculation: ； X pj It is the value at the corresponding position of the nearest neighbor, and p is the number of columns in the distance matrix corresponding to the number of rows in the original data matrix; Step 3: Calculate the dimensional correlation values ​​to obtain the final imputed values: First, calculate the deviation for each value, which is each value minus the mean of the statistics for that attribute: ； m0 is the number of non-missing values ​​in this attribute, x ij These are the corresponding statistical values; Next, calculate the influence of the dimension corresponding to the missing value. Perform the same operation on the observations of the K nearest neighbors (excluding the missing value), then sum them and take the mean to obtain the influence values ​​of the attribute dimensions corresponding to the complete data and the missing value data. ； r is the number of non-missing data points in this row of a data item, and also the number of dimensions that can be statistically analyzed to influence missing values ​​under real-world conditions. Coeff is... The corresponding covariance value; finally, the previously estimated substitute value x0 is added to the dimensionality correlation value, which is the final missing value imputation value. : 。 2. The method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to claim 1, characterized in that, It also includes data cleaning and evaluation of the cleaned data using root mean square error, including but not limited to the evaluation of outlier detection, removal, and missing value filling in electricity consumption data, using root mean square error to evaluate the data, and comparing the error between the processed electricity consumption data and the actual data.

3. The method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to claim 1, characterized in that, The outlier detection of the single-dimensional data is specifically based on statistical methods: according to relevant statistical methods, one of the following techniques is used: descriptive statistical analysis, box plots, or Laida's rule, to detect outliers in the single-dimensional electricity consumption data item by item.

4. The method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to claim 1, characterized in that, Outlier detection in multidimensional data based on the improved DBSCAN clustering algorithm: The DBSCAN parameters include: Eps neighborhood radius and MinPts density threshold, i.e. the minimum number of points in the cluster. The improved DBSCAN clustering algorithm is then used to detect outliers in the multidimensional dataset of user electricity current, voltage, electricity consumption, and power.

5. The method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to claim 1, characterized in that, The pre-filling of missing values ​​in electricity consumption data specifically involves data pre-filling using statistical rules and business rules, including: By calculating and analyzing the statistical characteristics of the data—median, mean, and mode—missing data values ​​are pre-filled. For the power archive data obtained by the middle platform, a data frame is generated, where each row represents a user ID and each column represents an attribute feature. For missing values ​​in the data, the median, mean, and mode are calculated according to the attribute of the corresponding column. If the data in the column has a normal symmetrical distribution, the mean is used to fill the gaps. If the data is skewed, the median is used to fill the gaps. If a data point appears frequently, the mode is used to fill the gaps. Fill in the missing values ​​based on the actual electricity usage scenarios of residents; Based on past user data, if a user has a low frequency of electricity use, low electricity demand, and low energy consumption, the missing values ​​for that user will be filled with 0.

6. The method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to claim 1, characterized in that, The outlier detection process for multidimensional data based on the improved DBSCAN clustering algorithm is as follows: Step 1: Based on the distribution characteristics of the original data, estimate the reasonable range of Eps using kernel density, and then determine the candidate range of MinPts through expectation. Step 2: Use the contour coefficient method to determine the optimal parameters from the candidate range; Step 3: The reshaped DBSCAN algorithm is used to cluster the multi-dimensional attribute data of current, voltage, power consumption, and power to find outlier noise points and set them to null.

7. A method for detecting and cleaning outliers in electricity consumption data based on DBSCAN and KNN algorithms according to any one of claims 1-6, characterized in that, The root mean square error is used to evaluate the cleaned data, including: After outlier detection, removal, and missing value imputation, an evaluation is performed using the root mean square error (RMSE) method. The error between the processed data and the actual data is compared. A portion of the existing accurate values ​​is randomly selected, corrected, and imputed. The RMSE of the corrected values ​​is then compared to the accurate values. The specific formula is as follows: ； Where, x i These are the original values. To fill in the numerical values; It uses the root mean square error (RMSE) as an indicator to evaluate the effectiveness of missing data imputation. The smaller the RMSE, the better the missing data imputation.

Images