An improved clustering electricity anomaly detection method based on Markov model

Abstract

The application relates to the technical field of power consumption anomaly detection, and specifically provides a Markov model-based improved power consumption anomaly detection method for clustering, which comprises the following steps: for power data preprocessing, a symbolic aggregate approximation method is used to reduce the dimension and symbolize the power consumption time sequence, and a Markov state transition probability model is constructed to represent the power consumption behavior mode of a user; for the problem that a traditional density peak value clustering algorithm relies on manual experience to select a truncation distance, a clustering anomaly detection model is provided; the relative distance is used to measure the difference in the Markov transition probability distribution of the user; a mycoparasitism optimization algorithm and a Bonferroni index are introduced to automatically optimize the truncation distance parameter; and the density peak value is used to identify the clustering center to complete user clustering and identify abnormal points. The application improves the precision and efficiency of power consumption anomaly detection, and provides technical support for power consumption safety management and anti-electricity stealing.

Description

Technical Field

[0001] This invention relates to the field of power consumption anomaly detection technology, specifically to an improved clustering power consumption anomaly detection method based on the Markov model. Background Technology

[0002] In the modern energy system architecture deeply integrated with smart grids and multi-energy complementarity, the safe and stable operation of the power system is crucial. However, abnormal electricity consumption behaviors frequently occur during actual electricity use. These behaviors not only lead to energy waste but may also cause damage to electrical equipment or even safety accidents, posing a serious threat to the reliability of user electricity supply and the stability of the power grid. Traditional manual inspection methods suffer from low efficiency, limited coverage, and high costs, making it difficult to meet the increasingly complex needs for detecting abnormal electricity consumption, resulting in a series of challenges such as economic losses, safety hazards, and increased carbon emissions. Therefore, developing an efficient and accurate method for detecting and analyzing abnormal electricity consumption is of great significance for improving the reliability of power system operation, ensuring user-side safety, and promoting the clean and low-carbon transition of energy. Summary of the Invention

[0003] To address the problems existing in the background technology, this invention aims to provide an improved clustering-based electricity consumption anomaly detection method based on the Markov model. It systematically studies the algorithm optimization and model innovation for electricity consumption anomaly detection, from data preprocessing and anomaly detection model improvement to multi-dimensional anomaly identification and classification. By integrating time series analysis, clustering algorithm optimization, and multi-criteria fusion techniques, an efficient and reliable electricity consumption anomaly detection system is constructed, providing theoretical support and technical pathways for the refined management and anomaly behavior identification of smart grids.

[0004] This invention provides an improved clustering-based method for detecting abnormal electricity consumption based on the Markov model, comprising the following steps:

[0005] S1. Data Preparation and Preprocessing: Obtain load data for the target area, delete users with a high proportion of missing values, and fill in the datasets of users with a low proportion of missing values; reconstruct the data distribution pattern through linear transformation methods and standardize the load data;

[0006] S2. Establishment of User State Transition Probability Matrix: Based on error comparison, determine the method for filling missing values ​​in the power data; determine the final number of characters and the division of the time axis through the error of the dataset and the actual power consumption pattern; establish the Markov state transition probability matrix in discrete states using symbolic power consumption data;

[0007] S3. Abnormal power consumption detection: The density peak clustering algorithm is used to cluster the data, and the outlier judgment criteria are set to identify outliers after clustering; the Bonferroni index and slime mold optimization algorithm are introduced to optimize the cutoff distance;

[0008] S4. Multi-criteria density peak clustering anomaly detection: Establish a multi-criteria density peak clustering anomaly detection model to identify outliers and construct feature indicators to reflect the electricity consumption behavior characteristics of abnormal users; use the K-means algorithm to classify abnormal electricity consumption behaviors, obtain typical electricity consumption anomaly categories, and analyze the possible causes of abnormal electricity consumption behaviors; construct a typical feature library of electricity safety hazards, distinguish various abnormal electricity consumption behaviors, and efficiently process abnormal electricity consumption behaviors according to specific circumstances.

[0009] Preferably, step S1 includes:

[0010] S11. The load data is filled in using piecewise linear interpolation, cubic Hermite polynomial interpolation, and cubic spline interpolation methods based on the real data environment.

[0011] S12. Standardize the load data using the following deviation normalization formula:

[0012] ;

[0013] In the formula, , They are the first The actual value and standardized value of a user's electricity consumption data , These are the minimum and maximum values ​​of the load data over m time periods, respectively.

[0014] Preferably, step S2 includes:

[0015] S21. Determining the number of approximate characters for symbol aggregation: The specific number of Markov states and the number of approximate characters for symbol aggregation need to be determined through experiments on real datasets;

[0016] S22. Establishment of the Markov state transition probability matrix: The characteristics are represented by a time-based Markov model.

[0017] Preferably, step S22 includes:

[0018] The standardized load dataset is converted into a symbol string using a symbolic aggregation approximation algorithm, where each letter corresponds to a discrete state in the Markov model. Considering the time-varying dynamic characteristics of user electricity consumption behavior, a Markov chain is modeled for each time period, and a state transition frequency matrix is ​​constructed to quantify the transition patterns between different states. Based on this state transition frequency matrix, the single-step transition probability estimate from state j to state i within period t is calculated using the following formula:

[0019] ;

[0020] In the formula, Let be the estimated single-step transition probability from state j to state i within period t. Let n be the frequency of state transitions from state j to state i within period t, and n be the total number of discrete states. It is the sum of the frequencies of all state transitions starting from state j within period t; the single-step transition probability estimate is an unbiased estimate of the state transition probability matrix of the corresponding period.

[0021] Preferably, step S3 includes:

[0022] S31. Calculation of relative distance: The difference between the state transition probability matrices of any customer's electricity consumption behavior is measured by measuring the relative distance to obtain typical consumer behavior dynamics;

[0023] S32. Improved Density Peak Clustering: Improved density peak clustering is used to identify user distributions of arbitrary cluster types, obtain groups with similar power consumption dynamic characteristics, and distinguish power consumption types with different electrical state data characteristics.

[0024] S33. Abnormal User Identification: An improved density peak clustering algorithm and an anomaly identification model with a set threshold are used to cluster and identify anomalies in the distance matrix of the difference between the quantized Markov state transition probability matrices.

[0025] Preferably, the slime mold optimization algorithm in step S3 optimizes the cutoff distance, and its foraging location is calculated using the following formula:

[0026] ;

[0027] In the formula, This represents the global optimal solution position vector for the slime mold population in the current iteration. For random numbers in Oscillation within a range Food weighting coefficients representing slime molds, as well as It refers to the positions of two random individuals within a slime mold, and the parameters... The value of gradually decreases linearly from 1 to 0 during the iteration process. yes Random numbers between;

[0028] Ferroni Index The calculation formula is as follows:

[0029] ;

[0030] In the formula, the Bonferroni index , used to quantify the non-uniformity of the local density distribution of electricity user samples participating in clustering, with a value range of [0,1]; The larger the value, the more concentrated the local density of the samples is in a few cluster center samples. The higher the cluster discrimination of the density peak cluster and the better the clustering effect. This method uses... The maximum value is the objective, and the cutoff distance is automatically optimized through the slime mold optimization algorithm; N is the total number of electricity user samples participating in density peak clustering; This is the sample index after all user samples are sorted in ascending order of local density values, with values ​​i=1,2,...,N; The local density of the j-th user sample Before calculation, the local density of all users needs to be sorted in ascending order, which satisfies... ; The sum of the local densities of the first i low-density user samples; This is the total cumulative sum of local densities for all user samples participating in the clustering; The proportion of the first i samples to the total number of samples corresponds to the horizontal axis of the Lorenz curve (cumulative sample share). The cumulative local density of the first i samples accounts for the proportion of the total density, corresponding to the vertical axis of the Lorenz curve (cumulative density share).

[0031] is the normalization coefficient, used to scale the total relative deviation to the [0,1] interval to ensure the comparability of the Bonferroni index under different sample sizes.

[0032] Preferably, the abnormal user identification in step S3 includes:

[0033] If the density value of a certain sample point Density relative to benchmark Small, and distance value Compared to the reference distance If the value is large, the sample point can be determined to be an outlier. The baseline density and baseline distance of each cluster are calculated as follows:

[0034] ;

[0035] ;

[0036] In the formula, and Corresponding to the c The baseline density and baseline distance of each cluster , These are empirically adjustable parameters; these two parameters together control the sensitivity of the detection threshold. The smaller, The larger the value, the fewer outliers the algorithm identifies according to this criterion. Indicates the first c The total number of samples contained in each cluster. This represents a subset consisting of all observed samples within the cluster.

[0037] Preferably, the multi-criteria density peak clustering anomaly detection model in step S4 includes two criteria, and a sample point is marked as a final abnormal user only when both criteria are met simultaneously.

[0038] The two judgment criteria are:

[0039] Criterion 1: The local density of a sample point is less than the reference density of its cluster, and the relative distance between the sample points is greater than the reference distance of its cluster.

[0040] Criterion 2: The local density of a sample point is less than the boundary density of its cluster.

[0041] Preferably, the boundary density of the cluster includes:

[0042] Step 1: Set the initial value of the boundary density of all clusters to 0;

[0043] Step 2: If two sample points belong to different clusters and the Euclidean distance between the two points is less than λ times the cutoff distance, then the average local density of the two sample points is taken as the boundary density of their respective clusters, where λ is an empirically adjustable parameter.

[0044] Step 3: Repeat step 2 to obtain multiple boundary density values ​​for each cluster after traversing all sample points;

[0045] Step 4: Select the minimum value among the multiple boundary density values ​​of each cluster as the final boundary density of that cluster.

[0046] Preferably, in step S4, feature indicators are constructed to reflect the characteristics of abnormal user electricity consumption behavior; the feature indicators include the upward trend of monthly electricity consumption. The downward trend in monthly electricity consumption The calculation formula is as follows:

[0047] ; ;

[0048] in, This is a sequence of deviations where the actual value is greater than the predicted value. This is a deviation sequence where the actual value is less than the predicted value. , These represent the lengths of the corresponding sequences.

[0049] The beneficial effects achieved by this invention are as follows:

[0050] First, this invention achieves effective dimensionality reduction by filling in missing values ​​in optimized time series data and applying symbolic aggregation approximation, and verifies that electricity consumption data possesses Markov properties, thereby constructing a discrete state transition probability matrix. This method significantly improves the extraction accuracy and computational efficiency of electricity consumption behavior features, providing a reliable theoretical and model foundation for abnormal electricity consumption detection and practical engineering applications.

[0051] Secondly, this invention relates to a Markov-improved density peak clustering anomaly detection method based on a slime mold optimization algorithm. This method utilizes an improved density peak clustering algorithm, symbolizing the differences between Markov state sequences using the KL distance metric, to identify data density peaks to determine cluster centers and complete cluster analysis. Addressing the issue of traditional density peak clustering algorithms relying on human experience for the cutoff distance, the method introduces the Bonferroni index and applies a slime mold optimization algorithm to automatically optimize this parameter, thereby determining the optimal cutoff distance and improving clustering accuracy and adaptability.

[0052] Third, based on the completed clustering, this invention establishes scientific outlier judgment criteria to detect outliers in different clusters. Multiple evaluation metrics are used to comprehensively assess the performance of the Markov-improved DPeaks clustering anomaly detection model. This model effectively improves the accuracy and reliability of anomaly identification, providing a new approach for detecting abnormal electricity consumption behavior, and has significant theoretical value and engineering application prospects.

[0053] Fourth, this invention employs a multi-criteria fusion and clustering method for detecting abnormal power consumption, effectively improving the accuracy and practicality of power anomaly identification. By constructing a multi-criteria-DPeaks clustering model, which integrates multiple detection standards, the accuracy and efficiency of identifying complex abnormal power consumption patterns are significantly improved, and a multi-dimensional evaluation system is established to comprehensively assess the model's performance.

[0054] Fifth, this invention employs principal component analysis and K-means clustering to achieve refined classification of abnormal users, forming typical abnormal patterns, and clearly revealing the characteristics and causes of various anomalies through visualization technology. This method not only reliably identifies abnormal users but also provides them with targeted management strategies, offering effective technical support for power companies' electricity consumption supervision and decision-making, and possesses high engineering application value. Attached Figure Description

[0055] Figure 1 This is a flowchart of the Markov-DPeaks clustering anomaly detection model in an embodiment of the present invention;

[0056] Figure 2 This is the average error under different Markov state numbers in the embodiments of the present invention;

[0057] Figure 3This is a histogram approximating the segmented aggregation of the dataset in this embodiment of the invention;

[0058] Figure 4 This is a user's seven-day electricity consumption data and its SAX graph in an embodiment of the present invention;

[0059] Figure 5 This is a decision graph used to find density peaks in an embodiment of the present invention;

[0060] Figure 6 This embodiment of the invention performs two-dimensional planar mapping on all customers using a multi-dimensional scale;

[0061] Figure 7 These are the electricity consumption curves of three typical users in this embodiment of the invention;

[0062] Figure 8 This is a schematic diagram of anomaly detection using the Markov-improved DPeaks clustering model in an embodiment of the present invention;

[0063] Figure 9 This is a comparison chart of the detection results in this embodiment of the invention with actual abnormal users;

[0064] Figure 10 This is a flowchart of the anomaly detection process of the multi-criteria-DPeaks clustering model in an embodiment of the present invention;

[0065] Figure 11 This is a schematic diagram of falsely detected users removed under multiple criteria in an embodiment of the present invention;

[0066] Figure 12 This is the distribution of the first type of user detection results on the actual coordinates in this embodiment of the invention;

[0067] Figure 13 This is the distribution of the second type of user detection results on the actual coordinates in this embodiment of the invention;

[0068] Figure 14 This is the distribution of the third type of user detection results on the actual coordinates in this embodiment of the invention;

[0069] Figure 15 This is the distribution of K-means clustering results of data points when K=6 in this embodiment of the invention; Detailed Implementation

[0070] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The present invention is not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] This invention provides an improved clustering-based method for detecting abnormal electricity consumption based on the Markov model, comprising the following steps:

[0072] S1. Data Preparation and Preprocessing: Obtain load data for the target area, delete users with a high proportion of missing values, and fill in the datasets of users with a low proportion of missing values; reconstruct the data distribution pattern through linear transformation methods and standardize the data.

[0073] S2. Establishment of User State Transition Probability Matrix: Based on error comparison, determine the method for filling missing values ​​in the power data; determine the final number of characters and the division of the time axis through the error of the dataset and the actual power consumption pattern; establish the Markov state transition probability matrix in discrete states using symbolic power consumption data;

[0074] S3. Abnormal power consumption detection: The density peak clustering algorithm is used to cluster the data, and the outlier judgment criteria are set to identify outliers after clustering; the Bonferroni index and slime mold optimization algorithm are introduced to optimize the cutoff distance and avoid the subjectivity of manually determining parameters;

[0075] S4. Multi-criteria density peak clustering anomaly detection: Establish a multi-criteria density peak clustering anomaly detection system to quickly and effectively identify outliers, construct feature indicators to reflect the electricity consumption behavior characteristics of abnormal users; use the K-means algorithm to classify abnormal electricity consumption behaviors, derive typical electricity consumption anomaly categories, and analyze the possible causes of abnormal electricity consumption behaviors; construct a typical feature library of electricity safety hazards, distinguish various abnormal electricity consumption behaviors, and efficiently process abnormal electricity consumption behaviors according to specific circumstances.

[0076] In some embodiments, the specific process of step S1 includes:

[0077] S11. The load data is filled in using interpolation methods such as piecewise linear interpolation, cubic Hermite polynomial interpolation, and cubic spline interpolation based on the real data environment.

[0078] S12. Standardize the load data using the following deviation normalization formula:

[0079] ;

[0080] In the formula, , They are the first The actual value and standardized value of a user's electricity consumption data. , These are the minimum and maximum values ​​of the load data over m time periods, respectively.

[0081] In some embodiments, step S2 includes the following steps:

[0082] S21. Determining the number of approximate characters for symbol aggregation: The specific number of Markov states and the number of approximate characters for symbol aggregation need to be determined through experiments on real datasets;

[0083] S22. Establishment of the state transition probability matrix: The characteristics are described using a time-based Markov model.

[0084] In some embodiments, the specific process of step S2 includes:

[0085] Each letter transformed from the standardized load dataset by the SAX algorithm corresponds to a discrete state in the next step of the Markov model. If we simplify the electricity consumption to three states, the electricity consumption level corresponds to 33.33% and 66.67% of the Cumulative Distribution Function (CDF). Defining the power probability distribution for a set of customers is accomplished by fitting a curve to a histogram of the power distribution. The most common distribution for power or total consumption is Weibull. The following equations represent the cumulative probability function and probability density function of the Weibull distribution, respectively:

[0086] ;

[0087] ;

[0088] In the formula, k represents the shape parameter. Indicates the scale parameter. Representing random variables in the two equations, cdf Weibull and pdf Weibull These are the cumulative probability function and probability density function of the Weibull distribution, respectively, which describe the cumulative probability and instantaneous probability density of the random variable.

[0089] By increasing the number of states, the difference between the SAX-transformed symbol string and the original load curve gradually decreases. However, too many states lead to an excessively large and sparse state transition probability matrix, resulting in the meaninglessness of the state transition probability matrix and the "curse of dimensionality" in the clustering process. Therefore, the number of states is a trade-off between the information loss of SAX and the size of the state transition probability matrix. The specific number of Markov states and the approximate number of characters for symbol aggregation need to be determined through experiments on real datasets.

[0090] Assume the observed sequence follows a containment pattern. m A discrete state space with various states is used, and the transition frequency matrix is ​​constructed to quantify the transition patterns between different states. The matrix elements... Reflects the number of transitions between specific states. j is The target state, for having n A string of symbols can be applied to a string of symbols. n A discrete Markov model corresponding to each state is used to model the dynamic characteristics of their consumption levels. However, customers' daily routines exhibit different dynamic characteristics at different times. Therefore, a time-based Markov model is employed to represent these characteristics. For each additional cycle, a Markov chain can be modeled. The calculation cycle is then... t single-step transition number matrix .Depend on The cycle can be further estimated. t State transition probability matrix .

[0091] ;

[0092] ;

[0093] Matrix elements reflect the number of transitions between specific states. Indicates time t From state j to state i The single-step transition probability estimation. It is the state transition probability matrix The unbiased estimate. Therefore, the following formula can be approximated as true:

[0094] ;

[0095] In some embodiments, step S3 includes the following steps:

[0096] S31. Calculation of Kullback-Leibler distance: Using relative distance (Kullback-Leibler) to measure the difference between the state transition probability matrices of arbitrary customer electricity consumption behavior, typical consumer behavior dynamics are obtained;

[0097] S32. Improved Density Peak Clustering: Improved density peak clustering is used to identify user distributions of arbitrary cluster types, obtain groups with similar power consumption dynamic characteristics, and distinguish power consumption types with different electrical state data characteristics.

[0098] S33. Abnormal User Identification: An improved density peak clustering algorithm and an anomaly identification model with a set threshold are used to cluster and identify anomalies in the distance matrix of the difference between the quantized Markov state transition probability matrices.

[0099] In the preferred embodiment, the calculation of relative distance in step S3 is the calculation of Kullback-Leibler distance. The specific process of calculating Kullback-Leibler distance and improving density peak clustering includes:

[0100] The Karnofsky distance (KL distance), also known as information divergence, is an important indicator for measuring the differences between different probability distributions. Therefore, in order to distinguish between distributions with state transition matrices... and For two Markov models, the KL distance is defined as:

[0101] ;

[0102] It should be noted that the KL distance is asymmetric, therefore it cannot be guaranteed. This is valid. To facilitate clustering, the symmetric KL distance between the two Markov models at period t is used. Defined as:

[0103] ;

[0104] Each customer is modeled using a TMarkov model across T time periods throughout the day. The KL distance is further extended to a T-period as follows:

[0105] ;

[0106] The KL distance between all customers is calculated using the above formula, resulting in the dissimilarity matrix.

[0107] Traditional density peak clustering algorithms often rely on human experience in selecting the cutoff distance parameter, which limits their applicability and accuracy. To overcome this limitation, a slime mold optimization algorithm is introduced into the DPeaks clustering process to optimize the cutoff distance, thus avoiding the subjectivity of manually determining the parameter. The formula for calculating its foraging location is as follows:

[0108] ;

[0109] In the formula, This represents the global optimal solution position vector for the slime mold population in the current iteration. For random numbers in Oscillation within a range Food weighting coefficients representing slime molds. as well as It refers to the positions of two random individuals within a slime mold, and the parameters... The value of gradually decreases linearly from 1 to 0 during the iteration process. yes Random numbers between intervals.

[0110] In the slime mold behavior model, the dynamic update process of individual location is... leading, Expressed as a formula:

[0111] ;

[0112] In the formula, , It is the first Fitness of individual slime molds. This represents the optimal fitness level for slime molds in the current iteration phase.

[0113] The formula for updating the value is as follows:

[0114] ;

[0115] In the formula, This represents the upper limit of the number of iterations. This indicates the current iteration number.

[0116] The weighting coefficients simulate the different changes in the propagation wave of the biological oscillator when slime mold senses different food concentrations. The formula is expressed as follows:

[0117] ;

[0118]

[0119] In the formula, for Random numbers between and This represents the best and worst fitness values ​​found during the iterative convergence process. Smelllndex This represents the sum of individual positions, sorted in descending order of fitness for each slime mold individual.

[0120] When slime molds search for food, they also isolate a portion of themselves for random exploration. Based on the analysis results, the locations of the slime molds are calculated as follows:

[0121] ;

[0122] In the formula, and These are the upper and lower limits of the slime mold individual's search for the solution space. and for Random numbers on the array. The parameter, 0.03, determines the proportion of randomly distributed individuals in the total slime mold population. The slime mold optimization algorithm assigns different weights to different targets based on biomimetic foraging behavior, exploring the optimal solution globally.

[0123] Before using density peak clustering (DPeaks) for outlier detection, an optimization algorithm is needed to refine the cutoff distance parameter, thus avoiding the subjectivity of manually determining the parameter. This relates to the cutoff distance parameter in the DPeaks clustering algorithm. To address the shortcomings of requiring manual settings, the Bonferroni index is introduced to measure the set of decision indicators. The distribution of . Among them, Bonferoni Index The calculation formula is as follows:

[0124] ;

[0125] The size is mainly affected by the density peaks of the first few terms. (The set...) Center front m The sample data corresponding to the item is set as the density peak point. Indicates the preceding The contribution of the cumulative decision index of each cluster center to the overall cumulative decision index. A higher value indicates that the cluster center contains more information, resulting in better clustering performance. The larger the value, the greater it will be. for The function to obtain When the value is at its maximum That is, the optimal cutoff distance. The solution process belongs to the optimal solution problem of nondeterministic polynomials. This invention uses the slime mold optimization algorithm for iterative solution.

[0126] In the preferred embodiment, the specific process of abnormal user identification in step S3 includes:

[0127] If the density value of a certain sample point Density relative to benchmark Small and distance value Compared to the reference distance A large value indicates that the sample point is an outlier. The baseline density and baseline distance for each cluster are calculated as follows:

[0128] ;

[0129] ;

[0130] In the formula, and Corresponding to the c The baseline density and baseline distance of each cluster. , These are empirically adjustable parameters; these two parameters together control the sensitivity of the detection threshold. The smaller, The larger the value, the fewer outliers the algorithm identifies based on this criterion. Indicates the first c The total number of samples contained in each cluster. This represents a subset consisting of all observed samples within the cluster.

[0131] The model evaluation module verifies detection performance through anomaly labels. This label data is completely decoupled from the anomaly detection model's identification process and is only used in the post-hoc evaluation stage. When the model achieves satisfactory evaluation metrics on experimental samples, it can be directly applied to samples lacking labels. Therefore, anomaly identification methods based on clustering analysis still fall under the category of unsupervised methods.

[0132] The fundamental problem of anomaly detection in power load is a binary classification with an imbalanced class distribution, making accuracy an inappropriate evaluation metric. For example, in a dataset of 100 power users, where 96% are normal and only 4% are abnormal, if the classifier identifies all samples as abnormal, its classification accuracy may reach 96%, but because it completely ignores the ability to identify the minority class, the model is effectively ineffective. In conclusion, it is unreasonable to solely evaluate anomaly detection models based on accuracy.

[0133] In evaluating power user anomaly identification models, the area under the receiver operating characteristic (ROC) curve (AUC) is a commonly used metric. This metric comprehensively reflects the classifier's detection rate and accuracy.

[0134] In some embodiments, step S4 includes the following steps:

[0135] S41. Multi-criteria-Improved DPeaks Clustering Anomaly Detection: Anomaly user multi-criteria is constructed based on the improved DPeaks clustering algorithm decision graph and the actual coordinates of the new feature set, which improves the detection effect of the anomaly detection model.

[0136] S42. Analysis of abnormal electricity consumption patterns based on feature clustering: Based on the abnormal users identified by the multi-criteria-DPeaks clustering anomaly detection model, feature indicators were further constructed using principal component analysis to comprehensively reflect the electricity consumption behavior characteristics of abnormal users, and K-means clustering was used to classify abnormal users.

[0137] In the preferred embodiment, step S4 includes the following specific processes:

[0138] The multi-criteria model identifies points that meet two single criteria as outliers. The specific rules for the two criteria are as follows:

[0139] Criterion 1: When the density value at a certain point Density relative to benchmark Small and its distance value Compared to the reference distance If the value is large, then the data point is determined to be an outlier.

[0140] Criterion 2: When the density value at a certain point Compared to the boundary density of the cluster where that point is located If the value is small, the data point is considered an outlier.

[0141] In this invention, a boundary point is defined as follows: if the Euclidean distance between two points is less than 1 / 2... times If these two points are located in two different clusters, then they correspond to the boundary points of those two clusters. There may be multiple sets of boundary points, and these boundary points can be used to determine the boundary density of each cluster. Calculations are performed, in which, c These represent different clusters. The process for obtaining criterion 2 is as follows:

[0142] Step 1: Set the boundary density of all clusters The initial value is set to 0;

[0143] Step 2: Take the average density of two points that meet the boundary value requirements as the boundary density of their respective clusters;

[0144] Step 3: Repeat step 2 to obtain multiple boundary densities for all clusters;

[0145] Step 4: Select the minimum boundary density in each cluster as the final value for that cluster. .

[0146] The parameters introduced above and The stringency of criterion 1 is constrained. (Numerical value) The smaller the value The larger the value, the fewer abnormal users will be identified according to criterion 1. And the parameter... The severity of criterion 2 is constrained; the smaller the value, the fewer abnormal users will be identified according to criterion 2. For multi-criteria detection, only anomalies that meet both of the above single criteria are marked as abnormal users.

[0147] The load characteristics of a power system can be quantitatively characterized by constructing feature indicators. This invention will extract features from complete daily and monthly user electricity consumption data after preprocessing based on four aspects: the overall performance, fluctuation, trend, and correlation of electricity consumption.

[0148] (1) The overall characteristics include the average daily electricity consumption, daily and monthly load factor, monthly peak-valley difference rate of electricity consumption, and the proportion of electricity consumption in the annual electricity consumption in each quarter. The daily and monthly load factor reflects the overall change in electricity consumption, the peak-valley difference rate reflects the magnitude of the change in electricity consumption, and the proportion of electricity consumption in each quarter reflects the distribution of electricity consumption.

[0149] (2) Fluctuation characteristics include the daily and monthly electricity consumption dispersion coefficients (the ratio of the daily and monthly electricity consumption standard deviation to the daily and monthly electricity consumption mean), the ratio of the daily and monthly electricity consumption dispersion coefficients to the industry's daily and monthly electricity consumption dispersion coefficients (industry electricity consumption is represented by the average electricity consumption of all users), and the fluctuation characteristics before and after the fluctuations. m The first and last month's electricity consumption difference. The coefficient of variation for each household's electricity consumption measures the degree of deviation of the user's electricity consumption curve from the average curve, and can reflect the fluctuation of electricity consumption. The first and last month's electricity consumption difference is based on long-term monthly electricity consumption data and can reflect the fluctuation characteristics of electricity consumption.

[0150] (3) The trend characteristics include the slope of the linear fit of the daily electricity consumption data. k The monthly electricity consumption data shows both upward and downward trends. The upward and downward trends were calculated using a simple moving average method. The monthly electricity consumption data was... It means that, then in Time before Simple moving average of data points S The calculation formula is:

[0151] ;

[0152] The steps for calculating trend characteristics are as follows:

[0153] Step 1: Treat the monthly average time series sequence of each user as a typical curve of the corresponding monthly electricity consumption;

[0154] Step 2: Calculate the simple moving average series of monthly electricity consumption for all users. S ;

[0155] Step 3: Check each user individually. t Time series and sequence Compare the electricity consumption. Greater than Points used (Data points where the actual value is greater than the predicted value) represent, Less than Points used (Data points where the actual value is less than the predicted value) are represented;

[0156] Step 4: Calculate the upward trend using the following formulas respectively. and downward trend ;

[0157] ;

[0158] ;

[0159] in, This is a sequence of deviations where the actual value is greater than the predicted value. This is a deviation sequence where the actual value is less than the predicted value. , These represent the lengths of the corresponding sequences.

[0160] (4) The correlation features include the daily electricity consumption sequence of each user. and typical daily electricity consumption (i.e., daily average series) The Pearson correlation coefficient of ) is expressed as follows:

[0161] ;

[0162] In clustering of anomalous users, the high-dimensionality of the feature set affects the visualization and analysis of the detection results. Therefore, dimensionality reduction of the original dataset is generally performed. Principal component analysis (PCA) is a commonly used feature set dimensionality reduction technique that can construct new, uncorrelated feature sets from the high-dimensional feature sets obtained during feature extraction. Furthermore, most of the key information of the original feature set can be preserved, enabling visualization and analysis of the original high-dimensional features in a low-dimensional space. Assuming... X Let represent the original feature set matrix. The calculation process for the new features is as follows:

[0163] (1) Standardize the data to obtain a centered matrix with column mean of 0. Calculate the covariance matrix of the original feature set matrix. eigenvalues , And its corresponding orthogonal vector, is The formula is as follows:

[0164] ;

[0165] In the formula, , .

[0166] (2) Before calculation The cumulative contribution rate of each new feature:

[0167] ;

[0168] (3) Assumption The expected value of the cumulative contribution rate, and let Based on this, the number of new features is determined. (Before calculation) One new feature:

[0169] ;

[0170] The main idea of ​​K-means clustering is as follows: First, select K initial centers within the sample space. Then, assign weights to the distances of the remaining samples to the centers, assigning each sample to the cluster corresponding to the nearest center. Next, iteratively update the cluster centers using a formula, recalculating the distances from each sample to the new cluster centers, and then re-performing the clustering operation for each sample. This process is repeated multiple times until the center points are corrected. Each iteration recalculates the distances from each sample to the updated center points and re-performs the clustering operation. This continues until a certain update of the center points achieves the optimal clustering result.

[0171] K-means cluster centers need to be iterated continuously, and the calculation formula is as follows:

[0172] ;

[0173] In the formula, n Represents the number of samples in different clusters. Samples representing different clusters.

[0174] The algorithm termination criterion is set as follows: after achieving optimal classification, the sum of squared Euclidean distances between all points and their corresponding cluster centers reaches the global minimum. Its mathematical expression is:

[0175] ;

[0176] In the formula, Representative cluster The central mean line, Representative cluster C The samples in.

[0177] The core of K-means clustering is the selection of initial cluster centers. The choice of initial cluster centers significantly impacts the final result. Poorly chosen initial values ​​can not only slow down convergence but also lead to invalid clustering or even empty clusters. There is no universal method to directly determine the optimal initial cluster centers. K Values ​​are commonly used to help select and evaluate different values. K Value-based strategies include: elbow rule, profile coefficient method, gap statistics method, hierarchical clustering method, business-based or prior knowledge-based method, stability method, etc.

[0178] This invention directly determines the optimal position using the elbow rule. K Values, this method is achieved by drawing different K This is achieved by plotting the curves corresponding to the sum of squared errors (SSE) or distortion within the cluster. Generally... K The value is negatively correlated with the sum of cluster errors because more clusters mean that the data points in each cluster are more closely packed. However, if K If the value is too large, the reduction in SSE becomes less significant, forming a "buck"-like inflection point. This inflection point is generally considered optimal. K value.

[0179] The sum of errors within a cluster can be expressed as:

[0180] ;

[0181] In the formula, Represents the sum of errors within the class. Indicates the number of clusters. express Number of samples within a class Points within the cluster Euclidean distance from the cluster center.

[0182] Reference Figures 1-15 This invention utilizes symbolic aggregation approximation and time-based Markov principles to construct a mathematical model of user electricity consumption behavior patterns. An improved density peak clustering technique is used to identify typical representative users of electricity consumption dynamics and to classify these users into different groups. An anomaly detection model is used to identify outliers after clustering, achieving intelligent identification of abnormal electricity consumption dynamics. Finally, the causes of abnormal electricity consumption behavior are investigated, and diverse abnormal electricity consumption behaviors are classified.

[0183] In this embodiment, five methods were used to fill in missing values: linear interpolation, post-missing value interpolation, shape-preserving piecewise cubic spline interpolation, piecewise cubic spline interpolation, and cubic Hermite polynomial interpolation. By calculating the mean squared error index, it was found that the shape-preserving piecewise cubic spline interpolation method had the smallest sum of squared deviations between the reconstructed sequence and the original observed data compared to traditional linear interpolation and piecewise cubic spline interpolation methods, as shown in Table 1. Therefore, the shape-preserving piecewise cubic spline interpolation method was adopted for handling missing values ​​in the dataset.

[0184] Table 1. Mean Squared Error of Five Missing Value Imputation Methods

[0185]

[0186] A symbolic aggregation approximation method is used to convert load curves into symbolic strings, addressing the issue of high dimensionality and large scale in large load curve datasets and effectively reducing the dimensionality of time-series power data. The key is to find the optimal number of states (characters) and the amplitude breakpoint of the ordinate. The optimal number of states (characters) is found using the average error under different Markov state numbers, and the amplitude breakpoint of the ordinate is found using a histogram represented by a piecewise aggregation approximation. Figure 2 To find the optimal number of states (characters) for the average error under different Markov state numbers, Figure 3 To find the amplitude breakpoints of the ordinate corresponding to the number of states in a histogram approximated by piecewise aggregation, consider a 28-day period of standardized electricity consumption data at a frequency of 30 minutes. The time axis divides each day into four time periods. This data can be represented as "cbbcabbcbbcbabaaacabcabccbba", with 3 symbols and a total of 28 characters, such as... Figure 4 As shown, this effectively reduces the complexity of electricity consumption data.

[0187] The state transition probability matrix of a user is shown below, reflecting the degree and probability of changes in the user's electricity consumption over four time periods. For example, the maximum value of 0.5965 in the first state transition probability matrix means that the user has a greater chance of remaining in state a, i.e., a lower level of electricity consumption, during the transition from stage 1 (00:00-06:30, 22:00-24:00, overnight stage) to stage 2 (06:30-11:30, morning stage).

[0188]

[0189]

[0190] The state transition probability matrix, as an important representation of user electricity consumption behavior characteristics, can be used as input data for cluster analysis. These user groups may exhibit common characteristics or trends in their electricity consumption behavior, which is of great significance for formulating personalized electricity service strategies, optimizing power resource allocation, and detecting anomalies.

[0191] Furthermore, by applying the KL distance calculation formula, a user distance matrix D is obtained, which is a diagonally symmetric distance matrix. This matrix clearly quantifies the differences in state transition probability matrices among different users. It intuitively reflects the differences in electricity consumption behavior patterns among different users.

[0192]

[0193] Then, the local density of each user is calculated independently according to the formula. and distance Indicators are mapped to decision graphs, such as Figure 5 As shown. Selecting the density peak point located in the upper right corner of the decision graph as the cluster center yields a total of 3 clusters. Figure 5 Different colors are used to mark them. Figure 6 A two-dimensional planar mapping of adjacent customers is presented. It can be seen that the customer distribution across four adjacent time periods is not spherical, and the proposed clustering technique can effectively address non-spherical data distribution. This allows customers to be divided into different groups based on their dynamic electricity consumption characteristics. The electricity consumption curve corresponding to each cluster center is considered as the typical electricity consumption pattern of that user group. After obtaining the cluster centers using symbolic aggregation approximation and the Markov-improved DPeaks algorithm, their daily electricity consumption is visualized as follows: Figure 7 .

[0194] An improved DPeaks clustering algorithm and a threshold-based anomaly detection model were used to cluster and identify anomalies in the distance matrix D representing the differences in the quantized Markov state transition probability matrices. The results are shown in the figure below. Figure 8 As shown, it divides all electricity users into three types. Figure 9 This demonstrates how normal and abnormal user labels from experimental data can be used to compare actual abnormal users with those identified by the model. The recall and precision obtained from the Markov-improved DPeaks clustering model experiments indicate the model's effectiveness. The experimental identification results are compared with the actual abnormal users in the dataset. Figure 9 The presence of numerous red concentric circles in the model demonstrates its effectiveness.

[0195] The multi-criteria-DPeaks clustering model is an improvement and refinement of the Markov-DPeaks clustering model. Its complete anomaly detection flowchart is as follows: Figure 10 As shown. The label information of the original dataset is retrieved, and the multiple criteria are compared with... Figure 8 Compare the false positives removed by a single criterion, such as Figure 11 As shown. The multi-criteria improved DPeaks clustering model uses multiple criteria to identify outliers in clusters, removing some falsely detected users based on single criteria. This solves the problems of low detection rate and high false detection rate of traditional anomaly detection models, enabling more accurate and reliable detection and identification of abnormal electricity consumption behavior.

[0196] like Figure 12 , 13 Users identified by both single criteria shown in Table 14 are the anomalous users identified under the multi-criteria model, which possesses the advantages of both single criteria. Table 2 shows that the AUC of all three criteria is greater than 0.9. Under the multi-criteria model, the model demonstrates good performance in anomaly detection. While the recall rate is slightly lower with multi-criteria, its precision far exceeds that of a single criterion. Analysis of the overall AUC shows that the multi-criteria model outperforms the single-criteria model in overall performance.

[0197] Table 2 Performance Comparison of DPeaks Clustering Models Based on Different Criteria

[0198]

[0199] When the number of clusters K=6, the clustering results of the abnormal user data points are as follows: Figure 15 As shown, this method provides a relatively clear classification of all abnormal users identified by the multi-criteria-improved DPeaks clustering anomaly detection model, which is beneficial for analyzing the causes of various abnormal electricity consumption behaviors.

[0200] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting abnormal electricity consumption based on improved clustering using a Markov model, characterized in that, It includes the following steps: S1. Data Preparation and Preprocessing: Obtain load data for the target area, delete users with a high proportion of missing values, and fill in the datasets of users with a low proportion of missing values; reconstruct the data distribution pattern through linear transformation methods and standardize the load data; S2. Establishment of User State Transition Probability Matrix: Based on error comparison, determine the method for filling missing values ​​in the power data; determine the final number of characters and the division of the time axis through the error of the dataset and the actual power consumption pattern; establish the Markov state transition probability matrix in discrete states using symbolic power consumption data; S3. Abnormal power consumption detection: The density peak clustering algorithm is used to cluster the data, and the outlier judgment criteria are set to identify outliers after clustering; the Bonferroni index and slime mold optimization algorithm are introduced to optimize the cutoff distance; S4. Multi-criteria density peak clustering anomaly detection: Establish a multi-criteria density peak clustering anomaly detection model to identify outliers and construct feature indicators to reflect the electricity consumption behavior characteristics of abnormal users; use the K-means algorithm to classify abnormal electricity consumption behaviors, obtain typical electricity consumption anomaly categories, and analyze the causes of abnormal electricity consumption behaviors; construct a typical feature library of electricity safety hazards, distinguish various abnormal electricity consumption behaviors, and handle abnormal electricity consumption behaviors according to specific circumstances.

2. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 1, characterized in that: Step S1 includes: S11. The load data is filled in using piecewise linear interpolation, cubic Hermite polynomial interpolation, and cubic spline interpolation methods based on the real data environment. S12. Standardize the load data using the following deviation normalization formula: ； In the formula, , They are the first The actual value and standardized value of a user's electricity consumption data , These are the minimum and maximum values ​​of the load data over m time periods, respectively.

3. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 1, characterized in that: Step S2 includes: S21. Determining the number of approximate characters for symbol aggregation: The specific number of Markov states and the number of approximate characters for symbol aggregation need to be determined through experiments on real datasets; S22. Establishment of the Markov state transition probability matrix: The characteristics are represented by a time-based Markov model.

4. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 3, characterized in that: Step S22 includes: The standardized load dataset is converted into a symbol string using a symbolic aggregation approximation algorithm, where each letter corresponds to a discrete state in the Markov model. Considering the time-varying dynamic characteristics of user electricity consumption behavior, a Markov chain is modeled for each time period, and a state transition frequency matrix is ​​constructed to quantify the transition patterns between different states. Based on this state transition frequency matrix, the single-step transition probability estimate from state j to state i within period t is calculated using the following formula: ； In the formula, Let be the estimated single-step transition probability from state j to state i within period t. Let be the frequency of state transitions from state j to state i within period t, and let n be the total number of discrete states. It is the sum of the frequencies of all state transitions starting from state j within period t; the single-step transition probability estimate is an unbiased estimate of the state transition probability matrix of the corresponding period.

5. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 1, characterized in that: Step S3 includes: S31. Calculation of relative distance: The difference between the state transition probability matrices of any customer's electricity consumption behavior is measured by measuring the relative distance to obtain typical consumer behavior dynamics; S32. Improved Density Peak Clustering: Improved density peak clustering is used to identify user distributions of arbitrary cluster types, obtain groups with similar power consumption dynamic characteristics, and distinguish power consumption types with different electrical state data characteristics. S33. Abnormal User Identification: An improved density peak clustering algorithm and an anomaly identification model with a set threshold are used to cluster and identify anomalies in the distance matrix of the difference between the quantized Markov state transition probability matrices.

6. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 5, characterized in that: In step S3, the slime mold optimization algorithm optimizes the cutoff distance, and the formula for calculating its foraging location is as follows: ； In the formula, This represents the position vector of the global optimal solution for the slime mold population in the current iteration. For random numbers in Oscillation within a range Food weighting coefficients representing slime molds, as well as It refers to the positions of two random individuals within a slime mold, and the parameters... The value of gradually decreases linearly from 1 to 0 during the iteration process. yes Random numbers between; Ferroni Index The calculation formula is as follows: ； In the formula, the Bonferroni index , used to quantify the non-uniformity of the local density distribution of electricity user samples participating in clustering, with a value range of [0,1]; The larger the value, the more concentrated the local density of the samples is in a few cluster center samples. The higher the cluster discrimination of the density peak cluster and the better the clustering effect. The maximum value is the objective, and the truncation distance is automatically optimized through the slime mold optimization algorithm; N represents the total number of electricity user samples participating in density peak clustering; This is the sample index after all user samples are sorted in ascending order of local density values, with values ​​i=1,2,...,N; The local density of the j-th user sample Before calculation, the local density of all users needs to be sorted in ascending order, which satisfies... ; The sum of the local densities of the first i low-density user samples; This is the total cumulative sum of local densities for all user samples participating in the clustering; The proportion of the first i samples to the total number of samples corresponds to the horizontal axis of the Lorenz curve. The proportion of the cumulative local density of the first i samples to the total density corresponds to the vertical axis of the Lorenz curve. is the normalization coefficient, used to scale the total relative deviation to the [0,1] interval to ensure the comparability of the Bonferroni index under different sample sizes.

7. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 6, characterized in that: Step S3, abnormal user identification, includes: If the density value of a certain sample point Density relative to benchmark Small, and distance value Compared to the reference distance If the value is large, the sample point can be determined to be an outlier. The baseline density and baseline distance of each cluster are calculated as follows: ； ； In the formula, and Corresponding to the c The baseline density and baseline distance of each cluster , These are empirically adjustable parameters; these two parameters together control the sensitivity of the detection threshold. The smaller, The larger the value, the fewer outliers the algorithm identifies according to this criterion. Indicates the first c The total number of samples contained in each cluster. This represents a subset consisting of all observed samples within the cluster.

8. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 1, characterized in that: The multi-criteria density peak clustering anomaly detection model described in step S4 includes two criteria, and a sample point is marked as an anomalous user only when both criteria are met simultaneously. The two criteria for judgment are: Criterion 1: The local density of a sample point is less than the reference density of its cluster, and the relative distance between the sample points is greater than the reference distance of its cluster. Criterion 2: The local density of a sample point is less than the boundary density of its cluster.

9. The method for detecting abnormal electricity consumption based on improved clustering using the Markov model according to claim 8, characterized in that: The boundary density of the cluster includes: Step 1: Set the initial value of the boundary density of all clusters to 0; Step 2: If two sample points belong to different clusters and the Euclidean distance between the two points is less than λ times the cutoff distance, then the average local density of the two sample points is taken as the boundary density of their respective clusters, where λ is an empirically adjustable parameter. Step 3: Repeat step 2 to obtain multiple boundary density values ​​for each cluster after traversing all sample points; Step 4: Select the minimum value among the multiple boundary density values ​​of each cluster as the final boundary density of that cluster.

10. The method for detecting abnormal electricity consumption based on improved clustering using a Markov model according to claim 1, characterized in that: In step S4, feature indicators are constructed to reflect the characteristics of abnormal user electricity consumption behavior; The characteristic indicators include the upward trend in monthly electricity consumption. The downward trend in monthly electricity consumption The calculation formula is as follows: ； ； in, This is a sequence of deviations where the actual value is greater than the predicted value. This is a deviation sequence where the actual value is less than the predicted value. , These represent the lengths of the corresponding sequences.

Images