Method for applying time sequence clustering model R-kshape to abnormal electric quantity detection

Abstract

The invention provides a method for detecting abnormal electric quantity by using a time sequence clustering model R-kshape, and the method comprises the steps: obtaining a data set which comprises a unit ID, a date, a moment, and on-grid electric quantity; performing data preprocessing on the acquired data set, wherein the preprocessing comprises missing value processing, zero electric quantity data deletion and reordering; feature engineering is carried out on the preprocessed data set, and derivative variables are constructed; establishing an R-kshape model based on the constructed derivative variables, wherein the R-kshape model is formed by fusing a rule strategy and kshape through logical association; and abnormal electric quantity detection is carried out based on the constructed R-kshape model. According to the method, on the basis of the time sequence clustering algorithm Kshape, the rule strategy (Rule) is fused to form a brand new improved model, the number of needed samples is small, and meanwhile, compared with abnormal electric quantity detection of pure rules and pure clustering, the method has a more accurate and more efficient recognition effect.

Description

Technical Field

[0001] The present invention relates to the field of machine learning algorithms, and in particular to a method for using a time series clustering model R-kshape for abnormal electricity quantity detection. Background Art

[0002] Abnormal power detection technology plays an important role in modern smart grids and energy management systems. As electricity demand continues to increase, the complexity and uncertainty of the power system also increase. Therefore, accurate monitoring and detection of abnormalities in power data is crucial for safe and stable operation of the power grid.

[0003] Traditional detection methods are usually based on rules and statistical methods, including box plots, establishing business rules, etc., which can be used for basic outlier detection, but the disadvantage is that the detection results are not accurate enough. After that, cluster recognition represented by kmeans was gradually used, and the detection effect was improved. However, the biggest disadvantage of kmeans clustering detection is that samples are isolated from each other during detection, while the actual power curve has many connections between adjacent points, that is, the changes between time series must be considered. Subsequently, the Kshape algorithm was proposed. It is good at clustering in time series data. Compared with kmeans clustering, it has further improved in processing time series data, but it is still not enough. It is usually used for user portraits, etc., and there is no case for abnormal power detection. Summary of the invention

[0004] In order to solve the shortcomings of traditional abnormal power detection methods, the present invention proposes a method for abnormal power detection using a time series clustering model R-kshape, and designs an improved time series clustering model R-kshape, which is based on the time series clustering algorithm Kshape and integrates a rule strategy (Rule) to form a new improved model, and is used for abnormal power detection for the first time; it is different from the abnormal power detection model of supervised learning (which requires a large number of pre-labeled samples and consumes manpower), and requires a small number of samples. At the same time, compared with abnormal power detection based on pure rules and pure clustering, it has a more accurate and efficient recognition effect.

[0005] A method for using a time series clustering model R-kshape for abnormal power detection includes the following steps:

[0006] Step 1: Obtain a data set, which includes: unit ID, date, time, and online power consumption;

[0007] Step 2, performing data preprocessing on the data set obtained in step 1, wherein the preprocessing includes missing value processing, deleting zero-point electricity data, and reordering;

[0008] Step 3: Perform feature engineering on the data set preprocessed in step 2 to construct derived variables;

[0009] Step 4: Establish an R-kshape model based on the derived variables constructed in step 3, wherein the R-kshape model is formed by integrating the rule strategy and kshape through logical association;

[0010] Step 5: Perform abnormal power detection based on the R-kshape model constructed in step 4.

[0011] Furthermore, the data set in step 1 is obtained from the data center through SQL.

[0012] Furthermore, step 2 performs data preprocessing on the data set obtained in step 1, specifically including:

[0013] (1) Missing value processing: Delete the data that are directly missing or marked as " / N";

[0014] (2) Delete data with zero electricity: If the online electricity of a data in the data set is 0, delete the data;

[0015] (3) Reordering: Reorder the data set in ascending order of time.

[0016] Furthermore, step 3 specifically includes:

[0017] Construct derivative variables of the previous and next moments: Use shift(1) and shift(-1) functions to process the online power, date, and time, respectively, and obtain 6 derivative variables: the online power at the previous non-zero point, the previous non-zero date, the previous non-zero time, the online power at the next non-zero point, the next non-zero date, and the next non-zero time;

[0018] The derived variable representing the multiple relationship between the current online power consumption and the online power consumption at the previous and next moments is constructed with the following formula:

[0019] Online power 1 = Online power / Online power at the previous non-zero point

[0020] Online power 2 = Online power / Online power at the next non-zero point.

[0021] Furthermore, the step 4 specifically includes:

[0022] Step 4.1: Build kshape time series clustering

[0023] (1) Select K initial time series as cluster centers. The cluster centers are calculated using the elbow method, where K is 5;

[0024] (2) Calculate the dynamic time warping (DTW) distance between each cluster center and each data point. The closer the dynamic time warping (DTW) distance, the higher the similarity.

[0025] (3) Assigning clusters: Based on the similarity metric, the proximity principle and the calculated DTW distance, each data point is assigned to the nearest cluster center, thus forming K clusters. The data points within each cluster have similar characteristics.

[0026] (4) Update cluster center: For each time series in a cluster, calculate the new cluster center by calculating the average value of the time series in the cluster;

[0027] (5) Iteration: repeat the process from (3) to (4) until the cluster center no longer changes significantly or reaches the preset iteration threshold;

[0028] (6) Output K clusters identified by kshape;

[0029] Step 4.2: Build the R-kshape model

[0030] (1) Mark the K clusters of kshape again, calculate the mean of the previous non-zero point online power and the next non-zero point online power of each cluster, and calculate the sum of the means = the previous non-zero point online power + the mean of the next non-zero point online power, automatically select the cluster with the largest sum of the means, mark it as a suspected abnormal point, and mark the rest of the clusters as 0;

[0031] (2) For the suspected outliers marked, each individual in the cluster where the suspected outliers are located is detected according to the anomaly detection rules. The anomaly detection rules for abnormal online power consumption are as follows:

[0032] Rule 1: (Online power 1>2) & (Online power 2>2) & (Online power>1)

[0033] Rule 2: Internet power consumption>1000

[0034] The "&" in rule 1 represents the logical relationship of "and", the logical relationship between rule 1 and rule 2 is "or", and the logical relationship between Kshape and the rules is that the suspected abnormal points identified by kshape meet rule 1 or rule 2, that is:

[0035] For each sample marked as a suspected anomaly point, after the anomaly detection rule strategy is used, the final abnormal power sample is screened out. The dimensions of each sample are: unit ID, date, time, online power, previous non-zero date, previous non-zero time, previous non-zero online power, next non-zero date, next non-zero time, next non-zero online power.

[0036] Based on kshape, the present invention integrates rule detection, proposes the R-kshape model and uses it for abnormal power detection for the first time. As an unsupervised learning model, the R-kshape model requires a small number of samples. Compared with the abnormal power detection methods of time series data based on pure rules (Rule strategy) and pure clustering (Kshape), it has a higher recognition rate and a relatively good recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a method for using the time series clustering model R-kshape for abnormal power detection in the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 The embodiment of the present invention provides a method for using a time series clustering model R-kshape for abnormal power detection, comprising the following steps:

[0040] Step 1: Get the dataset

[0041] The data set comes from the data center. The time is September 2023 in a certain region. The time granularity is hourly (that is, the variable "time"), and it is obtained through SQL. The data set includes the following dimensions: unit ID, date, time, and online power consumption.

[0042] Step 2: Preprocess the data set obtained in step 1, including:

[0043] (1) Missing value processing: There are two types of missing values ​​in the embodiment of the present invention: one is that the data is directly missing, and the other is marked as " / N". Since the purpose of the present invention is to identify abnormal points as accurately as possible, it is chosen to delete rather than fill these missing values.

[0044] (2) Delete the data of zero-point electricity. If the online electricity of the data is 0, delete the data.

[0045] (3) Reordering: There is a small amount of disordered data in the dataset, which will interfere with the model. We reorder the data in increasing order of time.

[0046] Step 3: Perform feature engineering on the data set preprocessed in step 2 to construct derived variables

[0047] Construct derivative variables of the previous and next moments: The processing objects are the online power, date, and time. Use the shift function of Python to process, and use the shift(1) and shift(-1) functions respectively. Since the points with a value of 0 have been deleted in the data cleaning stage, 6 derivative variables are obtained: the online power of the previous non-zero point, the date of the previous non-zero point, the previous non-zero point, the online power of the next non-zero point, the date of the next non-zero point, and the next non-zero point.

[0048] The variables at this time are still not enough to meet the modeling requirements, and lack the multiple relationship between the current online power consumption and the online power consumption at the previous and next moments. Therefore, we continue to construct derived variables, and have the following formula:

[0049] Online power 1 = Online power / Online power at the previous non-zero point

[0050] Online power 2 = Online power / Online power at the next non-zero point

[0051] Step 4: Build an R-kshape model based on the derived variables constructed in step 3

[0052] The step 4 specifically includes:

[0053] Step 4.1: Build kshape time series clustering

[0054] Compared with kmeans, kshape is better at time series clustering, and abnormal power detection is obviously a time series topic. The python3 used in this invention does not call any machine learning library, and only two basic libraries, numpy and pandas, are used to build kshape. The specific steps are as follows:

[0055] (1) Select K initial time series as cluster centers. The cluster centers are calculated using the elbow method, where K is 5.

[0056] (2) Calculate the dynamic time warping (DTW) distance between each cluster center and each data point. DTW distance is different from Euclidean distance and is good at processing data with specific time series characteristics. It calculates the distance between each data point and the cluster center. The closer the distance, the higher the similarity.

[0057] (3) Assign clusters. Based on the similarity metric, the proximity principle and the calculated DTW distance, each data point is assigned to the nearest cluster center, thus forming K clusters. The data points within each cluster have similar characteristics. For example, if the characteristics of a cluster are: the online power was low at the previous moment, the online power is high at the current moment, and the online power is low at the next moment, then the cluster may be judged as an abnormal group.

[0058] (4) Update the cluster center. For each time series within a cluster, calculate the new cluster center. Here, the average value of the time series within the cluster is calculated.

[0059] (5) Iteration: Repeat the process from (3) to (4) until the cluster center no longer changes significantly or reaches the preset iteration threshold. The iteration threshold set here is 500 times.

[0060] (6) Output the K clusters identified by kshape.

[0061] Step 4.2: Build the R-kshape model

[0062] R-kshape is formed by combining rule strategy and kshape through logical association. The process is as follows:

[0063] (1) Mark the K clusters of kshape again, calculate the mean of the previous non-zero point online power and the next non-zero point online power of each cluster, and calculate the sum of the means = the previous non-zero point online power + the mean of the next non-zero point online power, automatically select the cluster with the largest sum of the means, mark it as a suspected abnormal point, and mark the rest of the clusters as 0;

[0064] (2) For points marked as suspected outliers, the overall mean of the cluster meets the abnormal characteristics, but it does not mean that each individual of the cluster meets the abnormal characteristics. Therefore, each individual of the cluster needs to pass the anomaly detection rules.

[0065] Basis for establishing anomaly detection rules: The abnormal phenomenon of online power consumption is mainly a sudden sharp increase at a certain point in time. By consulting the data, there is no clear definition of the increase multiple, but it is at least greater than 2 times. At the same time, it is also necessary to consider that the online power consumption cannot be too low.

[0066] Therefore, the abnormal online power consumption is defined as follows:

[0067] Rule 1: (Online power 1>2) & (Online power 2>2) & (Online power>1)

[0068] Rule 2: Internet power consumption>1000

[0069] The "&" in rule 1 represents the logical relationship of "and", and the logical relationship between rule 1 and rule 2 is "or" (it should be noted that the rule strategies adopted in different scenarios are different). The logical relationship between Kshape and the rules is that the suspected anomaly points identified by kshape meet rule 1 or rule 2, that is:

[0070] For each sample marked as a suspected anomaly point, after the anomaly detection rule strategy is used, the final abnormal power sample is screened out. The dimensions of each sample are: unit ID, date, time, online power, previous non-zero date, previous non-zero time, previous non-zero online power, next non-zero date, next non-zero time, next non-zero online power.

[0071] In the Python program, the above R-kshape model construction process is implemented inside the custom R-kshape function, and the identified clusters are returned through the return statement.

[0072] Step 5: Perform abnormal power detection based on the R-kshape model constructed in step 4.

[0073] Results Analysis

[0074] This model is an unsupervised learning model, so there is no need to divide the training set. The data set is copied into three parts and run in three ways: rule strategy, kshape model, and R-kshape model. The results of manual identification in advance are used as the real abnormal power for comparison. The recognition rate (i.e., the accuracy of identification) and the recognition rate (i.e., the completeness of identification) are used as indicators, and the following definitions or formulas are given:

[0075] Number of matches: The number of identified anomalies that are also true anomalies

[0076] Recognition rate = number of matches / number of recognized anomalies

[0077] Recognition rate = number of matches / number of manual recognitions

[0078] The results are shown in Table 1 below:

[0079] Table 1 Results comparison

[0080]

[0081] From the above table, we can see that the R-kshape model has the highest recognition rate, reaching 93.3%. The main advantage of the Kshape model is that it can recognize all truly abnormal samples, but it also "accidentally" hurts many samples, identifying a large number of normal samples as abnormal. Finally, the recognition rate of Rule still needs to be improved.

[0082] Overall, the R-kshape model has the best effect.

[0083] Based on kshape, the present invention integrates rule detection, proposes the R-kshape model and uses it for abnormal power detection for the first time. As an unsupervised learning model, it requires a small number of samples. Compared with the time series data abnormal power detection methods of pure rules (Rule strategy) and pure clustering (Kshape), it has a higher recognition rate and a relatively good recognition rate.

[0084] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for abnormal power detection using a time series clustering model R-kshape, characterized in that: The steps include: Step 1: Obtain a data set, which includes: unit ID, date, time, and online power consumption; Step 2, performing data preprocessing on the data set obtained in step 1, wherein the preprocessing includes missing value processing, deleting zero-point electricity data, and reordering; Step 3: Perform feature engineering on the data set preprocessed in step 2 to construct derived variables; Step 4: Establish an R-kshape model based on the derived variables constructed in step 3, wherein the R-kshape model is formed by integrating the rule strategy and kshape through logical association; Step 5: Perform abnormal power detection based on the R-kshape model constructed in step 4.

2. The method for abnormal power detection using the time series clustering model R-kshape as claimed in claim 1 is characterized by: The data set in step 1 is obtained from the data center through SQL.

3. The method for abnormal power detection using the time series clustering model R-kshape as claimed in claim 1 is characterized by: Step 2 performs data preprocessing on the data set obtained in step 1, including: (1) Missing value processing: Delete the data that are directly missing or marked as " / N"; (2) Delete data with zero electricity: If the online electricity of a data in the data set is 0, delete the data; (3) Reordering: Reorder the data set in ascending order of time.

4. The method for abnormal power detection using the time series clustering model R-kshape as claimed in claim 1 is characterized by: Step 3 specifically includes: Construct derivative variables of the previous and next moments: Use shift(1) and shift(-1) functions to process the online power, date, and time, respectively, and obtain 6 derivative variables: the online power at the previous non-zero point, the previous non-zero date, the previous non-zero time, the online power at the next non-zero point, the next non-zero date, and the next non-zero time; The derived variable representing the multiple relationship between the current online power consumption and the online power consumption at the previous and next moments is constructed with the following formula: Online power 1 = Online power / Online power at the previous non-zero point Online power 2 = Online power / Online power at the next non-zero point.

5. The method for abnormal power detection using the time series clustering model R-kshape as claimed in claim 1 is characterized by: The step 4 specifically includes: Step 4.1: Build kshape time series clustering (1) Select K initial time series as cluster centers. The cluster centers are calculated using the elbow method, where K is 5; (2) Calculate the dynamic time warping (DTW) distance between each cluster center and each data point. The closer the dynamic time warping (DTW) distance, the higher the similarity. (3) Assigning clusters: Based on the similarity metric, the proximity principle and the calculated DTW distance, each data point is assigned to the nearest cluster center, thus forming K clusters. The data points within each cluster have similar characteristics. (4) Update cluster center: For each time series in a cluster, calculate the new cluster center by calculating the average value of the time series in the cluster; (5) Iteration: repeat the process from (3) to (4) until the cluster center no longer changes significantly or reaches the preset iteration threshold; (6) Output K clusters identified by kshape; Step 4.2: Build the R-kshape model (1) Mark the K clusters of kshape again, calculate the mean of the previous non-zero point online power and the next non-zero point online power of each cluster, and calculate the sum of the means = the previous non-zero point online power + the mean of the next non-zero point online power, automatically select the cluster with the largest sum of the means, mark it as a suspected abnormal point, and mark the rest of the clusters as 0; (2) For the suspected outliers marked, each individual in the cluster where the suspected outliers are located is detected according to the anomaly detection rules. The anomaly detection rules for abnormal online power consumption are as follows: Rule 1: (Online power 1>2) & (Online power 2>2) & (Online power>1) Rule 2: Internet power consumption>1000 The "&" in rule 1 represents the logical relationship of "and", the logical relationship between rule 1 and rule 2 is "or", and the logical relationship between Kshape and the rules is that the suspected abnormal points identified by kshape meet rule 1 or rule 2, that is: For each sample marked as a suspected anomaly point, after the anomaly detection rule strategy is used, the final abnormal power sample is screened out. The dimensions of each sample are: unit ID, date, time, online power, previous non-zero date, previous non-zero time, previous non-zero online power, next non-zero date, next non-zero time, next non-zero online power.

Images