Method and system for detecting abnormal power consumption of user based on energy internet big data
Through isolated forest algorithm, STL decomposition, MICE multiple fill and KPCA dimensionality reduction technology, the problems of missing and errors in energy data are solved, efficient power consumption anomaly detection is achieved, and energy resource scheduling and user experience are optimized.
Patent Information
- Application Number
- CN202510503136.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-22
- Publication Date
- 2025-08-08
AI Technical Summary
The prior art cannot effectively deal with the missing and error problems in energy data, resulting in inaccurate detection of power abnormalities and affecting energy resource scheduling and user experience.
The isolated forest algorithm is used to detect outliers, combine STL decomposition and MICE multiple filling model to repair missing values, use the KPCA algorithm to reduce dimensionality, and build a high-dimensional feature space anomaly detection model, and determine it with the power fluctuation coefficient.
It improves the accuracy of data quality and abnormal detection, optimizes energy resource scheduling, and improves user power consumption experience.
Smart Images
Figure CN120449030A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of energy internet and big data analysis technology, and in particular to a method and system for detecting abnormal electricity consumption of users based on energy internet big data. Background Art
[0002] Energy data processing and consumer electricity usage monitoring currently face numerous challenges. Firstly, the quality of collected energy data is poor, with widespread issues such as missing data, errors, inconsistencies, and redundancies. These issues directly compromise the accuracy of data processing results and lead to significant errors. For example, when analyzing energy consumption, missing data can prevent the results from truly reflecting actual conditions, while erroneous data can mislead decision-making. Secondly, existing data governance and repair methods are insufficient, making it difficult to effectively address these problematic data.
[0003] Existing technologies are insufficient for detecting anomalies in electricity usage. The lack of effective methods for detecting abnormal electricity usage in high-dimensional data streams prevents timely identification of abnormal user electricity usage through data analysis. This not only leads to irrational scheduling and allocation of energy resources, reducing energy efficiency, but also can negatively impact user experience due to inaccurate electricity usage information. For example, users who are unaware of anomalies in electricity usage may face problems such as power outages or unexpected charges. Summary of the Invention
[0004] This application provides a method and system for detecting abnormal user electricity consumption based on energy Internet big data to solve the problem of energy data quality, achieve accurate detection of abnormal user electricity consumption, and further optimize the scheduling and allocation of energy resources, thereby improving the user's electricity consumption experience.
[0005] The method for detecting abnormal electricity consumption of users based on energy internet big data provided in this application includes:
[0006] The energy consumption data is detected for outliers based on the isolation forest algorithm, the outlier score of each data point is calculated, and the outlier score is compared with the preset threshold to filter out the outliers.
[0007] The detected abnormal data are marked as missing values and repaired according to the data type. For seasonal factor data, they are decomposed into trend terms, seasonal terms and residual terms based on the STL decomposition method. The missing values in the residual terms are repaired using the cubic spline interpolation method, and the repaired residual terms are added to the trend terms and seasonal terms to restore the complete seasonal factor data. For comprehensive energy consumption data, the MICE multiple imputation model is adopted, combining seasonal factors and working day / holiday information to form a feature set. A prediction model is constructed for each missing variable. The prediction model predicts the estimated value of the current missing variable based on the other complete variable values and external environmental variables in the feature set. The estimated results of the missing values are generated multiple times based on the Gibbs sampling method, and the average or weighted average of these multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data.
[0008] After the data repair is completed, the KPCA algorithm is used to reduce the dimensionality of the multi-dimensional energy consumption data;
[0009] Based on the feature data after dimensionality reduction, a high-dimensional feature space anomaly detection model is constructed. This model is used to perform high-dimensional probability density modeling on the feature data after dimensionality reduction, calculate the probability density value of each sample in the model, and reflect the degree of abnormality of the sample in the feature space. At the same time, the power fluctuation coefficient of each user is further calculated to reflect the stability of the user's power consumption. The probability density value and the power fluctuation coefficient are combined as the basis for anomaly judgment. Based on the comparison results of the probability density value and the preset threshold value and the power fluctuation coefficient and the set threshold value, it is determined whether the user has abnormal power consumption behavior;
[0010] Output detection results and record abnormal user information for energy scheduling optimization and abnormal warning.
[0011] Preferably, the outlier score s(x) in the isolation forest algorithm is determined by the following formula:
[0012]
[0013] Where E(h(x)) is the average path length of sample x in all decision trees of the isolation forest, and c(n) is the average path length of the binary tree with a sample size of n. The calculation formula is:
[0014]
[0015] H(n-1) is the harmonic number of n-1.
[0016] As a preference, the calculation formula for the covariance matrix C in the KPCA algorithm is:
[0017]
[0018] Where N is the number of samples, For sample x i The mapped feature vector;
[0019] Solve the eigenvalue equation Cv=λv and use the kernel function K(x i , x j ) replaces the inner product operation, extracts the principal component features, takes the eigenvectors corresponding to the first m eigenvalues to form the matrix Q, and completes the dimensionality reduction projection: Y = QX.
[0020] Preferably, the calculation formula of the probability density value f(x) is:
[0021]
[0022] Where f(x) is the probability density value of sample x under the multivariate Gaussian distribution, x is the eigenvector after dimensionality reduction, d is the dimension of the eigenvector, Σ is the covariance matrix, |Σ| is the determinant of the covariance matrix Σ, μ is the mean vector of the eigenvector, (xu) T is the transpose of (x-μ), Σ -1 is the inverse matrix of the covariance matrix Σ, exp represents the exponential function;
[0023] User's power fluctuation coefficient CF i The calculation formula is:
[0024]
[0025] Among them, CF i is the power fluctuation coefficient of the i-th user, P ij is the electricity consumption of the i-th user on the j-th day, is the average daily electricity consumption of user i within n days, where n is the number of statistical days.
[0026] Preferably, the method for determining abnormal users is: comparing the probability density value f(x) with the set threshold value θ; if f(x) is less than θ and the corresponding user's power fluctuation coefficient is greater than the set threshold, it is determined that the user has abnormal power usage behavior.
[0027] The user abnormal electricity consumption detection system based on energy internet big data provided by the present invention includes:
[0028] The data collection module is used to collect user energy consumption data, including comprehensive energy consumption data and seasonal factor data;
[0029] Anomaly detection module, used to detect outliers in energy consumption data based on the isolation forest algorithm;
[0030] The data cleaning module is used to mark the detected abnormal data as missing values and repair the seasonal factor data and comprehensive energy consumption data based on the STL decomposition method and the MICE multiple imputation model. For the comprehensive energy consumption data, the missing values are estimated multiple times based on the Gibbs sampling method, and the average or weighted average of the multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data;
[0031] Dimensionality reduction analysis module, used to perform dimensionality reduction processing on the repaired energy consumption data based on the KPCA algorithm;
[0032] Anomaly determination module, used to determine abnormal power consumption behavior of users based on the probability density value and power fluctuation coefficient under the multivariate Gaussian distribution;
[0033] The result output module is used to record and output the detection results for use in energy scheduling optimization and abnormal warning.
[0034] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0035] The present invention effectively manages outliers and missing values in energy data through an outlier detection algorithm based on isolation forest and a data joint cleaning algorithm based on STL decomposition and MICE multiple filling model, improves data quality, and provides a guarantee for subsequent accurate data analysis. It also combines KPCA dimensionality reduction, multivariate Gaussian distribution probability density detection and electricity fluctuation coefficient, fully leveraging the superiority of KPCA in nonlinear high-dimensional data dimensionality reduction, and making up for the shortcomings of traditional PCA, TSNE, Umap and other methods in extracting multidimensional and complex features of electricity consumption data. At the same time, by introducing the electricity fluctuation coefficient, it overcomes the uncertainty of single probability density judgment in identifying anomalies in boundary samples and density variation areas, realizes the joint anomaly detection of multidimensional features and time fluctuation features, and significantly improves the accuracy, stability and practicality of electricity anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION
[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0038] like Figure 1 As shown, the method for detecting abnormal power consumption of users based on energy internet big data in this embodiment includes the following steps:
[0039] Step 1: Data collection and preprocessing
[0040] First, the system needs to collect energy consumption data from user energy-consuming devices and external data sources (such as meteorological data, holiday information, etc.). This data includes each user's comprehensive energy consumption data and seasonal factor data. The comprehensive energy consumption data includes but is not limited to daily electricity consumption, maximum load, and power factor. The seasonal factor data includes but is not limited to external meteorological parameters such as temperature, humidity, wind speed, and light intensity.
[0041] Step 2: Outlier Detection
[0042] The electricity consumption data of each user is input as a sample into the isolation forest algorithm. The isolation forest algorithm is an unsupervised learning method suitable for detecting outliers in high-dimensional data. The anomaly score s(x) is calculated for each data point. The formula is:
[0043]
[0044] Where E(h(x)) is the average path length of sample x in all decision trees of the isolation forest, and c(n) is the average path length of the binary tree with a sample size of n. The calculation formula is:
[0045]
[0046] n is the total number of samples involved in the calculation of the isolation forest algorithm, and H(n-1) is the harmonic number of n-1;
[0047] Compare the anomaly score with the preset threshold. If the anomaly score is greater than the preset threshold (such as 0.8), the data point is considered an outlier.
[0048] Step 3. Data repair
[0049] The detected abnormal data are marked as missing values and repaired according to the data type. For seasonal factor data (such as temperature, humidity, etc.), they are decomposed into trend items T based on the STL decomposition method. t , seasonal item S t and the residual term R t , satisfying Y t =T t +S t +R t, using the cubic spline interpolation method to fill in the missing values in the residual term, and then adding the patched residual term to the trend term and seasonal term to restore the complete seasonal factor data; for the comprehensive energy consumption data, the MICE multiple filling model is used for filling. The comprehensive energy consumption data is combined with seasonal factors and workday / holiday information to form a feature set. A prediction model is constructed for each missing variable. The prediction model predicts the estimated value of the current missing variable based on the other complete variable values and external environmental variables in the feature set. The Gibbs sampling method is used to generate the estimated results of the missing values multiple times, and the average or weighted average of these multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data;
[0050] This method assumes that missing data meet the MAR condition and is suitable for the MICE model;
[0051] Step 4: Data dimensionality reduction and feature extraction
[0052] After the data restoration is completed, the KPCA algorithm is used to reduce the dimensionality of the multi-dimensional energy consumption data. The KPCA algorithm can handle nonlinear data mapping and convert high-dimensional data into a low-dimensional feature space while retaining important information.
[0053] The calculation formula of the covariance matrix C in the KPCA algorithm is:
[0054]
[0055] Among them, N is the sample size, which means the number of users participating in the covariance calculation, i is the sample number, (x i ) is the i-th sample x i The eigenvector after mapping to high-dimensional space, T represents the matrix transpose operation;
[0056] Solve the eigenvalue equation Cv=λv and use the kernel function K(x i , x j ) replaces the inner product operation to extract the principal component features, takes the eigenvectors corresponding to the first m eigenvalues to form the matrix Q, and completes the dimensionality reduction projection: Y = QX, which represents the data after being reduced to m dimensions;
[0057] The KPCA dimension reduction dimension m is determined based on the cumulative variance contribution rate reaching more than 80%;
[0058] The KPCA algorithm can extract the main components in the nonlinear mapping feature space, which is suitable for dimensionality reduction of complex nonlinear relationships in energy consumption data. Compared with PCA, which can only process linear relationships, and TSNE and Umap, which have problems with uncertain dimension selection and poor stability, KPCA has a higher performance in pre-anomaly detection feature extraction.
[0059] Step 5: Anomaly Detection Model
[0060] Based on the feature data after dimensionality reduction, a high-dimensional probability density model is constructed. The data is modeled using multivariate Gaussian distribution. For each sample x, its probability density value f(x) in the model is calculated. The formula is:
[0061]
[0062] Where f(x) is the probability density value of sample x under the multivariate Gaussian distribution, x is the eigenvector after dimensionality reduction, d is the dimension of the eigenvector, Σ is the covariance matrix, |Σ| is the determinant of the covariance matrix Σ, μ is the mean vector of the eigenvector, (xu) T is the transpose of (x-μ), Σ -1 is the inverse matrix of the covariance matrix Σ, exp represents the exponential function;
[0063] At the same time, further calculate the power fluctuation coefficient CF of each user i , the formula is;
[0064]
[0065] Among them, CF i is the electricity consumption fluctuation coefficient of the i-th user, n is the number of statistical days, that is, the number of days the user consumes electricity in the statistical period, P ij is the electricity consumption of the i-th user on the j-th day, is the average daily electricity consumption of user i in n days, Σ represents the sum of j = 1 to n, is the square of the electricity consumption deviation, reflecting the deviation between the electricity consumption of the i-th user on the j-th day and his daily average electricity consumption;
[0066] The probability density value reflects the degree of abnormality in the distribution of samples in the feature space. However, considering that some users have high volatility but still fall within a reasonable range, a single probability density value can easily lead to missed or false detections. Therefore, the power fluctuation coefficient is introduced as an auxiliary indicator. The fluctuation coefficient can reflect the fluctuation and stability of users' daily power consumption, effectively compensating for the shortcomings of probability density judgment and improving the reliability of the judgment results.
[0067] Step 6: Abnormal user determination
[0068] Based on the previously calculated probability density value f(x) and the power fluctuation coefficient CF i , by comparison, if f(x) is less than the set threshold value θ, and CF i If the value is greater than the set threshold, the user is judged as an abnormal electricity user;
[0069] Step 7: Output and application of results
[0070] The detection results will be output to the user interface or database to record each user's electricity usage behavior, including the judgment results of normal and abnormal electricity usage. The abnormal user information will be used for subsequent energy scheduling optimization to adjust power distribution or take other response measures.
[0071] The user abnormal electricity consumption detection system based on energy internet big data in this embodiment includes a data acquisition module, an anomaly detection module, a data cleaning module, a dimensionality reduction analysis module, an anomaly determination module and a result output module.
[0072] The data collection module is used to collect users' comprehensive energy consumption data and seasonal factor data;
[0073] The anomaly detection module is used to detect outliers in energy consumption data based on the isolation forest algorithm;
[0074] The data cleaning module is used to mark the detected abnormal data as missing values and repair the seasonal factor data and comprehensive energy consumption data based on the STL decomposition method and the MICE multiple imputation model. For the comprehensive energy consumption data, the missing values are estimated multiple times based on the Gibbs sampling method, and the average or weighted average of the multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data.
[0075] The dimensionality reduction analysis module is used to perform dimensionality reduction processing on the repaired energy consumption data based on the KPCA algorithm;
[0076] The abnormality judgment module is used to judge the abnormal power consumption behavior of users based on the probability density value and power fluctuation coefficient under the multivariate Gaussian distribution;
[0077] The result output module is used to record and output the detection results for use in energy scheduling optimization and abnormal warning.
[0078] The method of the present invention fully integrates multiple advanced methods such as isolation forest, STL decomposition, cubic spline interpolation, MICE multiple filling, KPCA dimensionality reduction, multivariate Gaussian distribution probability density calculation and power fluctuation coefficient analysis, and has the following advantages:
[0079] 1. In response to the multiple types of missing and abnormal problems in energy consumption big data, it can classify and repair data according to data characteristics to improve data cleaning accuracy;
[0080] 2. Combining seasonal factors, holidays and external environmental variables to predict missing values is more comprehensive and flexible than traditional methods, enhancing model adaptability;
[0081] 3. The present invention uses the KPCA algorithm to perform dimensionality reduction processing on multidimensional energy consumption data. Compared with conventional dimensionality reduction algorithms such as PCA, TSNE, and Umap, KPCA can handle the complex mapping relationship between nonlinear features while retaining the main features of the data. It is particularly suitable for the nonlinear and multidimensional complex feature structures existing in energy consumption data, avoiding the problem of misjudgment of outliers caused by linear dimensionality reduction, thereby improving the discrimination accuracy and stability of subsequent anomaly detection;
[0082] 4. Based on the probability density value judgment based on multivariate Gaussian distribution, the power fluctuation coefficient is further introduced as an auxiliary judgment indicator. Single probability density judgment has the problem of being sensitive to local density and easily masking outliers. The power fluctuation coefficient reflects the temporal volatility of user power consumption behavior and complements the probability density value. It helps to distinguish normal fluctuations on the boundary of data distribution from true abnormal power consumption behavior, thereby improving the accuracy and robustness of anomaly judgment;
[0083] 5. The method and process are clear and suitable for deployment and application in actual energy Internet platforms, providing efficient and intelligent support for energy scheduling and energy safety warning.
[0084] This specification and drawings are merely illustrative of the present application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of this application. Thus, this application is intended to include such modifications and variations as fall within the scope of this application and its equivalents.
Claims
1. A method for detecting abnormal electricity consumption of users based on energy internet big data, characterized in that: include: The energy consumption data is detected for outliers based on the isolation forest algorithm, the outlier score of each data point is calculated, and the outlier score is compared with the preset threshold to filter out the outliers. The detected abnormal data are marked as missing values and repaired according to the data type. For seasonal factor data, they are decomposed into trend terms, seasonal terms and residual terms based on the STL decomposition method. The missing values in the residual terms are repaired using the cubic spline interpolation method, and the repaired residual terms are added to the trend terms and seasonal terms to restore the complete seasonal factor data. For comprehensive energy consumption data, the MICE multiple imputation model is adopted, combining seasonal factors and working day / holiday information to form a feature set. A prediction model is constructed for each missing variable. The prediction model predicts the estimated value of the current missing variable based on the other complete variable values and external environmental variables in the feature set. The estimated results of the missing values are generated multiple times based on the Gibbs sampling method, and the average or weighted average of these multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data. After the data repair is completed, the KPCA algorithm is used to reduce the dimensionality of the multi-dimensional energy consumption data; Based on the feature data after dimensionality reduction, a high-dimensional feature space anomaly detection model is constructed. This model is used to perform high-dimensional probability density modeling on the feature data after dimensionality reduction, calculate the probability density value of each sample in the model, and reflect the degree of abnormality of the sample in the feature space. At the same time, the power fluctuation coefficient of each user is further calculated to reflect the stability of the user's power consumption. The probability density value and the power fluctuation coefficient are combined as the basis for anomaly judgment. Based on the comparison results of the probability density value and the preset threshold value and the power fluctuation coefficient and the set threshold value, it is determined whether the user has abnormal power consumption behavior; Output detection results and record abnormal user information for energy scheduling optimization and abnormal warning.
2. The method according to claim 1, characterized in that The outlier score s(x) in the isolation forest algorithm is determined by the following formula: Where E(h(x)) is the average path length of sample x in all decision trees of the isolation forest, and c(n) is the average path length of the binary tree with a sample size of n. The calculation formula is: H(n-1) is the harmonic number of n-1.
3. The method according to claim 1, characterized in that The calculation formula of the covariance matrix C in the KPCA algorithm is: Where N is the number of samples, For sample x i The mapped feature vector; Solve the eigenvalue equation Cv=λv and use the kernel function K(x i , x j ) replaces the inner product operation, extracts the principal component features, takes the eigenvectors corresponding to the first m eigenvalues to form the matrix Q, and completes the dimensionality reduction projection: Y = QX.
4. The method according to claim 1, wherein The calculation formula for the probability density value f(x) is: Where f(x) is the probability density value of sample x under the multivariate Gaussian distribution, x is the eigenvector after dimensionality reduction, d is the dimension of the eigenvector, Σ is the covariance matrix, |Σ| is the determinant of the covariance matrix Σ, μ is the mean vector of the eigenvector, (xu) T is the transpose of (x-μ), Σ -1 is the inverse matrix of the covariance matrix Σ, exp represents the exponential function; User's power fluctuation coefficient CF i The calculation formula is: Among them, CF i is the power fluctuation coefficient of the i-th user, P ij is the electricity consumption of the i-th user on the j-th day, is the average daily electricity consumption of user i within n days, where n is the number of statistical days.
5. The method according to claim 1, wherein The method for determining abnormal users is: comparing the probability density value f(x) with the set threshold value θ. If f(x) is less than θ and the power fluctuation coefficient of the corresponding user is greater than the set threshold, it is determined that the user has abnormal power usage behavior.
6. A user abnormal electricity consumption detection system based on energy internet big data, characterized in that: include: Data collection module, used to collect user energy consumption data and external seasonal factor data; Anomaly detection module, used to detect outliers in energy consumption data based on the isolation forest algorithm; The data cleaning module is used to mark the detected abnormal data as missing values and repair the seasonal factor data and comprehensive energy consumption data based on the STL decomposition method and the MICE multiple imputation model. For the comprehensive energy consumption data, the missing values are estimated multiple times based on the Gibbs sampling method, and the average or weighted average of the multiple estimation results is taken as the final filling result of the missing value to obtain the final repaired comprehensive energy consumption data; Dimensionality reduction analysis module, used to perform dimensionality reduction processing on the repaired energy consumption data based on the KPCA algorithm; Anomaly determination module, used to determine abnormal power consumption behavior of users based on the probability density value and power fluctuation coefficient under the multivariate Gaussian distribution; The result output module is used to record and output the detection results for use in energy scheduling optimization and abnormal warning.