A method and system for detecting abnormal data in power systems
By combining a hybrid model of support vector machines and tree structures, integrating anomaly scores and setting an optimal threshold, the problem of missed detection rate in the detection of various anomalies in power data in existing technologies is solved, achieving more accurate anomaly judgment and system stability.
Patent Information
- Application Number
- CN202510625995.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from a high rate of missed anomaly detection when processing power data that contains multiple anomaly types and rich feature dimensions, making it difficult to accurately detect anomalies.
A hybrid model based on support vector machines and tree structures is adopted, combining the modeling ability of support vector machines for normal data and the high sensitivity of tree structures to outliers. By integrating the anomaly scores of different detection models and setting fusion, and adding an optimized threshold calculation method, more accurate anomaly detection is achieved.
It effectively improves the accuracy and robustness of anomaly detection, reduces the rate of missed anomaly data detection, and ensures the stable operation of the power system.
Smart Images

Figure CN123116033A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of power data technology, and in particular to a method and system for detecting abnormal data in power systems. Background Technology
[0002] With the accelerated construction of new power systems, the number of smart sensors in the power grid has increased significantly, generating massive amounts of power data that place new demands on the processing capabilities of data processing platforms. This power data encompasses various dimensions, including real-time power generation, consumption, and power output. However, due to factors such as equipment failure, grid instability, and external interference, the power data stored on data servers contains anomalous data. This anomalous data can severely impact power system stability assessment, data analysis, energy dispatch, and fault diagnosis, reducing the accuracy and operational efficiency of the power system. Therefore, a scientific and effective method is needed to effectively identify and process this anomalous data, ensuring the reliability and operational efficiency of the new power system. Existing methods for identifying anomalous data include statistical analysis, clustering-based methods, proximity-based methods, and machine learning methods. These methods have drawbacks, including parameter sensitivity, difficulty in handling high-dimensional data, and poor performance in datasets with a large number of anomalous data types.
[0003] The "Method for Detecting Abnormal Power Data" disclosed in Chinese patent literature, publication number CN119577649A, and publication date 2025-03-07, includes the following steps: Step 1: Collect power system operation data, and divide the power system operation data into several sliding windows by setting the sliding window size and sliding step size; Step 2: Compare the power system operation data within each pair of adjacent sliding windows to detect abnormal sliding windows; Step 3: Normalize the power system operation data within the abnormal sliding windows to obtain normalized data; Step 4: Calculate the similarity between each normalized data and normal power data using a similarity detection method, and identify the normalized data whose similarity exceeds a set similarity range as abnormal data. However, this technology is based on statistical analysis for detecting data anomalies in massive amounts of power data. It is suitable for situations where the abnormal data type is relatively simple. When the power data package contains multiple anomaly types and rich feature dimensions, it suffers from a high rate of missed detection of abnormal data, making accurate anomaly detection difficult. Summary of the Invention
[0004] The present invention aims to overcome the problem in the prior art that there is a high rate of missed detection of abnormal data and difficulty in accurately detecting abnormal data in power data containing multiple types of anomalies and rich feature dimensions, and provides a method and system for detecting abnormal data in power systems.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting abnormal data in a power system, comprising: Acquire power system data and perform data normalization and feature filtering; Train a support vector machine model and calculate its decision function value as the first outlier score; train an isolation forest model and calculate its outlier score as the second outlier score. The first and second outlier scores are weighted and summed to obtain the final outlier score. Anomaly thresholds are set based on minimizing the cost function, and data anomalies are determined based on the final anomaly score.
[0006] This invention employs a hybrid model based on support vector machines (SVMs) and tree structures. This model combines the SVM's ability to model normal data with the tree structure's high sensitivity to outliers, effectively improving the accuracy and robustness of anomaly detection. It can also better handle complex data structures and various anomaly types, thereby effectively reducing the false negative rate of anomaly data. By integrating the anomaly scores from different detection models, setting a fusion method, and adding an optimized threshold calculation method, more accurate anomaly judgment is achieved. This not only improves the accuracy of detection but also enhances the robustness of the system, ensuring the stable operation of the power system.
[0007] Preferably, the data normalization and feature filtering include: The minimum-maximum normalization method is used to normalize the power system data, mapping the data to the interval [0, 1]. The normalized power system data is initially screened through data cleaning; then feature screening is performed using either a feature screening based on preset indicators or a data-driven feature screening.
[0008] Preferably, the process of training the support vector machine model includes: Construct a hypersphere boundary to cover the normal data; The objective function is to minimize the weighted sum of slack variables and the sum of the hypersphere's volume; the constraint condition is that the distance of a data point beyond the hypersphere's boundary is less than or equal to the corresponding slack variable. The support vector machine model is then trained.
[0009] Preferably, the process of training the isolated forest model includes: A forest is formed by recursively constructing multiple isolated trees by randomly selecting features and split points, and the path length of each tree is calculated. The path length is the sum of the number of edges traversed by sample x from the root node to the leaf node of the tree and the corrected path. The corrected path is the average path length of the binary tree constructed from samples that share the same leaf node as sample x.
[0010] Preferably, the process of constructing an isolation tree includes: randomly selecting a feature from the dataset after feature filtering; randomly selecting a split point between the minimum and maximum values of the dataset with the selected feature; and splitting the dataset into two subsets according to the selected feature and the split point. For each subset, repeat the selection and segmentation steps for the splitting point until the segmentation stopping condition is met.
[0011] Preferably, the calculation of its decision function value as the first anomaly score includes: for the data sample x to be judged, the first anomaly score is obtained by subtracting the threshold of the decision function after weighted summation of the kernel function values between sample x and all support vectors; the weight of the kernel function value between sample x and the i-th support vector is the Lagrange multiplier of the i-th support vector.
[0012] Preferably, calculating its anomaly score as the second anomaly score includes: for a sample x out of a total of n data samples, The second outlier score is calculated by dividing the average path length of sample x in all isolated trees by the negative of the average path length of the binary tree constructed from n data samples, with 2 as the base.
[0013] Preferably, the weighting of the first abnormal score and the second abnormal score includes: the sum of the weights of the two abnormal scores is one; A preliminary optimal solution is obtained by traversing the ownership reorganization and merging of the power system data validation set through grid search. The weights of the preliminary optimal solution are then fine-tuned based on the sensitivity differences of different outlier data using the isolated forest model and support vector machine model to obtain the final weight allocation result.
[0014] Preferably, the cost function is the sum of the cost of false alarms and the cost of false alarms; The cost of a false alarm is the product of the number of false alarms below the anomaly threshold and the cost weight of a false alarm relative to a missed alarm. The cost of underreporting is 1 minus the difference between the cost weight of false positives and underreporting, multiplied by the number of underreporting under the anomaly threshold; the anomaly threshold calculated by minimizing the cost function is used as the final anomaly threshold for anomaly judgment.
[0015] A power system abnormal data detection system, comprising: The data preprocessing module acquires power system data and performs data normalization and feature filtering. The model training module trains the support vector machine model and the isolated forest model, and calculates the outlier scores for both. The weight allocation module assigns weights to the first and second abnormal scores and sums them to obtain the final abnormal score; the abnormal threshold setting module sets the abnormal threshold based on minimizing the cost function.
[0016] The present invention has the following beneficial effects: It uses a hybrid model based on support vector machines (SVMs) and tree structures, combining the modeling ability of SVMs for normal data with the high sensitivity of tree structures to outliers. This effectively improves the accuracy and robustness of anomaly detection and can better handle complex data structures and various anomaly types, thereby effectively reducing the false negative rate of anomaly data. Furthermore, by integrating anomaly scores from different detection models, setting a fusion method, and adding an optimized threshold calculation method, the present invention achieves more accurate anomaly judgment, not only improving detection accuracy but also enhancing system robustness and ensuring the stable operation of the power system. Attached Figure Description
[0017] Figure 1 This is a flowchart of the power system abnormal data detection method in this invention.
[0018] Figure 2 This is another flowchart of the power system abnormal data detection method in this invention. Detailed Implementation
[0019] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.
[0020] Existing methods for anomaly identification include statistical analysis, clustering-based methods, proximity-based methods, and machine learning methods. These methods suffer from drawbacks such as parameter sensitivity, difficulty in handling high-dimensional data, and poor performance in datasets with a large number of anomalous data types. To avoid these shortcomings and further improve the efficiency of anomaly identification, this invention constructs a hybrid model based on Support Vector Machines (SVM) and tree structures (such as isolated forests). This model effectively identifies and filters outliers in complex data, significantly reducing the proportion of anomalous data and thus improving data quality. This provides a reliable data foundation for the stable operation and accurate analysis of power systems.
[0021] like Figure 1 As shown, a method for detecting abnormal data in a power system includes: Acquire power system data and perform data normalization and feature filtering; Train a support vector machine model and calculate its decision function value as the first outlier score; train an isolation forest model and calculate its outlier score as the second outlier score. The first and second outlier scores are weighted and summed to obtain the final outlier score. Anomaly thresholds are set based on minimizing the cost function, and data anomalies are determined based on the final anomaly score.
[0022] It should be noted that this invention uses a hybrid model based on support vector machines and tree structures, combining the modeling ability of support vector machines for normal data with the high sensitivity of tree structures to outliers. This effectively improves the accuracy and robustness of anomaly detection and can better handle complex data structures and various anomaly types, thereby effectively reducing the false negative rate of anomaly data. By integrating the anomaly scores of different detection models, setting a fusion method, and adding an optimized threshold calculation method, more accurate anomaly judgment is achieved. This not only improves the accuracy of detection but also enhances the robustness of the system, ensuring the stable operation of the power system.
[0023] Specifically, a large amount of power system data is first acquired as the initial dataset. After data normalization and feature filtering, the preprocessing process is completed to obtain a dataset that meets the requirements for subsequent training. Data normalization is to eliminate the huge differences in the dimensions, features, and numerical ranges between different power system data to facilitate subsequent training calculations. Feature filtering is to select the datasets corresponding to important and necessary features and ignore irrelevant datasets, thereby reducing the amount of training calculations.
[0024] For the dataset after feature selection, both the Isolation Forest model and the Support Vector Machine (SVM) model are trained. For the data in the dataset, the objective function is to minimize the weighted sum of slack variables and the sum of the hypersphere's volume; the constraint is that the distance of a data point beyond the hypersphere's boundary is less than or equal to the corresponding slack variable. The SVM model is then trained by recursively constructing multiple isolated trees to form a forest through randomly selected features and split points, and calculating the path length of each tree.
[0025] After training the Support Vector Machine (SVM) model, the first anomaly score of the data sample to be evaluated is obtained by calculating the decision function value. Similarly, after training the Isolation Forest (IS) model, the second anomaly score is calculated for the same data sample. This yields the anomaly scores of the data sample under both models. The first and second anomaly scores are then weighted and summed to obtain the final anomaly score, which serves as the evaluation result for the data sample. The weights assigned to the two anomaly scores can be pre-set according to different models or dynamically adjusted based on actual needs.
[0026] After obtaining the final anomaly score of the data sample to be evaluated, it is compared with the anomaly threshold. If the final anomaly score is greater than the anomaly threshold, the data sample is judged to be anomaly; otherwise, it is considered normal data. The anomaly threshold can be calculated by minimizing a cost function that considers both false positive and false negative costs. The application of the cost function in threshold setting makes the threshold setting more scientific and reasonable, balancing the costs of false positives and false negatives to achieve the best detection results.
[0027] As a specific example, data normalization and feature filtering include: The minimum-maximum normalization method is used to normalize the power system data, mapping the data to the interval [0, 1]. The normalized power system data is initially screened through data cleaning; then feature screening is performed using either a feature screening based on preset indicators or a data-driven feature screening.
[0028] It should be noted that due to the high dimensionality of power data and the significant differences in the dimensions and numerical ranges among various features, the min-max normalization method is used to normalize the data, mapping it to the [0, 1] interval to eliminate the influence of dimensions. The core advantages of min-max normalization are its fixed output range and computational efficiency, making it particularly suitable for scenarios where power data exhibits large differences in feature dimensions and lacks significant outliers. If the data contains extreme values or has a complex distribution, it can be combined with other methods (such as robust normalization or nonlinear transformation) for optimization.
[0029] Specifically, the feature selection process consists of two steps. The first step is to perform preliminary data cleaning and screening, which includes: using the variance threshold method to remove features with variance close to 0 (such as the constant "Device Status Code_Not Enabled" field); removing missing values by deleting fields with a null value rate greater than 80% (such as "Backup Sensor Reading"); and deleting redundant fields by directly removing metadata that is irrelevant to the target (such as "Device ID" and "Record Timestamp").
[0030] After the initial screening is completed, a subsequent feature screening is performed to complete the final feature selection. You can choose to use either a feature screening based on preset indicators or a data-driven feature screening approach.
[0031] Feature initial screening based on preset indicators involves collaborating with power experts to develop a list of key indicators, such as: mandatory features (directly affecting equipment status): voltage, current, active power, harmonic distortion rate, temperature, etc.; excluded features (deemed irrelevant or redundant by experts): equipment serial number (unique identifier, statistically insignificant), historical alarm count (irrelevant to real-time status), etc. Datasets containing mandatory features are then selected for subsequent model training; or, after excluding excluded features, the remaining dataset is used for subsequent model training.
[0032] Data-driven feature selection refers to further screening through statistics and model evaluation. Examples include: correlation analysis (calculating the Pearson correlation coefficient between features and the target variable, retaining features with |r|>0.3); mutual information method (selecting the top 10% of features with the highest mutual information value to the target variable); and recursive feature elimination (RFE) (iteratively removing the lowest 20% of features using a random forest model). The datasets corresponding to the selected and retained features are then used for subsequent model training.
[0033] As a specific example, the process of training a support vector machine model includes: Construct a hypersphere boundary to cover the normal data; The objective function is to minimize the weighted sum of slack variables and the sum of the hypersphere's volume; the constraint condition is that the distance of a data point beyond the hypersphere's boundary is less than or equal to the corresponding slack variable. The support vector machine model is then trained.
[0034] It's worth noting that in the SVM (Support Vector Machine) model, a One-Class SVM model is used as the anomaly detection model. This is because One-Class SVM can construct a compact boundary containing only normal data by finding support vectors, utilize kernel functions to handle non-linear data, and can handle complex non-linear data distributions. It is also highly sensitive to outliers, effectively identifying anomalous data located outside the boundary. Furthermore, this model only requires normal samples for training, avoiding dependence on anomalous samples.
[0035] Specifically, the goal is to classify all non-outlier samples as positive. A hypersphere is used for this classification, and a spherical boundary is obtained around the data points in the feature space. The aim is to minimize the volume of this hypersphere, thereby minimizing the impact of outlier data. Assume the generated hypersphere parameters are a center o and a corresponding hypersphere radius r > 0. The hypersphere volume V(r) is minimized, and the center o is a linear combination of support vectors. Similar to traditional SVM methods, it can be required that the distance from all training data points xi to the center is strictly less than r. However, a slack variable ξ with a penalty coefficient C is constructed simultaneously. i This allows some data points to violate the boundary constraints of the hypersphere, thereby improving the model's tolerance for outlier data.
[0036] In real-world scenarios, data may contain a few outliers or noise that cannot be completely enclosed by the hypersphere. Directly forcing all data to strictly meet the constraints would cause the hypersphere to expand to an unreasonable range. Introducing the slack variable ξ... i The distance by which the i-th data point slightly exceeds the boundary can be quantized as ξ. i The objective function uses a penalty coefficient C to weight and sum these deviations, thus achieving a balance between the rigid requirement of minimizing the hypersphere volume and the flexibility to tolerate a small number of outliers. The optimization problem is as follows: ||x i -o||2≤r+ξ i i = 1, 2, 3...m ξ ι ≥0, i=1,2,...m The parameter *m* represents the total number of samples in the training dataset. On one hand, it provides flexibility to prevent the model from overfitting due to a few outliers (e.g., preventing the hypersphere from becoming excessively large just to cover a few noisy points). On the other hand, it adjusts the model's sensitivity to deviations by controlling the size of the penalty term *C*. If the penalty coefficient *C* is large, the model will more strictly penalize the slack variables, resulting in a compressed hypersphere volume but potentially missing some true data. Conversely, when *C* is small, the model allows for more deviations, resulting in a looser hypersphere but potentially stronger generalization ability. This design allows the model to freely adjust the tightness of the boundaries according to the distribution of the actual data.
[0037] Finally, the optimal center point o and radius r of the hypersphere, as well as the corresponding slack variable ξ, are determined. i The value of ξ is determined by the distance of these support vectors from the data points in the feature space. Essentially, it is a linear combination of support vectors (i.e., key data points located at the boundary of the hypersphere), similar to the weighted result of support vectors in traditional SVM. The radius r is determined by the distance of these support vectors from the center o. The resulting hypersphere can cover most normal data points with a minimal volume, while the slack variable ξ... i Then mark the offset magnitude of the points that are allowed to deviate from the boundary.
[0038] Furthermore, calculating the decision function value of the support vector machine model as the first anomaly score includes: for the data sample x to be judged, the first anomaly score is obtained by subtracting the threshold of the decision function after weighted summation of the kernel function values between sample x and all support vectors; the weight of the kernel function value between sample x and the i-th support vector is the Lagrange multiplier of the i-th support vector.
[0039] Specifically, the anomaly score of the One-Class SVM model is calculated as follows: For a data sample x, calculate its kernel function value with respect to all support vectors, and use the following formula: Where n sv α is the number of support vectors, automatically determined during the support vector machine model training process, and specifically depends on the data distribution and hyperparameters. i It is the Lagrange multiplier of the i-th support vector, k(x) i ,x) is the kernel function value between sample x and the i-th support vector x, and ρ is the threshold (intercept) of the decision function, which is automatically optimized by the training process.
[0040] As a specific example, the process of training an isolated forest model includes: A forest is formed by recursively constructing multiple isolated trees by randomly selecting features and split points, and the path length of each tree is calculated. The path length is the sum of the number of edges traversed by sample x from the root node to the leaf node of the tree and the corrected path. The corrected path is the average path length of the binary tree constructed from samples that share the same leaf node as sample x.
[0041] It's important to note that in tree-based models, the core idea of building an Isolation Forest algorithm is to construct multiple trees, forming a forest, by randomly selecting features and split values. Each tree is a binary tree used to isolate data points. The term "isolated" in the algorithm means that outliers are often far from most data points, thus requiring fewer splits in the tree to be isolated. Normal data points, being closer together, require more splits to be isolated. Therefore, Isolation Forest determines the degree of anomaly by evaluating how many splits a sample needs to be isolated.
[0042] It's worth noting that the calculation logic for PathLength is similar to that of a binary classification model. During prediction, it can output P(y=1), and iforest isolated forest can also output an anomaly score. The number of splits (i.e., the edges traversed by the data sample when it falls into a leaf node) is used as a metric, and for t trees, the average is taken.
[0043] The formula for calculating PathLength is: h(x) = e + c(T.size) Where e is the number of edges that sample x traverses from the root node to a leaf node of the tree, i.e., the number of splits. T.size represents the number of samples that share a leaf node with sample x, and C(T.size) is a correction value that represents the average path length of a binary tree constructed from T.size samples.
[0044] The formula for calculating c(n) is as follows: Where 0.5772156649 is Euler's constant.
[0045] Furthermore, the process of constructing an isolation tree includes: randomly selecting a feature from the feature-filtered dataset; randomly selecting a split point between the minimum and maximum values of the dataset with the selected feature; and splitting the dataset into two subsets based on the selected feature and the split point. For each subset, repeat the selection and segmentation steps for the splitting point until the segmentation stopping condition is met.
[0046] Specifically, isolated forest training includes: (1) Random feature selection: Randomly select a feature from the dataset after feature selection; (2) Randomly select a split point: On the selected feature, randomly select a split point. This split point is between the minimum and maximum values of the current dataset on that feature.
[0047] (3) Data segmentation: Based on the selected features and segmentation points, the dataset is divided into two subsets. One subset contains data less than or equal to the segmentation point, and the other contains data greater than the segmentation point.
[0048] (4) Recursively build the tree: Repeat steps 1-3 for each subset until one of the following conditions is met: there is only one data or no data in the subset; the tree reaches the preset height.
[0049] (5) Building a forest: Repeat the above process several times. Each time, a subset is randomly selected from the dataset after feature selection. Multiple trees are built by randomly selecting the subset and recursively dividing it to form a forest.
[0050] Furthermore, calculating the outlier score of the isolated forest model as the second outlier score involves: for a sample x out of a total of n data samples, The second outlier score is calculated by dividing the average path length of sample x in all isolated trees by the negative of the average path length of the binary tree constructed from n data samples, with 2 as the base.
[0051] Specifically, for each data point, the second outlier score is calculated using the following formula: Where h(x) is the path length of a sample on the Isolation Tree (iTree) of the Isolation Forest algorithm; E(h(x)) is the average path length of a sample on t iTrees; and c(n) is the average path length of a BST binary tree constructed from n samples.
[0052] As a specific implementation, the weight allocation of the first abnormal score and the second abnormal score can be implemented by setting fixed weights, and the sum of the weights of the two abnormal scores is one; after obtaining the two abnormal scores, the weighted calculation is performed directly according to their respective fixed weights; or, under the premise that the sum of the weights of the two abnormal scores is one, the two weights are dynamically adjusted and allocated.
[0053] like Figure 2 In the power system anomaly data detection method shown, the weight allocation for the two anomaly scores includes: obtaining a preliminary optimal solution by traversing the ownership reorganization and merging of the power system data validation set through grid search; The weights of the initial optimal solution are fine-tuned based on the sensitivity differences of different outlier data using the isolated forest model and the support vector machine model, to obtain the final weight allocation result.
[0054] It should be noted that this invention combines the first anomaly score of the One-Class SVM model with the second anomaly score of the Isolation Forest model using a weighted summation formula. After debugging on an actual test dataset, the weights are optimized on the validation set using grid search (the validation set is derived from a feature-filtered dataset, which is divided into training and validation sets in a certain proportion; the model is trained using the training set, and the weights are assigned to the trained model using the validation set). Furthermore, fine-tuning is done based on the differences in sensitivity of different models to different anomaly data (e.g., differences in sensitivity to current / voltage anomalies), assigning weight coefficients to the two models to obtain a more robust and accurate anomaly detection result. The weighted calculation formula is as follows: Where S final (x) is the final outlier score after fusion, S i (x) is the anomaly score of the i-th model, w i These are the weights assigned to the i-th model, and
[0055] In power system anomaly detection scenarios, the weight coefficient allocation of the hybrid model is achieved through the following steps: 1. Grid search parameter space definition: Based on model characteristics, a candidate range for weight coefficients is preset (e.g., [0.1, 0.9], [0.3, 0.7], etc.), forming a parameter grid. 2. Evaluation metrics (e.g., F1 score, AUC score, or false alarm / false negative cost function) are set to quantify the detection performance of different weight combinations. 3. Cross-validation to optimize weights: Using K-fold cross-validation, all weight combinations in the parameter grid are traversed on the validation set, recording the evaluation metric score corresponding to each weight group. 4. The weight combination with the highest score is selected as the initial optimal solution.
[0056] Fine-tuning of sensitivity differences in weights: Adjusting the weight range or step size based on the actual needs of power system anomaly detection and the varying sensitivities of different models to anomaly detection data. For example, if the Isolation Forest model is more sensitive to anomalies in specific equipment in historical data, its upper weight limit can be appropriately increased; if the SVM model performs stably in low-noise scenarios, its weight percentage can be retained; models more sensitive to current / voltage anomalies can have their weights appropriately increased, and so on. Verifying the rationality of weight allocation through expert experience avoids overfitting issues caused by purely data-driven approaches. Dynamic optimization and validation: Periodically re-executing the grid search based on new data and updating the weight coefficients to adapt to changes in system operating status.
[0057] As a specific implementation, an anomaly threshold is set based on minimizing a cost function, and the data anomaly is determined based on the final anomaly score. The cost function is the sum of the cost of false positives and the cost of false negatives. The cost of a false alarm is the product of the number of false alarms below the anomaly threshold and the cost weight of a false alarm relative to a missed alarm. The cost of underreporting is 1 minus the difference between the cost weight of false positives and underreporting, multiplied by the number of underreporting under the anomaly threshold; the anomaly threshold calculated by minimizing the cost function is used as the final anomaly threshold for anomaly judgment.
[0058] It should be noted that setting the anomaly threshold is a crucial step in the anomaly detection process. To ensure optimal detection results, a cost function is used to set the threshold. A cost function that minimizes the costs of false positives and false negatives is set, and the calculation formula is as follows: Cost(T)=λ*FP(T)+(1-λ)*FN(T) Where Cost(T) is the total cost under the anomaly threshold T, FP(T) is the number of false alarms under the anomaly threshold T, FN(T) is the number of false alarms under the anomaly threshold T, and λ is the cost weight of false alarms relative to false alarms, which is set according to the loss ratio of false alarms to false alarms in historical data.
[0059] Minimize Cost(T) to calculate the outlier threshold T: T = arg min Cost(T) For a data sample x, if the anomaly score Score(x) > T, then the data sample x is anomaly; otherwise, the data sample x is normal.
[0060] In addition to the method for detecting abnormal data in power systems, a system for detecting abnormal data in power systems is also provided, including: The data preprocessing module acquires power system data and performs data normalization and feature filtering. The model training module trains the support vector machine model and the isolated forest model, and calculates the outlier scores for both. The weight allocation module assigns weights to the first and second abnormal scores and sums them to obtain the final abnormal score; the abnormal threshold setting module sets the abnormal threshold based on minimizing the cost function.
[0061] It should be noted that this invention creates a hybrid model for power anomaly data detection based on the One-Class SVM support vector machine model and the isolated forest model; it provides a power anomaly data detection process based on the hybrid model, which integrates the anomaly scores of different models, performs weighted fusion settings, and adds an anomaly threshold calculation method based on minimizing the cost function, thereby achieving more accurate anomaly judgment; it not only improves the accuracy of detection, but also enhances the robustness of the system, ensuring the stable operation of the power system; for the weighted fusion method of anomaly scores, the decision function of the One-Class SVM support vector machine model and the anomaly scores of the isolated forest model are weighted and calculated to generate the final anomaly score, thereby combining the advantages of the two models to make up for their disadvantages and obtain a more accurate scoring result.
[0062] The above embodiments are further elaborations and descriptions of the present invention to facilitate understanding, and are not intended to limit the present invention in any way. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for detecting abnormal data in a power system, characterized in that, include: Acquire power system data and perform data normalization and feature filtering; Train the support vector machine model and calculate its decision function value as the first anomaly score; Train the isolated forest model and calculate its outlier score as the second outlier score; The first and second outlier scores are weighted and summed to obtain the final outlier score. Anomaly thresholds are set based on minimizing the cost function, and data anomalies are determined based on the final anomaly score.
2. The method for detecting abnormal data in a power system according to claim 1, characterized in that, The data normalization and feature filtering process includes: The minimum-maximum normalization method is used to normalize the power system data, mapping the data to the interval [0, 1]. The normalized power system data is initially screened through data cleaning; then feature screening is performed using either a feature screening based on preset indicators or a data-driven feature screening.
3. A method for detecting abnormal data in a power system according to claim 1 or 2, characterized in that, The process of training the support vector machine model includes: Construct a hypersphere boundary to cover the normal data; The objective function is to minimize the weighted sum of slack variables and the sum of the hypersphere's volume; the constraint condition is that the distance of a data point beyond the hypersphere's boundary is less than or equal to the corresponding slack variable. The support vector machine model is then trained.
4. A method for detecting abnormal data in a power system according to claim 1 or 2, characterized in that, The process of training the isolated forest model includes: A forest is formed by recursively constructing multiple isolated trees by randomly selecting features and split points, and the path length of each tree is calculated. The path length is the sum of the number of edges traversed by sample x from the root node to the leaf node of the tree and the corrected path. The corrected path is the average path length of the binary tree constructed from samples that share the same leaf node as sample x.
5. The method for detecting abnormal data in a power system according to claim 4, characterized in that, The process of constructing an isolation tree includes: randomly selecting a feature from the dataset after feature filtering; Randomly select a split point between the minimum and maximum values of the dataset with selected features; divide the dataset into two subsets based on the selected features and the split point; For each subset, repeat the selection and segmentation steps for the splitting point until the segmentation stopping condition is met.
6. The method for detecting abnormal data in a power system according to claim 3, characterized in that, The calculation of its decision function value as the first anomaly score includes: for the data sample x to be judged, The first anomaly score is obtained by subtracting the threshold of the decision function from the weighted sum of the kernel function values between sample x and all support vectors. The weights of the kernel function values between sample x and the i-th support vector are the Lagrange multipliers of the i-th support vector.
7. The method for detecting abnormal data in a power system according to claim 4, characterized in that, The calculation of its anomaly score as the second anomaly score includes: for sample x out of a total of n data samples, The second outlier score is calculated by dividing the average path length of sample x in all isolated trees by the negative of the average path length of the binary tree constructed from n data samples, with 2 as the base.
8. A method for detecting abnormal data in a power system according to claim 1, 2, 5, 6, or 7, characterized in that, The weighting of the first abnormal score and the second abnormal score includes: the sum of the weights of the two abnormal scores is one; A preliminary optimal solution was obtained by traversing the ownership reorganization and merger evaluation in the validation set of power system data through grid search; The weights of the initial optimal solution are fine-tuned based on the sensitivity differences of different outlier data using the isolated forest model and the support vector machine model, to obtain the final weight allocation result.
9. A method for detecting abnormal data in a power system according to claim 1, 2, 5, 6, or 7, characterized in that, The cost function is the sum of the cost of false alarms and the cost of false alarms; The cost of a false alarm is the product of the number of false alarms below the anomaly threshold and the cost weight of a false alarm relative to a missed alarm. The cost of underreporting is 1 minus the difference between the cost weight of false positives and underreporting, multiplied by the number of underreporting cases below the anomaly threshold; The anomaly threshold calculated by minimizing the cost function is used as the final anomaly threshold for anomaly judgment.
10. A power system abnormal data detection system, applicable to the abnormal data detection method as described in any one of claims 1-9, characterized in that, include: The data preprocessing module acquires power system data and performs data normalization and feature filtering. The model training module trains the support vector machine model and the isolated forest model, and calculates the outlier scores for both. The weight allocation module assigns weights to the first and second outlier scores and then sums them to obtain the final outlier score. The anomaly threshold setting module sets the anomaly threshold based on minimizing the cost function.
Citation Information
Patent Citations
Power abnormal data detection method
CN119577649A
