Detection method for abnormity of special item to which power grid reserve item belongs

Abstract

The invention provides a method for detecting anomalies of a special item to which a power grid reserve item belongs, and the method comprises the steps: carrying out the semantic segmentation and clustering analysis of the item name, and recognizing an abnormal item which does not accord with the classification of the corresponding special item. The method comprises the following steps: S1, project data extraction: performing standardization processing on project name data; s2, performing semantic segmentation, and extracting key feature information in a project name; s3, performing clustering analysis, performing clustering analysis on the extracted semantic features based on an adaptive DBSCAN algorithm, and extracting abnormal items; s4, matching identification: comparing a clustering abnormal value result with a standard special field, and identifying an abnormal classification to which the clustering abnormal value result belongs; and S5, outputting an abnormal result, outputting an abnormal detection result and providing early warning information. According to the invention, through combination of semantic segmentation and adaptive clustering analysis, the abnormal items which are not matched with the corresponding special items can be effectively identified, the accuracy and efficiency of anomaly detection are remarkably improved, and the method is suitable for data anomaly detection in power grid project management.

Description

Technical Field

[0001] The present invention relates to a special anomaly detection technology for a power grid reserve project, and belongs to the technical field of data analysis. Background Art

[0002] With the increasing complexity and scale of power grid reserve project management, how to efficiently and accurately identify and monitor anomalies in projects has become a key issue in ensuring the quality of power grid project management. Power grid reserve projects involve multiple special categories, and project names often contain complex semantic information, which not only reflects the nature of the project, but also relates to the special field to which it belongs. Therefore, anomaly detection for the special projects to which power grid reserve projects belong is of great significance for improving the accuracy and early warning capabilities of power grid project management.

[0003] Current anomaly detection methods are mostly based on semantic analysis and data mining techniques, especially in the automatic identification and classification of project names. However, existing technologies rely on traditional clustering algorithms such as K-means and DBSCAN when dealing with complex and diverse power reserve project data. These algorithms can help identify abnormal project names to a certain extent, but their limitations are significant. K-means requires the number of clusters to be set in advance, which is impractical in power reserve projects because the number of categories of different projects is often dynamic and unpredictable. Although DBSCAN can process noisy data, its hyperparameter selection is too fixed and it is difficult to adapt to the diversity and complexity of power reserve projects. Fixed hyperparameter settings cause existing algorithms to easily generate too many false positives or false negatives when faced with irregular semantic data.

[0004] Traditional project anomaly detection usually relies on clustering algorithms with fixed hyperparameters, such as the patent "Project Anomaly Detection Method and Device" (CN116257810A); the patent "Digital Medical Service Analysis Method and Server Combining Big Data Smart Medical Care" (CN114203312A), which is based on the classification of the labeled project information and predicted project information of the project to be detected, and the number of category labels is fixed. Special anomaly detection for power grid reserve projects requires not only accurate identification of projects that do not match the category to which the project belongs, but also in-depth analysis of the characteristics of abnormal projects to ensure the effectiveness of power grid project management. However, most of the current clustering algorithms lack adaptive capabilities and cannot dynamically adjust hyperparameters according to data characteristics, resulting in unstable detection results in different data sets. This problem is particularly prominent in the complex semantic environment of power grid reserve project names. Summary of the invention

[0005] Technical issues: In the management of power grid reserve projects, current anomaly detection methods generally rely on fixed hyperparameters, which is particularly insufficient when faced with dynamically changing and highly complex project data. Specifically, traditional clustering algorithms such as K-Means often have difficulty adapting to the natural distribution of data when dealing with multidimensional, multi-category, and high-noise data sets, resulting in inaccurate anomaly identification. In addition, the static setting of hyperparameters cannot take into account the possible changes in the characteristics of the data set over different time periods, such as the evolution of project names, the redefinition of categories, or changes in the external environment, which makes these methods prone to high false alarm and missed alarm rates in practical applications. As a result, when identifying abnormal projects, the system not only faces the problem of insufficient classification accuracy, but also the risk of being unable to quickly respond to changes in new data features, which seriously affects the decision-making efficiency and security of project management. Therefore, it is urgent to solve the problem of adaptive adjustment and dynamic optimization of hyperparameters to improve the flexibility and accuracy of clustering algorithms and ensure that potential anomalies in power grid reserve projects can be effectively captured.

[0006] Technical solution: The present invention proposes a method for data cleaning and anomaly detection of power grid reserve project names. The method first extracts the project name and special fields from the power grid reserve project management system through the API interface, and stores them uniformly in the data framework. In the data cleaning stage, regular expressions are used to remove invalid information (such as spaces, punctuation, invalid numbers) in the project name, and missing fields are marked as NaN and classified as abnormal projects. Then, the integrity of the project name is verified through the Pandas library to ensure that it meets the predefined word count and character type requirements, and manual inspection is performed when necessary. Subsequently, a custom dictionary containing special names and unit names is constructed, and the project name is segmented using the Jieba word segmentation library. Feature vectors are generated through the bag of words model, and the feature representation is optimized in combination with the Word2Vec pre-trained model. Next, the information gain feature selection technology is used to screen features with high information gain, and the principal component analysis (PCA) is used to reduce the dimension of the feature vector to improve data processing efficiency. In the cluster analysis stage, the Z-score of the feature vector after dimensionality reduction is standardized, and the adaptive DBSCAN algorithm is used for cluster analysis to identify low-density areas and noise items, which are stored in the abnormal sample set. The clustering effect is then evaluated by the silhouette coefficient. In the matching and identification process, the cosine similarity is used to calculate the similarity between the abnormal sample and the standard field, and the items are matched and classified according to the set similarity threshold, and a detection report is generated. Finally, the system exports the detection report in XML format and stores it in the database, establishes an incremental backup mechanism to automatically save historical reports every day, and provides a visual interface to display the dynamic situation of the abnormal detection results, including the distribution of abnormal items, trend changes and historical data comparison. This technical solution ensures the accuracy and reliability of power grid reserve project data through systematic data cleaning and anomaly detection.

[0007] Beneficial effects: By adaptively adjusting the hyperparameters of the clustering algorithm, the present invention solves the problems of false positives and false negatives caused by fixed hyperparameters and abnormal project labels in traditional methods, and can significantly improve the accuracy of anomaly detection when processing complex data of power grid reserve projects. Combining semantic segmentation technology and adaptive DBSCAN algorithm, it can accurately identify abnormal items in the project and provide classification processing suggestions for abnormal situations of different degrees. The present invention not only supports automated data cleaning and anomaly detection report generation, but also dynamically displays the detection results through a visual interface, so that power grid managers can intuitively understand the distribution and trend changes of abnormal items. The incremental backup mechanism of historical data also improves data storage efficiency and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart for detecting special abnormalities of a power grid reserve project provided by an embodiment of the present invention.

[0009] Figure 2 It is a flow chart of a DBSCAN clustering algorithm with adaptive parameters provided by an embodiment of the present invention.

[0010] Figure 3 This is a comparison of the evaluation indicators of the OBLAOA-DBSCAN proposed in the present invention on the clustering test data set. DETAILED DESCRIPTION

[0011] The data cleaning module of the method of the present invention is intended to eliminate redundant and irregular information in the names of power grid reserve projects, ensuring that subsequent processing is based on clean and compliant data. First, the project name and its special fields are extracted from the power grid reserve project management system through the API interface and stored in the data framework for unified management. In order to clean up invalid information in the project name, such as extra spaces, punctuation marks, and invalid numbers, regular expressions are used for processing.

[0012] When dealing with missing fields, we directly mark the missing items as NaN and treat them as abnormal items and put them into the abnormal item set. This operation simplifies the processing and avoids potential misjudgments. Next, we use the Pandas library to verify the data to ensure that the project name meets the predefined word limit, character type requirements and other integrity constraints. During this process, manual inspection is also required to ensure that all cleaned data has good integrity to avoid non-compliant items affecting subsequent processing.

[0013] After the semantic segmentation module starts, a custom dictionary is built, including the project name and unit name. The Jieba word segmentation library is used to segment the project name, and the continuous text string is divided into independent words. In order to enhance the accuracy of word segmentation, the custom dictionary is combined for further processing. Then, the bag-of-words model is used to convert the project name into a vector form. The number of occurrences of each word constitutes the feature vector of the project name, which is expressed as:

[0014] v=[n 1 ,n 2 ,…,n k ],

[0015] Among them, n i Indicates the number of occurrences of the i-th word in the project name. Next, combined with the Word2Vec pre-training model, the word segmentation results are embedded into the vector space to further optimize the feature representation. The vector representation is:

[0016] v ′ =f Word2Vec (v),

[0017] After that, the information gain feature selection technique is used to screen the features. The information gain calculation formula is as follows:

[0018] IG(T,A)=H(T)-H(T|A),

[0019] Among them, IG(T,A) represents the information gain of attribute A to target T, H(T) is the entropy of target T, and H(T|A) represents the conditional entropy after given attribute A. Features with high information gain are retained for subsequent processing.

[0020] In order to improve the efficiency of data processing, principal component analysis (PCA) is applied to reduce the dimension of the feature vector. The core of PCA is to reduce the dimension by maximizing the data variance. Its transformation formula is:

[0021] Z=W·(xu),

[0022] Among them, W is the projection matrix of PCA, x is the original eigenvector, and u is the mean.

[0023] In the cluster analysis stage, the feature vector generated by the semantic segmentation module is first Z-score normalized to eliminate the dimensional differences between different features. The Z-score normalization formula is:

[0024]

[0025] Among them, Z is the dimension reduction feature value, u is the mean of the feature, and ε is the standard deviation. This standardization step adjusts all features to a range of mean 0 and standard deviation 1 for unified processing.

[0026] Next, perform DBSCAN cluster analysis. DBSCAN is a density-based clustering algorithm suitable for noisy data. Its core parameters are ε (neighborhood radius) and MinPts (minimum number of neighbors). For each point, DBSCAN determines whether it is a core point based on the number of points in its neighborhood. If the number is greater than or equal to MinPts, and the neighborhood radius is less than ε, then the point belongs to a cluster. The formula is:

[0027] N(p)={qED|dis(p,q)≤θ}

[0028] Among them, N(p) represents the neighborhood of point p. After clustering the standardized feature matrix, the silhouette coefficient is used to evaluate the clustering effect: Among them, a is the average distance from the data point to its cluster, and b is the average distance from the data point to the nearest other cluster. The closer the silhouette coefficient is to 1, the better the clustering effect.

[0029] The specific process of DBSCAN is as follows:

[0030] (1) Neighborhood search: For each data point, search its neighborhood point set based on a given ε value (neighborhood radius).

[0031] (2) Identification of core points: If the neighborhood of a data point contains no less than MinPts points, the point is marked as a core point.

[0032] (3) Cluster expansion: For each core point, the size of the cluster is expanded by adding all the points in its neighborhood to the same cluster. This process is performed recursively until all reachable core points and their neighborhood points are included in the cluster.

[0033] (4) Identification of noise points: If a data point is neither a core point nor belongs to any cluster, it is considered a noise point.

[0034] When constructing an adaptive DBSCAN clustering method, the present invention proposes an improved DBSCAN to address the shortcomings of traditional DBSCAN in parameter adaptation, named DBSCAN optimized by the arithmetic optimization algorithm based on reverse learning (OBLAOA) (OBLAOA-DBSCAN). AOA is an optimization algorithm based on natural arithmetic processes, which has good global search capabilities, but may have local optimal problems in certain situations. The reverse learning technology further improves the algorithm's exploration ability by introducing a reverse search mechanism in the solution space.

[0035] In the specific implementation process, the introduction of OBL enables AOA to effectively explore potential excellent solutions and achieve better convergence in parameter search. We designed the workflow of OBLAOA, which includes the following steps:

[0036] (1) Initialization parameters: First, set the initial value range of ε and MinPts.

[0037] (2) Parameter search using AOA: In each iteration, AOA is used to perform a global search and the EPS and MinPts parameters are continuously adjusted according to the current clustering effect.

[0038] (3) Introducing the OBL mechanism: In each round of iteration, OBL generates candidate parameters for reverse search and further guides the optimization direction of the parameters by evaluating their performance in DBSCAN clustering.

[0039] (4) Clustering execution and performance evaluation: Through continuous iteration, the best performing parameter combination is selected, DBSCAN clustering is performed, and its effect is evaluated based on the silhouette coefficient.

[0040] In OBL, assuming that the current parameter solution set is x, and its value range in the solution space is [lower, upper], then the corresponding reverse solution x′ is

[0041] x ′ =lower+upper-x

[0042] The specific process of the AOA algorithm is as follows:

[0043] (1) Population initialization: A set of solutions is randomly generated as the initial population. Each solution corresponds to two parameters that need to be set for the DBSCAN algorithm.

[0044] (2) Fitness calculation: Perform DBSCAN clustering and calculate the fitness value of each solution based on the silhouette coefficient or other clustering effect evaluation indicators. The higher the fitness of the solution, the better the clustering effect of its corresponding parameter combination.

[0045] (3) Addition operation: Add the difference between the solution and other randomly selected solutions to the current solution, so that the solution can jump out of the current local area and move towards the possible global optimal direction.

[0046] (4) Subtraction operation: Subtract the difference between the solution and the solution with better performance, narrow the search range, and thus explore the possible optimal solution more carefully.

[0047] (5) Multiplication operation: By multiplying the solution with a random factor, the solution can be searched for new solutions in a larger range.

[0048] (6) Division operation: By dividing the current solution by a certain factor, the variation of the solution can be reduced and the optimal solution can be gradually approached.

[0049] (7) Introduction of OBL: After each iteration, the OBL mechanism generates the corresponding reverse solution. The goal of OBL is to form a mirror solution in the solution space so that synchronous searches can be performed in different regions of the solution space to avoid falling into local optimality.

[0050] (8) Fitness evaluation and selection: Based on the fitness value of each solution (including the reverse solution), the solution with the best performance is selected and retained for the next round of iteration.

[0051] (9) Convergence judgment and termination: When the change in the solution is lower than the set convergence standard, the algorithm ends and outputs the current optimal solution.

[0052] Through cluster analysis, the system can identify low-density areas in each category and treat these low-density items as abnormal items. At the same time, items marked as noise during the clustering process will also be extracted and stored in the abnormal sample set.

[0053] The matching recognition module mainly calculates the similarity between abnormal samples and standard fields through cosine similarity to evaluate the matching degree of abnormal items. The cosine similarity formula is:

[0054]

[0055] Among them, A and B are two feature vectors, and the dot product measures their directional similarity. In the specific implementation, the system can adjust the matching criteria by setting different thresholds, for example: similarity higher than 0.9: complete match, similarity between 0.7 and 0.9: partial match, similarity lower than 0.7: complete mismatch. All identified abnormal project names and their matching status are finally summarized into a detection report.

[0056] In the outlier result output stage, the system first exports the generated anomaly detection report in XML format and stores it in the specified database. The structured format of XML makes it easy to integrate the report with other systems. Next, the system establishes a data backup mechanism and uses incremental backup technology to automatically save historical reports every day to improve storage efficiency and avoid data redundancy. Incremental backup only saves data that has changed since the last backup, thereby reducing backup time and storage space usage. Finally, the system provides a visual interface that displays the dynamic situation of anomaly detection results through dashboards and charts, such as the distribution of abnormal items, trend changes, and historical data comparisons.

Claims

1. A method for detecting abnormalities of a special project of a power grid reserve project, characterized in that: The following steps are involved: Step 1: Perform data cleaning, eliminate redundant symbols and invalid information in project names, and correct irregular or missing data; Step 2: Identify the main words and additional information in the project name through semantic segmentation and extract key features; Step 3: Perform cluster analysis on the features of the project names and classify projects with similar contents; Step 4: Compare the clustering results with the predefined special categories to identify items that do not meet the requirements; Step 5: Output anomaly detection results, list all items identified as anomalies, analyze the causes and classifications of the anomalies, and provide corresponding warning information.

2. A method for detecting abnormalities of a special project of a power grid reserve project according to claim 1, characterized in that: The specific steps of data cleaning are as follows: Step 1: Use the API interface to extract the name and special fields of the power grid reserve project from the power grid reserve project management system, and store the extracted data in the data framework; Step 2: Use regular expressions to eliminate extra spaces before, after, and in the middle of the project name, match special symbols (punctuation marks, invalid numbers), and retain only meaningful text information; Step 3: Use data validation tools (Pandas library) to check whether each project name meets the predefined word limit and character type requirements, perform integrity constraint checks, and confirm the integrity of the cleaned data set.

3. A method for detecting abnormalities of a special project of a power grid reserve project according to claim 1, characterized in that: The specific steps of extracting key features by semantic cutting are as follows: Step 1: Build a custom dictionary, including project names and unit names. Step 2: Use the Jieba word segmentation library to segment the project name, split the continuous text string into independent words, and use a custom dictionary to enhance the accuracy of word segmentation; Step 3: Use the bag-of-words model to represent each project name as a vector, and combine it with the pre-trained Word2Vec word embedding model to map the word segmentation results into the vector space; Step 4: Use information gain feature selection technology to evaluate the importance of each feature vector, and apply principal component analysis to reduce the dimension of the features to improve the efficiency of data processing.

4. A method for detecting abnormalities of a special project of a power grid reserve project according to claim 1, characterized in that: The specific steps of cluster analysis feature are as follows: Step 1: Arrange the feature vectors processed by the semantic segmentation module into a feature matrix, standardize the feature matrix, and use the Z-score standardization method to adjust the mean of each feature to 0 and the standard deviation to 1 to eliminate the dimensional differences between different features; Step 2: Use the adaptive DBSCAN algorithm to perform cluster analysis on the standardized feature vectors using adaptive parameters, and use the silhouette coefficient to evaluate the clustering effect to verify the rationality and accuracy of the clustering results. Step 3: Based on the cluster analysis results, identify low-density areas in each category. The project names in these low-density areas are regarded as abnormal projects, and the project names marked as noise in the clustering process are extracted to form an abnormal sample set.

5. A method for detecting abnormalities of a special project of a power grid reserve project according to claim 1, characterized in that: The specific steps of matching and identifying abnormal sample sets are as follows: Step 1: Extract feature information from each item name in the abnormal sample set. And align with the feature information in the data frame, and use cosine similarity to calculate the similarity between the abnormal sample and the standard field to determine the degree of matching; Step 2: Set the similarity threshold to determine the matching status of the project name. Classify and mark abnormal samples based on the calculation results, specifically into "completely unmatched", "partially matched" or "meets the requirements", and take corresponding measures for different degrees of abnormal situations; Step 3: Summarize the matching and identification results, list all the project names and matching status identified as abnormal, and generate a detailed abnormal detection report. The report should include the project name, matching status, similarity value and handling suggestions.

6. A method for detecting abnormalities of a special project of a power grid reserve project according to claim 1, characterized in that: The specific steps of outputting the abnormal value result are as follows: Step 1: Use XML structured format to export the generated anomaly detection report and store it in the specified database; Step 2: Establish a data backup mechanism, automatically save historical reports every day, and use incremental backup technology to improve storage efficiency; Step 3: Establish a visualization interface, regularly update the dynamic display of anomaly detection results, and use dashboards and charts to display the distribution, change trends and historical data comparison of abnormal items.

7. An adaptive DBSCAN algorithm according to claim 4, characterized in that: The specific steps of parameter adaptation are as follows: Step 1: Determine the initial parameter range and set the reasonable initial value range of ε (neighborhood radius) and MinPts (minimum number of neighbors); Step 2: Update the parameters using the Backward Learning Based Arithmetic Optimization Algorithm (OBLAOA) to ensure that these parameters can converge quickly; Step 3: Use the generated initial parameters to perform DBSCAN clustering and calculate the silhouette coefficient of the clustering to quantify the clustering effect; Step 4: If the clustering effect is no longer significantly improved after multiple rounds of iterations, stop the optimization.

8. The arithmetic optimization algorithm based on reverse learning according to claim 7, characterized in that: The specific steps of parameter optimization are as follows: Step 1: Initialize hyperparameters and candidate solution sets; Step 2: Update the MOP and MOA values, generate random numbers, and decide whether to enter the exploration phase or the development phase based on the values ​​of these random numbers; Step 3: Select parameter update operators according to MOA value and MOP value and update candidate solution set; Step 4: After each iteration, calculate the fitness function values ​​of all candidate solutions, and use the opposite solution generation strategy (OBL) to find the opposite solution of the optimal solution based on the current optimal solution; Step 5: Calculate the fitness values ​​of the current solution and the opposing solution, and select the solution with better fitness as the optimal solution for the next round; Step 6: Repeat the above steps. When the improvement of the fitness value is lower than the set threshold, stop the optimization process and output the optimal parameters after final optimization.

Images