Method and system for screening redundant data of power system

Abstract

The invention discloses a method and a system for screening redundant data of a power system, and belongs to the technical field of data detection. In order to solve the problems of low data processing efficiency and poor nonlinear characteristic capture, each candidate item set is accurately counted, a frequent item set is screened out according to a preset support degree threshold value, the representativeness of the frequent item set is improved, the frequent item set in a fusion data set is mined, and the non-linear characteristic capture efficiency of the fusion data set is improved. The subsequent data processing steps are more accurate and efficient, the discretized data can directly reflect the contribution degree difference of different intervals to a model or analysis, the data are discretized through clustering, the space required for storing continuous numerical values is reduced, all possible item set pairs are generated for item sets in each cluster, and the data processing efficiency is improved. According to the method, the redundant data in the power system can be more accurately matched and identified as the strong association rule which serves as the potential association rule and is screened out, so that the screening accuracy of the redundant data is improved, and the risk of misjudgment and missed judgment is also reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data detection, and in particular to a method and system for screening redundant data in an electric power system. Background Art

[0002] With the development of science and technology, the power supply structure of the power system has undergone major changes. The power system will also carry out grid planning and energy storage investment to alleviate the peak-shaving pressure of the power system and ensure the absorption of new energy. Identifying and screening out redundant data is an important part of the massive data integration that supports the power system.

[0003] However, there are still the following problems in actual operation:

[0004] When processing large-scale power system data, traditional data mining methods often find it difficult to find a balance between data processing efficiency and accuracy. Frequent item sets that are too trivial will increase the computational complexity, while item sets that are too large may contain a large amount of non-critical information, affecting the accuracy of the analysis results. At the same time, power system data is often continuous and complex, and traditional linear analysis methods find it difficult to effectively capture its nonlinear characteristics. Summary of the Invention

[0005] The object of the present invention is to provide a method and system for screening redundant data in an electric power system to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for screening redundant data in a power system, comprising the following steps:

[0007] Data acquisition and preprocessing: regularly obtain raw data from power system data sources, store the raw data in a local database, and perform preprocessing operations;

[0008] Data fusion processing: fusing pre-processed multi-source heterogeneous energy data to generate a unified fusion data set;

[0009] Frequent item set mining: set item set rules, scan the fused data set layer by layer, calculate the item set support, prune the item set according to the support threshold, and iteratively generate frequent item sets of the flow data;

[0010] Data discretization processing: cluster analysis is performed on frequent item sets, data points are assigned to the nearest centroid, and the centroid position is updated repeatedly until the centroid is stable to form a discrete result;

[0011] Association rule construction: generating potential association rules from discrete results, calculating their support and confidence, and screening rules whose support and confidence exceed the preset thresholds as strong association rules;

[0012] Redundant data screening, applying strong association rules to the original or preprocessed data set, matching and marking redundant data, removing marked redundant data, generating and storing clean data sets, verifying the clean data sets to confirm that no redundant data is missing, and outputting a verification report.

[0013] Furthermore, a system for screening redundant data of a power system is applied to the above-mentioned method for screening redundant data of a power system, comprising:

[0014] Data acquisition module, used to:

[0015] Regularly obtain raw data from the power system data source and store it in a local database, the raw data including the power flow data set and auxiliary data for selecting the threshold value of the data item set support, and perform preprocessing operations on the raw data;

[0016] Data fusion processing module, used for:

[0017] The pre-processed multi-source heterogeneous energy data from different sources are fused to generate a fused data set;

[0018] Frequent itemset mining module, used for:

[0019] Set the item set rules, scan the fused data set layer by layer, calculate the support count of each item set, prune the item set according to the support count threshold, and repeat the iterative process to obtain the frequent item sets of the trend data;

[0020] Discrete processing modules for:

[0021] Discrete the frequent item sets, perform cluster analysis on the frequent item sets, assign the data points to the nearest centroid, form multiple discrete results, update the centroid position until the centroid no longer changes, and complete the discrete processing;

[0022] Association rule building blocks for:

[0023] Perform association operations on the frequent item sets in the discrete results respectively, calculate the support and confidence of the association rules, and when the support and confidence of the association rules reach the preset threshold, they are regarded as strong association rules;

[0024] Redundant data filtering module, used for:

[0025] Based on the constructed strong association rules, redundant data in the power system are matched and filtered out. The strong association rules are applied to the original data set or the preprocessed data set to identify and mark the data items that meet the redundant characteristics. The redundant data are removed from the data set to generate a clean data set.

[0026] Furthermore, the frequent itemset mining module includes:

[0027] Itemset generation unit, used for:

[0028] Receive the fused data set from the data fusion processing module, set the minimum item set length and the maximum item set length, and select all unique single items from the fused data set as the initial candidate item set Q1;

[0029] For each Q k , by combining the existing Q k-1 Generate new Q k Candidate item set, where k represents the candidate item set number, k>1, all newly generated Q k The candidate itemset is output as a candidate itemset list;

[0030] Support calculation unit, used for:

[0031] Receive the candidate item set list from the item set generation unit, traverse the fused data set, count each candidate item set, determine the number of times each candidate item set appears in the fused data set, and obtain the support;

[0032] Compare the support of each candidate item set with the preset support threshold. When the support of a candidate item set is less than the support threshold, the candidate item set is removed from the candidate item set list. After all candidate item sets are compared, a frequent item set list is obtained.

[0033] Connection collection building blocks for:

[0034] Receive a list of frequent itemsets, and for each Q in the frequent itemset list k Candidate item set, find and combine all possible Q k-1 Subset pairs, generate new Q k+1 candidate item set;

[0035] The newly generated Q k+1 The candidate item set is used as input and returned to the item set generation unit for iterative processing until no new frequent item sets are generated, and the frequent item sets of the power flow data are obtained.

[0036] Furthermore, the discrete processing module includes:

[0037] Cluster initialization unit, used to:

[0038] Receive the frequent itemsets of the power flow data as input from the frequent itemset mining module, select randomly selected data points as initial centroids, and output the selected initial centroids to the clustering iteration unit;

[0039] Clustering iteration unit, used to:

[0040] Receive the initial centroid position from the cluster initialization unit, traverse all data points in the frequent itemset of the power flow data, and assign the data points to the nearest centroid according to the Euclidean distance;

[0041] For each cluster, recalculate and update the centroid position of the cluster, and repeat the allocation and updating process until a preset number of iterations is reached;

[0042] The final clustering results are output to the association rule construction module.

[0043] Furthermore, the association rule construction module includes:

[0044] A rule generation unit, used to:

[0045] Receive the final clustering results from the discrete processing module, generate all possible item set pairs for the item sets in each cluster, and use the generated item set pairs as potential association rules;

[0046] Support confidence calculation unit, used for:

[0047] receiving potential association rules from a rule generation unit;

[0048] Traverse the preprocessed original data set and record the number of times each potential association rule appears in the data set. The support is calculated by dividing the number of times each potential association rule appears in the data set by the size of the data set.

[0049] For each potential association rule item set pair, the two item sets are regarded as rule A and rule B, and the conditional probability that rule B also occurs when rule A occurs is calculated to obtain the confidence level.

[0050] Output the support and confidence of each potential association rule.

[0051] Furthermore, the association rule building module further includes:

[0052] Rule filtering unit, used to:

[0053] Calculate the support and confidence of each potential association rule, compare the support and confidence with the preset threshold, and only when the support and confidence of a potential association rule exceed the preset threshold, it is considered as a strong association rule and the selected strong association rule is output;

[0054] Rule output unit, used for:

[0055] The strong association rules are formatted and output to the redundant data screening module.

[0056] Furthermore, the redundant data screening module includes:

[0057] Rule application unit, used to:

[0058] Receive the formatted strong association rules, load the preprocessed original data set from the local database, traverse each data item in the data set, match it with the strong association rules, and mark the data items that match the strong association rules as redundant;

[0059] Data filtering unit, used to:

[0060] Receive data items marked as redundant, traverse the dataset, remove all data items marked as redundant, generate a clean dataset without redundant data, and store it back to the local database.

[0061] Furthermore, the redundant data screening module further includes:

[0062] Screening verification unit for:

[0063] The clean dataset after screening is loaded, and the clean dataset is matched again using the same strong association rule to check whether there are any missing redundant data items, and a verification report is output. The verification report includes the amount of redundant data screened out and statistical information of the remaining dataset.

[0064] Furthermore, the redundant data screening module further includes:

[0065] A reference value setting module, configured to set a first redundant data amount reference value and a second redundant data amount reference value; wherein the first redundant data amount reference value is lower than the second redundant data amount reference value;

[0066] The first moment acquisition module is used to extract the moment of obtaining a clean data set after removing redundant data;

[0067] Data volume monitoring module, used to monitor the volume of redundant data in real time;

[0068] A second moment acquisition module is configured to acquire a moment corresponding to when the amount of redundant data reaches a preset first redundant data amount reference value;

[0069] A first redundant data generation time period acquisition module is configured to acquire a first redundant data generation time period when the moment when the clean data set is obtained by using the redundant data to be eliminated corresponds to the moment when the amount of redundant data reaches a preset redundant data amount reference value;

[0070] A third moment acquisition module is configured to acquire a moment corresponding to when the amount of redundant data reaches a preset second redundant data amount reference value;

[0071] A second redundant data generation time period acquisition module is configured to acquire a second redundant data generation time period when the moment when the clean data set is obtained by using the redundant data removed corresponds to the moment when the amount of redundant data reaches a preset redundant data amount reference value;

[0072] A redundant data generation index parameter acquisition module, configured to acquire redundant data generation index parameters using the first redundant data generation time period and the second redundant data generation time period;

[0073] The redundant data generation index parameter is obtained by the following formula:

[0074]

[0075] Wherein, Y represents a redundant data generation indicator parameter; R1 and R2 represent a first redundant data amount reference value and a second redundant data amount reference value, respectively; T1 and T2 represent a first redundant data generation time period and a second redundant data generation time period, respectively;

[0076] The redundant data generation indicator parameter is compared with a preset parameter threshold value. When the redundant data generation indicator parameter exceeds the preset parameter threshold value, the redundant data generation indicator parameter is judged to be abnormal and an abnormal alarm is issued.

[0077] Furthermore, the reference value setting module includes:

[0078] Redundant data information extraction module, used to extract the redundant data amount corresponding to each redundant data elimination in the historical operation records;

[0079] The redundant data amount extraction module is used to extract the redundant data amount corresponding to the current redundant data elimination;

[0080] A first redundant data amount reference value setting module is configured to obtain a first redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination;

[0081] The first redundant data amount reference value is obtained by the following formula:

[0082]

[0083] Wherein, R1 represents the first redundant data amount reference value; n represents the number of times redundant data is eliminated in the historical operation record; R i Indicates the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R minIndicates the minimum amount of redundant data corresponding to n redundant data eliminations in the historical operation records;

[0084] A second redundant data amount reference value setting module is configured to obtain a second redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination;

[0085] The second redundant data amount reference value is obtained by the following formula:

[0086]

[0087] Wherein, R2 represents the second redundant data amount reference value; n represents the number of times redundant data is eliminated in the historical operation record; R i Indicates the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R max Indicates the maximum amount of redundant data corresponding to the elimination of n redundant data in the historical operation record; R z Indicates the median value of the redundant data volume corresponding to n redundant data eliminations in the historical operation records.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] 1. By setting the minimum and maximum item set lengths, the present invention can accurately control the size range of frequent itemsets, ensuring that the mined frequent itemsets are neither too trivial nor too large, thereby improving the efficiency and accuracy of data processing. By accurately counting each candidate item set and screening out frequent itemsets based on a preset support threshold, it is ensured that only itemsets that frequently appear in the actual data set are retained, thereby improving the representativeness of frequent itemsets. By mining frequent itemsets in the fused data set, subsequent data processing steps are made more accurate and efficient, thereby improving the performance and effectiveness of the entire redundant data screening system.

[0090] 2. The present invention discretizes continuous and complex power system data into several representative clusters through cluster analysis, so that subsequent analysis can more easily capture the nonlinear characteristics of the data. The discretized data can directly reflect the difference in contribution of different intervals to the model or analysis. By discretizing the data through clustering, the space required to store continuous values ​​is reduced. The iteration and centroid update in the clustering process help to eliminate outliers or noise data, thereby reducing the interference of these noises on subsequent association rule mining. The discrete data set based on the clustering results can better support the mining of association rules.

[0091] 3. The present invention generates all possible item set pairs for the item sets in each cluster from the clustering results received by the discrete processing module as potential association rules, thereby ensuring the comprehensiveness and accuracy of the potential association rules. By traversing the preprocessed original data set, the support and confidence of each potential association rule are accurately calculated, thereby ensuring the reliability of the association rule measurement indicators, providing a solid data foundation for the subsequent screening of strong association rules. According to the preset support and confidence thresholds, the potential association rules are strictly screened, ensuring that only rules that simultaneously meet high support and high confidence can be regarded as strong association rules, thereby improving the practicality and effectiveness of the association rules. Based on the screened strong association rules, the redundant data screening module can more accurately match and identify redundant data in the power system, which not only improves the accuracy of redundant data screening, but also reduces the risk of misjudgment and missed judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of the screening method of the present invention;

[0093] Figure 2 Schematic diagram of the screening system module of the present invention. DETAILED DESCRIPTION

[0094] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0095] In order to solve the technical problems that traditional data mining methods often find it difficult to find a balance between data processing efficiency and accuracy when processing large-scale power system data, and that overly trivial frequent item sets increase computational complexity, while overly large item sets may contain a large amount of non-critical information, affecting the accuracy of analysis results, and that power system data is often continuous and complex, and traditional linear analysis methods have difficulty in effectively capturing its nonlinear characteristics, the present invention provides the following technical solutions:

[0096] See also Figure 1 A method for screening redundant data in a power system comprises the following steps:

[0097] Data acquisition and preprocessing: regularly obtain raw data from power system data sources, store the raw data in a local database, and perform preprocessing operations;

[0098] Data fusion processing: fusing pre-processed multi-source heterogeneous energy data to generate a unified fusion data set;

[0099] Frequent item set mining: set item set rules, scan the fused data set layer by layer, calculate the item set support, prune the item set according to the support threshold, and iteratively generate frequent item sets of the flow data;

[0100] Data discretization processing: cluster analysis is performed on frequent item sets, data points are assigned to the nearest centroid, and the centroid position is updated repeatedly until the centroid is stable to form a discrete result;

[0101] Association rule construction: generating potential association rules from discrete results, calculating their support and confidence, and screening rules whose support and confidence exceed the preset thresholds as strong association rules;

[0102] Redundant data screening, applying strong association rules to the original or preprocessed data set, matching and marking redundant data, removing marked redundant data, generating and storing clean data sets, verifying the clean data sets to confirm that no redundant data is missing, and outputting a verification report.

[0103] In the above-described embodiment, multi-source heterogeneous data is integrated through data fusion processing, eliminating redundancy and inconsistency between data and improving overall data quality. The frequent item set mining and association rule construction steps can automatically discover frequent patterns and strong association rules in the data, accurately identify redundant parts in the data, and automatically match and remove redundant data based on these rules, significantly improving the efficiency and accuracy of data processing. By screening out redundant data, the size of the data set is reduced, thereby reducing the cost of data storage and processing. At the same time, the clean data set also makes it easier to conduct subsequent data analysis and mining, improving overall data processing performance.

[0104] See also Figure 2 A system for screening redundant data of a power system, applied in the above-mentioned method for screening redundant data of a power system, comprises:

[0105] Data acquisition module, used to:

[0106] Regularly obtain raw data from the power system data source and store it in a local database, the raw data including the power flow data set and auxiliary data for selecting the threshold value of the data item set support, and perform preprocessing operations on the raw data;

[0107] Data fusion processing module, used for:

[0108] The pre-processed multi-source heterogeneous energy data from different sources are fused to generate a fused data set;

[0109] Frequent itemset mining module, used for:

[0110] Set the item set rules, scan the fused data set layer by layer, calculate the support count of each item set, prune the item set according to the support count threshold, and repeat the iterative process to obtain the frequent item sets of the flow data;

[0111] Discrete processing modules for:

[0112] Discrete the frequent item sets, perform cluster analysis on the frequent item sets, assign the data points to the nearest centroid, form multiple discrete results, update the centroid position until the centroid no longer changes, and complete the discrete processing;

[0113] Association rule building blocks for:

[0114] Perform association operations on the frequent item sets in the discrete results respectively, calculate the support and confidence of the association rules, and when the support and confidence of the association rules reach the preset threshold, they are regarded as strong association rules;

[0115] Redundant data filtering module, used for:

[0116] Based on the constructed strong association rules, redundant data in the power system are matched and filtered out. The strong association rules are applied to the original data set or the preprocessed data set to identify and mark the data items that meet the redundant characteristics. The redundant data are removed from the data set to generate a clean data set.

[0117] In the above embodiment, an automated screening process for redundant data in the power system is implemented. From data acquisition, preprocessing, fusion, frequent item set mining, discrete processing to association rule construction and redundant data screening, the entire process is highly automated, reducing manual intervention and improving processing efficiency and accuracy. At the same time, the screening of redundant data significantly reduces the size of the data set, thereby alleviating the burden of data storage and processing, saving storage resources, improving data processing speed, and reducing operating costs.

[0118] Frequent itemset mining module, including:

[0119] Itemset generation unit, used for:

[0120] Receive the fused data set from the data fusion processing module, set the minimum item set length and the maximum item set length, and select all unique single items from the fused data set as the initial candidate item set Q1;

[0121] For each Q k , by combining the existing Q k-1 Generate new Q k Candidate item set, where k represents the candidate item set number, k>1, all newly generated Q k The candidate itemset is output as a candidate itemset list;

[0122] Support calculation unit, used for:

[0123] Receive the candidate item set list from the item set generation unit, traverse the fused data set, count each candidate item set, determine the number of times each candidate item set appears in the fused data set, and obtain the support;

[0124] Compare the support of each candidate item set with the preset support threshold. When the support of a candidate item set is less than the support threshold, the candidate item set is removed from the candidate item set list. After all candidate item sets are compared, a frequent item set list is obtained.

[0125] Connection collection building blocks for:

[0126] Receive a list of frequent itemsets, and for each Q in the frequent itemset list k Candidate item set, find and combine all possible Q k-1 Subset pairs, generate new Q k+1 candidate item set;

[0127] The newly generated Q k+1 The candidate item set is used as input and returned to the item set generation unit for iterative processing until no new frequent item sets are generated, and the frequent item sets of the power flow data are obtained.

[0128] In the above embodiment, by setting the minimum item set length and the maximum item set length, the size range of the frequent itemsets can be accurately controlled to ensure that the mined frequent itemsets are neither too trivial nor too large, thereby improving the efficiency and accuracy of data processing. By accurately counting each candidate item set and filtering out frequent itemsets according to a preset support threshold, it is ensured that only itemsets that frequently appear in the actual data set are retained, thereby improving the representativeness of frequent itemsets.

[0129] In the above embodiment, by mining the frequent item sets in the fused data set, it can provide strong support for subsequent discrete processing, association rule construction and redundant data screening, making the subsequent data processing steps more accurate and efficient, thereby improving the performance and effect of the entire redundant data screening system, and simplifying the originally complex data processing process into clear steps and units, making the entire process easier to understand and implement.

[0130] Discrete processing modules, including:

[0131] Cluster initialization unit, used to:

[0132] Receive the frequent itemsets of the power flow data as input from the frequent itemset mining module, select randomly selected data points as initial centroids, and output the selected initial centroids to the clustering iteration unit;

[0133] Clustering iteration unit, used to:

[0134] Receive the initial centroid position from the cluster initialization unit, traverse all data points in the frequent itemset of the power flow data, and assign the data points to the nearest centroid according to the Euclidean distance;

[0135] For each cluster, recalculate and update the centroid position of the cluster, and repeat the allocation and updating process until a preset number of iterations is reached;

[0136] The final clustering results are output to the association rule construction module.

[0137] In the above embodiment, through cluster analysis, continuous and complex power system data can be discretized into several representative clusters, so that subsequent analysis can more easily capture the nonlinear characteristics of the data. The discretized data can directly reflect the differences in the contribution of different intervals to the model or analysis, making the data results easier to understand and explain. By discretizing the data through clustering, the space required to store continuous values ​​is reduced, especially in a big data environment such as the power system, which can significantly reduce storage costs. The iteration and centroid update in the clustering process help to eliminate outliers or noise data, because these data are difficult to be classified into clear clusters during the clustering process, thereby reducing the interference of these noises on subsequent association rule mining. The discrete data set based on the clustering results can better support the mining of association rules.

[0138] Association rule building blocks, including:

[0139] A rule generation unit, used to:

[0140] Receive the final clustering results from the discrete processing module, generate all possible item set pairs for the item sets in each cluster, and use the generated item set pairs as potential association rules;

[0141] Support confidence calculation unit, used for:

[0142] receiving potential association rules from a rule generation unit;

[0143] Traverse the preprocessed original data set and record the number of times each potential association rule appears in the data set. The support is calculated by dividing the number of times each potential association rule appears in the data set by the size of the data set.

[0144] For each potential association rule item set pair, the two item sets are regarded as rule A and rule B, and the conditional probability that rule B also occurs when rule A occurs is calculated to obtain the confidence level.

[0145] Output the support and confidence of each potential association rule;

[0146] Rule filtering unit, used to:

[0147] Calculate the support and confidence of each potential association rule, compare the support and confidence with the preset threshold, and only when the support and confidence of a potential association rule exceed the preset threshold, it is considered as a strong association rule and the selected strong association rule is output;

[0148] Rule output unit, used for:

[0149] The strong association rules are formatted and output to the redundant data screening module.

[0150] In the above embodiment, all possible item set pairs are generated for the item sets in each cluster from the clustering results received from the discrete processing module as potential association rules, thereby ensuring the comprehensiveness and accuracy of the potential association rules. By traversing the preprocessed original data set, the support and confidence of each potential association rule are accurately calculated, thereby ensuring the reliability of the association rule measurement indicators, providing a solid data foundation for the subsequent screening of strong association rules. According to the preset support and confidence thresholds, the potential association rules are strictly screened, ensuring that only rules that meet both high support and high confidence can be regarded as strong association rules, thereby improving the practicality and effectiveness of the association rules.

[0151] In the above embodiment, based on the screened strong association rules, the redundant data screening module can more accurately match and identify redundant data in the power system, which not only improves the accuracy of redundant data screening, but also reduces the risk of misjudgment and missed judgment. By removing redundant data, the data set of the entire power system is effectively purified and compressed, saving storage space and processing time, improving data quality and availability, and providing more accurate and valuable information for subsequent data analysis and decision support.

[0152] Redundant data screening module, including:

[0153] Rule application unit, used to:

[0154] Receive the formatted strong association rules, load the preprocessed original data set from the local database, traverse each data item in the data set, match it with the strong association rules, and mark the data items that match the strong association rules as redundant;

[0155] Data filtering unit, used to:

[0156] Receive data items marked as redundant, traverse the data set, remove all data items marked as redundant, generate a clean data set without redundant data, and store it back to the local database;

[0157] Screening verification unit for:

[0158] The clean dataset after screening is loaded, and the clean dataset is matched again using the same strong association rule to check whether there are any missing redundant data items, and a verification report is output. The verification report includes the amount of redundant data screened out and statistical information of the remaining dataset.

[0159] In the above embodiment, by accurately matching strong association rules, redundant items in the data set can be identified and marked, and these redundant items can be removed to generate a cleaner and more accurate data set, which significantly improves the quality of the data and provides a more reliable basis for subsequent data analysis and decision support. By screening out redundant data, data noise can be effectively reduced, the signal-to-noise ratio of the data can be improved, and the data analysis results can be closer to the actual situation. Compared with the original data set, the clean data set has a smaller data volume and a simpler structure. Therefore, it will be more efficient in the subsequent data processing process, which not only reduces computing time and resource consumption, but also improves the overall response speed of the system.

[0160] Specifically, the redundant data screening module further includes:

[0161] A reference value setting module, configured to set a first redundant data amount reference value and a second redundant data amount reference value; wherein the first redundant data amount reference value is lower than the second redundant data amount reference value;

[0162] The first moment acquisition module is used to extract the moment of obtaining a clean data set after removing redundant data;

[0163] Data volume monitoring module, used to monitor the volume of redundant data in real time;

[0164] A second moment acquisition module is configured to acquire a moment corresponding to when the amount of redundant data reaches a preset first redundant data amount reference value;

[0165] A first redundant data generation time period acquisition module is configured to acquire a first redundant data generation time period when the moment when the clean data set is obtained by using the redundant data to be eliminated corresponds to the moment when the amount of redundant data reaches a preset redundant data amount reference value;

[0166] A third moment acquisition module is configured to acquire a moment corresponding to when the amount of redundant data reaches a preset second redundant data amount reference value;

[0167] A second redundant data generation time period acquisition module is configured to acquire a second redundant data generation time period when the moment when the clean data set is obtained by using the redundant data removed corresponds to the moment when the amount of redundant data reaches a preset redundant data amount reference value;

[0168] A redundant data generation index parameter acquisition module, configured to acquire redundant data generation index parameters using the first redundant data generation time period and the second redundant data generation time period;

[0169] The redundant data generation index parameter is obtained by the following formula:

[0170]

[0171] Wherein, Y represents a redundant data generation indicator parameter; R1 and R2 represent a first redundant data amount reference value and a second redundant data amount reference value, respectively; T1 and T2 represent a first redundant data generation time period and a second redundant data generation time period, respectively;

[0172] The redundant data generation indicator parameter is compared with a preset parameter threshold value. When the redundant data generation indicator parameter exceeds the preset parameter threshold value, the redundant data generation indicator parameter is judged to be abnormal and an abnormal alarm is issued.

[0173] The technical effect of the above technical solution is: through the reference value setting module, the user can flexibly set the first redundant data volume reference value and the second redundant data volume reference value. This flexibility enables the module to adapt to different requirements for the sensitivity of redundant data in different application scenarios. The data volume monitoring module can monitor the data volume of redundant data in real time to ensure that the dynamic changes of redundant data are captured in time. The first moment acquisition module records the moment when the clean data set is generated, providing a benchmark for subsequent time period calculations. When the redundant data reaches the preset first and second redundant data volume reference values, the second moment acquisition module and the third moment acquisition module respectively record these key moments. Using these moments and the moment when the clean data set is generated, the first redundant data generation time period acquisition module and the second redundant data generation time period acquisition module can accurately identify the time period when redundant data is generated.

[0174] The Redundant Data Generation Index Parameter Acquisition module uses the identified time period and a preset redundant data reference value to calculate a redundant data generation index parameter using a formula. This parameter objectively reflects the generation of redundant data. The Redundant Data Anomaly Determination and Alarm module compares the calculated redundant data generation index parameter with a preset parameter threshold. If the parameter exceeds the threshold, a redundant data generation anomaly is determined and an immediate anomaly alarm is issued. This mechanism helps to promptly identify and address potential data issues, preventing negative impacts on subsequent data analysis. The entire module not only filters out redundant data but also conducts a comprehensive analysis of redundant data generation. This comprehensive approach makes it widely applicable in data management and analysis.

[0175] In summary, this technical solution, through a series of precisely designed submodules, effectively monitors, identifies, and processes redundant data. It not only improves data quality and accuracy but also promptly detects and addresses data anomalies, providing strong support for data analysis and decision-making.

[0176] Specifically, the reference value setting module includes:

[0177] Redundant data information extraction module, used to extract the redundant data amount corresponding to each redundant data elimination in the historical operation records;

[0178] The redundant data amount extraction module is used to extract the redundant data amount corresponding to the current redundant data elimination;

[0179] A first redundant data amount reference value setting module is configured to obtain a first redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination;

[0180] The first redundant data amount reference value is obtained by the following formula:

[0181]

[0182] Wherein, R1 represents the first redundant data amount reference value; n represents the number of times redundant data is eliminated in the historical operation record; R i Indicates the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R min Indicates the minimum amount of redundant data corresponding to n redundant data eliminations in the historical operation records;

[0183] A second redundant data amount reference value setting module is configured to obtain a second redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination;

[0184] The second redundant data amount reference value is obtained by the following formula:

[0185]

[0186] Wherein, R2 represents the second redundant data amount reference value; n represents the number of times redundant data is eliminated in the historical operation record; R i Indicates the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R max Indicates the maximum amount of redundant data corresponding to the elimination of n redundant data in the historical operation record; Rz Indicates the median value of the redundant data volume corresponding to n redundant data eliminations in the historical operation records.

[0187] The technical effect of the above technical solution is that, through the redundant data information extraction module and the redundant data volume extraction module, the technical solution can obtain the redundant data volume corresponding to each redundant data elimination in the historical operation record, as well as the redundant data volume of the current redundant data elimination. This provides a rich data foundation for setting subsequent reference values. Such a setting can, to a certain extent, avoid excessively high reference values ​​due to individual extreme values ​​(such as the maximum value), while also taking into account changes in current data and the overall distribution of historical data. By setting a first redundant data volume reference value and a second redundant data volume reference value, the technical solution provides a more flexible and accurate basis for subsequent redundant data monitoring and processing. When the amount of redundant data reaches or exceeds these reference values, corresponding processing mechanisms can be triggered, such as redundant data elimination and abnormality alarms. Because the reference values ​​are set based on both historical and current data, the technical solution can adapt to data changes and fluctuations to a large extent, and the reference value settings match the changes and fluctuations of the data. When the data distribution changes, the reference values ​​can also be adjusted accordingly, thereby enhancing the adaptability and robustness of the system.

[0188] In summary, this technical solution, by intelligently setting the first and second redundant data reference values, provides a more scientific, flexible, and accurate basis for monitoring and processing redundant data. This helps improve the efficiency and accuracy of data processing and reduces the impact of data anomalies on system performance.

[0189] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for screening redundant data in a power system, characterized in that: The following steps are involved: Data acquisition and preprocessing: regularly acquire raw data from the power system data source, store the raw data in the local database, and perform preprocessing operations; Data fusion processing: fusing the pre-processed multi-source heterogeneous energy data to generate a unified fusion data set; Frequent item set mining: set item set rules, scan the fused data set layer by layer, calculate the item set support, prune the item set according to the support threshold, and iteratively generate frequent item sets of the power flow data; Data discretization processing: cluster analysis is performed on frequent item sets, data points are assigned to the nearest centroid, and the centroid position is updated repeatedly until the centroid is stable to form a discrete result; Association rule construction: generating potential association rules from discrete results, calculating the support and confidence of association rules, and selecting rules whose support and confidence exceed the preset threshold as strong association rules; Redundant data screening, applying strong association rules to the original or preprocessed data set, matching and marking redundant data, removing marked redundant data, generating and storing clean data sets, verifying the clean data sets, confirming that no redundant data is missing, and outputting a verification report.

2. A system for screening redundant data of a power system, applied in the method for screening redundant data of a power system according to claim 1, characterized in that: include: Data acquisition module, used to: Regularly obtain raw data from the power system data source and store it in a local database, wherein the raw data includes a power flow data set and auxiliary data for selecting a threshold value of a data item set support, and perform preprocessing operations on the raw data; Data fusion processing module, used for: The pre-processed multi-source heterogeneous energy data from different sources are fused to generate a fused data set; Frequent itemset mining module, used for: Set the item set rules, scan the fused data set layer by layer, calculate the support count of each item set, prune the item set according to the support count threshold, and repeat the iterative process to obtain the frequent item set of the power flow data; Discrete processing modules for: Discretize the frequent itemsets, perform cluster analysis on the frequent itemsets, assign the data points to the nearest centroid, form multiple discrete results, update the centroid position until the centroid no longer changes, and complete the discrete processing; Association rule building blocks for: Perform association operations on the frequent item sets in the discrete results respectively, calculate the support and confidence of the association rules, and when the support and confidence of the association rules reach the preset threshold, they are regarded as strong association rules; Redundant data filtering module, used for: Based on the constructed strong association rules, the redundant data in the power system are matched and screened out. The strong association rules are applied to the original data set or the preprocessed data set, the data items that meet the redundant characteristics are identified and marked, the redundant data are removed from the data set, and a clean data set is generated.

3. A system for removing redundant data from a power system as claimed in claim 2, characterized in that: The frequent item set mining module comprises: Itemset generation unit, used for: Receive the fused data set from the data fusion processing module, set the minimum item set length and the maximum item set length, and select all unique single items from the fused data set as the initial candidate item set Q1; For each Q k , by combining the existing Q k-1 Generate new Q k Candidate item set, where k represents the candidate item set number, k>1, all newly generated Q k The candidate itemset is output as a candidate itemset list; Support calculation unit, used for: Receive the candidate item set list from the item set generation unit, traverse the fused data set, count each candidate item set, determine the number of times each candidate item set appears in the fused data set, and obtain the support; Compare the support of each candidate item set with the preset support threshold. When the support of a candidate item set is less than the support threshold, the candidate item set is removed from the candidate item set list. After all candidate item sets are compared, a frequent item set list is obtained. Connection collection building blocks for: Receive a list of frequent itemsets, and for each Q in the frequent itemset list k Candidate item set, find and combine all possible Q k-1 Subset pairs, generate new Q k+1 Candidate item set; The newly generated Q k+1 The candidate item set is taken as input and returned to the item set generation unit for iterative processing until no new frequent item sets are generated, and the frequent item sets of the power flow data are obtained.

4. A system for screening redundant data of a power system as claimed in claim 2, characterized in that: The discrete processing module comprises: Cluster initialization unit, used to: receiving frequent itemsets of power flow data as input from the frequent itemset mining module, selecting randomly selected data points as initial centroids, and outputting the selected initial centroids to the clustering iteration unit; Clustering iteration unit, used to: Receive the initial centroid position from the cluster initialization unit, traverse all data points in the frequent itemset of the power flow data, and assign the data points to the nearest centroid according to the Euclidean distance; For each cluster, recalculate and update the centroid position of the cluster, and repeat the allocation and updating process until a preset number of iterations is reached; The final clustering results are output to the association rule construction module.

5. A system for screening redundant data of a power system as claimed in claim 2, characterized in that: The association rule building module includes: A rule generation unit, used to: Receive the final clustering results from the discrete processing module, generate all possible item set pairs for the item sets in each cluster, and use the generated item set pairs as potential association rules; Support confidence calculation unit, used for: receiving potential association rules from a rule generation unit; Traverse the preprocessed original data set, record the number of times each potential association rule appears in the data set, and calculate the support by dividing the number of times each potential association rule appears in the data set by the size of the data set; For each potential association rule item set pair, the two item sets are regarded as rule A and rule B, and the conditional probability that rule B also occurs when rule A occurs is calculated to obtain the confidence level; Output the support and confidence of each potential association rule.

6. A system for screening redundant data in a power system as claimed in claim 5, characterized in that: The association rule building module further includes: Rule filtering unit, used to: Calculate the support and confidence of each potential association rule, and compare the support and confidence with the preset threshold. If and only if the support and confidence of a potential association rule exceed the preset threshold, it is regarded as a strong association rule, and the selected strong association rule is output; Rule output unit, used for: The strong association rules are formatted, and the formatted strong association rules are output to the redundant data screening module.

7. A system for screening redundant data in a power system as claimed in claim 2, characterized in that: The redundant data screening module comprises: Rule application unit, used to: Receive the formatted strong association rules, load the preprocessed original data set from the local database, traverse each data item in the data set, match it with the strong association rules, and mark the data items that match the strong association rules as redundant; Data filtering unit for: Receive data items marked as redundant, traverse the data set, remove all data items marked as redundant, generate a clean data set without redundant data, and store it back to the local database.

8. A system for removing redundant data from a power system as claimed in claim 7, characterized in that: The redundant data screening module further includes: Screening verification unit for: The clean data set after screening is loaded, and the clean data set is matched again using the same strong association rule to check whether there are any missing redundant data items, and a verification report is output, which includes the amount of redundant data screened out and statistical information of the remaining data set.

9. A system for screening redundant data in a power system as claimed in claim 7, characterized in that: The redundant data screening module further includes: A reference value setting module, used to set a first redundant data amount reference value and a second redundant data amount reference value; wherein the first redundant data amount reference value is lower than the second redundant data amount reference value; The first moment acquisition module is used to extract the moment of obtaining a clean data set after removing redundant data; A data volume monitoring module is used to monitor the volume of redundant data in real time; A second time acquisition module is used to acquire a time corresponding to when the amount of redundant data reaches a preset first redundant data amount reference value when the amount of redundant data reaches a preset first redundant data amount reference value; A first redundant data generation time period acquisition module, used to obtain the first redundant data generation time period when the time when the clean data set is obtained by using the redundant data to be eliminated and the time when the amount of redundant data reaches a preset redundant data amount reference value correspond to each other; A third moment acquisition module is used to acquire a moment corresponding to when the amount of redundant data reaches a preset second redundant data amount reference value when the amount of redundant data reaches a preset second redundant data amount reference value; A second redundant data generation time period acquisition module, used to obtain a second redundant data generation time period when the time when the clean data set is obtained by using the redundant data to be eliminated corresponds to the time when the amount of redundant data reaches a preset redundant data amount reference value; A redundant data generation index parameter acquisition module, used to acquire redundant data generation index parameters using the first redundant data generation time period and the second redundant data generation time period; The redundant data generation index parameter is obtained by the following formula: Wherein, Y represents a redundant data generation indicator parameter; R1 and R2 represent a first redundant data amount reference value and a second redundant data amount reference value, respectively; T1 and T2 represent a first redundant data generation time period and a second redundant data generation time period, respectively; The redundant data generation indicator parameter is compared with a preset parameter threshold value. When the redundant data generation indicator parameter exceeds the preset parameter threshold value, it is determined that the redundant data generation is abnormal and an abnormal alarm is issued.

10. A system for screening redundant data in a power system as claimed in claim 9, characterized in that: Reference value setting module, including: A redundant data information extraction module is used to extract the amount of redundant data corresponding to each redundant data removal in the historical operation records; A redundant data volume extraction module is used to extract the redundant data volume corresponding to the current redundant data removal; A first redundant data amount reference value setting module, used to obtain a first redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination; The first redundant data amount reference value is obtained by the following formula: Wherein, R1 represents the first redundant data amount reference value; n represents the number of redundant data removals in the historical operation record; R i represents the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R min Indicates the minimum amount of redundant data corresponding to the elimination of n redundant data in the historical operation records; A second redundant data amount reference value setting module, used to obtain a second redundant data amount reference value by combining the redundant data amount corresponding to each redundant data elimination in the historical operation record with the redundant data amount corresponding to the current redundant data elimination; The second redundant data amount reference value is obtained by the following formula: Wherein, R2 represents the second redundant data amount reference value; n represents the number of redundant data removals in the historical operation records; R i represents the amount of redundant data corresponding to the i-th elimination; R c Indicates the amount of redundant data corresponding to the current redundant data removal; R max Indicates the maximum amount of redundant data corresponding to the elimination of n redundant data in the historical operation records; R z Indicates the median value of the amount of redundant data corresponding to n redundant data removals in the historical operation records.

Images