Data processing method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a medium and a product, and the method comprises the steps: obtaining the operation data of a power system, carrying out the analysis, obtaining a clustering analysis data set, inputting the clustering analysis data set into an abnormal value detection model, determining and correcting an abnormal value, obtaining a data set corrected by the abnormal value, and carrying out the detection of the abnormal value. And deleting redundant data in the data set subjected to abnormal value correction through importance analysis, and inputting the data set without the redundant data into a data quality optimization model to obtain a target operation data processing result. The abnormal value detection model and the data quality optimization model are obtained by training accurate data meeting a preset data quality requirement in the historical operation data. According to the technical scheme of the embodiment of the invention, the problem that the currently obtained operation data is inaccurate is solved, the data quality can be improved through an intelligent algorithm and a machine learning model, the reliability and availability of the data are guaranteed, and reliable data support is provided for data analysis and decision making of a power system.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, apparatus, device, medium and product. Background Art

[0002] With the development of the economy and the growth of energy demand, the scale of the power system is becoming increasingly large, covering more power generation facilities, transmission lines, substation equipment and user terminals, which has led to an exponential growth in the amount of data, more diverse data sources, and a significant increase in the difficulty of data management and quality control. In order to realize the intelligent operation, monitoring and management of the power system, a large amount of real-time and accurate data needs to be analyzed and processed. High-quality data is essential for key applications such as operation risk management, equipment risk management, and power system risk management in the intelligent power system.

[0003] The operation of the power system is affected by many factors, such as weather conditions, equipment aging, load changes, etc., which makes the data highly dynamic and uncertain. At the same time, interference and noise in the power system may also affect the accuracy of the data. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device, equipment, medium and product, which can improve data quality through intelligent algorithms and machine learning models, ensure data reliability and availability, and provide reliable data support for power system data analysis and decision-making.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0006] Acquiring operation data of the power system, wherein the operation data includes operation data of power generation facilities, operation data of transmission lines, operation data of substation equipment and operation data of user terminals;

[0007] Perform cluster analysis on the running data to obtain a cluster analysis data set;

[0008] Inputting the cluster analysis data set into the outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set;

[0009] The data importance analysis is performed on the data set corrected by outliers by using the random forest importance ranking algorithm, and the redundant data in the data set corrected by outliers is deleted according to the importance analysis result to obtain a data set with redundant data removed;

[0010] Inputting the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data;

[0011] Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets the preset data quality requirements.

[0012] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0013] An operation data acquisition module is used to acquire the operation data of the power system, wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals;

[0014] A clustering module is used to perform cluster analysis on the running data to obtain a cluster analysis data set;

[0015] An outlier correction module is used to input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain an outlier-corrected data set;

[0016] A redundant data removal module is used to perform data importance analysis on the data set corrected by outliers through a random forest importance ranking algorithm, and delete redundant data in the data set corrected by outliers according to the importance analysis result to obtain a data set with redundant data removed;

[0017] A quality optimization module, used to input the data set with redundant data removed into the data quality optimization model to obtain a target operation data processing result of the operation data;

[0018] Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets the preset data quality requirements.

[0019] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0020] one or more processors;

[0021] A memory for storing one or more programs;

[0022] When the above one or more programs are executed by one or more processors, the above one or more processors implement the data processing method provided by any embodiment of the present invention.

[0023] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method as provided in any embodiment of the present invention.

[0024] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a data processing method as provided in any embodiment of the present invention.

[0025] The embodiments of the above invention have the following advantages or beneficial effects:

[0026] The embodiment of the present invention obtains the operation data of the power system, wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals; performs cluster analysis on the operation data to obtain a cluster analysis data set; inputs the cluster analysis data set into the outlier detection model, determines the outliers in the cluster analysis data set, and corrects the outliers to obtain an outlier-corrected data set; performs data importance analysis on the outlier-corrected data set by the random forest importance sorting algorithm, deletes the redundant data in the outlier-corrected data set according to the importance analysis result, and obtains a data set with redundant data removed; inputs the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data; wherein the outlier detection model and the data quality optimization model are obtained by training accurate data in historical operation data that meets the preset data quality requirements. The technical solution of the embodiment of the present invention solves the problem of inaccurate operation data currently obtained, and can improve data quality through intelligent algorithms and machine learning models, ensure data reliability and availability, and provide reliable data support for power system data analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0028] Figure 2 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0029] Figure 3 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0030] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;

[0031] Figure 5 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0033] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention is applicable to scenarios of power system data processing. The method can be executed by a data processing device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0034] like Figure 1 As shown, the data processing method of this embodiment includes the following steps:

[0035] S110: Acquire operation data of the power system.

[0036] Among them, the operating data includes power generation facility operating data, transmission line operating data, substation operating data and user terminal operating data.

[0037] On the premise of obtaining authorization, the data transmitted by each transmission node of the power system is collected and preprocessed, and log records and backups can be performed at the same time. Specifically, the identity of the personnel who perform the collection of multi-source data is authenticated, the target, scope, method and intended use of the collected data are clarified, and the collection authorization is obtained in compliance with relevant regulations and processes. By connecting professional business systems such as marketing systems, infrastructure systems and equipment management systems on the network nodes of power system data transmission, the operating data obtained includes power generation facility operating data, transmission line operating data, substation operating data and user terminal operating data. When data passes through the network node, it is captured. The captured data can be transmitted to the data processing server in real time after preprocessing, that is, the operating data of the power system is transmitted to the data processing server in real time.

[0038] At each critical link of data collection, decryption and processing, the system can automatically generate log records through the built-in log generation module. Each log record is assigned a globally unique identifier and the generated log records are stored in real time in a dedicated log database. The log database uses asymmetric key encryption technology and role-based access control.

[0039] S120: Perform cluster analysis on the operation data to obtain a cluster analysis data set.

[0040] In order to make large amounts of data easier to analyze and process, in this embodiment, cluster analysis is performed on the operating data, for example, cluster analysis is performed based on features determined by feature engineering to obtain at least one cluster analysis data set.

[0041] S130, inputting the cluster analysis data set into an outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set.

[0042] The outlier detection model can be obtained by training the isolation forest model, specifically by training the isolation forest model with accurate data that meets the preset data quality requirements in the historical operation data. Accurate data that meets the preset data quality requirements can be determined in the historical operation data according to preset screening rules. The preset data quality requirements can be determined based on experience, and this embodiment does not limit this. The outlier detection model constructs a decision boundary to distinguish accurate data from outliers based on the distribution pattern of accurate data learned in training and the difference between abnormal data and accurate data. Outliers can be corrected by a preset correction algorithm to obtain a data set corrected by outliers.

[0043] S140, performing data importance analysis on the data set corrected by the outliers by using a random forest importance ranking algorithm, and deleting redundant data in the data set corrected by the outliers according to the importance analysis result to obtain a data set with redundant data removed.

[0044] The data importance analysis is performed on the data set corrected for outliers using the random forest importance ranking algorithm. The redundant data features are determined based on the importance score of each feature of the data set in the importance analysis results, and the redundant data corresponding to the redundant data features are removed to obtain a data set without redundant data.

[0045] S150: Input the data set from which redundant data has been removed into a data quality optimization model to obtain a target operating data processing result of the operating data.

[0046] The data quality optimization model is obtained by training accurate data that meets the preset data quality requirements in the historical operation data. The data quality optimization model may include a trained long short-term memory network and a multi-layer perceptron network, which are respectively used to optimize the data quality of different types of data in the data set from which redundant data is removed, namely, time series data and structured data, and then fuse them to obtain the target operation data processing result of the operation data. Specifically, in the training stage, the time series data in the accurate data is used as input, and the time series patterns and laws of the time series data in the accurate data are learned by adjusting the parameters of the long short-term memory network, such as the weights of the forget gate, the input gate, and the output gate. These parameters are used in the prediction stage to extract and process the features of the time series data in the newly input data set from which redundant data is removed, so as to obtain the preliminary prediction results of the time series data in the data set from which redundant data is removed. For the multi-layer perceptron network, the structured data in the accurate data is used as input, and the intrinsic relationship and pattern of the structured data in the accurate data is learned by adjusting the connection weights and biases between the neurons in each layer. During the prediction, these learned parameters are used to process the structured data in the data set from which redundant data is removed, and the preliminary prediction results in the data set from which redundant data is removed are generated. For the preliminary prediction results of the two types of data, the fusion weight is determined by a cross-validation method, and the target operation data processing result is obtained by fusion according to the fusion weight. Compared with the original acquired operation data, the target operation data processing result has better data quality, is more accurate and concise, and can improve the stability of subsequent decision-making or data analysis.

[0047] The technical solution of this embodiment is to obtain the operation data of the power system, wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals; perform cluster analysis on the operation data to obtain a cluster analysis data set; input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain a data set corrected by the outliers; perform data importance analysis on the data set corrected by the outliers by the random forest importance sorting algorithm, delete the redundant data in the data set corrected by the outliers according to the importance analysis result, and obtain a data set with redundant data removed; input the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data; wherein the outlier detection model and the data quality optimization model are obtained by training accurate data in the historical operation data that meets the preset data quality requirements. The technical solution of the embodiment of the present invention solves the problem of inaccurate operation data currently obtained, and can improve data quality through intelligent algorithms and machine learning models, ensure data reliability and availability, and provide reliable data support for power system data analysis and decision-making.

[0048] Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present invention. This embodiment and the data processing method in the above embodiment belong to the same inventive concept, and further describes the process of cluster analysis and removal of redundant data. The method can be executed by a data processing device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0049] like Figure 2 As shown, the data processing method of this embodiment includes the following steps:

[0050] S210, obtaining the operation data of the power system.

[0051] Among them, the operating data includes power generation facility operating data, transmission line operating data, substation operating data and user terminal operating data.

[0052] In this embodiment, at each key link in the operation of data collection and processing, the system automatically generates detailed and accurate log records through the built-in log generation module. The log record contains the precise operation time, accurate to the millisecond level, and records the identity information and authority level of the specific operator who performs the operation. The operation content is recorded in detail, including the data collection source, collection time, collection volume, data processing method, data changes during processing, etc. At the same time, the source and target address of the data are recorded, including the network address, port number and other information of the sender and receiver. A globally unique identifier (UUID) is assigned to each log record so that specific log records can be quickly and accurately located and identified in subsequent query, tracking and audit processes.

[0053] The generated log records are stored in real time in a highly secure and reliable dedicated log database. The log database uses advanced encryption technology and access control strategies to ensure that only authorized personnel can access and read log data. Log data is stored in a structured manner for quick query and analysis. A strict log database backup plan is formulated to regularly back up log data to offline secure storage media, such as tapes, encrypted hard drives, etc. At the same time, the backup data is stored in a secure facility off-site to prevent data loss due to local disasters or accidents. Regularly check the integrity and readability of the backup data to ensure the availability and reliability of the backup data.

[0054] S220: Perform feature engineering on the operation data to obtain at least one data feature of the operation data.

[0055] A distributed computing framework environment is built through the Hadoop distributed file system, YARN and Spark computing framework, in which NumPy is used for data manipulation and preprocessing, Scitit-learn is used for machine learning algorithms, and Seaborn is used for data visualization.

[0056] In this embodiment, data processing related algorithms and models are set on different distributed nodes, and can be subsequently integrated through voting method or the like according to the parameter optimization results of the data processing related algorithms and models on different nodes.

[0057] Perform feature engineering on the preprocessed operation data to extract meaningful features. For example, for equipment operation status data, extract features such as equipment operation time, load rate, failure rate, and number of repairs; for operation plan data, extract features such as the planned start time, planned completion time, operation manager, number of operators, and operation location.

[0058] S230 , clustering the operating data using a hierarchical clustering algorithm according to data features to obtain a clustering data set to be optimized.

[0059] A hierarchical clustering algorithm is used to cluster data based on the extracted features. During the clustering process, the clustering quality is evaluated by adjusting the parameters of the clustering algorithm. The clustering results are analyzed and interpreted, and similar data points are classified into the same category for judgment.

[0060] Specifically, based on the extracted features, a hierarchical clustering algorithm is used to cluster the data. The agglomerative hierarchical clustering algorithm is used to first treat each data point as a separate cluster, then use the Euclidean distance to calculate the similarity between clusters, and merge the two clusters with the highest similarity into a new cluster; repeat this process until the preset number of clusters is reached or a certain termination condition is met. In the clustering process, the clustering results are optimized by adjusting the parameters of the clustering algorithm, such as the distance measurement method, connection criterion or the number of clusters. The silhouette coefficient indicator is used to evaluate the quality of clustering and select the optimal clustering result. The clustering results are analyzed and interpreted, similar data points are classified into the same category, and the integrity and consistency of the data are checked.

[0061] S240: Calculate the clustering statistics of the clustering data set to be optimized, determine the abnormal clustering data set according to the clustering statistics, and delete the abnormal data set to obtain the cluster analysis data set.

[0062] For each cluster, i.e. the cluster data set to be optimized, its cluster statistics, i.e. the center, radius, density and other statistics are calculated to better understand the distribution characteristics of the data. For inconsistent or abnormal clusters, they are determined as abnormal data sets, and the abnormal data sets are deleted or corrected to obtain the cluster analysis data set.

[0063] S250: Input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain an outlier-corrected data set.

[0064] In an optional implementation, the cluster analysis data set is input into an outlier detection model, outliers in the cluster analysis data set are determined, and the outliers are corrected to obtain a data set corrected by outliers. The cluster analysis data set can be input into an outlier detection model based on an isolation tree algorithm, and the outlier detection model performs an outlier judgment on the data in the cluster analysis data set based on the characteristics of the accurate data learned in the training phase to obtain outliers; a correction value corresponding to the outlier value is obtained through an outlier correction algorithm, and the correction value replaces the outlier value to obtain a data set corrected by outliers.

[0065] In view of the characteristics and requirements of power system data, a machine learning-based method, the Isolation Forest algorithm, is selected. Isolation Forest identifies outliers by randomly dividing the data space, and uses accurate data in historical operation data that meets the preset data quality requirements to train the selected anomaly detection model. During the training process, the parameters of the model, such as the number of trees in the Isolation Forest, are adjusted to improve the performance and accuracy of the model. The trained Isolation Forest anomaly detection model is used to perform online detection on the real-time collected data. The cluster analysis data set is input into the model, and the model anomaly detection model outputs the judgment result based on the characteristics of the accurate data learned in the training phase, and identifies whether the data point is an outlier. For the detected outliers, the correction value corresponding to the outlier is obtained by the preset outlier correction algorithm, and the correction value replaces the outlier value to obtain the outlier-corrected data set. Detailed analysis and processing can also be performed. If the anomaly is caused by data collection errors or data quality problems, data correction or data conversion is carried out in a timely manner; if the anomaly reflects the actual problems in the power system (such as planned work without tickets, unqualified operators, etc.), the corresponding early warning and emergency processing mechanism is triggered.

[0066] S260. Set decision tree parameters of the random forest importance ranking algorithm.

[0067] Determine the number of decision trees in the random forest and select them through methods such as cross-validation; then set parameters such as the maximum depth of each decision tree and the minimum number of samples for splitting to control the complexity of the model and prevent overfitting. When training the random forest model, for each decision tree, randomly extract samples from the data set (using sampling with replacement, i.e. Bootstrap sampling), and randomly select feature subsets for splitting node selection and decision tree growth. In this way, multiple independent but related decision trees are constructed to form a random forest.

[0068] S270, calculating feature importance scores for data features of the data set corrected for outliers according to the decision tree parameters, and obtaining feature importance score calculation results.

[0069] The importance score of each data feature is calculated through the feature importance evaluation mechanism inside the random forest model, Mean Decrease Impurity (MDI). In the MDI method, during the construction of the decision tree, the impurity reduction caused by each data feature when the node is split is recorded, and the impurity reduction of the same data feature in all decision trees is averaged to obtain the feature importance score of the data feature.

[0070] S280, determining redundant data features according to the feature importance score calculation result, determining data corresponding to the redundant data features as redundant data, deleting the redundant data in the data set corrected by the outliers, and obtaining a data set with redundant data removed.

[0071] According to the feature importance score of each data feature in the feature importance score calculation result, the redundant data feature is determined, and the column data corresponding to the redundant data feature is determined as redundant data.

[0072] In an optional embodiment, redundant data features are determined based on feature importance score calculation results, which can be to determine the feature importance score of the data feature of the data set corrected for outliers in the feature importance score calculation results, numerically compare the feature importance score with a preset feature importance threshold, and determine data features that are less than the preset feature importance threshold as redundant data features.

[0073] Remove the column data corresponding to the redundant data features from the data set, retain the data of the important features, and obtain a streamlined data set. In the process of removing redundant data, it is necessary to pay attention to maintaining the consistency and integrity of the data to avoid information loss or data structure damage due to data deletion.

[0074] S290. Input the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data.

[0075] Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets the preset data quality requirements.

[0076] The technical solution of this embodiment is to obtain the operation data of the power system; wherein the operation data includes the operation data of the power generation facilities, the operation data of the transmission lines, the operation data of the substation and the operation data of the user terminals; perform feature engineering on the operation data to obtain at least one data feature of the operation data; cluster the operation data by a hierarchical clustering algorithm according to the data features to obtain a clustering data set to be optimized; calculate the clustering statistics of the clustering data set to be optimized, determine the abnormal clustering data set according to the clustering statistics, delete the abnormal data set to obtain the clustering analysis data set; input the clustering analysis data set into the outlier detection model, determine the outliers in the clustering analysis data set, and correct the outliers to obtain a data set corrected by the outliers; set the decision tree parameters of the random forest importance ranking algorithm; calculate the feature importance score of the data features of the data set corrected by the outliers according to the decision tree parameters to obtain the feature importance score calculation result; determine the redundant data features according to the feature importance score calculation result, determine the data corresponding to the redundant data features as redundant data, delete the redundant data in the data set corrected by the outliers, and obtain the data set without redundant data. The data set with redundant data removed is input into the data quality optimization model to obtain the target operation data processing result of the operation data; wherein, the outlier detection model and the data quality optimization model are obtained by training accurate data in the historical operation data that meets the preset data quality requirements. The technical solution of the embodiment of the present invention solves the problem of inaccurate operation data currently obtained, and can improve the processability of data through cluster analysis and redundant data removal, provide a basis for subsequent data processing, and provide reliable data support for power system data analysis and decision-making.

[0077] In a specific data processing example, the anomaly detection model, clustering method and random forest algorithm can also be optimized in the training stage or application stage. Specifically, for the anomaly detection model, the data set is divided into a training set, a validation set and a test set according to a certain ratio. The training set is the correct data, and the validation set and the test set include the correct data and the abnormal data. The training set is used to train the model, the validation set is used to adjust the hyperparameters during the training process, and the test set is used to finally evaluate the performance of the model. After the model is trained with the training set, the performance of the model is evaluated on the validation set and the test set respectively. In the performance evaluation, the adjusted Rand coefficient is used to measure the clustering results and the true labels, the precision rate is used to evaluate the ability of the Isolation Forest anomaly detection model, and the determination coefficient is used to evaluate the predictive performance of the random forest model. The ROC curve can be drawn to observe the TPR and FPR at different thresholds, and the AUC-ROC value is calculated, and the threshold is adjusted according to the calculation results. The anomaly detection model can also be optimized by adjusting feature engineering and model fusion.

[0078] Specifically, the adjusted Rand coefficient is calculated by comparing the actual category and the clustering result. The value range is between [-1, 1]. The closer the value is to 1, the more consistent the clustering result is with the actual situation. Precision: The ratio of the number of samples detected as abnormal and indeed abnormal to the number of samples detected as abnormal. Coefficient of determination: Measures the degree of fit of the model to the data. The value range is between [0, 1]. The closer the value is to 1, the better the model fit is.

[0079] For clustering models, clustering evaluation indicators are calculated and compared with historical clustering results. For random forest models, MSE, MAE and R-squared of regression problems are calculated, and the values ​​and changing trends of evaluation indicators are analyzed. Clustering model optimization is achieved by adjusting the number of clusters and improving feature selection and transformation.

[0080] For random forest optimization, we can increase the number of trees, adjust the tree depth and split parameters, or use feature selection methods. We can use model-based hyperparameter optimization methods to tune the hyperparameters of random forest.

[0081] Anomaly detection models, clustering methods, and random forest algorithms can also be automatically tuned through particle swarm optimization algorithms. According to the key parameters of each algorithm, namely the anomaly detection model, clustering method and random forest algorithm, the position vector of the particle is formed. The value range of each parameter is determined according to its reasonable range in the algorithm. A certain number of particles are randomly generated, each particle represents a set of parameter combinations, and the position and speed of each particle are initialized. Based on the optimization objectives and constraints, a fitness function is constructed to evaluate the performance of the optimization data model under the parameter combination represented by each particle. The fitness value is calculated for each particle, and the individual optimal position and the global optimal position are updated. According to the individual optimal position and the global optimal position, the speed and position of the particle are updated according to the particle swarm optimization algorithm. When the position of the particle exceeds the legal value range of the parameter, it is adjusted to the boundary value. When the preset maximum number of iterations is reached or the fitness value meets the preset optimal standard, the iteration is stopped. The optimization results of each algorithm model are obtained from different nodes of the distributed framework, and the majority voting method, weighted average method or random subspace method are selected as the fusion method. The fusion weight is determined for the results of each model through experiments, and a data visualization interface is constructed to display the fused results. The results are provided with textual explanations and descriptions in combination with the process and knowledge of power system safety production operations, and experts in the field of power systems are invited to evaluate the fusion results and explanations.

[0082] Specifically, the key parameters of each algorithm in the optimization data model are sorted out in detail, such as the distance measurement parameters (such as Euclidean distance, Manhattan distance, etc.) and the number of clusters in the hierarchical clustering algorithm; the threshold parameters in the anomaly detection algorithm; the number of trees in the random forest and the feature selection ratio. Set a reasonable encoding method for each parameter, such as integer encoding, real number encoding or binary encoding, and determine its value range to ensure that the position of the particle in the search space has practical physical meaning and operability. Determine the size of the particle swarm based on computing resources and the complexity of the problem. For each particle, randomly generate an initial position within the value range of its parameters, representing a set of possible parameter combinations. Randomly initialize the particle's speed, and the speed range is set according to the parameter value range and the characteristics of the problem to control the particle's exploration ability in the search space.

[0083] Design a comprehensive fitness function to quantify and weighted sum the indicators such as accuracy, completeness, consistency, timeliness, reliability and availability of the data. For example, the accuracy weight is set to 0.3. For the quantification of each indicator, the relative deviation from the set target can be used as the measurement standard. For example, for accuracy, if the current accuracy is 90% and the target is 95%, the score of the accuracy indicator can be calculated as (90%-95%) / 95%. The scores of each indicator are converted to the [0,1] interval through normalization, and then weighted summed according to the weight to obtain the fitness value of the particle. In each iteration, for each particle, the parameter combination represented by its current position is applied to the ensemble model, and its fitness value is calculated through the fitness function. For each particle, its current fitness value is compared with its individual optimal fitness value. If it is better, the individual optimal position is updated. In the entire particle swarm, the individual optimal fitness values ​​of all particles are compared, the global optimal fitness value is found, and the corresponding global optimal position is recorded.

[0084] If the updated position of the particle exceeds the legal range of the parameter, it is handled according to the specific situation. For example, in the case of integer parameters, it can be forced to be set to the closest legal boundary value. For each parameter combination represented by the updated particle, check whether it meets the pre-set constraints. For example, whether the use of computing resources exceeds the carrying capacity of the server, whether the running time exceeds the preset upper limit, etc. If not, take the following measures to correct it: For minor violations of constraints, the particle's speed or position can be adjusted to make it gradually return to the legal area; for serious violations of constraints, the position and speed of the particle are regenerated to ensure that it restarts the search in a legal initial state. Set the maximum number of iterations, for example 100. When the number of iterations reaches this upper limit, stop the operation of the particle swarm optimization algorithm. Monitor the changes in the fitness value. If the improvement of the global optimal fitness value is less than a certain threshold (such as 0.1%) in several consecutive iterations (such as 10 times), the algorithm is considered to have converged and the iteration is stopped.

[0085] Deploy data collection modules on each node of the distributed framework to monitor and record the output results of each algorithm model in real time. For data clustering results, collect the cluster category and cluster center information of each data point. For anomaly detection results, obtain the data points marked as abnormal and their anomaly scores. For data after redundancy is removed, record statistical information such as data volume and feature distribution. If it is a classification problem, and the output results of multiple models are category labels, use the majority voting method, that is, count the number of times each category is predicted by different models, and select the category with the most predictions as the final fusion result. For numerical prediction problems, such as load prediction, the weighted average method can be used to determine the weights according to the historical prediction accuracy or other performance indicators of each model, and perform weighted average on the prediction values ​​of multiple models. Consider using methods in ensemble learning, such as the random subspace method in random forest, to randomly combine the results of multiple models to build a new fusion model.

[0086] Under different data sets and scenarios, the effects of different weight combinations on the performance of fusion results are tested respectively. Based on historical data and prior knowledge, the performance stability and accuracy of each model under different data distributions and business scenarios are analyzed. A dynamic weight adjustment mechanism is introduced to adjust the weights in real time according to the characteristics of real-time data and the recent performance of the model. A visualization interface is developed, and charts are used to display the fusion results. For example, a bar chart is used to display the data distribution in different clustering categories, and a line chart is used to display the changing trend of anomaly detection scores. Detailed text descriptions are provided for each visualization chart to explain the meaning and trend of the data. In combination with the professional knowledge and processes of power system safety production operations, in-depth explanations and analysis are provided for the results. A team of experts in the power system field, including senior engineers, data analysts, and business managers, is organized to conduct a comprehensive evaluation of the fusion results and interpretations. Based on their professional experience and actual business needs, the experts evaluate the accuracy, rationality, and practicality of the results, and put forward specific improvement opinions and suggestions. According to the feedback from experts, targeted adjustments and optimizations are made to the fusion method, weight determination strategy, and interpretation strategy.

[0087] Figure 3 This is a flowchart of a data processing method provided in an embodiment of the present invention. This embodiment and the data processing method in the above embodiment belong to the same inventive concept, and further describes the process of data processing by the data quality optimization model. The method can be executed by a data processing device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0088] like Figure 3 As shown, the data processing method of this embodiment includes the following steps:

[0089] S310: Obtain operation data of the power system.

[0090] Among them, the operating data includes power generation facility operating data, transmission line operating data, substation operating data and user terminal operating data.

[0091] S320: Perform cluster analysis on the operation data to obtain a cluster analysis data set.

[0092] S330, inputting the cluster analysis data set into the outlier detection model, determining the outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set.

[0093] S340, performing data importance analysis on the data set corrected by the outliers by using a random forest importance ranking algorithm, and deleting redundant data in the data set corrected by the outliers according to the importance analysis result to obtain a data set with redundant data removed.

[0094] S350: Input the time series data in the data set where redundant data is removed into the first data quality optimization network of the data quality optimization model to obtain a first model processing result of the time series data.

[0095] Among them, the first data quality optimization network is obtained based on long short-term memory network training.

[0096] Data with timestamps and numerical sequences are classified as time series data. The number, number of layers, and input and output dimensions of the first data quality optimization network are defined. The input layer receives the preprocessed time series data, which is processed by the long short-term memory network layer to extract time features, and finally outputs the first model processing result of the time series data through the fully connected layer.

[0097] During the training process, define the loss function, select the Adagrad optimization algorithm to adjust the learning rate, input the training set data into the joint model, regularly evaluate the model performance on the validation set, and adjust the hyperparameters based on the performance. Specifically, use the mean square error as the loss function of the first data quality optimization network, and the mean square error calculation formula is:

[0098]

[0099] Among them, y i is the true value, is the predicted value and N is the number of samples.

[0100] S360: Input the structured data in the data set from which redundant data is removed into a second data quality optimization network of the data quality optimization model to obtain a second model processing result of the structured data.

[0101] Among them, the second data quality optimization network is obtained based on multi-layer perceptron network training.

[0102] Classified and fixed-format data are classified as structured data and preprocessed. The number of layers, number of neurons in each layer, and activation function of the second data quality optimization network are designed. The input layer receives structured data, and after multiple layers of linear transformation and nonlinear activation, the second model processing results of the structured data are output.

[0103] During the training process, the cross entropy loss function is used as the loss function of the second data quality optimization network. The calculation formula of the cross entropy loss function is:

[0104]

[0105] Among them, p ij is the probability distribution of the true label, q ij is the probability distribution of the predicted label, and C is the number of categories;

[0106] The Adagrad algorithm automatically adjusts the learning rate of each parameter of the first data quality optimization network and the second data quality optimization network according to the square sum of the past gradients of the parameters. The update formula is:

[0107]

[0108] Among them, G t is the sum of squared gradients up to the tth step, and ∈ is a constant to prevent the denominator from being zero.

[0109] S370: Perform weighted summation on the first model processing result and the second model processing result to obtain a weighted summation result.

[0110] The processing result of the first model and the processing result of the second model are fused by weighted summation to obtain a weighted summation result.

[0111] S80. Input the weighted sum result into the fully connected layer of the data quality optimization model to obtain the target operation data processing result of the operation data.

[0112] The weighted summation result is input into the fully connected layer of the data quality optimization model. The number of neurons and activation function of the fully connected layer are predetermined according to the nature of the quality optimization task. The target operation data processing result is output through the fully connected layer.

[0113] The technical solution of this embodiment is to obtain the operation data of the power system; wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals; perform cluster analysis on the operation data to obtain a cluster analysis data set; input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain a data set corrected by the outliers; perform data importance analysis on the data set corrected by the outliers by using the random forest importance ranking algorithm, delete the redundant data in the data set corrected by the outliers according to the importance analysis result, and obtain a data set without redundant data; and sort the time series in the data set without redundant data. The column data is input into the first data quality optimization network of the data quality optimization model to obtain the first model processing result of the time series data; wherein the first data quality optimization network is obtained based on the long short-term memory network training; the structured data in the data set with redundant data removed is input into the second data quality optimization network of the data quality optimization model to obtain the second model processing result of the structured data; wherein the second data quality optimization network is obtained based on the multi-layer perceptron network training; the first model processing result and the second model processing result are weighted summed to obtain the weighted summation result; the weighted summation result is input into the fully connected layer of the data quality optimization model to obtain the target operation data processing result of the operation data. The technical solution of the embodiment of the present invention solves the problem of inaccurate operation data currently obtained, and can improve data quality through intelligent algorithms and machine learning models, ensure data reliability and availability, provide reliable data support for power system data analysis and decision-making, and obtain results by fusing different optimization models for different types of data to improve the accuracy of model prediction.

[0114] In a specific data processing example, for the long short-term memory network model, i.e., the first data quality optimization network, the internal structure of the long short-term memory network unit is designed in detail, including the number of neurons and the connection method of the input gate, forget gate, and output gate. The number of LSTM layers is determined, usually based on the complexity of the data and the length of the time series. For example, if the time series is long and the pattern is complex, the number of LSTM layers can be increased to better capture long-term dependencies. When defining the input and output dimensions, the input dimension should match the number of features of the time dependency data, and the output dimension is determined according to the goal of data quality optimization, such as the number of categories of the predicted correct values ​​or error types.

[0115] For the multilayer perceptron model, i.e., the second data quality optimization network, the number of neurons in each layer of the multilayer perceptron is carefully planned, and the network structure is usually constructed in an increasing or decreasing manner. For example, the number of neurons is gradually increased from the input layer to the hidden layer to increase the expressive power of the model, and then gradually decreased when approaching the output layer to avoid overfitting. Select a suitable activation function, such as ReLU, Sigmoid, or Tanh, based on the distribution of the data and task requirements. For the output layer, select a linear activation function or an appropriate classification activation function (such as Softmax) depending on whether the data quality optimization is regression or classification.

[0116] When fusing the outputs of the LSTM network model and the multilayer perceptron model, a weighted summation method is used, and appropriate weights need to be determined through experiments or based on prior knowledge. The final fully connected layer is used to synthesize the fused information and generate the final optimization result. The number of neurons and the choice of activation function depend on the nature of the data quality optimization.

[0117] Further, the loss function is defined, the Adagrad optimization algorithm is selected to adjust the learning rate, the training set data is input into the joint model, the model performance is regularly evaluated on the validation set, and the hyperparameters are adjusted according to the performance. For example, the weights and biases in the long short-term memory network and the multi-layer perceptron are randomly initialized, the training set data is input into the joint model, and the prediction value is obtained by forward calculation through the long short-term memory network layer and the multi-layer perceptron layer. For the long short-term memory network unit, its forward propagation formula is relatively complicated, mainly involving the calculation of the input gate, the forgetting gate, the output gate and the cell state, and the loss between the predicted value and the true value is calculated according to the selected loss function;

[0118] Based on the calculated loss, the gradient is calculated through the back propagation algorithm. For each parameter in the long short-term memory network and the multi-layer perceptron, its gradient is calculated according to the chain rule, and then the model parameters are updated according to the gradient calculated by the Adagrad optimization algorithm. This is repeated until the preset number of training rounds is reached or the loss converges to a certain threshold. During the training process, the model performance is regularly evaluated on the validation set, and the hyperparameters are adjusted according to the performance.

[0119] Specifically, taking a simple long short-term memory network unit as an example, the formula is as follows:

[0120] Input gate: i t =σ(W xi x t +W hi h t-1 +b i );

[0121] Forget gate: f t =σ(W xf xt +W hf h t-1 +b f );

[0122] Output gate: o t =σ(W xo x t +W ho h t-1 +b o );

[0123] Cell status update:

[0124]

[0125] Hidden state output: h t =o t ⊙tanh c t ;

[0126] Among them, σ is the Sigmoid function, ⊙ represents element-wise multiplication, W is the weight matrix, and b is the bias vector.

[0127] For a multi-layer perceptron, the forward propagation formula is: (l) =σ(W (l) a (l-1) +b (l) );

[0128] Among them, l represents the number of layers and a is the activation value of the neuron.

[0129] In this embodiment, the process of multi-source operation data quality optimization can also be monitored by indicators, comparison monitoring and log monitoring. The accuracy index is determined by using the F1 score to define the degree of match between the optimized data and the real and accurate data. The integrity index is measured by the proportion of complete data records in the total data records. The consistency index is determined by the logical and semantic consistency of the data. The time interval from the generation of data to the processing and error correction of the data is measured to determine the timeliness index. The calculation and recording frequency of the monitoring indicators is determined according to the importance and change speed of the data. The value of the monitoring indicators is calculated in real time during the data processing process, and the monitoring indicators are displayed in an intuitive way through visualization tools. According to the selected historical period, Data with good data quality confirmed through manual review is used as the benchmark data set. The currently processed data set is compared with the benchmark data set field by field at certain time intervals. The differences found are analyzed to determine whether they are changes in the data itself or problems in the data processing process. In each link of data collection, processing, error correction, and optimization, detailed operation logs are recorded and stored in a safe and reliable database. The logs are regularly analyzed to find possible problems or abnormal patterns. Association rule mining is used to conduct in-depth mining of log data, and thresholds and rules are set. When the monitoring indicators exceed the normal range, significant differences are found in the comparison, or abnormal patterns appear in the logs, the alarm mechanism is triggered.

[0130] Specifically, the F1 score calculation formula is:

[0131]

[0132] Where Precision is the accuracy, and the calculation formula is:

[0133]

[0134] Determine the frequency of calculation and recording of monitoring indicators based on the importance and rate of change of the data.

[0135] For key data and rapidly changing data, the monitoring frequency can be set to real-time or every few minutes; for relatively stable data, the monitoring frequency can be reduced to once an hour or a day. During data processing, the values ​​of the above monitoring indicators are calculated in real time. Use visualization tools (such as line charts, bar charts, dashboards, etc.) to display monitoring indicators in an intuitive way to quickly discover changing trends and abnormalities in data quality.

[0136] Record detailed operation logs at each stage of data collection, multi-algorithm processing, and optimization, including the source of the data, processing time, operator, algorithm and parameters used, data values ​​before and after processing, etc. Store logs in a secure and reliable database or file system to ensure the integrity and accessibility of logs, and establish log indexes to facilitate quick retrieval and query of specific log records. Analyze logs regularly to find possible problems or abnormal patterns, such as: frequent occurrence of a certain type of error, long processing time for a certain stage, abnormal operation of a specific operator, etc. Use data analysis tools and algorithms (such as association rule mining, cluster analysis, etc.) to conduct in-depth mining of log data to discover potential problems and trends. Set thresholds and rules to trigger alarm mechanisms when monitoring indicators exceed the normal range, significant differences are found in comparison, or abnormal patterns appear in the logs. Alarm information promptly notifies relevant personnel and provides detailed monitoring data and analysis results so that appropriate measures can be taken for improvement and optimization.

[0137] Figure 4 The present invention provides a schematic diagram of a data processing device according to an embodiment of the present invention. The present embodiment is applicable to scenarios of power system data processing. The data processing device can be implemented by software and / or hardware and integrated into a computer terminal device with application development function.

[0138] like Figure 4 As shown, the data processing device includes: an operation data acquisition module 410, a clustering module 420, an outlier correction module 430, a redundant data removal module 440 and a quality optimization module 450.

[0139] Among them, the operation data acquisition module 410 is used to obtain the operation data of the power system, wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals; the clustering module 420 is used to perform cluster analysis on the operation data to obtain a cluster analysis data set; the outlier correction module 430 is used to input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain a data set corrected by the outliers; the redundant data removal module 440 is used to perform data importance analysis on the data set corrected by the outliers through the random forest importance ranking algorithm, delete the redundant data in the data set corrected by the outliers according to the importance analysis result, and obtain a data set with redundant data removed; the quality optimization module 450 is used to input the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data; wherein the outlier detection model and the data quality optimization model are obtained by training with accurate data that meets the preset data quality requirements in the historical operation data.

[0140] The technical solution of this embodiment is to obtain the operation data of the power system, wherein the operation data includes the operation data of power generation facilities, the operation data of transmission lines, the operation data of substation equipment and the operation data of user terminals; perform cluster analysis on the operation data to obtain a cluster analysis data set; input the cluster analysis data set into the outlier detection model, determine the outliers in the cluster analysis data set, and correct the outliers to obtain a data set corrected by the outliers; perform data importance analysis on the data set corrected by the outliers by the random forest importance sorting algorithm, delete the redundant data in the data set corrected by the outliers according to the importance analysis result, and obtain a data set with redundant data removed; input the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data; wherein the outlier detection model and the data quality optimization model are obtained by training accurate data in the historical operation data that meets the preset data quality requirements. The technical solution of the embodiment of the present invention solves the problem of inaccurate operation data currently obtained, and can improve data quality through intelligent algorithms and machine learning models, ensure data reliability and availability, and provide reliable data support for power system data analysis and decision-making.

[0141] In an optional implementation, the clustering module 420 is specifically configured to:

[0142] Perform feature engineering on the operating data to obtain at least one data feature of the operating data; cluster the operating data using a hierarchical clustering algorithm based on the data features to obtain a clustering data set to be optimized; calculate the clustering statistics of the clustering data set to be optimized, determine the abnormal clustering data set based on the clustering statistics, and delete the abnormal data set to obtain a cluster analysis data set.

[0143] In an optional implementation, the outlier correction module 430 is specifically used to:

[0144] The cluster analysis data set is input into the anomaly detection model based on the isolation tree algorithm. The anomaly detection model makes an anomaly judgment on the data in the cluster analysis data set according to the characteristics of the accurate data learned in the training phase to obtain the outlier value. The correction value corresponding to the outlier value is obtained through the outlier correction algorithm, and the correction value replaces the outlier value to obtain the data set corrected by the outlier value.

[0145] In an optional implementation, the redundant data removal module 440 is specifically configured to:

[0146] The decision tree parameters of the random forest importance ranking algorithm are set; the feature importance scores of the data features of the data set corrected for outliers are calculated according to the decision tree parameters to obtain the feature importance score calculation results; the redundant data features are determined according to the feature importance score calculation results, the data corresponding to the redundant data features are determined as redundant data, and the redundant data in the data set corrected for outliers is deleted to obtain the data set with redundant data removed.

[0147] In an optional implementation, the redundant data removal module 440 is further configured to:

[0148] Determine the feature importance score of the data feature of the data set corrected for outliers in the feature importance score calculation result; compare the feature importance score with a preset feature importance threshold, and determine the data feature that is less than the preset feature importance threshold as a redundant data feature.

[0149] In an optional implementation, the quality optimization module 450 is specifically configured to:

[0150] Input the time series data in the data set with redundant data removed into the first data quality optimization network of the data quality optimization model to obtain the first model processing result of the time series data, wherein the first data quality optimization network is obtained based on the training of the long short-term memory network; input the structured data in the data set with redundant data removed into the second data quality optimization network of the data quality optimization model to obtain the second model processing result of the structured data, wherein the second data quality optimization network is obtained based on the training of the multi-layer perceptron network; perform weighted summation on the first model processing result and the second model processing result to obtain the weighted summation result; input the weighted summation result into the fully connected layer of the data quality optimization model to obtain the target operation data processing result of the operation data.

[0151] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.

[0153] like Figure 5As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0154] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0155] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0156] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0157] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0158] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RFID systems, tape drives, and data backup storage systems.

[0159] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the data processing method provided in the embodiment of the present invention, which includes:

[0160] Acquiring operation data of the power system, wherein the operation data includes operation data of power generation facilities, operation data of transmission lines, operation data of substation equipment and operation data of user terminals;

[0161] Perform cluster analysis on the running data to obtain a cluster analysis data set;

[0162] Inputting the cluster analysis data set into the outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set;

[0163] The data importance analysis is performed on the data set corrected by outliers by using the random forest importance ranking algorithm, and the redundant data in the data set corrected by outliers is deleted according to the importance analysis result to obtain a data set with redundant data removed;

[0164] Inputting the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data;

[0165] Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets the preset data quality requirements.

[0166] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the data processing method provided by any embodiment of the present invention is implemented. The method includes:

[0167] Acquiring operation data of the power system, wherein the operation data includes operation data of power generation facilities, operation data of transmission lines, operation data of substation equipment and operation data of user terminals;

[0168] Perform cluster analysis on the running data to obtain a cluster analysis data set;

[0169] Inputting the cluster analysis data set into the outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set;

[0170] The data importance analysis is performed on the data set corrected by outliers by using the random forest importance ranking algorithm, and the redundant data in the data set corrected by outliers is deleted according to the importance analysis result to obtain a data set with redundant data removed;

[0171] Inputting the data set with redundant data removed into the data quality optimization model to obtain the target operation data processing result of the operation data;

[0172] Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets the preset data quality requirements.

[0173] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0174] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0175] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0176] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, Python, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method provided in any embodiment of the present application.

[0178] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, Python, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0179] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0180] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: Acquiring operation data of the power system, wherein the operation data includes operation data of power generation facilities, operation data of transmission lines, operation data of substation equipment and operation data of user terminals; Performing cluster analysis on the operation data to obtain a cluster analysis data set; Inputting the cluster analysis data set into an outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set; Performing data importance analysis on the data set corrected by the outliers by using a random forest importance ranking algorithm, and deleting redundant data in the data set corrected by the outliers according to the importance analysis result to obtain a data set with redundant data removed; Inputting the data set from which redundant data has been removed into a data quality optimization model to obtain a target operating data processing result of the operating data; Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets preset data quality requirements.

2. The method according to claim 1, characterized in that The cluster analysis is performed on the operation data to obtain a cluster analysis data set, including: Performing feature engineering on the operation data to obtain at least one data feature of the operation data; Clustering the operating data using a hierarchical clustering algorithm according to the data features to obtain a clustering data set to be optimized; The clustering statistics of the clustering data set to be optimized are calculated, abnormal clustering data sets are determined according to the clustering statistics, and the abnormal data sets are deleted to obtain a cluster analysis data set.

3. The method according to claim 1, characterized in that The step of inputting the cluster analysis data set into an outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set includes: The cluster analysis data set is input into an anomaly detection model based on an isolation tree algorithm, and the anomaly detection model performs anomaly judgment on the data in the cluster analysis data set according to the features of the accurate data learned in the training phase to obtain anomaly values; The correction value corresponding to the outlier is obtained by an outlier correction algorithm, and the outlier is replaced by the correction value to obtain a data set corrected by the outlier.

4. The method according to claim 2, characterized in that: The method includes performing a data importance analysis on the data set corrected by the outliers by using a random forest importance ranking algorithm, and deleting redundant data in the data set corrected by the outliers according to the importance analysis result to obtain a data set with redundant data removed, including: Set the decision tree parameters for the random forest importance ranking algorithm; Calculating feature importance scores for data features of the data set corrected for outliers according to the decision tree parameters to obtain feature importance score calculation results; Redundant data features are determined according to the feature importance score calculation result, data corresponding to the redundant data features are determined as redundant data, and the redundant data in the data set corrected by the outliers is deleted to obtain a data set with redundant data removed.

5. The method according to claim 4, characterized in that The determining of redundant data features according to the feature importance score calculation result includes: Determining a feature importance score of a data feature of the data set corrected for outliers in the feature importance score calculation result; The feature importance score is numerically compared with a preset feature importance threshold, and the data feature that is less than the preset feature importance threshold is determined as a redundant data feature.

6. The method according to claim 1, characterized in that The step of inputting the data set from which redundant data has been removed into a data quality optimization model to obtain a target operation data processing result of the operation data includes: Inputting the time series data in the data set where redundant data is removed into a first data quality optimization network of a data quality optimization model to obtain a first model processing result of the time series data, wherein the first data quality optimization network is obtained based on long short-term memory network training; Inputting the structured data in the data set from which redundant data has been removed into a second data quality optimization network of the data quality optimization model to obtain a second model processing result of the structured data, wherein the second data quality optimization network is obtained based on multilayer perceptron network training; Performing weighted summation on the first model processing result and the second model processing result to obtain a weighted summation result; The weighted sum result is input into the fully connected layer of the data quality optimization model to obtain the target operation data processing result of the operation data.

7. A data processing device, characterized in that: include: An operation data acquisition module, used to acquire operation data of the power system, wherein the operation data includes operation data of power generation facilities, operation data of transmission lines, operation data of substation equipment and operation data of user terminals; A clustering module, used for performing cluster analysis on the operation data to obtain a cluster analysis data set; an outlier correction module, used for inputting the cluster analysis data set into an outlier detection model, determining outliers in the cluster analysis data set, and correcting the outliers to obtain an outlier-corrected data set; A redundant data removal module is used to perform data importance analysis on the data set corrected by outliers by using a random forest importance ranking algorithm, and delete redundant data in the data set corrected by outliers according to the importance analysis result to obtain a data set with redundant data removed; A quality optimization module, used for inputting the data set from which redundant data has been removed into a data quality optimization model to obtain a target operation data processing result of the operation data; Among them, the outlier detection model and data quality optimization model are obtained by training with accurate data in historical operation data that meets preset data quality requirements.

8. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6.

Images