Multi-source data quality improvement method and system and storage medium

Abstract

The invention relates to the field of data processing, in particular to a multi-source data quality improving method which is particularly suitable for processing and optimizing load data. Performing feature extraction on the multi-source load data to obtain load data containing a mode and a structure of the data; performing anomaly detection on the load data by applying a local anomaly factor LOF algorithm, and screening out abnormal data; correcting the detected abnormal data through a Pearson correlation analysis method; and based on the feature extraction and anomaly correction results, calculating the weight relationship between the load data features by applying an entropy weight method, and further optimizing the data correction and feature extraction process so as to improve the overall data quality and accuracy. According to the scheme, a reliable data basis is provided for big data analysis and operation state evaluation of the smart power grid, the efficiency and accuracy of data mining and analysis are effectively improved, and a solid guarantee is provided for optimal scheduling and decision support of the power grid.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multi-source data quality improvement method and system, and a storage medium, which are particularly suitable for processing and optimizing load data. Background Art

[0002] With the rapid development of smart grids, the breadth and complexity of data collection in distribution networks are increasing. These data come from multiple sources, including smart meters, sensors, and monitoring equipment, covering a variety of information such as load, voltage, and current. Due to the large number of data collection terminals, their wide distribution, and complex environments, data quality issues are becoming increasingly prominent, specifically manifested in data missing, anomalies, and redundancy. These problems not only reduce the credibility and use value of the data, but also bring challenges to the operation and optimization of smart grids.

[0003] Existing research mainly focuses on the following aspects: First, data missing processing is an important part of improving data quality. When dealing with missing data, traditional methods such as interpolation and regression analysis can fill data gaps to a certain extent, but they often face problems of low efficiency and insufficient precision in large-scale and multi-source data environments. Although interpolation is simple and easy to implement, its filling effect is limited when facing large-scale multi-source load data. Although regression analysis can provide higher accuracy, its complexity is high and it is difficult to meet the needs of real-time processing. Secondly, data anomaly detection is a key step to ensure data quality. Commonly used methods include statistical methods (such as the 3σ method) and machine learning methods (such as support vector machines (SVM) and isolation forests). However, these methods have certain limitations when dealing with high-dimensional and nonlinear data. When detecting data anomalies, statistical methods often rely on the distribution assumptions of the data, and their detection effect is poor for non-normally distributed data. Although machine learning-based methods have overcome this problem to a certain extent, their computational complexity is high and it is difficult to adapt to the real-time detection needs of large-scale data. Thirdly, data fusion and consistency are important issues in multi-source data processing. The fusion of multi-source data needs to consider the temporal and spatial consistency and logical consistency of the data. Traditional methods mainly rely on rule matching and expert systems, which are difficult to cope with the complex multi-source data fusion needs. The rule matching method relies on predefined rules, the maintenance and update requirements of the rule base are high, and it cannot cope with the dynamically changing data environment. Although the expert system can provide a certain degree of flexibility, its development and maintenance costs are high, and it is difficult to promote and apply it on a large scale. In addition, some studies have proposed a distribution network data quality improvement management platform architecture based on data dynamic governance and repair strategies, but most studies mainly focus on the design of the data quality management framework, and lack in-depth discussion of specific algorithms and implementation methods. In practical applications, these platforms often find it difficult to effectively deal with the quality issues of large-scale, multi-source, and heterogeneous data. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for improving the quality of multi-source data, comprising the following steps:

[0005] Extract features from multi-source load data to obtain load data containing data patterns and structures;

[0006] Applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data and filter out the abnormal data;

[0007] The detected abnormal data were corrected by Pearson correlation analysis;

[0008] Based on the above-mentioned feature extraction and anomaly correction results, the entropy weight method is used to calculate the weight relationship between load data features, and the data correction and feature extraction process is further optimized to improve the overall data quality and accuracy.

[0009] Furthermore, before extracting features from multi-source load data, the following steps are also included:

[0010] Clean multi-source load data by setting thresholds and rules to remove noise and obviously erroneous data points in the data;

[0011] For missing data that occurred during the collection process, interpolation and regression analysis data filling techniques were used to fill in the data gaps;

[0012] Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources;

[0013] On the basis of cleaning and filling, the multi-source data is integrated by using association analysis and rule matching technology, and the load data from different data sources are fused to obtain multi-source load data.

[0014] Furthermore, feature extraction of multi-source load data includes segmenting the load data into time series according to the periodic characteristics of the data to facilitate cluster analysis; the means algorithm divides n objects into K clusters C1…C k , each cluster has a high similarity; the optimization goal of the algorithm is to minimize the square error E of the cluster:

[0015]

[0016] in, C i The cluster center of the cluster. E is the sum of squared errors of the clusters, which indicates the quality of the clustering results. The smaller the squared error, the closer the data point is to the cluster center of the cluster to which it belongs, and thus the higher the similarity. In load data analysis, the goal is to make the load data characteristics in the same cluster more consistent for further anomaly detection and analysis. K is the number of clusters, which indicates the number of clusters divided in the K-means clustering algorithm. Each cluster represents a load pattern or feature; x is a data point, which represents a sample of load data; each data point corresponds to load data at a certain time or geographical location, such as specific measurements of current, voltage, etc. during a certain period of time. Through clustering, load data points with similar characteristics are classified into the same cluster. C i represents the i-th cluster, which represents the load data cluster divided by the clustering algorithm. The data points in each cluster have similar load characteristics. i C iThe cluster center represents the average value (i.e., the centroid) of all data points in the cluster. In the use scenario of load data, the cluster center represents the typical pattern of the load data in the cluster. i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center. In the load data scenario, this represents the difference between the load data at a specific moment and the typical load pattern of the cluster. By minimizing this difference, we can ensure that the data points in the cluster are as close to the cluster center as possible, thereby enhancing the similarity within the cluster. Through this method, the characteristic patterns of the load data can be extracted for subsequent anomaly detection and data correction operations.

[0017] Furthermore, applying the local outlier factor LOF algorithm to perform anomaly detection on the load data includes independently detecting different categories of load data to improve the accuracy of anomaly detection: the embodiment of independent detection is to process and analyze each category of data separately, rather than uniformly processing all data sets.

[0018] Here, by calculating the neighborhood distance of each data point, it means that each data point p is an independent detection object. This detection is based on the neighboring data points of each point, and the neighborhoods of data points of different categories are calculated separately, which is an indirect independent detection.

[0019] In this part, the calculation of local reachable density and local anomaly factor is actually performed independently for each data point p. These calculations are analyzed based on the neighborhood of each point to determine whether the point is an anomaly. This local analysis method shows that anomaly detection is performed in the local space of each data point, rather than a unified detection on the entire data set. Specifically, it includes the following four aspects of calculation:

[0020] (1) kth distance neighborhood: First, given a data set D, for any point p in the set, calculate the Euclidean distances between other points and point p and sort them from small to large. The kth point is recorded as the kth distance of the point, and all points within the kth distance are the kth distance neighborhood, recorded as N k (p), the neighborhood is used to define the local range of point p. The Euclidean distance can be expressed as:

[0021]

[0022] Where p and q represent two data points in the load data set in n-dimensional space, and p and q are load data recorded by different acquisition devices or at different time points. dist(p,q) is the Euclidean distance between two points. This distance measures the difference between data points in n-dimensional feature space. In the application of power load data, the Euclidean distance represents the degree of change of load between two time points. By calculating the distance, the data point closest to a load data point can be found and the neighborhood can be constructed.

[0023] (2) Reachable distance: After the kth distance neighborhood is determined, the reachable distance between point p and each point in its neighborhood is calculated to enhance the robustness of density calculation. The reachable distance between point p and point o in space is defined as the maximum value between the kth distance of point o and dist(p,o), denoted as reach-dist(p,o). The purpose of this step is to eliminate the influence of extreme distance points on density calculation and ensure more accurate density calculation;

[0024] ⑶ Local reachability density: Based on the reachable distance, calculate the local reachability density lrd of the data point p k (p) is the inverse of the average reachable distance between its neighboring points. The local reachable density reflects the density of point p in its local neighborhood, providing a basis for the subsequent calculation of the anomaly factor. Points with lower density are more likely to be abnormal:

[0025]

[0026] Among them, reach-dist(p,o) represents the reachable distance between point p and point o, which is defined as the maximum value of the kth distance of point o and dist(p,o); this parameter is used to measure the connectivity between load data point p and adjacent data point o. In application scenarios, this helps to distinguish the density of data points and identify whether abnormal data points are too "isolated" from surrounding data points. k (p) represents the local reachable density of data point p, which is the inverse of its average reachable distance in the neighborhood. This parameter reflects the density around data point p. In the application of load data, the local density of the load data point indicates the load change of the data point in the time period or area where it is located. If the density around the load data point is too low, it indicates that the point is an abnormal data point.

[0027] (4) Local outlier factor: Finally, the local outlier factor (LOF) of point p is calculated based on the ratio of the local reachability density of points p to other points in its neighborhood. The local outlier factor is the average of the ratios of the local reachability density of points in the neighborhood of point p to the local reachability density of point p, denoted as LOF k (p).

[0028]

[0029] Among them, LOF k (p) is the local outlier factor, which represents the ratio of the local reachability density of point p to the points in its neighborhood. k When the (p) value is around 1, it means that the data density of point p is equivalent to the data density of the points in its neighborhood. k If (p) is much greater than 1 or much less than 1, it means that point p is far away from other points and is an abnormal point. The larger the value of the local anomaly factor, the more isolated the point is relative to other points in the neighborhood and is a potential abnormal data point. The abnormality of point p is finally determined by the LOF value. In load data, this can help identify abnormal load points that are significantly different from the surrounding data, thereby improving data quality.

[0030] Through the above step-by-step calculation based on neighborhood, density and local anomaly factors, effective detection of abnormal points in load data is achieved, providing reliable support for improving data quality.

[0031] Furthermore, the detected abnormal data is corrected by Pearson correlation analysis. The data correction includes performing stepwise regression correction on the detected abnormal data based on multidimensional data characteristics to improve the accuracy of data correction:

[0032] The calculation formula of Pearson correlation coefficient is as follows:

[0033]

[0034] Among them, x1 and x2 are two n-dimensional data variables in the load data set, representing load data with different characteristics or different time periods. In the application scenario of power data, x1 and x2 are the measured values ​​of voltage and current, respectively, which are used to analyze the correlation between them. are the standard deviations of x1 and x2, respectively, indicating the fluctuation range of the load data. The larger the standard deviation of the load data, the more drastic the fluctuation. Cov(x1,x2) is the covariance between the two. The covariance calculation formula is as follows:

[0035]

[0036] The Pearson correlation coefficient is used to evaluate the correlation between two data. When the correlation coefficient is greater than a certain value, the two data are considered to have a strong correlation. In the scenario of load data, covariance can help determine the change relationship between different load variables (such as voltage and current).

[0037] Furthermore, the entropy weight method is used to calculate the weight relationship between load data features. The weight calculation includes dynamically adjusting the weight parameters to adapt to the changes in data features in different time periods and different environments. Specifically, the entropy between each variable is first calculated according to formula ⑺.

[0038]

[0039] E j The entropy of a load data feature reflects the amount of information about the feature. In a load data scenario, different load features (such as voltage, current, and power) may have different importance in different time periods. The larger the entropy value, the lower the amount of information about the feature.

[0040] The weights of each variable can also be obtained by formula ⑻.

[0041]

[0042] Among them, ω j is the weight of feature j, indicating the importance of this feature in data correction. In load data optimization, different load features have different impacts on the overall data quality, and weights are used to adjust the relative importance of different features. The larger the weight, the higher the influence of this feature in the data correction and optimization process. By dynamically adjusting these weights, the data correction strategy can be optimized in different environments and time periods, thereby improving the overall data quality.

[0043] According to another aspect of the present invention, a multi-source data quality improvement system is provided, comprising:

[0044] A feature extraction unit extracts features from multi-source load data to obtain load data containing patterns and structures of the data;

[0045] A detection unit, applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data to filter out the abnormal data;

[0046] A correction unit, which corrects the detected abnormal data by using the Pearson correlation analysis method;

[0047] The optimization unit, based on the aforementioned feature extraction and anomaly correction results, applies the entropy weight method to calculate the weight relationship between the load data features, and further optimizes the data correction and feature extraction process to improve the overall data quality and accuracy.

[0048] Furthermore, a data preprocessing unit is provided on the feature extraction unit, for processing the multi-source load data as follows:

[0049] Clean multi-source load data by setting thresholds and rules to remove noise and obviously erroneous data points in the data;

[0050] For missing data that occurred during the collection process, interpolation and regression analysis data filling techniques were used to fill in the data gaps;

[0051] Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources;

[0052] On the basis of cleaning and filling, the multi-source data is integrated by using association analysis and rule matching technology, and the load data from different data sources are fused to obtain multi-source load data.

[0053] Furthermore, the feature extraction unit extracts features from multi-source load data in the following manner:

[0054] According to the periodic characteristics of the data, the load data is segmented into time series; the means algorithm divides n objects into K clusters C1…C k , the optimization goal of the algorithm is to minimize the square error E of the cluster:

[0055]

[0056] in, C i The cluster center; E is the sum of squared errors of clusters, indicating the quality of clustering results; K is the number of clusters, each cluster represents a load mode or feature; x is a data point, representing a sample of load data; each data point corresponds to the load data at a certain time or a certain geographical location; C i represents the i-th load data cluster; u i C i The cluster center represents the average value of all data points in the cluster; ‖xu i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center; through this method, the characteristic pattern of load data is extracted for subsequent anomaly detection and data correction operations.

[0057] The present invention also provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0058] In summary, in order to cope with the multi-source and complexity of load data in smart grids, a multi-source data quality improvement method based on artificial intelligence is proposed. Through system preprocessing, feature extraction, anomaly detection, data correction and weight calculation, the integrity, accuracy and consistency of the data are effectively improved. This not only provides a reliable data foundation for big data analysis and operation status evaluation of smart grids, but also effectively improves the efficiency and accuracy of data mining and analysis, providing a solid guarantee for the optimization dispatch and decision support of power grids. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the process of the present invention;

[0060] Figure 2 It is a schematic diagram of the abnormal data detection process. DETAILED DESCRIPTION

[0061] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0062] refer to Figure 1-2 The present invention provides a multi-source data quality improvement method based on artificial intelligence. By performing multi-step processing on load data, the integrity, accuracy and consistency of the data are improved, providing a reliable data foundation for big data analysis and decision support of smart grids.

[0063] The technical solution of the present invention to solve the above technical problems is as follows:

[0064] The present invention provides a method for improving multi-source data quality based on artificial intelligence, and the method for improving multi-source data quality based on artificial intelligence includes:

[0065] Extract features from multi-source load data to obtain load data containing data patterns and structures;

[0066] Applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data and filter out the abnormal data;

[0067] The detected abnormal data were corrected by Pearson correlation analysis;

[0068] Based on the above-mentioned feature extraction and anomaly correction results, the entropy weight method is used to calculate the weight relationship between load data features, and the data correction and feature extraction process is further optimized to improve the overall data quality and accuracy.

[0069] Optionally, the multi-source load data is preprocessed before feature extraction. Data preprocessing is the first step in the multi-source data quality improvement method based on artificial intelligence, aiming to provide a high-quality data foundation for subsequent data processing steps. Data preprocessing includes the following aspects: data cleaning, missing data filling, standardization processing and data fusion. The specific plan is:

[0070] By preliminarily cleaning multi-source load data, noise and obviously erroneous data points in the data are removed. This step automatically detects and removes abnormal data by setting reasonable thresholds and rules to ensure the basic quality of the data.

[0071] For missing data that occurs during the collection process, interpolation, regression analysis and other data filling techniques are used to fill in the data gaps. During the filling process, the most suitable method is selected to predict and fill in the missing values ​​based on the time series characteristics and correlation of the data to ensure the continuity and integrity of the data.

[0072] Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources. Use normalization or standard deviation methods to convert the data to a unified scale range to facilitate subsequent feature extraction and analysis.

[0073] On the basis of cleaning and filling, load data from different data sources are fused to ensure data consistency and logical coherence. Using association analysis and rule matching technology, multi-source data is integrated to provide a unified and reliable data set for subsequent steps.

[0074] The data collected by the distribution network system has a strong periodicity. The K-means clustering method in data mining is used to analyze the multi-source load data and obtain the data characteristics of the collected multi-source load data.

[0075] The main idea of ​​the means algorithm is to divide n objects into K clusters C1…C k , each cluster has a high similarity. The optimization goal of the algorithm is to minimize the square error E of the clusters.

[0076]

[0077] in, C i E is the sum of squared errors of clusters, which indicates the quality of clustering results; K is the number of clusters, each cluster represents a load mode or feature; x is a data point, which represents a sample of load data; each data point corresponds to the load data at a certain time or geographical location; C i represents the i-th load data cluster; u i C i The cluster center represents the average value of all data points in the cluster; ‖xu i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center. Through this method, the characteristic pattern of load data can be extracted for subsequent anomaly detection and data correction operations.

[0078] The algorithm flow is as follows.

[0079] 1) Select K initial cluster centers from the data set D.

[0080] 2) Calculate the Euclidean distance between each data point in the set and the cluster center point (Formula 2), select the nearest cluster center point and include it in the class.

[0081] 3) After calculating all data points, recalculate the cluster center for each cluster.

[0082] 4) Determine whether the cluster center has changed. If so, return to step ⑵; if not, output the result.

[0083] Optionally, a local outlier factor (LOF) algorithm is applied to perform anomaly detection on the load data to filter out the abnormal data, wherein the core concepts of the LOF algorithm are as follows.

[0084] (1) kth distance neighborhood: Given a data set D, for any point p in the set, calculate the Euclidean distances between other points and point p and sort them from small to large. The kth point is recorded as the kth distance of the point, and all points within the kth distance are the kth distance neighborhood, recorded as N k (p), the neighborhood is used to define the local range of point p. The Euclidean distance can be expressed as:

[0085]

[0086] Where p and q are two data points in n-dimensional space, and dist(p,q) is the Euclidean distance between the two points. p and q are load data recorded by different acquisition devices or at different time points. dist(p,q) is the Euclidean distance between two points. This distance measures the difference between data points in n-dimensional feature space. In the application of power load data, the Euclidean distance indicates the degree of change in load between two time points. By calculating the distance, the data point closest to a load data point can be found and the neighborhood can be constructed.

[0087] (2) Reachable distance: After the kth distance neighborhood is determined, the reachable distance between point p and each point in its neighborhood is calculated to enhance the robustness of density calculation. The reachable distance between point p and point o in space is defined as the maximum value between the kth distance of point o and dist(p,o), denoted as reach-dist(p,o). The purpose of this step is to eliminate the influence of extreme distance points on density calculation and ensure more accurate density calculation.

[0088] ⑶ Local reachability density: Based on the reachable distance, calculate the local reachability density lrd of the data point p k (p) is the inverse of the average reachable distance between its neighboring points. The local reachable density reflects the density of point p in its local neighborhood, providing a basis for the subsequent calculation of the anomaly factor. Points with lower density are more likely to be abnormal:

[0089]

[0090] Where reach-dist(p,o) represents the reachable distance between point p and point o, which is defined as the maximum value of the kth distance of point o and dist(p,o); Lrd k (p) represents the local reachability density of data point p, which is the inverse of its average reachability distance in the neighborhood.

[0091] (4) Local outlier factor: Finally, the local outlier factor (LOF) of point p is calculated based on the ratio of the local reachability density of points p to other points in its neighborhood. The local outlier factor is the average of the ratios of the local reachability density of points in the neighborhood of point p to the local reachability density of point p, denoted as LOF k (p).

[0092]

[0093] According to the definition of local outlier factor, LOF k When the (p) value is around 1, it means that the data density of point p is equivalent to the data density of the points in its neighborhood. k If (p) is much larger than 1 or much smaller than 1, it means that point p is far away from other points and is an outlier.

[0094] Through the above-mentioned step-by-step calculation based on neighborhood, density and local anomaly factors, effective detection of abnormal points in load data is achieved, providing reliable support for improving data quality.

[0095] The data anomaly detection process based on K-means clustering and LOF algorithm is as follows Figure 2 shown.

[0096] By combining the two algorithms, we can extract the characteristics of the data set and filter out the abnormal data in the data set. Then, we can correct and fill the abnormal data through the multi-dimensional correlation of the data.

[0097] Here, the correlation analysis algorithm in this paper uses the Pearson correlation coefficient method to measure the correlation between data from different sources:

[0098] The calculation formula of Pearson correlation coefficient is as follows:

[0099]

[0100] Among them, x1 and x2 are n-dimensional data variables, are the standard deviations of x1 and x2, and the covariance between Cov(x1,x2). The formula for calculating covariance is as follows:

[0101]

[0102] The Pearson correlation coefficient is used to evaluate the correlation between two data. When the correlation coefficient is greater than a certain value, the two data are considered to have a strong correlation.

[0103] Optionally, the entropy weight method is used to calculate the weight relationship between the load data features, and the data correction process is optimized based on the calculation results.

[0104] (1) Determine weights using entropy weight method

[0105] Here, the entropy weight method determines the weight by the amount of information contained in the indicator, which can effectively use the data and eliminate the influence of subjective factors. First, the entropy between each variable is calculated according to formula ⑺.

[0106]

[0107] The weights of each variable can also be obtained by formula ⑻.

[0108]

[0109] Weight ω j It reflects the amount of information of the indicator and can quantify the impact of the indicator on the result. The larger the weight, the more influential the feature is in the data correction and optimization process. By dynamically adjusting these weights, data correction strategies can be optimized in different environments and time periods, thereby improving overall data quality.

[0110] (2) Distribution network data correction method based on multidimensional data characteristics

[0111] The distribution network data collection cycle is selected as T = 24h. The collected data set C = {c1, c2, ..., c M} contains M attributes (such as current, voltage, etc.). i The abnormal data set, the data correction method based on multidimensional data characteristics is as follows.

[0112] (a) Use K-means clustering method to extract features of historical data of various attributes in set C, and use LOF anomaly detection algorithm to screen out abnormal data sets and normal data sets.

[0113] (b) Select the normal data set of each attribute and calculate the Pearson coefficient between each attribute through association analysis. i , select m attributes that are strongly correlated with it to establish the Pearson correlation coefficient matrix Σ between each attribute, and use the entropy weight method to determine their respective weights w j ,j=1,2,…,m;

[0114] (c) Attribute c i The abnormal data set is denoted as For one of the abnormal data, record it as Select c from m strongly associated attributes i A strong correlation property c j , in its data set Find and The historical data points at the same time are recorded as The cluster center of the class is recorded as

[0115] (d) After selecting all m strongly correlated attributes, the optimal correction result of the original data is obtained by weighted summation:

[0116] (e) Select X i For the next abnormal data in the , repeat steps ⑶ and ⑷ until all abnormal data are repaired.

[0117] According to another aspect of the present invention, a multi-source data quality improvement system is provided, comprising:

[0118] A feature extraction unit extracts features from multi-source load data to obtain load data containing patterns and structures of the data;

[0119] A detection unit, applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data to filter out the abnormal data;

[0120] A correction unit, which corrects the detected abnormal data by using the Pearson correlation analysis method;

[0121] The optimization unit, based on the aforementioned feature extraction and anomaly correction results, applies the entropy weight method to calculate the weight relationship between the load data features, and further optimizes the data correction and feature extraction process to improve the overall data quality and accuracy.

[0122] As an improvement of the solution, a data preprocessing unit is provided on the feature extraction unit to process the multi-source load data as follows:

[0123] Clean multi-source load data by setting thresholds and rules to remove noise and obviously erroneous data points in the data;

[0124] For missing data that occurred during the collection process, interpolation and regression analysis data filling techniques were used to fill in the data gaps;

[0125] Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources;

[0126] On the basis of cleaning and filling, the multi-source data is integrated by using association analysis and rule matching technology, and the load data from different data sources are fused to obtain multi-source load data.

[0127] As an improvement of the solution, the feature extraction unit extracts features from multi-source load data in the following manner:

[0128] According to the periodic characteristics of the data, the load data is segmented into time series; the means algorithm divides n objects into K clusters C1…C k , the optimization goal of the algorithm is to minimize the square error E of the cluster:

[0129]

[0130] in, C i The cluster center; E is the sum of squared errors of clusters, indicating the quality of clustering results; K is the number of clusters, each cluster represents a load mode or feature; x is a data point, representing a sample of load data; each data point corresponds to the load data at a certain time or a certain geographical location; C i represents the i-th load data cluster; u i C i The cluster center represents the average value of all data points in the cluster; ‖xu i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center; through this method, the characteristic pattern of load data is extracted for subsequent anomaly detection and data correction operations.

[0131] The present invention also provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for improving the quality of multi-source data, characterized by: The following steps are included: Extract features from multi-source load data to obtain load data containing data patterns and structures; Applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data and filter out the abnormal data; The detected abnormal data were corrected by Pearson correlation analysis; Based on the above-mentioned feature extraction and anomaly correction results, the entropy weight method is used to calculate the weight relationship between load data features, and the data correction and feature extraction process is further optimized to improve the overall data quality and accuracy.

2. A method for improving multi-source data quality as claimed in claim 1, characterized in that: Before extracting features from multi-source load data, it also includes: Clean multi-source load data by setting thresholds and rules to remove noise and obviously erroneous data points in the data; For missing data that occurred during the collection process, interpolation and regression analysis data filling techniques were used to fill in the data gaps; Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources; On the basis of cleaning and filling, the multi-source data is integrated by using association analysis and rule matching technology, and the load data from different data sources are fused to obtain multi-source load data.

3. The method for improving multi-source data quality according to claim 1, characterized in that: The method for feature extraction of multi-source load data is as follows: According to the periodic characteristics of the data, the load data is segmented into time series; the means algorithm divides n objects into K clusters C1…C k , the optimization goal of the algorithm is to minimize the square error E of the cluster: in, C i The cluster center; E is the sum of squared errors of clusters, indicating the quality of clustering results; K is the number of clusters, each cluster represents a load mode or feature; x is a data point, representing a sample of load data; each data point corresponds to the load data at a certain time or a certain geographical location; C i represents the i-th load data cluster; u i C i The cluster center represents the average value of all data points in the cluster; ‖xu i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center; through this method, the characteristic pattern of load data is extracted for subsequent anomaly detection and data correction operations.

4. The method for improving multi-source data quality according to claim 1, characterized in that: Applying the local outlier factor LOF algorithm to perform anomaly detection on the load data point includes: First, given a data set D, for any point p in the set, calculate the Euclidean distances between other points and point p and sort them from small to large. The kth point is recorded as the kth distance of the point, and all points within the kth distance are the kth distance neighborhood, recorded as N k (p), the neighborhood is used to define the local range of point p; the Euclidean distance can be expressed as: Where p and q represent two data points in the load data set in n-dimensional space, respectively. p and q are load data recorded by different acquisition devices or at different time points. dist(p,q) is the Euclidean distance between the two points. Secondly, after the kth distance neighborhood is determined, the reachable distance between point p and each point in its neighborhood is calculated to enhance the robustness of density calculation; the reachable distance between point p and point o in space is defined as the maximum value between the kth distance of point o and dist(p,o), recorded as reach-dist(p,o); Again, based on the reachable distance, calculate the local reachable density lrd of the data point p k (p) is the reciprocal of the average reachable distance between its neighboring points: Where reach-dist(p,o) represents the reachable distance between point p and point o, which is defined as the maximum value of the kth distance of point o and dist(p,o); Lrd k (p) represents the local reachability density of data point p, which is the inverse of its average reachability distance in the neighborhood; Finally, the local outlier factor (LOF) of point p is calculated based on the ratio of the local reachable density of points in its neighborhood to the local reachable density of point p. The local outlier factor is the average of the ratios of the local reachable density of points in the neighborhood of point p to the local reachable density of point p, denoted as LOF k (p): Among them, LOF k (p) is the local outlier factor, which represents the ratio of the local reachability density of point p to the points in its neighborhood; LOF k When the (p) value is around 1, it means that the data density of point p is equivalent to the data density of the points in its neighborhood. k If (p) is much larger than 1 or much smaller than 1, it means that point p is far away from other points and is an abnormal point. The abnormality of point p is finally determined through the LOF value, thus achieving effective detection of abnormal points in load data.

5. The method for improving multi-source data quality according to claim 1, characterized in that: The abnormal data detected is corrected by Pearson correlation analysis method. The data correction includes performing stepwise regression correction on the detected abnormal data based on multidimensional data characteristics. The calculation formula of Pearson correlation coefficient is as follows: Among them, x1 and x2 are two n-dimensional data variables in the load data set, representing load data with different characteristics or different time periods. They are the standard deviation of x1 and x2, and the covariance between Cov(x1,x2). The covariance calculation formula is as follows:

6. A method for improving multi-source data quality as claimed in claim 1, characterized in that: The entropy weight method is used to calculate the weight relationship between load data features. The weight calculation includes dynamically adjusting the weight parameters to adapt to the changes in data features in different time periods and under different environments. Specifically, the entropy between each variable is calculated according to formula ⑺: E j The entropy of a load data feature reflects the amount of information about the feature; The weights of each variable are also obtained by formula ⑻; Among them, ω j is the weight of feature j, indicating the importance of the feature in data correction. The larger the weight, the higher the influence of the feature in the data correction and optimization process. By dynamically adjusting these weight relationships, the data correction strategy can be optimized in different environments and time periods.

7. A multi-source data quality improvement system, characterized in that: include, A feature extraction unit extracts features from multi-source load data to obtain load data containing patterns and structures of the data; A detection unit, applying a local outlier factor (LOF) algorithm to perform anomaly detection on the load data to filter out the abnormal data; A correction unit, which corrects the detected abnormal data by using the Pearson correlation analysis method; The optimization unit, based on the above-mentioned feature extraction and anomaly correction results, applies the entropy weight method to calculate the weight relationship between the load data features, and further optimizes the data correction and feature extraction process.

8. A multi-source data quality improvement system as claimed in claim 7, characterized in that: The feature extraction unit includes a data preprocessing unit, which is used to process the multi-source load data as follows: Clean multi-source load data by setting thresholds and rules to remove noise and obviously erroneous data points in the data; For missing data that occurred during the collection process, interpolation and regression analysis data filling techniques were used to fill in the data gaps; Standardize the load data from different data sources to eliminate the dimensional differences caused by the diversity of data sources; On the basis of cleaning and filling, the multi-source data is integrated by using association analysis and rule matching technology, and the load data from different data sources are fused to obtain multi-source load data.

9. A multi-source data quality improvement system as claimed in claim 7, characterized in that: The feature extraction unit extracts features from multi-source load data in the following way: According to the periodic characteristics of the data, the load data is segmented into time series; the means algorithm divides n objects into K clusters C1…C k , the optimization goal of the algorithm is to minimize the square error E of the cluster: in, C i The cluster center; E is the sum of squared errors of clusters, indicating the quality of clustering results; K is the number of clusters, each cluster represents a load mode or feature; x is a data point, representing a sample of load data; each data point corresponds to the load data at a certain time or a certain geographical location; C i represents the i-th load data cluster; u i C i The cluster center represents the average value of all data points in the cluster; ‖xu i ‖ 2 Represents the data point x and its cluster C i Cluster center u i The square of the Euclidean distance represents the deviation between each data point and its cluster center; through this method, the characteristic pattern of load data is extracted for subsequent anomaly detection and data correction operations.

10. A computer storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images