Abnormal data screening method and system based on PCA (Principal Component Analysis) and isolated forest algorithm

Abstract

The invention discloses an abnormal data screening method and system based on PCA and an isolated forest algorithm, and the method comprises the steps: 1, collecting the operation data of a distributed photovoltaic power station, and carrying out the preprocessing; 2, extracting features related to photovoltaic generating capacity anomaly detection from the preprocessed data to obtain feature data; 3, determining the principal component of PCA in combination with the information entropy of each feature, and carrying out the dimension reduction of the extracted feature data; 4, performing anomaly detection on the data after dimension reduction by combining a pre-trained isolated forest model and an auto-encoder reconstruction model to obtain an anomaly score of each piece of data; and 5, determining abnormal data according to the abnormal score and a preset first threshold value. According to the invention, the operation data of the high-dimensional distributed photovoltaic power station can be processed, and the nonlinear characteristics and dynamic changes of the data are fully captured, so that the efficiency and accuracy of photovoltaic power generation abnormal data screening in a complex data mode are improved.

Description

Technical Field

[0001] The present invention relates to the data anomaly detection technology in the field of photovoltaic power generation in the power industry, and particularly to an abnormal data screening method and system based on PCA and isolation forest algorithms. Background Art

[0002] In a distributed photovoltaic power generation system, accurately identifying and processing abnormal data is crucial for ensuring power generation efficiency and system stability. Traditional anomaly detection methods rely on rules and thresholds and are difficult to adapt to complex and changing data patterns.

[0003] In a distributed photovoltaic power generation system, accurately identifying and processing abnormal data is crucial for ensuring power generation efficiency and system stability. Traditional anomaly detection methods rely on rules and thresholds and are difficult to adapt to complex and changing data patterns.

[0004] The prior art document (CN114358160A) discloses a data anomaly detection method in a power system, which includes collecting the original data of users from a power grid server, performing data preprocessing, and then inputting the preprocessed data into a trained random forest for inference to obtain a decision result on whether the data is abnormal.

[0005] However, the deficiency of the prior art document is that it relies on the isolation forest algorithm for anomaly detection and only evaluates the anomaly degree of data points through the path length. Although this method can quickly identify abnormal points, it may not be accurate enough when dealing with complex data patterns, especially limited in capturing high-dimensional data and non-linear relationships. Summary of the Invention

[0006] To solve the deficiencies in the prior art, the present invention provides an abnormal data screening method and system based on PCA and isolation forest algorithms. This method improves the accuracy and efficiency of complex data anomaly detection by introducing an innovative principal component selection method and an anomaly score calculation formula.

[0007] Although the existing PCA and isolation forest algorithms perform well in data dimensionality reduction and anomaly detection, they still have limitations when dealing with photovoltaic power generation data. For example, PCA usually relies on the variance contribution rate when selecting principal components, and the isolation forest uses a fixed path length formula when calculating the anomaly score. These methods cannot fully capture the non-linear characteristics and dynamic changes of complex photovoltaic power generation data.

[0008] The present invention adopts the following technical solutions.

[0009] The first aspect of the present invention provides an abnormal data screening method based on PCA and isolation forest algorithms, which includes the following steps:

[0010] Step 1: Collect the operation data of the distributed photovoltaic power station and perform preprocessing;

[0011] Step 2: Extract the features related to the abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data;

[0012] Step 3: Determine the principal components of PCA by combining the information entropy of each feature, so as to reduce the dimension of the extracted feature data;

[0013] Step 4: Combine the pre-trained isolation forest model and the autoencoder reconstruction model to perform abnormal detection on the data after dimension reduction, and obtain the abnormal scores of each data;

[0014] Step 5: Determine the abnormal data according to the abnormal scores and the preset first threshold.

[0015] Optionally, the preprocessing includes outlier processing, and the outlier processing includes:

[0016] Classify the operation data of the distributed photovoltaic power station according to various categories to obtain multiple groups of operation data;

[0017] Sort each group of operation data from small to large, determine the first position value of the outlier threshold range as the operation data ranked at the 1 / 4 position, and determine the second position value of the outlier threshold range as the operation data ranked at the 3 / 4 position;

[0018] Determine the first boundary value and the second boundary value of the outlier threshold range according to the first position value and the second position value, and the outlier threshold range is less than the first boundary value or greater than the second boundary value;

[0019] Determine the outlier processing operation according to the degree of deviation of the outlier from the first boundary value or the second boundary value of the outlier threshold range.

[0020] Optionally, in Step 2, the features extracted related to the abnormal detection of photovoltaic power generation include at least one of the following:

[0021] Statistical features of power generation, time-domain features of power generation, model-based features of power generation, daily power generation change rate, and proportion of power consumption during peak-valley electricity price periods.

[0022] Optionally, in Step 3, determining the principal components of PCA by combining the information entropy of each feature, so as to reduce the dimension of the extracted feature data, includes:

[0023] Standardize the extracted feature data and calculate the covariance matrix of the standardized data;

[0024] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues and corresponding eigenvectors;

[0025] Calculate the information entropy of each feature in the standardized data according to the following formula:

[0026]

[0027] where H(X i ) is the information entropy of the i-th feature, and p(x ij ) is the probability distribution of the j-th data corresponding to the feature X i , and n i is the number of data corresponding to the feature X i ;

[0028] Perform a weighted sum of the feature values, information entropy, and their corresponding weights to determine the principal components;

[0029] Project the standardized data into the feature space composed of the principal components to obtain the dimensionality-reduced data.

[0030] Optionally, in step 4, combine the pre-trained isolation forest model and the autoencoder reconstruction model to perform anomaly detection on the dimensionality-reduced data to obtain the anomaly score of each data, including:

[0031] Input the dimensionality-reduced data into the pre-trained isolation forest model to obtain the path length corresponding to each isolation tree, and calculate the first anomaly score according to the path length;

[0032] Input the dimensionality-reduced data into the pre-trained autoencoder reconstruction model to obtain the reconstruction error, and calculate the second anomaly score according to the reconstruction error;

[0033] Calculate the anomaly score of each dimensionality-reduced data according to the first anomaly score and the second anomaly score.

[0034] Optionally, calculate the anomaly score of each dimensionality-reduced data according to the following formula based on the first anomaly score and the second anomaly score:

[0035] s combined (x) = α * s AE (x) + (1 - α) * s IF (x),

[0036] where α is the weight parameter, s AE (x) is the first anomaly score of the data x, and s IF (x) is the second anomaly score of the data x.

[0037] Optionally, calculate the first anomaly score according to the path length according to the following formula:

[0038]

[0039] where s IFThe first anomaly score of data x is (x), n is the number of data after PCA dimensionality reduction, h(x) is the height of the isolation tree where data x is located, E(h(x)) is the average path length of data x in all isolation trees, c(n) is the normalization factor of the expected path length, and H(i) is the harmonic number.

[0040] Optionally, the second anomaly score is calculated according to the reconstruction error by the following formula:

[0041] s AE (x) = ||x - x^||,

[0042] where s AE (x) is the second anomaly score, x is the data after PCA dimensionality reduction, x^ is the data obtained through the autoencoder reconstruction model, and ||x - x^|| is the reconstruction error.

[0043] The second aspect of the present invention provides an abnormal data screening system, and the system includes:

[0044] A data preprocessing module, which is used to collect the operation data of a distributed photovoltaic power station and perform preprocessing;

[0045] A feature extraction module, which is used to extract features related to abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data;

[0046] A dimensionality reduction module, which is used to determine the principal components of PCA in combination with information entropy, so as to perform dimensionality reduction on the extracted feature data;

[0047] An anomaly detection module, which is used to perform anomaly detection on the dimensionality-reduced data in combination with a pre-trained isolation forest model and an autoencoder reconstruction model, obtain the anomaly score of each data, and judge the abnormal data according to the anomaly score and a preset first threshold.

[0048] The third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, the above-mentioned abnormal data screening method based on PCA and isolation forest algorithm is implemented.

[0049] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned abnormal data screening method based on PCA and isolation forest algorithm is implemented.

[0050] Compared with the prior art, the beneficial effects of the present invention at least include:

[0051] The present invention determines the principal components of PCA by combining information entropy, thereby performing dimensionality reduction on the operation data of a distributed photovoltaic power station after feature extraction. It can process high-dimensional data and fully capture the non-linear features and dynamic changes of the data, thereby improving the efficiency and accuracy of screening abnormal data of photovoltaic power generation.

[0052] In the data preprocessing stage of the present invention, outlier processing is adopted, and outlier screening is preliminarily carried out before the dimensionality reduction operation, reducing the data volume for subsequent anomaly detection. In the feature extraction stage, the present invention comprehensively utilizes the statistical features of power generation, the time-domain features of power generation, the model-based features of power generation, the change rate of daily power generation, and the proportion of power consumption during peak-valley electricity price periods to comprehensively capture the multi-dimensional information of the data.

[0053] The present invention combines to calculate the anomaly score through multiple dimensions, which can more comprehensively evaluate the anomaly degree of data points, making the anomaly detection more sensitive, comprehensive and accurate, and thus better adapting to complex data patterns and significantly improving the accuracy and robustness of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0055] Figure 1 It is a schematic flowchart diagram of an abnormal data screening method based on PCA and isolation forest algorithm provided by an embodiment of the present invention;

[0056] Figure 2 It is a schematic flowchart diagram of another abnormal data screening method based on PCA and isolation forest algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions of the present invention in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the spirit of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0058] Combined with Figure 1 and Figure 2As shown in the figure, Embodiment 1 of the present invention provides an abnormal data screening method based on PCA (Principal Component Analysis) and the Isolation Forest algorithm, including the following steps:

[0059] Step 1: Collect the operation data of the distributed photovoltaic power station and perform preprocessing.

[0060] It can be understood that the operation data of the distributed photovoltaic power station includes but is not limited to voltage, current, and power.

[0061] The preprocessing in Step 1 includes handling missing values. Specifically, handling missing values includes: filling the missing values using the linear interpolation method. For time series data, linear interpolation is a simple and effective interpolation method, which assumes that the data change between two known data points is linear. The specific formula is:

[0062]

[0063] where (x1, y1) and (x2, y2) are known data points, x is the point to be interpolated, and y is the interpolation result.

[0064] Optionally, in Step 1, the preprocessing further includes outlier handling. Specifically, outlier handling includes:

[0065] Classify the operation data of the distributed photovoltaic power station according to various categories to obtain multiple groups of operation data;

[0066] Sort each group of operation data from small to large, determine the first position value of the outlier threshold range as the operation data at the 1 / 4 position, and determine the second position value of the outlier threshold range as the operation data at the 3 / 4 position.

[0067] Calculate the first boundary value and the second boundary value of the outlier threshold range based on the first position value and the second position value. The outlier threshold range is less than the first boundary value or greater than the second boundary value;

[0068] Determine the outlier handling operation according to the degree of deviation of the outlier from the first boundary value or the second boundary value of the outlier threshold range.

[0069] In this embodiment, the categories of operation data can be current, voltage, and power. For example, classify the obtained current and voltage data to obtain current - type operation data and voltage - type operation data. Taking the current - type operation data as an example, sort the current - type operation data from small to large, find the values at the 25% and 75% positions, which are the first position value Q1 and the second position value Q2 respectively. Calculate the first boundary value low and the second boundary value high of the outlier threshold range according to the first position value and the second position value as follows:

[0070] low = Q1 - 1.5 * IQR

[0071] high = Q2 + 1.5 * IQR

[0072] IQR = Q2 - Q1

[0073] The outlier threshold range is less than the first boundary value low or greater than the second boundary value high. The running data greater than high or less than low is determined as an outlier. And the outlier is marked. Then, for the marked outlier, according to the deviation degree of its characteristic value from the boundary of the outlier threshold range, operations such as retention, correction, or deletion are selected. In this way, preliminary outlier processing of the running data can improve the speed and accuracy of subsequent high-dimensional complex data processing. By selecting operations such as retention, correction, or deletion according to the deviation degree of the characteristic value of the marked outlier from the boundary of the outlier threshold range, rather than simply deleting any abnormal data.

[0074] In this embodiment, absolute deviation, relative deviation, standard deviation, Z-score, percentile deviation, Mahalanobis distance, etc. can be used to calculate the deviation degree. The conditions for selecting operations such as retention, correction, or deletion according to the deviation degree can be set by those skilled in the art according to the actual application situation.

[0075] In this embodiment, preprocessing is performed on outliers that significantly deviate from the normal range. These outliers may be caused by data entry errors, measurement errors, equipment failures, etc., and directly interfere with data analysis and model training. Therefore, cleaning is required at an early stage, which can improve data quality and ensure the accuracy of subsequent analysis and modeling.

[0076] Step 2: Extract features related to abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data.

[0077] The features related to abnormal detection of photovoltaic power generation extracted in Step 2 include at least one of the following: statistical features of power generation, time-domain features of power generation, model-based features of power generation, daily power generation change rate, and the proportion of power consumption during peak-valley electricity price periods.

[0078] In this embodiment, the statistical features may include calculating the mean, median, maximum, and minimum of the data; the frequency-domain features may be extracting the frequency-domain features using Fourier transform; the time-domain features may include calculating the autocorrelation, skewness, and kurtosis of the data; the model-based features are the features extracted from the data by establishing a mathematical model, which can capture the time dependence and dynamic changes in the data. For example, an autoregressive model can identify trends and seasonal patterns in the data, while residual analysis can help identify outliers that the model cannot explain. These features help improve the accuracy of anomaly detection. The daily power generation change rate may include calculating the change rate of the daily power generation; the electricity consumption ratio during peak-valley electricity price periods may include the ratio of the electricity consumption during peak periods to that during valley periods.

[0079] It can be understood that the above features are different feature types and describe the photovoltaic power generation data from different perspectives. When performing dimensionality reduction processing, different types of features will be regarded as different dimensions. By combining these multi-dimensional features together to form a feature vector, and then performing dimensionality reduction processing on this feature vector, the complexity of the data can be reduced and the efficiency of the model can be improved.

[0080] Step 3: Determine the principal components of PCA by combining the information entropy of each feature, so as to perform dimensionality reduction on the extracted feature data.

[0081] Specifically, in Step 3, determining the principal components of PCA by combining the information entropy of each feature, so as to perform dimensionality reduction on the extracted feature data, includes:

[0082] Step 3.1: Standardize the extracted feature data and calculate the covariance matrix of the standardized data.

[0083] The specific formula is: where X is the extracted feature data, μ is the mean of the data after feature extraction, and σ is the standard deviation.

[0084] In this way, the influence of different dimensions can be eliminated, ensuring that the mean of each feature is 0 and the variance is 1, and ensuring that the influence of each feature on the dimensionality reduction result is equivalent.

[0085] Step 3.2: Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues and corresponding eigenvectors.

[0086] In this embodiment, the covariance matrix reflects the correlation between features. Calculate the covariance matrix of the standardized data according to the following formula,

[0087]

[0088] where X standardized is the standardized data matrix, Σ is the covariance matrix, and N is the number of data points in the data matrix.

[0089] Step 3.3: Calculate the information entropy of each feature in the standardized data.

[0090] Specifically, calculate the information entropy of each feature in the standardized data according to the following formula:

[0091]

[0092] where H(X i ) is the information entropy of the i-th feature, and p(x ij ) is the probability distribution of the j-th data corresponding to the feature X i , and n i is the number of data corresponding to the feature X i .

[0093] Step 3.4: Determine the principal components according to the eigenvalue and the information entropy.

[0094] Optionally, perform weighted summation according to the eigenvalue, the information entropy and their corresponding weights to determine the principal components.

[0095] In this embodiment, the obtained sums can be sorted from large to small, and the first k sums and the corresponding eigenvectors are selected as the principal components. It is also possible to first sort the eigenvalues from large to small, select the first p eigenvalues, then calculate the information entropy of the first p eigenvalues, sort the information entropy from large to small, and select the eigenvectors corresponding to the first q information entropy as the principal components, where q ≤ p.

[0096] The present invention determines the principal components of PCA by combining the information entropy, performs dimensionality reduction processing on the operation data of the distributed photovoltaic power station after feature extraction, can process high-dimensional data, and fully captures the non-linear features and dynamic changes of the data, thereby improving the efficiency and accuracy of screening abnormal data of photovoltaic power generation.

[0097] Step 3.5: Project the standardized data into the feature space composed of the principal components to obtain the data after dimensionality reduction.

[0098] Specifically, X reduced = X standardized P. Where X reduced is the data after dimensionality reduction, and P is the matrix composed of the first k principal components.

[0099] Step 4: Combine the pre-trained isolation forest model and the autoencoder reconstruction model to perform anomaly detection on the data after dimensionality reduction to obtain the anomaly score of each data.

[0100] In this embodiment, the parameter settings of the Isolation Forest include the number of trees and the subsample size. For example, the number of trees is set to 100 and the subsample size is set to 256. An Isolation Forest model is constructed by randomly selecting features and split values and recursively splitting until each data point is isolated or the maximum tree depth is reached. For each data point, calculate its path length in all isolation trees and obtain the average value. At the same time, calculate the Euclidean distance between the original data point and the data point after reconstruction by the autoencoder. By combining the anomaly scores calculated based on the path length and the anomaly scores obtained from the reconstruction error of the autoencoder, a comprehensive anomaly score is calculated.

[0101] Step 4 specifically includes:

[0102] Step 4.1: Input the dimension-reduced data into the pre-trained Isolation Forest model to obtain the path length corresponding to each isolation tree, and calculate the first anomaly score according to the path length.

[0103] Specifically, the first anomaly score is calculated according to the path length by the following formula:

[0104]

[0105] where s IF (x) is the first anomaly score of data x, n is the number of data after PCA dimension reduction, h(x) is the height of the isolation tree where data x is located, E(h(x)) is the average path length of data x in all isolation trees, c(n) is the normalization factor of the expected path length, and H(i) is the harmonic number, which can be approximated as H(i)≈

[0106] ln(i)+0.5772156649 (Euler's constant).

[0107] In Step 4.1, first set the parameters of the Isolation Forest model. The parameter settings of the Isolation Forest include the number of trees and the subsample size. For example, the number of trees is set to 100 and the subsample size is set to 256. Then, randomly select features and split values and recursively split until each data point is isolated or the maximum tree depth is reached. The Isolation Forest algorithm identifies outliers by constructing multiple isolation trees. Each tree recursively splits the data by randomly selecting features and split values until each data point is isolated at a leaf node. For each data point x, calculate its path length h(t,x) in each tree and obtain the average value Eh(x) of the path lengths of all trees. The path length is the distance from the root node to the leaf node. Outliers usually have shorter path lengths because they are different and easier to isolate. The closer the first anomaly score is to 1, the more likely it is to be an outlier.

[0108] In this embodiment, the average path length is calculated by the following formula:

[0109]

[0110] Among them, h(t, x) is the path length of the data point x in the t-th tree, and T is the total number of isolated trees.

[0111] Step 4.2: Input the dimension-reduced data into the pre-trained autoencoder reconstruction model to obtain the reconstruction error, and calculate the second anomaly score according to the reconstruction error.

[0112] Specifically, the second anomaly score is calculated according to the following formula based on the reconstruction error:

[0113] s AE (x) = ||x - x^||,

[0114] Among them, s AE (x) is the second anomaly score, x is the data after PCA dimension reduction, x^ is the data obtained through the autoencoder reconstruction model, and ||x - x^|| is the reconstruction error.

[0115] It can be understood that using the reconstruction error of the autoencoder as the anomaly score is actually the Euclidean distance (or L2 distance) between the original data point and the data point reconstructed by the autoencoder. This is a non-negative real number, indicating the degree of difference between the data point and the data reconstructed by the autoencoder. The higher the second anomaly score, the greater the difference between the data point and the normal data, and the more likely it is an outlier.

[0116] Step 4.3: Calculate the anomaly score of each dimension-reduced data according to the first anomaly score and the second anomaly score.

[0117] The anomaly score of each dimension-reduced data is calculated according to the following formula based on the first anomaly score and the second anomaly score:

[0118] s combined (x) = α * s AE (x) + (1 - α) * s IF (x),

[0119] Among them, α is the weight parameter, s AE (x) is the first anomaly score of the data x, and s IF (x) is the second anomaly score of the data x. The weight parameter α is adjusted according to the characteristics of the data, for example, optimized through cross-validation.

[0120] In this embodiment, the isolated forest model and the autoencoder reconstruction model are combined to perform anomaly detection on the dimensionality-reduced data, and the first anomaly score and the second anomaly score are calculated respectively, which can effectively reduce the misjudgment caused by a single method and can identify anomaly points more comprehensively. Since the autoencoder reconstruction model can capture the non-linear features of the data, combining with the isolated forest model can further improve the robustness and accuracy of the detection, and can maintain a high detection performance under different data distributions and noise levels.

[0121] Step 5: Determine the abnormal data according to the anomaly score and the preset first threshold;

[0122] Specifically, compare the anomaly score with the preset first threshold. The preset first threshold can be set as a certain percentile of the anomaly score, such as the 95th percentile, and thus identify the abnormal data points. If s(x) is greater than the preset first threshold, then x is regarded as an anomaly point.

[0123] Taking a distributed photovoltaic power station as an example, the operation data volume of the power station in one month is 2,880 data points (with a recording frequency of once every 15 minutes). Using the method provided in Embodiment 1 of the present invention, 1,000 abnormal data points can be identified, accounting for 1% of the total data.

[0124] Furthermore, the method further includes:

[0125] Step 6: Determine whether to trigger the alarm mechanism according to the abnormal data.

[0126] Specifically, display the abnormal data points in the form of a list and use a chart to display the distribution of the abnormal data. For severely abnormal data, the system automatically sends an alarm email to the maintenance personnel of the photovoltaic power station. The alarm mechanism can be triggered for data with an anomaly score greater than the preset second threshold (for example, it can be set to 99%). The abnormal data can include data attributes, and whether to trigger the alarm mechanism can also be judged in combination with the data attributes. In this way, the abnormal data can be processed hierarchically, improving the work efficiency of subsequent relevant personnel.

[0127] Embodiment 2 of the present invention provides an abnormal data screening system that runs the abnormal data screening method based on PCA and the isolated forest algorithm as described in Embodiment 1. The system includes:

[0128] A data preprocessing module, which is used to collect the operation data of the distributed photovoltaic power station and perform preprocessing;

[0129] A feature extraction module, which is used to extract the features related to the abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data;

[0130] A dimensionality reduction module, which is used to determine the principal components of PCA in combination with information entropy, so as to perform dimensionality reduction on the extracted feature data;

[0131] An anomaly detection module, which is used to combine a pre-trained isolation forest model and an autoencoder reconstruction model to perform anomaly detection on the dimension-reduced data, obtain the anomaly score of each data, and determine the anomaly data according to the anomaly score and a preset first threshold.

[0132] Optionally, the system further includes:

[0133] An alarm module, which is used to determine whether to trigger an alarm mechanism according to the anomaly data.

[0134] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, it implements the anomaly data screening method based on PCA and the isolation forest algorithm described in Embodiment 1.

[0135] Embodiment 4 of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the anomaly data screening method based on PCA and the isolation forest algorithm described in Embodiment 1.

[0136] Compared with the prior art, the beneficial effects of the present invention at least include:

[0137] The present invention determines the principal components of PCA by combining information entropy, thereby performing dimension reduction processing on the operation data of the distributed photovoltaic power station after feature extraction. It can process high-dimensional data and fully capture the non-linear features and dynamic changes of the data, thereby improving the efficiency and accuracy of screening photovoltaic power generation anomaly data.

[0138] The present invention adopts outlier processing in the data preprocessing stage, preliminarily screens outliers before the dimension reduction operation, and reduces the data volume for subsequent anomaly detection. In the feature extraction stage, the present invention comprehensively utilizes the statistical features of power generation, the time-domain features of power generation, the model-based features of power generation, the daily power generation change rate, and the proportion of power consumption during peak-valley electricity price periods to comprehensively capture the multi-dimensional information of the data.

[0139] The present invention combines to calculate the anomaly score through multiple dimensions, which can more comprehensively evaluate the anomaly degree of data points, make the anomaly detection more sensitive, comprehensive and accurate, and thus better adapt to complex data patterns, significantly improving the accuracy of anomaly detection.

[0140] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or subsequent. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0141] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement aspects of the present disclosure.

[0142] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in a groove having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not to be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0143] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing devices, or may be downloaded to an external computer or an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0144] Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. An abnormal data screening method based on PCA and isolation forest algorithm, characterized in that, The method includes the following steps: Step 1: Collect the operation data of the distributed photovoltaic power station and perform preprocessing; Step 2: Extract the features related to the abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data; Step 3: Determine the principal components of PCA by combining the information entropy of each feature, so as to reduce the dimension of the extracted feature data; Step 4: Combine the pre-trained isolation forest model and the autoencoder reconstruction model to perform abnormal detection on the dimension-reduced data, and obtain the abnormal score of each data; Step 5: Determine the abnormal data according to the abnormal score and the preset first threshold.

2. The abnormal data screening method based on PCA and isolation forest algorithm according to claim 1, wherein In Step 1, the preprocessing includes outlier processing, and the outlier processing includes: Classify the operation data of the distributed photovoltaic power station according to various categories to obtain multiple groups of operation data; Sort each group of operation data from small to large, determine the first position value of the outlier threshold range as the operation data at the 1 / 4 position, and determine the second position value of the outlier threshold range as the operation data at the 3 / 4 position; Determine the first boundary value and the second boundary value of the outlier threshold range according to the first position value and the second position value, and the outlier threshold range is less than the first boundary value or greater than the second boundary value; Determine the outlier processing operation according to the degree of deviation of the outlier from the first boundary value or the second boundary value of the outlier threshold range.

3. The abnormal data screening method based on PCA and isolation forest algorithm according to claim 1, characterized in that In Step 2, the features extracted related to the abnormal detection of photovoltaic power generation include at least one of the following: Statistical features of power generation, time-domain features of power generation, model-based features of power generation, daily power generation change rate, and proportion of power consumption in peak-valley electricity price periods.

4. The abnormal data screening method based on PCA and isolation forest algorithm according to claim 1, characterized in that In Step 3, determining the principal components of PCA by combining the information entropy of each feature, so as to reduce the dimension of the extracted feature data, includes: Standardize the extracted feature data and calculate the covariance matrix of the standardized data; Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues and corresponding eigenvectors; Calculate the information entropy of each feature in the standardized data according to the following formula: where \(H(X i )\) is the information entropy of the \(i\)-th feature, and \(p(x ij )\) is the probability distribution of the \(j\)-th data corresponding to the feature \(X i \), and \(n i \) is the number of data corresponding to the feature \(X i \); Perform weighted summation on the eigenvalues, information entropy and their corresponding weights to determine the principal components; Project the standardized data into the feature space composed of the principal components to obtain the dimension-reduced data.

5. The abnormal data screening method based on PCA and isolation forest algorithm according to any one of claims 1 to 4, characterized in that In Step 4, combining the pre-trained isolation forest model and the autoencoder reconstruction model to perform abnormal detection on the dimension-reduced data, and obtaining the abnormal score of each data, includes: Input the dimension-reduced data into the pre-trained isolation forest model to obtain the path length corresponding to each isolation tree, and calculate the first abnormal score according to the path length; Input the dimension-reduced data into the pre-trained autoencoder reconstruction model to obtain the reconstruction error, and calculate the second abnormal score according to the reconstruction error; The anomaly score of each dimension-reduced data is calculated based on the first anomaly score and the second anomaly score.

6. The anomaly data screening method based on PCA and isolation forest algorithm according to claim 5, characterized in that The anomaly score of each dimension-reduced data is calculated according to the following formula based on the first anomaly score and the second anomaly score: s combined (x) = α * s AE (x) + (1 - α) * s IF (x), where α is a weight parameter, and s AE (x) is the first anomaly score of data x, and s IF (x) is the second anomaly score of data x.

7. The anomaly data screening method based on PCA and isolation forest algorithm according to claim 6, characterized in that The first anomaly score is calculated according to the path length by the following formula: where s IF (x) is the first anomaly score of data x, n is the number of data after PCA dimensionality reduction, h(x) is the height of the isolation tree where data x is located, E(h(x)) is the average path length of data x in all isolation trees, c(n) is the normalization factor of the expected path length, and H(i) is the harmonic number.

8. The abnormal data screening method based on PCA and isolation forest algorithm according to claim 6, characterized in that, The second anomaly score is calculated according to the reconstruction error by the following formula: s AE f(x) = ||x - x^||, where s AE (x) is the second anomaly score, x is the data after PCA dimensionality reduction, x^ is the data obtained through the autoencoder reconstruction model, and ||x - x^|| is the reconstruction error.

9. The abnormal data screening system for the abnormal data screening method based on PCA and isolation forest algorithm according to any one of claims 1 to 8, characterized in that, The system includes: A data preprocessing module, configured to collect operation data of a distributed photovoltaic power station and perform preprocessing; A feature extraction module, configured to extract features related to abnormal detection of photovoltaic power generation from the preprocessed data to obtain feature data; A dimension reduction module, configured to determine the principal components of PCA in combination with information entropy, so as to perform dimension reduction on the extracted feature data; An anomaly detection module, configured to perform anomaly detection on the dimension-reduced data in combination with a pre-trained isolation forest model and an autoencoder reconstruction model, obtain the anomaly score of each data, and determine the anomaly data according to the anomaly score and a preset first threshold.

10. An electronic device, including a processor and a storage medium; characterized in that The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the anomaly data screening method based on PCA and isolation forest algorithm according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the steps of the anomaly data screening method based on PCA and isolation forest algorithm according to any one of claims 1-8 are implemented.

Images