A method for handling data anomalies in photovoltaic power prediction acquisition devices

Abstract

This application discloses a method for handling data anomalies in photovoltaic power prediction acquisition devices, belonging to the field of power technology. The method includes: synchronously acquiring and storing multidimensional meteorological data and actual power generation data of the photovoltaic system to obtain raw data; performing linear correlation and nonlinear correlation analysis of the raw data using the maximum information coefficient to obtain core input features; identifying and marking anomalies in the core input features and removing the anomalies to obtain de-anomaly-removed data; and performing iterative interpolation on the de-anomaly-removed data to obtain imputed data and outputting it. This method solves the problems of insufficient accuracy in photovoltaic power data processing and inaccurate identification of key information in existing technologies, thereby improving the accuracy of photovoltaic power data analysis and processing.

Description

Technical Field

[0001] This application belongs to the field of power technology, specifically, it relates to a method for handling data anomalies in a photovoltaic power prediction-related data acquisition device. Background Technology

[0002] Driven by the "dual carbon" goals, photovoltaic (PV) power generation, as a core component of clean energy, has seen its installed capacity continuously expand. PV power output is significantly affected by meteorological factors (solar radiation, temperature, humidity, etc.), exhibiting strong fluctuations and intermittency. The accuracy of ultra-short-term forecasts directly determines the security of grid dispatch and the efficiency of PV power absorption. Meteorological data and actual power generation data are the core inputs to PV power forecasting models, and data quality (completeness, accuracy, and validity) directly determines the upper limit of the forecasting model's performance.

[0003] Currently, the meteorological data collected by photovoltaic power plants includes multi-dimensional variables such as module temperature, ambient temperature, air pressure, relative humidity, total radiation, direct radiation, and diffuse radiation, while simultaneously recording actual power generation data. However, this type of data faces three core challenges in its collection and transmission:

[0004] 1. Feature redundancy and lack of effective information: Among the multidimensional meteorological variables, some variables have very low correlation with photovoltaic power (such as air pressure and relative humidity), while some variables have very strong mutual coupling (such as ambient temperature and relative humidity). Directly inputting them into the model will lead to multicollinearity, increase the computational burden, and may also obscure the role of core features.

[0005] 2. Outlier Interference: Sensor malfunctions, signal transmission packet loss, extreme weather, and other factors can lead to outliers in the data (such as zero power under high irradiance or sudden temperature rises and falls in modules). Traditional detection methods include the 3-sigma method, a classic outlier detection method in statistics based on the assumption of a normal distribution. Its core idea is that, assuming the data follows a normal distribution, the probability of a value falling within the mean (μ) and three standard deviations (3σ) is 99.73%. Therefore, data points exceeding the μ ± 3σ range are considered outliers (i.e., low-probability events). However, this method is limited to single-dimensional data; univariate detection methods struggle to identify logical anomalies that violate multivariate coupling relationships and cannot adapt to the seasonal and daily periodic variations of photovoltaic data.

[0006] 3. Missing Value Issues: Removing outliers creates data gaps. Directly deleting samples containing missing values ​​reduces data volume and introduces bias. Traditional missing value imputation methods include mean imputation and similar-day imputation. Mean imputation fills the gap by calculating the mean of the data points on either side of the missing value, but this can cause the imputation result to lose its consistency when multiple consecutive missing values ​​exist. Similar-day imputation fills the gap by selecting data from the period most similar to the missing value, but this method alters the original data's volatility and has low accuracy.

[0007] Existing solutions to the above problems have significant limitations: feature selection relies solely on linear correlation analysis (such as using only the Pearson coefficient), neglecting nonlinear correlation features; anomaly detection uses the 3-sigma method, which cannot adapt to the non-stationary nature of the data; missing value imputation often employs simple linear interpolation or similarity-based imputation, failing to fully utilize the interaction information between variables. These shortcomings result in poor data quality after preprocessing, thus limiting the accuracy improvement of subsequent prediction models and becoming a technical bottleneck in the field of ultra-short-term photovoltaic power prediction. Summary of the Invention

[0008] To address the technical problems existing in the prior art, this application discloses a method for handling data anomalies in photovoltaic power prediction-related acquisition devices, specifically:

[0009] A method for handling data anomalies in photovoltaic power prediction-related acquisition devices, the method comprising:

[0010] The system acquires and stores multidimensional meteorological data and actual power generation data of photovoltaic systems simultaneously to obtain raw data.

[0011] The original data is subjected to linear correlation and nonlinear correlation analysis based on the maximum information coefficient to obtain the core input features;

[0012] The core input features are used to identify and mark abnormal data, and the abnormal data is removed to obtain the data after anomaly removal.

[0013] The anomaly-removed data is iteratively interpolated to obtain the filled data, which is then output.

[0014] Optionally, the simultaneous acquisition and storage of multi-dimensional meteorological data and actual power generation data of the photovoltaic system to obtain raw data includes:

[0015] Acquire multidimensional meteorological data of the photovoltaic system and the acquisition time of the multidimensional meteorological data;

[0016] The actual power generation data of the photovoltaic system corresponding to the acquisition time is obtained to obtain synchronously collected multidimensional meteorological data and actual power generation data;

[0017] The synchronously collected multidimensional meteorological data and actual power generation data are stored to obtain the raw data.

[0018] Optionally, linear correlation and nonlinear correlation analysis with maximum information coefficient are performed on the original data to obtain core input features, including:

[0019] The original data is preprocessed based on the Z-Score normalization method, and the normalization equation is as follows:

[0020] ,

[0021] Where, x i The i-th value represents the data feature; μ represents the mean; σ represents the standard deviation; X std Standard values ​​representing data characteristics;

[0022] The linear correlation of the preprocessed data is determined by the following equation:

[0023] ,

[0024] in, Let x be the correlation coefficient between variables X and Y, with a value range of [-1, 1]. i y i Let represent the i-th data in variables X and Y, respectively. and , where are the average values ​​of variables X and Y respectively, and n is the sample size;

[0025] Based on mutual information theory, the maximum information coefficient of the preprocessed data is obtained to acquire nonlinear correlations. The equation for the maximum information coefficient is:

[0026] ,

[0027] Where I(X, Y) represents mutual information; n x n y B represents the number of grid cells; B represents the upper limit of the number of grid cells.

[0028] Based on the comparison results of the nonlinear association and linear correlation, the feature with the highest correlation is obtained to obtain the core input feature.

[0029] Optionally, the core input features are used to identify and mark abnormal data, and the abnormal data is removed to obtain anomaly-free data, including:

[0030] A multidimensional feature vector is constructed from the core input features. The multidimensional feature vector is represented as follows:

[0031] X=[a1, a2, ... a m ] T ,

[0032] Where, a1~a n This represents the core input features; m is the number of core input features;

[0033] Obtain the Mahalanobis distance of the core input features. The equation for determining the Mahalanobis distance is:

[0034] ,

[0035] in, The mean vector of a multidimensional feature vector; ∑ -1 D represents the inverse of the covariance matrix; M (a) j ) represents the j-th core input feature;

[0036] Based on the dual-window distance detection method, abnormal data within the core input features are obtained;

[0037] Abnormal data is removed from the core input features to obtain the de-abnormalized data.

[0038] Optionally, the method based on dual-window distance detection, which acquires abnormal data within the core input features, includes:

[0039] Set the size of the window to cover the core input features; the larger window W... L Covering several weeks of data in the core input features, small window W S Covers several days of data in the core input features;

[0040] In the small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within;

[0041] W in the small window S Large window W L Slide inwards to get the larger window W L All anomalies within.

[0042] Optionally, the small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within include:

[0043] In the small window W S Using the mean of the Nemaduan distance as a benchmark, obtain the small window W. S The dynamic threshold for fluctuations exceeding the baseline value is determined by the following equation:

[0044] ,

[0045] in, This represents the mean distance within the small window, reflecting the central tendency of the local data. This represents the maximum distance within the small window. S represents the minimum distance within the small window; S(·) represents the scaling function used to shrink the portion exceeding the baseline; t represents the time when the photovoltaic system occurs, which is the core input feature.

[0046] For small window W S The abnormal data within is identified using the following equation:

[0047] ,

[0048] in, The sample x at time t t The results of the anomaly detection; Normal indicates normal data; Anomaly indicates abnormal data; Indicates sample x t Mahalanobis distance;

[0049] In the outlier data, the feature that contributes the most to the increase in Mahalanobis distance is identified, and the contribution determination equation is as follows:

[0050] ,

[0051] Among them, D original D represents the Mahalanobis distance of the original data rows; modified (f) represents the Mahalanobis distance after replacing the value of row feature f with its sliding window mean; C(f) represents the contribution of feature f; f represents the feature.

[0052] Optionally, the iterative interpolation process performed on the anomaly-removed data to obtain and output the interpolated data includes:

[0053] Data after anomaly removal Missing values ​​were imputed using the median to obtain the complete matrix. And set the maximum number of iterations G, where Q represents the original dataset; q1~q p p represents the core input feature vector in the dataset; p represents the number of core input feature vectors.

[0054] The core input feature vectors are sorted in ascending order of missing rate, and features and labels are constructed. The equations for constructing features and labels are as follows:

[0055] ,

[0056] Among them, X s X represents the s-th variable that needs to be filled. -s Indicates that except for X s A matrix consisting of all other variables except I; obs Indicates the row number of valid data in column s; I mis This indicates the row number where data is missing in column s; Represents the input features of the training set; Indicates the target labels of the training set; The input features of the prediction set are represented by s; s represents the variable index.

[0057] Based on the maximum number of iterations or the normalized root mean square error, determine whether the upper limit of the number of filling iterations has been reached, and store and output the filling data when the upper limit of the number of filling iterations is reached.

[0058] Optionally, the step of determining whether the upper limit for the number of filling iterations has been reached based on the maximum number of iterations or the normalized root mean square error, and storing and outputting the filling data when the upper limit for the number of filling iterations is reached, includes:

[0059] The training data is fitted using a random forest model to handle nonlinear relationships. The random forest algorithm model is as follows:

[0060] ,

[0061] Among them, f s This represents the training result of the random forest model; Train() represents the training algorithm;

[0062] The predicted set is input into the random forest model to obtain the estimated values ​​of the variables, and then input into the matrix. The corresponding position is represented by the equation:

[0063] ,

[0064] in, Represents the estimated value of a variable;

[0065] The difference between two adjacent iterations is obtained to determine whether convergence has occurred. If convergence is achieved, the normalized root mean square error (RMSE) is used for determination. The equation for determining the normalized RMSE is:

[0066] ,

[0067] in, This represents the normalized root mean square error result. This indicates the result of the next iteration; This represents the result of the previous iteration;

[0068] When the normalized root mean square error is less than a preset threshold, or when the maximum number of iterations is reached, the upper limit of the number of filling iterations is reached, and the filling data is stored and output.

[0069] The beneficial effects of this application include:

[0070] 1. Comprehensive and accurate feature selection, with optimal retention of core features and elimination of redundancy. This invention employs a dual analysis method of Pearson correlation coefficient + MIC, which can capture both linear correlations and uncover nonlinear coupling relationships. The direct technical effect is that no core features are missed, redundant variables are completely eliminated, and the quality of input features is significantly improved. This reduces the computational burden of subsequent models and avoids underfitting caused by missing effective features. The final technical effect is that the comprehensiveness and accuracy of feature selection are greatly improved compared to existing technologies, providing a high-quality input foundation for the prediction model.

[0071] 2. High anomaly detection accuracy, significantly reduced false alarm and missed detection rates. This invention addresses multivariate coupling problems through Mahalanobis distance, adapts to the seasonality and daily periodicity of data through a dual-window approach, and enhances the dynamics and stability of the damped scaling threshold. The direct technical effect is the ability to identify logical anomalies that violate physical laws while adapting to changes in data statistical characteristics over different time periods; the outlier missed detection rate can be reduced by 10% compared to existing technologies, and the effective data retention rate can be increased by 20%. The ultimate technical effect is that the overall accuracy of anomaly detection is improved by more than 30% compared to existing technologies, providing precise markers for filling gaps in the detection process.

[0072] 3. High accuracy in missing value imputation while preserving the complete inter-variable relationship structure: This invention's random forest iterative imputation fully utilizes multivariate interaction information, iteratively optimizing and converging to the optimal imputation result without requiring assumptions about data distribution. The direct technical effect is that the relative error between the imputed value and the true value is ≤3%, far lower than existing imputation methods. Simultaneously, it fully preserves the linear and non-linear relationship structure between variables, avoiding model training bias caused by imputation distortion. The final technical effect is a 20% improvement in missing value imputation accuracy compared to existing technologies, and the data quality meets the input requirements of high-precision prediction models.

[0073] 4. Strong adaptability, compatible with data characteristics of different photovoltaic power plants. The dual-window size, dynamic threshold parameters, and random forest iteration count of this invention can be adaptively adjusted according to the geographical environment, meteorological conditions, and data characteristics of different photovoltaic power plants. The direct technical effect is that the model can adapt to data from different weather types such as sunny, cloudy, and overcast days, as well as the differences between photovoltaic power plants in different regions. The ultimate technical effect is that it can be widely applied to various distributed and centralized photovoltaic power plants, demonstrating strong engineering practicality. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments of this application or the prior art will be briefly introduced below. Obviously, the following description is only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0075] Figure 1 A flowchart illustrating a method for handling data anomalies in a photovoltaic power prediction acquisition device, provided in an embodiment of this application.

[0076] Figure 2 This document presents a flowchart illustrating the outlier detection process of a data anomaly handling method for a photovoltaic power prediction-related acquisition device, as provided in an embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, in the embodiments of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0078] The existing system architecture includes: a data acquisition module: a meteorological sensor array deployed at the photovoltaic power station (collecting component temperature, ambient temperature, air pressure, relative humidity, and radiance), and a power generation acquisition unit (sampling frequency 5 minutes / time), simultaneously acquiring multidimensional meteorological data and actual power generation data; a feature selection module: using the Pearson correlation coefficient single analysis method to calculate the linear correlation coefficient between each meteorological variable and power generation, setting a fixed threshold (e.g., |r|≥0.3), selecting variables that meet the threshold condition as model input, and eliminating other variables; and an outlier detection module: employing the Mahalanobis distance method with a fixed threshold, an effective metric for outlier detection in multivariate data. It considers the correlation between variables and the scale differences between variables, and can identify outliers that are difficult to detect using Euclidean distance. This method calculates the window distance centered on each data point sequentially and compares it with a fixed upper threshold. Data rows exceeding the threshold are identified as abnormal and set to null. The missing value imputation module uses a random forest imputation method, a "multi-decision tree ensemble" prediction model. Its core logic for imputing missing values ​​is to divide the original dataset into training and test sets. Variables with missing values ​​are treated as the test set, while other complete variables are treated as the training set. The random forest model learns the correlation between features and the target, and finally, the prediction results are used to imput the missing values. The output module outputs the imputed dataset to the photovoltaic power prediction model, while simultaneously storing the original and preprocessed data for traceability. The main problems with existing technologies include: incomplete feature selection and an imbalance between redundant removal and effective feature retention. Existing methods only use Pearson correlation coefficients for linear correlation analysis, while photovoltaic systems are complex nonlinear systems with numerous nonlinear correlations between meteorological variables and power generation, such as the nonlinear coupling between total radiation and power. Furthermore, existing methods do not consider the mutual coupling between variables, only removing variables with low linear correlation and failing to identify redundant coupled variables. The causal chain is as follows: Single linear correlation analysis → Omission of nonlinear correlation features + Inability to identify coupled redundant variables → Low input feature quality → Increased model computational burden + Limited prediction accuracy. Anomaly detection accuracy is low, with high false negative and false positive rates. Existing methods use Mahalanobis distance detection with fixed thresholds, which, on the one hand, cannot identify specific feature anomalies, leading to empty rows of data. On the other hand, photovoltaic data exhibits significant seasonality and daily periodicity; fixed thresholds cannot adapt to the non-stationary changes in the data, resulting in normal high-value data in winter being misjudged as anomalies and abnormally low-value data in summer being missed. The causal chain is: Fixed threshold detection → Inability to identify specific feature anomalies + Inability to adapt to the non-stationary nature of the data → High false negative and false positive rates for outliers → Residual data noise + Loss of valid data → Subsequent model training bias. Missing value imputation accuracy is low, with only one prediction imputation performed for each variable containing missing values. In the feature matrix used by this method, other variables may also contain missing values.These missing values ​​are roughly filled in the first step using the mean or median. This coarse imputation distorts the true relationships between variables, causing the model to learn data based on incorrect information. If inappropriate data is used for initial imputation, the random forest model will predict based on this skewed information, spreading the bias to all missing values. This bias cannot be identified and corrected in a single step, ultimately causing the entire dataset to systematically deviate from the true distribution. The causal chain is: simple random forest imputation → over-reliance on initial imputation + failure to utilize iterative feedback between variables → amplification of initial bias → distortion of statistical properties of imputed data → unreliable input to the prediction model.

[0079] The purpose of this application is to address the technical shortcomings in photovoltaic power prediction, such as feature redundancy, outlier interference, and missing value influence between meteorological data recorded by data acquisition devices and actual power generation data, leading to low data quality and insufficient accuracy of subsequent prediction models. The basic solution is as follows: First, a feature correlation analysis method combining Pearson correlation coefficient and maximum information coefficient (MIC) is used to screen out core meteorological features strongly correlated with photovoltaic power output and eliminate redundant variables. Second, a Mahalanobis distance anomaly detection algorithm with dual-window dynamic threshold is used to accurately identify and label anomalous data in multivariate coupled scenarios. Finally, an iterative imputation algorithm based on random forest is used to reconstruct and fill in the missing values ​​after anomaly labeling, forming a high-quality clean dataset. This provides reliable input for ultra-short-term photovoltaic power prediction models and is adaptable to the meteorological data characteristics of different photovoltaic power plants. Figure 1 The diagram shown is a flowchart of a data anomaly handling method for a photovoltaic power prediction-related acquisition device provided in an embodiment of this application. Specifically:

[0080] S110. Acquire and store multi-dimensional meteorological data and actual power generation data of the photovoltaic system simultaneously to obtain raw data.

[0081] S120. Perform linear correlation and nonlinear correlation analysis on the original data to obtain the core input features.

[0082] S130. Identify and mark abnormal data for the core input features, and remove the abnormal data to obtain data after anomaly removal.

[0083] S140. Perform iterative interpolation on the anomaly-removed data to obtain the filled data and output it.

[0084] The following will provide a detailed explanation of all the steps above:

[0085] As described in step S110, the purpose of this step is to determine and synchronize the environmental data and power generation data within the photovoltaic system, so that the correlation between these two types of data can be obtained in subsequent analysis and applied to the operational data analysis of the photovoltaic system. Specifically:

[0086] Acquire multidimensional meteorological data of the photovoltaic system and the acquisition time of the multidimensional meteorological data;

[0087] The actual power generation data of the photovoltaic system corresponding to the acquisition time is obtained to obtain synchronously collected multidimensional meteorological data and actual power generation data;

[0088] The synchronously collected multidimensional meteorological data and actual power generation data are stored to obtain the raw data.

[0089] This step is executed based on the multi-source data acquisition module, which is the data foundation of the system. This module enables the synchronous acquisition and storage of multi-dimensional meteorological data and actual power generation data, ensuring data integrity and spatiotemporal consistency. Core components include: component temperature sensor, ambient temperature sensor, barometric pressure sensor, relative humidity sensor, total radiation sensor, direct radiation sensor, diffuse radiation sensor, power generation acquisition unit, and data storage unit. Core function: Synchronously acquires multi-dimensional meteorological data and actual power generation data every 5 minutes, achieving spatiotemporal synchronization and secure storage of data, providing complete and consistent raw data for subsequent preprocessing stages.

[0090] As described in step S120, the purpose of this step is to analyze the core data among the various types of data obtained, thereby laying the foundation for subsequent data analysis work. Specifically:

[0091] 3. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that the step of performing linear correlation and nonlinear correlation analysis on the original data to obtain core input features includes:

[0092] The original data is preprocessed based on the Z-Score normalization method, and the normalization equation is as follows:

[0093] ,

[0094] Where, x i The i-th value represents the data feature; μ represents the mean; σ represents the standard deviation; X std Standard values ​​representing data characteristics;

[0095] The linear correlation of the preprocessed data is determined by the following equation:

[0096] ,

[0097] in, Let x be the correlation coefficient between variables X and Y, with a value range of [-1, 1]. i y i Let represent the i-th data in variables X and Y, respectively. and , where are the average values ​​of variables X and Y respectively, and n is the sample size;

[0098] Based on mutual information theory, the maximum information coefficient of the preprocessed data is obtained to acquire nonlinear correlations. The equation for the maximum information coefficient is:

[0099] ,

[0100] Where I(X, Y) represents mutual information; n x n y B represents the number of grid cells; B represents the upper limit of the number of grid cells.

[0101] Based on the comparison results of the nonlinear association and linear correlation, the feature with the highest correlation is obtained to obtain the core input feature.

[0102] This step is performed by the feature correlation analysis module, which solves the problem of incomplete feature screening in existing technologies. Through dual analysis of linear correlation of Pearson correlation coefficient and nonlinear correlation of maximum information coefficient (MIC), it achieves the screening of core features and the elimination of redundant variables.

[0103] Among them, the data preprocessing sub-step involves performing Z-Score standardization on the collected raw data to eliminate the impact of dimensional differences on correlation analysis.

[0104] Among them, the sub-step for calculating the Pearson correlation coefficient: The Pearson correlation coefficient is mainly used to measure the degree of linear correlation between two continuous variables.

[0105] Among them, the MIC calculation sub-step is based on mutual information theory. For the same set of data, the maximum information coefficient between variables is calculated according to the formula to capture nonlinear correlation.

[0106] In the feature selection sub-step: by comparing the analysis results of Pearson correlation coefficient and MIC, a dual selection rule is set to obtain the most core input features for photovoltaic power prediction. Other variables are eliminated because their correlation with the target variable is too low, in order to reduce model complexity and provide high-quality input features for subsequent steps.

[0107] As described in step S130, the purpose of this step is to use a dual-window method to detect abnormal data, thereby removing obviously abnormal data, and then performing subsequent numerical insertion processing based on this data. Specifically:

[0108] 4. A method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that the step of identifying and marking abnormal data in the core input features, and removing the abnormal data to obtain anomaly-free data, includes:

[0109] A multidimensional feature vector is constructed from the core input features. The multidimensional feature vector is represented as follows:

[0110] X=[a1, a2, ... a m ] T ,

[0111] Where, a1~a n This represents the core input features; m is the number of core input features;

[0112] Obtain the Mahalanobis distance of the core input features. The equation for determining the Mahalanobis distance is:

[0113] ,

[0114] in, The mean vector of a multidimensional feature vector; ∑ -1 D represents the inverse of the covariance matrix; M (a) j ) represents the j-th core input feature;

[0115] Based on the dual-window distance detection method, abnormal data within the core input features are obtained;

[0116] Abnormal data is removed from the core input features to obtain the de-abnormalized data.

[0117] The dual-window distance detection method acquires abnormal data within the core input features, including:

[0118] Set the size of the window to cover the core input features; the larger window W... L Covering several weeks of data in the core input features, small window W S Covers several days of data in the core input features;

[0119] In the small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within;

[0120] W in the small windowS Large window W L Slide inwards to get the larger window W L All anomalies within.

[0121] The small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within include:

[0122] In the small window W S Using the mean of the Nemaduan distance as a benchmark, obtain the small window W. S The dynamic threshold for fluctuations exceeding the baseline value is determined by the following equation:

[0123] ,

[0124] in, This represents the mean distance within the small window, reflecting the central tendency of the local data. This represents the maximum distance within the small window. S represents the minimum distance within the small window; S(·) represents the scaling function used to shrink the portion exceeding the baseline; t represents the time when the photovoltaic system occurs, which is the core input feature.

[0125] For small window W S The abnormal data within is identified using the following equation:

[0126] ,

[0127] in, The sample x at time t t The results of the anomaly detection; Normal indicates normal data; Anomaly indicates abnormal data; Indicates sample x t Mahalanobis distance;

[0128] In the outlier data, the feature that contributes the most to the increase in Mahalanobis distance is identified, and the contribution determination equation is as follows:

[0129] ,

[0130] Among them, D original D represents the Mahalanobis distance of the original data rows; modified (f) represents the Mahalanobis distance after replacing the value of row feature f with its sliding window mean; C(f) represents the contribution of feature f; f represents the feature.

[0131] This step is performed by the dual-window dynamic threshold anomaly detection module, which is one of the core innovations of this invention. It solves the problem of low anomaly detection accuracy in the prior art by using a combination algorithm of Mahalanobis distance, dual windows, and dynamic threshold to achieve accurate identification and labeling of abnormal data.

[0132] This method utilizes Mahalanobis distance to handle multivariate coupling problems, captures the seasonal and periodic trends of data through a dual time window mechanism, and introduces a damped scaling strategy to construct a dynamic threshold in order to achieve accurate identification and removal of outliers.

[0133] In order to quantify the degree to which sample points deviate from the normal data cluster, and to consider the correlation between variables, a multidimensional feature vector is constructed for the core variables after feature selection.

[0134] The dual-window distance calculation sub-step includes: setting the large window W. L This allows it to cover several weeks of data, with a small window W S To cover several days of data, W S In W L Inward sliding. W L W is used to calculate the Mahalanobis distance distribution under seasonal background. S Used to capture daily periodic fluctuations.

[0135] Among them, for the damped scaling dynamic threshold generation sub-step: with W S The mean of the Nemadus distance is used as a benchmark, and the fluctuations exceeding the benchmark are damped and smoothed.

[0136] Among them, among them, This represents the mean distance within the small window, reflecting the central tendency of the local data. This represents the maximum distance within the small window. S represents the minimum distance within the small window, S(·) represents the scaling function used to shrink the part that exceeds the baseline; t represents the moment when the photovoltaic system of the core input feature occurs. When the local fluctuation is severe, the threshold will increase or decrease accordingly to avoid false alarms.

[0137] For rows identified as outliers, it is further determined which specific feature contributed the most to the increased distance. The feature with the highest contribution is then identified as an outlier and marked as null. For example... Figure 2 The diagram shown is an outlier detection flowchart of a data anomaly processing method for a photovoltaic power prediction-related acquisition device provided in an embodiment of this application.

[0138] As described in step S140, the purpose of this step is to perform interpolation processing on the data after outlier removal, and then output the resulting interpolated value. Specifically:

[0139] Data after anomaly removal Missing values ​​were imputed using the median to obtain the complete matrix. And set the maximum number of iterations G, where Q represents the original dataset; q1~q p p represents the core input feature vector in the dataset; p represents the number of core input feature vectors.

[0140] The core input feature vectors are sorted in ascending order of missing rate, and features and labels are constructed. The equations for constructing features and labels are as follows:

[0141] ,

[0142] Among them, X s X represents the s-th variable that needs to be filled. -s Indicates that except for X s A matrix consisting of all other variables except I; obs Indicates the row number of valid data in column s; I mis This indicates the row number where data is missing in column s; Represents the input features of the training set; Indicates the target labels of the training set; The input features of the prediction set are represented by s; s represents the variable index.

[0143] Based on the maximum number of iterations or the normalized root mean square error, determine whether the upper limit of the number of filling iterations has been reached, and store and output the filling data when the upper limit of the number of filling iterations is reached.

[0144] The step of determining whether the upper limit for the number of filling iterations has been reached based on the maximum number of iterations or the normalized root mean square error, and storing and outputting the filling data when the upper limit for the number of filling iterations is reached, includes:

[0145] The training data is fitted using a random forest model to handle nonlinear relationships. The random forest algorithm model is as follows:

[0146] ,

[0147] Among them, f s This represents the training result of the random forest model; Train() represents the training algorithm;

[0148] The predicted set is input into the random forest model to obtain the estimated values ​​of the variables, and then input into the matrix. The corresponding position is represented by the equation:

[0149] ,

[0150] in, Represents the estimated value of a variable;

[0151] The difference between two adjacent iterations is obtained to determine whether convergence has occurred. If convergence is achieved, the normalized root mean square error (RMSE) is used for determination. The equation for determining the normalized RMSE is:

[0152] ,

[0153] in, This represents the normalized root mean square error result. This indicates the result of the next iteration; This represents the result of the previous iteration;

[0154] When the normalized root mean square error is less than a preset threshold, or when the maximum number of iterations is reached, the upper limit of the number of filling iterations is reached, and the filling data is stored and output.

[0155] This step is performed by the random forest iterative imputation module, which is another core innovation of this invention. It solves the problem of low accuracy in missing value imputation in the prior art by using a random forest + iterative optimization method to reconstruct missing values ​​using multivariate interaction information.

[0156] The data initialization sub-step involves initializing the original dataset X=(X1,X2,...) ,X p First, the median is used to initially impute all missing values, resulting in the complete matrix X. (0) Then set the maximum number of iterations T.

[0157] The training sub-step for the random forest model involves fitting the training data using a random forest regressor.

[0158] The convergence determination sub-step involves calculating the difference between two adjacent iterations to determine whether convergence has occurred. For continuous variables, the normalized root mean square error is used as the criterion.

[0159] The data output process is executed by the data output module, which saves the filled data and finally samples and records detailed time information of the feature data and photovoltaic output data used as input to the prediction model at 5-minute intervals, providing reliable input for the subsequent power prediction model and supporting data traceability and operation and maintenance management.

[0160] The beneficial effects of this application include:

[0161] 1. Comprehensive and accurate feature selection, with optimal retention of core features and elimination of redundancy. This invention employs a dual analysis method of Pearson correlation coefficient + MIC, which can capture both linear correlations and uncover nonlinear coupling relationships. The direct technical effect is that no core features are missed, redundant variables are completely eliminated, and the quality of input features is significantly improved. This reduces the computational burden of subsequent models and avoids underfitting caused by missing effective features. The final technical effect is that the comprehensiveness and accuracy of feature selection are greatly improved compared to existing technologies, providing a high-quality input foundation for the prediction model.

[0162] 2. High anomaly detection accuracy with significantly reduced false alarm and missed detection rates. This invention addresses multivariate coupling problems through Mahalanobis distance, adapts to the seasonality and daily periodicity of data using a dual-window approach, and enhances the dynamics and stability of the damped scaling threshold. The direct technical effect is the ability to identify logical anomalies that violate physical laws while adapting to changes in data statistical characteristics across different time periods; the outlier missed detection rate can be reduced by 10% compared to existing technologies, and the effective data retention rate can be increased by 20%. The ultimate technical effect is that the overall accuracy of anomaly detection is improved by more than 30% compared to existing technologies, providing precise markers for filling gaps in the detection process.

[0163] 3. High accuracy in missing value imputation while preserving the complete inter-variable relationship structure: This invention's random forest iterative imputation fully utilizes multivariate interaction information, converging to the optimal imputation result through iterative optimization, without requiring assumptions about data distribution. The direct technical effect is that the relative error between the imputed value and the true value is ≤3%, far lower than existing imputation methods. Simultaneously, it completely preserves the linear and non-linear relationship structure between variables, avoiding model training bias caused by imputation distortion. The final technical effect is that the missing value imputation accuracy is improved by 20% compared to existing technologies, and the data quality meets the input requirements of high-precision prediction models.

[0164] 4. Strong adaptability, compatible with data characteristics of different photovoltaic power plants. The dual-window size, dynamic threshold parameters, and random forest iteration count of this invention can be adaptively adjusted according to the geographical environment, meteorological conditions, and data characteristics of different photovoltaic power plants. The direct technical effect is that the model can adapt to data from different weather types such as sunny, cloudy, and overcast days, as well as the differences between photovoltaic power plants in different regions. The ultimate technical effect is that it can be widely applied to various distributed and centralized photovoltaic power plants, demonstrating strong engineering practicality.

[0165] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to computer program instructions. The aforementioned computer program can be stored in a non-volatile storage medium, and when executed, it performs the steps of the above method embodiments. Alternatively, if the integrated unit of the present invention is implemented as a software functional module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for handling data anomalies in a photovoltaic power prediction-related data acquisition device, characterized in that, The processing method includes: The system acquires and stores multidimensional meteorological data and actual power generation data of photovoltaic systems simultaneously to obtain raw data. The original data is subjected to linear correlation and nonlinear correlation analysis based on the maximum information coefficient to obtain the core input features; The core input features are used to identify and mark abnormal data, and the abnormal data is removed to obtain the data after anomaly removal. The anomaly-removed data is iteratively interpolated to obtain the filled data, which is then output.

2. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that, The simultaneous acquisition and storage of multi-dimensional meteorological data and actual power generation data of the photovoltaic system to obtain raw data includes: Acquire multidimensional meteorological data of the photovoltaic system and the acquisition time of the multidimensional meteorological data; The actual power generation data of the photovoltaic system corresponding to the acquisition time is obtained to obtain synchronously collected multidimensional meteorological data and actual power generation data; The synchronously collected multidimensional meteorological data and actual power generation data are stored to obtain the raw data.

3. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that, The linear correlation and nonlinear correlation analysis of the original data to obtain core input features includes: The original data is preprocessed based on the Z-Score normalization method, and the normalization equation is as follows: ， Where, x i The i-th value represents the data feature; μ represents the mean; σ represents the standard deviation; X std Standard values ​​representing data characteristics; The linear correlation of the preprocessed data is determined by the following equation: ， in, Let x be the correlation coefficient between variables X and Y, with a value range of [-1, 1]. i y i Let represent the i-th data in variables X and Y, respectively. and , where are the average values ​​of variables X and Y respectively, and n is the sample size; Based on mutual information theory, the maximum information coefficient of the preprocessed data is obtained to acquire nonlinear correlations. The equation for the maximum information coefficient is: ， Where I(X, Y) represents mutual information; n x n y B represents the number of grid cells; B represents the upper limit of the number of grid cells. Based on the comparison results of the nonlinear association and linear correlation, the feature with the highest correlation is obtained to obtain the core input feature.

4. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that, The core input features are used to identify and mark abnormal data, and the abnormal data is removed to obtain anomaly-free data, including: A multidimensional feature vector is constructed from the core input features. The multidimensional feature vector is represented as follows: X=[a1,a2, ,a m ] T , Where, a1~a n This represents the core input features; m is the number of core input features; Obtain the Mahalanobis distance of the core input features. The equation for determining the Mahalanobis distance is: ， in, The mean vector of a multidimensional feature vector; ∑ -1 D represents the inverse of the covariance matrix; M (a) j ) represents the j-th core input feature; Based on the dual-window distance detection method, abnormal data within the core input features are obtained; Abnormal data is removed from the core input features to obtain the de-abnormalized data.

5. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 4, characterized in that, The dual-window distance detection method acquires abnormal data within the core input features, including: Set the size of the window to cover the core input features; the larger window W... L Covering several weeks of data in the core input features, small window W S Covers several days of data in the core input features; In the small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within; W in the small window S Large window W L Slide inwards to get the larger window W L All anomalies within.

6. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 5, characterized in that, The small window W S The feature that contributes most to the increase of Mahalanobis distance is obtained within the small window W. S Anomalies within include: In the small window W S Using the mean of Mahalanobis distance as the benchmark, obtain the small window W. S The dynamic threshold for fluctuations exceeding the baseline value is determined by the following equation: ， in, This represents the mean distance within the small window, reflecting the central tendency of the local data. This represents the maximum distance within the small window. S represents the minimum distance within the small window; S(·) represents the scaling function used to shrink the portion exceeding the baseline; t represents the time when the photovoltaic system occurs, which is the core input feature. For small window W S The abnormal data within is identified using the following equation: ， in, The sample x at time t t The results of the anomaly detection; Normal indicates normal data; Anomaly indicates abnormal data; Indicates sample x t Mahalanobis distance; In the outlier data, the feature that contributes the most to the increase in Mahalanobis distance is identified, and the contribution determination equation is as follows: ， Among them, D original D represents the Mahalanobis distance of the original data rows; modified (f) represents the Mahalanobis distance after replacing the value of row feature f with its sliding window mean; C(f) represents the contribution of feature f; f represents the feature.

7. The method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 1, characterized in that, The iterative interpolation process performed on the anomaly-removed data to obtain and output the interpolated data includes: Data after anomaly removal Missing values ​​were imputed using the median to obtain the complete matrix. And set the maximum number of iterations G, where Q represents the original dataset; q1~q p p represents the core input feature vector in the dataset; p represents the number of core input feature vectors. The core input feature vectors are sorted in ascending order of missing rate, and features and labels are constructed. The equations for constructing features and labels are as follows: ， Among them, X s X represents the s-th variable that needs to be filled. -s Indicates that except for X s A matrix consisting of all other variables except I; obs Indicates the row number of valid data in column s; I mis This indicates the row number where data is missing in column s; Represents the input features of the training set; Indicates the target labels of the training set; The input features of the prediction set are represented by s; s represents the variable index. Based on the maximum number of iterations or the normalized root mean square error, determine whether the upper limit of the number of filling iterations has been reached, and store and output the filling data when the upper limit of the number of filling iterations is reached.

8. A method for handling data anomalies in a photovoltaic power prediction-related acquisition device according to claim 7, characterized in that, The step of determining whether the upper limit for the number of filling iterations has been reached based on the maximum number of iterations or the normalized root mean square error, and storing and outputting the filling data when the upper limit for the number of filling iterations is reached, includes: The training data is fitted using a random forest model to handle nonlinear relationships. The random forest algorithm model is as follows: ， Among them, f s This represents the training result of the random forest model; Train() represents the training algorithm; The predicted set is input into the random forest model to obtain the estimated values ​​of the variables, and then input into the matrix. The corresponding position is represented by the equation: ， in, Represents the estimated value of a variable; The difference between two adjacent iterations is obtained to determine whether convergence has occurred. If convergence is achieved, the normalized root mean square error (RMSE) is used for determination. The equation for determining the normalized RMSE is: ， in, This represents the normalized root mean square error result. This indicates the result of the next iteration; This represents the result of the previous iteration; When the normalized root mean square error is less than a preset threshold, or when the maximum number of iterations is reached, the upper limit of the number of filling iterations is reached, and the filling data is stored and output.

Images