A data feature engineering processing method for improving distributed photovoltaic prediction accuracy

Abstract

The application relates to a data feature engineering processing method for improving distributed photovoltaic prediction accuracy, which comprises the following steps: initial data identification based on an iForest algorithm; data reconstruction through super-resolution reconstruction-double-channel convolutional neural network to obtain reconstructed data; correlation analysis of the obtained reconstructed data through a Pearson correlation coefficient; and finding an optimal time offset input into a physical model through correlation analysis and Granger Causality Test (GCT), transforming wind speed and wind direction, and normalizing meteorological data to input data into a data-driven model for processing. The data processing of the physical model in the application is specially used for processing the space-time correlation of field data, and finally, feature generation technology is introduced to realize data feature extraction maximization of the data-driven model; missing data is identified and supplemented, so that the data missing problem of the distributed photovoltaic power station is solved, and the missing data after the supplement has higher correlation with the original data due to the double-channel processing of the neural network.

Description

A Data Feature Engineering Processing Method to Improve the Prediction Accuracy of Distributed Photovoltaic Power Generation Technical Field

[0001] This invention relates to the field of power system forecasting technology, and in particular to a data feature engineering processing method for improving the accuracy of distributed photovoltaic forecasting. Background Technology

[0002] Under pressure from carbon emissions and the depletion of fossil fuels, solar photovoltaic (PV) power generation has become the fastest-growing energy form in recent years. According to the International Energy Agency (IEA), the global PV market reached at least 942 GW in 2021. A large part of this exponential growth is due to the emergence of small-scale, distributed systems on the rooftops of residential and commercial buildings. Distributed PV capacity is projected to reach 530 GW by 2024. The uncertainty of PV power generation negatively impacts grid stability, reliability, and dispatch. Accurate and efficient short-term PV power forecasting plays a crucial role in improving grid stability, economic dispatch, and ensuring power quality.

[0003] Currently, short-term photovoltaic (PV) forecasting methods can be divided into two main categories: physical modeling methods and data-driven methods. Physical modeling methods include "white-box" methods that use analytical equations to describe the internal energy conversion mechanisms of PV systems, while data-driven methods employ statistical and machine learning algorithms. Most current research utilizes data-driven methods.

[0004] With the tremendous growth of photovoltaic (PV) power generation, short-term forecasting methods play a crucial role in the reliable and economical operation of power systems. For distributed PV systems, the diversity of their locations, the variety of meteorological information, and limitations of data acquisition equipment present greater challenges to forecasting methods, as shown in Figure 1. Unlike large-scale, geographically concentrated PV power plants, distributed PV devices are geographically dispersed. Furthermore, communication and monitoring equipment for distributed PV devices is often partially missing, leading to the loss of some power and meteorological data. Therefore, achieving high-precision short-term forecasts for distributed PV is more complex than for PV power plants. Accurate forecasting using either physical models or data-driven models requires relatively high-quality and feature-rich data. However, data gaps are severe in new PV systems, and their distribution is relatively dispersed. Therefore, the accuracy and generalization ability of the aforementioned forecasting tools for distributed PV are significantly insufficient, necessitating the development of a data feature engineering processing method to improve the accuracy of distributed PV forecasting. Summary of the Invention

[0005] To address the shortcomings of missing and scattered distributed photovoltaic (PV) data, the present invention aims to provide a data feature engineering processing method to improve the generalization ability and accuracy of short-term PV forecasting.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a data feature engineering processing method for improving the accuracy of distributed photovoltaic prediction, the method comprising the following sequential steps:

[0007] (1) Initial data identification based on iForest algorithm: The data obtained by the distributed photovoltaic station has anomalies and missing values. The iForest algorithm is used to clean the outliers in the initial data.

[0008] (2) Super-resolution reconstruction - dual-channel convolutional neural network for data reconstruction: The cleaned data still has missing data. The missing data is filled in by super-resolution reconstruction - dual-channel convolutional neural network to obtain the reconstructed data;

[0009] (3) Conduct correlation analysis: Analyze the correlation of the reconstructed data using the Pearson correlation coefficient;

[0010] (4) Data processing for physical model and data-driven model: Find the optimal time offset by correlation analysis and Granger causality test (GCT) and input it into the physical model. Transform wind speed and wind direction and normalize meteorological data and input them into the data-driven model for processing.

[0011] The step (1) specifically refers to the iForest algorithm comprising two stages: the first stage is to construct an isolated forest composed of trees, and the second stage is to determine the degree of anomaly.

[0012] The judgment of the degree of abnormality specifically refers to:

[0013] After obtaining t iTrees, iForest is constructed. In each tree, x is searched for sample points in the sample set. The anomaly index is calculated using formula (1) to determine the degree of anomaly.

[0014]

[0015] In the formula, S(x) is the anomaly index of the detected sample, with a value between 0 and 1; E(h(x)) is the average path length of x to be measured on iTree; and c(v) is the average search path length of the binary tree composed of v points in the training data x.

[0016]

[0017] In the formula, h(x) = ln(x) + ξ, where ξ is Euler's constant;

[0018] Based on formula (1), the following conclusions can be drawn:

[0019] 1) S(x) = 1 indicates that all samples are abnormal;

[0020] 2) S(x) = 0 indicates that there are no outliers in all samples;

[0021] 3) S(x) is between (0,1), indicating that there are no obvious outliers in all samples.

[0022] Step (2) specifically refers to: super-resolution is an underdetermined function that maps defective missing data x to estimated complete data y, denoted as f. θ :x→y, super-resolution mapping f θ Implemented by a convolutional neural network, this network takes the defective data, i.e., short vectors, as input features, with a length of d. f and output length d c d f <d c The estimated complete data, i.e., the long vector, consists of three parts: feature extraction, information supplementation, and reconstruction.

[0023] Given p instances and d f Input of each feature The feature extraction part extracts features from X. The features of each instance are represented by m feature vectors, and each vector has a length of d. f These features It contains abstract feature information of the input X;

[0024] Next, the information supplementation part contains a global residual connection and n local residual blocks, which supplement the missing information into the feature vector. In each local residual block, the convolutional layer performs a non-linear mapping between the input and output feature spaces.

[0025] Finally, the reconstruction part integrates the feature vectors of each instance into α sub-vectors, each of length d. f These subvectors Rearranged into estimated complete data Subvectors are generated in parallel using convolution operations, and the output of the super-resolution reconstruction-dual-channel convolutional neural network is used to estimate the complete data by reconstructing the original data.

[0026] Step (3) specifically refers to: analyzing the correlation between major meteorological factors and photovoltaic output in meteorological data using the Pearson correlation coefficient (PCC).

[0027]

[0028] Where, r XY cov(X,Y) and cov(X,Y) represent the PCC value and covariance of the time series variables, respectively, and σ X and σ Y PCC represents the standard deviation of variables X and Y, and E(·) represents the expected value of the variables. The larger the absolute value of PCC, the stronger the correlation between the variables.

[0029] In step (4), the data processing of the physical model refers to: using the physical distributed photovoltaic prediction sub-model to estimate and predict the starting point of distributed photovoltaic power, including estimating or predicting the power generation of all distributed photovoltaic power generation systems in the considered area, and introducing the optimal time offset method into the physical distributed photovoltaic prediction sub-model based on Granger causality test GCT and Pearson correlation coefficient PCC interpolation, and selecting the optimal time offset physical sub-model.

[0030] The Granger causality test (GCT) is introduced to test the causal relationship between time series variables. If the lagged variable X in the statistical model can explain variable Y, then X is considered a Granger factor of Y, and the direction and value of the time offset are determined by the Granger causality test (GCT).

[0031]

[0032] Where, α i β i These are the coefficients of the unrestricted regression model, ε. i It is univariate white noise, s is the highest lag term, Y t Y represents the current sample. t-i Representing previous samples, the non-causal null hypothesis corresponds to H0: β j =0,j=1,…,s, by comparing the estimation accuracy, the Granger causal relationship between variables Y and X is determined; for each distributed photovoltaic power generation system, the optimal value of the highest lag term s is determined using the Akayk criterion;

[0033] According to the Granger causality test (GCT), total radiation data is randomly selected: First, the total radiation data with a 15-minute time interval is densified using cubic spline interpolation to generate dense data with a 5-minute time interval; by moving the dense total radiation data left and right, the Pearson correlation coefficient (PCC) with photovoltaic power is calculated sequentially, moving one point at a time. The direction and value of the offset are determined by the Granger causality test (GCT), and the time offset corresponding to the maximum Pearson correlation coefficient (PCC) value is selected as the optimal time offset.

[0034] The data-driven data processing refers to:

[0035]

[0036]

[0037] Among them, D sin D cos Y represents the daily cyclical characteristics. sin Y cos Representing the cyclical characteristics of each year, t dur Indicates the duration from a certain point in time to the current point in time;

[0038] Wind speed V W Wind direction D W The variables are converted into the horizontal and vertical components W of the wind information. x W y As shown below:

[0039]

[0040] Before input data drives the model, all types of data are normalized. Finally, all numerical weather prediction (NWP) data undergoes feature scaling, using interval scaling to normalize the input features. All feature values ​​are scaled to the interval [-1, 1].

[0041]

[0042] In the formula, x r x represents the r-th sample value of feature r. max and x min x′ represents the maximum and minimum values ​​of feature x, respectively. r This represents the normalized value.

[0043] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the collected data is preprocessed by identification, interpolation, and then a super-resolution reconstruction-dual-channel convolutional neural network is used to process the removed or missing data; Second, the data processing of the physical model in the present invention is specifically used to handle the spatiotemporal correlation of field data, in which Granger causality test is used, and finally feature generation technology is introduced to maximize the data feature extraction of the data-driven model; Third, since the missing data is identified and interpolated by the super-resolution reconstruction-dual-channel convolutional neural network, the data missing problem of distributed photovoltaic power stations is made up for, because the dual-channel processing of the neural network makes the interpolated missing data highly correlated with the original data; Fourth, distributed photovoltaics have a wide distribution range and poor generalization ability, so Granger causality test and Pearson correlation coefficient are added to the physical model prediction to effectively solve the problem of the spatiotemporal nature of distributed photovoltaic power stations; Fifth, feature generation technology is added to the data-driven prediction, which maximizes the extraction of data features of photovoltaic stations in different regions and improves the generalization ability of the prediction. Attached Figure Description

[0044] Figure 1 is a schematic diagram of distributed photovoltaic power generation.

[0045] Figure 2 is a flowchart of iForest's data cleaning process;

[0046] Figure 3 is a schematic diagram illustrating the correlations between different features in the dataset. Detailed Implementation

[0047] A data feature engineering processing method for improving the accuracy of distributed photovoltaic forecasting includes the following sequential steps:

[0048] (1) Initial data identification based on iForest algorithm: The data obtained by the distributed photovoltaic station has anomalies and missing values. The iForest algorithm is used to clean the outliers in the initial data.

[0049] (2) Super-resolution reconstruction - dual-channel convolutional neural network for data reconstruction: The cleaned data still has missing data. The missing data is filled in by super-resolution reconstruction - dual-channel convolutional neural network to obtain the reconstructed data;

[0050] (3) Conduct correlation analysis: Analyze the correlation of the reconstructed data using the Pearson correlation coefficient;

[0051] (4) Data processing for physical model and data-driven model: Find the optimal time offset by correlation analysis and Granger causality test (GCT) and input it into the physical model. Transform wind speed and wind direction and normalize meteorological data and input them into the data-driven model for processing.

[0052] The iForest algorithm, proposed by Liu et al., is an unsupervised anomaly detection algorithm suitable for continuous data, used to detect and discover outliers. Furthermore, the Isolation Forest algorithm boasts high computational efficiency and accuracy, is sensitive to globally sparse points, and is applicable to high-dimensional data and large datasets. It is suitable for handling outliers in wind and photovoltaic power generation measurement data.

[0053] As shown in Figure 2, step (1) specifically refers to the iForest algorithm comprising two stages: the first stage is to construct an isolated forest composed of trees, and the second stage is to determine the degree of anomaly.

[0054] The judgment of the degree of abnormality specifically refers to:

[0055] After obtaining t iTrees, iForest is constructed. In each tree, x is searched for sample points in the sample set. The anomaly index is calculated using formula (1) to determine the degree of anomaly.

[0056]

[0057] In the formula, S(x) is the anomaly index of the detected sample, with a value between 0 and 1; E(h(x)) is the average path length of x to be measured on iTree; and c(v) is the average search path length of the binary tree composed of v points in the training data x.

[0058]

[0059] In the formula, h(x) = ln(x) + ξ, where ξ is Euler's constant;

[0060] Based on formula (1), the following conclusions can be drawn:

[0061] 1) S(x) = 1 indicates that all samples are abnormal;

[0062] 2) S(x) = 0 indicates that there are no outliers in all samples;

[0063] 3) S(x) is between (0,1), indicating that there are no obvious outliers in all samples.

[0064] Step (2) specifically refers to: super-resolution is an underdetermined function that maps defective missing data x to estimated complete data y, denoted as f. θ :x→y, super-resolution mapping f θ Implemented by a convolutional neural network, this network takes the defective data, i.e., short vectors, as input features, with a length of d. f and output length dc d f <d c The estimated complete data, i.e., the long vector, consists of three parts: feature extraction, information supplementation, and reconstruction.

[0065] Given p instances and d f Input of each feature The feature extraction part extracts features from X. The features of each instance are represented by m feature vectors, and each vector has a length of d. f These features It contains abstract feature information of the input X;

[0066] Next, the information supplementation part contains a global residual connection and n local residual blocks, which supplement the missing information into the feature vector. In each local residual block, the convolutional layer performs a non-linear mapping between the input and output feature spaces.

[0067] Finally, the reconstruction part integrates the feature vectors of each instance into α sub-vectors, each of length d. f These subvectors Rearranged into estimated complete data Subvectors are generated in parallel using convolution operations, and the output of the super-resolution reconstruction-dual-channel convolutional neural network is used to estimate the complete data by reconstructing the original data.

[0068] Like many other renewable energy sources, photovoltaic power is highly dependent on weather conditions. Step (3) specifically refers to: analyzing the correlation between major meteorological factors in the meteorological data and photovoltaic output using the Pearson correlation coefficient (PCC).

[0069]

[0070] Where, r XY cov(X,Y) and cov(X,Y) represent the PCC value and covariance of the time series variables, respectively, and σ X and σ Y Figure 3 shows the correlation coefficient between numerical weather prediction (NWP) and photovoltaic power. The standard deviations of variables X and Y are represented by , and E(·) represents the expected value of the variables. The larger the absolute value of PCC, the stronger the correlation between the variables.

[0071] In step (4), the data processing of the physical model refers to: using the physical distributed photovoltaic prediction sub-model to estimate and predict the starting point of distributed photovoltaic power, including estimating or predicting the power generation of all distributed photovoltaic power generation systems in the considered area, and introducing the optimal time offset method into the physical distributed photovoltaic prediction sub-model based on Granger causality test GCT and Pearson correlation coefficient PCC interpolation, and selecting the optimal time offset physical sub-model.

[0072] The starting point for using physical sub-models to estimate and predict distributed photovoltaic (PV) power is the so-called bottom-up strategy. This involves estimating or predicting the power generation of all distributed PV systems within the considered area. Distributed PV power generation exhibits strong spatiotemporal correlations within adjacent areas. In most cases, the number of meteorological measurement points is far less than the number of distributed PV systems. Sometimes only numerical weather prediction (NWP) data is available. Optimizing the time offset of the raw meteorological data helps improve the accuracy and reliability of the physical model, as it allows for a more appropriate match between the meteorological data and the physical model.

[0073] The Granger causality test (GCT) is introduced to test the causal relationship between time series variables. If the lagged variable X in the statistical model can explain variable Y, then X is considered a Granger factor of Y, and the direction and value of the time offset are determined by the Granger causality test (GCT).

[0074]

[0075] Where, α i β i These are the coefficients of the unrestricted regression model, ε. i It is univariate white noise, s is the highest lag term, Y t Y represents the current sample. t-i Representing previous samples, the non-causal null hypothesis corresponds to H0: β j =0,j=1,…,s, by comparing the estimation accuracy, the Granger causal relationship between variables Y and X is determined; for each distributed photovoltaic power generation system, the optimal value of the highest lag term s is determined using the Akayk criterion;

[0076] According to the Granger causality test (GCT), total radiation data is randomly selected: First, the total radiation data with a 15-minute time interval is densified using cubic spline interpolation to generate dense data with a 5-minute time interval; by moving the dense total radiation data left and right, the Pearson correlation coefficient (PCC) with photovoltaic power is calculated sequentially, moving one point at a time. The direction and value of the offset are determined by the Granger causality test (GCT), and the time offset corresponding to the maximum Pearson correlation coefficient (PCC) value is selected as the optimal time offset.

[0077] Feature generation (FG) is a novel feature engineering technique applied to distributed photovoltaic (PV) power prediction. The goal of FG is to extract independent features from raw data for data-driven model learning. During feature generation, the periodicity of timestamps and wind information from weather data are encoded. Raw timestamp data, such as year, month, day, hour, and minute information, is not suitable as input encoding. Therefore, it is re-encoded based on daily and yearly periodicity to help the data-driven model improve its ability to capture long-term dependencies.

[0078] The data-driven data processing refers to:

[0079]

[0080]

[0081] Among them, D sin D cos Y represents the daily cyclical characteristics. sin Y cos Representing the cyclical characteristics of each year, t dur Indicates the duration from a certain point in time to the current point in time;

[0082] Wind speed V W Wind direction D W The variables are converted into the horizontal and vertical components W of the wind information. x W y As shown below:

[0083]

[0084] Before input data drives the model, all types of data are normalized. Finally, all numerical weather prediction (NWP) data undergoes feature scaling, using interval scaling to normalize the input features. All feature values ​​are scaled to the interval [-1, 1].

[0085]

[0086] In the formula, x r x represents the r-th sample value of feature r. max and x min x′ represents the maximum and minimum values ​​of feature x, respectively. r This represents the normalized value.

[0087] In summary, this invention first identifies, imputes, and preprocesses the collected data, then uses a super-resolution reconstruction-dual-channel convolutional neural network to process the removed or missing data. The data processing of the physical model in this invention is specifically designed to handle the spatiotemporal correlation of field data, employing Granger causality testing. Finally, feature generation technology is introduced to maximize the extraction of data features from the data-driven model. Because the super-resolution reconstruction-dual-channel convolutional neural network identifies and imputes missing data, it compensates for the data loss problem in distributed photovoltaic power stations. Due to the dual-channel processing of this neural network, the imputed missing data has a high correlation with the original data.

Claims

1. A data feature engineering processing method for improving the accuracy of distributed photovoltaic forecasting, characterized in that: The method includes the following sequential steps: (1) Initial data identification based on the iForest algorithm: The data obtained from the distributed photovoltaic station has anomalies and missing values. The iForest algorithm is used to clean the anomalies in the initial data; (2) Data reconstruction using super-resolution reconstruction-dual-channel convolutional neural network: The cleaned data still has missing values. The super-resolution reconstruction-dual-channel convolutional neural network is used to interpolate the missing data to obtain the reconstructed data; (3) Correlation analysis: The Pearson correlation coefficient is used to analyze the correlation of the reconstructed data; (4) Data processing of physical model and data-driven model: The optimal time offset is found by correlation analysis and Granger causality test (GCT) and input into the physical model to process the wind. The speed and wind direction are transformed and the meteorological data is normalized and input data is used to drive the model for processing; in step (4), the data processing of the physical model refers to: using the physical distributed photovoltaic prediction sub-model to estimate and predict the starting point of distributed photovoltaic power, including estimating or predicting the power generation of all distributed photovoltaic power generation systems in the considered area, and introducing the optimal time offset method into the physical distributed photovoltaic prediction sub-model based on Granger causality test GCT and Pearson correlation coefficient PCC interpolation, and selecting the optimal time offset physical sub-model; introducing Granger causality test GCT to test the causal relationship between time series variables, if the lagged variable X in the statistical model can explain the variable Y, then X is considered to be a Granger factor of Y, and the direction and value of the time offset are determined by Granger causality test GCT: (6) Among them, 、 These are the coefficients of an unrestricted regression model. It is univariate white noise, where s is the highest lag term. Indicates the current sample, Representing previous samples, the non-causal null hypothesis corresponds to By comparing estimation accuracy, the Granger causality between variables Y and X is determined; for each distributed photovoltaic power generation system, the optimal value of the highest lag term s is determined using the Akaik criterion; according to the Granger causality test (GCT), total radiation data is randomly selected: first, the total radiation data with 15-minute time intervals is densified using cubic spline interpolation to generate dense data with 5-minute time intervals; by moving the dense total radiation data left and right, the Pearson correlation coefficient (PCC) with photovoltaic power is calculated sequentially, moving one point each time, with the direction and value of the offset determined by the Granger causality test (GCT), and the time offset corresponding to the maximum Pearson correlation coefficient (PCC) value is selected as the optimal time offset; the data-driven data processing refers to: , (7) , (8) Among them, ， It represents the cyclical characteristics of each day. ， Represents the cyclical characteristics of each year. Indicates the duration from a certain point in time to the current point in time; wind speed and wind direction The variables were converted into horizontal and vertical components of the wind information. ， As shown below: , (9) Before input data drives the model, all types of data are normalized. Finally, all numerical weather prediction (NWP) data undergoes feature scaling, using interval scaling to normalize the input features. All feature values ​​are scaled to the interval [-1, 1]. In equation (10), The r-th sample value represents feature r. and These represent the maximum and minimum values ​​of feature x, respectively. This represents the normalized value.

2. The data feature engineering processing method for improving the accuracy of distributed photovoltaic prediction according to claim 1, characterized in that: Step (1) specifically refers to the following: The iForest algorithm includes two stages: the first stage is to construct an isolated forest composed of trees, and the second stage is to determine the degree of anomaly; the degree of anomaly determination specifically refers to: after obtaining t iTrees, constructing iForest, searching for x (i.e., sample points in the sample set) in each tree, and calculating the anomaly index using formula (1) to determine the degree of anomaly: (1) In the formula, It is the anomaly index of the detected samples, with a value between 0 and 1. The requirement is to measure the average path length of x on iTree. The average search path length of the binary tree composed of v points in the training data x is: (2) In the formula, ， It is Euler's constant; according to formula (1), the following conclusions are drawn: 1) =1 indicates that all samples are abnormal; 2) =0 indicates that there are no outliers in all samples; 3) A value between (0,1) indicates that there are no obvious outliers in all samples.

3. The data feature engineering processing method for improving the accuracy of distributed photovoltaic prediction according to claim 1, characterized in that: Step (2) specifically refers to: super-resolution is an underdetermined function that maps defective missing data x to estimated complete data y, expressed as: Super-resolution mapping Implemented by a convolutional neural network, this network takes the defective data, i.e., short vectors, as input features, with a length of [missing information]. and output length is ， < The estimated complete data, i.e., the long vector, consists of three parts: feature extraction, information supplementation, and reconstruction; given a... One instance and Input of each feature The feature extraction part extracts features from X. The features of each instance are represented by m feature vectors, and the length of each vector is m. These features The first part contains abstract feature information of the input X; then, the information supplementation part contains a global residual connection and n local residual blocks to supplement the missing information into the feature vector. In each local residual block, the convolutional layer performs a non-linear mapping between the input and output feature spaces; finally, the reconstruction part integrates the feature vectors of each instance into a single feature vector. There are n subvectors, each with a length of n. These subvectors Rearranged into estimated complete data Subvectors are generated in parallel using convolution operations, and the output of the super-resolution reconstruction-dual-channel convolutional neural network is estimated from the reconstruction of the original data.

4. The data feature engineering processing method for improving the accuracy of distributed photovoltaic prediction according to claim 1, characterized in that: Step (3) specifically refers to: analyzing the correlation between major meteorological factors and photovoltaic output in meteorological data using the Pearson correlation coefficient (PCC). (5) Among them, and These represent the PCC value and covariance of the time series variable, respectively. and Representing variables standard deviation PCC represents the expected value of a variable. The larger the absolute value of PCC, the stronger the correlation between the variables.

Images