Data identification and truth value compensation method for new energy prediction
By combining the segmented two-way quartering method with a neural network, outliers in renewable energy power generation data are identified and eliminated, and true value compensation is performed, which solves the problem of uneven data quality in renewable energy power generation forecasting and improves the forecast accuracy and reliability.
Patent Information
- Application Number
- CN202510734339.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-04
- Publication Date
- 2025-09-19
AI Technical Summary
In existing renewable energy power generation prediction technologies, the quality of raw data varies greatly, traditional abnormal data identification methods are difficult to adapt to complex working conditions, and time series prediction models are highly sensitive to abnormal data, resulting in accumulated prediction errors and affecting prediction accuracy and reliability.
The segmented two-direction quartering method is used to identify and eliminate outliers, combined with neural networks for true value compensation, and the BP neural network training model is used for data filling and anomaly identification to improve data availability.
It improves the data accuracy and reliability of new energy power generation forecasts, effectively corrects abnormal data and reduces forecast errors.
Smart Images
Figure CN120675035A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power systems, and in particular relates to a data identification and true value compensation method for new energy prediction. Background Art
[0002] As the global energy mix shifts toward cleaner, low-carbon energy, the share of renewable energy generation, represented by photovoltaic and wind power, continues to rise. However, renewable energy generation is affected by multiple factors, including weather conditions and equipment operating conditions, resulting in significant volatility and uncertainty in output. This poses significant challenges to power system scheduling, market transactions, and safe and stable operation.
[0003] Existing new energy power prediction technologies generally face the following problems:
[0004] 1. Raw data quality varies. Factors such as sensor noise, communication anomalies, and extreme weather conditions lead to a large number of outliers and missing values in historical data. Traditional abnormal data identification methods based on threshold judgment or statistical filtering are difficult to adapt to the multidimensional data characteristics under complex working conditions and are prone to loss of effective information.
[0005] 2. Time series prediction models are highly sensitive to abnormal data. Existing compensation technologies mostly use linear interpolation or mean filling, which fails to fully consider the spatiotemporal correlation and nonlinear dynamic evolution of renewable energy output sequences, resulting in the accumulation of prediction errors along the time dimension.
[0006] The above problems restrict the generalization ability and engineering practicality of the prediction model. Therefore, developing a new compensation method that takes into account both data identification accuracy and computational efficiency has become the key to improving the reliability of new energy prediction. Summary of the Invention
[0007] The problem to be solved by the present invention is to provide a data identification and true value compensation method for new energy prediction, which can improve the availability of data and enhance the accuracy of new energy prediction.
[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a data identification and true value compensation method for new energy prediction, comprising the following steps:
[0009] S1: Input photovoltaic power generation data or wind power generation data;
[0010] S2: Set the main independent variables and dependent variables;
[0011] S3: Identify and remove missing values and outliers to obtain data 1;
[0012] S4: If the data is used for regression prediction, the output data 1 is used for regression prediction; if the data is used for time series prediction, the missing values and outliers are compensated by the true value of the neural network.
[0013] Furthermore, in S1, the data volume is historical data for at least one year, and the data resolution is one point per hour.
[0014] Furthermore, in S1, the photovoltaic power generation data includes independent variables and dependent variables, the independent variables of the photovoltaic power generation data include temperature, humidity, wind speed and light intensity, and the dependent variables of the photovoltaic power generation data include photovoltaic power; the independent variables of the wind power generation data include wind speed, wind direction and temperature, and the dependent variables of the wind power generation data include wind power.
[0015] Furthermore, in S2, the main independent variable of the photovoltaic power generation data is light intensity, and the main dependent variable of the photovoltaic power generation data is photovoltaic output power; the main independent variable of the wind power generation data is wind speed, and the main dependent variable of the wind power generation data is wind power output power.
[0016] Furthermore, the S3 includes the following steps:
[0017] S31: Use Python to identify missing points in the data file, delete the rows with missing data, and delete the rows where the value of the main independent variable is less than or equal to 0;
[0018] S32: For the new energy prediction problem, the quartile method is improved to a segmented two-way quartile method to identify outliers in the direction of the main independent variables and dependent variables.
[0019] Furthermore, the S32 includes the following steps:
[0020] S321: Sort the data according to the size of the main independent variable value, slice the sorted data, and calculate the number of slices according to the following formula:
[0021]
[0022] Where N slice is the number of slices, is the rounding symbol, k is the segment density coefficient, y gap is the data interval, and its formula is as follows:
[0023]
[0024] Where y v_max ,y v_min is the maximum and minimum value of the dependent variable, N sum is the total amount of data;
[0025] S322: After completing the first data slice, for the i-th data slice, calculate the median m of the dependent variable i , lower quartile m1i and the upper quartile m 3i , calculate the interquartile range G mi , the formula is as follows, G mi =m 3i -m 1i
[0026] The normal range of the slice data is:
[0027] range=[m 1i -1.5*G mi ,m 3i +1.5*G mi ]
[0028] Mark the points outside the normal range in each slice as abnormal;
[0029] S323: Reorganize the data and sort the data according to the size of the dependent variable. Calculate the number of slices according to the following formula:
[0030]
[0031] Where x v_max , x v_min is the maximum and minimum value of the main independent variable;
[0032] Slice the data twice, and use the method in S322 to identify outliers based on the median, lower quartile, and upper quartile of the main independent variable for each slice, and mark the abnormal data;
[0033] S324: Delete the data marked as abnormal.
[0034] Furthermore, the S4 includes the following steps:
[0035] S41: Fill missing values by interpolating and supplementing the missing values in each column of the original data to obtain data 2;
[0036] S42: Introduce the classic BP neural network, use the independent variables in data 1 as the input of the BP neural network, and the dependent variables as the output of the BP neural network for training, and obtain the training model model_predict;
[0037] S43: Perform outlier identification. For data 2, use the segmented two-way quartering method and the neural network method to mark abnormal situations respectively. The data that are determined as abnormal points by both methods are defined as abnormal data.
[0038] S44: Perform true value compensation for abnormal data, and determine the point as abnormal data. The dependent variable is y predict Replace, complete the true value compensation, and obtain data 3;
[0039] S45: Output data 3 for time series prediction.
[0040] Furthermore, the S43 includes the following steps:
[0041] S431: According to the steps of S321-S323, the abnormal data is marked by the segmented two-direction four-division method;
[0042] S432: Input the independent variable of data 2 into the model_predict model to obtain the predicted value y of all data predict , compare the dependent variable of each data with the predicted value, obtain the relative error value, and use the neural network method to mark abnormal data;
[0043] S433: The data points marked as abnormal by both methods are identified as abnormal data.
[0044] Furthermore, in S432, the calculation formula of the relative error value is as follows:
[0045]
[0046] Points with an MSE less than 20% are marked as outliers.
[0047] The advantages and positive effects of the present invention are:
[0048] To address the data distortion problem during photovoltaic or wind power generation prediction, the present invention proposes a data identification method based on the segmented two-direction quartering method for regression prediction; based on the regression prediction data, for the time series prediction problem, a neural network-based abnormal data identification and true value compensation method is proposed. This method can improve data availability and enhance the accuracy of new energy prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is an overall flow chart of an embodiment of the present invention.
[0050] Figure 2 3 is a comparison chart of the original data of the embodiment of the present invention and the data after processing by this solution. DETAILED DESCRIPTION
[0051] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0052] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0053] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.
[0054] The embodiments of the present invention are further described below with reference to the accompanying drawings:
[0055] like Figure 1 As shown, a data identification and true value compensation method for new energy prediction includes the following steps:
[0056] S1: Input photovoltaic power generation data or wind power generation data. Specifically, the data volume is at least one year of historical data, and the data resolution is one point per hour. In this embodiment, the data volume is one year of historical data for a specific location, and the data resolution is one point per hour. The photovoltaic power generation data includes independent variables and dependent variables. The independent variables of the photovoltaic power generation data include temperature, humidity, wind speed, and light intensity, and the dependent variable of the photovoltaic power generation data includes photovoltaic power. The independent variables of the wind power generation data include wind speed, wind direction, and temperature, and the dependent variable of the wind power generation data includes wind power.
[0057] S2: Set the main independent variables and dependent variables. Specifically, the main independent variable for photovoltaic power generation data is sunlight intensity, and the main dependent variable for photovoltaic power generation data is photovoltaic output power. The main independent variable for wind power generation data is wind speed, and the main dependent variable for wind power generation data is wind power output power.
[0058] S3: Identify and remove missing values and outliers to obtain data 1. Specifically, S3 includes the following steps:
[0059] S31: Use Python to identify missing points in the data file, delete the rows with missing data, and delete the rows where the value of the main independent variable is less than or equal to 0.
[0060] S32: For the new energy prediction problem, the quartile method is improved to a segmented two-way quartile method to identify outliers in the direction of the main independent variables and dependent variables. Specifically, S32 includes the following steps:
[0061] S321: Sort the data according to the size of the main independent variable value, slice the sorted data, and calculate the number of slices according to the following formula:
[0062]
[0063] Where N slice is the number of slices, is the rounding symbol, k is the segment density coefficient, which is 10 in this embodiment, and y gap is the data interval, and its formula is as follows:
[0064]
[0065] Where y v_max ,y v_min is the maximum and minimum value of the dependent variable, N sum The total amount of data.
[0066] S322: After completing the first data slice, for the i-th data slice, calculate the median m of the dependent variable i , lower quartile m 1i and the upper quartile m 3i , calculate the interquartile range G mi , the formula is as follows, G mi =m 3i -m 1i
[0067] The normal range of the slice data is:
[0068] range=[m 1i -1.5*G mi ,m 3i +1.5*G mi ]
[0069] Each point outside the normal range in each slice is marked as abnormal.
[0070] S323: Reorganize the data and sort the data according to the size of the dependent variable. Calculate the number of slices according to the following formula:
[0071]
[0072] Where x v_max , x v_min are the maximum and minimum values of the main independent variable.
[0073] The data is sliced twice, and the method in S322 is used to identify outliers for each slice based on the median, lower quartile, and upper quartile of the main independent variable, and mark abnormal data.
[0074] S324: Delete the data marked as abnormal.
[0075] At this point, the missing values and outliers in the original data have been removed.
[0076] S4: If the data is used for regression prediction, then output data 1 for regression prediction; if the data is used for time series prediction, in order to maintain the time series of the data, missing values and outliers are compensated by the true value of the neural network and cannot be directly deleted. Specifically, S4 includes the following steps:
[0077] S41: Fill missing values. Fill missing values in the original data by interpolation for each column to obtain data 2.
[0078] S42: Introduce the classic BP neural network, use the independent variables in data 1 as the input of the BP neural network, and the dependent variables as the output of the BP neural network for training, and obtain the training model model_predict.
[0079] S43: Perform outlier identification. For data 2, use the segmented two-way quartering method and the neural network method to mark abnormal situations respectively. The data that are determined as abnormal points by both methods are defined as abnormal data. Specifically, S43 includes the following steps:
[0080] S431: According to steps S321-S323, perform segmented two-direction quartering method to mark abnormal data.
[0081] S432: Input the independent variable of data 2 into the model_predict model to obtain the predicted value y of all data predict , compare the dependent variable of each data with the predicted value, get the relative error value, and use the neural network method to mark abnormal data. The calculation formula of the relative error value is as follows:
[0082]
[0083] Points with an MSE less than 20% are marked as outliers.
[0084] S433: The data points marked as abnormal by both methods are identified as abnormal data.
[0085] S44: Perform true value compensation for abnormal data. For the points determined to be abnormal data, the dependent variable is replaced with ypredict to complete the true value compensation and obtain data 3.
[0086] S45: Output data 3 for time series prediction.
[0087] Taking wind power generation data as an example, this embodiment is simulated and verified to obtain a comparison chart between the original data and the processed data of this embodiment, as shown in FIG. Figure 2 As shown. Figure 2 It can be seen that the original data contains a large number of unreasonable abnormal points, such as wind speeds between -200 and 0 and points after 100, or the situation where the wind power of some data points at the same wind speed has a large difference. However, the data processed by this embodiment show an obvious positive correlation between wind power and wind speed, and the wind power at the same wind speed is distributed within a reasonable range, which is in line with common sense. This shows that the data identification and true value compensation method of this embodiment can effectively screen and correct abnormal data, proving the effectiveness and feasibility of this embodiment.
[0088] The advantages and positive effects of the present invention are:
[0089] To address the data distortion problem during photovoltaic or wind power generation prediction, the present invention proposes a data identification method based on the segmented two-direction quartering method for regression prediction; based on the regression prediction data, for the time series prediction problem, a neural network-based abnormal data identification and true value compensation method is proposed. This method can improve data availability and enhance the accuracy of new energy prediction.
[0090] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.
Claims
1. A data identification and true value compensation method for new energy prediction, characterized by: The following steps are included: S1: Input photovoltaic power generation data or wind power generation data; S2: Set the main independent variables and dependent variables; S3: Identify and remove missing values and outliers to obtain data 1; S4: If the data is used for regression prediction, the output data 1 is used for regression prediction; if the data is used for time series prediction, the missing values and outliers are compensated by the true value of the neural network.
2. The data identification and true value compensation method for new energy prediction according to claim 1, characterized in that: In S1, the data volume is historical data of at least one year, and the data resolution is one point per hour.
3. The data identification and true value compensation method for new energy prediction according to claim 1 or 2, characterized in that: In S1, the photovoltaic power generation data includes independent variables and dependent variables. The independent variables of the photovoltaic power generation data include temperature, humidity, wind speed and light intensity, and the dependent variables of the photovoltaic power generation data include photovoltaic power; the independent variables of the wind power generation data include wind speed, wind direction and temperature, and the dependent variables of the wind power generation data include wind power.
4. The data identification and true value compensation method for new energy prediction according to claim 3 is characterized by: In S2, the main independent variable of the photovoltaic power generation data is light intensity, and the main dependent variable of the photovoltaic power generation data is photovoltaic output power; the main independent variable of the wind power generation data is wind speed, and the main dependent variable of the wind power generation data is wind power output power.
5. The data identification and true value compensation method for new energy prediction according to claim 1 or 2, characterized in that: Said S3 comprises the following steps, S31: Use Python to identify missing points in the data file, delete the rows with missing data, and delete the rows where the value of the main independent variable is less than or equal to 0; S32: For the new energy prediction problem, the quartile method is improved to a segmented two-way quartile method to identify outliers in the direction of the main independent variables and dependent variables.
6. The data identification and true value compensation method for new energy prediction according to claim 5, characterized in that: The S32 The following steps are included: S321: Sort the data according to the size of the main independent variable value, slice the sorted data, and calculate the number of slices according to the following formula: Where N slice is the number of slices, is the rounding symbol, k is the segment density coefficient, y gap is the data interval, and its formula is as follows: Where y v_max ,y v_min is the maximum and minimum value of the dependent variable, N sum is the total amount of data; S322: After completing the first data slice, for the i-th data slice, calculate the median m of the dependent variable i , lower quartile m 1i and the upper quartile m 3i , calculate the interquartile range G mi , the formula is as follows, G mi =m 3i -m 1i The normal range of the slice data is: range=[m 1i -1.5*G mi ,m 3i +1.5*G mi ] Mark the points outside the normal range in each slice as abnormal; S323: Reorganize the data and sort the data according to the size of the dependent variable. Calculate the number of slices according to the following formula: Where x v_max , x v_min is the maximum and minimum value of the main independent variable; Slice the data twice, and use the method in S322 to identify outliers based on the median, lower quartile, and upper quartile of the main independent variable for each slice, and mark the abnormal data; S324: Delete the data marked as abnormal.
7. The data identification and true value compensation method for new energy prediction according to claim 6, characterized in that: Said S4 comprises the following steps, S41: Fill missing values by interpolating and supplementing the missing values in each column of the original data to obtain data 2; S42: Introduce the classic BP neural network, use the independent variables in data 1 as the input of the BP neural network, and the dependent variables as the output of the BP neural network for training, and obtain the training model model_predict; S43: Perform outlier identification. For data 2, use the segmented two-way quartering method and the neural network method to mark abnormal situations respectively. The data that are determined as abnormal points by both methods are defined as abnormal data. S44: Perform true value compensation for abnormal data, and determine the point as abnormal data. The dependent variable is y predict Replace, complete the true value compensation, and obtain data 3; S45: Output data 3 for time series prediction.
8. The data identification and true value compensation method for new energy prediction according to claim 7, characterized in that: The S43 The following steps are included: S431: According to the steps of S321-S323, the abnormal data is marked by the segmented two-direction four-division method; S432: Input the independent variable of data 2 into the model_predict model to obtain the predicted value y of all data predict , compare the dependent variable of each data with the predicted value, obtain the relative error value, and use the neural network method to mark abnormal data; S433: The data points marked as abnormal by both methods are identified as abnormal data.
9. The data identification and true value compensation method for new energy prediction according to claim 8, characterized in that: In the S432, the calculation formula of the relative error value is as follows: Points with an MSE less than 20% are marked as outliers.