Method for predicting power consumption condition and diagnosing abnormal data by using self-adaptive ARIMA (autoregressive integrated moving average) model

Abstract

The invention discloses a method for predicting a power consumption condition by using a self-adaptive ARIMA model. The method comprises the following steps: S1, collecting data; s2, data processing; s3, constructing and training a model; s4, electricity price and demand prediction; and S5, performing model evaluation. The invention further discloses a power utilization condition abnormal data diagnosis method which is applied to the method for predicting the power utilization condition through the self-adaptive ARIMA model. According to the method for predicting the electricity consumption condition, the ARIMA model is used for predicting the electricity price and the demand, the ARIMA model is applied to the electricity price and demand data in the electricity market so as to predict the electricity price and the demand in the future, the model parameters are dynamically adjusted according to the historical data and the current market condition, and the prediction accuracy and stability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power market, and in particular to a method for predicting power consumption status and abnormal data diagnosis by using an adaptive ARIMA model. Background Art

[0002] With the development of the global economy, the importance of the power industry has attracted more and more attention. However, the uncertainty and complexity of the power market make it difficult to predict electricity prices and demand. The existing methods for predicting electricity prices and demand have certain defects and limitations, such as low prediction accuracy and long prediction time.

[0003] Therefore, it is necessary to design a method for predicting power consumption and abnormal data diagnosis using an adaptive ARIMA model to solve the problems of low prediction accuracy and long prediction time in existing prediction technologies. Summary of the invention

[0004] The present invention aims to solve the shortcomings in the prior art and proposes a method for predicting power consumption and abnormal data diagnosis using an adaptive ARIMA model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A method for predicting power consumption using an adaptive ARIMA model specifically comprises the following steps: S1. Data collection: Collect the current historical power data, real-time power data and website predicted power data of the target power market through the target website; S2. Data processing: After normalizing the historical power data to between 0 and 1, the data is divided into a training set and a test set, and the time series data is converted into supervised learning data; S3, model construction and training: construct an ARIMA model, perform model training on the training set, and according to the minimum information criterion, first cycle to obtain the best P and Q values, and then cycle to obtain the best D value, where P represents the order of the AR model, Q represents the order of the MA model, and D represents the number of differences; S4. Electricity price and demand forecast: Use the trained ARIMA model, use the test set to make forecasts and restore the forecast results to data; S5. Model evaluation: Calculate the prediction error of the ARIMA model and visualize the prediction results.

[0006] Furthermore, the power data includes power price data and power demand data.

[0007] Furthermore, according to the minimization of information criterion, first looping to obtain the best P and Q values, and then looping to obtain the best D value is: using an ARMA model with a maximum P value of M1 and a maximum Q value of M2, the P and Q values ​​are traversed from 1 to their respective maximum values, and according to the minimization of information criterion, the best order of the ARMA model, that is, the best P and Q values, is selected, wherein M1 is a positive integer from 8 to 15, and M2 is a positive integer from 8 to 15; setting the range of D to 0 to N, writing a for loop to combine the best P, Q and D values ​​and substitute them into model.fit(), and obtaining the best D value according to the minimization of information criterion; wherein N is a positive integer greater than or equal to 4.

[0008] Furthermore, the minimization information criterion is an AIC information criterion.

[0009] Furthermore, the calculating of the prediction error of the ARIMA model includes comparing the prediction results of the ARIMA model with the real-time power data to obtain corresponding prediction errors.

[0010] Furthermore, the model evaluation index includes one or more of root mean square error, mean square error, mean absolute error, and R square.

[0011] Furthermore, the prediction refers to a prediction for the next day.

[0012] Furthermore, the historical power data refers to the power data of Y days in history at the current time, where Y=3, 4, 5, 6, 7, 8.

[0013] The present invention also discloses a method for diagnosing abnormal data of power usage conditions. In the method for predicting power usage conditions, a box line method or a cap method is used to diagnose and process abnormal data in the historical power data.

[0014] Furthermore, the abnormal data diagnosis method for power usage of the present invention further includes, before restoring the data of the prediction results, performing abnormal range detection on the prediction results and eliminating abnormal values.

[0015] Compared with the prior art, the method provided by the present invention uses an adaptive ARIMA model to predict power consumption and abnormal data diagnosis method. The present invention uses an adaptive ARIMA model to predict electricity usage conditions, uses the ARIMA model to predict electricity prices and demand, applies the ARIMA model to electricity prices and demand data in the electricity market to predict future electricity prices and demand, and dynamically adjusts model parameters according to historical data and current market conditions, thereby improving the accuracy and stability of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a method for predicting power consumption using an adaptive ARIMA model according to an embodiment of the present invention; Figure 2 This is a flow chart of a method for predicting power consumption and abnormal data diagnosis using an adaptive ARIMA model according to an embodiment of the present invention; Figure 3 This is a comparison chart of electricity price prediction using an adaptive ARIMA model to predict electricity usage conditions according to an embodiment of the present invention; Figure 4 This is a demand forecast comparison diagram for predicting electricity usage using an adaptive ARIMA model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Example

[0019] like Figure 1 to Figure 2 As shown, a method for predicting power consumption using an adaptive ARIMA model specifically includes the following steps: The specific steps include: S1. Data collection: Collect the current historical power data, real-time power data and website predicted power data of the target power market through the target website.

[0020] Specifically, data collection technology is used to collect power market power data. Python is used to access the electricity price and demand panel data on the AEMO website at specific locations. The data is accessed once at regular intervals as a scheduled task, and the data is stored in kafka, MySQL, CK (ClickHouse) database management systems for analysis and prediction. The interval time can be 5 minutes, 6 minutes, 8 minutes, etc. Power data includes electricity price data and electricity demand data, etc.

[0021] Among them, the historical power data refers to the historical power data of Y days at the current time. In one embodiment, Y=3, 4, 5; in other embodiments, Y can be other values, such as 6, 7, 8, 9, etc.; it can be understood that the value of Y should take into account timeliness and accuracy.

[0022] (1) Target website and type of data collected: Based on the electricity market price and electricity demand data, the following types of data are collected: 'REGION' String COMMENT 'Region', 'SETTLEMENTDATE'DateTime COMMENT'Settlement time', 'TOTALDEMAND'Float32 COMMENT'Total demand', 'RRP'Float32 COMMENT'Recommended retail price', 'PERIODTYPE' String COMMENT 'Period type'.

[0023] (2) Request strategy: Use Python's Requests library to send an HTTP request to the website, where the three parameters cookies, headers, and json_data are all dictionary types. Cookies contains the cookie information to be sent, headers contains the request header information, and json_data contains the JSON data to be sent; use the HTTP POST method to send a JSON data to the target website, and carry the corresponding cookie and request header information; based on the website's real-time electricity price, it is refreshed every 5 minutes, and a scheduled task is set to reissue the request every five minutes; it can be understood that the refresh time and scheduled task time can also select other values ​​according to actual needs, such as 3 minutes, 6 minutes, 8 minutes, etc.

[0024] (3) Data cleaning: Clean and process the collected data. Each time a request is made, a list data file will be returned. The time span of the data is 3 days before and after the current time node, including historical electricity prices, real-time electricity prices and future predicted electricity prices. The real-time electricity price (PERIODTYPE=='ACTUAL') and predicted electricity price (PERIODTYPE=='FORECAST') data are filtered out. First, convert the list to Dateframe data format and change the region to a detailed address, including: 'NSW1':'New South Wales', 'QLD1':'Queensland', 'SA1':'South Australia', 'TAS1':'Tasmania', 'VIC1':'Victoria'; Secondly, add two new time columns based on the original data: 'local_Time' DateTime COMMENT 'local time', 'createTime' DateTime COMMENT 'data creation time'; It is understandable that if the original time column SETTLEMENTDATE is in the Australian time zone (UTC+10), according to the background scheduling requirements, the scheduling time is unified to the Shanghai time zone (UTC+8), local_Time is the original time zone, createTime is the data generation time, which can be understood as the time when the data collected by the current visit website is stored in the local MySQL, which is Shanghai time (UTC+8). This time is set to facilitate later comparative analysis.

[0025] Understandably, if the original time column SETTLEMENTDATE is in other time zones outside of Australia or Shanghai, the scheduling time will be unified to the Shanghai time zone (UTC+8) according to the background scheduling requirements.

[0026] (4) Data storage: The cleaned and processed data is written into MySQL and the local CK (ClickHouse) database management system in real time according to the data collection time.

[0027] (5) Regularly update data collection procedures: Regularly update data collection procedures, check code logic and website data status, and ensure the timeliness and accuracy of data.

[0028] (6) Comply with data collection regulations and website agreements: Comply with data collection specifications and website agreements to avoid affecting the target website.

[0029] S2. Data processing: After normalizing the historical power data to between 0 and 1, the data is divided into a training set and a test set, and the time series data is converted into supervised learning data.

[0030] Specifically, in the code, the historical power data is first read and normalized to between 0 and 1; then, the normalized historical power data is divided into two data sets, namely a training set and a test set, and the time series data is converted into supervised learning data.

[0031] S3, model construction and training: construct an ARIMA model, perform model training on the training set, and according to the minimum information criterion, first cycle to obtain the optimal P and Q values, and then cycle to obtain the optimal D value.

[0032] Specifically, after S2 is completed, the ARIMA model is constructed and the training set is used for model training. The adaptive ARIMA algorithm automatically selects the best ARIMA model to fit the given time series data; the three parameters P, D, and Q in ARIMA (P, D, Q) represent the order of the AR model, the number of differences, and the order of the MA model, respectively. Specifically, The P parameter represents the order of the AR model, that is, the number of lag terms retained in the autoregressive model; the AR model means that there is a linear relationship between the value at the current moment and the value at the previous P moments, so the values ​​at the previous P moments need to be retained as the input of the prediction model; The D parameter represents the number of differences, that is, the number of times the original time series is differentiated; differentiation refers to converting the original time series into its first-order or multi-order difference series, thereby eliminating the non-stationarity of the time series; through differentiation, the non-stationary time series is converted into a stationary time series, making it easier to predict; The Q parameter represents the order of the MA model, that is, the number of lag terms retained in the moving average model; the MA model means that there is a linear relationship between the current moment value and the residuals of the previous Q moments, so the residuals of the previous Q moments need to be retained as the input of the prediction model.

[0033] The specific implementation method is as follows: (1) Cycle to get the best P, Q value: Use the ARMA model in the StatsModels library to select the best ARMA model order function; automatically select the best ARMA model order according to the given minimum information criterion; the parameters are explained as follows: dataset1: time series data to be fitted; ic: information criterion, which can be in list form, including "AIC", "BIC", "HQIC", "FPE" indicators; trend: the type of trend term, which can be "c" (constant term), "nc" (no trend term), or others; max_ar: maximum value of AR order; max_ma: maximum value of MA order; Use AIC as the information criterion to minimize, and select the ARMA model with the maximum AR order (maximum value of P) and the maximum MA order (maximum value of Q) both being 10; the function returns a dictionary containing the information criterion values ​​for each order, as well as the optimal ARMA orders P and Q at which the information criterion value is minimized.

[0034] Among them, each order is all the combined values ​​of P and Q. For example, when the maximum values ​​of P and Q are both 10, all the combined values ​​of P and Q are: P=1, Q=1, 2,...,10; P=2, Q=1, 2,...,10; … P=10, Q=1, 2,...,10.

[0035] When the maximum values ​​of P and Q are M1 and M2 respectively, all combinations of P and Q are: P=1, Q=1, 2,..., M2; P=2, Q=1, 2,..., M2; … P=M1, Q=1, 2,..., M2.

[0036] In other embodiments, M1 and M2 may take other values, for example, both are 8, or both are 12, or are 8 and 10 respectively, or are 10 and 8 respectively, and so on; M1 and M2 may be equal or unequal.

[0037] In one embodiment, M1 is a positive integer from 8 to 15, and M2 is a positive integer from 8 to 15.

[0038] (2) Cycle to obtain the best D value: Set the value range of D to 0 to N, write a for loop to combine the P, Q values ​​and order D selected above and substitute them into model.fit(), use the minimization information criterion (such as the AIC information criterion) again, retain the best parameters, and the result will produce the best P, Q, and D values ​​suitable for this group of data, that is, the best D value. At this point, the ARIMA model training is completed, and the ARIMA model with the best P, Q, and D values ​​is used for subsequent model prediction; where N is a positive integer greater than or equal to 4.

[0039] S4. Electricity price and demand forecast: Use the trained ARIMA model, use the test set to make forecasts and restore the forecast results to data.

[0040] Regarding the real-time prediction of electricity prices and demand, using the trained ARIMA model, we can intercept the historical X1 days of test set data of the current time, perform a scheduled task every X2 hours to perform day-ahead prediction, and use X3 minutes as a point to predict the next day, totaling 1440 / X3 points, and restore the prediction results to time series data, store them in MySQL for data overwriting and updating, and store the local CK without deduplication (meaning that when storing in the local ClickHouse database, the data is directly inserted without deduplication), and combine the following S5 step to perform model evaluation; wherein, in one embodiment, X1 is 3, 4, 5, 6, 7, 8, X2 is 2, 3, 4, and X3 is 20, 30, 40, 60.

[0041] For example, in one embodiment, the test set data of the current 7 days of history is intercepted, and a scheduled task is performed every 2 hours to perform day-ahead prediction. The next day is predicted at a point every half an hour, for a total of 48 points. The prediction results are restored to time series data and stored in MySQL for data overwriting and updating. The local CK is stored without deduplication, and the model is evaluated in combination with the following S5 step.

[0042] In another embodiment, the test set data of the current 6 days of history is intercepted, and a scheduled task is performed every 3 hours to perform day-ahead prediction. The next day is predicted for 20 minutes at a point, for a total of 72 points. The prediction results are restored to time series data and stored in MySQL for data overwriting and updating. The local CK is stored without deduplication, and the model is evaluated in combination with the following S5 step.

[0043] S5. Model evaluation: Calculate the prediction error of the ARIMA model and visualize the prediction results.

[0044] In one embodiment, calculating the prediction error of the ARIMA model includes comparing the prediction results of the ARIMA model with the real-time power data to obtain corresponding prediction errors.

[0045] In another embodiment, calculating the prediction error of the ARIMA model includes comparing the prediction result of the ARIMA model with the website predicted power data and the real-time power data to obtain a corresponding prediction error.

[0046] In one embodiment, the indicators of ARIMA model evaluation include one or more of root mean square error, mean square error, mean absolute error, and R square; the corresponding error diagnosis methods are root mean square error diagnosis method, mean square error diagnosis method, mean absolute error diagnosis method, and R square diagnosis method, where: The root mean square error diagnostic method is used to measure the difference between the predicted value and the true value. RMSE is the square root of MSE. It squares the difference between the predicted value and the true value, averages it, and then squares it. The smaller the value, the smaller the difference between the predicted value and the true value, and the better the prediction ability of the model. The mean square error diagnostic method is used to measure the difference between the predicted value and the true value; MSE is the average of the squares of the difference between the predicted value and the true value. The smaller the value, the smaller the difference between the predicted value and the true value, and the better the predictive ability of the model; The mean absolute error diagnostic method is used to measure the difference between the predicted value and the true value; MAE is the average of the absolute values ​​of the difference between the predicted value and the true value. The smaller the value, the smaller the difference between the predicted value and the true value, and the better the predictive ability of the model; The R-squared diagnostic method is used to measure the model's ability to explain the data; the value range of R-squared is 0 to 1. The closer the value is to 1, the better the model's ability to explain the data, and the closer it is to 0, the worse the model's ability to explain the data; the R-squared method is to calculate the square of the difference between the predicted value and the true value and then divide it by the total variance. The larger the value, the better the model's predictive ability.

[0047] In one embodiment, the prediction results may be visualized by drawing a comparison graph of the predicted values ​​and the true values ​​to display the prediction results, or by drawing a comparison table of the predicted values ​​and the true values ​​to display the prediction results, or / and by presenting the indicators of model evaluation (such as root mean square error, mean square error, mean absolute error, R square, etc.) in a visualized manner, etc.

[0048] For example, Figure 3 By means of a comparison chart, the electricity price forecast value (i.e. Figure 3 The actual electricity price and the electricity price forecast given on the website (i.e. Figure 3 The three values ​​of "Electricity Price Forecast Value - Website") are combined to achieve the effect of visualizing the prediction results.

[0049] For example, Figure 4 By means of a comparison chart, the demand forecast value (i.e. Figure 3 The actual demand value and the demand forecast value given on the website (i.e. Figure 3 The three values ​​of "demand forecast value - website") are combined to achieve the effect of visualizing the prediction results; among them, demand refers to electricity demand, or electricity consumption demand.

[0050] For this patent, the predicted value and the true value are relative to the power data, that is, the predicted value and the true value refer to the predicted value of the power data and the true value of the power data respectively.

[0051] The present invention also provides a method for diagnosing abnormal data of electricity usage conditions, which is applied to the aforementioned method for predicting electricity usage conditions. The abnormal data diagnosis method judges the abnormal data in the historical power data based on the problems of missing data and abnormal values ​​in the original data, regards the data exceeding the upper and lower limits as abnormal values, and marks and processes the abnormal values.

[0052] In one embodiment, the abnormal data diagnosis method uses the box-and-line method to diagnose and process abnormal values ​​in historical power data: calculate the quartiles (Q1, Q2, Q3) and interquartile range (IQR) of the data set; calculate the upper limit (Q3+1.5×IQR) and lower limit (Q1-1.5×IQR) according to the interquartile range IQR, regard the data exceeding the upper and lower limits as abnormal values, and mark the abnormal values; this method aims to process the abnormal data as the mean on the basis of retaining about 75% of the original data, so that the overall data is more stable and real, and improve the model accuracy to a certain extent.

[0053] In one embodiment, the abnormal data diagnosis method uses a cap method to diagnose and process abnormal values ​​in the historical power data, and the missing historical power data is filled with the mean; the cap method defines the data range through simple upper and lower limits, and can quickly identify and filter out abnormal values.

[0054] In one embodiment, the abnormal data diagnosis method further includes, before restoring the prediction results, performing abnormal range detection on the prediction results and eliminating abnormal values; wherein, the abnormal range detection on the prediction results and the elimination of abnormal values ​​may adopt the box-line method or the cap method.

[0055] The present invention uses an adaptive ARIMA model to predict electricity consumption and an abnormal data diagnosis method, uses the ARIMA model to predict electricity prices and demand, and dynamically adjusts model parameters according to historical data and current market conditions to improve the accuracy and stability of the prediction. The present invention can be applied to electricity price and demand prediction in the power market and to abnormal data diagnosis. The abnormal data is processed as a mean value on the basis of retaining about 75% of the original data, so that the overall data is more stable and real, and the model accuracy is improved to a certain extent.

[0056] 1. High accuracy: Dynamically adjust model parameters based on historical data and current market conditions to improve forecast accuracy and stability.

[0057] 2. Efficiency: Future electricity prices and demand can be quickly predicted to improve the efficiency and reliability of the electricity market.

[0058] 3. Flexibility: It can be customized according to different market demands and data characteristics to meet the needs of different users.

[0059] The method of the present invention can be applied to the following scenarios: 1. Electricity market regulation: Regulators can use our approach to forecast future electricity prices and demand to oversee how the market is performing.

[0060] 2. Power Generation Companies: Power generation companies can use our methods to predict future electricity prices and demand to optimize power generation plans and investment decisions.

[0061] 3. Electricity users: Electricity users can use our method to predict future electricity prices and demand to optimize electricity usage plans and reduce electricity costs.

[0062] Therefore, the present invention provides a method for predicting electricity consumption and abnormal data diagnosis using an adaptive ARIMA model, which has the advantages of high accuracy, high efficiency and flexibility, and can be applied to electricity price and demand forecasting in the power market and diagnosis of abnormal data.

[0063] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for predicting electricity consumption using an adaptive ARIMA model, characterized in that: The specific steps include: S1. Data collection: Collect the current historical power data, real-time power data and website predicted power data of the target power market through the target website; S2. Data processing: After normalizing the historical power data to between 0 and 1, the data is divided into a training set and a test set, and the time series data is converted into supervised learning data; S3, model construction and training: construct an ARIMA model, perform model training on the training set, and according to the minimum information criterion, first cycle to obtain the best P and Q values, and then cycle to obtain the best D value, where P represents the order of the AR model, Q represents the order of the MA model, and D represents the number of differences; S4. Electricity price and demand forecast: Use the trained ARIMA model, use the test set to make forecasts and restore the forecast results to data; S5. Model evaluation: Calculate the prediction error of the ARIMA model and visualize the prediction results.

2. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: The power data includes power price data and power demand data.

3. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: According to the minimum information criterion, the optimal P and Q values ​​are first obtained by looping, and then the optimal D value is obtained by looping: Use the ARMA model with the maximum P value of M1 and the maximum Q value of M2. The P and Q values ​​are traversed from 1 to their respective maximum values. According to the minimum information criterion, the best ARMA model order, that is, the best P and Q values, are selected, where M1 is a positive integer from 8 to 15, and M2 is a positive integer from 8 to 15; Set the range of D to 0 to N, write a for loop to combine the optimal P, Q and D values ​​and substitute them into model.fit(), and obtain the optimal D value according to the minimization information criterion; where N is a positive integer greater than or equal to 4.

4. The method for predicting power consumption using an adaptive ARIMA model according to claim 3, characterized in that: The minimization information criterion is the AIC information criterion.

5. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: The calculating the prediction error of the ARIMA model includes comparing the prediction results of the ARIMA model with the real-time power data to obtain corresponding prediction errors.

6. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: The model evaluation index includes one or more of root mean square error, mean square error, mean absolute error, and R square.

7. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: The prediction refers to a prediction for the next day.

8. The method for predicting power consumption using an adaptive ARIMA model according to claim 1, characterized in that: The historical power data refers to the power data of Y days in history at the current time, where Y=3, 4, 5, 6, 7, 8.

9. A method for diagnosing abnormal data of power usage, applied to the method for predicting power usage according to any one of claims 1 to 8, characterized in that: The abnormal data in the historical power data is diagnosed and processed by using the box line method or the cap method.

10. The method for diagnosing abnormal power consumption data according to claim 9, characterized in that: It also includes, before restoring the data of the prediction results, performing abnormal range detection on the prediction results and eliminating abnormal values.

Images