Power demand forecasting system and method guided by variant Transformer

Abstract

The present invention belongs to the field of electric power engineering technology, and discloses a system and method for predicting power demand guided by a variant Transformer. The prediction system includes: a data preprocessing module, a variant multi-scale Transformer model, a prediction module, and a state space model. The present invention normalizes the power demand time series data within a given time period before the current moment through the data preprocessing module; performs multi-scale processing on the power demand time series data through the variant multi-scale Transformer model, and processes the power demand time series data at the corresponding scale through a dual attention mechanism; obtains power demand prediction features through the prediction module; and obtains power demand prediction results through the state space model. The present invention can accurately predict future power demand to assist in decisions such as power production, distribution, and management.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power engineering, relates to power demand forecasting, and in particular to a power demand forecasting system and method guided by a variant Transformer. Background Art

[0002] Since the 20th century, research on forecasting of power-related time series data has been ongoing, for example, power demand forecasting, power generation forecasting, power transmission forecasting, and power price forecasting.

[0003] In the modern era, the development of power systems is highly efficient and has diverse sources, but the supply of these energy sources is still unstable and inaccurate. Most electricity needs to be reserved in advance, so more accurate technology is needed to achieve management and prediction. Therefore, accurate prediction of power data has become an important demand and challenge in today's society, which is of great significance to power management, distribution and production.

[0004] However, the current electricity demand forecasting methods should fully consider the diversity of electricity sources. The production characteristics of some renewable energy sources vary over time, and the electricity production that relies on these energy sources also shows greater randomness and timing. In addition, there is a certain long-term dependency between these energy sources and a variety of other unknown factors. These different features have different scales, and the traditional Transformer structure is insufficient in dealing with multi-scale problems. For example, the DeepAR model cannot model the complex dependencies between variables, does not adequately handle the interactions of external factors, and is prone to error accumulation problems in multi-step forecasts. The PrEF model can effectively alleviate this problem and has better interpretability, which can reveal the relationship between time series and external factors. However, the lack of multi-scale transformation may result in the inability to fully capture the characteristics of different scales in the power time series, thereby limiting the model's ability to model complex dependencies and multi-scale patterns. Summary of the invention

[0005] The purpose of the present invention is to provide a power demand forecasting system guided by variant Transformer to accurately predict future power demand in order to assist in decision-making on power production, distribution, and management.

[0006] Another object of the present invention is to provide a power demand forecasting method guided by variant Transformer.

[0007] The present invention mainly solves the following problems: (1) achieving higher prediction accuracy; (2) solving the large randomness and timing characteristics of power production; and (3) solving the deficiencies of the model in processing characteristic data of different scales from a technical aspect.

[0008] The core idea of ​​the present invention is to use a variant multi-scale Transformer structure with an adaptive path to achieve effective modeling of multi-scale time series data. By adaptively adjusting the multi-scale modeling process, it is possible to dynamically process the characteristics of power demand data at different time resolutions and time distances, thereby improving the prediction accuracy. Moreover, in order to cope with the inherent randomness and uncertainty of power demand data, the present invention introduces a probabilistic state space model to map the observed data and external factors into the state space for probabilistic reasoning, thereby learning the probability distribution of power demand and reducing the uncertainty of the prediction. Finally, the data from the past 48 hours is used to predict the power demand for the next 24 hours, providing strong support for regional power dispatching, management and planning.

[0009] Based on the above inventive ideas, the present invention provides a power demand forecasting system guided by a variant Transformer, which includes:

[0010] A data preprocessing module is used to normalize the power demand time series data within a specified time period before the current moment;

[0011] A variant multi-scale Transformer model is used to perform multi-scale processing on power demand time series data, and process the power demand time series data at corresponding scales through a dual attention mechanism to capture short-term fluctuations and long-term trends in power demand; the variant multi-scale Transformer model includes a multi-scale data construction module for dividing the power demand time series data into multiple scales, a Transformer module for processing power demand time series data at different scales, and a multi-scale aggregation module; the Transformer module processes the power demand time series data through a dual attention mechanism of internal attention and external attention to obtain output at corresponding scales; the multi-scale aggregation module is used to perform weighted aggregation on the outputs of the Transformer modules at different scales to obtain the output of the variant multi-scale Transformer model;

[0012] The prediction module is used to obtain the power demand prediction features based on the output of the variant multi-scale Transformer model;

[0013] The state space model is used to obtain the power demand forecast result within a specified time period after the current moment based on the power demand forecast characteristics.

[0014] In one implementable manner, the electricity demand time series data includes electricity demand and related features; the related features include timestamp (universal time UTC, time format is "year / month / day hour:minute"), city temperature, city rainfall, day of the week (used to reflect the periodic changes in electricity demand during the week), whether it is a holiday (the electricity demand pattern on holidays is usually different from that on weekdays), etc.

[0015] The data preprocessing module is used to normalize the power demand time series data within a specified time period before the current moment, and the normalized power demand time series data is used as input data of the variant multi-scale Transformer model.

[0016] In one implementation, a variant multi-scale Transformer model is used for power demand forecasting. First, a module is constructed by multi-scale data, using a series of windows of different sizes. Raw power demand time series data Multi-scale division is performed. Each window size S divides the time series data into P = H / S segments (X 1 ,X 2 ,…,X P ), each fragment The power demand time series data contains S time steps, where d represents the data dimension. This multi-scale division combined with the dual attention mechanism can simultaneously capture the short-term fluctuations and long-term trends of power demand and effectively perform multi-scale modeling on time series data.

[0017] The Transformer module includes a data reconstruction unit and a dual attention unit. The data reconstruction unit is used to perform Fourier transform and inverse transform on the input power demand time series data to obtain reconstructed power demand time series data; the dual attention unit is used to process the reconstructed power demand time series data to obtain output at a corresponding scale.

[0018] In a preferred implementation, in order to improve the model efficiency and generalization ability, the data reconstruction unit uses discrete Fourier transform (DFT) to process the power demand time series data, analyzes the power demand time series data, and selects the top K in descending order of amplitude. f The main frequency components (that is, in descending order of amplitude, select the K with the largest amplitude f The main frequency components) are obtained by inverse DFT (IDFT) to obtain the main periodic patterns. Reconstruct the signal containing the main periodic information to guide multi-scale modeling, where A represents the amplitude and Φ represents the phase. Then, the reconstructed power demand time series data X sea As input to the dual attention unit.

[0019] The dual attention unit includes intra-patch attention and inter-patch attention;

[0020] The internal attention calculation formula is:

[0021]

[0022] in, denote the query vector, key vector and value vector respectively; d m Represents dimension; each Capturing the dependencies of short-term electricity demand changes within each segment, for example, the correlation between different minutes within the same hour (if d contains minute-level data);

[0023] The external attention calculation formula is:

[0024]

[0025] in, Represent the query vector, key vector and value vector respectively; d′ m Representation dimension; external attention captures the long-term dependencies between different fragments, for example, the correlation between different hours and different days, thereby capturing the effects of seasonality or cycles.

[0026] The output of the dual attention unit is expressed as:

[0027]

[0028] The multi-scale aggregation module performs weighted aggregation on the outputs of the Transformer module at different scales. The present invention adopts a routing mechanism to obtain the weights of different scales. Specifically, the routing mechanism uses a softmax function to calculate the weight of each scale:

[0029] R(X trans )=Softmax(X trans W K +∈·Softplus(X trans W noise ));

[0030] in, W r and represents a learnable parameter; outputs seasonal or periodic patterns (i.e., external attention calculation results Add it to the original power demand time series data X, and then perform linear mapping Linear(·) to transform and merge them along the time dimension to obtain Xtrans In order to enhance sparsity, only the scales corresponding to the first K largest weights are retained in descending order of weights, and the other weights are reset to zero.

[0031] Finally, the multi-scale aggregation module performs weighted aggregation on the outputs of Transformer modules of different scales according to the weights obtained by the routing mechanism:

[0032]

[0033] Among them, T i (·) is the transformation function to align time dimensions of different scales, is an indicator function that ensures that only the selected scales participate in the aggregation, and finally obtains the output of the variant multi-scale Transformer model. Represents a block size S i The output of the multi-scale Transformer, since different block sizes produce different time dimensions, the multi-scale aggregation module first performs the transformation function T i (·) to align the time dimensions of different scales.

[0034] In one possible implementation, the prediction module uses a fully connected neural network layer for preliminary prediction of long time series to obtain power demand prediction features.

[0035] In one possible implementation, the state space model (SSM) further processes and predicts the power demand forecast features obtained by the forecast module. The state space model can capture the potential trends and seasonal changes in the power demand time series data and provide more accurate multi-step forecast results while taking into account the uncertainty of the forecast. The state space model is described as:

[0036]

[0037] in, represents the implicit state at time t, including time patterns such as trend and seasonality, and represents the potential state of electricity demand at time t, d s It is the dimension of the implicit state; Represents the state transfer matrix, which transfers the implicit state from the previous moment to the current moment, describing the evolution of the implicit state over time, for example, the persistence of a trend or the periodicity of a seasonal pattern; The implicit state L t Converted to observable power demand value X t ; and Indicates systematic bias; and represent the variance of the implicit state and power demand forecast respectively (here we assume that L t The vectors in are independent, so Q t and R t is a diagonal matrix). Since the power demand time series data has been normalized, the power demand value X t After performing inverse normalization, the final power demand forecast value can be obtained.

[0038] In a preferred implementation, the state space model can also model the nonlinear relationship and conditional probability distribution in the power demand time series data, and use the activation function δ(·) (such as the Tanh function) to obtain the probability distribution as follows:

[0039]

[0040] Therefore, the state space model uses the power demand forecast characteristics as initial values ​​or prior information to build a dynamic model to capture the time evolution of power demand. Through the iterative update of implicit states, the state space model can generate multi-step power demand forecast results containing uncertainty information. t and R t The size of controls the uncertainty range of the prediction. The larger the value, the higher the uncertainty. Ultimately, the state space model provides not only a prediction value at a point in time, but a prediction result containing probability distribution, which more comprehensively reflects the uncertainty of power demand prediction.

[0041] The present invention also provides a method for predicting power demand guided by a variant Transformer, using the above prediction system, and performing the following steps:

[0042] S1 normalizes the power demand time series data within a specified time period before the current moment through a data preprocessing module;

[0043] S2 processes the power demand time series data at multiple scales through one or more sequentially configured variant multi-scale Transformer models, and processes the power demand time series data at corresponding scales through a dual attention mechanism;

[0044] S3 obtains power demand prediction characteristics through a prediction module according to the output of step S2;

[0045] S4 obtains the power demand forecast result within a specified time period after the current moment through the state space model based on the power demand forecast characteristics.

[0046] In the above step S2, when there are more than two variant multi-scale Transformer models, the variant multi-scale Transformer models are connected by residual connections; the concatenation result of the input and output of the previous variant multi-scale Transformer model is used as the output of the next variant multi-scale Transformer model.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1) The present invention introduces a variant multi-scale Transformer structure, which effectively solves the limitations of the traditional Transformer in processing multi-scale features, improves the accuracy and stability of power demand forecasting, and performs better in the face of noise and nonlinear relationships;

[0049] 2) The present invention uses an adaptive path adjustment strategy (including a dual attention mechanism and a routing mechanism) to dynamically adjust the modeling process according to the input data, thereby enhancing the model's ability to cope with short-term fluctuations and long-term trends, and improving the accuracy and robustness of the prediction of complex power demand;

[0050] 3) The present invention combines the nonlinear neural state space model (SSM) with the variant multi-scale Transformer to achieve probability distribution modeling of power demand, reduce the uncertainty in the prediction, and provide power managers with richer decision-making basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the principle of a power demand forecasting system guided by a variant Transformer provided in Example 1 of the present invention.

[0052] Figure 2 This is a flow chart of a method for predicting power demand guided by a variant Transformer provided in Example 2 of the present invention.

[0053] Explanation of terms

[0054] Transformer is a neural network architecture based on the self-attention mechanism, which was originally used to handle natural language processing (NLP) tasks. Its main feature is to capture the correlation between different positions in the input sequence through the "self-attention" module (Self-Attention), without relying on the sequential characteristics of traditional recurrent neural networks (such as LSTM and GRU). Transformer processes the input sequence hierarchically through the encoder and decoder structure, where the encoder is used to generate feature representations of the input sequence, and the decoder uses these representations to generate the output sequence. The architecture also introduces positional encoding to maintain sequence information. Transformer is widely used in multimodal learning and large-scale sequence modeling due to its parallel computing capabilities and superior feature capture capabilities. For its detailed implementation method, please refer to [Vaswani, A., et al. "Attention is All You Need." NeurIPS, 2017].

[0055] The state space model (SSM) is a widely used probabilistic modeling method for time series data. The model represents the time series as a series of latent states that evolve over time, allowing the model to handle uncertainty and dependencies in the data. Specifically, the SSM contains state transition equations and observation equations that link latent states to observed data to capture temporal correlations and dependencies between external factors. By maintaining a traceable reasoning path, the SSM effectively reduces error accumulation in predictions and improves the robustness and accuracy of the model. For implementation details, refer to [Rangapuram, SS, et al. "Deep State Space Models for Time Series Forecasting." NeurIPS, 2018]. DETAILED DESCRIPTION

[0056] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0057] Example 1

[0058] The main purpose of the present invention is to accurately predict the power demand at future times by analyzing and modeling historical power demand data and related external factors (such as temperature, humidity, holidays, etc.). This is of great significance for the efficient management, resource scheduling and optimization of the power system, helping power companies to plan power generation, energy storage and distribution strategies in advance, thereby meeting user needs and reducing operating costs. In addition, by capturing short-term fluctuations and long-term trends in power demand, the present invention can cope with the randomness and uncertainty of power demand, improve the accuracy of prediction and the stability of the power grid system.

[0059] Based on this, this embodiment provides a power demand forecasting system guided by variant Transformer, which integrates time resolution and time distance to achieve multi-scale time series modeling, so as to more accurately predict power demand. First, the variant multi-scale Transformer structure is used to model the time series of power demand to process the characteristics of time series data with different scales. In the modeling stage, according to the input dynamic time series data, the multi-scale modeling process is adaptively adjusted to achieve dynamic modeling, thereby improving the accuracy of the model. Next, considering the randomness and uncertainty of the data, this embodiment uses a probabilistic state space model for feature processing. This step converts the observation results and external factors into the parameter space of the state space model to perform probabilistic time series reasoning, learn the distribution of power demand, and reduce the risk caused by uncertainty. Finally, the data of the past 48 hours can be used to predict the power demand for the next 24 hours, assisting regional power scheduling, management, planning and other purposes.

[0060] In combination with the above modeling process, the power demand forecasting system guided by the variant Transformer provided in this embodiment is as follows: Figure 1 As shown, it includes: a data preprocessing module, a variant multi-scale Transformer model, a prediction module and a state space model.

[0061] (1) Data preprocessing module

[0062] The data preprocessing module is used to normalize the power demand time series data within a specified time period before the current moment.

[0063] The initial time series data set is cleaned according to different data properties and feature categories to obtain the power demand and related features at each time point to form the power demand time series data. The related features include timestamp (UTC, time format is "year / month / day hour:minute"), city temperature, city rainfall, day of the week (to reflect the periodic changes in power demand during the week), whether it is a holiday (the power demand pattern on holidays is usually different from that on weekdays), etc.

[0064] Through the data preprocessing module, each feature of the power demand time series data in the specified time period before the current moment is normalized, the value is mapped to the [0,1] interval, and the scaler of each feature is saved for subsequent inverse transformation. The normalized power demand time series data is used as the input data of the variant multi-scale Transformer model.

[0065] (2) Variant Multi-Scale Transformer Model

[0066] The variant multi-scale Transformer model is used to perform multi-scale processing on the time series data of electricity demand, and processes the time series data of electricity demand at the corresponding scale through the dual attention mechanism to capture the short-term fluctuations and long-term trends of electricity demand.

[0067] More than one variant multi-scale Transformer model can be set as required. In this embodiment, two variant multi-scale Transformer models are set, and the residuals of the two variant multi-scale Transformer models are connected in sequence. The first variant multi-scale Transformer model takes the normalized power demand time series data as input, and the second variant multi-scale Transformer model takes the splicing result of the input and output of the first variant multi-scale Transformer model as input, and the splicing result of the input and output of the second variant multi-scale Transformer model is used as the input of the prediction module.

[0068] The variant multi-scale Transformer model includes a multi-scale data construction module for dividing the power demand time series data into multiple scales, a Transformer module for processing the power demand time series data at different scales, and a multi-scale aggregation module.

[0069] The multi-scale data building block uses a series of windows of different sizes Raw power demand time series data Multi-scale division is performed. Each window size S divides the time series data into P = H / S segments (X 1 ,X 2 ,…,X P ), each fragment The power demand time series data contains S time steps, where d represents the data dimension (in this embodiment, d=6). This multi-scale division combined with the dual attention mechanism can simultaneously capture the short-term fluctuations and long-term trends of power demand and perform effective multi-scale modeling of time series data.

[0070] The Transformer module processes the power demand time series data through the dual attention mechanism of internal attention and external attention to obtain the output at the corresponding scale. The Transformer module includes a data reconstruction unit and a dual attention unit.

[0071] The data reconstruction unit is used to perform Fourier transform and inverse transform on the input power demand time series data to obtain reconstructed power demand time series data. The data reconstruction unit uses discrete Fourier transform (DFT) to process the power demand time series data, analyze the power demand time series data, and select the first K in descending order of amplitude. f The main frequency components are obtained by inverse DFT (IDFT) to obtain the main periodic patterns. The signal containing the main periodic information is reconstructed to guide multi-scale modeling, where A represents the amplitude and Φ represents the phase. Then, the reconstructed power demand time series data X sea As input to the dual attention unit.

[0072] The dual attention unit is used to process the reconstructed power demand time series data to obtain output at the corresponding scale. The dual attention unit includes intra-patch attention and inter-patch attention.

[0073] The internal attention calculation formula is:

[0074]

[0075] in, denote the query vector, key vector and value vector respectively; d m Represents dimension; each Capture the dependencies of short-term electricity demand changes within each segment, for example, the correlation between different minutes within the same hour (if d contains minute-level data).

[0076] The external attention calculation formula is:

[0077]

[0078] in, Represent the query vector, key vector and value vector respectively; d′ m Representation dimension; external attention captures the long-term dependencies between different fragments, for example, the correlation between different hours and different days, thereby capturing the effects of seasonality or cycles.

[0079] The output of the dual attention unit is expressed as:

[0080]

[0081] The multi-scale aggregation module is used to perform weighted aggregation on the output of the Transformer module at different scales to obtain the output of the multi-scale Transformer model. This embodiment uses a routing mechanism to obtain weights of different scales. Specifically, the routing mechanism uses a softmax function to calculate the weight of each scale:

[0082] R(X trans )=Softmax(X trans W K +∈·Softplus(X trans W noise ));

[0083] in, W r and represents a learnable parameter; outputs seasonal or periodic patterns (i.e., external attention calculation results Add it to the original power demand time series data X, and then perform linear mapping Linear(·) to transform and merge them along the time dimension to obtain X trans In order to enhance sparsity, only the scales corresponding to the first K largest weights are retained in descending order of weights, and the other weights are reset to zero.

[0084] Finally, the multi-scale aggregation module performs weighted aggregation on the outputs of Transformer modules of different scales according to the weights obtained by the routing mechanism:

[0085]

[0086] Among them, T i (·) is the transformation function to align time dimensions of different scales, It is an indicator function that ensures that only the selected scales participate in the aggregation, and finally obtains the output of the variant multi-scale Transformer model.

[0087] (3) Prediction module

[0088] The prediction module is used to obtain the power demand prediction features based on the output of the variant multi-scale Transformer model.

[0089] In this embodiment, the prediction module uses a fully connected neural network layer for preliminary prediction of long time series to obtain power demand prediction characteristics.

[0090] (4) State space model

[0091] The state space model is used to obtain the power demand forecast result within a specified time period after the current moment based on the power demand forecast characteristics.

[0092] The state space model (SSM) further processes and predicts the power demand forecast features obtained by the forecast module. The state space model can capture the potential trends and seasonal changes in the power demand time series data and provide more accurate multi-step forecast results while considering the uncertainty of the forecast. The state space model is described as:

[0093]

[0094] in, represents the implicit state at time t, including time patterns such as trend and seasonality, and represents the potential state of electricity demand at time t, d s It is the dimension of the implicit state; Represents the state transfer matrix, which transfers the implicit state from the previous moment to the current moment, describing the evolution of the implicit state over time, for example, the persistence of a trend or the periodicity of a seasonal pattern; The implicit state L t Converted to observable power demand value X t ; and Indicates systematic bias; and represent the variance of the implicit state and power demand forecast respectively (here we assume that L t The vectors in are independent, so Q t and R t is a diagonal matrix).

[0095] Since the power demand time series data has been normalized, the power demand value X t After performing inverse normalization, the final power demand forecast value can be obtained.

[0096] The state space model can also model the nonlinear relationship and conditional probability distribution in the power demand time series data. The probability distribution is obtained by using the activation function δ(·) (such as the Tanh function) as follows:

[0097]

[0098] Therefore, the state space model uses the power demand forecast characteristics as initial values ​​or prior information to build a dynamic model to capture the time evolution of power demand. Through the iterative update of implicit states, the state space model can generate multi-step power demand forecast results containing uncertainty information. t and R tThe size of controls the uncertainty range of the prediction. The larger the value, the higher the uncertainty. Ultimately, the state space model provides not only a prediction value at a point in time, but a prediction result containing probability distribution, which more comprehensively reflects the uncertainty of power demand prediction.

[0099] Example 2

[0100] like Figure 2 As shown, this embodiment provides a method for predicting power demand guided by a variant Transformer, using the above prediction system provided in Example 1, and following the steps below:

[0101] S1 normalizes the power demand time series data within a specified time period before the current moment through a data preprocessing module.

[0102] S2 processes the electricity demand time series data at multiple scales through one or more sequentially set variant multi-scale Transformer models, and processes the electricity demand time series data at corresponding scales through a dual attention mechanism.

[0103] In this embodiment, the prediction system sets up two variant multi-scale Transformer models, and the residuals of the two variant multi-scale Transformer models are connected in sequence. The first variant multi-scale Transformer model takes the normalized power demand time series data as input, and the second variant multi-scale Transformer model takes the splicing result of the input and output of the first variant multi-scale Transformer model as input. The splicing result of the input and output of the second variant multi-scale Transformer model is used as the output of step S2, that is, as the input of the prediction module.

[0104] The steps for multi-scale processing of power demand time series data using the variant multi-scale Transformer model are as follows:

[0105] (a) Through the multi-scale data construction module, a series of windows of different sizes are used to divide the original power demand time series data into multiple scales;

[0106] (b) For the power demand time series data at any scale, the power demand time series data is processed by the Transformer module. This step includes the following sub-steps:

[0107] (b1) performing Fourier transform and inverse transform on the input power demand time series data through a data reconstruction unit to obtain reconstructed power demand time series data;

[0108] (b2) The reconstructed power demand time series data is processed by internal attention and external attention through the dual attention unit, and the two processing results are added to obtain the output result of the Transformer module;

[0109] (c) The output of the Transformer module at different scales is weightedly aggregated through the multi-scale aggregation module to obtain the output result of the variant multi-scale Transformer model.

[0110] S3 obtains the power demand prediction characteristics through the prediction module according to the output of step S2.

[0111] S4 obtains the power demand forecast result within a specified time period after the current moment through the state space model based on the power demand forecast characteristics.

[0112] As mentioned above, the power demand value X t After denormalization, the final power demand forecast value is obtained.

[0113] In addition, in this step, the probability distribution of power demand can also be obtained through the state space model.

[0114] Application Examples

[0115] This application example is from CND (CENTRO NACIONALDE The electricity demand time series data are collected by the DESPACHO) platform (https: / / www.cnd.com.pa / index.php / informes-disponibles), as shown in Table 1, including the hourly electricity consumption in Panama during 2017-2018 and 2019-2020. The power range is 380.6 (MW·h)-1635.9 (MW·h) and 85.2 (MW·h)-1754.9 (MW·h), and the average power demand is 1194 and 1217.5 respectively. The temperature is [19.8, 27.8] and [20.22, 39.1] respectively, the humidity is [17.5, 90.6] and [16, 100] respectively, and the precipitation (mm) is [0, 45.8] and [0, 52.0] respectively.

[0116] Table 1 Relevant information of experimental data of the present invention

[0117] Dataset Panama 17-18 Panama 19-20 Duration 2017-2018 2019-2020 Time interval 1hour 1hour Power consumption range (MW·h) [380.6,1635.9] [85.2,1754.9] Average power demand 1194.0 1217.5 temperature [19.8,27.8] [20.22,39.1] humidity [17.5,90.6] [16,100] Precipitation (mm) [0,45.8] [0,52.0]

[0118] As mentioned above, the initial time series data set is cleaned according to different data properties and feature categories to obtain the electricity demand and related features at each time point; the related features include: timestamp (universal standard time UTC, time format is "year / month / day hour: minute"), city temperature, city rainfall; also includes related time features: the day of the week, which is used to reflect the periodic changes in electricity demand during the week; whether it is a holiday, the electricity demand pattern on holidays is usually different from that on weekdays.

[0119] The power demand time series data after the above data cleaning is divided into data sets. Each subset used in this application example is divided into a training set (50%), a validation set (25%) and a test set (25%). For the power demand time series data, the power demand and time characteristics of the past two days (48 hours) are used to predict the hourly electricity consumption of the next day (next 24 hours). This division method can ensure the time continuity of the model and the reliability of the training effect.

[0120] Each feature of each dataset in the training set, validation set, and test set is normalized separately, the values ​​are mapped to the interval [0,1], and the scaler of each feature is saved separately for subsequent inverse transformation. Next, all features are combined into an array and the first 100 data points are removed. Then, the data is split into an input sequence X (data of the past T time steps) and a target sequence Y (data of the future horizon time steps) to meet the data input form of the model.

[0121] First, the power demand forecasting system guided by the variant Transformer is trained using the training set according to steps S2-S4 given in the previous embodiment 2, and the weighted sum of the Copula loss and the marginal likelihood loss is used as the loss function. The Adam optimizer is used for gradient optimization according to the loss value.

[0122] The loss function is expressed as follows:

[0123]

[0124] Among them, Σ represents the covariance matrix of the sample matrix of the predicted power demand target sequence; σ i Represents the variance of the diagonal samples of the predicted target sequence sample matrix; represents the expectation of the predicted target sequence sample under the multivariate normal distribution; z represents the sample vector sampled from the multivariate normal distribution under the covariance matrix; p(x) represents the probability distribution of the actual target sequence sample x; Σ x Represents the covariance matrix of the actual target sequence sample matrix; μ x represents the actual target sequence sample mean vector; d represents the dimension of the data.

[0125] Verify through the validation set; according to the feedback of the validation set, further tune the hyperparameters of the prediction system (such as learning rate, scale weight of the routing mechanism, etc.) to ensure the best prediction effect under multi-scale conditions, that is, the loss value tends to be stable. Finally, the prediction performance of the prediction system is evaluated using the test set, and quantified using common indicators such as RMSE and MAE to evaluate the effectiveness of the prediction system.

[0126] The power demand forecasting system guided by variant Transformer in the present invention and the PrEF method in the prior art are tested on two data sets, Panama 17-18 and Panama 19-20. The test results are shown in Table 2, and four indicators, RMSE, MAE, Log p, and CRPS, are used for evaluation. The root mean square error (RMSE) and mean absolute error (MAE) are used to measure the prediction residual; the logarithmic density Logp and the continuous ranking probability score (CRPS) are used to evaluate the quality of the power demand distribution.

[0127] Table 2 Effect of power demand forecasting on application datasets

[0128]

[0129] Note: The best result is in bold, and the second best result is underlined.

[0130] From the experimental results in Table 2, it can be seen that the prediction system of the present invention outperforms the index values ​​of other models in four indicators (RMSE, MAE, Log p and CRPS), and the prediction system achieves the best performance on all data sets. Compared with the comparison model, the present invention not only proposes an innovative multi-scale variant Transformer structure to adapt to the short-term fluctuations and long-term trends of power demand, but also more effectively handles the randomness and uncertainty problems in power demand.

[0131] It can be seen that the present invention can significantly improve the accuracy and robustness of power demand forecasting by combining multi-scale modeling and probabilistic forecasting. Multi-scale aggregation and soft routing mechanisms enable the model to capture the characteristics of different time scales, while the state space model further optimizes the reliability of the forecast, providing more comprehensive and accurate support for power system scheduling and management.

[0132] The introduction of the other comparative experimental models in the table is as follows:

[0133] HA (Historical Average): HA predicts future demand based on historical averages and is usually used as a baseline model. This method is simple and easy to use, but has limited performance when processing data with complex time dependencies.

[0134] ARIMA: A statistical model that analyzes and predicts stationary time series data by combining autoregression (AR), difference (I) and moving average (MA) (see [Cheng-Ming Lee, et al. "Short-term load forecasting using lifting scheme and ARIMA models." Expert Systems with Applications. 2011.]).

[0135] SVR: A regression model based on support vector machine, which uses support vectors in high-dimensional feature space for prediction. SVR has good performance in processing nonlinear data (see [T. Hong, et al. "Probabilistic electric load forecasting: A tutorial review." International Journal of Forecasting. 2016.]).

[0136] LSTM and GRU: LSTM is widely used to handle power demand forecasting tasks because it can capture long-term dependencies in time series data. Power demand data usually shows obvious periodicity and long-term trends, and the forget gate, input gate, and output gate structure of LSTM enable it to avoid the gradient vanishing problem in traditional RNN, maintain the memory of long-term information, and effectively capture the seasonality and volatility of power demand. GRU is a simplified version of LSTM (see [J.Chung, et al. "Empirical evaluation of gated recurrent neural networks on sequencemodeling." arXiv.2014.]).

[0137] CNN-RNN: In the task of electricity demand forecasting, the CNN-RNN architecture combines the properties of convolutional neural networks (CNNs) to capture local temporal features of data and uses recurrent neural networks (RNNs) to model long-term temporal dependencies (see [X.Guo, et al. "A short-term load forecasting model of multi-scale CNN-LSTMhybrid neural network considering the real-time electricity price." Energy Reports. 2020.]).

[0138] GRU-VAE: Used to generate forecast samples and model data uncertainty in power demand forecasting (see [Y.Qiu,et al. "Aggregate model for power load forecasting based on conditionalautoencoder."International Conference on Intelligent Computing.2021.]).

[0139] DSSM: Combines the latent state and observed data in the time series to predict the long-term trend and cyclical changes in electricity demand. It is used to generate forecast samples and model the uncertainty of data in electricity demand forecasting (see [S.Rangapuram, et al. "Deep state space models for time series forecasting." NeurIPS, 2018.]).

[0140] DeepAR: Probabilistic modeling and forecasting of electricity demand using autoregressive recurrent neural networks (see [D.Salinas, et al. "DeepAR: Probabilistic forecasting with autoregressiverecurrent networks." International Journal of Forecasting, 2020.]).

[0141] PrEF: A probabilistic framework for electricity forecasting that aims to capture the uncertainty dependencies in electricity time series data and establish the interactive relationship between electricity data and various external factors by combining a nonlinear neural state space model (SSM) with a copula augmentation mechanism (see [Z. Wang, et al. "PrEF: Probabilistic electricity forecasting via Copula-augmented state space model." AAAI, 2022.]).

[0142] TPEDF: A transformer-guided probabilistic electricity demand forecasting framework that aims to capture the global and local dependencies of electricity demand time series, as well as the association between complex external features and electricity demand (see [Y.Cao, et al. "Probabilistic Electricity Demand Forecasting with Transformer-GuidedState Space Model." 2022IEEE 5th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), 2022.]).

[0143] Although the present invention is described herein with reference to the embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that a person skilled in the art can make various other specific modifications and combinations that do not deviate from the essence of the present invention based on the technical inspirations disclosed by the present invention, and these modifications and combinations are still within the protection scope of the present invention.

Claims

1. A power demand forecasting system guided by variant Transformer, characterized in that: include: A data preprocessing module is used to normalize the power demand time series data within a specified time period before the current moment; A variant multi-scale Transformer model is used to perform multi-scale processing on power demand time series data, and process the power demand time series data at corresponding scales through a dual attention mechanism to capture short-term fluctuations and long-term trends in power demand; the variant multi-scale Transformer model includes a multi-scale data construction module for dividing the power demand time series data into multiple scales, a Transformer module for processing power demand time series data at different scales, and a multi-scale aggregation module; the Transformer module processes the power demand time series data through a dual attention mechanism of internal attention and external attention to obtain output at corresponding scales; the multi-scale aggregation module is used to perform weighted aggregation on the outputs of the Transformer modules at different scales to obtain the output of the variant multi-scale Transformer model; The prediction module is used to obtain the power demand prediction features based on the output of the variant multi-scale Transformer model; The state space model is used to obtain the power demand forecast result within a specified time period after the current moment based on the power demand forecast characteristics.

2. The power demand forecasting system guided by variant Transformer according to claim 1, characterized in that: The power demand time series data includes power demand and related features; the related features include timestamp, city temperature, city rainfall, day of the week, and whether it is a holiday.

3. The power demand forecasting system guided by variant Transformer according to claim 1, characterized in that: Build modules from multi-scale data, using a range of windows of different sizes The original power demand time series data X is divided into multiple scales; each window size S divides the time series data into P = H / S segments (X 1 ,X 2 ,…,X P ), each fragment X i Contains electricity demand data for S time steps.

4. The power demand forecasting system guided by variant Transformer according to claim 1, characterized in that: The Transformer module includes a data reconstruction unit and a dual attention unit. The data reconstruction unit is used to perform Fourier transform and inverse transform on the input power demand time series data to obtain reconstructed power demand time series data; the dual attention unit is used to process the reconstructed power demand time series data to obtain output at a corresponding scale.

5. The power demand forecasting system guided by variant Transformer according to claim 4, characterized in that: The data reconstruction unit processes the power demand time series data using discrete Fourier transform and selects the first K f The main frequency components are obtained by inverse Fourier transform. A represents amplitude, Φ represents phase, as the reconstructed power demand time series data; with the reconstructed power demand time series data X sea As input to the dual attention unit.

6. The power demand forecasting system guided by variant Transformer according to claim 4 or 5, characterized in that: The dual attention unit includes internal attention and external attention; The internal attention calculation formula is: in, denote the query vector, key vector and value vector respectively; d m Represents dimension; The external attention calculation formula is: in, Represent the query vector, key vector and value vector respectively; d′ m Represents dimension; The output of the dual attention unit is expressed as:

7. The power demand forecasting system guided by variant Transformer according to claim 4 or 5, characterized in that: The multi-scale aggregation module performs weighted aggregation on the outputs of the Transformer modules at different scales; a routing mechanism is used to obtain weights of different scales; the routing mechanism uses a softmax function to calculate the weight of each scale: R(X trans )=Softmax(X trans W r +∈·Softplus(X trans W noise )); in, W r and Represents a learnable parameter; the result of external attention calculation Add it to the original power demand time series data X, and then perform linear mapping Linear(·) to get X trans ; Finally, the multi-scale aggregation module performs weighted aggregation on the outputs of Transformer modules of different scales according to the weights obtained by the routing mechanism: Among them, T i (·) is the transformation function to align time dimensions of different scales, It is an indicator function, and the first K corresponding scales are selected in descending order of weight.

8. The power demand forecasting system guided by variant Transformer according to claim 1, characterized in that: The prediction module uses a fully connected neural network layer for preliminary prediction of long time series to obtain the power demand prediction characteristics.

9. The power demand forecasting system guided by variant Transformer according to claim 1, characterized in that: The state space model is described as: Among them, L t represents the implicit state at time t; A t , C t represents the state transfer matrix; B t , D t Indicates system deviation; Q t , R t denote the variance of the implicit state and power demand forecast values, respectively.

10. The power demand forecasting system guided by variant Transformer according to claim 9, characterized in that: The state space model uses the activation function δ(·) to obtain the probability distribution as follows:

11. A method for predicting power demand guided by variant Transformer, characterized in that: Using the prediction system according to any one of claims 1 to 10, the following steps are performed: S1 normalizes the power demand time series data within a specified time period before the current moment through a data preprocessing module; S2 processes the power demand time series data at multiple scales through one or more sequentially configured variant multi-scale Transformer models, and processes the power demand time series data at corresponding scales through a dual attention mechanism; S3 obtains power demand prediction characteristics through a prediction module according to the output of step S2; S4 obtains the power demand forecast result within a specified time period after the current moment through the state space model based on the power demand forecast characteristics.

12. The method for predicting power demand guided by variant Transformer according to claim 11, characterized in that: In step S2, when there are more than two variant multi-scale Transformer models, the variant multi-scale Transformer models are connected by residual connections; the concatenation result of the input and output of the previous variant multi-scale Transformer model is used as the output of the next variant multi-scale Transformer model.

Images