Power load prediction method based on improved Transform architecture

Abstract

According to the power load prediction method based on the improved Transform architecture, multi-granularity periodic feature coding and a two-stage self-attention mechanism TSSA are innovatively designed, and the problems existing in a standard Transform model are effectively solved. In addition, according to the method, a KV Cache streaming reasoning technology is adopted, and calculation redundancy is thoroughly eliminated. At the same time, the innovative statistical pooling feature aggregation SP mechanism significantly enhances the robustness of the model to the local random fluctuation of the input data. According to the method, multi-layer expansion convolution is creatively applied to the window-level feature sequence, the long-term periodic mode is effectively captured under the condition that the size of the TSSA window is not increased, and the problem of calculated amount explosion caused by directly increasing the window is avoided. According to the method, the original advantages of the Transform model are kept, meanwhile, systematic optimization is carried out according to the special requirements of power load prediction, the prediction precision and the calculation efficiency are remarkably improved, and the method has important theoretical value and practical significance.

Description

Technical Field

[0001] This invention belongs to the field of electricity load forecasting technology, specifically relating to an improved Transformer model for electricity load forecasting. Background Technology

[0002] As the scale and complexity of global power grid systems continue to grow, the importance of load forecasting in power system operation is becoming increasingly prominent. High-precision load forecasting can not only improve the operating efficiency of power systems and reduce operating costs, but also optimize the allocation of power resources and reduce energy waste, which is of great significance for achieving smart grids and sustainable energy development.

[0003] In recent years, deep learning methods such as recurrent neural networks like LSTM and GRU have made breakthroughs in nonlinear modeling, but they still face inherent defects such as gradient vanishing and low computational efficiency, especially when processing long-term historical data (several days to several weeks).

[0004] The Transformer model, with its powerful ability to model long-range dependencies, offers a new approach to power load forecasting. However, the standard Transformer model suffers from two key bottlenecks: 1) The computational complexity of the self-attention mechanism increases quadratically with the sequence length (O(n)). 2 )) It is difficult to efficiently process long historical window data; 2) It lacks a dedicated intra-frame attention mechanism, which makes it impossible to make full use of the local feature association of load data (Reference: [2] Zhou et al., "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting", AAAI, 2021). These limitations make the practical application of the standard Transformer model in power load forecasting tasks challenging. Summary of the Invention

[0005] The present invention mainly aims at the problems that the existing models have limited information transmission ability on long-sequence data and are prone to gradient vanishing or explosion, resulting in poor accuracy in multi-step prediction. Although the standard Transformer model can theoretically capture long-range dependencies, its computational complexity is high and it lacks a dedicated intra-frame attention layer, leading to the inability to efficiently model long sequences. A power load prediction method based on an improved Transformer model is proposed. In the present invention, an innovative improved Transformer architecture is proposed. Through multi-dimensional optimization design, while maintaining the advantages of the Transformer model, its performance in power load prediction tasks is significantly improved. This solution not only solves the limitations of existing methods but also provides a new technical path for accurate load prediction in the smart grid environment.

[0006] The above problems of the present invention are mainly solved by the following technical solutions:

[0007] A power load data prediction method based on an improved Transformer neural network model includes the following steps:

[0008] S1. Propose a feature field construction and field encoding scheme; in the power load prediction task, the factors to be considered include time (TIME), temperature (TEMP), electricity price (PRICE), holiday

[0009] HOLIDAY, and load (LOAD); when performing neural network modeling, these factors need to be vectorized to construct an input feature vector X t ∈R 8 , for time step t, X t =

[0010] [TIME t (4), TEMP t (1), PRICE t (1), HOLIDAY t (1), LOAD t (1)];

[0011] S2. Propose a streaming window function; in the real-time prediction task, for the historical first N-frame features X t-N:t , based on the encoder of the transformer, construct a window function W(·) to capture local change rules; the window length is set to N, and the window shift is set to S < N, that is, there is an overlap between windows; considering the generalization of the window function, the features need to be normalized, that is:

[0012] Mean(·) and Std(·) are the mean and standard deviation respectively, and the X ' data follows a normal distribution; further, position encoding within the window is used to obtain X” = PE(X' ), where PE is the position encoding matrix;

[0013] S3. A two-level self-attention coding (TSSA) mechanism is proposed. The two-level self-attention coding design is proposed, namely, establishing intra-frame self-attention and inter-frame self-attention sequentially. The self-attention function is assumed to be y1, y2, ... = SA(x1, x2, ...), x i For the input feature x at time t, the feature is... t ={x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD}, where: x TIME For time-related features, x TEMP For temperature characteristics, x PRICE For electricity price characteristics, x HOLIDAY For holiday characteristics, x LOAD For the load value; each field vector is first projected onto the same dimension size: x i ∈{x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD},w i Let b be the field weight matrix. i For the field bias vector, we get (h is the uniform projection dimension), and then intra-frame self-attention calculation is performed to obtain The inter-frame self-attention calculation results at multiple time steps are as follows:

[0014] h1,h2,…=SA inter (g1,g2,…)

[0015] S4. Employ streaming inference based on KV Cache; extract the attention weights W from the window function model. Q W K W V Calculate and cache the key and value vectors for each frame; calculate the attention result based on the query vector of the current frame and the KV cache;

[0016] S5. Proposes in-window feature aggregation based on statistical pooling (SP). For the feature sequence obtained by the two-stage self-attention calculation in S3, H t =(h t-N+1 ,h t-N+2 ,…,h t The window representation G is obtained by pooling statistical data. t =Concat(Mean(H)t ),Var(H t ), where Mean(·) and Std(·) are the mean operator and variance operator, respectively;

[0017] S6. Propose a wide-view prediction based on dilated convolution; obtain the local representation G by modeling the local area using TSSA+SP. t Then, multiple local representations are further inferred through dilated convolution to obtain the prediction result at a certain time step. t =Conv(G t-N+1 G t-N+2 ,…,G t );

[0018] S7 employs a supervised learning framework based on the root mean square error (RMSE), specifically optimized for time series prediction tasks. This module calculates the predicted value for each frame based on the prediction sequence output by module S5. Compared with the real label Y t The mean square error between the two values ​​is calculated, and the model parameter θ is adjusted by gradient descent to make the prediction as close as possible to the true value.

[0019] The beneficial effects of this invention are

[0020] (1) This invention designs a multi-granularity feature encoding and a two-level attention mechanism to effectively extract the spatiotemporal coupling features of power load. Targeting the multi-period characteristics of load data, a four-level time encoding of "year-month-week-day" and normalized representations of physical quantities such as temperature and electricity price are adopted to construct an 8-dimensional feature vector to enhance the interpretability of the model's input features. Furthermore, intra-frame self-attention (SA_intra) is used to mine cross-dimensional nonlinear correlations such as temperature-electricity price-load, and inter-frame self-attention (SA_inter) is combined to capture the time-varying patterns of the sequence, effectively overcoming the shortcomings of insufficient feature interaction in existing time-series models.

[0021] (2) This invention proposes a fusion architecture of streaming window function, KV Cache, and statistical pooling to achieve efficient long sequence modeling. The sequence is segmented into local segments using a sliding window mechanism, and a key-value matrix of cached historical frames is calculated using KV Cache-based attention, reducing redundant computation by more than 50%. After Z-score normalization within the window, a two-level attention mechanism is used to extract local features, and then window-level representations are generated through statistical pooling (Mean / Var). This design maintains the global modeling capability of the Transformer while enhancing the model's robustness to power grid fluctuations through local statistics.

[0022] (3) This invention constructs a dilated convolution-attention collaborative inference network, breaking through the field-of-view limitations of existing models. Targeting the long-cycle dependence of power load, multi-scale dilated convolution operations are implemented on the window-level representation sequence, exponentially expanding the receptive field to capture macroscopic patterns such as annual and monthly cycles. This hybrid architecture combines the dynamic weight allocation of self-attention with the stable feature extraction of convolution, maintaining O(n) computational complexity while improving prediction accuracy by 12.6% compared to the standard Transformer, exhibiting particularly strong stability under scenarios of sudden changes in power grid load. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process in an embodiment of the present invention.

[0024] Figure 2 This is an overall framework diagram of an embodiment of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, a power load forecasting method based on an improved Transformer model includes the following steps:

[0027] S1. A feature field construction and encoding scheme is proposed. In the task of power load forecasting, this paper mainly examines the following factors: time, temperature, electricity price, and holidays. These factors are closely related to residential electricity consumption. Furthermore, for neural network modeling, we need to vectorize these factors to construct the input feature vector X. t ∈R 8 X t =

[0028] [TIME t (4), TEMP t (1),PRICE t (1),HOLIDAY t (1),LOAD t (1)].

[0029] The specific steps are as follows:

[0030] Time: Considering the generalization of the model, the time periodicity is encoded according to different granularities. Taking hourly data as an example, the main periodicity adopts the quantization method of "mid-year month", "mid-month day", "mid-week day", and "mid-day hour". For example, if a frame of time record is "March 15, 2025, Saturday, 20:10", its feature encoding is: [3 / 12, 15 / 30, 6 / 7, 20 / 24]. Keeping 2 decimal places, the time vector is: [0.25, 0.50, 0.86, 0.83].

[0031] Temperature (TEMP): The quantization method also adopts a normalization strategy. Assuming the maximum city temperature is 50 degrees, a frame with a temperature of 25 degrees can be encoded as 0.5.

[0032] Electricity price (PRICE): Electricity price is also an important influencing factor, and the quantification method also adopts a normalization strategy.

[0033] Holidays: Holidays are also an important load factor considered in this paper. Here, information on statutory holidays can be collected and given as a Boolean feature vector.

[0034] Load: The quantification result of the load can be normalized by using the historical highest value as the denominator.

[0035] Step S1 is as follows:

[0036] S11. Based on historical load data, weather type, electricity price, and statutory holiday information, multi-dimensional feature encoding is performed. For time: considering the generalization of the model, encoding is performed according to the time periodicity at different granularities. A multi-period normalization strategy is adopted to decompose the timestamp into: annual period: [month / 12], monthly period: [date / 30], weekly period: [day of the week / 7], and intraday period: [hour / 24]. For example, "2025-03-15 20:10" is encoded as [0.25, 0.50, 0.86, 0.83], effectively preserving the periodicity of day and night, week and month. TIME t =[month / 12, day / 30, weekday / 7, hour / 24].

[0037] S12. For environmental and market factors, such as temperature (TEMP), electricity price (PRICE), and holidays (HOLIDAY): dynamic normalization is used. The formula for calculating temperature (TEMP) is: TEMP t= observed_temp / extreme_high_temp, where extreme_high_temp is the historical extreme high temperature in the region. Assuming the maximum urban temperature is 50 degrees, a temperature of 25 degrees in a certain frame can be encoded as 0.5. For the electricity price (PRICE), the calculation formula: PRICE t = (price - min_price) / (max_price - min_price). For holidays (HOLIDAY): A feature vector is given using a boolean type, HOLIDAY t ∈ {0, 1}, where 1 represents national statutory holidays and adjusted holidays.

[0038] S13. Load (LOAD): The quantization result of the load can be normalized using the historical maximum value as the denominator. The calculation formula: LOAD t = load / max_load, parameter: max_load: the maximum load in the past three years.

[0039] S2. Propose a streaming window function. In real-time prediction tasks, for the historical first N-frame features X t-N:t , a window function W(·) is constructed based on the Transformer encoder to capture local change patterns. Here, the window length is N, and the window shift is set to S < N, meaning there is overlap between windows. Considering the generalization of the window function, the features need to be normalized, that is:

[0040] Mean(·) and Std(·) are the mean and standard deviation respectively, X '

[0041] The data follows a normal distribution. Further, position encoding within the window is used to obtain X” = PE(X ' ).

[0042] Step S2 is specifically as follows:

[0043] S21. The streaming window function encodes the historical first N-frame features through a Transformer-based encoder, uses a sliding window mechanism with a window length of N and a window shift of S (S < N) to achieve local overlapping sampling, and normalizes the features within the window (subtract the mean and divide by the standard deviation). At the same time, position encoding within the window is introduced to capture temporal dependence relationships.

[0044] S22. The role of the streaming window function is to dynamically learn the local change patterns of the features within the window using the attention mechanism of the Transformer, ensure temporal continuity through overlapping windows, improve the generalization of the model through normalization, and retain temporal information through position encoding, thereby enhancing the representation ability of historical features in real-time prediction tasks.

[0045] S23. For time step t, the sliding window contains the most recent N-frame features X t-N:t ∈{x t-N ,x t-N+1 ,...,x t}∈R N×d , where x i ∈R d is the original feature vector of the i-th frame (from S1 encoding). Then, Z-score normalization is performed on all features within the window: where is the mean vector within the window, is the standard deviation vector within the window. To retain the temporal information, sinusoidal positional encoding is added to each frame within the window. Let the position pos ∈ {0, 1,..., N - 1} (relative position within the window): PE(pos, 2k) = sin(pos / 10000 2k / d ), where k is the dimension index (0 ≤ k < d / 2), d is the feature dimension, and finally the positional encoding matrix PE ∈ R N×d . The i-th row of corresponds to the position pos = i. Finally, the normalized features and the positional encoding are added frame by frame to obtain the final window representation: Expanded into matrix form: where is the normalized feature matrix, and PE is the positional encoding matrix, and its i-th row PE i = PE(pos = i).

[0046] S3. A Two-Stage Self-Attention (TSSA) mechanism is proposed. In constructing the window function, the standard Transformer self-attention paradigm only considers the inter-frame relationships. However, in the power load forecasting task, there are also strong relationships among the fields within a frame. For example, the relationship between LOAD / TIME is the early peak period of electricity consumption. Therefore, this paper proposes a two-stage self-attention encoding design, that is, constructing intra-frame self-attention and inter-frame self-attention successively. Assume the self-attention function is: y1, y2,... = SA(x1, x2,...), where x i is the feature. For the input feature x t at time t = {x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD}, where: x TIME is the time-related feature, x TEMP is the temperature feature, x PRICEFor electricity price characteristics, x HOLIDAY For holiday characteristics, x LOAD For the load value; each field vector is first projected onto the same dimension size: x i ∈

[0047] {x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD},w i Let b be the field weight matrix. i For the field bias vector, we get (h is the uniform projection dimension), and then intra-frame self-attention calculation is performed to obtain The inter-frame self-attention calculation results at multiple time steps are as follows:

[0048] h1,h2,…=SA inter (g1,g2,…)

[0049] Step S3 is as follows:

[0050] S31. Existing time series models (such as LSTM and TCN) typically model multivariate time series as a whole, implicitly assuming that "all variables share the same time series pattern." However, in reality, relationships between fields may be instantaneous (e.g., a temperature rise immediately affects air conditioning load but has no effect on industrial load), and time series patterns may be variable-specific (e.g., temperature has a diurnal cycle, while rainfall is sudden). Based on these issues, Two-Stage Self-Attention (TSSA) encoding is proposed. It models two key dependencies in time series in stages: 1) Intra-Frame Dependency: the association between different feature fields within the same time step (e.g., the interaction between temperature, humidity, and load); 2) Inter-Frame Dependency: the temporal dynamics between different time steps (e.g., periodic changes or trends in load). By decoupling these two dependencies, TSSA can capture the complex patterns of multivariate time series more precisely.

[0051] S32. Intra-Frame Self-Attention performs self-attention calculations on multivariate fields (such as temperature) within each time step, modeling the nonlinear coupling relationships between fields. Example: In electricity load forecasting, increased temperature may lead to increased air conditioning load, but is unrelated to industrial load—intra-frame attention learns this conditional dependency between fields. It captures the dependencies between different feature fields within the same time step. Input window X t-N:t ∈R N×d For each frame x within the window iProject the fields onto a uniform dimension h and calculate Q / K / V:

[0052] Q intra (query) = x i W Q ∈R h

[0053] K intra (key) = x i W K ∈R h

[0054] V intra (value) = x i W V ∈R h

[0055] Among them W Q W K W V ∈R d×h It is a learnable projection matrix.

[0056] Calculate inter-field attention:

[0057] Output enhanced frame features

[0058] S33. Inter-frame self-attention models long-term dependencies in sequences over time, capturing trends, periodicity, and abrupt event changes. Examples include daily periodicity (high during the day, low at night) or abnormal fluctuations during holidays in electricity load. It captures temporal dependencies across different time steps. For sequences... Calculate inter-step attention:

[0059]

[0060] in, Then calculate each head independently:

[0061]

[0062] Among them W O ∈R h×h Finally, residual connections are performed to preserve the original feature information, layer normalization is applied to stabilize the training process, and temporal augmentation features are output.

[0063]

[0064] Where H intra For in-field attention output ∈ R N×h H interFor the time-interval attention output ∈ R N×h .

[0065] S4. Streaming inference based on KV Cache is adopted. To address the redundant computation caused by overlap in sliding window inference, this method extracts the attention weights from the window function model.

[0066] W Q W K W V We compute and cache the key and value vectors for each frame. When calculating attention within the window function, we directly compute the attention result based on the query vector of the current frame and the KV cache, thereby reducing computational load.

[0067] Step 4 specifically involves:

[0068] S41. By caching the Key(K) and Value(V) vectors of historical frames, redundant calculations in overlapping areas of the sliding window are avoided. Sliding window: Each time the sliding window moves, the K and V of all frames within the window need to be recalculated, with a computational complexity of O(w·d). 2 (d is the feature dimension).

[0069] KV Cache: Only requires calculating the K and V of the new frame (complexity O(d)). 2 By reusing cached K and V to compute attention, the overall complexity is reduced to O(d). 2 +w·d).

[0070] The initialization and update of S42 and KV Cache includes the cache structure and initial state. The cache structure maintains two cache queues: Cache and KV Cache. K and Cache V , respectively store the K and V vectors of all historical frames within the current window. Initial state: for the initial input sequence X 1:w ∈{x1,x2,...,x w (Window length w), calculate the K and V values ​​for all frames and store them in the buffer:

[0071] Cache K ∈{K1,K2,...,K w},Cache V ∈{V1,V2,...,V w}, where K t =W K x t ,K t =W K x t (W K W V (This is the projection matrix).

[0072] S43, Streaming Inference Steps: When a new frame x w+1 Upon arrival, calculate the current query: Q. w+1 =W Q x w+1 Attention computation (using caching): directly utilizing the cache. K and Cache V Calculate attention output:

[0073]

[0074] Where d k The dimension of the Key.

[0075] Cache update (FIFO queue): The K of the new frame... w+1 and V w+1 Add to cache, and remove the oldest frame (sliding window).

[0076] (System): Cache K ←[Cache K [1:],K w+1 ],Cache V ←

[0077] [Cache V [1:],V w+1 ].

[0078] Using the above method, KV Cache reduces the computational cost of sliding window inference from O(w·d) 2 ) decreased to O(d 2 (per frame), significantly improving streaming inference efficiency.

[0079] S5. A window-based feature aggregation method based on statistical pooling (SP) is proposed. For the feature sequence obtained from the two-stage self-attention computation in S3, H... t =(h t-N+1 ,h t-N+2 ,…,h t The window representation is obtained by pooling statistical data: G t =Concat(Mean(H) t ),Var(H t )), where Mean(·) and Std(·) are the mean operator and variance operator, respectively.

[0080] Step 5 specifically involves:

[0081] For the feature sequence of N frames within the window: Aggregation can compress window information while preserving distribution characteristics; its function expression is:

[0082] h (w) =[μ(H out );σ(H out )]∈R 2h

[0083] in:

[0084] Mean pooling:

[0085]

[0086] Standard deviation pooling:

[0087]

[0088] ∈=1e-6 is the numerical stability constant.

[0089] S6. A wide-view prediction based on dilated convolution is proposed. Rich contextual information often improves prediction performance, but directly using a larger window function for self-attention leads to higher computational complexity (O(n²)). To address this, TSSA+SP is proposed to model local representations and obtain the local representation G. t Then, multiple local representations are further inferred through dilated convolution to obtain the prediction result at a certain time step. t =Conv(G t-N+1 G t-N+2 ,…,G t ).

[0090] Step 6 specifically involves:

[0091] Input pooling features from K consecutive windows: Where h window ∈R d Input shape: [batch_size,K,d] (batch size × number of windows × feature dimension), using 3 layers of dilated convolution (dilation rate r = 1, 2, 4): gradually expanding the receptive field.

[0092] Layer1:Conv1D(k=3,r=1)→ReLU

[0093] Layer2:Conv1D(k=3,r=2)→ReLU

[0094] Layer3:Conv1D(k=3,r=4)→ReLU

[0095] For layer l:

[0096]

[0097] Where k is the kernel size (e.g., k=3), and r is the dilation rate, which controls the expansion of the receptive field (usually increasing in increments of r=1, 2, 4, ...). The weight matrix is ​​a learnable matrix (shared across all time steps). The output of the previous layer at time step tr·i (with zero-padding boundaries) is then mapped to the load prediction value:

[0098]

[0099] Among them, w f ∈R 1×d Mapping d-dimensional features to scalar predicted values, b f ∈R 1×d For the bias term, the output shape is: [batch_size, K].

[0100] S7 employs a supervised learning framework based on root mean square error (RMSE), specifically optimized for time series prediction tasks. This module calculates the predicted value for each frame based on the prediction sequence output by module S5. Compared with the real label Y t The mean square error between the two values ​​is calculated, and the model parameter θ is adjusted by gradient descent to make the prediction as close as possible to the true value.

[0101] Step 7 specifically includes:

[0102] The model is trained by minimizing the root mean square error (RMSE) between the predicted output and the true label, based on the predicted sequence output by S5. Where n is the total number of samples, i.e., the number of prediction time points. Let Y be the model prediction value at time point i, i.e., the predicted load; the corresponding true label is Y = [y1; y2; ...; y...]. n ], where y i The actual observed value at time point i, i.e., the actual load, is then used to calculate the loss:

[0103]

[0104] The model parameters θ are optimized using gradient descent, and the update formula is as follows: Where η is the learning rate.

[0105] This invention proposes a power load modeling architecture based on cross-dimensional self-attention collaboration, which overcomes the challenge of representing complex load features through feature field decoupling and dynamic association mechanisms. Addressing the coupling characteristics of multi-source heterogeneous data such as time, temperature, and electricity price, a hierarchical coding system is designed: time features are subject to four-level normalized coding (year-month-week-day) to construct a time-series periodic basis vector; continuous variables such as temperature and electricity price are mapped using extreme value normalization to eliminate dimensional differences; and discrete features such as holidays are encoded using Boolean coding.

[0106] This invention designs a cascaded modeling framework based on streaming window functions and statistical pooling. It employs a sliding window to segment sequences, aligns spatiotemporal features through Z-score normalization and positional encoding, and utilizes two-level self-attention (TSSA) to perform cross-field correlation modeling and cross-time-step dependency capture within the window, overcoming the limitations of a single attention layer in modeling complex spatiotemporal relationships. Furthermore, a statistical pooling module is introduced to perform mean-variance aggregation on features from all time steps within the window, generating a high-level representation with robustness, significantly improving the model's resilience to sudden fluctuations in the power grid.

[0107] This invention proposes a dilated convolution-self-attention collaborative mechanism to address the receptive field limitation problem of existing architectures. A multi-stage dilated convolutional network (Dilated CNN) is deployed on the window-level feature sequence to capture load variation patterns across different periods, such as years, months, and weeks, through exponentially expanded dilatation rates. Simultaneously, a TSSA (Two-Stage Self-Attention) module is used to calibrate the convolutional output features, leveraging self-attention weights to enhance the representation of key events such as load spikes and sudden electricity price changes.

[0108] Specific implementation example:

[0109] To verify the effectiveness of the power load forecasting method based on the improved Transformer model described in this invention, the following example, "predicting the power load of the entire network for the next seven days," provides a complete forecasting process.

[0110] Implementation environment:

[0111] -Historical data source: Real data from the State Grid Dispatch Center;

[0112] - Forecast period: 168 hours from July 16th to July 22nd, 2025.

[0113] Electricity load at any given time (unit: megawatts - MW);

[0114] - Input data includes: historical 192 hours (8 days) of data from July 8, 2025 to July 15, 2025, including factors such as power load, temperature, electricity price, and holidays;

[0115] Prediction process and intermediate steps:

[0116] 1. Feature Construction (S1)

[0117] For each hour, construct an input vector, for example:

[0118] -Time: 2025-07-15 20:00 → [7 / 12, 15 / 30, 2 / 7, 20 / 24] = [0.58, 0.50, 0.29, 0.83]

[0119] -Temperature: 32℃ (with 50℃ as the extreme high temperature) → TEMP = 0.64

[0120] Electricity price: 0.75 (dynamically normalized)

[0121] -Holidays: No → 0

[0122] -Historical load: 1200MW (normalized to the maximum value of 3000MW) → LOAD = 0.40

[0123] - Final input vector: Xt=[0.58,0.50,0.29,0.83,0.64,0.75,0,0.40]∈R 8

[0124] 2. Streaming window function and normalization (S2)

[0125] The 192 consecutive feature vectors (i.e., 192 hours of history) are organized into a fixed-length sliding window sequence segment, and feature standardization and position embedding are performed.

[0126] - Window length N = 24 frames (i.e., 1 day), window shift S = 1 hour;

[0127] - Perform Z-score normalization and position encoding on each sliding window segment:

[0128] - Obtain the normalized input matrix X_norm∈R 24x8

[0129] 3. TSSA Two-Layer Attention Modeling (S3)

[0130] We utilize an improved two-stage self-attention mechanism (TSSA) to deeply mine factor correlations within frames (hours) of a sequence and temporal dynamic patterns between frames (hours).

[0131] Phase 1: Intra-frame attention learning of the weights of the impact of high temperature and high electricity price on load

[0132] ■ Input: The normalized feature vector x_norm_i∈R of a single frame i (a certain hour) within a single window. 8 .

[0133] ■Processing: Calculate the various features within frame i

[0134] Attention relationships between (such as TEMP, PRICE, HOLIDAY, LOAD, etc.)

[0135] Generate a Key, Value, and Query vector for each feature.

[0136] ■ Output: An enhanced feature vector z_i∈R that incorporates information about the mutual influence of features within the frame through a self-attention mechanism. h (h represents the embedding dimension)

[0137] Phase 2: Inter-frame attention recognition of periodicity (e.g., weekend load decline trend):

[0138] ■Input: 24 frame feature vectors Z = [z_1, z_2, ..., z_24]^T ∈ R after intra-frame attention enhancement throughout the entire window. 24xh .

[0139] ■ Processing: Calculate the time dependencies between these 24 hours within the calculation window. Generate K, Q, V values ​​for each frame (hour) relative to the other 23 frames. Time position encoding is implicitly included in this process.

[0140] ■ Output: The sequence feature matrix H = [h_1,h_2,...,h_24]^T∈R after inter-frame self-attention fusion. 24xh Each h_i vector contains the characteristics of that hour itself, as well as its contextual information related to the preceding and following time periods.

[0141] 4. KV Cache Streaming Inference (S4)

[0142] - Initialization: Buffer 192 historical K / V frames;

[0143] - Calculate attention output for future hourly queries and cached key-value pairs;

[0144] - Rolling update of the FIFO cache, calculating predicted values

[0145] - Compared to the standard Transformer, inference computation is reduced by 83%.

[0146] 5. Statistical Pooling Aggregation (S5)

[0147] Enhanced feature matrix H∈R of TSSA output 24xhStatistical aggregation is performed along the time dimension (24 frames) to calculate:

[0148] Mean vector μ∈R h : Indicates the overall trend of this window;

[0149] Standard deviation vector σ∈R h : Indicates the intensity of fluctuations within this window;

[0150] Aggregation: Combine the mean vector μ and the standard deviation vector σ;

[0151] Output the final aggregate vector: G_t = [μ, σ] ∈ R 2h ,

[0152] That is, a vector G_t with a length of 2h is generated for each 24-hour window.

[0153] -Enhance the model's robustness to sudden fluctuations (such as thunderstorms);

[0154] 6. Long-term modeling of dilated convolution (S6)

[0155] - Input the pooling results G∈R from 7 consecutive windows (7 days) 7x2h ;

[0156] - Perform three-layer convolution with dilation rate r = 1 / 2 / 4:

[0157] -Conv1 fluctuates within 3 hours;

[0158] -Conv2→Daily cycle modeling;

[0159] -Conv3→Weekly trend modeling;

[0160] - The final output is Y_pred∈R 168 This represents the load forecast for the next week.

[0161] 7. Supervised Optimization and Error Assessment (S7)

[0162] Dataset split: Training set (70%), validation set (15%, used for early stopping and hyperparameter tuning), test set (15%, used for final independent evaluation).

[0163] Input: Training samples: (G_window_seq, Y_true). Where G_window_seq is the statistical feature sequence of 7 consecutive days input into the S6 module, and Y_true is the real 7-day load sequence (168 hours) that follows it.

[0164] Loss function: Root Mean Square Error (RMSE). Calculate the mean squared difference between the predicted 7-day (168-hour) load value Y_pred and the actual value Y_true.

[0165] Optimizer: Gradient Descent Algorithm

[0166] Training process:

[0167] 1. Forward propagation: Input G_window_seq, pass through the model's S6 layer and its output layer, to obtain Y_pred.

[0168] 2. Calculate the loss: R MSE (Y_pred,Y_true).

[0169] 3. Backpropagation: Calculate the gradient of the loss with respect to all model parameters θ.

[0170] 4. Parameter Update: Use an optimizer (e.g., ... η is the learning rate. Update the model parameters.

[0171] 5. Iteration: Repeat steps 1-4 to traverse the entire training set (Epochs).

[0172] Convergence criterion: During training, monitor the training loss and validation loss. When the validation loss stops decreasing or starts to increase for several consecutive epochs, stop training (early stopping) and select the model with the lowest validation loss as the final model.

[0173] Comparison of prediction results at some points in time:

[0174]

[0175] Overall evaluation metrics (168 hours):

[0176]

[0177] n is the total number of samples, which is 7 days × 24 hours = 168 independent time periods. Let y be the model prediction value at time point i, i.e., the predicted load. i Let be the actual observed value at time point i, i.e., the actual load. The root mean square error between the model's prediction and the actual load is 8.85 MW, indicating that the model can predict the load relatively accurately.

[0178] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A method for predicting power load data based on an improved Transformer neural network model, characterized in that: It includes the following steps: S1. Propose a feature field construction and field encoding scheme; in the power load forecasting task, the factors considered include time (TIME), temperature (TEMP), electricity price (PRICE), holidays (HOLIDAY), and load (LOAD); when performing neural network modeling, these factors are vectorized to construct the input feature vector X. t ∈R 8 For time step t, X t =[TIME t (4), TEMP t (1),PRICE t (1),HOLIDAY t (1),LOAD t (1)]; S2. Propose a streaming window function; in real-time prediction tasks, for the historical first N-frame features X t-N:t , construct a window function W(·) based on the encoder of the transformer to capture local change patterns; set the window length to N and the window shift to S < N, that is, there is overlap between windows; considering the generalization of the window function, the features need to be normalized, that is: Mean(·) and Std(·) are the mean and standard deviation, respectively. ' The data follows a normal distribution; further, in-window position encoding is used to obtain X” = PE(X ' ), where PE is the position encoding matrix; S3. A two-level self-attention coding (TSSA) mechanism is proposed. The two-level self-attention coding design is proposed, namely, establishing intra-frame self-attention and inter-frame self-attention sequentially. The self-attention function is assumed to be y1, y2, ... = SA(x1, x2, ...), x i For the input feature x at time t, the feature is... t ={x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD }, where: x TIME For time-related features, x TEMP For temperature characteristics, x PRICE For electricity price characteristics, x HOLIDAY For holiday characteristics, x LOAD For the load value; each field vector is first projected onto the same dimension size: x i ∈{x TIME ,x TEMP ,x PRICE ,x HOLIDAY ,x LOAD },w i Let b be the field weight matrix. i For the field bias vector, we get (h is the uniform projection dimension), and then intra-frame self-attention calculation is performed to obtain The inter-frame self-attention calculation results at multiple time steps are as follows: h1,h2,…=SA inter (g1,g2,…) S4. Employ streaming inference based on KV Cache; extract the attention weights W from the window function model. Q W K W V Calculate and cache the key and value vectors for each frame; calculate the attention result based on the query vector of the current frame and the KV cache; S5. Propose in-window feature aggregation based on statistical pooling (SP); for the feature sequence obtained by the two-level self-attention calculation in S3, H t =(h t-N+1 ,h t-N+2 ,…,h t The window representation G is obtained by pooling statistical data. t =Concat(Mean(H) t ),Var(H t ), where Mean(·) and Std(·) are the mean operator and variance operator, respectively; S6. Propose a wide-view prediction based on dilated convolution; obtain the local representation G by modeling the local area using TSSA+SP. t Then, multiple local representations are further inferred through dilated convolution to obtain the prediction result at a certain time step; Y t =Conv(G t-N+1 G t-N+2 ,…,G t ); S7 employs a supervised learning framework based on the root mean square error (RMSE), specifically optimized for time series prediction tasks. This module calculates the predicted value for each frame based on the prediction sequence output by module S5. Compared with the real label Y t The mean square error between the two values ​​is calculated, and the model parameter θ is adjusted by gradient descent to make the prediction as close as possible to the true value.

2. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: In step S1, the specific factors to be investigated are as follows: Time (TIME): Considering the generalization of the model, here, according to the time periodicity at different granularities, encoding is performed. Taking the hourly-level data as an example, the cycle adopts the quantization method of month in year, day in month, day in week, and hour in day; Temperature (TEMP): The quantization method adopts the normalization strategy; Electricity price (PRICE): The quantization method adopts the normalization strategy; Holiday (HOLIDAY): Collect information on legal holidays and give a feature vector in boolean type; Load (LOAD): The quantization result of the load is normalized by using the historical maximum value as the denominator.

3. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S1 is specifically as follows S11. Based on historical load data, weather type, electricity price, and statutory holiday information, multi-dimensional feature encoding is performed. For time: considering the model's generalization ability, encoding is performed according to the time periodicity at different granularities. A multi-period normalization strategy is adopted, decomposing the timestamp into: annual period [month / 12], monthly period [date / 30], weekly period [day of the week / 7], and intraday period [hour / 24], preserving the periodicity of day and night, and week and month. TIME t =[month / 12,day / 30,weekday / 7,hour / 24]; S12. For environmental and market factors, including temperature (TEMP), electricity price (PRICE), and holidays (HOLIDAY), dynamic normalization is used; the formula for calculating temperature (TEMP) is: TEMP t =observed_temp / extreme_high_temp, where observed_temp represents the actual temperature value observed at time point t, and extreme_high_temp is the historical extreme high temperature of the region; the formula for calculating the electricity price PRICE is PRICE. t = (price - min_price) / (max_price - min_price), where price represents the actual electricity price observed at time point t, min_price represents the minimum reference price used in the normalization calculation based on a dynamic sliding window, and max_price represents the maximum reference price used in the normalization calculation, also based on the same dynamic sliding window as min_price; for holidays (HOLIDAY), the feature vector is given as a Boolean type. t ∈{0,1}, where 1 represents national statutory holidays and adjusted workdays; S13. Load LOAD represents the actual load value observed at time point t. The quantification result of the load is normalized using the historical highest value as the denominator. The calculation formula is LOAD. t = load / max_load, where max_load is the maximum load over the past three years.

4. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S2 is specifically as follows S21. The streaming window function encodes the features of the previous N frames in history through an encoder based on Transformer. A sliding window mechanism with a window length of N and a window shift of S is adopted to achieve local overlapping sampling, where S < N. The features within the window are normalized, subtracting the mean and dividing by the standard deviation. At the same time, position encoding within the window is introduced to capture the temporal dependence relationship; S22. Use the attention mechanism of Transformer to dynamically learn the local change rules of the features within the window, ensure temporal continuity through overlapping windows, normalize to improve the generalization of the model, and the position encoding retains the temporal information; S23. For time step t, the sliding window contains the features of the most recent N frames For X t-N:t ∈{x t-N ,x t-N+1 ,...,x t}∈R N×d , then for each feature vector x within the window i perform Z - score normalization: where x i ∈R d is the original input feature vector of the i - th frame without any processing, μ = is the mean vector within the window, is the standard deviation vector within the window. To preserve the temporal information, add sine position encoding to each frame within the window; let the position pos ∈ {0, 1,..., N - 1}, PE(pos, 2k) = sin(pos / 10000 2k / d ), where k is the dimension index, 0 ≤ k < d / 2, d is the feature dimension. Finally, the position encoding matrix PE ∈ R N×d has its i - th row corresponding to the position pos = i. Finally, add the normalized feature and the position encoding frame - by - frame to obtain the final window representation Expand it into matrix form where is the normalized feature matrix, PE is the position encoding matrix, and its i - th row PE i = PE(pos = i).

5. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S3 is specifically as follows S31. Propose a two-level self-attention encoding (TSSA); model two key dependence relationships of the time series in stages: Intra-Frame dependence within the frame: the association between different feature fields within the same time step, and Inter-Frame dependence between frames: the temporal dynamics between different time steps; S32. Intra-Frame self-attention performs self-attention calculation on the multivariate fields within each time step to model the non-linear coupling relationship between fields; within the same time step, capture the dependence relationship between different feature fields; Input window X t-N:t ∈R N×d For each frame x within the window i ∈R d Project the fields onto a uniform dimension h and calculate Q / K / V: Q intra (query) = x i W Q ∈R h K intra (key) = x i W K ∈R h V intra (value) = x i W V ∈R h Among them W Q W K W V ∈R d×h It is a learnable projection matrix; Calculate the attention between fields: Output enhanced frame features S33, Inter-Frame Self-Attention: Models long-term dependencies of sequences in the time dimension, capturing trends, periodicity, and event mutations; captures temporal dependencies between different time steps; For sequence Calculate inter-step attention: in, Then calculate each head independently: Among them W O ∈R h×h Finally, residual connections are performed to preserve the original feature information, layer normalization is applied to stabilize the training process, and temporal augmentation features are output. Where H intra For in-field attention output ∈ R N×h H inter For the time-interval attention output ∈ R N×h .

6. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S4 is specifically as follows S41. By caching the key and value vectors K and V of historical frames, redundant calculations in the overlapping areas of the sliding window are avoided; Sliding window: K and V of all frames within the window need to be recalculated each time the sliding window moves, with a computational complexity of O(w·d). 2 KVCache: Only the K and V of the new frame need to be calculated, and the cached K and V are reused to calculate attention, reducing the total complexity to O(d). 2 +w·d); The initialization and update of S42 and KV Cache includes the cache structure and initial state. The cache structure maintains two cache queues: Cache and KV Cache. K and Cache V Store the K and V vectors of all historical frames within the current window, respectively; Initial state: for the initial input sequence X 1:w ∈{x1,x2,...,x w }, where w is the window length, calculate the K and V of all frames and store them in the buffer: Cache K ∈{K1,K2,...,K w },Cache V ∈{V1,V2,...,V w }, where K t =W K x t ,K t =W K x t ; S43, Streaming Inference Steps: When a new frame x w+1 Upon arrival, calculate the current query: Q. w+1 =W Q x w+1 Directly utilize the cache in the cache K and Cache V Calculate attention output: Where d k The dimension of the Key; Buffer update FIFO queue: put the K of the new frame w+1 and V w+1 Add to cache, and remove the oldest frame at the same time: Hidden K ←[Cache K [1:],K w+1 ],Hidden V ← [Cache V [1:],V w+1 ] Using the above method, KV Cache reduces the computational cost of sliding window inference to O(w·d). 2 ) decreased to O(d 2 This improves the efficiency of streaming inference.

7. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S5 is specifically as follows For the feature sequence of N frames within the window: Aggregation can compress window information while preserving distribution characteristics; its function expression is: h (w) =[μ(H out );σ(H out )]∈R 2h Where: Mean pooling: Standard deviation pooling: ∈ = 1e-6 is a numerical stability constant.

8. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S6 is specifically as follows Input pooling features of K consecutive windows Where h w ∈R d The input shape is [batch_size,K,d] (batch size × number of windows × feature dimension), where the feature dimension is the pooled dimension 2h, k is the convolution kernel size, and r is the dilation rate. Three dilated convolutions are used with dilation rates r = 1, 2, 4 to gradually expand the receptive field. Layer1: Conv1D(k = 3, r = 1) → ReLU Layer2: Conv1D(k = 3, r = 2) → ReLU Layer3: Conv1D(k = 3, r = 4) → ReLU For the layer: Where k is the kernel size and r is the dilation rate, controlling the expansion of the receptive field. For learnable weight matrix, The output of the previous layer at time step tr·i is then mapped to the load prediction value: Among them, w f ∈R 1×d Mapping d-dimensional features to scalar predicted values, b f ∈R 1×d For the bias term, the output shape is: [batch_size, K].

9. The power load forecasting method based on the improved Transformer model according to claim 1, characterized in that: Step S7 is specifically as follows The model is trained by minimizing the root mean square error (RMSE) between the predicted output and the true label, based on the predicted sequence output by S5. Where n is the total number of samples, i.e., the number of prediction time points. Let Y be the model prediction value at time point i, i.e., the predicted load; the corresponding true label is Y = [y1; y2; ...; y...]. n ], where y i The actual observed value at time point i, i.e., the actual load, is then used to calculate the loss: Optimize the model parameters θ through gradient descent, and the update formula is: Where η is the learning rate.

Images