A load prediction method and system based on Transformer and diffusion decoder

Abstract

This invention relates to the field of power load forecasting technology. Specifically, it relates to a load forecasting method and system based on Transformer and diffusion decoder, aiming to solve the problems of traditional models failing to capture long-range temporal dependencies and multivariate coupling characteristics of load sequences, and failing to accurately reconstruct load waveform details, resulting in large prediction deviations across multiple time scales. This invention preprocesses multi-source time-series data of load, renewable energy output, and meteorology to extract five types of explicit temporal features. It uses a layered frozen pre-trained Transformer encoder to extract high-dimensional temporal features and fuses them with the temporal features to form a comprehensive feature matrix. By improving the cross-attention interaction of the diffusion model and the reverse denoising of the diffusion decoder, a multi-scale load forecast sequence is generated. After two stages of training and model optimization, the performance is evaluated by root mean square relative error and mean absolute error. This effectively captures complex temporal patterns, improves prediction accuracy, and provides reliable support for refined power grid scheduling and renewable energy consumption.

Description

Technical Field

[0001] This invention relates to the field of power load forecasting technology, and more specifically, to a load forecasting method and system based on Transformer and diffusion decoder. Background Technology

[0002] Power load forecasting is an important technology, specifically applied to the accurate forecasting of loads across multiple time scales in power systems. Its core lies in improving forecast accuracy through long-range feature mining and refined sequence generation to meet the core needs of power grid dispatching and renewable energy consumption. Power load time-series data is influenced by multiple factors, including meteorological conditions, renewable energy output, time cycles, and electricity consumption behavior, exhibiting complex characteristics of strong nonlinearity, non-stationarity, and long-range dependence. Traditional forecasting models struggle to fully capture the long-range time-series dependencies and multivariate coupling characteristics of load sequences, and the sequence generation methods cannot accurately reproduce the detailed fluctuation patterns of load waveforms. This leads to significant deviations between multi-time-scale load forecasting results and actual loads, hindering the support for refined power grid dispatching and safe, stable operation. To address this technical challenge, we present a load forecasting method and system based on Transformer and diffusion decoder. Summary of the Invention

[0003] The purpose of this invention is to provide a load prediction method and system based on Transformer and diffusion decoder to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, one objective of this invention is to provide a load prediction method based on Transformer and diffusion decoder, comprising the following steps: S1. Acquire and preprocess multi-source historical time-series data, including power system load data, new energy output data and meteorological data. Clean, fill and normalize the data, extract and filter core input features, and extract five types of explicit time features corresponding to each data point: hour, day, month, season, and weekday / holiday. S2. Input the preprocessed time series data into the adapted pre-trained Transformer encoder to extract high-dimensional temporal depth features; wherein, the pre-trained Transformer encoder adopts a layered freezing strategy, its front coding layer parameters are frozen, and only the back coding layer and output projection layer parameters are adjusted, and the length of the back window of the model input is dynamically configured according to the prediction length. S3. The various explicit temporal features extracted in step S1 are mapped into dense vectors through the embedding layer, and all temporal feature vectors are concatenated. Then, the concatenated temporal feature vectors are concatenated with the high-dimensional temporal depth features output by the pre-trained Transformer encoder in the feature dimension to form a comprehensive feature matrix, so as to achieve lossless fusion of temporal features and periodic features. S4. Based on the improved diffusion model, load prediction sequence is generated. The comprehensive feature matrix obtained in step S3 and the noise are input into the cross-attention layer for feature interaction. Then, the diffusion decoder performs the reverse denoising process to gradually reconstruct the future multi-timescale load prediction sequence from pure random noise. S5. A two-stage process is adopted to train and optimize the model. The first stage is the pre-training stage, which jointly trains the adjustable part of the pre-trained Transformer encoder and the diffusion decoder to learn general temporal representation and basic denoising ability. The second stage is the adjustment stage, which uses pure random noise as the input of the diffusion decoder and fixes all parameters of the pre-trained Transformer encoder, only adjusting the diffusion decoder to retain the waveform reconstruction ability of the model while avoiding future information leakage. S6. Input the time series data to be predicted into the model trained in step S5, output the load prediction results for the corresponding time scale, and use the root mean square relative error (RMSRE) and mean absolute error (MAE) as core indicators to evaluate the prediction performance.

[0005] A second objective of this invention is to provide a system for implementing a load prediction method based on Transformer and diffusion decoder as described in any one of the above-mentioned methods, comprising: The data preprocessing unit is used to acquire and preprocess multi-source historical time series data, extract and filter core input features, and extract five types of explicit time features for each data point: hour, day, month, season, and weekday / holiday. The feature extraction and fusion unit includes an adapted pre-trained Transformer encoder and an embedding layer. The Transformer encoder is used to receive processed time series data and extract high-dimensional time series deep features. The embedding layer is used to map various explicit time features into dense vectors and concatenate them. Finally, the concatenated time feature vectors are concatenated with the high-dimensional time series deep features in the feature dimension to form a comprehensive feature matrix. The sequence prediction generation unit is built based on an improved diffusion model, including a cross-attention layer and a diffusion decoder, which are used to interact the comprehensive feature matrix with noise through a cross-attention mechanism and perform a reverse denoising process to generate future multi-timescale load prediction sequences. The model optimization and evaluation unit is used to execute a two-stage model training process, which includes a pre-training phase and an adjustment phase, to optimize model parameters. It is also used to receive the time series data to be predicted during the prediction phase, output the load prediction results, and evaluate the prediction performance using root mean square relative error and mean absolute error.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a model architecture combining a Transformer encoder and a diffusion decoder. The encoder accurately captures the long-range temporal dependencies of power load sequences and the coupling characteristics of multiple variables such as weather, renewable energy output, and electricity consumption behavior. The diffusion decoder enables refined reconstruction of the load waveform, effectively addressing the problems of traditional models in capturing long-range dependencies of nonlinear and non-stationary load sequences and insufficient waveform detail reproduction. This significantly improves the accuracy of load forecasting across multiple time scales and substantially reduces prediction bias. The scheme can deeply integrate multiple influencing factors and complete feature selection and optimization, enhancing the model's generalization ability and adaptability to complex electricity consumption scenarios. It achieves accurate end-to-end prediction without relying on extensive manual feature engineering. Furthermore, the accurate prediction results provide reliable data support for refined grid scheduling and renewable energy consumption, effectively reducing scheduling decision errors, facilitating the smooth integration of renewable energy into the grid, ensuring the safe and stable operation of the grid, and comprehensively improving the intelligence level and operational efficiency of grid scheduling. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the overall workflow of the present invention; Figure 2 This is a schematic diagram of the overall structure of the present invention; The meanings of the labels in the diagram are as follows: 1. Data preprocessing unit; 2. Feature extraction and fusion unit; 3. Sequence prediction and generation unit; 4. Model optimization and evaluation unit. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] Please see Figure 1 As shown, one of the objectives of this embodiment is to provide a load prediction method based on Transformer and diffusion decoder, including the following steps: S1. Acquire and preprocess multi-source historical time series data, including power system load data, new energy output data and meteorological data. Clean, fill and normalize the data, extract and filter core input features, and extract five types of explicit time features corresponding to each data point: hour, day, month, season, and weekday / holiday. S2. Input the preprocessed temporal data into the adapted pre-trained Transformer encoder to extract high-dimensional temporal depth features. The pre-trained Transformer encoder adopts a layered freezing strategy, where the parameters of the front encoding layer are frozen, and only the parameters of the back encoding layer and the output projection layer are adjusted. The length of the back window of the model input is dynamically configured according to the prediction length. S3. The various explicit temporal features extracted in step S1 are mapped into dense vectors through the embedding layer, and all temporal feature vectors are concatenated. Then, the concatenated temporal feature vector is concatenated with the high-dimensional temporal depth features output by the pre-trained Transformer encoder in the feature dimension to form a comprehensive feature matrix, so as to achieve lossless fusion of temporal features and periodic features. S4. Based on the improved diffusion model, load prediction sequence is generated. The comprehensive feature matrix obtained in step S3 and the noise are input into the cross-attention layer for feature interaction. Then, the diffusion decoder performs the reverse denoising process to gradually reconstruct the future multi-timescale load prediction sequence from pure random noise. S5. A two-stage process is adopted to train and optimize the model. The first stage is the pre-training stage, which jointly trains the adjustable part of the pre-trained Transformer encoder and the diffusion decoder to learn general temporal representation and basic denoising ability. The second stage is the adjustment stage, which uses pure random noise as the input of the diffusion decoder and fixes all the parameters of the pre-trained Transformer encoder, only adjusting the diffusion decoder to retain the waveform reconstruction ability of the model while avoiding future information leakage. S6. Input the time series data to be predicted into the model trained in step S5, output the load prediction results for the corresponding time scale, and use the root mean square relative error (RMSRE) and mean absolute error (MAE) as core indicators to evaluate the prediction performance.

[0010] Historical time-series data, including power system load data, renewable energy output data, and meteorological data, were acquired. The historical time-series data were normalized, and the normalized historical time-series data were triple-validated using Pearson correlation coefficient, variance inflation factor (VIF), and multicollinearity condition number. Four types of input features were selected: average temperature, precipitation, photovoltaic output, and system load. For each time-series data point, five types of explicit time features were extracted: hour, day, month, season, and weekday / holiday. The preprocessed dataset was divided into training set, validation set, and test set in an 8:1:1 ratio, with the training set containing a complete year of time-series data.

[0011] In S2, the adaptation process of the pre-trained Transformer encoder includes: The four types of input features are mapped to a linear projection layer. For the word embedding dimension of the pre-trained model, the input back-view window length is configured as a fixed multiple of the prediction length, and a layered freezing strategy is adopted to freeze the word embedding. The parameters of the pre-trained model's front encoding layer are adjusted, only the parameters of the rear encoding layer and the output projection layer are adjusted.

[0012] Positional encoding information is injected into the input sequence, and the initial input representation is fed into the Transformer encoder. The long-range dependencies and multivariate coupling features of the sequence are captured through a multi-head self-attention mechanism. The Transformer encoder finally outputs a high-dimensional temporal representation vector.

[0013] The five types of explicit temporal features extracted from S1 are mapped into dense vectors through the embedding layer. All temporal feature vectors are then concatenated to obtain the overall temporal feature embedding vector. This overall temporal feature embedding vector is then concatenated to the feature dimension of the high-dimensional temporal representation vector output by the Transformer encoder to form a comprehensive feature matrix.

[0014] In S4, load forecast sequences are generated based on an improved diffusion model, including: The integrated feature matrix and noise are input into the cross-attention layer for feature interaction. The integrated feature matrix is ​​used as the key and value, and the intermediate representation of the decoder is used as the query. The diffusion decoder adopts the Transformer decoder structure and is configured with multi-head cross-attention and mask self-attention. Cosine scheduling is used to control the noise addition process.

[0015] The diffusion model performs a forward noise addition process, gradually adding Gaussian noise to the original load data. The diffusion model then performs a reverse denoising process, gradually recovering the true load data from the pure Gaussian noise. The model is trained by predicting the added noise through a denoising neural network.

[0016] The first stage is the pre-training stage, which jointly trains the adjustable part of the pre-trained Transformer encoder and the diffusion decoder to learn general temporal representations and basic denoising capabilities. The second stage is the tuning stage, which uses pure random noise as the input to the diffusion decoder and fixes all parameters of the pre-trained Transformer encoder, only tuning the diffusion decoder.

[0017] In S6, the root mean square relative error (RMSRE) and mean absolute error (MAE) are used as core indicators to evaluate forecast performance, and a rolling forecast method is used to calculate the monthly forecast accuracy.

[0018] Further explanation is needed. First, three types of multi-source historical time-series data were collected from the target area: power system load data, renewable energy output data, and meteorological data, with a time granularity of 1 hour / point and a time span covering a full year. The collected raw time-series data underwent cleaning, and linear interpolation was used to fill in missing and outlier values. For periods with excessively large missing values, the corresponding day-ahead forecast values ​​were used to supplement the data, ensuring the continuity and integrity of the time-series data. Then, all the cleaned numerical time-series data were normalized using Z-Score standardization, mapping the data to a standard distribution with a mean of 0 and a variance of 1, effectively improving the convergence speed and stability of subsequent model training. Next, the normalized historical time-series data underwent triple feature verification using Pearson correlation coefficient, variance inflation factor (VIF), and multicollinearity condition number, strictly adhering to the Pearson correlation coefficient... Feature selection was conducted using criteria such as coefficients > 0.1, variance inflation factor (VIF) < 5, and multicollinearity condition number < 100. This resulted in the precise identification of four core input features: average temperature, precipitation, photovoltaic output, and system load. After selecting these core features, for each time-series data point, five explicit time features—hourly, daily, monthly, seasonal, and weekday / holiday—were extracted from the corresponding timestamp. This provides a foundation for the subsequent fusion of time-series and periodic features. Finally, the fully preprocessed dataset, after data cleaning, normalization, triple-validation selection, and time feature extraction, was divided into training, validation, and test sets in a fixed ratio of 8:1:1. The training set ensured to include a complete year of time-series data to cover the full-cycle load variation patterns, while the test set covered typical operating scenarios such as high-temperature, high-load summers and winter heating. This provided standardized and reasonably distributed data support for model training, parameter validation, and performance testing.

[0019] In the S2 module, the adapted pre-trained Transformer encoder ( The entire process of adaptation and feature extraction is completed. Based on four core input features, linear projection mapping, back window configuration, hierarchical parameter freezing, positional encoding injection, and multi-head self-attention feature extraction are performed sequentially to finally output a high-dimensional temporal representation vector. The specific implementation method is as follows: First, linear projection adaptation is performed on the four input features of average temperature, precipitation, photovoltaic output, and system load. The original input features are mapped to the linear projection layer. The formula for calculating the word embedding dimension of the pre-trained model using linear projection is: in The original input matrix consists of four types of input features. These are the weight parameters for the linear projection layer. For bias terms, To achieve dimension-aligned projected feature matrices, ensuring a perfect match between the input feature dimensions and the word embedding dimensions of the pre-trained model, a back-look window length is then configured based on the prediction length, setting the back-look window length of the input sequence to a fixed multiple of the prediction length. ,Right now This fixed-multiple configuration allows the input sequence to cover the entire historical time series, accurately matching the historical feature dependencies required for prediction length, and providing a suitable sequence length basis for long-range dependency capture. Next, a layered freezing strategy is used to manage the parameters of the pre-trained model, directly freezing them. All parameters of the pre-trained model's front-end encoding layer (weights and biases remain unchanged from their initial pre-training values) are used, with only the parameters of the rear-end encoding layer and output projection layer being made available for iterative adjustment. This preserves the basic feature extraction capabilities learned by the pre-trained model from massive amounts of general time-series data while allowing the model to adapt to the specific scenario features of power load time-series prediction, balancing model generalization and scenario adaptability. After parameter adaptation, positional encoding information is injected into the input sequence to address the issue of the Transformer encoder's inability to perceive temporal position. The positional encoding uses a sine-cosine positional encoding method, calculated using the following formula: ,in For the temporal position index of the input sequence, Encode the dimension index for location. The feature dimension of the pre-trained model; the position encoding matrix With the projected input feature matrix Adding elements together, we obtain the final input sequence containing the injected position information: This allows the encoder to perceive the positional relationships within the temporal sequence. The initial input sequence... The adapted Transformer encoder captures long-range dependencies and multivariate coupling features of the temporal sequence through a multi-head self-attention mechanism. The multi-head self-attention calculation formula is as follows: The formula for calculating single-head attention is: in These are query, key, and value matrices, respectively. For the number of attention heads, For single-head attention feature dimensions, For single-head attention projection weights, The multi-head attention outputs projection weights; through parallel computation of multi-head self-attention, it simultaneously captures the long-range temporal dependence of the load time series, as well as the coupled correlation features of four types of variables: temperature, precipitation, photovoltaic, and load. Finally, the Transformer encoder aggregates and upscales the temporal features that integrate long-range dependence, multivariate coupling, and location information, ultimately outputting a high-dimensional temporal representation vector. This vector fully carries the coupling relationship of the four types of input features, the long-range dependency pattern of the time sequence, and the positional encoding information, providing high-quality, high-dimensional time sequence feature input for subsequent prediction and decoding.

[0020] The five explicit time features extracted from S1—hourly, daily, weekly, monthly, and seasonal features—are sequentially embedded, mapped, concatenated, and fused to form a comprehensive feature matrix. The specific implementation method is as follows: First, five types of explicit temporal features are fed into independent learnable embedding layers. Each embedding layer, based on a pre-defined unified embedding dimension d_emb, uses learnable weight matrices and bias terms to map discrete categorical temporal features one by one into dense vectors with unified dimensions and dense information, achieving a non-linear transformation from discrete temporal features to continuous dense vectors. Then, the dense vectors corresponding to the five types of temporal features are sequentially concatenated along the feature dimension, preserving the independent semantic information of each type of temporal feature. The concatenated vector yields a holistic temporal feature embedding vector that integrates the complete information of the five types of explicit temporal features. Finally, the holistic temporal feature embedding vector is concatenated along the feature dimension into the high-dimensional temporal representation vector output by the Transformer encoder, keeping the temporal length dimension unchanged and only expanding the feature dimension. This concatenation of the feature dimension achieves deep fusion of temporal representation features and temporal features, ultimately forming a comprehensive feature matrix that carries long-range temporal dependencies and multivariate coupling features while integrating explicit temporal feature information. This provides a complete, high-dimensional, multimodal feature input for the subsequent decoding module.

[0021] Accurate prediction of temporal loads is achieved based on an improved diffusion model. First, the integrated feature matrix obtained from previous fusion and randomly initialized Gaussian noise are simultaneously input into a cross-attention interaction layer to achieve deep interaction and information alignment of multi-source features. The cross-attention layer uses the integrated feature matrix as the key (K) and value (V) vectors, and the intermediate implicit representation of the diffusion decoder as the query (Q) vector. Through a multi-head cross-attention mechanism, it achieves deep coupling between global features and local temporal features, providing highly adaptable feature input for the subsequent diffusion process. The diffusion decoder used in this study is a dedicated Transformer decoder structure, configured with both a multi-head cross-attention module and a masked self-attention module. The masked self-attention module uses an upper triangular mask matrix to shield future temporal information, retaining only the effective transmission of historical temporal features to ensure the causal compliance of load prediction. The multi-head cross-attention module continuously connects to the previously obtained integrated feature matrix, completing real-time interaction between external global features and internal temporal features. The two attention modules work together to achieve deep feature extraction and information interaction. The forward noise addition process of the diffusion model employs a cosine scheduling strategy to control the rhythm and amplitude of noise addition, gradually adding Gaussian noise to the original true load time series data until the original data is converted into a pure Gaussian noise distribution. Its forward noise addition recursive formula strictly follows the definition: In the formula: This is the load time series data after adding noise at step t. For the time series data at step t-1, Let be the noise scheduling coefficient at step t. Add standard Gaussian noise at step t and satisfy the following conditions: The noise scheduling coefficient is generated by a cosine function, and the formula is: In the formula: , Here, $T$ represents the upper and lower limits of the scheduling coefficient, and $T$ represents the total number of noise-adding steps. This represents the current noise addition step. After the current noise addition is completed, the diffusion model performs a reverse denoising process, iteratively recovering the true load time-series forecast data from the pure Gaussian noise distribution. Each step of the reverse process predicts the Gaussian noise added in the current step using a denoising neural network, and then gradually removes the noise components of the corresponding step based on the predicted noise, until a load forecast sequence that closely matches the true distribution is recovered. Its reverse denoising recursive formula strictly follows the definition: In the formula: The noise at step t is predicted by the denoising neural network. It is the cumulative product of the scheduling coefficients in the previous t steps. The noise figure for the reverse process. The noise is random Gaussian noise (set to 0 in the final step). The training process of the diffusion model focuses on optimizing the noise prediction error of the denoising neural network. A supervised training loss function is constructed, which adopts the mean squared error form and strictly follows the definition: Randomly sample the number of noisy steps during training Obtain the noisy data for the corresponding step. With real noise ,Will With steps The noise is input into the denoising neural network to obtain the predicted noise. The network parameters are continuously optimized based on the loss function until the noise prediction error converges, thus completing the overall training of the improved diffusion model.

[0022] The first stage is the pre-training stage, whose core objective is to enable the model to learn general temporal representations and basic denoising capabilities. This involves jointly training the adjustable part of the pre-trained Transformer encoder with the diffusion decoder. First, pre-processed multi-source temporal features are input into the pre-trained Transformer encoder, whose adjustable part extracts general temporal representations. These representations are then fed into the diffusion decoder, simultaneously performing forward denoising and backward denoising iterations based on a given training loss function. The first stage involves jointly optimizing the adjustable parameters of the encoder and the diffusion decoder, allowing the model to fully learn the general patterns of load time series and basic denoising rules. The second stage is the adjustment stage, specifically for fine-tuning the diffusion decoder. First, all parameters of the pre-trained Transformer encoder are fixed, and its weights are no longer updated. Then, pure random noise is used as the input to the diffusion decoder, and only the parameters of the diffusion decoder are fine-tuned. Under pure random noise input, the diffusion decoder still performs specific fitting optimization based on the aforementioned loss function. The encoder parameters are kept frozen throughout, and only the denoising and sequence generation capabilities of the decoder are optimized, allowing the model to ultimately meet the accurate output requirements of power load forecasting, thus completing the entire two-stage training.

[0023] In the S6 forecasting performance evaluation phase, a quantitative evaluation system based on Mean Absolute Error (MAE) and Root Mean Square Relative Error (RMSRE) was strictly constructed according to requirements. A rolling forecasting model was simultaneously adopted to complete full-cycle forecasting, and monthly forecast accuracy was accurately calculated. All indicator calculations and process execution strictly followed established formulas and specifications. First, the core evaluation indicators were calculated, with MAE used to measure the absolute deviation between the predicted and actual values. The calculation formula strictly adhered to a predetermined form: In the formula, To assess the total number of samples, For the first The actual load value of each sample. For the first The model's predicted value for each sample is used to visually reflect the overall absolute deviation of the model's predictions. The root mean square relative error (RMSRE) measures the relative deviation of the predicted value from the true value and is a core relative indicator for evaluating prediction accuracy. Its calculation formula strictly follows a predetermined form: The symbols in the formula have the same meaning as the MAE index, which effectively eliminates the influence of dimensions and accurately reflects the relative error level of the model prediction. Subsequently, a rolling forecasting method is used to perform full-cycle forecasting and evaluation. Based on a fixed-length sliding window, a forecast sequence is generated point-by-point according to the time series. The window length is set to the number of monthly sampling points based on the characteristics of the monthly load data. After the forecast for each time point is completed, the actual value of that point is immediately included in the evaluation sample, and the window slides forward by one time point simultaneously. This process of point-by-point forecasting, evaluation, and sliding is maintained throughout, fully replicating the real-world scenario of the entire monthly load forecasting process and ensuring a high degree of consistency between the evaluation results and the actual application scenario. Finally, based on the full-sequence results of the rolling forecast, the monthly forecast accuracy is calculated. The calculation formula strictly follows a predetermined form, using the monthly root mean square relative error as a benchmark. The monthly forecast accuracy is defined as 1 minus the monthly root mean square relative error. The specific formula is as follows: In the formula, The root mean square relative error of the monthly full-sample rolling forecast is given by this formula. The monthly forecast accuracy can be directly obtained through this formula, which accurately quantifies the monthly forecast performance of the model and provides an objective quantitative evaluation basis for subsequent model optimization and parameter tuning.

[0024] The second objective of this invention is to provide a system for implementing a load prediction method based on Transformer and diffusion decoder, including any one of the above-mentioned methods, comprising: The data preprocessing unit 1 is used to acquire and preprocess multi-source historical time series data, extract and filter core input features, and extract five types of explicit time features for each data point: hour, day, month, season, and weekday / holiday. The feature extraction and fusion unit 2 includes an adapted pre-trained Transformer encoder and an embedding layer. The Transformer encoder is used to receive the processed time series data and extract high-dimensional time series deep features. The embedding layer is used to map various explicit time features into dense vectors and concatenate them. Finally, the concatenated time feature vectors and high-dimensional time series deep features are concatenated in the feature dimension to form a comprehensive feature matrix. The sequence prediction generation unit 3 is built based on an improved diffusion model, including a cross-attention layer and a diffusion decoder, which is used to interact the comprehensive feature matrix and noise through the cross-attention mechanism and perform a reverse denoising process to generate future multi-timescale load prediction sequences. The model optimization and evaluation unit 4 is used to execute a two-stage model training process that includes a pre-training stage and an adjustment stage to optimize model parameters. It is also used to receive the time series data to be predicted in the prediction stage, output the load prediction results, and evaluate the prediction performance using root mean square relative error and mean absolute error.

[0025] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A load prediction method based on Transformer and diffusion decoder, characterized in that: Includes the following steps: S1. Acquire and preprocess multi-source historical time-series data, including power system load data, new energy output data and meteorological data. Clean, fill and normalize the data, extract and filter core input features, and extract five types of explicit time features corresponding to each data point: hour, day, month, season, and weekday / holiday. S2. Input the preprocessed time series data into the adapted pre-trained Transformer encoder to extract high-dimensional temporal depth features; wherein, the pre-trained Transformer encoder adopts a layered freezing strategy, its front coding layer parameters are frozen, and only the back coding layer and output projection layer parameters are adjusted, and the length of the back window of the model input is dynamically configured according to the prediction length. S3. The various explicit temporal features extracted in step S1 are mapped into dense vectors through the embedding layer, and all temporal feature vectors are concatenated. Then, the concatenated temporal feature vectors are concatenated with the high-dimensional temporal depth features output by the pre-trained Transformer encoder in the feature dimension to form a comprehensive feature matrix, so as to achieve lossless fusion of temporal features and periodic features. S4. Based on the improved diffusion model, load prediction sequence is generated. The comprehensive feature matrix obtained in step S3 and the noise are input into the cross-attention layer for feature interaction. Then, the diffusion decoder performs the reverse denoising process to gradually reconstruct the future multi-timescale load prediction sequence from pure random noise. S5. A two-stage process is adopted to train and optimize the model. The first stage is the pre-training stage, which jointly trains the adjustable part of the pre-trained Transformer encoder and the diffusion decoder to learn general temporal representation and basic denoising ability. The second stage is the adjustment stage, which uses pure random noise as the input of the diffusion decoder and fixes all parameters of the pre-trained Transformer encoder, only adjusting the diffusion decoder to retain the waveform reconstruction ability of the model while avoiding future information leakage. S6. Input the time series data to be predicted into the model trained in step S5, output the load prediction results for the corresponding time scale, and use the root mean square relative error (RMSRE) and mean absolute error (MAE) as core indicators to evaluate the prediction performance.

2. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: Historical time-series data, including power system load data, renewable energy output data, and meteorological data, are acquired. This historical time-series data is then normalized, and triple verification is performed on the normalized data using Pearson correlation coefficient, variance inflation factor (VIF), and multicollinearity condition number. Four types of input features are selected: average temperature, precipitation, photovoltaic output, and system load. For each time-series data point, five explicit time features are extracted: hour, day, month, season, and weekday / holiday. The preprocessed dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio, with the training set containing a complete year of time-series data.

3. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: In S2, the adaptation process of the adapted pre-trained Transformer encoder includes: The four types of input features are mapped to a linear projection layer. For the word embedding dimension of the pre-trained model, regarding the prediction length, the length of the input back-view window is configured as a fixed multiple of the prediction length, and a layered freezing strategy is adopted to freeze the word embedding. The parameters of the pre-trained model's front encoding layer are adjusted, only the parameters of the rear encoding layer and the output projection layer are adjusted.

4. The load prediction method based on Transformer and diffusion decoder according to claim 3, characterized in that: S2 further includes: injecting positional encoding information into the input sequence, inputting the initial input representation into the Transformer encoder, capturing the long-range dependencies and multivariate coupling features of the sequence through a multi-head self-attention mechanism, and the Transformer encoder finally outputting a high-dimensional temporal representation vector.

5. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: The five types of explicit temporal features extracted in S1 are mapped into dense vectors through the embedding layer. All temporal feature vectors are then concatenated to obtain the overall temporal feature embedding vector. This overall temporal feature embedding vector is then concatenated to the feature dimension of the high-dimensional temporal representation vector output by the Transformer encoder to form a comprehensive feature matrix.

6. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: In S4, the generation of the load prediction sequence based on the improved diffusion model includes: The integrated feature matrix and noise are input to the cross-attention layer for feature interaction, where the integrated feature matrix is ​​used as the key and value, and the intermediate representation of the decoder is used as the query. The diffusion decoder adopts the Transformer decoder structure and is configured with multi-head cross-attention and mask self-attention. The noise addition process is controlled by cosine scheduling.

7. The load prediction method based on Transformer and diffusion decoder according to claim 6, characterized in that: The diffusion model performs a forward noise addition process, gradually adding Gaussian noise to the original load data. The diffusion model then performs a reverse denoising process, gradually recovering the true load data from the pure Gaussian noise. The model is trained by predicting the added noise through a denoising neural network.

8. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: The first stage is the pre-training stage, in which the adjustable part of the pre-trained Transformer encoder and the diffusion decoder are trained together to learn general temporal representations and basic denoising capabilities. The second stage is the adjustment stage, in which pure random noise is used as the input of the diffusion decoder, and all parameters of the pre-trained Transformer encoder are fixed, while only the diffusion decoder is adjusted.

9. The load prediction method based on Transformer and diffusion decoder according to claim 1, characterized in that: In S6, the root mean square relative error (RMSRE) and mean absolute error (MAE) are used as core indicators to evaluate forecast performance, and a rolling forecast method is used to calculate the monthly forecast accuracy.

10. A system for implementing a load forecasting method based on Transformer and diffusion decoder as described in any one of claims 1-9, characterized in that, include: The data preprocessing unit (1) is used to acquire and preprocess multi-source historical time series data, extract and filter core input features, and extract five types of explicit time features for each data point: hour, day, month, season, and weekday / holiday. The feature extraction and fusion unit (2) includes an adapted pre-trained Transformer encoder and an embedding layer. The Transformer encoder is used to receive processed time series data and extract high-dimensional time series deep features. The embedding layer is used to map various explicit time features into dense vectors and concatenate them. Finally, the concatenated time feature vectors are concatenated with the high-dimensional time series deep features in the feature dimension to form a comprehensive feature matrix. The sequence prediction generation unit (3) is built based on the improved diffusion model, including a cross-attention layer and a diffusion decoder, which is used to interact the comprehensive feature matrix with noise through the cross-attention mechanism and perform a reverse denoising process to generate future multi-timescale load prediction sequences. The model optimization and evaluation unit (4) is used to execute a two-stage model training process including a pre-training stage and an adjustment stage to optimize model parameters, and to receive the time series data to be predicted in the prediction stage, output the load prediction results, and evaluate the prediction performance using root mean square relative error and mean absolute error.

Images