Knowledge distillation and time self-attention additive neural network-based interpretable load prediction method

Abstract

The invention discloses an interpretable load prediction method based on knowledge distillation and a time self-attention additive neural network, and the method comprises the steps: collecting the historical load and meteorological data of a power grid as the input characteristics of a model, carrying out the detection of a data quartile abnormal value, dividing the data quartile abnormal value into a training set, a test set and a verification set, and carrying out the detection of the data quartile abnormal value; standardization and abnormal value filling are carried out through Z-shaped orthogonalization and linear filling, and finally, a tensor form meeting the model input requirement is converted through a sliding window; designing a knowledge distillation'teacher-student 'framework based on multiple scales and multiple cycles; constructing a time self-attention additive neural network TSA-NAM as a student model; calculating a shape function representing the contribution degree and the characteristic value in the sub-network to obtain the interpretability of the characteristic dimension; exporting the attention weight of the time self-attention module to obtain the interpretability of the time dimension; performing simulation verification; according to the method, high reliability and high precision are guaranteed, and meanwhile, multi-dimensional interpretability is brought to power load prediction.

Description

Technical Field

[0001] This invention relates to the field of smart grid dispatching technology, and in particular to an interpretable load forecasting method based on knowledge distillation and time-dependent attention additive neural networks. Background Technology

[0002] With the establishment of the construction goals for new power systems, the promotion of increased renewable energy installed capacity, and the reform of electricity market policies, traditional load forecasting methods are becoming increasingly inadequate for the current grid environment with its complex source-load coupling relationships. To address the challenges of real-time power supply and demand balance and ensure the safe and stable operation of the power grid, high-precision load forecasting methods have become a crucial support for building a reliable new power system.

[0003] Regional load prediction methods based on artificial intelligence "black box" models can better fit the nonlinear relationship between input and output. Currently, these methods can be divided into traditional machine learning methods and deep learning methods. Traditional machine learning methods include random forests, support vector machines, extreme gradient boosting, and artificial neural networks, etc. However, with the development of new power systems, these methods cannot adapt to the increasingly complex load sequences. Deep learning methods can perform large-scale, high-dimensional data analysis, and through recurrent neural networks, convolutional neural networks, generative adversarial networks, and other methods for feature extraction, they improve the model's generalization ability and effectively enhance load prediction accuracy.

[0004] While existing research has achieved high accuracy by constructing high-capacity models, the deep, stacked neural networks create complex, multi-layered abstractions. This inherent "black box" characteristic makes the model's decision-making process difficult to trace, raising security and reliability issues in practical applications. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and provide an interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural networks, comprising the following steps: Step 1: Collect historical load and meteorological data of the power grid as input features of the model. After detecting outliers in the quartiles of the data, divide it into training set, test set and validation set. Then, standardize it and impute outliers through Z-orthogonalization and linear filling. Finally, convert it into tensor form that meets the input requirements of the model through a sliding window. Step 2: Design a "teacher-student" framework for knowledge distillation based on multiple scales and multiple periods: Decompose the output of the teacher model into multi-scale time domain and key period frequency domain signals through one-dimensional convolutional downsampling and sparse Fourier transform, so as to transfer the high-precision deep "knowledge" of the teacher model to the interpretable student model. Step 3: Construct a temporal self-attention additive neural network TSA-NAM as a student model: The additive neural network can independently learn the nonlinear relationship between each feature and the load through parallel sub-networks. By adding a parallel temporal self-attention module to the sub-network, the dependence and importance of each feature in the time dimension can be dynamically captured. Step 4: Calculate the shape function representing the relationship between contribution and feature value in the sub-network using a percentile-based binning aggregation method to obtain the interpretability of the feature dimension; obtain the interpretability of the time dimension by deriving the attention weights of the temporal self-attention module. Step 5: Simulation verification to analyze the effectiveness of the model.

[0007] Preferably, step 1 includes the following process: The standard calculation process for using quartiles to detect outliers is as follows: ; ; in: Q 1 is the lower quartile. Q 3 is the upper quartile. IQR Interquartile range; not R norm Data within the specified range was identified as outliers; outliers and missing values ​​were handled using linear interpolation, calculated as follows: ; n t Data values ​​to be filled in; s The distance between the next available data and the data to be filled; n t+s Indicates the next available data; n t-s This represents the previous available data; the data is standardized using Z-Score, which is calculated as follows: ; in: , σ denoted as the mean and variance of the feature quantities, respectively, and n' is the standardized value; Standardization brings the input features to the same order of magnitude as the power load data; finally, a sliding window is used to convert the data into tensor form.

[0008] Preferably, the specific process of converting the input data into a tensor form that meets the model input requirements through a sliding window in step 1 is as follows: First, define the window parameters and the number of time steps for extracting historical features and the number of future time steps to be predicted. M and N Subsequently, a sliding window is used to generate samples: starting from the beginning of the time series, one time step is slid in at a time to extract continuous samples. N A historical value is used as input, corresponding to its subsequent... M The nth value is output as the nth t The input and output of a sample can be expressed by the following formula: ; ; in: X t and Y t They are respectively t Input and output of each sample; finally, mathematical expression: given a time series The sample set can be represented as: .

[0009] Preferably, step 2 includes the following process: Step 2.1 Knowledge Distillation The "Teacher-Student" framework adopts three core modules: multi-scale distillation, multi-period distillation, and supervised learning. Multi-scale distillation is used to transfer knowledge from the representation of the same time series at different sampling frequencies. One-dimensional convolution operation is used to downsample the original scale to generate predictions at a more detailed time resolution. Repeated downsampling operation can obtain multi-scale sets of teacher and student models. Step 2.2, Multi-Period Distillation, guides the student model to learn and replicate the deep periodicity knowledge captured by the teacher model in the time series by aligning the periodic distribution of the teacher and student models in the frequency domain; the original prediction sequence is converted from a time domain signal to a frequency domain signal using Discrete Fourier Transform and amplitude calculation (Amp); subsequently, to highlight important frequencies and reduce noise interference from secondary frequencies, the frequency domain distribution is converted into a probability distribution using the Softmax function; finally, the periodic distribution of the teacher and student models is aligned using KL divergence as the loss function. Step 2.3: Use real data as a benchmark to supervise the student model's predicted output, quantifying the direct difference between the student model's predicted output and the real data, ensuring the authenticity and effectiveness of the model's predictions; the specific process of the supervised learning loss function is expressed by the following formula: ; in:Y truth As a baseline, real future load data, Y student For student model predicted values, L hard The loss function for supervised learning; Step 2.4: The total loss function in the distillation process is composed of the soft label loss and the supervised learning loss function, and can be expressed by the following formula: ; ; ; ; in: L scale The loss function represents the multi-scale distillation; by using the mean squared error (MSE) loss across all scales and summing the results, the teacher-student model can achieve prediction alignment at different temporal resolutions. L period This is the loss function for multi-cycle distillation; L soft and L totel These are represented as the soft tag and distillation total loss functions, respectively. λ and δ The weights for multi-scale distillation / soft labeling are respectively, when λ The larger the value, the greater the trend of learning load changes in the student model from the teacher model's predictions. λ The smaller the value, the more the student model focuses on learning the periodic characteristics of the load. δ Similarly, weights are used to allocate the student model's learning from real data and the teacher model's predictions. Q teacher This represents the periodic probability distribution of the teacher model after Softmax calculation; Q student This represents the periodic probability distribution of the student model after Softmax calculation; Indicates time resolution as m The predicted values ​​of the teacher model under the given conditions; Indicates time resolution as m The predicted values ​​of the student model. Preferably, the process of step 2.1 can be expressed by the following formula: ; ; in, Time resolution is mThe predicted value of the teacher or student model; stride is the one-dimensional convolution step size; y x Represents a multi-scale set of predictions from teacher or student models; M This represents the number of times the downsampling operation is repeated.

[0010] Preferably, the calculation process in step 2.2 can be expressed by the following formula: ; ; in: Y x For the predicted values ​​of the teacher or student model, A x Output the calculated spectrum for the teacher or student model. , These are the numbers in the spectrum diagram. i , j The amplitude of each; T o To predict the step size; by adjusting the temperature parameter. τ To control the probability distribution, when τ The smaller the value, the more concentrated the output probability distribution will be on the most critical periodic information for load forecasting, filtering out secondary frequencies; conversely, the distribution of probability weights will be smoother. Q x This is the periodic probability distribution of the teacher or student model calculated using Softmax.

[0011] Preferably, step 3 includes the following process: A parallel TSA module is constructed to dynamically capture the dependencies and importance of each feature in the time dimension, providing interpretability in the time dimension and improving the accuracy of load forecasting without affecting the transparency of the "white box" model. The nonlinear relationship between different features and loads is learned through neural networks. Then, the additive structure preserves the independent influence of each feature on the model output. Finally, additive aggregation is performed to obtain the final predicted output. NAM is expressed by the following formula: ; in: g Indicates the target variable to be output. f n ( x n ) as the representative of the first n Input features x The shape function, ε For residual terms; Each of NAM f n (x n All of these are derived from neural network fitting and visualized. f n ( x n )and x n The shape function is used to understand the relationship between different features and the output; the NAM model uses a multilayer perceptron as its basic building block; the model constructs an independent feature network for each input feature, and each feature network learns in parallel to capture the nonlinear relationship between different features and the target variable, in order to fit the shape function. f n ( x n The outputs of all feature networks are additively aggregated to obtain the final output of the NAM network. The TSA module calculates attention weights at different time steps, enabling the model to adaptively focus on key time points in the time series, thereby improving the model's prediction accuracy. The magnitude of the attention weights can reflect the importance of information at each time point to the final prediction, thus enhancing the model's interpretability. After the raw data is split according to features, it is fed into its dedicated sub-network. First, the attention weights at different time steps are dynamically calculated through the self-attention module to identify key node information in the sequence. Then, the processed weighted feature sequence is passed to the MLP for further calculation to obtain the shape function of the feature with respect to the load. Finally, the results of all feature sub-networks are additively aggregated to obtain the final predicted value. Global and local interpretability are obtained through the attention network and the additive structure of the model.

[0012] Preferably, the calculation process for attention weights is as follows: ; ; ; ; in: X n For the first n The input vector of each feature, T i Indicates the time step of the input features. W 1, W 2 represents the weight matrix; b 1, b 2 represents the bias term of the weight matrix. h att The result is the calculation result of the hidden layer; β logitsOriginal attention score; β n This is the final attention weight vector.

[0013] Preferably, step 4 includes the following process: A percentile-based binning aggregation method is used to calculate and mine the shape function representing the relationship between contribution and eigenvalues. The process is as follows: ; ; ; ; ; in: For the first n The first feature i scalar eigenvalues The length of the load sequence L No. l 1 eigenvalue, The feature contribution vector is calculated by the model; the feature values ​​are discretized by binning and divided into... K Each interval To indicate the first k Each interval in the prediction step size t The set on; by calculating the first k The eigenvalues ​​and average contribution values ​​of each interval are used as the X-axis and Y-axis in a two-dimensional coordinate system, respectively, to obtain... K A discrete point; by linearly connecting the discrete points, a shape function is obtained, which provides global interpretability for the model.

[0014] Preferably, step 5 includes the following process: S5.1, data preprocessing; S5.2 Model parameter settings; S5.3, Accuracy Comparison Experiment; S5.4, Distillation and Ablation Experiment; S5.5, Interpretability Analysis.

[0015] Beneficial effects of this invention: 1. This invention, while ensuring high reliability and high accuracy, brings multi-dimensional interpretability to power load forecasting. Multi-dimensional interpretability can help business personnel understand model decisions, diagnose local prediction errors, and enable data-driven guidance in feature engineering. Current research still has room for expansion. In addition to the multi-period and multi-scale distillation method of this invention, future work will continuously improve the accuracy of interpretable load models through self-distillation and multi-teacher distillation, and promote the development of artificial intelligence applications in new power systems. 2. This invention trains student models using a high-precision teacher model and a "multi-scale, multi-period" distillation framework, which can effectively improve the prediction accuracy of interpretable models; 3. This invention constructs a parallel temporal attention mechanism for additive neural networks, which can improve the predictive performance of the model and further provide interpretability in the temporal dimension; 4. The KD-TSA-NAM model can provide multi-dimensional interpretability. NAM can decouple the contributions of each feature and provide global and local interpretability, while TSA can provide local interpretability. Attached Figure Description

[0016] Figure 1 This is a flowchart of an interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural networks; Figure 2 A schematic diagram of a multi-scale, multi-period knowledge distillation framework; Figure 3 The structure diagram of NAM; Figure 4 This is a schematic diagram of the overall architecture of the TSA-NAM interpretable prediction model; Figure 5 This is a schematic diagram of the correlation coefficient analysis matrix; Figure 6 A comparison chart of prediction curves from different models for different day types; Figure 7 Heatmap of the contribution of each global feature to different time steps; Figure 8 A schematic diagram illustrating feature extraction using a sparse self-attention mechanism; Figure 9 A schematic diagram of attention weights at each time step for different features; Figure 10 This diagram illustrates the contribution of different characteristics to load forecasting. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] Example 1: An interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural networks includes the following steps: Step 1: Collect historical load and meteorological data of the power grid as input features of the model. After detecting outliers in the quartiles of the data, divide it into training set, test set and validation set. Then, standardize it and impute outliers through Z-orthogonalization and linear filling. Finally, convert it into tensor form that meets the input requirements of the model through a sliding window.

[0019] Step 2: Design a "teacher-student" framework based on multi-scale and multi-period knowledge distillation. By using one-dimensional convolutional downsampling and sparse Fourier transform, the output of the teacher model is decomposed into multi-scale time domain and key periodic frequency domain signals, so as to transfer the high-precision deep "knowledge" of the teacher model to the interpretable student model.

[0020] Step 3: Construct a Temporal Self-Attention Additive Neural Network (TSA-NAM) as the student model. The additive neural network can independently learn the nonlinear relationship between each feature and the load through parallel sub-networks. Add parallel temporal self-attention modules to the sub-networks to dynamically capture the dependencies and importance of each feature in the time dimension.

[0021] Step 4: Calculate the shape function representing the relationship between contribution and feature values ​​in the sub-network using a percentile-based binning aggregation method to obtain the interpretability of the feature dimension. Then, derive the attention weights of the temporal self-attention module to obtain the interpretability of the temporal dimension.

[0022] Step 5: Simulation verification to analyze the effectiveness of the model.

[0023] In step 1, a series of methods are used to preprocess the data, converting the raw data into tensor form that meets the model input requirements; specifically, this includes: The standard calculation process for using quartiles to detect outliers is as follows: ; ; in: Q 1 is the lower quartile. Q 3 is the upper quartile. IQR The interquartile range. Not present. R norm Data within the specified range was identified as outliers. This paper uses linear interpolation to handle outliers and missing values. The calculation method for linear interpolation is as follows: ; n t Data values ​​to be filled in; s The distance between the next available data and the data to be filled; n t+s Indicates the next available data; n t-s This represents the previous available data. The data is standardized using Z-Score, calculated as follows: ; in: , σHere, n represents the mean and variance of the feature quantities, respectively, and n' is the standardized value. Standardization brings the input features and the power load data to the same order of magnitude. Finally, a sliding window is used to convert the data into tensor form. The specific process of converting the input data into a tensor form that meets the model input requirements using a sliding window in step 1 is as follows: First, define the window parameters and the number of time steps for extracting historical features and the number of future time steps to be predicted. M and N Subsequently, a sliding window is used to generate samples: starting from the beginning of the time series, one time step is slid in at a time to extract continuous samples. N A historical value is used as input, corresponding to its subsequent... M The nth value is output as the nth t The input and output of a sample can be expressed by the following formula: ; ; in: X t and Y t They are respectively t Input and output of each sample; finally, mathematical expression: given a time series The sample set can be represented as: .

[0024] In step 2, the distillation framework proposes three core modules: multi-scale distillation, multi-period distillation, and supervised learning. Multi-scale distillation transfers knowledge between the representations of the same time series at different sampling frequencies. It downsamples the original scale using one-dimensional convolution to generate predictions at a more detailed temporal resolution. Repeated downsampling operations yield multi-scale sets of teacher and student models, and the process can be expressed by the following formula: ; ; Time resolution is m The predicted value of the teacher or student model; stride is the one-dimensional convolution step size; y x Represents a multi-scale set of predictions from teacher or student models; MThe number of times the downsampling operation is repeated. Multi-period distillation aims to guide the student model to learn and replicate the deep periodicity knowledge captured by the teacher model in the time series by aligning the periodic distribution of the teacher and student models in the frequency domain. The original predicted sequence is converted from a time-domain signal to a frequency-domain signal using Discrete Fourier Transform and amplitude calculation (Amp). Subsequently, to highlight important frequencies and reduce noise interference from minor frequencies, the frequency-domain distribution is converted to a probability distribution using the Softmax function. Finally, the periodic distribution of the teacher and student models is aligned using KL divergence as the loss function. The calculation process can be represented by the following formula: ; ; in: Y x For the predicted values ​​of the teacher or student model, A x Output the calculated spectrum for the teacher (student) model. , These are the numbers in the spectrum diagram. i , j The amplitude of each; T o To predict the step size; by adjusting the temperature parameter. τ To control the probability distribution, when τ The smaller the value, the more concentrated the output probability distribution will be on the most critical periodic information for load forecasting, filtering out secondary frequencies; conversely, the distribution of probability weights will be smoother. Q x The periodic probability distribution of the teacher (student) model is calculated using Softmax. Real data is used as a benchmark to supervise the student model's predicted output, quantifying the direct difference between the student model's predicted output and the real data, ensuring the authenticity and effectiveness of the model's predictions. The specific process of the supervised learning loss function can be expressed by the following formula: ; in: Y truth As a baseline, real future load data, Y student For student model predicted values, L hard Let be the supervised learning loss function. The total loss function in the distillation process is composed of the soft-label loss and the supervised learning loss function, and can be expressed by the following formula: ; ; ; ; in: L scale This represents the loss function for multi-scale distillation. By using the mean squared error (MSE) loss across all scales and summing the sums, we achieve prediction alignment of the teacher-student model at different temporal sharpness levels. L period This is the loss function for multi-period distillation. L soft and L totel These are represented as the soft tag and distillation total loss functions, respectively. λ and δ The weights for multi-scale distillation / soft labeling are respectively, when λ The larger the value, the greater the trend of learning load changes in the student model from the teacher model's predictions. λ The smaller the value, the more the student model focuses on learning the periodic characteristics of the load. δ Similarly, weights are used to allocate the student model's learning from real data and the teacher model's predictions. Q teacher This represents the periodic probability distribution of the teacher model after Softmax calculation; Q student This represents the periodic probability distribution of the student model after Softmax calculation; Indicates time resolution as m The predicted values ​​of the teacher model under the given conditions; Indicates time resolution as m The predicted values ​​of the student model.

[0025] In step 3, a student model for load forecasting using TSA-NAM is established. The additive structure of NAM can independently learn the nonlinear relationship between each feature and the load through parallel sub-networks and provides interpretability of the feature dimension. A parallel TSA module is constructed to dynamically capture the dependencies and importance of each feature in the time dimension, providing interpretability in the time dimension and improving the accuracy of load forecasting without affecting the transparency of the "white box" model.

[0026] The NAM model is a self-interpretive deep learning model. It combines the interpretability of traditional additive structures with the fitting capabilities of deep learning. First, a neural network learns the nonlinear relationship between different features and loads. Then, the additive structure preserves the independent influence of each feature on the model output. Finally, additive aggregation yields the final predicted output. NAM can be expressed using the following formula: ; in: g Indicates the target variable to be output. f n ( xn ) as the representative of the first n Input features x The shape function, ε For the residual terms. Each of the terms in NAM f n ( x n All of these are derived from neural network fitting and visualized. f n ( x n )and x n The shape function is used to understand the relationship between different features and the output. The NAM model uses a multilayer perceptron (MLP) as its basic building block. The model constructs an independent feature network for each input feature. Each feature network learns in parallel, capturing the non-linear relationship between different features and the target variable to fit the shape function. f n ( x n The outputs of all feature networks are additively aggregated to obtain the final output of the NAM network. The structure of NAM can be... Figure 3 As shown.

[0027] A parallel TSA module is proposed to mine the contribution of elements at different time steps in the input feature vector to the output of the shape function. The TSA module calculates attention weights at different time steps, enabling the model to adaptively focus on key time points in the time series, thereby improving the model's prediction accuracy. The magnitude of the attention weights reflects the importance of information at each time point to the final prediction, thus enhancing the model's interpretability. The calculation process of the attention weights is as follows: ; ; ; ; in: X n For the first n The input vector of each feature, T i Indicates the time step of the input features. W 1, W 2 represents the weight matrix. b 1, b 2 represents the bias term of the weight matrix. h att The result of the hidden layer calculation.β logits For the original attention score, β n This is the final attention weight vector.

[0028] The overall architecture of the TSA-NAM prediction model is as follows: Figure 4 As shown, the raw data, after being split by feature, is fed into its dedicated sub-network. First, the attention weights at different time steps are dynamically calculated through a self-attention module to identify key node information in the sequence. Then, the processed weighted feature sequence is passed to the MLP for further calculation to obtain the shape function of the feature with respect to the load. Finally, the results of all feature sub-networks are additively aggregated to obtain the final predicted value. Global and local interpretability are obtained through the attention network and the additive structure of the model.

[0029] In step 4, a percentile-based binning aggregation method is used to calculate and mine the shape function representing the relationship between contribution and feature value. The process is as follows: ; ; ; ; ; in: For the first n The first feature i scalar eigenvalues The length of the load sequence L No. l Each feature value. This is the feature contribution value vector calculated by the model. The feature values ​​are discretized by binning, dividing them into... K Each interval To indicate the first k Each interval in the prediction step size t The set on. By calculating the first k The eigenvalues ​​and average contribution values ​​of each interval are used as the X-axis and Y-axis in a two-dimensional coordinate system, respectively, to obtain... K A discrete point; by linearly connecting the discrete points, a shape function is obtained, which provides global interpretability for the model.

[0030] In step 5, the effectiveness of the model and interpretable analysis is analyzed through simulation verification. Specifically, this includes the following steps: S5.1 Data Preprocessing To verify the effectiveness of the proposed method, this paper uses meteorological and load datasets from a region in Central my country from January 2021 to August 2024 for example analysis; January 2021 to July 2023 is used as the training set; and August 2023 to August 2024 is used as the test set. The experiment employs a rolling window model, using meteorological information from the two days prior to the prediction time and historical load data as input features to predict the load curve for the next day.

[0031] Table 1. Dataset information and preprocessing methods

[0032] Mean absolute percentage error (MAPE), root mean square error percentage (RMSE), and mean absolute error (MAE) are used to determine the percentage error between the predicted value and the true value.

[0033] ; ; ; in: k To predict the total number of targets; For the first i The true value of each sample; For the first i Predicted values ​​for each sample; I RMSE , I MAE , I MAPE A smaller value indicates higher prediction accuracy. The Spearman correlation coefficient measures the correlation between different features, and its calculation formula is as follows: ; in: r The Spearman correlation coefficient. d i For the first i The rank difference of each sample data k The total number of data pairs; the result can be calculated. Figure 5 As shown. By Figure 5 It can be seen that the correlation between temperature and load is quite significant, because rising temperature will affect residents' electricity consumption behavior; humidity and air pressure show a negative correlation with historical load; and the correlation between surface radiation and historical load is second only to temperature.

[0034] S5.2 Model Parameter Settings The model framework of this invention is built on Python 3.9 and a PyTorch 2.7.1 deep learning environment. The PC configuration for training the model is as follows: Intel(R) Core(TM) i5-12400F CPU, 16 GB RAM, and NVIDIA RTX 4060 GPU. The accuracy of the proposed method in load forecasting is verified by comparing it with KD-TSA-NAM (the method presented in this paper), Transformer, LSTM, KD-NAM, NAM, and GAM models. The final parameters of each model are obtained through the Adam optimizer. The parameters of each model, distillation parameters, and training parameters are set as shown in Tables 2 and 3.

[0035] Table 2. Teacher-Student Model and Distillation Parameters: Model Parameter Settings

[0036] Table 3 Training parameters for each model

[0037] S5.3 Precision Comparison Experiment The effectiveness of this method is verified by comparing the prediction accuracy of each model for different day types (weekdays, holidays, and special holidays). Table 4 shows the overall prediction performance of each model as follows: Table 4. Comparison of performance metrics for each model on the test set

[0038] As shown in Table 4, the method mentioned in this invention has higher accuracy in the numerical analysis compared with other models; the error is reduced by approximately 30.2% and 43.8% compared with traditional black-box models such as Transformer and LSTM; secondly, compared with KD-NAM, it shows that the method in this paper can effectively improve the model fitting ability after adding a temporal self-attention mechanism; finally, it has a significant advantage in prediction accuracy compared with interpretable models such as NAM and GAM, which shows that combining the knowledge distillation framework and the temporal attention mechanism can significantly improve the accuracy of interpretable models.

[0039] The accuracy and robustness of each model are further verified by comparing load forecasting performance indicators under different day types. The forecast curves and performance indicators of different models under each day type are as follows: Figure 6 As shown in Figure 1 and Table 5. Due to the confidentiality requirements of the data provider, only the normalized results are presented in the prediction curves.

[0040] Table 5 Performance Indicators of Load Forecasting for Different Daily Types

[0041] Combination Figure 6As shown in Table 5, the KD-TSA-NAM student model obtained through distillation can effectively fit the behavior of the teacher model and has good accuracy under any day type. Compared with the Transformer black-box model, its error is reduced by 37.2%, 43.1%, and 41.1% under different day types, respectively. Compared with the NAM model, which also has interpretability, the error is reduced by approximately 66.6%, 73.9%, and 92.9% under different day types. This further verifies that the prediction performance of the proposed method is robust and accurate under various day types. This indicates that the student model improves the periodic modeling ability through the multi-period distillation mechanism. Secondly, peak and trough values ​​(4:00 and 20:00) can be observed. The predicted curves of all models showed a high degree of agreement with the actual values. This is because the load composition during these two periods is relatively simple, consisting of basic load and residential load, which have relatively strong regularity. During the secondary peak period (11:00), the predicted bias of some models increased significantly. This is because the load during this period is composed of industrial, commercial, and residential loads. Temporary adjustments to industrial production plans and the high sensitivity of commercial electricity consumption to real-time weather conditions lead to increased load uncertainty during this period, thus increasing the difficulty of prediction. In addition, during special holidays such as the Spring Festival, the load prediction performance of the models decreased compared to other day types. This is because the load characteristics of special holidays differ significantly from those of regular working days, and the fitting effect is relatively poor due to the smaller number of relevant samples.

[0042] In summary, the method of this invention exhibits excellent performance in terms of prediction accuracy and robustness. Furthermore, it possesses superior interpretability, helping operators better understand the model's decision-making process and providing greater transparency for practical business applications.

[0043] S5.4 Distillation and Ablation Experiment To verify the effectiveness of each component in the distillation method of this invention, an ablation experiment was designed to remove key modules one by one to evaluate their impact on TSA-NAM prediction performance. The specific experimental control group settings are as follows, and the results are shown in Table 6.

[0044] Method 1 (Removing Multi-Scale Distillation): Only multi-cycle distillation and supervised learning are retained.

[0045] Method 2 (Removing Multi-Period Distillation): Only multi-scale distillation and supervised learning are retained.

[0046] Method 3 (Traditional Knowledge Distillation): The output of the teacher model is directly used as soft labels and real data (hard labels) for training.

[0047] Method 4 (Baseline Model): TSA-NAM is trained directly using only hard labels.

[0048] Table 6 Ablation Experiment Results

[0049] Table 6 leads to the following conclusions: First, comparing methods 3 and 4, it is evident that the knowledge distillation mechanism can effectively improve prediction accuracy, but the effect is relatively limited. Second, comparing methods 1, 2, and 3, both introducing multi-scale distillation and multi-period distillation individually demonstrate superior performance compared to traditional distillation methods, proving the effectiveness of these two modules. Finally, the proposed method reduces errors by an average of approximately 47.56% compared to method 4, and by approximately 37.26%, 37.62%, and 43.99% respectively compared to methods 1-3. This indicates that combining multi-scale and multi-period mechanisms can create complementary advantages, helping the student model learn to fit the multi-scale output capabilities of the teacher model and also learn the periodic characteristics of the teacher model's predictions, thereby most effectively improving the prediction accuracy of the student model.

[0050] S5.5, Interpretability Analysis S5.5.1 Global Interpretability This section verifies the self-interpretive capability of the proposed method in load forecasting. First, this subsection analyzes the model's global interpretability, aiming to reveal the contribution of input features to the prediction results. Global interpretability not only facilitates the construction of feature functions but also helps business personnel understand and compare the importance of each input feature in the final prediction result.

[0051] The contribution of each input feature to the prediction of each future time point under different values ​​was analyzed and compared, and its internal decision-making logic was visualized. Figure 7 This study demonstrates the impact of input characteristics such as temperature, air pressure, humidity, surface radiation, and historical load on the load forecast for the next 24 hours (96 forecast steps). Analysis shows that the influence of temperature characteristics on load forecasting is mainly concentrated in the morning (steps 1 to 32) and evening (steps 72 to 96). During the evening period, the influence of temperature is shown in darker colors on the left and right sides, while the central area is lighter, indicating that the model effectively captures the sensitivity of air conditioning load to meteorological changes. Furthermore, the influence of air pressure, humidity, and surface radiation characteristics presents a horizontal stripe pattern, indicating that these characteristics play a periodic moderating role in the model, and their contribution is significantly dependent on the forecast time. Further analysis... Figure 7 (e) It can be seen that historical load is the most critical input feature, and its influence gradually transitions from blue on the left to red on the right, reflecting the significant positive correlation between historical load and the prediction target. By comprehensively comparing the influence range of different features on the model prediction, the importance of each input feature is ranked as follows: historical load, temperature, surface radiation, humidity, and air pressure. To quantify the accuracy of the shape functions of different features across different samples, the shape function containing the error of each feature can be determined by... Figure 8 As shown.

[0052] S5.5.2, Local Interpretability Analyzing the local interpretability of the model provides an important way to evaluate and improve the model, helping business personnel to identify key information from historical sequences, correct potential errors in the model, and enhance the reliability of load forecasting.

[0053] To facilitate macroscopic analysis, the original 192 time steps were downsampled to 48 time steps through aggregation. Figure 9 This paper presents the temporal self-attention weights for different features in a typical winter sample. Temperature features primarily focus on historical data from two days prior to the prediction point (before step 24), indicating that the impact of temperature on load has a time lag. The model tends to integrate and predict temperature trends within this time period to assess its final impact on load. Humidity's attention weights are sparser compared to other features, concentrated on a few discrete historical moments. Surface horizontal radiation shows a more singular focus, while the attention for air pressure is relatively even across its historical sequence, lacking a clear focus. Historical load attention does not solely rely on short-term data near the prediction point; its attention peaks at earlier historical moments (step 15, during the midday peak two days prior to the prediction point). The model can identify and utilize potential diurnal autocorrelation in load data to capture key historical information. This demonstrates that the model adaptively allocates attention weights based on the differentiated dependency patterns of different features and their inherent attributes (such as periodicity and time lag), thereby accurately capturing key historical climate and load information and providing crucial support for the accuracy and rationality of predictions.

[0054] Using the same sample as an example, the contribution of each input feature to a single prediction can be easily quantified based on the model's shape function. Figure 10 The data shows the contribution values ​​of different features predicted for this sample in the morning (time step 24). The contribution value analysis shows that temperature and historical load are the dominant driving factors for this prediction, followed by the influence of surface radiation and air pressure. Finally, the contribution values ​​of humidity and the bias term are further corrected, and the final prediction value at this time is obtained by summing the contribution values ​​of all parts.

[0055] This embodiment proposes a novel load forecasting framework based on knowledge distillation and the TSA-NAM interpretable model. Simulation results lead to the following conclusions: 1) Training student models using a high-precision teacher model and a "multi-scale, multi-period" distillation framework can effectively improve the prediction accuracy of interpretable models; 2) Constructing a parallel temporal attention mechanism for additive neural networks can improve the model's predictive performance and further provide interpretability in the temporal dimension; 3) The KD-TSA-NAM model can provide multidimensional interpretability. NAM can decouple the contributions of each feature and provide global and local interpretability, while TSA can provide local interpretability.

[0056] In summary, this invention, while ensuring high reliability and accuracy, brings multi-dimensional interpretability to power load forecasting. This multi-dimensional interpretability helps business personnel understand model decisions, diagnose local prediction errors, and enables data-driven guidance in feature engineering. Current research still has room for expansion. In addition to the multi-period and multi-scale distillation methods of this invention, future work will continuously improve the accuracy of interpretable load models through self-distillation and multi-teacher distillation, promoting the development of artificial intelligence applications in new power systems.

[0057] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. An interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural networks, characterized in that, Includes the following steps: Step 1: Collect historical load and meteorological data of the power grid as input features of the model. After detecting outliers in the quartiles of the data, divide it into training set, test set and validation set. Then, standardize it and impute outliers through Z-orthogonalization and linear filling. Finally, convert it into tensor form that meets the input requirements of the model through a sliding window. Step 2: Design a "teacher-student" framework based on multi-scale and multi-period knowledge distillation: Decompose the output of the teacher model into multi-scale time domain and key period frequency domain signals through one-dimensional convolutional downsampling and sparse Fourier transform, so as to transfer the high-precision deep "knowledge" of the teacher model to the interpretable student model. Step 3: Construct a temporal self-attention additive neural network TSA-NAM as a student model: The additive neural network can independently learn the nonlinear relationship between each feature and the load through parallel sub-networks. By adding a parallel temporal self-attention module to the sub-network, the dependence and importance of each feature in the time dimension can be dynamically captured. Step 4: Calculate the shape function representing the relationship between contribution and feature value in the sub-network using a percentile-based binning aggregation method to obtain the interpretability of the feature dimension; The interpretability of the time dimension is obtained by deriving the attention weights of the time self-attention module; Step 5: Simulation verification to analyze the effectiveness of the model.

2. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, Step 1 includes the following process: The standard calculation process for using quartiles to detect outliers is as follows: ； ； in: Q 1 is the lower quartile. Q 3 is the upper quartile. IQR Interquartile range; not R norm Data within the specified range was identified as outliers; outliers and missing values ​​were handled using linear interpolation, calculated as follows: ； n t Data values ​​to be filled in; s The distance between the next available data and the data to be filled; n t+s Indicates the next available data; n t-s This represents the previous available data; the data is standardized using Z-Score, which is calculated as follows: ； in: , σ denoted as the mean and variance of the feature quantities, respectively, and n' is the standardized value; Standardization brings the input features to the same order of magnitude as the power load data; finally, a sliding window is used to convert the data into tensor form.

3. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, The specific process of converting the input data into a tensor form that meets the model input requirements using a sliding window in step 1 is as follows: First, define the window parameters and the number of time steps for extracting historical features and the number of future time steps to be predicted. M and N ; Subsequently, a sliding window is used to generate samples: starting from the beginning of the time series, one time step is slid in at a time to extract continuous samples. N A historical value is used as input, corresponding to its subsequent... M The nth value is output as the nth t The input and output of a sample can be expressed by the following formula: ； ； in: X t and Y t They are respectively t Input and output of each sample; finally, mathematical expression: given a time series The sample set can be represented as: 。 4. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, Step 2 includes the following process: Step 2.1, Knowledge Distillation: The "Teacher-Student" framework adopts three core modules: multi-scale distillation, multi-period distillation, and supervised learning. Multi-scale distillation is used to transfer knowledge from the representation of the same time series at different sampling frequencies. One-dimensional convolution operations are used to downsample the original scale to generate predictions at a more detailed temporal resolution. Repeated downsampling operations can obtain multi-scale sets of teacher and student models. Step 2.2, Multi-Period Distillation, guides the student model to learn and replicate the deep periodicity knowledge captured by the teacher model in the time series by aligning the periodic distribution of the teacher and student models in the frequency domain; the original prediction sequence is converted from a time domain signal to a frequency domain signal using Discrete Fourier Transform and amplitude calculation (Amp); subsequently, to highlight important frequencies and reduce noise interference from secondary frequencies, the frequency domain distribution is converted into a probability distribution using the Softmax function; finally, the periodic distribution of the teacher and student models is aligned using KL divergence as the loss function. Step 2.3: Use real data as a benchmark to supervise the student model's predicted output, quantifying the direct difference between the student model's predicted output and the real data, ensuring the authenticity and effectiveness of the model's predictions; the specific process of the supervised learning loss function is expressed by the following formula: ； in: Y truth As a baseline, real future load data, Y student For student model predicted values, L hard The loss function for supervised learning; Step 2.4: The total loss function in the distillation process is composed of the soft label loss and the supervised learning loss function, and can be expressed by the following formula: ； ； ； ； in: L scale The loss function represents the multi-scale distillation; by using the mean squared error (MSE) loss across all scales and summing the results, the teacher-student model can achieve prediction alignment at different temporal resolutions. L period This is the loss function for multi-cycle distillation; L soft and L totel These are represented as the soft tag and distillation total loss functions, respectively. λ and δ The weights for multi-scale distillation / soft labeling are respectively, when λ The larger the value, the greater the trend of learning load changes in the student model from the teacher model's predictions. λ The smaller the value, the more the student model focuses on learning the periodic characteristics of the load. δ Similarly, weights are used to allocate the student model's learning from real data and the teacher model's predictions. Q teacher This represents the periodic probability distribution of the teacher model after Softmax calculation; Q student This represents the periodic probability distribution of the student model after Softmax calculation; Indicates time resolution as m The predicted values ​​of the teacher model under the given conditions; Indicates time resolution as m The predicted values ​​of the student model.

5. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, The process in step 2.1 can be expressed by the following formula: ； ； in, Time resolution is m The predicted value of the teacher or student model; stride is the one-dimensional convolution step size; y x Represents a multi-scale set of predictions from teacher or student models; M This represents the number of times the downsampling operation is repeated.

6. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, The calculation process in step 2.2 can be expressed by the following formula: ； ； in: Y x For the predicted values ​​of the teacher or student model, A x Output the calculated spectrum for the teacher or student model. , These are the numbers in the spectrum diagram. i , j The amplitude of each; T o To predict the step size; by adjusting the temperature parameter. τ To control the probability distribution, when τ The smaller the value, the more concentrated the output probability distribution will be on the most critical periodic information for load forecasting, filtering out secondary frequencies; conversely, the distribution of probability weights will be smoother. Q x This is the periodic probability distribution of the teacher or student model calculated using Softmax.

7. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, Step 3 The process includes the following: A parallel TSA module is constructed to dynamically capture the dependencies and importance of each feature in the time dimension, providing interpretability in the time dimension and improving the accuracy of load forecasting without affecting the transparency of the "white box" model. The nonlinear relationship between different features and loads is learned through neural networks. Then, the additive structure preserves the independent influence of each feature on the model output. Finally, additive aggregation is performed to obtain the final predicted output. NAM is expressed by the following formula: ； in: g Indicates the target variable to be output. f n ( x n ) as the representative of the first n Input features x The shape function, ε For residual terms; Each of NAM f n ( x n All of these are derived from neural network fitting and visualized. f n ( x n )and x n The shape function is used to understand the relationship between different features and the output; the NAM model uses a multilayer perceptron as its basic building block; the model constructs an independent feature network for each input feature, and each feature network learns in parallel to capture the nonlinear relationship between different features and the target variable, in order to fit the shape function. f n ( x n The outputs of all feature networks are additively aggregated to obtain the final output of the NAM network. The TSA module calculates attention weights at different time steps, enabling the model to adaptively focus on key time points in the time series, thereby improving the model's prediction accuracy. The magnitude of the attention weights can reflect the importance of information at each time point to the final prediction, thus enhancing the model's interpretability. After the raw data is split according to features, it is fed into its dedicated sub-network. First, the attention weights at different time steps are dynamically calculated through the self-attention module to identify key node information in the sequence. Then, the processed weighted feature sequence is passed to the MLP for further calculation to obtain the shape function of the feature with respect to the load. Finally, the results of all feature sub-networks are additively aggregated to obtain the final predicted value. Global and local interpretability are obtained through the attention network and the additive structure of the model.

8. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 7, characterized in that, The calculation process for attention weights is as follows: ； ； ； ； in: X n For the first n The input vector of each feature, T i Indicates the time step of the input features. W 1, W 2 represents the weight matrix; b 1, b 2 represents the bias term of the weight matrix. h att The result is the calculation result of the hidden layer; β logits Original attention score; β n This is the final attention weight vector.

9. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, Step 4 includes the following process: A percentile-based binning aggregation method is used to calculate and mine the shape function representing the relationship between contribution and eigenvalues. The process is as follows: ； ； ； ； ； in: For the first n The first feature i scalar eigenvalues The length of the load sequence L No. l 1 eigenvalue, The feature contribution vector is calculated by the model; the feature values ​​are discretized by binning and divided into... K Each interval To indicate the first k Each interval in the prediction step size t The set on; by calculating the first k The eigenvalues ​​and mean contribution values ​​of each interval are used as the X-axis and Y-axis in a two-dimensional coordinate system, respectively, to obtain... K A discrete point; by linearly connecting the discrete points, a shape function is obtained, which provides global interpretability for the model.

10. The interpretable load prediction method based on knowledge distillation and temporal self-attention additive neural network according to claim 1, characterized in that, Step 5 includes the following process: S5.1, data preprocessing; S5.2 Model parameter settings; S5.3, Accuracy Comparison Experiment; S5.4, Distillation and Ablation Experiment; S5.5, Interpretability Analysis.

Images