A method for electric load forecasting based on small sample data set

Abstract

A method for predicting electric load of a small sample data set, in which an improved Transformer model is constructed, each sequence data in the constructed data set is introduced into an embedding layer as a single column, the embedding layer maps each sequence data to obtain high-dimensional data, and the high-dimensional data is respectively calculated by a spatial attention mechanism layer and a channel attention mechanism layer. The output of the attention mechanism calculation result is divided into two steps, one of which enters a normalization layer alone and is retained as a cell state, and the other step enters a normalization layer after subtracting the output result of the embedding layer and performs forward propagation, the result after forward propagation is subtracted from the result of the normalization layer input, and enters the normalization layer again, the output of the normalization layer is residually connected with the retained cell state, and the electric load prediction result is output through the output layer.

Description

Technical Field

[0001] The present invention relates to the technical field of power collection load detection, and in particular to a power collection load prediction method with a small sample data set. Background Art

[0002] As an indispensable energy source in life and industrial production, the importance of stable supply of electricity for maintaining social activities and economic development is self-evident. However, since electricity load is affected by many factors such as weather changes, economic activities, and population growth, it presents highly dynamic and nonlinear characteristics, which brings great challenges to power load forecasting. Traditional load forecasting methods often rely on statistical models or machine learning techniques, which perform well when processing large sample data. Nowadays, many power load data are collected with days as the minimum sampling unit. The data set formed by this sampling method generally has two problems. First, sampling power data by day will result in too little data. Second, data sampled by day is prone to large sequence fluctuations due to emergencies. Therefore, in the case of such a small sample data set, traditional methods are difficult to capture patterns and trends in the data, resulting in poor prediction accuracy.

[0003] In recent years, the rise of deep learning technology has opened up new paths for the study of power systems. Among them, the Transformer model has achieved great success in the field of natural language processing due to its excellent ability to process sequence data. The model relies on the self-attention mechanism to capture the relationship between elements in the sequence, which makes it a powerful tool for processing electric load data with time series characteristics. However, due to the complex network structure of the Transformer model, it is easy to cause overfitting problems when using the Transformer model for training small sample data sets.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The present invention provides a method for predicting electric load of a small sample data set. First, the weather conditions of the day and whether the day is a holiday are introduced as dynamic variables in the data set to enrich the data set. Secondly, the encoding layer part of the Transformer model is used to change the original residual connection structure to a model structure that subtracts dimension by dimension, and introduces a spatial attention mechanism and a channel attention mechanism to replace the multi-head attention mechanism in the Transformer model, and introduces a cell state to save the calculation result of the attention mechanism. Finally, the output of the last layer of the encoding layer is residually connected to the cell state, so that the trend information in the sequence data can be paid attention to, and the residual information in the sequence data can also be captured, and finally the occurrence of overfitting can be prevented.

[0006] A method for predicting electric load using a small sample data set includes:

[0007] The electricity consumption data with a minimum collection unit of day and a data volume of less than 1,000 points is collected to form sequence data that changes with the date.

[0008] An improved Transformer model is constructed, which has an embedding layer, a spatial attention mechanism layer, a channel attention mechanism layer, a forward propagation layer, a gating mechanism fusion layer, a residual reduction, a normalization layer, a cell state, a residual addition, and an output layer from bottom to top. The sequence data enters the embedding layer, and the embedding layer maps the sequence data to obtain higher-dimensional data. The mapped high-dimensional data enters the spatial attention mechanism layer and the channel attention mechanism layer for spatial attention mechanism calculation and channel attention mechanism calculation respectively. The two results enter a forward propagation layer for forward propagation respectively, and the two attention mechanism results are fused. The two forward propagation results enter the fusion layer to obtain the final attention mechanism calculation result.

[0009] The improved Transformer model is trained and the power consumption prediction results are obtained by inputting the improved Transformer model.

[0010] In the method for predicting electric load using a small sample data set, the embedding layer only includes one linear layer to map the feature dimension of the input data to high-dimensional data.

[0011] In the described method for predicting electric load of a small sample data set, the spatial attention mechanism layer is composed of a convolution layer and a Sigmoid activation function layer, and the importance relationship of the time relationship of each sequence is captured by calculating the maximum value in the sequence length dimension and the average value of each two items. The channel attention mechanism layer is composed of a maximum pooling layer, an average pooling layer, a forward propagation layer and a Sigmoid activation function layer, wherein the forward propagation layer also includes two linear layers and a Relu activation function layer. The sequence length dimension data is compressed by the maximum pooling layer and the average pooling layer, and after compression, the importance relationship between different features is respectively assigned through the forward propagation layer.

[0012] The method for predicting electric load for a small sample data set is characterized in that the forward propagation layer only contains one linear layer. After the two attention mechanisms are calculated, they are respectively passed through a forward propagation layer to correct the dimension and mine the potential relationship between the data. The forward propagation after the channel attention mechanism layer is mainly for correcting the dimension, and the forward propagation after the spatial attention mechanism layer is for mining the potential relationship between the data.

[0013] In the method for predicting electric load of a small sample data set, the fusion layer determines the weights given to space and channel based on the fully connected layer of the Sigmoid activation function. Specifically, the output of the fusion mechanism can be expressed by the following formula:

[0014]

[0015] in is the result of the spatial attention mechanism after passing through the forward propagation layer, is the result of the channel attention mechanism passing through the forward propagation layer, is the weight assigned to the fully connected layer of the Sigmoid activation function. When it is close to 1, the spatial attention mechanism accounts for a larger proportion. When it is close to 0, the channel attention mechanism accounts for a larger proportion.

[0016] In the method for predicting electric load using a small sample data set, after the normalization layer calculates the mean and variance of each feature, the data is mapped to (0, 1) to eliminate the dimensional differences between different features.

[0017] In the method for predicting electric load using a small sample data set, the forward propagation layer includes a linear layer, which performs same-dimensional mapping on the data and mines the potential relationship after the data is reduced.

[0018] In the electric load prediction method for a small sample data set, the output layer includes two linear layers. The first linear layer performs feature fusion and maps the final feature dimension to 1 dimension. The second linear layer performs sequence dimension fusion and maps the final sequence length dimension to 1 dimension, thereby outputting the final prediction result.

[0019] In the method for predicting electric load using a small sample data set, the data set is divided into two parts: the training set accounts for 70% of the total data volume, the test set accounts for 20% of the total data volume, and the validation set accounts for 10% of the total data volume. The improved Transformer model is trained with the training set, and the improved Transformer model after training is tested with the test set and verified with the validation set.

[0020] In the method for predicting electric load with a small sample data set, the mean square error is selected Backpropagation of the model as loss function, learning rate Set to 0.0001 and select mean absolute error As a reference indicator for validating the model on the test set:

[0021] ,

[0022] ,

[0023] in, represents the true value of the data, represents the predicted value of the data,

[0024] First, define 10 rounds of model training and observe the loss on the model validation set to evaluate the generalization ability of the model.

[0025] In the method for predicting electric load with a small sample data set, the improved Transformer model is trained, and the optuna library is used for hyperparameter tuning. The hyperparameters include the input sequence length, the model embedding dimension, the number of encoders, and the number of feature maps in the forward propagation process. The entire model is trained using the optimal parameters trained by the optuna library. The batch size is also set to 32, and the number of training rounds is set to 30. The change curves of the loss functions of the model training set and the validation set are observed at this time. The learning rate scheduler is used to adjust the learning rate, and the learning rate is changed within a fixed period. The length of the first period T_0 is set to 8, that is, the learning rate is reduced every 8 training rounds, and the growth mode T_mult of the subsequent periods is set to 2. By analogy, the minimum value eta_min that the learning rate can reach is set to When the learning rate drops to this value, it will no longer drop, and the eta_min value will continue to be used for training. No local minimum occurs during the training process, and the model training effect should be optimal.

[0026] Compared with the prior art, the present invention has the following advantages: the present invention introduces four dynamic influencing factors, namely, the maximum temperature, the minimum temperature, the wind speed and whether it is a holiday, as variables into the model for the power load data collected on a daily basis, thereby improving the prediction effect of the model. The present invention proposes a spatial attention mechanism and a channel attention mechanism to replace the multi-head attention mechanism in the Transformer model structure, wherein the spatial attention mechanism mainly captures the importance of different time steps, and by merging the mean of each two items and the maximum value of the sequence, the peak value and trend of the sequence can be paid attention to at the same time. The channel attention mainly calculates the importance between different features, and introduces a gating mechanism to fuse the results of the two attention mechanisms. The model can automatically adjust the contribution ratio of the two attention mechanisms according to the characteristics of the input data, thereby achieving more effective feature selection and weight allocation. The two attention mechanisms are combined and used, which can more efficiently capture the dependency between time and features in the input sequence, respectively, and solve the problem that the multi-head attention mechanism of the Transformer model is difficult to capture local dependencies. The present invention improves the residual connection block in the Transformer model structure into a dimension-by-dimension decreasing module, which not only pays attention to the trend information of the sequence, but also captures the residual information in the sequence propagation process, and effectively prevents the overfitting problem. The present invention introduces the cell state structure to retain the calculation results of the attention mechanism and store the importance of each variable. The invention finally performs residual analysis on the cell state and the results of the normalization layer, which can not only explore the mutual influence between different variables to improve the prediction accuracy of the model, but also effectively alleviate the gradient explosion and gradient vanishing phenomena that are prone to occur in deep networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The drawings in the specification are only for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.

[0028] In the attached picture:

[0029] Figure 1 Box plot before outlier treatment;

[0030] Figure 2 Box plot after outlier processing;

[0031] Figure 3 To improve the overall structure diagram of the Transformer model;

[0032] Figure 4 This is the schematic diagram of the spatial attention mechanism layer;

[0033] Figure 5 This is the schematic diagram of the channel attention mechanism layer;

[0034] Figure 6 It is the schematic diagram of the standardized layer;

[0035] Figure 7 This is the loss of the model after 10 rounds of training;

[0036] Figure 8 This is the loss function diagram of the model using the optimal parameters;

[0037] Fig. 9 This is the model loss function diagram after adjusting the learning rate;

[0038] Fig.10 This is a comparison chart between the model prediction value and the true value after adjusting the learning rate;

[0039] Fig.11 It is the loss function diagram of Transformer model;

[0040] Fig.12 This is a comparison chart of the Transformer model's predicted value and the actual value.

[0041] The present invention is further explained below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0042] The specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0043] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the present invention. The scope of protection of the present invention shall be determined by the attached claims.

[0044] To facilitate understanding of the embodiments of the present invention, further explanation will be given below by taking specific embodiments as examples in conjunction with the accompanying drawings, and each of the accompanying drawings does not constitute a limitation on the embodiments of the present invention.

[0045] like Figures 1 to 12 As shown in FIG. 1 , the electric load prediction method for a small sample data set includes the following steps:

[0046] In the electricity consumption data with a minimum collection unit of day and a data volume of less than 1000 points, that is, the data is a column and is the electricity consumption data of a certain building on the same day (kWh), multiple features are introduced, including the highest temperature (℃), the lowest temperature (℃), the wind speed (m / s) and whether the day is a holiday (1 for yes, 0 for no). Before building the model, the data set is first tested for missing values ​​and outliers. If the missing data does not exceed 5, the deletion method is used to delete the missing data. If it exceeds 5, the average value of the previous and next data is used to fill the data. The abnormal data detection defines that data that is higher than 4 times the standard deviation of the overall data is abnormal data. If the number of abnormal data is not more than 5, the deletion method is used to directly delete the abnormal data. If the number of abnormal data is more than 5, the abnormal data is first marked as 0, and then the average value of the previous and next data is used to fill.

[0047] Construct an improved Transformer model, which has an embedding layer, a spatial attention mechanism layer, a channel attention mechanism layer, a forward propagation layer, a gating mechanism fusion layer, a residual reduction, a normalization layer, a cell state, a residual addition, and an output layer from bottom to top.

[0048] First, the five features in the data set, namely power consumption, maximum temperature, minimum temperature, wind speed, and whether it is a holiday, which change with the date, enter the embedding layer. The embedding layer maps the sequence data to obtain higher-dimensional data.

[0049] The mapped high-dimensional data enters the spatial attention mechanism layer and the channel attention mechanism layer respectively to calculate the spatial attention mechanism and the channel attention mechanism. After calculating the results of the spatial attention mechanism and the channel attention mechanism, the two results enter a forward propagation layer respectively.

[0050] After the two attention mechanism results are forward propagated separately, the two attention mechanism results are fused, and the two forward propagation results enter the fusion layer to obtain the final attention mechanism calculation result.

[0051] The output of the attention mechanism calculation result is divided into two steps. One step is to enter the attention mechanism calculation result into a normalization layer separately and retain it as a cell state. The other step is to subtract the attention mechanism calculation result from the embedding layer output result and then enter the normalization layer.

[0052] The result obtained after decreasing and entering the normalization layer enters a forward propagation layer.

[0053] The result after forward propagation is decremented with the output of the normalization layer and enters a normalization layer again.

[0054] The output result of the second normalization layer is residually connected with the above-retained cell state, and the power consumption prediction result is output through the output layer.

[0055] After the model is built, the model is trained to the best for subsequent use. First, the data set is divided into training set, test set and validation set. The four parameters in the training process, namely the proportion of data set division, number of training rounds, learning rate size, and loss function, are defined respectively. At the same time, the five parameters of the model, namely the input sequence length, model embedding dimension, number of encoders, number of forward propagation feature maps, and convolution kernel size, are defined. After the parameters are defined, the model is trained. A small number of rounds are used for the first training, and the changes in the loss functions of the training set and validation set are observed. When the loss functions of the training set and validation set no longer decrease, the model converges.

[0056] After a few rounds of training, we tuned the five model parameters, including input sequence length, model embedding dimension, number of encoders, number of forward propagation feature maps, and convolution kernel size, to find the optimal parameters. We also used a dynamic learning rate adjustment strategy to help the model converge better during training. Then, we selected a higher round of training and observed the changes in the loss functions of the training set and the validation set.

[0057] In another embodiment, a method for predicting electric load with a small sample data set includes the following steps:

[0058] The power consumption data with a minimum collection unit of day and a data volume of less than 1000 points is collected to form a sequence data that changes with the date. The power consumption data includes the highest temperature, the lowest temperature, the wind speed, and whether the day is a holiday.

[0059] An improved Transformer model is constructed, which has an embedding layer, a spatial attention mechanism layer, a channel attention mechanism layer, a forward propagation layer, a gating mechanism fusion layer, a residual reduction, a normalization layer, a cell state, a residual addition, and an output layer from bottom to top. The sequence data enters the embedding layer, and the embedding layer maps the sequence data to obtain higher-dimensional data. The mapped high-dimensional data enters the spatial attention mechanism layer and the channel attention mechanism layer for spatial attention mechanism calculation and channel attention mechanism calculation respectively. The two results enter a forward propagation layer for forward propagation respectively, and the two attention mechanism results are fused. The two forward propagation results enter the fusion layer to obtain the final attention mechanism calculation result.

[0060] The output of the attention mechanism calculation result is divided into two steps. One step is to enter the attention mechanism calculation result separately into a normalization layer and retain it as a cell state. The other step is to subtract the attention mechanism calculation result from the embedding layer output result and enter the normalization layer. The result obtained after decrementing and entering the normalization layer enters a forward propagation layer. The result after forward propagation is decremented with the output result of the normalization layer and enters a normalization layer again. The output result of the second normalization layer is residually connected with the retained cell state, and the power consumption prediction result is output through the output layer.

[0061] Improved Transformer model for training, first divide the data set, divide the data set into training set, test set and validation set, define the proportion of data set division, number of training rounds, learning rate size and loss function parameters in the training process respectively, at the same time, define the input sequence length, model embedding dimension, number of encoders, number of forward propagation feature maps and convolution kernel size parameters of the model, start training the model after the parameters are defined, select fewer rounds for the first training, and observe the loss function changes of the training set and validation set. When the loss functions of the training set and validation set no longer decrease, the model converges. After fewer rounds of training, the parameters of the five models, input sequence length, model embedding dimension, number of encoders, number of forward propagation feature maps and convolution kernel size, are tuned to find the optimal parameters, and a dynamic learning rate adjustment strategy is used to help the model converge better during the training process. Then, a higher round is selected for training again, and the loss function changes of the training set and validation set are observed to complete the training of the improved Transformer model. The test set is input into the improved Transformer model to obtain the power consumption prediction result.

[0062] In a preferred embodiment of the method for predicting electric load of a small sample data set, the embedding layer only includes one linear layer to map the feature dimension of the input data to high-dimensional data.

[0063] In a preferred embodiment of the electric load prediction method for a small sample data set, the spatial attention mechanism layer is composed of a convolution layer and a Sigmoid activation function layer, and the importance relationship of each sequence in time relationship is captured by calculating the maximum value in the sequence length dimension and the average of each two items. The channel attention mechanism layer is composed of a maximum pooling layer, an average pooling layer, a forward propagation layer and a Sigmoid activation function layer, wherein the forward propagation layer also includes two linear layers and a Relu activation function layer. The sequence length dimension data is compressed through the maximum pooling layer and the average pooling layer, and after compression, the importance relationship between different features is respectively assigned through the forward propagation layer.

[0064] In a preferred implementation of the method for predicting electric load for a small sample data set, the forward propagation layer only includes one linear layer. After the two attention mechanisms are calculated, they are respectively passed through a forward propagation layer to correct the dimension and mine the potential relationship between the data. The forward propagation after the channel attention mechanism layer is mainly for correcting the dimension, and the forward propagation after the spatial attention mechanism layer is for mining the potential relationship between the data.

[0065] In a preferred implementation of the method for predicting electric load using a small sample data set, after the normalization layer calculates the mean and variance of each feature, the data is mapped to (0, 1) to eliminate the dimensional differences between different features.

[0066] In a preferred implementation of the method for predicting electric load with a small sample data set, the output layer includes two linear layers, the first linear layer performs feature fusion and maps the final feature dimension to 1 dimension, and the second linear layer performs sequence dimension fusion and maps the final sequence length dimension to 1 dimension, thereby outputting the final prediction result.

[0067] In a preferred implementation of the method for predicting electric load with a small sample data set, the data set is divided, the training set accounts for 70% of the total data volume, the test set accounts for 20% of the total data volume, and the validation set accounts for 10% of the total data volume. The improved Transformer model is trained with the training set, and the improved Transformer model after training is tested with the test set, and verified with the validation set.

[0068] In a preferred implementation of the method for predicting electric load with a small sample data set, the mean square error is selected Backpropagation of the model as loss function, learning rate Set to 0.0001 and select mean absolute error As a reference indicator for validating the model on the test set:

[0069] ,

[0070] ,

[0071] in, represents the true value of the data, represents the predicted value of the data,

[0072] First, define 10 rounds of model training and observe the loss on the model validation set to evaluate the generalization ability of the model.

[0073] In a preferred implementation of the method for predicting electric load with a small sample data set, the optuna library is used for hyperparameter tuning, and the hyperparameters include the input sequence length, the model embedding dimension, the number of encoders, and the number of feature maps in the forward propagation process. The entire model is trained using the optimal parameters trained by the optuna library, and the batch size is also set to 32, and the number of training rounds is set to 30 rounds. The change curves of the loss functions of the model training set and the validation set are observed at this time, and the cosine annealing learning rate scheduler is used to adjust the learning rate. The learning rate is changed within a fixed period, and the length of the first period T_0 is set to 8, that is, the learning rate is reduced every 8 training rounds, and the growth mode T_mult of the subsequent periods is set to 2, and so on. The minimum value eta_min that the learning rate can reach is set to That is, when the learning rate drops to this value, it will no longer drop, and continue to use the value of eta_min for training. No local minimum occurs during the training process, and the model training effect should be optimal.

[0074] In one embodiment,

[0075] 1) Introduce dynamic changing features such as temperature and wind speed as predictable sequence data, and map each column separately through the embedding layer. .

[0076] 2) After mapping the features, the spatial attention mechanism and channel attention mechanism are calculated respectively. Assume an input tensor ,in Indicates the batch size, represents the sequence length, Represents the embedding dimension.

[0077] First, the spatial attention mechanism is calculated. The main purpose of the spatial attention mechanism design is to focus on important time points in the input sequence such as peaks and overall trends.

[0078] 1. For each sample, calculate the maximum value of the entire sequence in the time dimension and the average value of every two time steps. This step helps the model identify the key time points in the sequence and the trend of the entire time series:

[0079] ,

[0080] in For the entire sequence, is the data value of the first time step, is the data value of the second time step, and the calculated The shape is , The shape is .

[0081] 2. Concatenate the maximum value and the average value of every two time steps along the second dimension to form a new tensor , to capture both peak and trend information.

[0082] 3. Convert the dimension of the sequence. . Through a 1D convolutional layer To calculate the attention weights between different time points in each dimension, generate the attention weight matrix, and use the Sigmoid function Normalize the attention weights to ensure that the weights are mapped to (0,1). ,in Indicates the calculation of Sigmoid function. The obtained attention weight matrix For each element in A , the Sigmoid function calculation formula is:

[0083]

[0084] 4. To apply the attention calculation results to each dimension of the mapped data, the attention weight matrix Expand to the same shape as input X :

[0085]

[0086] in Represents the attention weight matrix obtained Expand and get the expanded result .

[0087] 5. Finally, the expanded attention weights are multiplied element-wise with the input tensor to achieve weighting at different time points.

[0088]

[0089] in Represents element-by-element multiplication. The output of the spatial attention mechanism obtained at this time Through the above steps, the spatial attention mechanism can help the model focus on the key time points in the input sequence and assign higher weights to these time points, thereby improving the model's ability to capture important features.

[0090] Secondly, calculate the channel attention mechanism. The main purpose of the channel attention mechanism is to focus on the important features in the input features.

[0091] 1. First, rearrange the dimensions of the input tensor so that the feature dimension becomes the second dimension. This helps with the subsequent pooling operation.

[0092]

[0093] in Represents the rearrangement of the dimensions of the tensor. .

[0094] 2. Calculate the global average and maximum value of each feature channel. This step helps the model capture the importance and distribution of different features.

[0095]

[0096] in Represents a rearranged tensor Take the average value along the feature dimension, Represents a rearranged tensor Find the maximum value along the feature dimension, and The shapes of .

[0097] 3. Add the results of average pooling and maximum pooling along the third dimension, the time dimension, to obtain a new tensor :

[0098]

[0099] 4. Use two fully connected layers to further compress the feature dimension and obtain the attention weight vector

[0100]

[0101] in For forward propagation, it includes the following steps:

[0102]

[0103]

[0104]

[0105] in is the weight matrix of the first linear mapping, is the weight matrix of the second linear mapping, is the bias matrix of the first linear mapping, is the bias matrix of the second linear mapping, is the activation function.

[0106]

[0107] 5. Use Sigmoid function Normalize the attention weights:

[0108]

[0109] in Represents the calculation of the Sigmoid activation function, and the calculation formula is:

[0110]

[0111] Where a represents each value in the attention weight matrix.

[0112] 6. Finally, the attention weight With input Perform matrix multiplication to obtain the weighted output Y:

[0113]

[0114] in Represents the transpose of the attention weight vector. At this time, the output of the channel attention mechanism is obtained Through the above steps, the channel attention mechanism can help the model focus on important features in the input features and assign higher weights to these features, thereby improving the model's ability to learn key features.

[0115] Finally, the results of the two attention mechanisms are fused through a gated fusion mechanism. Before passing through the gating mechanism, the calculation results of the two attention mechanisms must first pass through two fully connected layers to change their shape.

[0116]

[0117]

[0118] in and They are the calculation results of the spatial attention mechanism and the channel attention mechanism, Represents a linear mapping of the results of the spatial attention mechanism, Represents a linear mapping of the channel attention mechanism results. After mapping, the shapes of the two attention mechanisms are , so the gated fusion mechanism can be used to fuse the two calculation results. The gated fusion mechanism is based on the fully connected layer of the Sigmoid activation function to determine the weights given to space and channels. Specifically, the output of the fusion mechanism can be expressed by the following formula:

[0119]

[0120] in is the weight value generated by the Sigmoid activation function, and the attention mechanism outputs the result .when When the value is close to 1, the model relies more on the attention weight calculation of the temporal relationship; conversely, when When the value is close to 0, the model relies more on the attention weight calculation of the relationship between features. This dynamic weighting mechanism enables the model to flexibly adjust the importance of the time dimension and feature dimension according to the characteristics of the data.

[0121] 3) The residual connection module of the original Transformer model is changed to a residual reduction module to alleviate the overfitting problem in the original Transformer model, and the calculation result of the attention mechanism is retained as a cell state, and finally a residual connection is made with the result of the forward propagation. This can not only explore the mutual influence relationship between different variables and improve the prediction accuracy, but also alleviate the gradient explosion and gradient disappearance problems in long time series problems. Output of the attention mechanism layer Perform residual reduction to obtain the residual trend ,and :

[0122]

[0123] 4) In order to ensure the independence of variables and avoid the influence of sampling misalignment or time lag on the intercorrelation of modeling variables, each variable is standardized separately.

[0124]

[0125] in represents the input sequence for each feature, Represents the average value of each feature sequence. Represents the standard deviation of each special diagnosis sequence.

[0126] All sequences are normalized to Gaussian distribution as (variable) tags to reduce the differences caused by inconsistent measurements. The output after the attention mechanism layer is , at this time, the result after layer normalization is retained as the cell state :

[0127]

[0128] in Represents each feature sequence in the calculation result of the attention mechanism, Represents the average value of each feature sequence in the calculation result of the attention mechanism. Represents the standard deviation of each feature sequence in the calculation results of the attention mechanism.

[0129] The remaining trend will decrease Also perform layer normalization, namely:

[0130]

[0131] in represents the remaining sequence after decrementing, It represents the average value of each characteristic sequence of the remaining sequence after decreasing. Represents the standard deviation of each characteristic sequence of the remaining sequence after decreasing. .

[0132] 5) After a forward propagation layer The remaining trend Perform feature mapping again, and the result after mapping is recorded as , and the mapped With the standardized layer Perform residual reduction again to obtain :

[0133]

[0134]

[0135] in Represents the linear mapping of the forward propagation process in the model.

[0136] 6) After the last layer normalization, the output result is recorded as , at this time will The residual is added to the cell state C, and then the load prediction result is output through the final linear layer. The final output result is recorded as .

[0137] .

[0138] in Represents the linear mapping of the output layer.

[0139] In one embodiment, the method includes,

[0140] Step 1: Improve the model structure such as Figure 3 As shown in the figure, the entire model contains 10 key modules: embedding layer, spatial attention mechanism layer, channel attention mechanism layer, gating mechanism fusion layer, residual reduction, normalization layer, cellular layer, residual addition, forward propagation and output layer. The entire model is as follows from bottom to top:

[0141] Embedding layer: The embedding layer contains only one linear layer, and its purpose is to map the feature dimension of the input data to a high dimension to capture the potential relationship between features.

[0142] Spatial Attention: The spatial attention layer consists of a convolutional layer and a sigmoid activation function layer. It captures the importance of the temporal relationship of each sequence by calculating the maximum value of the sequence length dimension and the average of each two items.

[0143] Channel Attention Mechanism Layer: The channel attention mechanism layer consists of a maximum pooling layer, an average pooling layer, a forward propagation layer, and a Sigmoid activation function layer. The forward propagation layer also includes two linear layers and a Relu activation function layer. The maximum pooling layer and the average pooling layer are used to compress the sequence length dimension data. After compression, the importance relationship between different features is given through the forward propagation layer.

[0144] Feed-forward layer: The feed-forward layer contains only one linear layer. After the two attention mechanisms are calculated, they pass through a feed-forward layer to correct the dimension and mine the potential relationship between the data. The feed-forward after the channel attention mechanism layer is mainly to correct the dimension, and the feed-forward after the spatial attention mechanism layer is mainly to mine the potential relationship between the data. The feed-forward layer in the entire process is to mine the potential information of the standardized data and perform a transformation of the same dimension.

[0145] Gating mechanism fusion layer (Sigmoid): Based on the calculation results of the two attention mechanisms, weights are generated and assigned to the two attention mechanisms respectively, and the influence of different attention mechanisms on the output results is dynamically adjusted according to the weights.

[0146] Residual reduction: To solve the overfitting problem of the Transformer model on small sample data sets, the residual reduction module is used to replace the residual addition module of the traditional Transformer model. Specifically, after the attention mechanism is calculated, the embedded sequence is subtracted from the result of the attention mechanism calculation to reduce data redundancy. Similarly, after forward propagation, it is subtracted from the result of the normalization layer to continue to eliminate redundancy in the data.

[0147] Normalization layer (LayerNorm): After calculating the mean and variance of each feature, the data is mapped to (0,1) directly to eliminate the dimensional differences between different features.

[0148] Cell state (C): The cell state is the final result of the calculation result of the retained attention mechanism after the normalization layer. The main purpose of introducing the cell state is to retain the influence of the output results between and within sequences.

[0149] Residual addition: Residual addition is to add the retained cell state to the result of the last normalization in order to reintroduce the influence of the attention mechanism results on the final output results and prevent the occurrence of gradient explosion or gradient disappearance during training.

[0150] Output layer (Projection): The output layer mainly consists of two linear layers. The first linear layer performs feature fusion and maps the final feature dimension to 1 dimension. The second linear layer performs sequence dimension fusion and maps the final sequence length dimension to 1 dimension, thereby outputting the final prediction result.

[0151] Step 2: After understanding the structure of the entire model, let’s introduce the overall data flow of the model. Specifically, for an input sequence ,in

[0152]

[0153] The input sequence , that is, the first column is the electricity consumption data of a building for 16 days, the second column is the highest temperature of each day in the 16 days, the third column is the lowest temperature of each day in the 16 days, the fourth column is the wind speed of each day in the 16 days, and the last column is whether the day of the 16th day is a holiday, if yes, it is 1, otherwise it is 0. Using this sequence data, you can use the 16-day historical data to predict the electricity consumption in the next 1-3 days. By predicting the data in the next 1-3 days, you can observe the approximate electricity consumption in the next 1-3 days. If the prediction results show that the electricity consumption in the next day will be very high, you can take preventive measures to avoid overloading the power grid. On the contrary, if the prediction results show that the electricity consumption in the next day does not reach the ideal peak, you can increase some production activities to ensure the stable progress of production work.

[0154] The input sequence enters the model from bottom to top through the embedding layer. Figure 3 The Embedding layer shown in is the embedding layer. The embedding layer uses a linear layer to map the feature dimensions of the data, that is, the embedded ,in , , Represents the batch size fed into the model at one time, Indicates that the input sequence contains several features, Represents the dimension after feature mapping, where are fixed values, which are related to the composition of the data set fed into the model each time. is a hyperparameter, which can be defined as any positive integer. In the field of power load forecasting, The values ​​of are mapped to high dimensions, so that the mutual influence between different features can be explored. In this example, the input sequence Mapping to 512 dimensions fully explores the potential information between data. is the weight matrix of the linear mapping of the embedding layer, is the bias matrix of the linear mapping of the embedding layer, that is, , ,because and At the beginning, the data is randomly generated, so at this time, after mapping for:

[0155] at this time .

[0156] After the embedding layer, the data will be parallely fed into Figure 3 The spatial attention mechanism layer (SpatialAttention) and channel attention mechanism (Channel Attention) are shown in the figure. First, let’s introduce the spatial attention mechanism layer. The spatial attention mechanism mainly consists of a 1D convolution layer and a Sigmoid activation function layer. The convolution kernel size of the convolution layer is You can choose any positive integer but it cannot be greater than the length of the input sequence, and padding is performed to keep the length of the sequence unchanged. The padding number Depend on It is concluded that Indicates the rounding calculation. The input size of the convolution layer is determined by the mapping size of the data after the embedding layer, and the output size of the convolution layer is set to 1. In this example, the convolution kernel size is set to 3, so the padding amount , after feature mapping, the mapping size of the data is 512, so the input size of the convolution layer is 512. After the data enters this layer, the maximum value of each feature sequence and the average value between every two time steps are calculated. After the two calculation results are obtained, the two results are spliced. As shown below:

[0157]

[0158] The first line is the maximum value of each feature, and the remaining lines are the average values ​​of every two time steps. To use the convolutional layer to perform calculations along the time dimension, the dimension is converted. , using a convolution kernel size of 3 and padding of 1 pair Calculate the attention weight, that is, in each column of data, slide 3 time steps each time for weighted calculation. For example, in the first column of data, the calculation of attention using convolution should be Similarly, the calculation method for the remaining 511 columns of data is the same. After the convolution layer calculation is completed, the output size is set to 1, so the convolution calculation is performed through the Sigmoid activation function to map the calculation result to 0-1 to obtain the attention calculation score. At this time, the attention score matrix should be:

[0159] , the attention score Transpose it and expand it to the original sequence dimension. The expanded attention score is:

[0160]

[0161] The attention mechanism score is multiplied point by point by the original input sequence to obtain the spatial attention mechanism calculation result, that is,

[0162]

[0163] Calculated .

[0164] Then, the channel attention mechanism is calculated. The channel attention mechanism consists of a maximum pooling layer, an average pooling layer, a forward propagation layer, and a Sigmoid activation function layer. The forward propagation layer contains a linear mapping layer that maps the data to different dimensions, a relu activation function layer that performs nonlinear activation on the data, and another dimension restoration linear layer that maps the dimension back to the original feature dimension of the data. The adjustable parameter is the size of the data after mapping in the forward propagation layer, and the size of the parameter can be defined as a positive integer of any size. Similarly, for the mapped

[0165]

[0166] First, Transpose to calculate the maximum pooling layer and the average pooling layer. After transposing, calculate the average and maximum values ​​of each dimension. The maximum and average matrices obtained are:

[0167]

[0168]

[0169] The two calculated matrices are added and enter the forward propagation layer. In this example, the number of forward propagation layer mappings is defined as 256. After passing through the forward propagation layer, the output result is equal:

[0170]

[0171] The output result passes through the Sigmoid activation function layer to obtain the attention weight matrix , at this time the attention weight matrix is ​​equal to:

[0172]

[0173] The obtained attention weight matrix is ​​combined with the transposed input Perform matrix multiplication, that is

[0174]

[0175] At this time the output matrix .

[0176] After the two attention mechanisms are calculated, they pass through two linear layers respectively. The role of the linear layer is to restore the dimension and continue to capture the potential relationship of the attention mechanism calculation results. The spatial attention mechanism passes through a linear layer for same-dimensional transformation, and the channel attention mechanism passes through a linear layer that converts 1 dimension to 512 dimensions. First, the output result of the spatial attention mechanism is obtained after a same-dimensional mapping. , the mapping result is Should be:

[0177]

[0178] For the calculation results of the channel attention mechanism, the settlement results of the channel attention mechanism are first transformed into dimensions, that is, , and then in order to convert the dimension to the sequence after the embedding layer remain the same, so The feature dimension is mapped to 512 dimensions, that is, , obtained after mapping Should be:

[0179]

[0180] The spatial attention mechanism obtained at this time has the same dimension as the channel attention mechanism. Next, the calculation results of the two attention mechanisms are fused. In the fusion part, the gating mechanism is selected to fuse the calculation results of the two attention mechanisms. The gating mechanism will derive a weight based on the calculation results of the two attention mechanisms. , and then respectively as well as The final output is obtained by weighted summing the calculation results of the two attention mechanisms. In this example, it is assumed that the generated , and the proportion of spatial attention mechanism is relatively large, that is, Therefore, the final output of the attention mechanism is Should be:

[0181]

[0182] Next, keep the calculation result of the attention mechanism just now, and keep it as a cell state after a layer of normalization layer , that is, each column is standardized by calculating the mean and standard deviation, such as the mean of the first column , standard deviation , so the first value of the first column is normalized as , and so on, the calculations for other columns are the same, so the cell state is obtained The calculation result is:

[0183]

[0184] And at this time

[0185] Preserve cell state After that, the mapped sequence Subtract the calculation result of the attention mechanism to get the result at this time for:

[0186]

[0187] The calculated results are standardized layer by layer to eliminate the differences between dimensions, and the calculated results at this time are obtained. for:

[0188]

[0189] and The calculation results are After a linear forward propagation of the same dimension, that is, the size of the mapping is also 512, the result after mapping is recorded as for

[0190]

[0191] The linear propagation layer will pass through this layer Subtract the forward propagation layer , the calculation result at this time is recorded as , the result obtained by subtracting the two results is for:

[0192]

[0193] After getting the output result , After another normalization layer, the normalized result is obtained and compared with the previously retained cell structure. Add them together and the output result is , the calculation result is for:

[0194]

[0195] In calculating the residual and After that, the model enters the output layer, which consists of two linear layers. The first linear layer changes the feature dimension of the model, that is, mapping the 512-dimensional data back to 1 dimension, and the second linear layer changes the sequence length dimension of the model, that is, mapping the 16-step data to 1 dimension to predict the power consumption in the next day, so as to provide a reference for the operation of the power grid. The calculation results at this time are , first pass through the first linear layer, map the 512-dimensional data to 1-dimensional, and get the output result at this time :

[0196]

[0197] The output result is , the output result Perform dimensional transformation, that is, swap the second and third dimensions, and then enter the next linear layer for sequence length adjustment to obtain the final output result .

[0198] If the output result is higher than the average value of the input sequence (770) but not higher than the peak value of the input sequence (1337), appropriate measures should be taken in advance to use electricity more efficiently. If the building is equipped with green electricity such as photovoltaic power generation, it can be prepared in advance to ensure stable electricity consumption. The output result is higher than the peak value of 1337, which means that the power consumption on the next day may be extreme. We should be prepared for emergencies in advance and perform equipment maintenance and repairs in advance to deal with large-scale power outages caused by peak power consumption. The output result is lower than the average value of 770. The corresponding buildings can increase some production activities to improve production efficiency and increase electricity consumption to maintain the stable operation of the power grid.

[0199] Step 3: After understanding the structure of the entire model and the flow of data, the following describes how to actually train a model, including data selection and processing, the flow of data during actual training, and the mathematical formula calculations of key steps in each layer. In terms of data selection, a data set of electricity consumption in a building in Shanghai is selected. The data is a single column with days as the sampling unit, containing two years of data from 2022 to 2023, that is, 730 data. First, the original data is processed with missing values ​​and outliers. In this example, there are no missing values ​​in the data set. For outliers, box plots are drawn to determine whether there are anomalies, and outliers are defined as values ​​that are 4 times the standard deviation of the original data. The box plots before and after removing the outliers are compared. Figure 1 , Figure 2 As shown by Figure 1 It can be seen that there are obvious anomalies in the data, with the maximum value being higher than 10,000, but the number of outliers is small. From the figure, it can be seen that there are only two outliers. Therefore, the deletion method is used to directly remove the outliers. The box plot of the data set after removal is as follows: Figure 2 As shown. Deleting outliers can ensure that model training will not be affected by outliers, thereby ensuring the training effect of the model. Then, the highest temperature, lowest temperature, and wind speed of the day are introduced through the official meteorological website data, and the holidays library is used to determine whether the day is a holiday. By introducing four dynamic variables to enrich the original data set, the prediction accuracy is improved. The description of the constructed data set is as follows:

[0200]

[0201] Step 4: After constructing the data set, divide the data set into 70% of the total data, the test set accounts for 20% of the total data, and the validation set accounts for 10% of the total data. Due to the small amount of data, the batch size of the training set, test set, and validation set is set to 16. The division of training data, test data, and validation data is completed.

[0202] Step 5: Define a set of parameters to start training the model. The required parameters and the first defined parameter values ​​are shown in the following table:

[0203]

[0204] After defining the parameters, use them to train the model.

[0205] Step 6: For the input sequence The shape is , where the first dimension is the batch size, which has been mentioned in step 2 when dividing the data set. The second dimension is the input sequence length, which has been defined in step 3 when defining the parameters. The third dimension is the number of features. The five features mentioned in step 1 when constructing the data set are power consumption, maximum temperature, minimum temperature, wind speed, and whether it is a holiday. First, the sequence First enter the embedding layer to map the low-dimensional data to high-dimensional data. After feature mapping The shape is .

[0206] Step 7: After the embedding layer, the model enters the calculation of the spatial attention mechanism and the channel attention mechanism in parallel. In the spatial attention mechanism calculation layer, the maximum value of each sequence in the input sequence is first calculated. And the average of the step lengths between each two , , , concatenate the maximum value with the average of the two steps to capture both the peak and the trend in the sequence. ,in For splicing operation, , in order to capture the attention weight of the time step, the concatenated Transpose, then The shape is In step 3, the convolution kernel size is defined as 3, so the shape of the convolution kernel should be (512, 1, 3). The calculation result is obtained by convolution. , and use the Sigmoid function to map the calculation results to between (0, 1). , and obtain the importance weight of each time step, and extend the calculation results to the same as the input The same shape, the calculation result The second dimension in the result is copied 512 times, ,in To represent the expansion of the dimension and convert it into the original dimension, , now do a point-by-point multiplication , Represents the dot multiplication operation. The shape is , the three dimensions are batch size, sequence length, and feature dimension.

[0207] Step 8: Since the spatial attention mechanism is parallel to the channel attention mechanism, the channel attention mechanism is calculated at the same time. After the embedding layer To facilitate subsequent calculations, first input Transpose, then , and then use global average pooling and global maximum pooling to calculate the maximum and average values ​​of each feature channel, and the calculated average value and the maximum value The shapes of , add the average and maximum values ​​along the third dimension, the time dimension, to get a new tensor .Will Transpose to perform full connection calculation, then Then two fully connected layers are used to further compress the feature dimension and obtain the weight vector, which first passes through the first linear layer and uses the forward propagation feature map number parameter defined in step 3. , Represents the first linear layer, at this time The shape is ,go through The function performs nonlinear activation. , and then pass through a linear layer to map the sequence back to the original dimension to obtain the attention weight, that is, , Represents the second linear layer, at this time The shape should be , the attention weights are normalized using the Sigmoid function. , and finally the attention weight With input Perform matrix multiplication to obtain weighted output ,in Represents the transpose of the attention weight vector, and the output of the channel attention mechanism is The shape is , where the first dimension is the batch size, the second dimension is the feature dimension, and the third dimension is the sequence length.

[0208] Step 9: After calculating the results of the two attention mechanisms in parallel, the calculation results of the two attention mechanisms are respectively passed through a forward propagation layer to extract features from the calculation results again and change the dimensions for subsequent fusion. The calculation results of the spatial attention mechanism at this time , after a linear layer with the same latitude transformation, the features are extracted again. ,at this time The shape is still The calculation result of the channel attention mechanism is , first Perform dimensional conversion to convert the feature dimension to the third dimension to facilitate linear mapping. After the conversion, remap the 1-dimensional feature back to 512 dimensions. ,at this time The shape is converted to .

[0209] Step 10: Calculate the results of the two attention mechanisms after the linear layer as well as The merging is done through a gating unit. The gating fusion mechanism determines the weights given to space and channels based on the fully connected layer of the Sigmoid activation function. , through dynamic training to adjust the weights of the spatial attention mechanism and the channel attention mechanism, the output result is The shape is .

[0210] Step 11: Save the calculated results of the fusion attention mechanism as a separate cell state , The shape and The same is , and at the same time, using the original input sequence Dimension by dimension means subtracting the calculation result of the attention mechanism along the feature dimension , and the calculation results are ,at this time .

[0211] Step 12: After reducing the dimension by dimension, perform layer standardization to convert the result after reducing the dimension by dimension Treat each dimension of as a variable and calculate the mean and variance of this dimension:

[0212]

[0213] in represents the remaining sequence after decrementing, It represents the average value of each characteristic sequence of the remaining sequence after decreasing. Represents the standard deviation of each characteristic sequence of the remaining sequence after decreasing. . Similarly, the cell state that is preserved The same normalization process is also performed:

[0214]

[0215] in Represents each feature sequence in the calculation result of the attention mechanism, Represents the average value of each feature sequence in the calculation result of the attention mechanism. Represents the standard deviation of each feature sequence in the calculation results of the attention mechanism.

[0216] Map each dimension of data to between (0,1) to eliminate the dimensional differences between variables. Cell status The shapes of .

[0217] Step 13: Calculation results after layer standardization Continue forward propagation and continue to explore the potential relationship between data through a forward propagation layer of the same dimensional transformation. , the result after forward propagation is The shape is ,Will The result of normalization with the previous layer Carry out dimension-by-dimension reduction to obtain the calculation result , ,at this time The shape is .

[0218] Step 14: Reduce the result dimension by dimension Normalize the layers again:

[0219]

[0220] in represents the remaining sequence after decrementing, It represents the average value of each characteristic sequence of the remaining sequence after decreasing. Represents the standard deviation of each characteristic sequence of the remaining sequence after decreasing. .

[0221] The calculation results and cell status Add the residuals, that is, add them dimension by dimension to get the calculated result ,at this time The shape is The residual structure of this design can not only introduce the importance of time and features and improve prediction accuracy, but also effectively alleviate the problems of gradient explosion and gradient disappearance.

[0222] Step 15: In the final output layer, two linear layers are used for prediction. Since the final prediction target is the power consumption, the 512-dimensional feature column is first mapped to 1 dimension, that is,

[0223]

[0224] At this time, the output result after the first linear layer The shape is , and then adjust the dimension of the output result to adjust the sequence length dimension to the third dimension. , when corresponding to data collected in days, the data of the previous days are used to predict the power consumption of the next day, to observe in advance whether abnormal events will cause abnormal power consumption and optimize the overall efficiency of the generator set. Therefore, the sequence length dimension is mapped to one dimension, that is,

[0225]

[0226] The output result is in the shape of , the first dimension is the batch size, the second dimension is the number of predicted features, and the third dimension is the length of the predicted sequence. Convert the dimensions. , in this case the first dimension is the batch size, the second dimension is the predicted sequence length, and the third dimension is the number of predicted features.

[0227] Step 16: After understanding the operation process of the entire model, officially start training the model and select the mean square error Backpropagation of the model as loss function, learning rate It is set to 0.0001, and to ensure the objectivity of the experimental results, the mean absolute error is selected As a reference indicator for validating the model on the test set:

[0228]

[0229]

[0230] in, represents the true value of the data, represents the predicted value of the data,

[0231] First, define 10 rounds of model training and observe the loss on the model validation set to evaluate the generalization ability of the model. Figure 7 It can be seen that the model did not converge after 10 rounds of training, which means that the model parameters and the number of training rounds need to be redefined.

[0232] Step 17: To find the optimal parameters of the model and make the model converge, use the optuna library to tune the hyperparameters. The optuna library contains a variety of parameter optimization methods. The hyperparameters that need to be tuned include the input sequence length, the model embedding dimension, the number of encoders, and the number of feature maps in the forward propagation process. Define the model to be trained for 30 rounds and the optuna library to be trained for 10 rounds, that is, the optuna library is trained for one round and the model is trained 30 times. The optimal model parameters obtained by the optuna library are as follows:

[0233]

[0234] Step 18: Use the optimal parameters trained by the optuna library to train the entire model. The batch size is also set to 32, and the number of training rounds is set to 30. Observe the change curve of the loss function of the model training set and the validation set at this time. Figure 8 As shown in the figure, it can be seen that the model has tended to converge, but it is observed that the model seems to fall into a local minimum in the first ten rounds of training, resulting in the final model The convergence effect is not optimal, so a dynamic learning rate adjustment strategy is used. The cosine annealing learning rate scheduler is used to adjust the learning rate and change the learning rate within a fixed period. The scheduler needs to set the following parameters:

[0235]

[0236] Set T_0 to 8, that is, the learning rate is reduced every 8 rounds of training, and T_mult is set to 2, that is, after the first cycle, the next cycle is The learning rate is reduced for rounds, and so on, setting eta_min to , that is, when the learning rate drops to this value, it will no longer drop, and continue to use the value of eta_min for training. Fig. 9 It can be seen that after the model lowered the learning rate, there was no local minimum in the training process. At this time, the model training effect should reach the optimal value. The comparison between the predicted value and the true value is plotted by Fig.10 As shown in the figure, the prediction results of the model are very close to the actual test set data. At this time, the performance of the model is observed by the evaluation indicators on the test set. The two evaluation indicators are , , and The results are relatively close, indicating that the error distribution of the prediction results is relatively uniform, there are no major prediction anomalies, and the performance of the model is very good, which can be used as a practical application model. For the convenience of subsequent use, save the trained model at this time.

[0237] Step 19: To verify the effectiveness of the built model, compare it with the original Transformer model. Use the optuna library to find the optimal parameters of the original Transformer model. The optimal parameters of the original Transformer are:

[0238]

[0239] Use the optimal parameters to train the Transformer model and observe the model loss function and model fitting effect. The loss function graph is as follows: Fig.11 As shown, the fitting effect is Fig.12As shown in the figure, although the model has not overfitted, the model performance is unstable. The validation set error fluctuated significantly during the 18th and 19th rounds of training, while the training set error has been decreasing. Therefore, the original Transformer still has the risk of overfitting. Fig.12 Comparison between the true value and the predicted value shows that the predicted results fluctuate greatly between the 60th and 80th sample points in the test set, and the two evaluation indicators are , . and Compared with the improved model, the error is larger and and The difference is relatively large, indicating that the predicted value has a large anomaly. Therefore, the improved Transformer model in this paper has obvious advantages over the original Transformer model in predicting small sample data sets with daily collection units.

[0240] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A method for predicting electric load based on a small sample data set, characterized in that: The steps include: The electricity consumption data with a minimum collection unit of day and a data volume of less than 1,000 points is collected to form sequence data that changes with the date. An improved Transformer model is constructed, which has an embedding layer, a spatial attention mechanism layer, a channel attention mechanism layer, a forward propagation layer, a gating mechanism fusion layer, a residual reduction, a normalization layer, a cell state, a residual addition, and an output layer from bottom to top. The sequence data enters the embedding layer, and the embedding layer maps the sequence data to obtain higher-dimensional data. The mapped high-dimensional data enters the spatial attention mechanism layer and the channel attention mechanism layer to calculate the spatial attention mechanism and the channel attention mechanism respectively. After calculating the results of the spatial attention mechanism and the channel attention mechanism, the two results enter a forward propagation layer for forward propagation respectively. The two attention mechanism results are forward propagated separately. After that, the two attention mechanism results are fused, and the two forward propagation results enter the fusion layer to obtain the final attention mechanism calculation result. The output of the attention mechanism calculation result is divided into two steps. One step is to enter the attention mechanism calculation result into a standardization layer alone and retain it as a cell state. The other step is to subtract the attention mechanism calculation result from the output result of the embedding layer and enter the standardization layer. The result obtained after decrementing and entering the standardization layer enters a forward propagation layer. The result after forward propagation is decremented with the output result of the standardization layer and enters a standardization layer again. The output result of the second standardization layer is residually connected with the retained cell state, and the power consumption prediction result is output through the output layer. The improved Transformer model is trained and the power consumption prediction results are obtained by inputting the improved Transformer model.

2. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The embedding layer only contains a linear layer to map the feature dimension of the input data to high-dimensional data.

3. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The spatial attention mechanism layer consists of a convolutional layer and a Sigmoid activation function layer. It captures the importance of the temporal relationship of each sequence by calculating the maximum value in the sequence length dimension and the average of each two items. The channel attention mechanism layer consists of a maximum pooling layer, an average pooling layer, a forward propagation layer and a Sigmoid activation function layer. The forward propagation layer also includes two linear layers and a Relu activation function layer. The sequence length dimension data is compressed through the maximum pooling layer and the average pooling layer. After compression, the importance relationship between different features is given respectively through the forward propagation layer.

4. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The forward propagation layer only contains one linear layer. After the two attention mechanisms are calculated, they pass through a forward propagation layer to correct the dimension and explore the potential relationship between the data. The forward propagation after the channel attention mechanism layer is to correct the dimension, and the forward propagation after the spatial attention mechanism layer is to explore the potential relationship between the data.

5. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The fusion layer is based on the fully connected layer of the Sigmoid activation function to determine the weights given to space and channels. The output of the fusion mechanism is expressed by the following formula: ， in, is the result of the spatial attention mechanism after passing through the forward propagation layer, is the result of the channel attention mechanism passing through the forward propagation layer, is the weight assigned to the fully connected layer of the Sigmoid activation function. When it is close to 1, the spatial attention mechanism accounts for a larger proportion. When it is close to 0, the channel attention mechanism accounts for a larger proportion.

6. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: After calculating the mean and variance of each feature, the normalization layer maps the data to (0,1) to eliminate the dimensional differences between different features.

7. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The forward propagation layer includes a linear layer to map the data into the same dimension and explore the potential relationship after the data is reduced.

8. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The output layer contains two linear layers. The first linear layer performs feature fusion and maps the final feature dimension to 1 dimension. The second linear layer performs sequence dimension fusion and maps the final sequence length dimension to 1 dimension, thereby outputting the final prediction result.

9. The method for predicting electric load based on a small sample data set according to claim 1, characterized in that: The data set is divided into 70% of the total data volume for training set, 20% of the total data volume for test set, and 10% of the total data volume for validation set. The improved Transformer model is trained with the training set, and the improved Transformer model after training is tested with the test set and verified with the validation set.

10. The method for predicting electric load based on a small sample data set according to claim 9, characterized in that: Select mean square error Backpropagation of the model as loss function, learning rate Set to 0.0001 and select mean absolute error As a reference indicator for validating the model on the test set: ， ， in, represents the true value of the data, represents the predicted value of the data, Define the model training for 10 rounds and observe the loss on the model validation set to evaluate the generalization ability of the model.

Images