A deep learning wind power prediction method based on multi-stage attention mechanism

Abstract

The present invention discloses a deep learning wind power prediction method based on a multi-stage attention mechanism. Aiming at the problem that wind power is difficult to predict, the present invention designs a multi-stage attention long short-term memory network prediction model, and the specific steps are as follows: first, convolution and attention mechanism are integrated to form a first-level attention to extract multi-dimensional features of dynamic changes in wind power data; then, the feature vector after spatial importance weight allocation is input into a long short-term memory network (LSTM) to extract time series features; then, a second-level attention mechanism is used to assign different weights to hidden state features at different times; finally, in order to avoid the problem of low quality of hidden state extraction of the second-level attention caused by the randomness of wind power data, a multi-stage attention and LSTM unit are connected in parallel to obtain the designed model by using a random difference structure. Compared with the existing machine learning and deep learning methods, it can more accurately predict wind power.

Description

Technical Field

[0001] The present invention relates to the field of wind power prediction, and in particular to a deep learning wind power prediction method based on a multi-stage attention mechanism. Background Art

[0002] With the development of human society and the continuous expansion of industrialization, the global consumption of fossil energy is increasing, leading to the continuous aggravation of environmental pollution. In recent years, renewable energy has received widespread attention as an alternative to traditional fossil fuels. Among renewable energy sources, wind energy, as a pollution-free renewable energy, has gradually become the focus of energy development in various countries due to its advantages such as rapid growth, high utilization efficiency, and easy large-scale deployment and installation. However, due to the inherent randomness and intermittency of wind energy, the output power of wind farms is extremely unstable and easily affected by a variety of time-varying meteorological factors. This will bring huge challenges to the safe grid connection of wind power and the reasonable allocation of resources. Accurate and real-time wind power prediction can provide effective guidance information for wind power resource scheduling, which is crucial to the safe, stable optimization and operation of wind farms. Therefore, wind power prediction technology has been widely studied, its prediction methods have been continuously broadened, and the prediction reliability has been continuously improved.

[0003] At present, wind power prediction methods can be divided into physical methods, statistical methods and machine learning methods. Physical methods require the use of a large amount of multi-source data and computing resources such as numerical weather forecasts (NWP). In short-term and ultra-short-term predictions, the prediction accuracy of physical methods is often not ideal. Statistical methods usually use linear models and have poor ability to process nonlinear data. Machine learning methods can process complex and nonlinear data, and data-driven artificial intelligence prediction methods are currently the most widely used methods. However, the shallow structure of machine learning methods only shows strong nonlinear fitting capabilities in small-scale data sets. When the data set is a large-scale data set containing many features and a large number of samples, this shallow structure is difficult to extract deep features. With the continuous development of artificial intelligence, deep learning plays an increasingly important role in the field of wind power prediction. At present, the most widely used deep learning models include: Convolutional Neural Networks (CNN), Deep Belief Networks (DBN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), etc. The use of deep networks can more accurately capture the nonlinear mapping relationship between historical wind power data and characteristic information, thereby completing the prediction of nonlinear and non-stationary power data and obtaining better prediction results.

[0004] In addition, forecasting methods that combine the individual advantages of a single forecasting model have become popular in recent years, and a large number of studies have shown that forecasting methods based on a combination of multiple models usually perform better than using only one model in predicting time series. By analyzing the problem to be solved and integrating the attention mechanism into the classic deep learning model, the problem-solving ability of the deep learning model can also be improved. Although existing deep learning modeling methods have performed well in wind power forecasting, most of these deep learning models are stacked with different modules, such as coupling convolutional networks with recurrent neural networks and directly embedding attention mechanisms. In order to improve accuracy, modules are stacked blindly, and no reasonable deep learning framework is proposed for wind power problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a deep learning wind power prediction method based on a multi-stage attention mechanism, which enhances the model's ability to capture the spatiotemporal characteristics of wind power historical data and improves the accuracy of wind power prediction.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows: a deep learning wind power prediction method based on a multi-stage attention mechanism, the method comprising the following steps:

[0007] Step 1: Collect the original historical wind speed, wind direction, and power series data of the wind farm within two years. Raw ∈R m×n ;

[0008] Step 2: Collect the data D Raw ∈R m×n Divide into training set, validation set and test set;

[0009] Step 3: Input the data obtained in step 2 into the convolution layer to obtain a feature map, and perform similarity calculation on the feature map to form a first-level attention mechanism to fully extract spatial features.

[0010] Step 4: Input the feature map obtained in step 3 into LSTM to fully extract the time features;

[0011] Step 5: The LSTM output obtained in step 4 and the output of the first-level attention mechanism are input into the second-level attention mechanism to perform spatiotemporal feature fusion, improve the quality of the time-step hidden state, and output the prediction result;

[0012] Step 6 uses the training set data for training. When the verification error on the verification set is the smallest, the model prediction performance is evaluated on the test set.

[0013] As a further description of the above scheme: the historical wind farm operation data in step 1, D Raw ={X, y}, X∈R m×(n-1) , y∈R m×1 , where m is the number of samples, n = f × l, f is the number of features; the wind power data is treated as a multivariate time series, l is the number of delayed variables, and y is the predicted variable, i.e., the actual value of wind power.

[0014] As a further description of the above scheme: In step 2, the data is divided into D according to the time sequence. y1 , D y2 Two years of data; D y1 The first three quarters are used as training sets, D y1 The data of the last quarter is the validation set, D y2 The data points do not participate in the training process and are all used as test sets.

[0015] As a further description of the above scheme: Step 3 uses 1×1 convolution to capture spatial features, and generates N blocks through the convolution module to obtain 3×N feature maps containing rich information, which are used as the conversion of query (Q), key (K), and value (V) of the attention mechanism. The specific calculation formula is as follows:

[0016]

[0017] in, represents the convolutional map, d k is the dimension of the key vector, and the softmax operation is performed along that specific dimension to ensure that the sum of the output weights is 1.

[0018] As a further description of the above scheme: Step 4 inputs the 3×N feature maps containing rich information into the LSTM through the fully connected layer to fully extract the time features. The training of the LSTM passes through three unit gates in sequence. The specific calculation process is as follows:

[0019] (1) At time t, the hidden layer node output vector h of the previous time t-1 After being concatenated with the current input vector X(t), it is input into the model. After passing through the forget gate, the gate value f of the forget gate can be obtained. t , calculated as follows:

[0020] f t =σ(w f [h t-1 ,X(t)]+b f )

[0021] Among them, σ represents the sigmoid function, w f represents the forget gate weight vector; bf represents the forget gate bias vector, f t Represents the output of the forget gate, and its value is between 0 and 1, 0 means forgetting all state information, and 1 means keeping all;

[0022] (2) After that, the data is fed into the input gate, which determines what information in the current input will be stored in the cell state; it consists of two parts: a sigmoid layer that determines which values ​​we will update, and a tanh layer that creates a new vector of candidate values ​​that can be added to the state, which is calculated as follows:

[0023] i t =σ(w i [h t-1 ,X(t)]+b i )

[0024]

[0025] Among them, i t Represents the input gating value, which is between 0 and 1; Indicates the candidate value of the state, which determines the size of the state update value, w i , w c , b i , b c Represent the corresponding weights and biases respectively;

[0026] (3) Then, the unit state is updated. In each step, it forgets some old information based on the output of the forget gate and adds new candidate values, which are controlled by the input gate; the calculation is as follows:

[0027]

[0028] Among them, c t represents the updated state value, ⊙ represents the Hadamard Product;

[0029] (4) Finally, the output gate controls which parts of the cell state will be output. The sigmoid layer is used to determine which parts of the cell state will be output. The cell state is passed through tanh to obtain a value between -1 and 1, and it is multiplied by the output of the sigmoid gate. This will only output the part that determines the output. This is calculated as follows:

[0030] o t =σ(w o [h t-1 ,X(t)]+b o )

[0031] h t =ot ⊙tanh(c t )

[0032] Among them, t Indicates the gate value of the output gate, which is also between 0 and 1. t represents the hidden layer output at time t, w o , b o Represent the corresponding weights and biases respectively.

[0033] As a further description of the above scheme: the second-level attention mechanism is introduced in step 5, mainly to calculate the weight distribution of hidden states at different time steps. The calculation of weights is not only related to the output, but also to the spatial attention score of the input, taking into account the spatiotemporal dependency of the global information of wind power data; the specific calculation process of the second-level attention mechanism is as follows:

[0034] (1) The LSTM network outputs the latent vector h = [h1, h2, ..., h t ], calculate the attention score at the current time t:

[0035] q t =vtanh(wh t-1 +b)

[0036] Among them, q t represents the attention score at the previous t moments, v represents the key-value vector w, and b represents the corresponding weight and bias respectively;

[0037] (2) To avoid the impact of the randomness of wind power data, the information of the first-level attention mechanism is added to calculate the latent feature vector c after time importance distribution. t :

[0038]

[0039] Among them, Z is the spatial feature, σ i With Z i are the first-level attention output and temporal latent feature vector at time i respectively;

[0040] (3) The latent vector c obtained by the second-level attention mechanism t It is concatenated with the spatial information Z transmitted by the residual network, and the output Y is calculated through linear mapping. t :

[0041]

[0042] Among them, W t represents the weight at time t, and B represents the fixed bias of the final output layer.

[0043] As a further description of the above scheme: Step 6 inputs the training set and uses the small batch stochastic gradient descent algorithm to train the model. The specific training steps are as follows:

[0044] (1) Initialize the MA-LSTM network parameters and normalize the training set data; perform standardization processing, and the standardization formula is shown as follows:

[0045]

[0046] in, is the normalized data set, μ is the sample mean, and σ is the sample standard deviation;

[0047] (2) Obtain the predicted value on the validation set and calculate the loss function. The formula of the loss function L is as follows:

[0048]

[0049] Among them, Y real，i is the true value, Y pre，i is the predicted value, m is the number of samples;

[0050] (3) Repeat steps 3 to 5 as the model training process until the loss function L converges and the prediction performance of the model is obtained on the test set.

[0051] The characteristics of the present invention are as follows: Due to the inherent intermittent and random nature of wind speed, the existing time series prediction method is difficult to capture the complex nonlinear relationship between wind speed and wind power, resulting in low prediction accuracy. Aiming at the problem that wind power is difficult to predict, the present invention introduces a multi-stage attention mechanism based on a deep learning method, and designs a multi-stage attention long short-term memory network (MA-LSTM) prediction model to improve the accuracy of wind power prediction. The specific steps are: first, the convolution and attention mechanism are integrated to form a first-level attention to extract the multi-dimensional features of the dynamic changes of wind power data; then the feature vector after the spatial importance weight allocation is input into the long short-term memory network (LSTM) to extract the time series features; then the second-level attention mechanism is used to assign different weights to the hidden state features at different times; finally, in order to avoid the problem of low quality of the hidden state extracted by the second-level attention due to the randomness of wind power data, the multi-stage attention and LSTM units are connected in parallel to obtain the designed model.

[0052] Compared with the prior art, the present invention has the following beneficial effects: the prediction method provided by the present invention enhances the model's ability to capture the spatiotemporal characteristics of wind power historical data, and can more accurately predict wind power compared with existing machine learning and deep learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the flow chart of the proposed method MA-LSTM wind power prediction;

[0054] Figure 2 This is an example diagram of the MA-LSTM structure of the proposed method;

[0055] Figure 3 This is a performance comparison chart of different prediction methods in 2-hour prediction (data from March 2012); DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] Implementation Cases:

[0058] The actual wind energy data for this case study comes from the International New Energy Laboratory, and all samples and characteristics are used for the experiment.

[0059] Step 1: Select the data from the International New Energy Laboratory for 2011 and 2012, and collect 16MW wind turbine data with a sampling time of 5 minutes in 2011 and 2012. After downsampling, 210,242 samples are obtained with a sampling interval of 15 minutes.

[0060] Step 2: In the experiment, the modeling data is divided into three parts: training set, validation set and test set, as shown in Table 1. In order to further demonstrate the model's ability to predict wind power, the data for the entire year of 2012 are not used in training and validation, but only as a test set to be predicted.

[0061] Table 1 Division of the dataset used in the experiment

[0062]

[0063] Step 3: Use 1×1 convolution to capture spatial features, and set the number of convolution layers to 1. And generate N blocks through the convolution module to obtain 3×N feature maps containing rich information, which are used as the conversion of query (Q), key (K), and value (V) of the attention mechanism. The specific calculation formula is as follows:

[0064]

[0065] in, represents the convolutional map, d k is the dimension of the key vector, and the softmax operation is performed along a specific dimension to ensure that the sum of the output weights is 1. The number of convolutional layers is set to 1, and the convolution kernel size is set to 1×1.

[0066] Step 4: Input the 3×N feature maps containing rich information into LSTM through the fully connected layer to fully extract the temporal features. The training of LSTM passes through three unit gates in sequence. The specific calculation process is as follows:

[0067] (1) At time t, the hidden layer node output vector h of the previous time t-1 It is concatenated with the input vector X(t) at the previous moment and then input into the model. After passing through the forget gate, the gate value f of the forget gate can be obtained. t , calculated as follows:

[0068] f t =σ(w f [h t-1 ,X(t)]+b f )

[0069] Among them, σ represents the sigmoid function, w f represents the forget gate weight vector; b f represents the forget gate bias vector, f t Represents the output of the forget gate, and its value is between 0 and 1, where 0 means forgetting all state information and 1 means keeping all of it.

[0070] (2) After that, the data is fed into the input gate, which determines what information in the current input will be stored in the cell state. It consists of two parts: a sigmoid layer that determines which values ​​we will update, and a tanh layer that creates a new vector of candidate values ​​that can be added to the state. This is calculated as follows:

[0071] i t =σ(w i [h t-1 ,X(t)]+b i )

[0072]

[0073] Among them, i t Represents the input gating value, which is between 0 and 1; Indicates the candidate value of the state, which determines the size of the state update value, w i , w c , b i , b c Represent the corresponding weights and biases respectively.

[0074] (3) Then, the unit state is updated. In each step, it forgets some old information based on the output of the forget gate and adds new candidate values, which are controlled by the input gate. The calculation is as follows:

[0075]

[0076] Among them, c t represents the updated state value, and ⊙ represents the Hadamard Product.

[0077] (4) Finally, the output gate controls which parts of the cell state are output. Use the sigmoid layer to determine which parts of the cell state are output. Pass the cell state through tanh (to obtain a value between -1 and 1) and multiply it by the output of the sigmoid gate. This will only output the part that determines the output. This is calculated as follows:

[0078] o t =σ(w o [h t-1 ,X(t)]+b o )

[0079] h t =o t ⊙tanh(c t )

[0080] Among them, t Indicates the gate value of the output gate, which is also between 0 and 1. t represents the hidden layer output at time t, w o , b o Represent the corresponding weights and biases respectively.

[0081] The number of hidden layer nodes of LSTM is set to 64 nodes, the number of layers is set to 2, and the learning rate is set to 0.002.

[0082] Step 5: The second-level attention mechanism is introduced to calculate the weight distribution of hidden states at different time steps. The weight calculation is not only related to the output, but also to the spatial attention score of the input, taking into account the spatiotemporal dependency of the global information of wind power data. The specific calculation process of the second-level attention mechanism is as follows:

[0083] (1) The LSTM network outputs the latent vector h = [h1, h2, ..., h t ], calculate the attention score at the current k moment:

[0084] q t =vtanh(wh t-1 +b)

[0085] Among them, q t represents the attention score at the previous t moments, v represents the key-value vector w, and b represents the corresponding weight and bias respectively;

[0086] (2) To avoid the impact of the randomness of wind power data, the information of the first-level attention mechanism is added to calculate the latent feature vector c after time importance distribution. t :

[0087]

[0088] (3) The latent vector c obtained by the second-level attention mechanism t It is concatenated with the spatial information Z transmitted by the residual network, and the output Y is calculated through linear mapping. t :

[0089]

[0090] Step 6: Input the training set and train the model using the mini-batch stochastic gradient descent algorithm. All weights and biases in steps 4 and 5 are calculated by back propagation. The specific training steps are as follows:

[0091] (1) Initialize the MA-LSTM network parameters and normalize the training set data. Perform standardization processing, and the standardization formula is as follows:

[0092]

[0093] in, is the normalized data set, μ is the sample mean, and σ is the sample standard deviation.

[0094] (2) Obtain the predicted value on the validation set and calculate the loss function. The formula of the loss function L is as follows:

[0095]

[0096] Among them, Y real，i is the true value, Y pre,i is the predicted value.

[0097] (3) Repeat steps 3 to 5 as the model training process until the loss function L converges and the prediction performance of the model is obtained on the test set. The training rounds are set to 200 and the batch size is set to 64.

[0098] The present invention compares the following methods:

[0099] (1) Persistence, a classic benchmark prediction method for time series;

[0100] (2) The classic machine learning method support vector machine regression (SVR);

[0101] (3) LSTM, a deep learning method widely used in wind power prediction;

[0102] (4) Commonly used combined model prediction method CNN-LSTM;

[0103] (5) MA-LSTM, a deep learning wind power prediction method based on multi-stage attention mechanism (the method proposed in the present invention).

[0104] In order to reduce the error caused by the operation, all prediction results are taken as the average of 10 times, removing the best and worst results. This paper uses the root mean square error RMSE and the determination coefficient R 2 Two indicators are used to evaluate the performance of the model.

[0105]

[0106] Among them, y test,i and represent the true value and the predicted value respectively. represents the average value of the true value, N test Indicates the number of samples used to evaluate model performance.

[0107] The experiment tested the prediction performance of various prediction methods and models in three scenarios: 1 hour, 2 hours, and 4 hours in advance. The results are shown in Table 2.

[0108] Table 2 Comparison of prediction performance of different prediction methods

[0109]

[0110] From Table 2, Figure 3 The following conclusions can be drawn:

[0111] The MA-LSTM model provided by the present invention has the best performance in all cases. Compared with the best-performing CNN-LSTM model in the January-March forecast, the R 2 The 1-hour, 2-hour and 4-hour advance predictions were improved by 5.34%, 6.39% and 14.17% respectively; as the prediction time scale increases, the advantages of the proposed method become more and more obvious, and the 4-hour advance prediction performance is significantly improved. The MA-LSTM model proposed in the present invention has greatly improved the lag in the prediction results, can most closely track the trend of the actual power value, and has a strong wind power prediction capability.

[0112] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A deep learning wind power prediction method based on a multi-stage attention mechanism, characterized in that: The method comprises the following steps: Step 1: Collect the original historical wind speed, wind direction and power series data of the wind farm within two years. Raw ∈R m×n ; Step 2: Collect the data D Raw ∈R m×n Divide into training set, validation set and test set; Step 3: Input the data obtained in step 2 into the convolution layer to obtain a feature map, and perform similarity calculation on the feature map to form a first-level attention mechanism to fully extract spatial features. Step 4: Input the feature map obtained in step 3 into LSTM to fully extract the time features; Step 5: The LSTM output obtained in step 4 and the output of the first-level attention mechanism are input into the second-level attention mechanism to perform spatiotemporal feature fusion, improve the quality of the time-step hidden state, and output the prediction result; Step 6: Use the training set data for training. When the verification error on the verification set is the smallest, evaluate the model prediction performance on the test set. The step 3 uses 1×1 convolution to capture spatial features, and generates N blocks through the convolution module to obtain 3×N feature maps containing rich information, which are used as the conversion of query Q, key K, and value V of the attention mechanism. The specific calculation formula is as follows: in, represents the convolutional map, d k is the dimension of the key vector, and the softmax operation is performed along a specific dimension to ensure that the sum of the output weights is 1; In step 4, 3×N feature maps containing rich information are input into LSTM through the fully connected layer to fully extract the time features. The training of LSTM passes through three unit gates in sequence. The specific calculation process is as follows: (1) At time t, the hidden layer node output vector h of the previous time t-1 After being concatenated with the current input vector X(t), it is input into the model. After passing through the forget gate, the gate value f of the forget gate can be obtained. t , calculated as follows: f t =σ(w f [h t-1 ,X(t)]+b f ) Among them, σ represents the sigmoid function, w f represents the forget gate weight vector; b f represents the forget gate bias vector, f t Represents the output of the forget gate, and its value is between 0 and 1, where 0 means forgetting all state information and 1 means keeping all of it; (2) After that, the data is fed into the input gate, which determines what information in the current input will be stored in the cell state; it consists of two parts: a sigmoid layer that determines which values ​​we will update, and a tanh layer that creates a new vector of candidate values ​​that can be added to the state, which is calculated as follows: i t =σ(w i [h t-1 ,X(t)]+b i ) Among them, i t Represents the input gating value, whose value is between 0 and 1; Indicates the candidate value of the state, which determines the size of the state update value, w i , w c , b i , b c Represent the corresponding weights and biases respectively; (3) Then, the unit state is updated. In each step, it forgets some old information based on the output of the forget gate and adds new candidate values, which are controlled by the input gate; the calculation is as follows: Among them, c t represents the updated state value, ⊙ represents the Hadamard Product; (4) Finally, the output gate controls which parts of the cell state will be output. The sigmoid layer is used to determine which parts of the cell state will be output. The cell state is passed through tanh to obtain a value between -1 and 1, and it is multiplied by the output of the sigmoid gate. This will only output the part that determines the output. This is calculated as follows: o t =σ(w o [h t-1 ,X(t)]+b o ) h t =o t ⊙tanh(c t ) Among them, t Represents the gate value of the output gate, which is also between 0 and 1. t represents the hidden layer output at time t, w o , b o Represent the corresponding weights and biases respectively; The second-level attention mechanism is introduced in step 5, mainly to calculate the weight distribution of hidden states at different time steps. The calculation of weights is not only related to the output, but also to the spatial attention score of the input, taking into account the spatiotemporal dependency of the global information of wind power data. The specific calculation process of the second-level attention mechanism is as follows: (1) The LSTM network outputs the latent vector h = [h1, h2, ..., h t ], calculate the attention score at the current time t: q t =vtanh(wh t-1 +b) Among them, q t represents the attention score at the previous t moments, v represents the key-value vector w, and b represents the corresponding weight and bias respectively; (2) To avoid the impact of the randomness of wind power data, the information of the first-level attention mechanism is added to calculate the latent feature vector c after time importance distribution. t : Among them, Z is the spatial feature, σ i With Z i are the first-level attention output and temporal latent feature vector at time i respectively; (3) The latent vector c obtained by the second-level attention mechanism t It is concatenated with the spatial information Z transmitted by the residual network, and the output Y is calculated through linear mapping. t : Among them, W t represents the weight at time t, and B represents the fixed bias of the final output layer.

2. The deep learning wind power prediction method based on multi-stage attention mechanism according to claim 1 is characterized in that: Step 1: Collect the original historical wind speed, wind direction and power series data of the wind farm within two years. Raw ={X, y}, X∈R m×(n-1) , y∈R m×1 , where m is the number of samples, n = f × l, f is the number of features; the wind power data is treated as a multivariate time series, l is the number of delayed variables, and y is the predicted variable, i.e., the actual value of wind power.

3. The deep learning wind power prediction method based on multi-stage attention mechanism according to claim 1 is characterized in that: In step 2, the data is divided into D y1 , D y2 Two years of data; D y1 The first three quarters are used as training sets, D y1 The data of the last quarter is the validation set, D y2 The data does not participate in the training process and is all used as a test set.

4. The deep learning wind power prediction method based on multi-stage attention mechanism according to claim 1 is characterized in that: In step 6, the training set is input and the model is trained using a small batch stochastic gradient descent algorithm. The specific training steps are as follows: (1) Initialize the MA-LSTM network parameters and normalize the training set data; perform standardization processing, and the standardization formula is shown as follows: in, is the normalized data set, μ is the sample mean, and σ is the sample standard deviation; (2) Obtain the predicted value on the validation set and calculate the loss function. The formula of the loss function L is as follows: Among them, Y real，i is the true value, Y pre，i is the predicted value, m is the number of samples; (3) Repeat steps 3 to 5 as the model training process until the loss function L converges and the prediction performance of the model is obtained on the test set.

Images