A short-term wind speed prediction method fusing time features and cross attention

Abstract

The application is a kind of short-term wind speed prediction method fusing time characteristics and cross attention, which comprises the following contents: using a sliding window, sample segmentation is performed on a wind speed data set, feature normalization processing is performed on the segmented samples, wind speed sequence, sine value of wind direction angle, cosine value of wind direction angle, ambient temperature and corresponding time stamp are obtained, time stamp information is extracted, and periodic encoding of different time units is performed on the time stamp information to obtain time characteristics with values in the range of [-1, 1]; the wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature and the time stamp conversion characteristics are grouped; a CA-LSTM-TCN model is constructed. By adding time encoding features, the memory and learning ability of the model for wind speed sequence period information can be improved, and when the input time sequence is long, LSTM, TCN and other networks can still effectively process long-term time sequence information, improving the accuracy of wind speed prediction.

Description

Technical Field

[0001] This invention relates to the fields of new energy power generation technology and deep learning technology, and in particular to a short-term wind speed prediction method that integrates time features and cross-attention. Background Technology

[0002] Wind is characterized by its intermittency, fluctuation, and randomness, making accurate wind speed prediction a significant challenge. The accuracy of wind speed prediction is directly related to the extent of wind energy development and utilization; higher prediction accuracy leads to greater wind energy development and utilization. For a long time, many researchers have been dedicated to exploring accurate wind speed prediction models.

[0003] Traditional machine learning methods are widely used in sequence prediction due to their advantages of small data volume, simple structure, and low hardware requirements. However, considering the non-stationarity and complexity of wind speed sequences, signal analysis-based wind speed prediction methods decompose the wind speed sequence, treating each subsequence as an independent prediction task, thus ignoring the correlation between tasks.

[0004] Therefore, this application proposes a short-term wind speed prediction method with better prediction accuracy. Summary of the Invention

[0005] This invention overcomes the shortcomings of existing wind speed prediction technologies and provides a short-term wind speed prediction method that integrates time features and cross-attention. It aims to better process historical wind speed information, better extract deeply integrated features, and improve the accuracy of wind speed prediction.

[0006] The technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention provides a short-term wind speed prediction method that integrates temporal features and cross-attention, the prediction method comprising the following:

[0008] Using a sliding window, the wind speed dataset is segmented, and the segmented samples are normalized to obtain the wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the corresponding timestamp. The timestamp information is extracted and periodically encoded in different time units to obtain time features with values ​​in the range of [-1,1].

[0009] The wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the time features after timestamp conversion are grouped into three groups: the first group is the time features, the second group is the wind speed sequence, and the third group is the other covariate features, which include the sine value of the wind direction angle, the cosine value of the wind direction angle, and the ambient temperature.

[0010] The CA-LSTM-TCN model is constructed as follows: It consists of three parallel fully connected branches. The outputs of two adjacent fully connected branches are each connected to a cross-attention module. The outputs of the two cross-attention modules are then processed by average pooling and connected to two parallel LSTM networks and a TCN network, respectively. The LSTM and TCN networks have the same input features. The output features of the LSTM and TCN networks are processed by the cross-attention modules and then passed through an unfolded layer to connect to a fully connected output layer. The output of this fully connected output layer is the output of the CA-LSTM-TCN model. The input of the CA-LSTM-TCN model consists of three different sets of features, and each set of features is processed by a fully connected branch.

[0011] Secondly, this invention provides a short-term wind speed prediction method that integrates temporal features and cross-attention. The prediction method is as follows:

[0012] Using a sliding window, the wind speed dataset is segmented, and the segmented samples are normalized to obtain the wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the corresponding timestamp. The timestamp information is extracted and periodically encoded in different time units to obtain time features with values ​​in the range of [-1,1].

[0013] The wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the time features after timestamp conversion are grouped into three groups: the first group is the time features, the second group is the wind speed sequence, and the third group is the other covariate features, which include the sine value of the wind direction angle, the cosine value of the wind direction angle, and the ambient temperature.

[0014] The three groups of data are input into three fully connected networks with three branches to extract shallow features. The time features of the first group and the wind speed sequence of the second group are processed by the first and second fully connected branches, respectively, and then fused through the first cross-attention module. At the same time, the wind speed sequence of the second group and the other covariate features of the third group are processed by the second and third fully connected branches, respectively, and then fused through the second cross-attention module. The outputs of the first and second cross-attention modules are then processed by average pooling and input into the parallel LSTM network and TCN network, respectively.

[0015] The outputs of the LSTM and TCN networks are fused through a third cross-attention module (CA). The last two dimensions of the fused features are then expanded into a single-dimensional feature, which is finally passed through a fully connected output layer to obtain the final wind speed prediction.

[0016] The wind speed dataset contains four dimensions: time information, wind speed, wind direction, and ambient temperature. The data is segmented by setting a fixed-length sliding window to obtain samples. The wind speed value to be predicted is set as the label wind speed. The dataset is segmented using a sliding window with a selected sliding window size of 72+L0. The input data dimension is 72×4, the prediction output dimension is L0×1, and L0 represents the number of time moments to predict the wind speed value. The step size of the sliding window is 1.

[0017] The process of periodic encoding involves extracting the minutes, hours, and day of the week from the date and encoding them.

[0018]

[0019] In the formula, T1, T2 and T3 are the encoded time feature values.

[0020] The cross-attention module includes a left branch and a right branch, and the specific calculation process is as follows:

[0021]

[0022] In the formula, It is the left branch input. The last dimension of data, It is the input data for the right branch, f l (·) is a linear mapping function; then initialize the three weight matrices W. q W k and W v Its dimension is C×(C / h), where C represents the number of input features in the last dimension, determined by the input data, and h represents the number of attention points, which is a hyperparameter in the cross-attention module. The weighting process of the attention is as follows:

[0023]

[0024]

[0025] In the formula, A is the redistribution weight matrix, and the next step is to calculate...

[0026]

[0027]

[0028] In the formula, Z l This is the final output vector of the cross-attention module. The input data for the left branch has two dimensions;

[0029] The computation process of the state vectors of the three fully connected branches after passing through two cross-attention modules can be summarized as follows:

[0030] Z1=CA1([H1,H2]), Z2=CA2([H2,H3])

[0031] In the formula, Z1 and Z2 are the feature vectors after cross-attention fusion, and CA1 and CA2 are two different cross-attention modules.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) The prediction method of this invention extracts and encodes timestamp features, expanding the dimension of input data features and providing periodic information for model learning, fully considering the correlation between time and wind speed. By adding time-encoded features, the model's ability to remember and learn periodic information of wind speed sequences can be improved. When the input time series is long, networks such as LSTM and TCN can still effectively process long-term time series information, improving the accuracy of wind speed prediction.

[0034] (2) The method of this invention adopts a variable grouping structure, dividing the input data into a time feature group, a wind speed sequence group, and other covariate groups. Through the cross-attention module, wind speed information is fused with time features and other covariates respectively, which strengthens the correlation between wind speed sequence and external information and takes into account time correlation and spatial correlation.

[0035] (3) The method of this invention processes the fused feature vector by inputting it into two different network modules (LSTM and TCN networks). Since the LSTM and TCN networks have different structures and are connected in parallel, they also have different focuses in feature learning. A hybrid deep learning CA-LSTM-TCN model is constructed by using LSTM, TCN and cross-attention to predict wind speed sequences. The cross-attention module fuses the outputs of the two networks to fully explore the correlation information between the data, which can improve the model performance and the prediction accuracy of wind speed. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall process of a short-term wind speed prediction method that integrates time features and cross-attention proposed in this invention.

[0037] Figure 2 This is a schematic diagram of sliding window segmentation for a short-term wind speed prediction method that integrates temporal features and cross-attention proposed in this invention.

[0038] Figure 3 This is a schematic diagram of the cross-attention module in a short-term wind speed prediction method that integrates time features and cross-attention proposed in this invention.

[0039] Figure 4This is a schematic diagram of the LSTM network structure for a short-term wind speed prediction method that integrates temporal features and cross-attention proposed in this invention.

[0040] Figure 5 This is a schematic diagram of the TCN network structure for a short-term wind speed prediction method that integrates temporal features and cross-attention proposed in this invention.

[0041] Figure 6 A comparison chart of prediction errors (MAE) (m / s) for different models.

[0042] Figure 7 A comparison chart of prediction errors (MAPE) (%) for different models. Detailed Implementation

[0043] To make the technical solution of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings.

[0044] like Figure 1-7 As shown, a short-term wind speed prediction method that integrates temporal features and cross-attention is proposed. This method employs the addition of temporal features and a cross-attention module to improve the accuracy of wind speed prediction. The prediction method includes the following:

[0045] First, a sliding window is used to segment the wind speed dataset. Then, feature normalization is performed on the segmented samples to obtain the wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the corresponding timestamp. The timestamp information is extracted and periodically encoded in different time units to obtain time features with values ​​in the range of [-1,1].

[0046] The wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the time features after timestamp conversion are grouped into three groups: the first group is the time features, the second group is the wind speed sequence, and the third group is the other covariate features, which include the sine value of the wind direction angle, the cosine value of the wind direction angle, and the ambient temperature.

[0047] The CA-LSTM-TCN model is constructed as follows: it includes three parallel fully connected branches. The outputs of two adjacent fully connected branches are each connected to a cross-attention module. The outputs of the two cross-attention modules are then processed by average pooling and connected to two parallel LSTM networks and TCN networks, respectively. The LSTM and TCN networks have the same input features. The output features of the LSTM and TCN networks are processed by the cross-attention modules and then passed through an unfolded layer to connect to a fully connected output layer. The output of this fully connected output layer is the output of the CA-LSTM-TCN model. The input of the CA-LSTM-TCN model consists of three different sets of features: the first set is time features, the second set is target wind speed features, and the third set is other covariate features. Each of the three sets of features is processed by a fully connected branch.

[0048] The three groups of data are input into three fully connected networks with three branches to extract shallow features. The time features of the first group and the wind speed sequences of the second group are processed by the first and second fully connected branches, respectively, and then fused through the first cross-attention module. At the same time, the wind speed sequences of the second group and the other covariate features of the third group are processed by the second and third fully connected branches, respectively, and then fused through the second cross-attention module. The outputs of the first and second cross-attention modules are processed by average pooling and then input into the parallel LSTM and TCN networks, respectively. The outputs of the LSTM and TCN networks are fused through the third cross-attention module (CA). The last two dimensions of the fused features are expanded into a single dimension and finally passed through a fully connected output layer to obtain the final wind speed prediction value.

[0049] LSTM and TCN networks share the same input features. However, due to their different network structures, their ability to learn from the input data during training varies, resulting in different output features. By combining the diverse learning capabilities of the two networks, they are connected in parallel for simultaneous learning.

[0050] This invention is implemented in detail according to the following steps:

[0051] Step 1: Split the dataset using a sliding window;

[0052] Step 2: Extract timestamp information from the segmented sample data and perform periodic encoding;

[0053] Step 3: Group the input variables based on time-encoding features. The first group consists of time features, which include three dimensions; the second group consists of wind speed sequences, which include one dimension; and the third group consists of other covariate features, including the sine of the wind direction angle, the cosine of the wind direction, and the ambient temperature, which together have three dimensions.

[0054] Step 4: Input the grouped variables into a three-branch fully connected layer, and then use cross attention to fuse them;

[0055] Step 5: The fused features are input into LSTM and TCN networks respectively for deep feature extraction, then the output features of the two networks are fused through the attention module, and finally the features are expanded and input into a fully connected network to learn the final predicted wind speed.

[0056] In step 1, the original wind speed dataset contains four dimensions: time information, wind speed, wind direction, and ambient temperature. The data is segmented using a fixed-length sliding window to obtain samples. The wind speed value to be predicted is set as the label wind speed. The dataset is segmented using a sliding window with a size of 72 + L0, where the input data dimension is 72 × 4, the prediction output dimension is L0 × 1, and L0 represents the number of time points for which the wind speed value is predicted. The step size of the sliding window is 1. The process is as follows: Figure 2 As shown, the entire dataset is divided into training, validation, and test sets in a 7:2:1 time series ratio.

[0057] In step 2, the timestamp information is used for periodic encoding. This mainly involves extracting the minutes, hours, and day of the week from the date for encoding.

[0058]

[0059] In the formula, T1, T2, and T3 are the encoded time feature values, which are restricted to the range [-1, 1]. The sine and cosine values ​​of the wind direction variable are calculated as wind direction features, and the original wind direction data is deleted to avoid redundancy of feature information.

[0060] In step 3, all variables are divided into three groups. The first group consists of time features, containing three dimensions; the second group consists of wind speed sequences, containing one dimension; and the third group consists of other covariate features, including the sine, cosine, and ambient temperature, totaling three dimensions. These are denoted as time information X1, wind speed information X2, and ambient temperature information X3, respectively. X1 contains three features: T1, T2, and T3; X2 contains only one feature: wind speed; and X3 contains three features: the sine, cosine, and ambient temperature of the wind direction angle.

[0061] In step 4, X1, X2, and X3 are input into the three fully connected branches for feature mapping to obtain the hidden state vector. The calculation process can be represented as follows:

[0062] H1=f1(X1), H2=f2(X2), H3=f3(X3)

[0063] In the formula, f1(·), f2(·) and f3(·) represent three different fully connected layers; H1, H2 and H3 are the hidden state vectors of their corresponding outputs, and their dimensions depend on the number of neurons in the fully connected layer.

[0064] The structure of the cross-attention module is as follows: Figure 3 As shown, the specific calculation process is as follows:

[0065]

[0066] In the formula It is the left branch input. The last dimension of data, It is the input data for the right branch, f l (·) is a linear mapping function. Then, the three weight matrices W are initialized. q W k and W v Its dimension is C×(C / h), where C represents the dimension of the input features, determined by the input data, and h represents the number of attention points, a hyperparameter in the cross-attention module that can be adjusted. The weighting process of attention is as follows:

[0067]

[0068]

[0069] In the formula, A is the redistribution weight matrix, and the specific formula for the softmax function is: z i Let i be a variable, C be the number of variables, and i = an integer equal to 1 - C. Then the next calculation process is as follows:

[0070]

[0071]

[0072] In the formula Z l This is the final output vector of the cross-attention module. The input data for the left branch has two dimensions, one of which is merged with the right branch.

[0073] The computation process of the state vectors of the three fully connected branches after passing through two cross-attention modules can be summarized as follows:

[0074] Z1=CA1([H1,H2]), Z2=CA2([H2,H3])

[0075] In the formula, Z1 and Z2 are the feature vectors after cross-attention fusion, and CA1 and CA2 are two different cross-attention modules.

[0076] In step 5, vectors Z1 and Z2 are added together and averaged to obtain Z. a Z a = (Z1+Z2) / 2;

[0077] Z a Input the LSTM network and TCN network respectively.

[0078] LSTM network structure as follows Figure 4 As shown. The formulas for calculating the gates and activation functions of LSTM are as follows:

[0079] Input gate: i t =σ(W i ·[h t-1 ,x t ]+b i )

[0080] Forgotten Gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0081] Output gate: o t =σ(W o ·[h t-1 ,x t ]+b o )

[0082] Neuron state update: h t =o t ·tanh(c t )

[0083] Neuron output calculation:

[0084] Activation function sigmoid:

[0085] Activation function tanh:

[0086] In the formula, σ is the activation function Sigmoid, tanh is also an activation function, and W i and W c It is the input gate weight matrix, b i and b c It is the input gate bias vector; W f It is the forget gate weight matrix, b f W is the forget gate bias vector; o It is the output gate weight matrix, b o It is the output gate bias vector.

[0087] The structure of a TCN (Temporal Convolutional Network) is as follows: Figure 5 As shown. The formula for calculating TCN is as follows:

[0088] Dilated causal convolution:

[0089] FRN Normalization: z i =max(y i ,τ)

[0090] Swish activation function: f(x) = x·sigmoid(βx)

[0091] Dropout function: r = rand(·),

[0092] In the formula, d represents the sampling rate of the causal convolution, and k represents the size of the convolution kernel; in FRN normalization, γ, α, and τ are learnable parameters that are gradually adjusted by the network during training, and ε is a manually set hyperparameter, which is generally set to 0.01; in the Swish activation function, β is a manually set hyperparameter. When β = 0, the Swish activation function becomes the linear function x / 2, and when β = ∞, the Swish activation function becomes 0 or x; in the Dropout function, rand represents a random function that generates random numbers between 0 and 1, and p is a hyperparameter that represents the probability of inactivation, with a value ranging from 0 to 1.

[0093] The computation process for LSTM and TCN networks can be summarized as follows:

[0094] M1 = LSTM(Z) a M2 = TCN(Z) a )

[0095] Then, after passing through the cross-attention module, we obtain:

[0096] D = CA([M1,M2])

[0097] In the formula, D is the fused feature vector, which is a three-dimensional matrix of N×L×C. After the expansion layer, the last two dimensions are merged into N×(L×C). N represents the number of samples, i.e. the number of sliding windows, L is the length of the input data (72 in this embodiment), and C is the number of channels. Finally, the final predicted output value is obtained after passing through a fully connected layer.

[0098] Y = D·W F +b F

[0099] In the formula W F Let b be the weight matrix of the fully connected layer. FHere is the bias vector of the fully connected layer; the predicted output Y = [y1, y2, ..., y] is given by a step size of L. L ] T .

[0100] During network training, an early stopping mechanism is used to monitor the loss value of the CA-LSTM-TCN model. Training stops when the loss value no longer decreases within a certain number of iterations (e.g., 7 iterations); otherwise, training continues until the set number of iterations is completed. Before network training, hyperparameters such as learning rate, batch size, and loss function need to be specified. In this method, the commonly used loss function is the mean squared error (MSE) regression loss, the batch size is set to 64, and the learning rate is set to 0.001.

[0101] The mean squared error function and the linear exponential function are combined to form a new loss function for model training, as shown in the following equation.

[0102]

[0103] α and β are adjustment parameters; by selecting appropriate α and β, the robustness and asymmetry of the estimation can be controlled. The larger α is, the stronger the asymmetry and the greater the penalty for positive errors.

[0104] Figure 6 and Figure 7 The results of different prediction models for MAE and MAPE at different prediction durations are shown respectively.

[0105] To verify the effectiveness of this invention, under the same input features, the algorithm of this invention (CA-LSTM-TCN) was compared with other deep learning algorithms (MLR, RNN, GRU, LSTM, TCN, and LSTM-TCN two networks cascaded together) using two representative evaluation metrics, MAE and MAPE. Experimental results show that, compared with other deep learning networks, the prediction algorithm in this paper has higher prediction accuracy and better results.

[0106] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0107] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A short-term wind speed prediction method fusing time features and cross-attention, characterized in that, The process of this prediction method is as follows: Using a sliding window, the wind speed dataset is segmented, and the segmented samples are normalized to obtain the wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the corresponding timestamp. The timestamp information is extracted and periodically encoded in different time units to obtain time features with values ​​in the range of [-1,1]. The wind speed sequence, the sine value of the wind direction angle, the cosine value of the wind direction angle, the ambient temperature, and the time features after timestamp conversion are grouped into three groups: the first group is the time features, the second group is the wind speed sequence, and the third group is the other covariate features, which include the sine value of the wind direction angle, the cosine value of the wind direction angle, and the ambient temperature. The three groups of data are input into three fully connected networks with three branches to extract shallow features. The time features of the first group and the wind speed sequence of the second group are processed by the first and second fully connected branches, respectively, and then fused through the first cross-attention module. At the same time, the wind speed sequence of the second group and the other covariate features of the third group are processed by the second and third fully connected branches, respectively, and then fused through the second cross-attention module. The outputs of the first and second cross-attention modules are then processed by average pooling and input into the parallel LSTM network and TCN network, respectively. The outputs of the LSTM and TCN networks are fused through a third cross-attention module. The last two dimensions of the fused features are then expanded into a single-dimensional feature, which is finally passed through a fully connected output layer to obtain the final wind speed prediction.

2. The fusion time feature and cross-attention based short-term wind speed prediction method according to claim 1, characterized in that, The wind speed dataset contains four dimensions: time information, wind speed, wind direction, and ambient temperature. The data is segmented by setting a fixed-length sliding window to obtain samples. The wind speed value to be predicted is set as the label wind speed. The dataset is segmented using a sliding window with a selected sliding window size of 72+L0. The input data dimension is 72×4, the prediction output dimension is L0×1, and L0 represents the number of time moments to predict the wind speed value. The step size of the sliding window is 1.

3. The fusion time feature and cross-attention based short-term wind speed prediction method according to claim 1, characterized in that, The process of periodic encoding involves extracting the minutes, hours, and day of the week from the date and encoding them. ， In the formula , and are the encoded time characteristic values.

4. The fusion time feature and cross-attention based short-term wind speed prediction method according to claim 3, characterized in that, The first group consists of time features, which include three dimensions; the second group consists of wind speed sequences, which include one dimension; and the third group consists of other covariate features, including the sine of the wind direction angle, the cosine of the wind direction angle, and ambient temperature, which together have three dimensions.

5. The fusion time feature and cross-attention based short-term wind speed prediction method according to claim 3, characterized in that, The cross-attention module includes a left branch and a right branch, and the specific calculation process is as follows: ， ， In the formula, It is the left branch input. The last dimension of data, This is the input data for the right branch. It is a linear mapping function; then, three weight matrices are initialized. and Its dimensions are , This indicates the number of input features in the last dimension, determined by the input data. The number of attention points represents a hyperparameter in the cross-attention module. The weighting process for attention is as follows: ， ， In the formula, A is the redistribution weight matrix, and the next step is to calculate... ， ， In the formula, This is the final output vector of the cross-attention module. The input data for the left branch has two dimensions; The computation process of the state vectors of the three fully connected branches after passing through two cross-attention modules can be summarized as follows: ， ， In the formula, and is a cross-attention fused feature vector, and are two different cross-attention modules, , and are hidden state vectors corresponding to the outputs of the three fully connected branches.

Images