Wind speed prediction method based on sparse attention mechanism and three-dimensional convolution

Abstract

The invention discloses a wind speed prediction method based on a sparse attention mechanism and three-dimensional convolution, and the method comprises the following steps: obtaining a historical data set which comprises historical ground observation data, historical circulation elevation data and historical wind speed data in a historical time period; taking each time point in a historical time period and historical ground observation data and historical circulation elevation data corresponding to each time point as input, taking historical wind speed data corresponding to each time point as output, training a neural network, and obtaining a wind speed prediction model; the neural network comprises a time sequence prediction model and a double-branch three-dimensional convolution model; and inputting the future target time point and the future ground observation data and the future elevation data corresponding to the future target time point into a wind speed prediction model, and outputting a target wind speed corresponding to the future target time point through the wind speed prediction model. The problems that a traditional prediction model cannot capture three-dimensional space features and is low in prediction precision are solved.

Description

Technical Field

[0001] The present invention belongs to the field of wind speed prediction and relates to a prediction method, and specifically to a wind speed prediction method based on sparse attention mechanism and three-dimensional convolution. Background Art

[0002] With the growing development of the global transportation network, the safety of airport operations and the punctuality of flights have become the focus of global attention. Inaccurate wind speed forecasts may lead to unstable airport operations, which in turn affects key links such as flight takeoff and landing, route planning, etc. In addition to its impact on the aviation field, wind speed forecasts are also of great significance in other fields, such as wind power generation, tourist attractions, etc., because these fields also rely on accurate wind speed information. Especially in wind power generation, accurate wind speed forecasts can improve power generation efficiency and reduce energy costs. Therefore, improving the accuracy of wind speed forecasts has an important impact on the operation and development of all walks of life. The introduction of deep learning technology can provide more comprehensive and accurate theoretical support for wind speed forecasting. Through deep learning, the complex relationships in meteorological data can be better captured, the spatiotemporal resolution and accuracy of wind speed forecasts can be improved, and a more reliable theoretical basis can be provided for the safety and punctuality of airport flights.

[0003] The common informer model uses the sparse attention mechanism (ProbSparse Self-attention). Its main principle is to find the most active part of the value in Q and calculate the inner product with K. In the long sequence prediction, more attention can be paid to the features with strong correlation and the remaining features are replaced by the average vector, which can make the time complexity and memory usage smaller. Although the informer model performs well, it cannot capture a series of features in three-dimensional space, resulting in complex and changeable terrain still posing obstacles to high-precision numerical forecasts. Traditional weather forecasts are difficult to provide high-precision wind speed forecasts under complex terrain conditions. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a wind speed prediction method based on sparse attention mechanism and three-dimensional convolution, which solves the problem that the traditional prediction model cannot capture three-dimensional spatial features and has low prediction accuracy.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A wind speed prediction method based on sparse attention mechanism and three-dimensional convolution includes the following steps:

[0007] S1: Obtain historical data sets, which include historical ground observation data, historical circulation elevation data, and historical wind speed data within a historical period;

[0008] S2: Taking each time point in the historical time period and the historical ground observation data and historical circulation elevation data corresponding to each time point as input, and taking the historical wind speed data corresponding to each time point as output, the neural network is trained to obtain a wind speed prediction model, and the neural network includes a time series prediction model and a dual-branch three-dimensional convolution model;

[0009] S3: Input the future target time point and the future ground observation data and future elevation data corresponding to the future target time point into the wind speed prediction model, and output the target wind speed corresponding to the future target time point through the wind speed prediction model.

[0010] The beneficial effects of the above scheme are:

[0011] (1) The present invention integrates different types of data features and can better adapt to time series prediction problems in other regions. By comprehensively utilizing various types of data features, the present invention can demonstrate strong versatility and adaptability in different geographical environments, thereby enhancing the applicability of the model.

[0012] (2) The present invention utilizes a large amount of text data for training and adopts a dual-branch three-dimensional convolutional structure, so that it can learn the features before and after a certain moment, so that the model has stronger context understanding ability when processing time series data, can better cope with extreme weather conditions and complex spatiotemporal relationships, and improve the accuracy and stability of prediction.

[0013] Furthermore, the method further comprises:

[0014] Get the validation dataset;

[0015] The wind speed prediction model is evaluated using the validation dataset.

[0016] Furthermore, before step S2, the method further includes:

[0017] The historical data set is divided into training set, test set and data set in a ratio of 7:2:1 according to the length of the time interval of the historical time period;

[0018] Step S2 specifically includes:

[0019] Each time point in the historical time period in the training set and the historical ground observation data and historical circulation elevation data corresponding to each time point are taken as input, and the historical wind speed data corresponding to each time point in the training set is taken as output. The neural network is trained to obtain a wind speed prediction model.

[0020] The beneficial effect of the above further scheme is that the independence of data is maintained during the model development and evaluation process, and the generalization ability of the model can be effectively evaluated.

[0021] Furthermore, in step S2, the attention mechanism formula of the time series prediction model is:

[0022]

[0023] Among them, Q, K, and V represent the matrices of query, key, and value respectively. represents the result of the scaled dot product, d k represents the dimension of K;

[0024] The formula for sparse attention mechanism formula is:

[0025]

[0026] Among them, α is a hyperparameter that controls the degree of sparsity, and S represents a sparse subset of attention weights.

[0027] The beneficial effect of the above further scheme is: the introduction of a sparse self-attention mechanism can maintain high computational efficiency when processing long sequence data, while taking into account the use of global and local information.

[0028] Furthermore, in step S2, the dual-branch three-dimensional convolutional model includes an input layer, a three-dimensional convolutional layer, an activation layer and a pooling layer.

[0029] Furthermore, the calculation formula for calculating the value at (x, y, z) in the j-th feature map in the output of the i-th layer in the three-dimensional convolutional layer is:

[0030]

[0031] Among them, φ represents the nonlinear activation function, m represents the index of the feature map of the (i-1)th layer, represents the value at position (p, q, r) associated with the feature map, P i , Q i , R i They represent the length, width and depth of the convolution kernel in the i-th convolutional layer respectively.

[0032] Furthermore, the formula of the loss function in the wind speed prediction model is:

[0033]

[0034] in, represents the Gaussian kernel function.

[0035] The beneficial effects of the above further scheme are: the error can be mapped from the original feature space to a higher dimensional space, the prediction error of nonlinear data can be quantified in the nonlinear space, the robustness to outliers is stronger, and the stability of the results is enhanced.

[0036] Furthermore, the wind speed prediction model is evaluated using a validation dataset, including:

[0037] The wind speed prediction model was evaluated using mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE), Nash efficiency coefficient (NSE) and correlation coefficient (R) methods.

[0038] The beneficial effects of the above further scheme are: helping to improve the prediction ability and generalization ability of the wind speed prediction model, and guiding the subsequent model improvement and optimization work. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of a wind speed prediction method based on sparse attention mechanism and three-dimensional convolution.

[0040] Figure 2 This is the structural diagram of the wind speed prediction model.

[0041] Figure 3 Schematic diagram of the convolution kernel in the dual-branch 3D convolution model. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, a wind speed prediction method based on sparse attention mechanism and three-dimensional convolution includes the following steps:

[0044] S1: Obtain historical data sets, which include historical ground observation data, historical circulation elevation data, and historical wind speed data within a historical time period.

[0045] In step S1, the historical circulation elevation data may be extracted from ERA5 published by the European Center of Medium-range Weather Forecast (ECMWF). The historical ground observation data may be obtained after preprocessing ground data and satellite data.

[0046] S2: Take each time point in the historical time period and the historical ground observation data and historical circulation elevation data corresponding to each time point as input, and take the historical wind speed data corresponding to each time point as output, train the neural network, and obtain a wind speed prediction model. The neural network includes a time series prediction model and a dual-branch three-dimensional convolution model.

[0047] Before step S2, the historical data set can be divided into a training set, a test set and a data set in a ratio of 7:2:1 according to the time interval length of the historical time period, and each time point in the historical time period in the training set and the historical ground observation data and historical circulation elevation data corresponding to each time point are used as input, and the historical wind speed data corresponding to each time point in the training set are used as output to train the neural network and obtain a wind speed prediction model.

[0048] Partitioning historical data sets can maintain data independence during model development and evaluation, and can effectively evaluate the generalization ability of the model.

[0049] like Figure 2 As shown, Figure 2 The wind speed prediction model can include a time series prediction model and a dual-branch three-dimensional convolution model. Figure 2 In the example, the input of the time series prediction model can be ground observation data, which can be processed by the time series prediction model through a multi-head sparse self-attention mechanism, distillation, and multi-head sparse self-attention mechanism, and output a feature map. The input of the dual-branch 3D convolution model can be circulation data, which can be processed by convolution, maximum pooling, and convolution in the dual-branch 3D convolution model, and output a feature map. The linear layer can process the feature map output by the time series prediction model and the feature map output by the dual-branch 3D convolution model, and output the final processing result.

[0050] In step S2, the time series prediction model can be an Informer model, which is an advanced time series prediction model that combines the self-attention mechanism and the convolutional neural network. It can effectively handle long-term dependencies and local feature extraction to achieve more accurate time series prediction. The framework of the Informer model consists of an encoder, a decoder, and a predictor. The encoder part is responsible for extracting features from the input sequence, and the decoder part is responsible for generating prediction results.

[0051] The Informer model uses a self-attention mechanism and convolutional neural network, combining global and local information to achieve effective modeling of long sequence data, and has significant advantages in long-term prediction of meteorological data such as temperature, humidity, wind speed, etc.

[0052] The Informer model includes three notable features: ProbSparse self-attention mechanism, distillation, and generative decoder.

[0053] In the ProbSparse self-attention mechanism, the time complexity and memory usage of the Informer model are reduced from O(L^2) to O(Llog L), and it has quite good performance in sequence dependency alignment. From the traditional self-attention formula, a mechanism that can assign different weights to each position in the sequence so that the model can perform weighted processing according to the importance of each position in the sequence.

[0054] In step S2, the attention mechanism formula of the time series prediction model is:

[0055]

[0056] Among them, Q, K, and V represent the matrices of query, key, and value respectively. represents the result of the scaled dot product, d k Represents the dimension of K.

[0057] In the attention mechanism formula of the time series prediction model, the softmax function is used to normalize the attention weight to a probability distribution to ensure that the weighted sum is 1. The final attention representation is obtained by multiplying the weighted value V by the attention weight.

[0058] In long sequence data, the fully connected self-attention mechanism will bring a large computational overhead, so the sparse method is introduced to reduce the computational complexity. The sparse self-attention mechanism is implemented by introducing the sub-interval attention subset. The formula for sparse attention mechanism formula is:

[0059]

[0060] Among them, α is a hyperparameter that controls the degree of sparsity, and S represents a sparse subset of attention weights.

[0061] In the formula for sparse attention mechanism formula, α is a hyperparameter that controls the degree of sparsity, which is used to balance the trade-off between full connection and sparse self-attention. S is a binary matrix that represents a sparse subset of attention weights. In the sparse self-attention mechanism, only the attention weights of some positions are retained, and the weights of other positions are set to 0. The '+' in the formula means combining the sparse attention weights with the fully connected attention weights to reduce the computational complexity while retaining global information. The introduction of S makes the attention weight matrix more sparse when calculating, that is, only the attention weights between a few positions are non-zero, thereby reducing the amount of calculation. By introducing the sparse self-attention mechanism, it is possible to maintain high computational efficiency when processing long sequence data, while taking into account the use of global and local information.

[0062] Distillation emphasizes dominant attention by halving the input of cascaded layers, effectively processing extremely long input sequences.

[0063] In this embodiment, although the generative decoder is simple in concept, it performs a single forward prediction for a long time series, rather than a step-by-step prediction, which greatly improves the inference speed of long sequence prediction.

[0064] In step S2, the convolution kernel of the dual-branch 3D convolution model is as follows: Figure 3 As shown in the figure. The convolution kernel of a three-dimensional convolution is three-dimensional. Multiple three-dimensional convolution kernels of the same size will be scanned on multiple channels respectively, and the elements at corresponding positions will be added to obtain the corresponding spatiotemporal features. The three-dimensional convolution operation can capture the spatiotemporal correlation of data from near to far by increasing the depth of the convolution layer while taking into account the sequential relationship in the time dimension. Figure 3 In the figure, H represents the time window size used by the time series, which can also be defined as the depth, and W and T represent the length and width of the grid image, respectively. Figure 3 Here, k represents the size of the convolution kernel used in the three-dimensional convolution. The traditional two-dimensional convolution kernel can only perform convolution on the length and width of the image, while the three-dimensional convolution kernel can convolve in the time dimension and effectively extract time features.

[0065] Optionally, the dual-branch 3D convolution model may include an input layer, a 3D convolution layer (3D Convolution), an activation layer (Active Layer) and a pooling layer (Pooling layer).

[0066] The input layer can receive an input sequence, which can be the ERA5 reanalysis data released by the European Center of Medium-range Weather Forecast (ECMWF). Each time step corresponds to a grid map of the current time.

[0067] In the 3D convolution layer, the time steps of each batch of the input sequence will be passed to the 3D convolution layer in sequence. Compared with the 2D convolution, the 3D convolution has one more dimension, which can be the time dimension in the spatiotemporal sequence or the continuous frames in the video. The calculation formula for calculating the value at (x, y, z) in the j-th feature map in the output of the i-th layer in the 3D convolution layer is:

[0068]

[0069] Among them, φ represents the nonlinear activation function, m represents the index of the feature map of the (i-1)th layer, represents the value at position (p, q, r) associated with the feature map, P i , Q i , R i They represent the length, width and depth of the convolution kernel in the i-th convolutional layer respectively.

[0070] According to the above formula, each position in the feature map obtained by 3D CNN is correlated with several consecutive frames in the input data, indicating that three-dimensional convolution can well capture the characteristics of data in adjacent time and space.

[0071] The activation layer is an important component of the two-branch three-dimensional convolutional model. Its main function is to introduce nonlinear transformations so that the two-branch three-dimensional convolutional model can better fit complex nonlinear functional relationships and learn more complex patterns, thereby improving the expressive power of the two-branch three-dimensional convolutional model.

[0072] The pooling layer can be used to reduce the spatial size of the feature map. While shrinking the feature map, it always retains the most important features (or information) in each pooling area, thereby reducing the amount of computation in subsequent layers and avoiding overfitting. The pooling operation is performed by specifying the size of the pooling area and the stride of the operation, which is similar to the convolution operation.

[0073] The spatiotemporal features extracted by the two branches are merged, and the extracted spatiotemporal features are further combined with the text features extracted from the ground observation data, and the global spatiotemporal features can be obtained in the process.

[0074] In the present invention, due to the nonlinear problem that is common in long time series prediction data, the traditional deep learning model cannot directly evaluate the error caused by nonlinearity in linear space. Therefore, the improved Gaussian kernel loss function can be used to map the error from the original feature space to a higher-dimensional space, and quantify the prediction error of nonlinear data in nonlinear space.

[0075] Therefore, let Φ be the original feature space R n The nonlinear mapping function to the reproducing kernel Hilbert space (RKHS) H is of the form: Φ:R n →H. The inner product in kernel space can be defined as follows:

[0076]

[0077] The square of the distance between data points in RKHS can be expressed using the Mercer kernel as follows:

[0078]

[0079] In the present invention, a Gaussian kernel function can be used, and the Gaussian kernel function is as follows:

[0080]

[0081] Therefore, the square of the distance between data points in RKHS can be expressed as:

[0082]

[0083] Therefore, the formula of the loss function in the wind speed prediction model can be:

[0084]

[0085] in, represents the Gaussian kernel function.

[0086] S3: Input the future target time point and the future ground observation data and future elevation data corresponding to the future target time point into the wind speed prediction model, and output the target wind speed corresponding to the future target time point through the wind speed prediction model.

[0087] After the wind speed prediction model is trained, the wind speed prediction model can be evaluated using the validation data set using methods such as mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE), Nash efficiency coefficient (NSE), and correlation coefficient (R).

[0088] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A wind speed prediction method based on sparse attention mechanism and three-dimensional convolution, characterized in that: The method comprises: S1: Acquire a historical data set, wherein the historical data set includes historical ground observation data, historical circulation elevation data, and historical wind speed data within a historical time period; S2: taking each time point in the historical time period and the historical ground observation data and historical circulation elevation data corresponding to each time point as input, and taking the historical wind speed data corresponding to each time point as output, training a neural network to obtain a wind speed prediction model, wherein the neural network includes a time series prediction model and a dual-branch three-dimensional convolution model; S3: Input the future target time point and the future ground observation data and future elevation data corresponding to the future target time point into the wind speed prediction model, and output the target wind speed corresponding to the future target time point through the wind speed prediction model.

2. The method according to claim 1, characterized in that The method further comprises: Get the validation dataset; The wind speed prediction model is evaluated using the validation dataset.

3. The method according to claim 1, characterized in that Before step S2, the method further includes: Dividing the historical data set into a training set, a test set, and a data set in a ratio of 7:2:1 according to the time interval length of the historical time period; Step S2 specifically includes: Each time point in the historical time period of the training set and the historical ground observation data and historical circulation elevation data corresponding to each time point are used as input, and the historical wind speed data corresponding to each time point in the training set is used as output, and the neural network is trained to obtain the wind speed prediction model.

4. The method according to claim 1, characterized in that: In step S2, the attention mechanism formula of the time series prediction model is: Among them, Q, K, and V represent the matrices of query, key, and value respectively. represents the result of the scaled dot product, d k represents the dimension of K; The formula for sparsely defining the attention mechanism formula is: Among them, α is a hyperparameter that controls the degree of sparsity, and S represents a sparse subset of attention weights.

5. The method according to claim 1, characterized in that In step S2, the dual-branch three-dimensional convolutional model includes an input layer, a three-dimensional convolutional layer, an activation layer and a pooling layer.

6. The method according to claim 5, characterized in that The calculation formula for calculating the value at (x, y, z) in the j-th feature map in the i-th layer output in the three-dimensional convolutional layer is: Among them, φ represents the nonlinear activation function, m represents the index of the feature map of the (i-1)th layer, represents the value at position (p, q, r) associated with the feature map, P i , Q i , R i They represent the length, width and depth of the convolution kernel in the i-th convolutional layer respectively.

7. The method according to any one of claims 1 to 6, characterized in that: The formula of the loss function in the wind speed prediction model is: in, represents the Gaussian kernel function.

8. The method according to claim 2, characterized in that: The using the validation data set to evaluate the wind speed prediction model comprises: The wind speed prediction model was evaluated using mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE), Nash efficiency coefficient (NSE) and correlation coefficient (R) methods respectively.

Images