Transformer ultra-short-term wind power prediction model improved by introducing partial hypothesis

Abstract

The application relates to a Transformer ultra-short-term wind power prediction model improved by introducing a partial hypothesis, which comprises a PlicLSTM space-time feature extraction module and a power prediction module based on MHA; the PlicLSTM space-time feature extraction module extracts local key information according to wind direction and distance factors; and the power prediction module based on MHA pays attention to global correlation information among all stations. Through accurate prediction of wind power, the application assists in optimizing the operation strategy of a wind farm, reduces operation risks, and promotes wind power grid connection; meanwhile, the application provides reliable prediction data support for an energy dispatching department, so as to support balanced dispatching and energy optimization utilization of a power system, and helps to improve the management efficiency of the wind farm and the stable operation of the power system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, artificial intelligence and wind power prediction, and particularly relates to a Transformer ultra-short-term wind power prediction model improved by introducing a partiality assumption. BACKGROUND

[0002] With the gradual increase of the proportion of wind power in the power system, the uncertainty of wind power has a greater impact on the safe and stable operation of the power system, the utilization efficiency of wind power, and the generation cost. Wind power prediction (WPP) can increase the ability of the power grid to accommodate wind power, improve the economy of the power system operation, and has an important role in power grid dispatching and wind farm operation management.

[0003] Transformer is a sequence-to-sequence (Seq2Seq) architecture model with multi-head attention (MHA) mechanism as the core, which is applied to machine translation tasks, and then introduced into the field of computer vision, improving the effect of image classification, target detection and semantic segmentation tasks. Existing work applies Transformer to the task of ultra-short-term wind power prediction, and compared with previous methods, the accuracy is improved, but there is still a lack of partiality assumption in the application of this field.

[0004] The locality assumption is a priori knowledge from the model designer, which assumes that the correlation between elements in space exists in the relationship of near large and far small. The model should mainly focus on the local elements in the near place, and relatively ignore the weakly correlated elements in the far place. The locality assumption constrains the objective function of the model, so that the model learns in a smaller solution space, achieving higher learning efficiency. Existing research shows that in the task of ultra-short-term wind power prediction, there is a local high correlation between the output power of wind turbines in the same dominant wind direction. However, the MHA mechanism in the Transformer calculates the global attention between all elements, without modeling the relative position relationship between elements. This mechanism ignores the locality assumption, which has two effects on modeling. ① In the case of insufficient training data, the model is easily disturbed by noise and is difficult to learn the key information of the local high correlation of wind turbines in the near distance and the dominant wind direction, ultimately affecting the prediction effect. ② In the case of sufficient training data, locality becomes a factor limiting the effect of the model. On the one hand, the locality assumption itself can be considered as "indirect training data", which is the priori knowledge that researchers have summarized from the data in advance. The model learns more training data to compensate for the lack of priori knowledge. On the other hand, the locality assumption is not comprehensive, perfect and correct, but one-sided and overall correct. Learning the global correlation between elements will enable more accurate prediction results. For example, when the current leading VisionTransformer (ViT) model is pre-trained on a large-scale dataset and then fine-tuned on a task dataset, if the sample size used for pre-training exceeds 100M, ViT begins to show advantages over CNNs, and the larger the dataset size, the more obvious the advantages of ViT.

[0005] In summary, the MHA mechanism of the Transformer lacks the locality assumption, which makes the Transformer have stronger global relationship expression ability, but also makes it more difficult to train, and it is difficult to learn the key information of local high correlation on small-scale datasets, limiting the model's effectiveness. The solution is to introduce a locality structure into the model or use a pre-trained model on a large-scale dataset as a backbone network for fine-tuning. In the field of computer vision, multiple Transformer-related tasks have shown that introducing a locality structure can improve the performance and training stability of the Transformer. For example, ViTAE method integrates multiple scale dilated convolution layers into the Transformer, which performs well on small-scale datasets. ConViT proposes a soft convolution module that allows the model to dynamically adjust the weight of the locality structure in different sizes of datasets or at different stages of training. In addition, the pre-training-fine-tuning method has also been verified to be effective, such as ViT, which begins to show advantages over CNNs when the pre-training sample size exceeds 100M. SUMMARY

[0006] The application aims to overcome the deficiencies of the prior art, and provides a Transformer ultra-short-term wind power prediction model improved by introducing a partiality assumption, which combines a feature extraction module based on spatiotemporal locality and a prediction module based on global attention, and solves the problem that MHA is difficult to learn local key information on a small-scale data set.

[0007] The application solves the technical problem by the following technical scheme:

[0008] A Transformer ultra-short-term wind power prediction model improved by introducing a partiality assumption, characterized by comprising a PlicLSTM spatiotemporal feature extraction module and a power prediction module based on MHA, wherein the PlicLSTM spatiotemporal feature extraction module extracts local key information according to wind direction and distance factors, and the power prediction module based on MHA focuses on global correlation information between all sites.

[0009] Moreover, the PlicLSTM spatiotemporal feature extraction module introduces a spatiotemporal correlation assumption in the ultra-short-term WPP into the power prediction model, and focuses on local key information located on the same dominant wind direction when extracting spatiotemporal features; the PlicLSTM spatiotemporal feature extraction module adopts a convolution mode according to the wind direction path for feature fusion, and is an extension of the LSTM structure, and focuses on one-way transmission of time series information and information of adjacent time points when fusing time series information.

[0010] Moreover, the PlicLSTM spatiotemporal feature extraction module is a cyclic structure in the horizontal direction, and the PlicLSTM Cell processes time series data in a cyclic manner, and the input of a single time point in the time series input is denoted as X i , X i is a three-dimensional tensor, and the dimension is 5x h x w ; wherein 5 is the number of channels, corresponding to 5 data features of the conversion power, wind speed, temperature, pressure and air density of each wind turbine in the data set, h , w and H 0 , H 1 ,... H n are the number of wind turbines in the selected rectangular region in the vertical and horizontal directions respectively, and the PlicLSTM spatiotemporal feature extraction module is a stacked structure in the vertical direction, and the output of the previous layer

[0011] Moreover, the MHA-based power prediction module takes the local spatiotemporal features extracted by the PlicLSTM spatiotemporal feature extraction module as input, and uses the MHA module to further extract the global correlation between stations, establishing the relationship between local spatiotemporal features and output power among all stations.

[0012] The MHA-based power prediction module is part of the Transformer Encoder Layer structure, including a multi-head attention module that calculates the attention matrix between stations in the spatial dimension, residual connections, and a normalization module. The MHA-based power prediction module can be stacked with multiple layers, and the final output dimension is ( h × w , hidden_dim The resulting matrix corresponds to the region. h × w Each site; the power prediction module based on MHA is followed by a linear layer, which will... hidden_dim The dimension is reduced to 1, after softmax The output dimension after activation function is ( h × w The normalized prediction result of 1) is multiplied by the equivalent rated power of the corresponding station and then denormalized to obtain the final power prediction value.

[0013] Furthermore, the PlicLSTM spatiotemporal feature extraction module is a PlicLSTM with a recurrent structure. Each recurrent unit is called a PlicLSTM Cell, which is an extension of the LSTM Cell structure. It expands a one-dimensional vector into a fused feature matrix output by the PLICN module. This matrix is ​​a three-dimensional tensor, with the three dimensions being the number of channels in the feature matrix (Channel), the vertical dimension (Height), and the horizontal dimension (Width). Therefore, the input of PlicLSTM in model training is a five-dimensional tensor, where the first dimension is the training batch size (batch_size), the second dimension is the time series length (Seq_len), and the last three dimensions are the feature matrix dimensions (Channel×Height×Width).

[0014] remember t The input data of the PlicLSTM Cell at time t is X t The module's long-term memory is C t Short-time output is H t The PLICN module is denoted as FPlic(∙), and the forget gate is denoted as... f t The input gate is denoted as i t The output gate is denoted as ot , the PlicLSTM Cell is calculated as shown in equations (1) to (6);

[0015] wherein the symbol W , the symbol b The related variables are all network parameters to be learned, the symbol represents Hadamard product, that is, the corresponding position elements of the same specification matrix are multiplied, and σ(·) represents an activation function, in the experiment Sigmoid , tanh (·) represents tanh an activation function.

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] The PlicLSTM focuses on the key features of spatial and temporal locality in the ultra-short-term WPP task. The spatial feature extraction part PLICN strengthens the prior assumption of spatial locality compared with CNN, adopts the feature fusion mode according to the wind direction path convolution, ignores the data points far away or located in the non-dominant wind direction, and the time feature extraction part LSTM pays more attention to the sequence of time series and the key information of local high correlation of adjacent time points rather than global information compared with the MHA mechanism in the Transformer, and has higher robustness on small-scale data sets or in response to noise.

[0023] Moreover, the power prediction module based on MHA further extracts the global correlation between stations on the basis of the local spatiotemporal features extracted by the PlicLSTM spatiotemporal feature extraction module, establishes the relationship between the local spatiotemporal features and the output power among all stations, and the local spatiotemporal features output by the PlicLSTM H n hidden_dim, h, w need to be reshaped into a shape input of h w , hidden_dim first. The dimension reshaping is performed in a row-first manner, the prediction module adopts the MHA mechanism to calculate the global attention matrix between all stations, and the data after dimension reshaping is denoted as I , the calculation mode of this part is shown in equations (7) to (11); ​​

[0024] in, , , All of these are parameters to be trained. express The number of dimensions in the last dimension; Concat(∙) is a tensor join operation. n For the number of heads;

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] The output of the power prediction module is denoted as O The layer normalization operation is denoted as LayerNorm(∙), and the calculation method of the residual and normalized part is shown in Equation (12);

[0031]

[0032] O Dimensions are ( h ×w, hidden_dim The prediction module is followed by a linear layer to adjust the hidden layer dimensions. hidden_dim Reduced to 1, then softmax After activation function, output h × w The normalized predicted power of each station can be obtained by inverse normalization, which yields the predicted power value for the target time point.

[0033] The advantages and beneficial effects of this invention are as follows:

[0034] This invention combines the PlicLSTM feature extraction module, which assumes spatiotemporal locality, with a global attention prediction module. Ablation experiments and comparative experiments show that the PlicLSTM module can improve the robustness of the model on small-scale datasets, compensate for the shortcomings of MHA in learning locally highly correlated key information, and MHA can further improve the prediction accuracy based on PlicLSTM. The PlicLSTM-MHA model has higher prediction accuracy than comparative models such as Transformer in wind power prediction tasks within 1 hour. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the PlicLSTM-MHA model of the present invention;

[0036] Figure 2 PlicLSTM Cell loop structure diagram of the present application;

[0037] Figure 3 Row-major flattening diagram of the present application. DETAILED DESCRIPTION

[0038] The present application will be further described in the following specific examples, which are only descriptive and not limiting, and cannot limit the protection scope of the present application.

[0039] The technical solution of the present application is a Transformer ultra-short-term wind power prediction model improved by introducing a partial hypothesis, which is composed of two parts, namely a PlicLSTM spatio-temporal feature extraction part and a power prediction part based on MHA. The overall structure of the model is shown in Figure (1). The PlicLSTM spatio-temporal feature extraction part extracts local key information according to wind direction and distance factors, and the power prediction part based on MHA focuses on the global correlation information between all sites. Both of them focus on different aspects of wind power prediction problems and complement each other.

[0040] PlicLSTM introduces the spatio-temporal correlation hypothesis in ultra-short-term WPP into the wind power prediction model, and focuses on local key information located in the same dominant wind direction when extracting spatio-temporal features, which can solve the problem that the MHA mechanism of Transformer is difficult to learn local high correlation key information on small-scale data sets. In space, PlicLSTM uses PLICN to convolve according to the wind direction path, ignoring data points that are far away or located in non-dominant wind directions, which conforms to the spatial locality of the ultra-short-term WPP scenario. In time, PlicLSTM is an extension of the LSTM structure, focusing on one-way transmission of time series information and information of adjacent time points when fusing time series information.

[0041] In terms of model structure, PlicLSTM is a loop structure, and PlicLSTM Cell processes time series data in a loop. Let the input of a single time point in the time series input be X i , X i is a three-dimensional tensor with dimensions of 5× h × w . Among them, 5 is the number of channels, corresponding to 5 data features in the data set, including the converted power of each wind turbine, wind speed, temperature, pressure, and air density, h , w are the number of wind turbines in the selected rectangular area in the longitudinal and transverse directions, respectively. PlicLSTM is a stacked structure in the longitudinal direction, and the output of the previous layer ( H0 , H 1 , ..., H n This will be used as the input for the next layer of PlicLSTM.

[0042] The power prediction section uses the local spatiotemporal features extracted by PlicLSTM as input and further extracts the global correlation between stations using the MHA module, establishing the relationship between local spatiotemporal features and output power across all stations. The prediction module is part of the Transformer Encoder Layer structure, including a multi-head attention module that calculates the attention matrix between each station in the spatial dimension, residual connections, and a normalization module. Multiple layers can be stacked in the power prediction section, with the final output dimension being (…). h × w , hidden_dim The resulting matrix corresponds to the region. h × w Each site. A linear layer follows the prediction section, which will... hidden_dim The dimension is reduced to 1, after softmax The output dimension after activation function is ( h × w The normalized prediction result of 1) is multiplied by the equivalent rated power of the corresponding station and then denormalized to obtain the final power prediction value.

[0043] 1) Spatiotemporal feature extraction part of PlicLSTM

[0044] PlicLSTM is a recurrent structure, as shown in Figure (2). A single recurrent unit is called a PlicLSTM Cell, which is an extension of the LSTM Cell structure. In the LSTM model, the data at a single time step in the time series is represented by a one-dimensional vector. In PlicLSTM, this one-dimensional vector is extended into a fusion feature matrix output by the PLICN module. This matrix is ​​a three-dimensional tensor, with the three dimensions being the number of channels in the feature matrix (Channel), the vertical dimension (Height), and the horizontal dimension (Width). Therefore, the input of PlicLSTM in model training is a five-dimensional tensor, where the first dimension is the training batch size (batch_size), the second dimension is the time series length (Seq_len), and the last three dimensions are the feature matrix dimensions (Channel×Height×Width).

[0045] remember t The input data of the PlicLSTM Cell at time t is X t The module's long-term memory is C t Short-time output isH t , the PLICN module is denoted as FPlic(·), and the forget gate is denoted as f t , the input gate is denoted as i t , the output gate is denoted as o t The PlicLSTM Cell is calculated as shown in equations (1) to (6). Wherein, the symbol W , the symbol b The related variables are network parameters to be learned, the symbol represents Hadamard product, that is, the corresponding position elements of the same specification matrix are multiplied, and σ(·) represents an activation function, and in the experiment, the Sigmoid , tanh (·) represents tanh an activation function.

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] The PlicLSTM focuses on the key features of spatial and temporal locality in the ultra-short-term WPP task. Among them, the spatial feature extraction part PLICN strengthens the prior assumption of spatial locality compared with CNN, adopts the feature fusion mode according to the wind direction path convolution, ignores the data points far away or located in the non-dominant wind direction, and can realize efficient use of training samples on a small-scale data set. The time feature extraction part LSTM pays more attention to the sequence of time series and the key information of local high correlation between adjacent time points than the MHA mechanism in the Transformer, and does not pay attention to global information, which has higher robustness on a small-scale data set or when dealing with noise.

[0053] In addition, the PlicLSTM has the characteristics of one-way transmission of time series information, that is, the output of the model about t is only calculated by the input before t and t , and does not contain information after t . This feature conforms to the causality in the ultra-short-term WPP and has logical rationality.

[0054] 2) Power prediction part based on MHA

[0055] This part further extracts the global correlation between stations based on the local spatiotemporal features extracted by PlicLSTM, and establishes the relationship between local spatiotemporal features and output power among all stations.

[0056] As shown in Figure (1), the local spatiotemporal features output by PlicLSTM H n ( hidden_dim, h, w It needs to be reshaped into ( ) first. h × w , hidden_dim Shape input and dimension reshaping are performed in a row-first manner, and the flattening principle is shown in Figure (3). The prediction part uses the MHA mechanism to calculate the global attention matrix among all stations, and the dimension-reshaped data is denoted as I The calculation method for this part is shown in equations (7) to (11). , , All of these are parameters to be trained. express The number of dimensions in the last dimension; Concat(∙) is a tensor join operation. n The number of attention heads.

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] The output of the power prediction module is denoted as O The layer normalization operation is denoted as LayerNorm(∙), and the calculation method of the residual and normalized part is shown in Equation (12).

[0063]

[0064] O Dimensions are ( h ×w, hidden_dim The prediction module is followed by a linear layer to adjust the hidden layer dimensions. hidden_dim Reduced to 1, then softmax After activation function, output h × w The normalized predicted power of each station can be obtained by inverse normalization, which yields the predicted power value for the target time point.

[0065] While embodiments of the present application and the figures are disclosed for purposes of illustrating the present application, it is understood that various substitutions, modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application and the following claims, therefore, the scope of the present application is not limited to the disclosed embodiments and figures.

Claims

1. A Transformer ultra-short-term wind power prediction model improved by introducing a partial hypothesis, characterized in that: The PlicLSTM spatiotemporal feature extraction module extracts local key information according to wind direction and distance factors, and the power prediction module based on MHA pays attention to global correlation information between all stations. The PlicLSTM spatiotemporal feature extraction module introduces the spatiotemporal correlation assumption in the ultra-short-term WPP into the power prediction model, and pays attention to local key information located in the same dominant wind direction when extracting spatiotemporal features; the PlicLSTM spatiotemporal feature extraction module uses PLICN to perform feature fusion in the form of wind direction path convolution, and is an extension of the LSTM structure, and pays attention to one-way transmission of time series information and information of adjacent time points when fusing time series information. The PlicLSTM spatio-temporal feature extraction module is a transverse loop structure, and the PlicLSTM Cell processes the time series data in a loop, and the input of a single time in the time series input is X i , X i is a three-dimensional tensor, and the dimension is 5x h x w ; wherein 5 is the number of channels, corresponding to the five data characteristics of the conversion power, wind speed, temperature, pressure and air density of each wind turbine in the data set, h , w are the number of wind turbines in the selected rectangular area in the longitudinal and transverse directions respectively, and the PlicLSTM spatio-temporal feature extraction module is a superposition structure in the longitudinal direction, and the output of the previous layer H 0 , H 1 ,... H n will be used as the input of the next layer. The power prediction module based on MHA takes the local spatiotemporal features extracted by the PlicLSTM spatiotemporal feature extraction module as input, further extracts the global correlation between stations using the MHA module, and establishes the relationship between local spatiotemporal features and output power between all stations. The MHA-based power prediction module is part of the Transformer Encoder Layer structure, including a multi-head attention module that calculates the attention matrix between stations in the spatial dimension, residual connections, and a normalization module. The MHA-based power prediction module can be stacked with multiple layers, and the final output dimension is ( h × w , hidden_dim The resulting matrix corresponds to the region. h × w Each site; the power prediction module based on MHA is followed by a linear layer, which will... hidden_dim The dimension is reduced to 1, after softmax The output dimension after activation function is ( h × w The normalized prediction result of 1) is multiplied by the equivalent rated power of the corresponding station and then denormalized to obtain the final power prediction value; In the spatial feature extraction part, PLICN strengthens the prior assumption on spatial locality compared with CNN, and uses the method of convolution according to wind direction path to perform feature fusion, ignoring data points that are far away or located in non-dominant wind directions.

2. The Transformer ultra-short-term wind power prediction model improved by introducing the part hypothesis according to claim 1, wherein: The PlicLSTM spatiotemporal feature extraction module is a PlicLSTM with a loop structure, and a single loop unit is called a PlicLSTM Cell, which is an extension based on the LSTM Cell structure. The one-dimensional vector is extended to the fusion feature matrix output by the PLICN module. The matrix is a three-dimensional tensor, and the three dimensions are the channel number Channel of the feature matrix, the vertical dimension number Height of the feature matrix, and the horizontal dimension number Width. Therefore, the input of the PlicLSTM in model training is a five-dimensional tensor, wherein the first dimension is the training batch size batch_size, the second dimension is the time series length Seq_len, and the last three dimensions are the feature matrix dimensions Channel×Height×Width. Record t The input data of the PlicLSTM Cell at the moment is X t The long-term memory quantity of the module is C t The short-term output is H t The PlicN module is recorded as FPlic(·), the forget gate is recorded as f t The input gate is recorded as i t The output gate is recorded as o t The PlicLSTM Cell calculation mode is shown in formula (1) to formula (6). wherein the symbol W , the symbol b The relevant variables are all network parameters to be learned, the symbol represents Hadamard product, that is, the elements in the corresponding positions of the matrices of the same specification are multiplied, and σ(·) represents an activation function, which is Sigmoid , tanh σ(·) represents tanh an activation function. The PlicLSTM focuses on the spatiotemporal locality key features in the ultra-short-term WPP task. The time feature extraction part LSTM pays more attention to the sequence of time series information and the local high correlation key information of adjacent time points than the MHA mechanism in the Transformer, rather than global information, and has higher robustness on small-scale data sets or in the presence of noise.

3. The Transformer ultra-short-term wind power prediction model improved by introducing the part hypothesis according to claim 1, characterized in that: The MHA-based power prediction module further extracts the global correlation between stations on the basis of the local spatiotemporal features extracted by the PlicLSTM spatiotemporal feature extraction module, establishes the relationship between the local spatiotemporal features and the output power between all stations, and the local spatiotemporal features output by the PlicLSTM H n ( hidden_dim, h, w ) need to be first reshaped into a shape input of ( h x w , hidden_dim ), and the dimension reshaping is performed in a row-first manner. The prediction module adopts an MHA mechanism to calculate the global attention matrix between all stations. Let the data after dimension reshaping be I , and the calculation manner of this part is shown in formulas (7) to (11). wherein, , , are parameters to be trained, denotes the number of dimensions of the last dimension, Concat(·) is the tensor concatenation operation, n is the number of attention heads; The power prediction module outputs are denoted as O , and the layer normalization operation is denoted as LayerNorm(·). The residual and normalization part is calculated as shown in equation (12). O dimension is h ×w, hidden_dim ), the prediction module is followed by a linear layer that reduces the hidden layer dimension to 1, and after an activation function, the output is hidden_dim × softmax normalized prediction power for h × w sites, and after denormalization, the prediction power value for the target time point is obtained.

Images