A wind power prediction method based on a transformer and an improved mask matrix

Abstract

The application provides a wind power prediction method based on a Transformer and an improved mask matrix, which adds original data containing missing values to a position matrix, and then multiplies the position matrix by an improved mask matrix to construct active missing, so as to obtain time series data with position information and mask information, the time series data is used as an input matrix of an encoder in the Transformer to obtain reconstructed data, and a loss between an active missing part in the reconstructed data and an active missing part in the original data is calculated; according to the reconstructed data and the position matrix, missing values in the original data are interpolated, when the Transformer model converges, the result of interpolating the original data by using the reconstructed data is the final interpolation result; finally, a wind power function is predicted by using the data set obtained through interpolation. The application can accurately predict wind power in the case that the data set is incomplete.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wind power generation, and simultaneously belongs to the fields of machine learning and big data analysis, and particularly relates to a wind power prediction method based on a Transformer and an improved mask matrix. BACKGROUND

[0002] Wind power prediction is highly dependent on the integrity of data. However, due to some human factors such as data loss and deletion, and some irresistible factors such as sensor abnormalities and equipment aging, data loss will occur, so that the data set used by the downstream prediction task is mostly an incomplete data set. Generally, the processing method for missing data is direct deletion and missing interpolation. Direct deletion is simple and fast, but has great limitations, so in most cases, missing data is interpolated. There are many interpolation methods, including manual interpolation, statistical interpolation method, and the most popular machine learning interpolation method. Manual interpolation, as the original interpolation method, is time-consuming and labor-intensive. In the era of machine learning, machine learning interpolation methods can capture some hidden features through machine learning, thereby better interpolating data, saving manpower, shortening time, and improving interpolation accuracy.

[0003] However, the current machine learning interpolation methods such as deep belief networks and GANs predict missing data by capturing the features of existing data. If the overall missing part is too large, or the data used to predict the missing values is too small, the prediction accuracy of the missing data will be reduced. Therefore, in the prior art, it is difficult to accurately and effectively predict wind power when the data set is incomplete. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a wind power prediction method based on a Transformer and an improved mask matrix. The method realizes machine learning interpolation of wind power sequence based on a Transformer framework and an improved mask matrix, thereby accurately predicting wind power when the data set is incomplete.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0006] A wind power prediction method based on a Transformer and an improved mask matrix, comprising:

[0007] Step 1: Obtain raw data. The raw data consists of meteorological data and wind turbine data over a period of time. The raw data is represented as a three-dimensional matrix of time-meteorological data-wind turbine data. The period of time is divided into multiple time windows. Within each time window, a two-dimensional matrix of time and multi-dimensional features is formed. The multi-dimensional features include temperature, humidity, atmospheric pressure, and power data.

[0008] Step 2: Add the original data to the position matrix, and then multiply it by the improved mask matrix to construct active missing data, resulting in time series data (i.e., a new matrix) with position and mask information. This time series data serves as the input matrix for the encoder in the Transformer. The improved mask matrix is ​​labeled differently for the three types of data in the original data: those that were originally missing, those that were actively missing, and those that were not missing.

[0009] Step 3: Calculate the loss between the actively missing parts in the reconstructed data obtained by the Transformer and the actively missing parts in the original data using the loss function.

[0010] Step 4: Based on the reconstructed data and the position matrix, impute the missing values ​​in the original data, and calculate the loss between the imputed data and the original data using the interpolation formula;

[0011] Step 5: When the loss between the actively missing part in the reconstructed data and the true value of the actively missing part in the original data converges, that is, when the Transformer model converges, the result of interpolating the original data using the reconstructed data is the final interpolation result.

[0012] Step 6: Use the interpolated dataset to predict wind power function.

[0013] In one embodiment, step 2, the position matrix is ​​obtained by adding the time series fixed position coding matrix and the time delay coding matrix, where the time delay coding... When incorporated into location information, the time-delay coding matrix formula is as follows:

[0014]

[0015] In the formula, The values ​​in the mask matrix, For the tth i The time delay encoded value of the j-th feature at time step. For the tth i-1 The time delay encoding value of the j-th feature at time n, where N is the size of the sliding time window and D is the dimension of the feature.

[0016] In one embodiment, the improved mask matrix has three optional parameters: -x, 0 and x, where -x represents originally missing, 0 represents active missing, and x represents not missing, and the pattern of the improved mask matrix is as follows:

[0017] where M ∈ R N×D

[0018] where: M is an N × D matrix, is one of the three optional parameters of the mask matrix, when the data in the original data is originally missing, the value of the data corresponding position on the mask matrix is -x, when the data in the original data is active missing, the value of the data corresponding position on the mask matrix is -0, and when the data in the original data is not missing, the value of the data corresponding position on the mask matrix is x.

[0019] In one embodiment, the step 3, the loss calculation formula is as follows:

[0020] L = X · (tanh(M))' - De(z) · (tanh(M))'

[0021] where L is the loss value between the active missing part of the reconstructed data and the active missing part of the original data, X is the original data, De() is the decoder in the Transformer, z is the low-dimensional vector after the encoder, which is the data feature captured by the Transformer encoder, and De(z) is the reconstructed data obtained by the Transformer.

[0022] In one embodiment, the step 4, the position of the missing value in the original data is obtained by the mask matrix and the position matrix, and then the same position data in the reconstructed data is used to fill the original data.

[0023] In one embodiment, the step 4, the interpolation formula is:

[0024] L inputed = X · RELU(tanh(M)) + De(z) · (1- RELU(tanh(M)))

[0025] where L inputed is the loss between the interpolated data and the original data, which uses the Relu function and the tanh function.

[0026] ​​​Compared with the prior art, the application combines the Transformer overall framework and the characteristics of the wind power sequence, uses position coding and an improved mask matrix to reconstruct the missing data. Through the reconstruction of the missing data, the missing part in the original data is filled, the integrity of the data set is ensured, the situation that the prediction accuracy is reduced due to the incomplete data set is avoided, and the accuracy of the wind power prediction under the incomplete data set is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The overall flowchart of the application.

[0028] Figure 2 The results of the comparative test of the application on the IMWP data set with a missing rate of 30%. DETAILED DESCRIPTION

[0029] The embodiments of the application will be described in detail below with reference to the accompanying drawings and examples.

[0030] In order to combine the mainstream machine learning imputation method with the characteristics of the wind power sequence, the purpose of the application is to provide a wind power prediction method based on Transformer and an improved mask matrix. The method realizes data sequence imputation based on the Transformer framework and the improved mask matrix. The original wind power data with missing values is processed using the position matrix and the improved mask matrix. The processed data with mask information and position information is used as the input of the encoder in the Transformer. The data is reconstructed using the Transformer model. According to the data features captured by the encoder, the decoder outputs a reconstructed matrix without missing values. The model is optimized according to the improved loss calculation method. After optimization, the original data is imputed using the reconstructed data according to the improved mask matrix and the position matrix.

[0031] Specifically, as shown in Figure 1 The method of the application includes the following steps:

[0032] Step 1, obtaining the original data.

[0033] In the application, the original data is meteorological data and wind turbine data within a period of time, such as temperature, humidity, atmospheric pressure, power data, etc. Since a time window is used, the original data is a three-dimensional matrix of time-meteorological data-wind turbine data. If a period of time is divided into multiple time windows, a two-dimensional matrix of time and multiple features is formed in each time window. The multiple features include temperature, humidity, atmospheric pressure, and power data. In these data, due to human factors such as data loss, or some irresistible factors such as equipment aging, the data has missing values.

[0034] Step 2: Introduce location information and mask information into the raw data.

[0035] First, the original data with missing values ​​is added to the position matrix for position marking. Then, it is multiplied with the improved mask matrix to construct active missing values ​​and mark the missing data information, thus obtaining time series data with position and mask information, i.e., a new matrix. This time series data is used as the input matrix of the encoder in the Transformer model, and the decoder obtains the reconstructed time series.

[0036] For position embedding, since this invention uses the traditional Transformer model as the base model, it does not include recursion or convolution. To enable the model to learn temporal information, a fixed version of position embedding is used, embedding position information in the encoder. Simultaneously, to help the model better learn the hidden information of the time series, delay encoding is added to the position information. The formula for the time delay coding matrix is ​​shown below:

[0037]

[0038] In the formula, The values ​​in the mask matrix, For the tth i The time delay encoded value of the j-th feature at time step. For the tth i-1 The time delay encoding value of the j-th feature is given by time, N is the size of the sliding time window (which can be set to 1 week or 1 day), and D is the dimension of the feature.

[0039] Therefore, the position encoding of this invention is obtained by adding a time-series fixed position encoding and a time-delay encoding; that is, the position matrix is ​​obtained by adding a time-series fixed position encoding matrix and a time-delay encoding matrix. The value of the time-delay encoding is determined based on the mask matrix of the original data. Then, the position encoding is added to the original data to obtain the original data containing position information.

[0040] In the original data of this invention, "originally missing" refers to data that was originally missing at the corresponding position, meaning that there was no data at that position. "Not missing" means that data was originally present at the corresponding position. To achieve the purpose of this invention, "active missing" is also introduced. Active missing refers to artificially creating a missing value in the input matrix so that the original position contains data, in order to calculate the loss during model training.

[0041] For the originally missing data in the original data, the loss between the reconstructed data after the transformation and the originally missing data cannot be calculated, that is, the final reconstruction effect of the "missing data" cannot be compared because the true value is missing. Therefore, the original data is constructed to simulate the originally missing data in the original data. Since the missing data constructed actively has a true value, the loss between the reconstructed data and the missing data can be calculated to measure the training effect of the model.

[0042] Specifically, in the improved mask matrix, the labels of the three types of data, originally missing, actively missing and not missing, in the original data are different. Therefore, when multiplying the original data by the improved mask matrix, the active missing can be constructed.

[0043] For example, the improved mask matrix has three optional parameters: -x, 0 and x. Among them, -x represents originally missing, 0 represents actively missing, and x represents existing, and the default type is float32. The purpose here is to construct an input matrix, so that the value at the position of the originally missing data cannot be 0, the value at the position of the actively missing data must be 0, and the value at the position of the not missing data cannot be 0. Therefore, in the mask matrix, -x represents originally missing, because the original data at this time is the original data after position embedding, in which the value at the position of the originally missing data is not nan or 0, so after multiplication by -x, the value is also not 0; the value at the position of the actively missing data must be 0, so 0 represents actively missing, so after multiplication by 0, the value is 0; the value at the position of the not missing data cannot be 0, so after multiplication by x, the value is also not 0. For example, if you want to actively construct 30% missing, then record the position of the originally existing data with a quantity size of 30% of the total amount of data on the existing data, and then set the value of the corresponding position in the mask matrix to 0.

[0044] The style of the improved mask matrix M is as follows:

[0045] Where M ∈ R N×D

[0046] In the formula, M is an N × D matrix, -x is one of the three optional parameters of the mask matrix, when the data in the original data is originally missing, the value at the position corresponding to the data is -x, when the data in the original data is actively missing, the value at the position corresponding to the data is -0, when the data in the original data is not missing, the value at the position corresponding to the data is x.

[0047] ​Wherein, the value of x can be freely selected, but the selected value is preferably subjected to a tanh activation function and exceeds the default float32, and after -x, 0, x is subjected to relu, tanh and tanh derivation, it will tend to -1 and 1; j represents the number of features, for example, 0 represents the first feature, and j represents the j-1th feature; t0 represents the first set of multivariate data in the time window, t i represents the i-1th set of multivariate data in the time window. Specifically, the mask matrix only uses the derivative of the tanh function when calculating the reconstruction loss, and the derivative of the tanh function is 1 at 0 and approximately 0 on both sides. When calculating the interpolation loss, the mask matrix needs to use the tanh activation function and the relu activation function in turn.

[0048] The improved mask matrix will obtain a new matrix with mask information and position information after being multiplied by the original data with position information, which is the input matrix of the Transformer encoder. Since the mask matrix has different labels for the three types of data in the original data, namely, originally missing data, actively missing data and non-missing data, in the input matrix, the data at the originally missing position is the sum of the original data and the value of the corresponding position matrix multiplied by -x, the data at the actively missing position is 0, and the data at the non-missing position is the sum of the original data and the value of the corresponding position matrix multiplied by x.

[0049] Step 3, calculate the reconstruction loss.

[0050] Through the loss function, the loss between the actively missing part of the reconstruction data obtained by the Transformer and the actively missing part of the original data is calculated to improve the performance of the model.

[0051] By using the improved mask matrix, two types of missing data can be labeled, so when calculating the reconstruction loss, only the reconstruction loss between the actively constructed missing data can be calculated, and the loss calculation formula is as follows:

[0052] L=X·(tanh(M))'-De(z)·(tanh(M))

[0053] Wherein, L is the loss value between the actively missing part of the reconstruction data and the actively missing part of the original data, X is the original data, De() is the decoder in the Transformer, z is the low-dimensional vector after the encoder, which is the data feature captured by the Transformer encoder, and De(z) is the reconstruction data obtained by the Transformer. Since the derivative of the tanh() function is used as the activation function, the value in the matrix M in the improved mask matrix will be directly equal to 1 when it exceeds 100.

[0054] Step 4, data interpolation.

[0055] According to the reconstructed data and the position matrix, the missing values in the original data are interpolated, and the loss between the interpolated data and the original data is calculated by the interpolation formula.

[0056] Since the data originally missing in the original data has been marked in the improved mask matrix, the position of the originally missing data is known, and thus, through the mask matrix and the position matrix, the specific position of the missing values in the original data can be obtained. Then, the original data is filled using the data at the same position in the reconstructed data.

[0057] The interpolation formula in this step is:

[0058] L inputed = X RELU (tanh (M)) + De (z) (1 RELU (tanh (M)))

[0059] Wherein, L inputed is the loss between the interpolated data and the original data, which uses the Relu function and the tanh function.

[0060] Step 5, when the loss between the actively missing part of the reconstructed data and the true value of the actively missing part of the original data converges, i.e. the Transformer model converges, the result of using the reconstructed data to interpolate the original data is the final interpolation result.

[0061] Here, the actively missing part of the reconstructed data refers to the predicted data at the position of the actively missing data. In the input matrix, there are actively missing, originally missing and non-missing data, which are generated after passing through the Transformer encoder and decoder. The actively missing data in the input matrix also passes through the Transformer encoder and decoder to obtain the reconstruction and obtain the predicted data.

[0062] The true value of the actively missing part of the original data refers to the data of the actively missing part of the original data. Because it is actively missing, these data actually exist, but in order to construct the missing, the value of the mask matrix corresponding to the position is set to 0, which is equivalent to missing. Here, the true value refers to the originally data of the actively missing part of the original data.

[0063] Step 6, using the data set obtained by interpolation to perform wind power function prediction.

[0064] In a specific embodiment of the present invention, the hardware is a computer, configured with the following hardware environment: CPU: 2 Intel Xeon 6130 processors (2.1GHz / 16c) / 2666MHz / 10.4GT; GPU: 6 x 16GB TESLA-P100 4096b P CAC; Memory: 16 x 32GB ECC Registered DDR4 2666; Software environment: Operating system: Ubuntu 16.04; Deep learning frameworks: Tensorflow, pandas, scikit-learn; Languages ​​and development environment: Python 3.6, Anaconda 3, PyCharm 2020.

[0065] This example uses the IMWP data from a wind farm in Inner Mongolia in 2019 as the analysis object. The IMWP dataset is divided into two parts: meteorological data (NWP) and power data, totaling 14,980 records. The meteorological data (NWP) includes date, time, floor height, wind speed, wind direction, humidity, temperature, air pressure, and air density data, some of which are shown in Table 1. The power data includes date, time, wind farm number, collector number, short-term raw power, short-term predicted power, etc., some of which are shown in Table 2. This data was acquired by the Wind Farm Operation Monitoring and Data Acquisition System (SCADA). Due to harsh environmental conditions, SCADA system sensor malfunctions, and communication failures, there are certain anomalies and missing data, with nearly 4,300 records (30%) missing. In this example, the proposed method, VAE-GAN, GAN, KNN, and MEAN are used for imputation on the IMWP dataset with a 30% missing rate. Then, a random forest regression model and an LSTM neural network model are used to predict future wind power data. The results are shown in Table 2. Figure 2 As shown. (Through) Figure 2 It can be seen that the dataset interpolated by the method proposed in this invention will provide a dataset that is closer to real data for subsequent downstream tasks, and will also improve the performance of downstream models.

[0066] While the present invention has been described through preferred embodiments, it is not limited to the embodiments described herein, and various changes and modifications are made without departing from the scope of the invention.

[0067] Table 1 Historical Weather Data in IMWP

[0068]

[0069] Table 2 Historical power data in IMWP

[0070]

Claims

1. A wind power prediction method based on Transformer and improved mask matrix, characterized in that, include: Step 1: Obtain raw data. The raw data consists of meteorological data and wind turbine data over a period of time. The raw data is represented as a three-dimensional matrix of time-meteorological data-wind turbine data. The period of time is divided into multiple time windows. Within each time window, a two-dimensional matrix of time and multi-dimensional features is formed. The multi-dimensional features include temperature, humidity, atmospheric pressure, and power data. Step 2: Add the original data to the position matrix, and then multiply it by the improved mask matrix to construct active missing data, resulting in time series data (i.e., a new matrix) with position and mask information. This time series data serves as the input matrix for the encoder in the Transformer. The improved mask matrix is ​​labeled differently for the three types of data in the original data: those that were originally missing, those that were actively missing, and those that were not missing. Step 3: Calculate the loss between the actively missing parts in the reconstructed data obtained by the Transformer and the actively missing parts in the original data using the loss function. Step 4: Based on the reconstructed data and the position matrix, impute the missing values ​​in the original data, and calculate the loss between the imputed data and the original data using the interpolation formula; Step 5: When the loss between the actively missing part in the reconstructed data and the true value of the actively missing part in the original data converges, that is, when the Transformer model converges, the result of interpolating the original data using the reconstructed data is the final interpolation result. Step 6: Use the interpolated dataset to predict wind power function.

2. The wind power prediction method based on Transformer and improved mask matrix according to claim 1, characterized in that, In step 2, the position matrix is ​​obtained by adding the time series fixed position coding matrix and the time delay coding matrix, where the time delay coding... When incorporated into location information, the time-delay coding matrix formula is as follows: In the formula, The values ​​in the mask matrix, For the tth i The time delay encoded value of the j-th feature at time step. For the tth i-1 The time delay encoding value of the j-th feature at time n, where N is the size of the sliding time window and D is the dimension of the feature.

3. The wind power prediction method based on Transformer and improved mask matrix according to claim 1, characterized in that, The improved mask matrix has three optional parameters: -x, 0, and x, where -x indicates that the mask was originally missing, 0 indicates that it was intentionally missing, and x indicates that it was not missing. The improved mask matrix has the following format: in Where M is an N×D matrix, It is one of the three optional parameters of the mask matrix. When the data in the original data is missing, the corresponding position of that data is displayed. A value of -x indicates that when data in the original dataset is actively missing, the corresponding position of that data is missing. A value of -0 indicates that the data at the corresponding position in the original data is not missing. The value is x.

4. The wind power prediction method based on Transformer and improved mask matrix according to claim 1, characterized in that, In step 3, the loss calculation formula is as follows: L=X·(tanh(M))′-De(z)·(tanh(M))′ Where L is the loss value between the actively missing part in the reconstructed data and the actively missing part in the original data, X is the original data, De() is the decoder in Transformer, z is the low-dimensional vector after the encoder, which is the data feature captured by the Transformer encoder, and De(z) is the reconstructed data obtained after Transformer.

5. The wind power prediction method based on Transformer and improved mask matrix according to claim 1, characterized in that, In step 4, the positions of missing values ​​in the original data are obtained through the mask matrix and the position matrix, and then the data at the same positions in the reconstructed data are used to fill in the missing values ​​in the original data.

6. The wind power prediction method based on Transformer and improved mask matrix according to claim 1, characterized in that, In step 4, the interpolation formula is: L inputed =X·RELU(tanh(M))+De(z)·(1-RELU(tanh(M))) Among them, L inputed To account for the loss between the interpolated data and the original data, the ReLU function and the tanh function were used.

Images