Long-term wind power prediction method based on improved Bert

Abstract

The invention discloses a long-term wind power prediction method based on improved Bert. The method mainly comprises the following steps of: firstly, preprocessing an abnormal value and time information of fan information in the data concentration; thirdly, processing the data by using a full connection layer and a convolutional layer to obtain a high-dimensional representation of fan information with time sequence information and time information; a full connection layer is used for processing the multi-head self-attention mechanism, and a shared query key matrix and a value matrix needed by the multi-head self-attention mechanism are obtained respectively. Then, a random rotation method is used to map the data matrix into different hash buckets, and a multi-head self-attention mechanism is used in each hash bucket. And finally, obtaining a final result by using residual connection and layer normalization processing. According to the improved Bert model provided by the invention, the prediction accuracy can be improved while the time complexity is reduced to O (nlogn).

Description

Technical Field

[0001] The present invention belongs to the field of renewable energy and deep learning, and specifically relates to a long-term wind power prediction method based on improved BERT. Background Art

[0002] Wind power generation is a renewable energy source that helps reduce dependence on fossil fuels and promote sustainable development. However, wind power generation is greatly affected by weather and seasons, resulting in unstable power generation. Therefore, the prediction of wind power has very important application value. Wind power forecasting can be divided into ultra-short-term (within 30 minutes), short-term (30 minutes-6 hours), medium-term (6 hours-1 day) and long-term (more than 1 day) according to the forecast time range. Long-term wind power forecasting can effectively promote the integration of renewable energy such as wind energy, ensure that it plays a greater role in the energy system, and promote energy transformation. For power dispatching, long-term wind power forecasting provides important supporting information for power dispatching plans.

[0003] Recently, deep learning has been widely used in time series prediction tasks, among which Transformer-based models benefit from the self-attention mechanism and perform well in long time series prediction tasks. The self-attention mechanism can better pay attention to the more correlated parts of the input sequence, thereby effectively capturing long-distance dependencies.

[0004] However, the time complexity of the original self-attention mechanism module is very high, with a time complexity of O(n 2 ). The improved Bert model proposed in the present invention can reduce the time complexity to O(n log n) and improve the accuracy of prediction. Summary of the invention

[0005] Aiming at the problem of long-term wind power prediction, the present invention provides a long-term wind power prediction method based on improved BERT. Traditional Transformer-based models are composed of two parts: the Encoder layer and the Decoder layer, while this model only has the Encoder layer, which makes the model have a small number of parameters. In addition, this model uses the self-attention mechanism improved by the local sensitive hashing technology based on Angular, which reduces the time complexity of the original self-attention mechanism from O(n 2 ) is reduced to O(n log n), which reduces the computational cost and improves the accuracy of wind power prediction.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A long-term wind power prediction method based on improved Bert includes the following steps:

[0008] Step 1) The missing values ​​in the data set are filled using the backward filling method, the processed data are normalized, and the timestamps of the data are layered.

[0009] Step 2), input the preprocessed information of each wind turbine into a convolution layer and perform absolute position encoding. At the same time, input the timestamp after layered processing into a fully connected layer. Then, the information matrix of each wind turbine processed by the convolution layer and absolutely position encoded is added element by element to the timestamp matrix of each wind turbine processed by the fully connected layer, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism.

[0010] In step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers. Then, the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket number using the mapping relationship, and sorted in chronological order within the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket.

[0011] Step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model.

[0012] As a further optimization scheme of the long-term wind power prediction method based on the improved Bert of the present invention, it is characterized in that, in the step 1), the missing values ​​in the data set are filled by a backward filling method, and the processed data are normalized, and then the timestamps of the data are layered:

[0013] First, the abnormal values ​​of some features of the wind turbine information in the data set are set as thresholds. For example, if the wind turbine power value is less than 0, it is set to 0; if the angle values ​​of the three blades of the wind turbine are greater than 89, it is set to 89. Then, the missing values ​​in the data set are filled using the backward filling method, that is, the data at the moment after the missing data is used for filling.

[0014] Then, the mean and standard deviation of all features of the wind turbine information in the data set are calculated, and the data are normalized using the mean and standard deviation. The normalization formula is as follows:

[0015]

[0016] Among them, data represents the value of a specific feature, mean represents the mean of a specific feature, and std represents the standard deviation of a specific feature.

[0017] Finally, the timestamp features in the data set are layered. The timestamp is divided into three layers: hours, minutes, and seconds, and the hours, minutes, and seconds of the timestamp are scaled respectively. The scaling formula is as follows:

[0018]

[0019] Among them, Stamp h 、Stamp m 、Stamp s Respectively represent the hours, minutes, and seconds values ​​in the timestamp.

[0020] As a further optimization scheme of the long-term wind power prediction method based on improved Bert of the present invention, it is characterized in that, in the step 2), the pre-processed information of each wind turbine is input into a convolution layer and absolutely position encoded. At the same time, the timestamp after layered processing is input into a fully connected layer. Then, the information matrix of each wind turbine processed by the convolution layer and absolutely position encoded is added element by element to the timestamp matrix of each wind turbine processed by the fully connected layer, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information, and the high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism respectively:

[0021] First, the preprocessed information of the angles of the three blades of each wind turbine, the ambient temperature, the generated power, etc. is input into a one-dimensional convolution filter with a convolution kernel size of 3, a stride of 1, a padding of 1, and no bias. The original data in the data set is projected onto the preset dimension to obtain the embedded representation of the wind turbine information. The formula is as follows:

[0022] u=Conv1d(X)

[0023] in, represents the embedded representation of the obtained wind turbine information, Represents the preprocessed wind turbine information data. s, p, d, and f represent the length of the input sequence, the length of the predicted sequence, the embedding dimension, and the number of features of the sequence, respectively. Conv1d() represents a one-dimensional convolution operation.

[0024] Next, the absolute position embedding method is used to retain the timing information of the original data. Specifically, the sin function is used to process the value of the specific feature of the even index position of the fan information data, and the cos function is used to process the value of the feature of the odd index specific position of the fan information data to obtain the timing information. The absolute position embedding formula is as follows:

[0025]

[0026] in, Indicates the timing information, pos indicates the index of the specific time position of the fan information, The position index representing the specific features of the fan information data at each moment. (For each fan, a batch of fan information is a two-dimensional matrix, each row represents the fan information at each moment, and its columns represent the features contained in the fan information data).

[0027] At the same time, the stratified timestamp data is input into a fully connected layer without bias to obtain the time information. The formula is as follows:

[0028] SE=Linear(X mark )

[0029] in, Indicates time information. represents the timestamp data after stratification, m represents the total number of timestamp types (the value in this embodiment is 3, representing hours, minutes, and seconds respectively), and Linear() represents a fully connected layer.

[0030] Then, the data matrices u, PE, and SE obtained in the above steps are added element by element to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The calculation formula is as follows:

[0031]

[0032] in, represents the high-dimensional representation of wind turbine information with timing information and time information, α represents a custom hyperparameter (when the data is normalized, α is set to 1), and + represents an element-by-element addition operation.

[0033] Finally, the high-dimensional wind turbine information Input into a fully connected layer without bias to obtain the shared query key matrix required by the multi-head self-attention mechanism. The calculation formula is as follows:

[0034]

[0035] in, Represents the shared query key matrix required for the multi-head self-attention mechanism.

[0036] At the same time, the high-dimensional wind turbine information Input into another fully connected layer without bias to obtain the value matrix required by the multi-head self-attention mechanism. The formula is defined as follows:

[0037]

[0038] in, A matrix of values ​​required to represent a multi-head self-attention mechanism.

[0039] As a further optimization scheme of the long-term wind power prediction method based on the improved BERT of the present invention, it is characterized in that in the step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers, and then the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket number by using the mapping relationship, and sorted in time sequence in the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket:

[0040] First, the obtained shared query key matrix QK and value matrix V are split into multiple heads of the same size so that the multi-head self-attention mechanism can be applied later. The calculation formula is as follows:

[0041] QK after =Split(QK)

[0042] V after =Split(V)

[0043] in, represents the shared query key matrix after splitting, Represents the value matrix after splitting, n head Represents the number of heads of the multi-head self-attention mechanism, and Split() represents the split operation.

[0044] In order to describe the implementation process more concisely, the following steps S1 to S7 are defined:

[0045] S1. Split the shared query key matrix QK after Multiply it with the randomly generated normal distribution matrix to achieve the purpose of random rotation. The calculation formula is as follows:

[0046] QK new =QK after ×R

[0047] in, represents the query key matrix after random rotation, Represents a matrix of randomly generated normal distribution, l bucket Indicates the size of each hash bucket.

[0048] S2. QK obtained after random rotation new The elements in are reversed one by one, and then combined with QK new The final matrix is ​​obtained by splicing, and the calculation formula is as follows:

[0049] QK final =Reverse(QK new )+QK new

[0050] in, It represents the final processed shared query key matrix, Reverse() represents the inversion operation, and + represents the concatenation operation.

[0051] S3. Get the shared query key matrix QK final The maximum value of each row is used. The index corresponding to the maximum value is used as the hash mapping result corresponding to each sequence data, and then added to the preset offset to obtain the hash bucket number corresponding to this sequence data. The calculation formula is as follows:

[0052] buckets=max(QK final )+i*n bucket

[0053] in, Represents the hash bucket number matrix corresponding to each sequence data, i∈{0,...,n hash -1} represents the hash round number, n hash Indicates the total number of hash rounds, n bucket Represents the number of hash buckets, and max() is used to obtain the index corresponding to the maximum value in each row of the matrix.

[0054] S4. Query key matrix QK after Sum value matrix V after In the first dimension, the hash bucket numbers are sorted from small to large, and then the hash buckets with the same number are sorted in chronological order. The calculation formula is as follows:

[0055] QK sort =Sort time (Sort num (QK afer ))

[0056] V sort =Sort time (Sort num (V after ))

[0057] in, represents the sorted shared query key matrix, Represents the sorted value matrix, Sort time () indicates sorting in chronological order. num () means sorting by hash bucket number.

[0058] S5. Share the query key matrix QK sort Copy it into two matrices, namely Q sort and K sort , as the query matrix and key matrix respectively. For the key matrix Ksort Sum value matrix V sort , so that it contains not only the data in the current hash bucket, but also the data in the previous hash bucket. Then, the scaled dot product function is used to calculate the attention score, and then the calculated attention score is converted into a probability value through the softmax function, and multiplied with the value matrix to get the output. The calculation formula is as follows:

[0059]

[0060] in, Represents the value obtained by the i-th hash bucket, represents the query matrix required to calculate the i-th hash bucket, represents the key matrix required to calculate the i-th hash bucket, Represents the value matrix required to calculate the i-th hash bucket, and softmax() represents the softmax operation.

[0061] S6. Calculate the value z obtained from all hash buckets i Splice to get the complete output sequence. The calculation formula is as follows:

[0062] Z = concat(z i )

[0063] Where i∈{0,...,n hash -1} indicates the hash round number, Represents the complete sequence output, and concat() represents the concatenation operation.

[0064] S7. Use Sort time (), Sort num () The sorting information provided during sorting restores the complete sequence output to the original time sequence order. The calculation formula is as follows:

[0065] Z sort =ReSort(Z)

[0066] in, Represents the output after the order is restored. ReSort() means restoring the temporal order according to the provided sorting information.

[0067] Repeat the above steps S1 to S7, using different random rotations each time, and finally perform weighted summation on the outputs obtained from each calculation. The calculation formula is as follows:

[0068]

[0069] in, Represents the result of weighted average of all hash rounds, represents the output matrix of the kth round of hashing, w k Represents the output matrix of the kth round of hashing The weight of .

[0070] Finally, concatenate each head in the multi-head self-attention mechanism to obtain the output Z of the multi-head self-attention mechanism final , the calculation formula is as follows:

[0071] z final =concat(Z sum )

[0072] in, Represents the final output of the multi-head self-attention mechanism.

[0073] 1. As a further optimization scheme of the long-term wind power prediction method based on the improved BERT of the present invention, it is characterized in that in the step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model:

[0074] First, the final output result Z of the multi-head self-attention mechanism final High-dimensional representation of wind turbine information Perform residual connection and layer normalization, then input into two convolutional layers, perform layer normalization, and finally input into a fully connected layer containing bias to obtain the final output of the model. The calculation formula is as follows:

[0075]

[0076] out2=Con1v(Con1v(LayerNorm(out1)))

[0077] out3 = LayerNorm (out1 + out2)

[0078] out=Linear(out3)

[0079] in, Represents the result after the first residual connection, represents the result after layer normalization and two one-dimensional convolutional layers. Represents the result after residual connection and layer normalization, Indicates the final output of the model. Con1v() indicates a one-dimensional convolution operation, and LayerNorm() indicates a layer normalization operation.

[0080] Transpose the final result out and intercept the data of the last p moments to get the predicted value. The formula is defined as follows:

[0081]

[0082] in, Represents the final model prediction result, and Cut() means intercepting the data of the last p moments.

[0083] In the process of training the model, the Adam optimizer is used to minimize the MSELoss loss function and learn the model parameters. The MSELoss loss function formula is defined as follows:

[0084]

[0085] Among them, truth n Represents the true value at the nth moment, res n Represents the prediction at the nth moment. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is an overall implementation flow chart of the present invention.

[0087] Figure 2 It is a diagram representing input data in an embodiment of the present invention.

[0088] Figure 3 It is a model structure diagram of the improved Bert model in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are not intended to limit the protection scope of the present invention:

[0090] like Figure 1 As shown, the present invention discloses a long-term wind power prediction method based on improved Bert, comprising the following steps:

[0091] Step 1) The missing values ​​in the data set are filled using the backward filling method, the processed data are normalized, and the timestamps of the data are layered.

[0092] Step 2), input the preprocessed information of each wind turbine into a convolution layer and perform absolute position encoding. At the same time, input the timestamp after layered processing into a fully connected layer. Then, the information matrix of each wind turbine processed by the convolution layer and absolutely position encoded is added element by element to the timestamp matrix of each wind turbine processed by the fully connected layer, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism.

[0093] In step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers. Then, the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket number using the mapping relationship, and sorted in chronological order within the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket.

[0094] Step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model.

[0095] In step 1), the missing values ​​in the data set are filled using a backward filling method, and the processed data are normalized, and then the timestamps of the data are layered:

[0096] First, the abnormal values ​​of some features of the wind turbine information in the data set are set as thresholds. For example, if the wind turbine power value is less than 0, it is set to 0; if the angle values ​​of the three blades of the wind turbine are greater than 89, it is set to 89. Then, the missing values ​​in the data set are filled using the backward filling method, that is, the data at the moment after the missing data is used for filling.

[0097] Then, the mean and standard deviation of all features of the wind turbine information in the data set are calculated, and the data are normalized using the mean and standard deviation. The normalization formula is as follows:

[0098]

[0099] Among them, data represents the value of a specific feature, mean represents the mean of a specific feature, and std represents the standard deviation of a specific feature.

[0100] Finally, the timestamp features in the data set are layered. The timestamp is divided into three layers: hours, minutes, and seconds, and the hours, minutes, and seconds of the timestamp are scaled respectively. The scaling formula is as follows:

[0101]

[0102] Among them, Stamp h 、Stamp m 、Stamp s Respectively represent the hours, minutes, and seconds values ​​in the timestamp.

[0103] In the step 2), if Figure 2 As shown in the figure, the pre-processed information of each wind turbine is input into a convolutional layer and absolutely position encoded. At the same time, the timestamp after layered processing is input into a fully connected layer. Then, the information matrix of each wind turbine processed by the convolutional layer and absolutely position encoded is added element by element to the timestamp matrix of each wind turbine processed by the fully connected layer, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism:

[0104] First, the preprocessed information of the angles of the three blades of each wind turbine, the ambient temperature, the generated power, etc. is input into a one-dimensional convolution filter with a convolution kernel size of 3, a stride of 1, a padding of 1, and no bias. The original data in the data set is projected onto the preset dimension to obtain the embedded representation of the wind turbine information. The formula is as follows:

[0105] u=Conv1d(X)

[0106] in, represents the embedded representation of the obtained wind turbine information, Represents the preprocessed wind turbine information data. s, p, d, and f represent the length of the input sequence, the length of the predicted sequence, the embedding dimension, and the number of features of the sequence, respectively. Conv1d() represents a one-dimensional convolution operation.

[0107] Next, the absolute position embedding method is used to retain the timing information of the original data. Specifically, the sin function is used to process the value of the specific feature of the even index position of the fan information data, and the cos function is used to process the value of the feature of the odd index specific position of the fan information data to obtain the timing information. The absolute position embedding formula is as follows:

[0108]

[0109] in, Indicates the timing information, pos indicates the index of the specific time position of the fan information, The position index representing the specific features of the fan information data at each moment. (For each fan, a batch of fan information is a two-dimensional matrix, each row represents the fan information at each moment, and its columns represent the features contained in the fan information data).

[0110] At the same time, the stratified timestamp data is input into a fully connected layer without bias to obtain time information. The formula is as follows:

[0111] SE=Linear(X mark )

[0112] in, Indicates time information. represents the timestamp data after stratification, m represents the total number of timestamp types (the value in this embodiment is 3, representing hours, minutes, and seconds respectively), and Linear() represents a fully connected layer.

[0113] Then, the data matrices u, PE, and SE obtained in the above steps are added element by element to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The calculation formula is as follows:

[0114]

[0115] in, represents the high-dimensional representation of wind turbine information with timing information and time information, α represents a custom hyperparameter (when the data is normalized, α is set to 1), and + represents an element-by-element addition operation.

[0116] Finally, the high-dimensional wind turbine information Input into a fully connected layer without bias to obtain the shared query key matrix required by the multi-head self-attention mechanism. The calculation formula is as follows:

[0117]

[0118] in, Represents the shared query key matrix required for the multi-head self-attention mechanism.

[0119] At the same time, the high-dimensional wind turbine information Input into another fully connected layer without bias to obtain the value matrix required by the multi-head self-attention mechanism. The formula is defined as follows:

[0120]

[0121] in, A matrix of values ​​required to represent a multi-head self-attention mechanism.

[0122] In step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers. Then, the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket numbers using the mapping relationship, and sorted in time sequence within the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket:

[0123] First, the obtained shared query key matrix QK and value matrix V are split into multiple heads of the same size so that the multi-head self-attention mechanism can be applied later. The calculation formula is as follows:

[0124] QK after =Split(QK)

[0125] V after =Split(V)

[0126] in, represents the shared query key matrix after splitting, Represents the value matrix after splitting, n head Represents the number of heads of the multi-head self-attention mechanism, and Split() represents the split operation.

[0127] In order to describe the implementation process more concisely, the following steps S1 to S7 are defined:

[0128] S1. Split the shared query key matrix QK after Multiply it with the randomly generated normal distribution matrix to achieve the purpose of random rotation. The calculation formula is as follows:

[0129] QK new =QK after ×R

[0130] in, represents the query key matrix after random rotation, Represents a matrix of randomly generated normal distribution, l bucket Indicates the size of each hash bucket.

[0131] S2. QK obtained after random rotation new The elements in are reversed one by one, and then combined with QK new The final matrix is ​​obtained by splicing, and the calculation formula is as follows:

[0132] QK final =Reverse(QK new )+QK new

[0133] in, It represents the final processed shared query key matrix, Reverse() represents the inversion operation, and + represents the concatenation operation.

[0134] S3. Get the shared query key matrix QK final The maximum value of each row is used. The index corresponding to the maximum value is used as the hash mapping result corresponding to each sequence data, and then added to the preset offset to obtain the hash bucket number corresponding to this sequence data. The calculation formula is as follows:

[0135] buckets=max(QK final )+i*n bucket

[0136] in, Represents the hash bucket number matrix corresponding to each sequence data, i∈{0,...,n hash -1} represents the hash round number, n hash Indicates the total number of hash rounds, n bucket Represents the number of hash buckets, and max() is used to obtain the index corresponding to the maximum value in each row of the matrix.

[0137] S4. Query key matrix QK after Sum value matrix V after In the first dimension, the hash bucket numbers are sorted from small to large, and then the hash buckets with the same number are sorted in chronological order. The calculation formula is as follows:

[0138] QK sort =Sort time (Sort num (QK after ))

[0139] V sort =Sort time (Sort num (V after ))

[0140] in, represents the sorted shared query key matrix, Represents the sorted value matrix, Sort time () indicates sorting in chronological order. num () means sorting by hash bucket number.

[0141] S5. Share the query key matrix QK sort Copy it into two matrices, namely Q sort and K sort , as the query matrix and key matrix respectively. For the key matrix K sort Sum value matrix V sort, so that it contains not only the data in the current hash bucket, but also the data in the previous hash bucket. Then, the scaled dot product function is used to calculate the attention score, and then the calculated attention score is converted into a probability value through the softmax function, and multiplied with the value matrix to get the output. The calculation formula is as follows:

[0142]

[0143] in, Represents the value obtained by the i-th hash bucket, represents the query matrix required to calculate the i-th hash bucket, represents the key matrix required to calculate the i-th hash bucket, Represents the value matrix required to calculate the i-th hash bucket, and softmax() represents the softmax operation.

[0144] S6. Calculate the value z obtained from all hash buckets i Splice to get the complete output sequence. The calculation formula is as follows:

[0145] Z = concat(z i )

[0146] Where i∈{0,...,n hash -1} indicates the hash round number, Represents the complete sequence output, and concat() represents the concatenation operation.

[0147] S7. Use Sort time (), Sort num () The sorting information provided during sorting restores the complete sequence output to the original time sequence order. The calculation formula is as follows:

[0148] Z sort =ReSort(Z)

[0149] in, Represents the output after the order is restored. ReSort() means restoring the temporal order according to the provided sorting information.

[0150] Repeat the above steps S1 to S7, using different random rotations each time, and finally perform weighted summation on the outputs obtained from each calculation. The calculation formula is as follows:

[0151]

[0152] in, Represents the result of weighted average of all hash rounds, represents the output matrix of the kth round of hashing, wk Represents the output matrix of the kth round of hashing The weight of .

[0153] Finally, concatenate each head in the multi-head self-attention mechanism to obtain the output Z of the multi-head self-attention mechanism final , the calculation formula is as follows:

[0154] Z final =concat(Z sum )

[0155] in, Represents the final output of the multi-head self-attention mechanism.

[0156] In step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model:

[0157] like Figure 3 As shown in the model structure diagram, the final output result Z of the multi-head self-attention mechanism final High-dimensional representation of wind turbine information Perform residual connection and layer normalization, then input into two convolutional layers, perform layer normalization, and finally input into a fully connected layer containing bias to obtain the final output of the model. The calculation formula is as follows:

[0158]

[0159] out2=Con1v(Con1v(LayerNorm(out1)))

[0160] out3 = LayerNorm (out1 + out2)

[0161] out=Linear(out3)

[0162] in, Represents the result after the first residual connection, represents the result after layer normalization and two one-dimensional convolutional layers. Represents the result after residual connection and layer normalization, Indicates the final output of the model. Con1v() indicates a one-dimensional convolution operation, and LayerNorm() indicates a layer normalization operation.

[0163] Transpose the final result out and intercept the data of the last p moments to get the predicted value. The formula is defined as follows:

[0164]

[0165] in, Represents the final model prediction result, and Cut() means intercepting the data of the last p moments.

[0166] In the process of training the model, the Adam optimizer is used to minimize the MSELoss loss function and learn the model parameters. The MSELoss loss function formula is defined as follows:

[0167]

[0168] Among them, truth n Represents the true value at the nth moment, res n Represents the predicted value at the nth moment. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0169] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A long-term wind power prediction method based on improved Bert, comprising the following steps: Step 1) The missing values ​​in the data set are filled using the backward filling method, the processed data are normalized, and the timestamps of the data are layered. Step 2), input the preprocessed information of each wind turbine into a convolution layer and perform absolute position encoding. At the same time, input the timestamp after layered processing into a fully connected layer. Then, the information matrix of each wind turbine processed by the convolution layer and absolutely position encoded is added element by element to the timestamp matrix of each wind turbine processed by the fully connected layer, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism. In step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers. Then, the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket number using the mapping relationship, and sorted in chronological order within the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket. Step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model.

2. According to the method for long-term wind power prediction based on improved Bert according to claim 1, it is characterized in that: In step 1), the missing values ​​in the data set are filled using a backward filling method, and the processed data are normalized, and then the timestamps of the data are layered: First, the abnormal values ​​of some features of the wind turbine information in the data set are set as thresholds. For example, if the wind turbine power value is less than 0, it is set to 0; if the angle values ​​of the three blades of the wind turbine are greater than 89, it is set to 89. Then, the missing values ​​in the data set are filled using the backward filling method, that is, the data at the moment after the missing data is used for filling. Then, the mean and standard deviation of all features of the wind turbine information in the data set are calculated, and the data are normalized using the mean and standard deviation. The normalization formula is as follows: Among them, data represents the value of a specific feature, mean represents the mean of a specific feature, and std represents the standard deviation of a specific feature. Finally, the timestamp features in the data set are layered. The timestamp is divided into three layers: hours, minutes, and seconds, and the hours, minutes, and seconds of the timestamp are scaled respectively. The scaling formula is as follows: Among them, Stamp h 、Stamp m 、Stamp s Respectively represent the hours, minutes, and seconds values ​​in the timestamp.

3. The long-term wind power prediction method based on improved Bert according to claim 2 is characterized in that: In step 2), the pre-processed information of each wind turbine is input into a convolution layer and absolute position encoding is performed. At the same time, the timestamp after layered processing is input into a fully connected layer. Then, the information matrix of each wind turbine after convolution layer processing and absolute position encoding is added element by element to the timestamp matrix of each wind turbine after full connection layer processing, so as to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The high-dimensional representation of the wind turbine information is simultaneously input into two different fully connected layers to obtain the shared query key matrix and value matrix required by the multi-head self-attention mechanism respectively: First, the preprocessed information of the angles of the three blades of each wind turbine, the ambient temperature, the generated power, etc. is input into a one-dimensional convolution filter with a convolution kernel size of 3, a stride of 1, a padding of 1, and no bias. The original data in the data set is projected onto the preset dimension to obtain the embedded representation of the wind turbine information. The formula is as follows: u=Conv1d(X) in, represents the embedded representation of the obtained wind turbine information, Represents the preprocessed wind turbine information data. s, p, d, and f represent the length of the input sequence, the length of the predicted sequence, the embedding dimension, and the number of features of the sequence, respectively. Conv1d() represents a one-dimensional convolution operation. Next, the absolute position embedding method is used to retain the timing information of the original data. Specifically, the sin function is used to process the value of the specific feature of the even index position of the fan information data, and the cos function is used to process the value of the feature of the odd index specific position of the fan information data to obtain the timing information. The absolute position embedding formula is as follows: in, Indicates the timing information, pos indicates the index of the specific time position of the fan information, The position index representing the specific features of the fan information data at each moment. (For each fan, a batch of fan information is a two-dimensional matrix, each row represents the fan information at each moment, and its columns represent the features contained in the fan information data). At the same time, the stratified timestamp data is input into a fully connected layer without bias to obtain the time information. The formula is as follows: SE=Linear(X mark ) in, Indicates time information. represents the timestamp data after stratification, m represents the total number of timestamp types (the value in this embodiment is 3, representing hours, minutes, and seconds respectively), and Linear() represents a fully connected layer. Then, the data matrices u, PE, and SE obtained in the above steps are added element by element to obtain a high-dimensional representation of the wind turbine information with timing information and time information. The calculation formula is as follows: in, represents the high-dimensional representation of wind turbine information with timing information and time information, α represents a custom hyperparameter (when the data is normalized, α is set to 1), and + represents an element-by-element addition operation. Finally, the high-dimensional wind turbine information Input into a fully connected layer without bias to obtain the shared query key matrix required by the multi-head self-attention mechanism. The calculation formula is as follows: in, Represents the shared query key matrix required for the multi-head self-attention mechanism. At the same time, the high-dimensional wind turbine information Input into another fully connected layer without bias to obtain the value matrix required by the multi-head self-attention mechanism. The formula is defined as follows: in, A matrix of values ​​required to represent a multi-head self-attention mechanism.

4. The long-term wind power prediction method based on improved Bert according to claim 3 is characterized in that: In step 3), the shared query key matrix is ​​randomly rotated and mapped to hash buckets with different numbers. Then, the high-dimensional representation of the wind turbine information is sorted from small to large according to the hash bucket numbers using the mapping relationship, and sorted in time sequence within the hash buckets with the same number, and then a multi-head self-attention mechanism is used in each hash bucket: First, the obtained shared query key matrix QK and value matrix V are split into multiple heads of the same size so that the multi-head self-attention mechanism can be applied later. The calculation formula is as follows: QK after =Split(QK) V after =Split(V) in, represents the shared query key matrix after splitting, Represents the value matrix after splitting, n head Represents the number of heads of the multi-head self-attention mechanism, and Split() represents the split operation. In order to describe the process more concisely, the following steps S1 to S7 are defined: S1. Split the shared query key matrix QK after Multiply it with the randomly generated normal distribution matrix to achieve the purpose of random rotation. The calculation formula is as follows: QK new =QK after ×R in, represents the query key matrix after random rotation, Represents a matrix of randomly generated normal distribution, l bucket Indicates the size of each hash bucket. S2. QK obtained after random rotation new The elements in are reversed one by one, and then combined with QK new The final matrix is ​​obtained by splicing, and the calculation formula is as follows: QK final =Reverse(QK new )+QK new in, It represents the final processed shared query key matrix, Reverse() represents the inversion operation, and + represents the concatenation operation. S3. Get the shared query key matrix QK final The maximum value of each row is used. The index corresponding to the maximum value is used as the hash mapping result corresponding to each sequence data, and then added to the preset offset to obtain the hash bucket number corresponding to this sequence data. The calculation formula is as follows: buckets=max(QK final )+i*n bucket in, Represents the hash bucket number matrix corresponding to each sequence data, i∈{0,...,n hash -1} represents the hash round number, n hash Indicates the total number of hash rounds, n bucket Represents the number of hash buckets, and max() is used to obtain the index corresponding to the maximum value in each row of the matrix. S4. Query key matrix QK after Sum value matrix V after In the first dimension, the hash bucket numbers are sorted from small to large, and then the hash buckets with the same number are sorted in chronological order. The calculation formula is as follows: QK sort =Sort time (Sort num (QK after )) V sort =Sort time (Sort num (V after )) in, represents the sorted shared query key matrix, Represents the sorted value matrix, Sort time () indicates sorting in chronological order. num () means sorting by hash bucket number. S5. Share the query key matrix QK sort Copy it into two matrices, namely Q sort and K sort , as the query matrix and key matrix respectively. For the key matrix K sort Sum value matrix V sort , so that it contains not only the data in the current hash bucket, but also the data in the previous hash bucket. Then, the scaled dot product function is used to calculate the attention score, and then the calculated attention score is converted into a probability value through the softmax function, and multiplied with the value matrix to get the output. The calculation formula is as follows: in, Represents the value obtained by the i-th hash bucket, represents the query matrix required to calculate the i-th hash bucket, represents the key matrix required to calculate the i-th hash bucket, Represents the value matrix required to calculate the i-th hash bucket, and softmax() represents the softmax operation. S6. Calculate the value z obtained from all hash buckets i Splice to get the complete output sequence. The calculation formula is as follows: Z=concat(z i ) where i∈{0,...,n hash -1} indicates the hash round number, Represents the complete sequence output, and concat() represents the concatenation operation. S7. Use Sort time (), Sort num () The sorting information provided during sorting restores the complete sequence output to the original time sequence order. The calculation formula is as follows: WITH sort =ReSort(Z) in, Represents the output after the order is restored, and ReSort() means restoring the temporal order according to the provided sorting information. Repeat the above steps S1 to S7, using different random rotations each time, and finally perform weighted summation on the outputs obtained from each calculation. The calculation formula is as follows: in, Represents the result of weighted average of all hash rounds, represents the output matrix of the kth round of hashing, w k Represents the output matrix of the kth round of hashing The weight of . Finally, concatenate each head in the multi-head self-attention mechanism to obtain the output Z of the multi-head self-attention mechanism final , the calculation formula is as follows: From final =concat(Z sum ) in, Represents the final output of the multi-head self-attention mechanism.

5. The long-term wind power prediction method based on improved Bert according to claim 4 is characterized in that: In step 4), the final result after being processed by the multi-head self-attention mechanism is residually connected with the high-dimensional representation of the wind turbine information, and layer normalization is performed, and then input into the two convolutional layers, and residual connection and layer normalization are performed to obtain the final output of the model: First, the final output result Z of the multi-head self-attention mechanism final High-dimensional representation of wind turbine information Perform residual connection and layer normalization, then input into two convolutional layers, perform layer normalization, and finally input into a fully connected layer containing bias to obtain the final output of the model. The calculation formula is as follows: out2=Con1v(Con1v(LayerNorm(out1))) out3 = LayerNorm (out1 + out2) out=Linear(out3) in, Represents the result after the first residual connection, represents the result after layer normalization and two one-dimensional convolutional layers. Represents the result after residual connection and layer normalization, Indicates the final output of the model. Con1v() indicates a one-dimensional convolution operation, and LayerNorm() indicates a layer normalization operation. Transpose the final result out and intercept the data of the last p moments to get the predicted value. The formula is defined as follows: RES=Cut(out T ) in, Represents the final model prediction result, and Cut() means intercepting the data of the last p moments. In the process of training the model, the Adam optimizer is used to minimize the MSELoss loss function and learn the model parameters. The MSELoss loss function formula is defined as follows: Among them, truth n Represents the true value at the nth moment, res n Represents the predicted value at the nth moment.

Images