Load forecasting method based on attention mechanism and spatiotemporal features

Abstract

A load forecasting method using an attention mechanism to fuse spatiotemporal features, wherein an electric load data set is obtained and divided into a training set and a test set; weather data is collected and sorted by importance using a random forest algorithm, and the top three significant features of the weather data are selected as model inputs; VMD is applied to decompose the electric load data into multiple local components, each component corresponds to a vibration mode within a frequency range, and the parameters in VMD are iteratively optimized using a RIME algorithm to obtain a subsequence component with the least complexity; an energy consumption forecasting model using an attention mechanism to fuse spatiotemporal features is constructed based on the decomposed subsequence components and the top three selected meteorological features as model inputs, and a BiTCN model and a BiGRU model are used in parallel to simultaneously process time series data, and feature enhancement is performed after passing through a cross-attention layer to obtain a forecasting result.

Description

Technical Field

[0001] The present invention relates to the technical field of single-phase PWM inverter circuits, and in particular to a load forecasting method integrating time and space features with an attention mechanism. Background Art

[0002] Existing energy consumption forecasting can be roughly divided into two categories: one is traditional forecasting methods such as regression analysis, time series method, and gray forecasting model; the other is artificial intelligence forecasting methods such as support vector machine and artificial neural network. Regression analysis and time series method are mainly based on statistical principles, using historical data to build statistical models to estimate and analyze future energy consumption. Artificial neural network imitates the human brain to establish mathematical operation models, can fit any complex nonlinear relationship, and has the ability to distribute and process complex mathematical problems. Although there are many methods for applying neural networks to energy consumption, due to the complexity of energy consumption characteristics, a single machine learning method has the disadvantages of high error rate, high computational complexity, and low computational efficiency, and it is difficult to achieve a high prediction accuracy. In view of the problem that energy consumption load has strong randomness and low prediction accuracy of traditional methods, the existing technology mostly uses VMD to decompose the data in the input model into subsequences of different characteristic frequencies to reduce the complexity of load characteristics and improve prediction accuracy. However, most of the parameters of VMD are set manually, which has an artificial impact on the prediction results and the universality of the prediction results is slightly poor. Therefore, how to deal with energy consumption load, decompose it reasonably, and accurately predict energy consumption data is a hot topic in current research. Summary of the invention

[0003] The present invention provides a load forecasting method that integrates spatiotemporal features with an attention mechanism. The BiTCN network is used to capture the global spatial features in the data, the BiGRU network is used to extract the temporal dependency of the data, and then a cross-attention mechanism is used to effectively fuse the spatial features and the temporal features. By calculating the attention weights, the correlation between different positions in the spatiotemporal features is learned, the features in the spatiotemporal series data are better captured, the implicit information is fully mined, and the energy consumption load is accurately predicted.

[0004] A load forecasting method that integrates spatiotemporal features with an attention mechanism includes:

[0005] Obtain a power load data set, normalize or standardize the power load data, use a sliding window to process the power load data, and generate a training set and a test set;

[0006] Collect local weather data sets, build a random forest model, and use the feature importance method of the random forest algorithm to sort the feature importance, draw a feature importance graph, and select the top three weather features based on importance.

[0007] Variational mode decomposition (VMD) is applied to decompose the power load data to obtain the subsequence components with the minimum complexity.

[0008] According to the decomposed subsequence components and the first three selected weather features as model input, an energy consumption prediction model integrating spatiotemporal features with attention mechanism is constructed. The BiTCN model and BiGRU model are used in parallel to receive input sequences at the same time to realize prediction.

[0009] In the load forecasting method that integrates spatiotemporal features with the attention mechanism, the weather data set contains 11 dimensions of weather features in total. The first 10 weather features are selected as input variables, and the load value at the current moment is used as the target variable. The data is divided into a training set and a test set. The weather features include date, temperature, air pressure, relative humidity, water vapor content per kilogram of air, light radiation, air density, wind speed, wind direction, rainfall, and load value.

[0010] In the load forecasting method that integrates spatiotemporal features with the attention mechanism, variational mode decomposition (VMD) is used to decompose power load data, and sample entropy is selected as the optimal fitness function to decompose it into multiple modal components (IMF). The sample entropy value of each IMF is calculated, and the number of modal components K and the penalty factor alpha are iteratively optimized through the RIME algorithm, and the center frequency, bandwidth and mode of each mode are updated. When the convergence condition is met, the iterative process stops, the optimal parameters are obtained, and the VMD is decomposed using the optimal parameters.

[0011] In the load prediction method of integrating spatiotemporal features with the attention mechanism, the BiTCN model and the BiGRU model operate in parallel and receive the input sequence at the same time, and then obtain the fused forward and reverse sequence information respectively, and pass the sequence information of the BiTCN model as the query and the sequence information of the BiGRU model as the key and value to the cross attention layer for feature enhancement, and output through the fully connected layer to achieve prediction.

[0012] In the load forecasting method that integrates the attention mechanism with spatiotemporal features, the missing values ​​in the power load data are filled by interpolation, the maximum threshold and the minimum threshold are set, and the abnormal values ​​in the power load data that are higher than the maximum threshold or lower than the minimum threshold are deleted.

[0013] In the load forecasting method of integrating spatiotemporal features with the attention mechanism, the power load data is subjected to Z-Score normalization, and the Z-Score function is as follows:

[0014] (1)

[0015] In the formula, is the current i-th data point, is the mean value of the entire data set, σ is the standard deviation of the entire data set, is the standardized score of the ith data point.

[0016] In the load prediction method of the attention mechanism integrating spatiotemporal features, the RIME algorithm is used to iteratively optimize the parameters in VMD, including: first, taking the load value as the original signal, clarifying the parameters to be optimized as the number of modal components K and the penalty factor alpha, then selecting the sample entropy as the fitness function to quantitatively evaluate the pros and cons of the parameter combination, and before starting the optimization, first preset an individual extreme value for each parameter representing the optimal solution of each parameter in the current iteration process, and then set the population size and the maximum number of iterations as the termination conditions of the optimization, call the RIME algorithm, iteratively update the speed information and position information of each parameter, and in each iteration, update the individual extreme value of each parameter according to the calculation result of the fitness function, find the global optimal parameter, and then use the global optimal parameter to perform VMD decomposition on the load value to obtain the subsequence component with the minimum complexity and call the RIME optimization algorithm.

[0017] In the load forecasting method that integrates the attention mechanism with spatiotemporal features, sample entropy is selected as the optimal fitness function, the sample entropy value of each IMF is calculated, the number of iterations and the population size are set, the optimization range of the number of modal components K and the penalty factor alpha is set, and the optimal solution within the range is found through the RIME algorithm. The optimized number of modal components K and the penalty factor alpha are automatically obtained, and the VMD is decomposed using the optimal parameters to obtain multiple optimized IMF values.

[0018] In the load forecasting method of integrating spatiotemporal features with the attention mechanism, when VMD is decomposed, the values ​​of K and alpha are preset to obtain multiple IMF values, and the sample entropy of formula (2) is selected as the fitness function to minimize the entropy value. The sample entropy calculation formula is as follows:

[0019] (2)

[0020] In the formula, N represents the total number of data points, r represents the similarity threshold, and They represent the probability that the distance between vectors is less than r when the embedding dimension is m and m+1, respectively. Represents the sample entropy of a time subsequence of length m.

[0021] In the load forecasting method of the attention mechanism fusing spatiotemporal features, constructing an energy consumption prediction model of the attention mechanism fusing spatiotemporal features includes: in the BiTCN model, passing the sequence data X=(x1, x2, ..., xT) as input to the forward TCN, thereby obtaining a forward output result R, and in order to capture the reverse semantic information of the sequence, the original sequence X is reversed to obtain X'=(xT, xT-1, ..., x1), and it is input into the reverse TCN, thereby obtaining a reverse output result R'.

[0022] In the load forecasting method that integrates spatiotemporal features with the attention mechanism, the forward output R and the reverse output R' are integrated, and the integrated result is normalized to obtain a temporal feature that integrates the forward and reverse semantic information. At the same time, the sequence data is passed to BiGRU to capture the temporal features of the sequence, and multi-feature fusion is performed. Then, through the cross-attention mechanism, the attention weights between different positions are calculated, the input sequence is weighted and summed, and the key features are input into the fully connected layer for prediction to obtain the prediction result.

[0023] In the load forecasting method of integrating spatiotemporal features with the attention mechanism, the energy consumption prediction model integrating spatiotemporal features with the attention mechanism includes a bidirectional temporal convolutional network module, a cross attention mechanism module, and a bidirectional gated recurrent unit. The bidirectional temporal convolutional network module adds a weight normalization layer, an activation function, a Dropout layer, and a bidirectional time block to the TCN convolutional layer. After the input data is processed by two layers of dilated causal convolutional layers, a direct mapping of the input is added for residual connection.

[0024] In the load forecasting method that integrates spatiotemporal features with the attention mechanism, the bidirectional time block contains two TCN modules. The input x passes through the forward time convolution block to obtain the result of forward propagation. At the same time, the input x is reversed in the time dimension, and then passes through the backward time convolution block to obtain the result of backward propagation. The results of forward and backward propagation are spliced ​​in the feature dimension, and the Dropout layer is applied to prevent overfitting to obtain the output result.

[0025] In the load forecasting method that integrates spatiotemporal features with the attention mechanism, the cross-attention mechanism module calculates attention on two different sequences to establish an association relationship between the two sequences. It splits the input tensor into two parts, and then uses one part as the query set x and the other part as the key value set. The similarity between the query and the key value is calculated by dot product, and weight normalization and weighted summation are performed to obtain the attention weight to achieve information fusion. The bidirectional gated recurrent unit module combines the bidirectional characteristics of the bidirectional recurrent neural network and the gating mechanism of the gated recurrent unit, and trains two independent GRU networks at the same time, so that the model can simultaneously consider the past and future information of the sequence data and introduce update gates and reset gates to control the flow of information to process the dependencies in the time series data. By stacking multiple bidirectional GRU layers, the model's ability to represent and characterize sequence data is improved.

[0026] In the load forecasting method of integrating spatiotemporal features with the attention mechanism, the cross attention function is as follows:

[0027] (3)

[0028] Where Q is the query matrix, K is the key matrix, and V is the value matrix. is the dimension of the key vector and Softmax is the mapping function.

[0029] Compared with the prior art, the present invention has the following advantages: The present invention uses the VMD decomposition method optimized by RIME to decompose the time series. This method overcomes the limitation of the traditional VMD decomposition method that requires manual parameter adjustment. The parameters of VMD are automatically adjusted by the RIME algorithm, making the load decomposition process more intelligent and efficient. Moreover, the introduction of sample entropy as a fitness function can ensure that the load component with the minimum time complexity is obtained during the decomposition process, which not only improves the efficiency of signal decomposition, but also enables better learning and extraction of feature information.

[0030] The present invention adopts a combined prediction model of BiTCN-Attention-BiGRU. BiTCN extracts forward and backward time information in the input sequence to obtain better feature extraction capabilities. Traditional TCN only considers the forward convolution calculation of the input sequence, while ignoring the impact of backward information on the prediction results. At the same time, the advantages of BiGRU in processing timing are combined, and key information is captured through the Attention mechanism, so that the model can accurately capture and extract key features in complex power load data. By closely combining the VMD decomposition method optimized by RIME with the BiTCN-Attention-BiGRU prediction model, the prediction accuracy of the model is significantly improved, thereby obtaining a more accurate energy consumption prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The drawings in the specification are only for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.

[0032] In the attached picture:

[0033] Figure 1 It is the BiTCN structure diagram of the present invention;

[0034] FIG2 is a diagram of the BiTCN-Attention-BiGRU model architecture of the present invention;

[0035] FIG3 is a flow chart of load forecasting of the present invention;

[0036] FIG4 is a diagram of a power load sequence of the present invention;

[0037] FIG5 is a characteristic correlation diagram of the present invention;

[0038] FIG6 is an exploded view of the VMD of the present invention;

[0039] FIG7 is a decomposition diagram of RIME-VMD optimization of the present invention;

[0040] FIG8 is a comparison diagram of the original model of the present invention;

[0041] FIG. 9 is a comparison diagram of the optimized models of the present invention.

[0042] The present invention is further explained below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0043] The specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0044] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the present invention. The scope of protection of the present invention shall be determined by the attached claims.

[0045] To facilitate understanding of the embodiments of the present invention, further explanation will be given below by taking specific embodiments as examples in conjunction with the accompanying drawings, and each of the accompanying drawings does not constitute a limitation on the embodiments of the present invention.

[0046] like Figures 1 to 9 As shown, a load forecasting method that integrates the attention mechanism with spatiotemporal features includes:

[0047] Obtain a power load data set, normalize or standardize the power load data, use a sliding window to process the power load data, and generate a training set and a test set;

[0048] Collect local weather data sets, build a random forest model, and use the feature importance method of the random forest algorithm to sort the feature importance, draw a feature importance graph, and select the top three weather features based on importance.

[0049] Variational mode decomposition (VMD) is applied to decompose the power load data to obtain the subsequence components with the minimum complexity.

[0050] According to the decomposed subsequence components and the first three selected weather features as model input, an energy consumption prediction model integrating spatiotemporal features with attention mechanism is constructed. The BiTCN model and BiGRU model are used in parallel to receive input sequences at the same time to realize prediction.

[0051] In the preferred implementation of the load forecasting method that integrates the attention mechanism with spatiotemporal features, the weather data set contains a total of 11 dimensions of weather features. The top 10 weather features are selected as input variables, and the load value at the current moment is used as the target variable. The data set is divided into a training set and a test set. The weather features include date, temperature, air pressure, relative humidity, water vapor content per kilogram of air, light radiation, air density, wind speed, wind direction, rainfall, and load value.

[0052] In the preferred implementation of the load forecasting method that integrates the attention mechanism with spatiotemporal features, variational mode decomposition (VMD) is used to decompose the power load data, and the sample entropy is selected as the optimal fitness function to decompose it into multiple modal components (IMF). The sample entropy value of each IMF is calculated, and the number of modal components K and the penalty factor alpha are iteratively optimized through the RIME algorithm, and the center frequency, bandwidth and mode of each mode are updated. When the convergence condition is met, the iterative process stops, the optimal parameters are obtained, and the VMD is decomposed using the optimal parameters.

[0053] In the preferred implementation of the load prediction method that integrates spatiotemporal features with the attention mechanism, the BiTCN model and the BiGRU model operate in parallel and simultaneously receive the input sequence, and then obtain the fused forward and reverse sequence information respectively, and use the sequence information of the BiTCN model as the query, and the sequence information of the BiGRU model as the key and value to pass to the cross attention layer for feature enhancement, and output through the fully connected layer to achieve prediction.

[0054] In a preferred implementation of the load forecasting method that integrates the attention mechanism with spatiotemporal features, missing values ​​in the power load data are filled using interpolation, a maximum threshold and a minimum threshold are set, and abnormal values ​​in the power load data that are higher than the maximum threshold or lower than the minimum threshold are deleted.

[0055] In a preferred implementation of the load forecasting method using the attention mechanism to integrate spatiotemporal features, the power load data is subjected to Z-Score normalization, and the Z-Score function is as follows:

[0056] (1)

[0057] In the formula, is the current i-th data point, is the mean value of the entire data set, σ is the standard deviation of the entire data set, is the standardized score of the ith data point.

[0058] In the preferred implementation of the load forecasting method integrating the attention mechanism with spatiotemporal features, the RIME algorithm is used to iteratively optimize the parameters in the VMD, including: first, taking the load value as the original signal, clarifying the parameters to be optimized as the number of modal components K and the penalty factor alpha, then selecting the sample entropy as the fitness function to quantitatively evaluate the quality of the parameter combination, and before starting the optimization, first presetting an individual extreme value for each parameter representing the optimal solution of each parameter in the current iteration process, and then setting the population size and the maximum number of iterations as the termination conditions for the optimization, calling the RIME algorithm, and iteratively updating the speed information and position information of each parameter, In each iteration, the individual extreme value of each parameter is updated according to the calculation result of the fitness function, the global optimal parameter is found, and then the load value is decomposed by VMD using the global optimal parameter to obtain the subsequence component with the minimum complexity. The RIME optimization algorithm is called, and the sample entropy is selected as the optimal fitness function. The sample entropy value of each IMF is calculated, the number of iterations and the population size are set, the optimization range of the number of modal components K and the penalty factor alpha is set, and the optimal solution within the range is found through the RIME algorithm. The optimized number of modal components K and the penalty factor alpha are automatically obtained, and the optimal parameters are used to decompose VMD to obtain multiple optimized IMF values.

[0059] In the preferred implementation of the load forecasting method integrating the attention mechanism with spatiotemporal features, during VMD decomposition, the values ​​of K and alpha are preset to obtain multiple IMF values, and the sample entropy of formula (2) is selected as the fitness function to minimize the entropy value. The sample entropy calculation formula is as follows:

[0060] (2)

[0061] In the formula, N represents the total number of data points, r represents the similarity threshold, and They represent the probability that the distance between vectors is less than r when the embedding dimension is m and m+1, respectively. Represents the sample entropy of a time subsequence of length m.

[0062] In a preferred implementation of the load prediction method of the attention mechanism fusion spatiotemporal features, constructing an energy consumption prediction model of the attention mechanism fusion spatiotemporal features includes: in the BiTCN model, firstly, passing the sequence data X=(x1, x2,..., xT) as input to the forward TCN, thereby obtaining a forward output result R; in order to capture the reverse semantic information of the sequence, the original sequence X is reversed to obtain X'=(xT, xT-1, ..., x1), and it is input into the reverse TCN, thereby obtaining a reverse output result R'; then, the forward output R and the reverse output R' are integrated, and the integrated result is normalized to obtain a temporal feature that combines the forward and reverse semantic information; at the same time, the sequence data is passed to the BiGRU to capture the temporal features of the sequence, and multi-feature fusion is performed; then, the attention weights between different positions are calculated through the cross attention mechanism, the input sequence is weighted summed, and the key features are input into the fully connected layer for prediction to obtain the prediction result.

[0063] In the preferred implementation of the load forecasting method that integrates the attention mechanism with spatiotemporal features, the energy consumption prediction model that integrates the attention mechanism with spatiotemporal features includes a bidirectional temporal convolutional network module, a cross-attention mechanism module, and a bidirectional gated recurrent unit. The bidirectional temporal convolutional network module adds a weight normalization layer, an activation function, a Dropout layer, and a bidirectional time block to the TCN convolutional layer. After the input data is processed by two layers of dilated causal convolutional layers, a direct mapping of the input is added for residual connection.

[0064] In a preferred implementation of the load forecasting method that integrates the attention mechanism with spatiotemporal features, the bidirectional time block includes two TCN modules, and the input x passes through a forward time convolution block to obtain the result of forward propagation. At the same time, the input x is reversed in the time dimension, and then passes through a backward time convolution block to obtain the result of backward propagation. The results of forward and backward propagation are spliced ​​in the feature dimension, and a Dropout layer is applied to prevent overfitting to obtain the output result.

[0065] In the preferred implementation of the load forecasting method of the attention mechanism integrating spatiotemporal features, the cross-attention mechanism module calculates attention on two different sequences to establish an association relationship between the two sequences. It splits the input tensor into two parts, and then uses one part as the query set x and the other part as the key value set. The similarity between the query and the key value is calculated by dot product, and weight normalization and weighted summation are performed to obtain the attention weight to achieve information fusion. The bidirectional gated recurrent unit module combines the bidirectional characteristics of the bidirectional recurrent neural network and the gating mechanism of the gated recurrent unit, and trains two independent GRU networks at the same time, so that the model can simultaneously consider the past and future information of the sequence data and introduce update gates and reset gates to control the flow of information to process the dependencies in the time series data. By stacking multiple bidirectional GRU layers, the model's ability to represent sequence data is improved. The cross-attention function is as follows:

[0066] (3)

[0067] Where Q is the query matrix, K is the key matrix, and V is the value matrix. is the dimension of the key vector and Softmax is the mapping function.

[0068] In another embodiment, a load forecasting method using an attention mechanism to fuse spatiotemporal features comprises the following steps:

[0069] Obtain a power load data set, normalize or standardize the power load data, use a sliding window to process the power load data, and generate a training set and a test set;

[0070] Collect local weather data sets, a total of 11 dimensions of weather features, including date, temperature, air pressure, relative humidity, water vapor content per kilogram of air, light radiation, air density, wind speed, wind direction, rainfall, load value, select the first 10 weather features as input variables, the current load value as the target variable, and divide them into training set and test set, build a random forest model, and use the feature importance method of the random forest algorithm to sort the feature importance, draw a feature importance graph, and select the top three weather features according to importance.

[0071] Variational mode decomposition (VMD) is applied to decompose power load data. Sample entropy is selected as the optimal fitness function and decomposed into multiple modal components (IMF). The sample entropy value of each IMF is calculated. The number of modal components K and penalty factor alpha are iteratively optimized through the RIME algorithm. The center frequency, bandwidth and mode of each mode are updated. When the convergence condition is met, the iterative process stops and the optimal parameters are obtained. The optimal parameters are used to decompose VMD to obtain the subsequence components with the minimum complexity.

[0072] According to the decomposed sub-sequence components and the first three selected weather features as model input, an energy consumption prediction model that integrates spatiotemporal features with an attention mechanism is constructed. The BiTCN model and the BiGRU model are used in parallel to receive the input sequence at the same time, and the forward and reverse sequence information of the fusion are obtained respectively. The sequence information of the BiTCN model is used as the Query, and the sequence information of the BiGRU model is used as the Key and Value to be passed to the cross-attention layer for feature enhancement, which is then output through the fully connected layer to achieve prediction.

[0073] In one embodiment, the load forecasting method using the attention mechanism to fuse spatiotemporal features includes:

[0074] Obtain the power load data set, handle missing values ​​and outliers, normalize or standardize the data, use sliding windows to process time series data, and divide the data into training sets and test sets to ensure data consistency.

[0075] Collect weather data, including temperature, wind speed, meteorological information, etc. Use the random forest algorithm to select the top three significant features and use them as model input to reduce the redundancy of input data.

[0076] VMD is applied to decompose the power load data into multiple local components, each of which corresponds to a vibration mode within a frequency range. The RIME algorithm is used to iteratively optimize the parameters in VMD to obtain the subsequence components with the least complexity. According to the decomposed components, the first three meteorological features are selected as the model input. An energy consumption prediction model that integrates spatiotemporal features with an attention mechanism is constructed. The BiTCN model and the BiGRU model are used in parallel to process the sequence data at the same time. After the cross-attention layer, feature enhancement is performed to achieve high-precision prediction. Among them,

[0077] For missing values, interpolation is used to fill in the data. For some outliers that are too high or too low, the maximum and minimum thresholds are set to delete them. The data is normalized by Z-Score, the sliding window size is set, and the training set and test set are divided to ensure the consistency of the data. The Z-Score function is as follows:

[0078]

[0079] Dynamically changing features such as temperature and wind speed are introduced as predictable sequence data. The importance of each meteorological factor is ranked using the random forest algorithm to achieve feature selection, thereby improving the training efficiency and prediction performance of the model.

[0080] The process of parameter optimization based on the RIME-VMD decomposition method: First, the parameters to be optimized are the number of modal components K and the penalty factor alpha, and a maximum number of iterations is set as the termination condition for optimization. Then, the sample entropy is selected as the fitness function to quantitatively evaluate the pros and cons of parameter combinations. Before starting the optimization, an individual extreme value is preset for each parameter, which represents the optimal solution of each parameter in the current iteration process. Then, the velocity information and position information of each parameter are iteratively updated based on the RIME algorithm. In each iteration, the individual extreme value of each parameter is updated according to the calculation result of the fitness function to find the global optimal solution. The optimized VMD decomposition method is then used to process the original sequence, decomposing the complex load sequence into subsequence components with lower complexity.

[0081] Construct a BiTCN-Attention-BiGRU prediction model, such as Figure 2 As shown in the figure, the specific process is as follows: In the BiTCN model, the sequence data X = (x1, x2, ..., xT) is first passed as input to the forward TCN to obtain a forward output result R. In order to capture the reverse semantic information of the sequence, the original sequence X is reversed to obtain X' = (xT, xT-1, ..., x1), and it is input into the reverse TCN to obtain a reverse output result R'. Then, the forward output R and the reverse output R' are integrated, and the integrated result is normalized to obtain a temporal feature that combines the forward and reverse semantic information. At the same time, the sequence data is passed to BiGRU to capture the temporal features of the sequence, and multi-feature fusion is performed. Then, through the cross-attention mechanism, the sequence information of the BiTCN model is used as the query, and the sequence information of the BiGRU model is used as the key and value. It is passed to the cross-attention layer for feature enhancement and output through the fully connected layer to achieve high-precision prediction. Specifically, the cross-attention function is shown below:

[0082]

[0083] In one embodiment, Figure 3 As shown, for the problem of long sequence load forecasting, the present invention combines the frost optimization algorithm with the BiTCN and BiGRU network models that introduce the Attention mechanism, and proposes a BiTCN-Attention-BiGRU combined forecasting model based on the frost optimization algorithm. The present invention provides a load forecasting method that integrates the attention mechanism with spatiotemporal features, and the method includes:

[0084] 1. Obtain a public power transformer dataset and a local weather dataset from a certain region in China, including data from July 2016 to June 2018. Each data point contains 8-dimensional features, including the recording date of the data point, 6 different types of transformer load characteristics (high useful load, high useless load, medium useful load, medium useless load, low useful load, low useless load) and load value. Each data point is sampled once an hour. The sequence diagram of the load value changing over time is as follows: Figure 4 The weather dataset has 10 features, including some geographical measurements such as air pressure, temperature, humidity, rainfall, etc.

[0085] 2. Clean the power data as necessary. For data with missing values, use the linear interpolation method of formula (1) to fill in the missing values, set the maximum threshold to 60, the minimum threshold to 0, and remove outliers that are too large or too small. Perform Z-score standardization on the data to unify the data distribution. Encode the time characteristics corresponding to the load into periodic characteristics, and then combine them with the load characteristics and meteorological factors (air pressure, temperature, wind speed, etc.) to form the input vector of the prediction model.

[0086] (1)

[0087] In the formula, x represents the time position of the missing value, y represents the missing value to be filled, x0 and y0 represent the previous moment and value of the missing value, respectively, and x1 and y1 represent the next moment and value of the missing value, respectively.

[0088] 3. Perform VMD initial decomposition on the cleaned data, select the load value as the decomposition signal, set the initial parameters, including the number of modal components K and the penalty factor alpha, and perform decomposition to obtain multiple IMF values. The decomposition results are as follows: Figure 6 As shown, the sample entropy of formula (2) is selected as the fitness function to minimize the entropy value. The sample entropy calculation formula is as follows:

[0089] (2)

[0090] In the formula, N represents the total number of data points, r represents the similarity threshold, They represent the probability that the distance between vectors is less than r when the embedding dimension is m and m+1, respectively. Represents the sample entropy of a time subsequence of length m.

[0091] 4. Call the RIME optimization algorithm, select sample entropy as the best fitness function, calculate the sample entropy value for each IMF, set the number of iterations and population size, set the optimization range of K and alpha, set the search range of the modal component number K to (2, 8), and set the search range of the penalty factor alpha to (100, 3000). Use the RIME algorithm to optimize the parameters of VMD, automatically obtain the optimized parameters K and alpha, and use the optimal parameters to decompose VMD to obtain multiple optimized IMF values. The decomposition results are as follows: Figure 7 shown.

[0092] The RIME optimization algorithm uses soft frost search strategy, hard frost perforation mechanism, and active greedy selection mechanism to simulate the movement of soft frost particles to perform algorithm search. It develops the algorithm by simulating the crossover behavior between hard frost agents to ensure that the population evolves in a better direction in each iteration.

[0093] The calculation formulas for the soft frost search strategy and hard frost perforation mechanism are as follows:

[0094]

[0095]

[0096] In the formula, represents the jth position of the newly generated search agent in the i-th dimension, represents the position of the current optimal search agent in the jth dimension, Represents a random number used to increase the randomness of the search. is an angle parameter related to the number of iterations, simulating the change in the growth direction of frost particles. is a step function control coefficient used to adjust the search step size, and h is used to further adjust the search step size. and Respectively represent the upper and lower bounds of the search space in the i-th dimension, Used to decide whether to apply the hard frost perforation mechanism, Used to evaluate the performance of this agent.

[0097] 5. Select the features with significant influence in weather factors. The weather data set has 11 features, including date, temperature, air pressure, relative humidity, water vapor content per kilogram of air, light radiation, air density, wind speed, wind direction, rainfall, and load value. Random forest algorithm is used to sort weather-related factors, and load value is used as the target variable. The ratio of training set to test set is 8:2. The feature importance method of random forest algorithm is used, and mean square error is used as the loss function. According to the average value of all decision tree prediction results, the importance of each meteorological factor is sorted, and a feature importance diagram is drawn to perform feature selection. The screening results are as follows: Figure 5 As shown, the first three temperature, air pressure, and light radiation are selected as input features and recorded as X1, X2, and X3 to reduce the redundancy of data input.

[0098] 6. According to the RIME algorithm, the decomposed components are optimized and combined with the selected meteorological features as model input to construct an energy consumption prediction model that integrates the attention mechanism and spatiotemporal features.

[0099] 7. Define the BiTCN-Ateention-BiGRU model. The overall framework of the model proposed in this invention is as follows Figure 2 As shown in the figure, it mainly consists of three parts: bidirectional temporal convolutional network, cross attention mechanism, and bidirectional gated recurrent unit.

[0100] Bidirectional temporal convolutional network module, such as Figure 1 As shown in the figure, the module adds a weight normalization layer, an activation function, a Dropout layer, and a bidirectional time block on the basis of the TCN convolution layer. Each residual block has two causal expansion convolution operations. After each convolution operation, a weight normalization layer, an activation function layer, and a Dropout layer are added. At the same time, a 1*1 convolutional neural network is added to the TCN network to keep the original input dimension unchanged and add it. Among them, the weight normalization layer can effectively speed up the calculation speed, and the Dropout layer prevents the model from overfitting. In order to capture more information, a bidirectional time block is added, which contains two TCN modules. The input x passes through the forward time convolution block to obtain the result of forward propagation. At the same time, the input x is reversed in the time dimension, and then passed through the backward time convolution block to obtain the result of backward propagation. The results of forward and backward propagation are spliced ​​in the feature dimension, and Dropout is applied to prevent overfitting to obtain the output result.

[0101] The cross attention mechanism module refers to calculating attention on two different sequences to establish the association relationship between the two sequences. Its basic principle is to split the input tensor into two parts, and then use one part as the query set x and the other part as the key value set. The similarity between the query and the key is calculated by dot product, and the weight normalization and weighted summation are performed to obtain the attention weight to achieve information fusion.

[0102] The bidirectional gated recurrent unit module combines the bidirectional characteristics of the bidirectional recurrent neural network and the gating mechanism of the gated recurrent unit, and trains two independent GRU networks at the same time, so that the model can simultaneously consider the past and future information of the sequence data. In addition, the update gate and reset gate are introduced to control the flow of information, which can better capture the long-term dependencies in the time series data, effectively avoid the problems of gradient disappearance and gradient explosion, and effectively handle the dependencies in the time series data. By stacking multiple bidirectional GRU layers, the model's ability to represent sequence data can be improved.

[0103] The calculation formula for the reset gate is:

[0104]

[0105] In the formula, represents the input data at the t-th time step, express The reset gate weight parameters, represents the hidden state at time t-1, express The reset gate weight parameters of represents the bias term for resetting the gate, represents the sigmoid activation function; t≤10;

[0106] The calculation formula of the update gate is:

[0107]

[0108] In the formula, express The update gate weight parameter, express The update gate weight parameter, represents the bias parameter of the update gate;

[0109] The BiTCN-Attention-BiGRU prediction module is as follows Figure 2 As shown in the figure, the module consists of an input layer, a BiTCN module, an Attention mechanism, a BiGRU module, and an output layer. Each layer in the module is described in detail as follows:

[0110] (1) Model input: In the input layer, each load feature and its corresponding influencing factors are input, totaling 14-dimensional features, including 6-dimensional original features, 5-dimensional RIME optimized decomposition vectors, and 3-dimensional meteorological features. Feature fusion is performed as model input and the 14-dimensional features are input into the BiTCN module and BiGRU module.

[0111] (2) The BiTCN module in this example includes two layers of BiTCN models:

[0112] The first-layer BiTCN model contains forward time blocks and reverse time blocks. Each time block has an input dimension of 14, an output dimension of 32, a causal convolution kernel size of 3, a dilation factor of 1, a padding size of 2, a step size of l, and a dropout of 0.5;

[0113] The second-layer BiTCN model contains forward time blocks and reverse time blocks. Each time block has an input dimension of 64, an output dimension of 64, a causal convolution kernel size of 3, a dilation factor of 2, a padding size of 4, a stride of l, and a dropout of 0.5;

[0114] (3) The BiGRU module in this example contains a total of 2 layers of BiGRU models:

[0115] The first-layer BiGRU model has an input dimension of 14, a forward GRU hidden unit number of 32, a backward GRU hidden unit number of 32, and an output dimension of 64.

[0116] The second-layer BiGRU model has an input dimension of 64, a forward GRU hidden unit number of 64, a backward GRU hidden unit number of 64, and an output dimension of 128.

[0117] The bidirectional parameter of each layer is true.

[0118] (4) Cross-attention mechanism module: the output of the BiTCN layer is used as the query, and the output of the BiGRU layer is used as the key and value. The dot product operation is first performed on the query and key, and the attention weight is obtained through the softmax function. Then, the attention weight is weighted and summed over the value sequence.

[0119] (5) Model output: fully connected layer, output layer. The fully connected layer parameters include weights and biases. The weight matrix shape is (128, 1), the bias is 1, and the output dimension is 1.

[0120] In summary, the overall process of the BiTCN-Ateention-BiGRU model is as follows: first, the global spatial features are extracted through the BiTCN network, and at the same time, the data is extracted through the BiGRU network to extract the temporal features, and then the cross-attention mechanism is used to fuse the spatial and temporal features. By calculating the attention weights, feature enhancement fusion is performed, and the information in the sequence is aggregated through sequence average pooling. Finally, it passes through the fully connected layer for high-precision prediction.

[0121] 8. Model training: Set the epoch number to 50, batch_size to 64, the ratio of training set to test set to 8:2, the training set has 13937 items, the test set has 3484 items, use sliding window prediction, the window size is set to 12, the loss function is MSE, the optimizer is Adam, and the learning rate is 0.003

[0122] 9. Output forecast results: The forecast result is the load value for the next day, and the evaluation index is the determination coefficient R 2 , root mean square error RMSE, mean absolute error MAE, to make an overall evaluation of the fitting of the model prediction. The calculation formulas of the three indicators are as follows:

[0123]

[0124]

[0125]

[0126] 10. Set up comparative experiments and ablation experiments, and predict evaluation indicators: Use TCN, LSTM, RIME-TCN, and RIME-LSTM as comparative experiments. Use RIME-BiTCN-A-BiGRU as ablation experiments.

[0127] The comparison of BiTCN-Ateention-BiGRU models is shown below Figure 8 As shown in the figure, there are four curves: true value, TCN, LSTM, and BiTCN-Ateention-BiGRU. The RIME algorithm is added to each model for optimization, and the model comparison chart after RIME optimization is obtained as shown in the figure. Fig. 9 As shown, there are four curves including true value, RIME-TCN, RIME-LSTM, and RIME-BiTCN-A-BiGRU. The specific indicator results are shown in Table 1 below.

[0128] The hardware environment of this experiment is Core i7-9700K, memory is 16G, operating system is Window10, GPU model is NVIDIA Ge Force RTX2060, development framework is Pytorch, compilation language is Python3.7.2, and development environment is PyCharm

[0129] Table 1 Test results comparison table

[0130]

[0131] From Table 1, through R 2, RMSE, and MAE, it can be seen that the BiTCN-A-BiGRU model in this example is significantly better than the TCN model and the LSTM model. 2 As high as 0.96, but there is still room for improvement. After the RIME algorithm optimizes each model, the evaluation indicators of each model have increased significantly. It can be seen that the prediction accuracy and reliability of the RIME optimization algorithm are good, and the R 2 The accuracy is 0.98, the RMSE is 0.44, and the MAE is 0.55. It can be seen that the model in this example has high accuracy and stability in data fitting and prediction, and has a stronger ability to interpret data, and can well capture the trends and characteristics of the data. In summary, the various indicators of the method of the present invention are generally improved compared with the prediction method, and the RIME optimization algorithm effectively improves the performance of the model and further improves the prediction accuracy.

[0132] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A load forecasting method that integrates temporal and spatial features with an attention mechanism, characterized in that: The steps include: Obtain a power load data set, normalize or standardize the power load data, use a sliding window to process the power load data, and generate a training set and a test set; Collect local weather data sets, build a random forest model, and use the feature importance method of the random forest algorithm to sort the feature importance and draw a feature importance graph; The top three weather features are selected according to their importance, and the variational mode decomposition VMD is applied to decompose the power load data to obtain the subsequence components with the least complexity. In the application of variational mode decomposition VMD to decompose the power load data, the sample entropy is selected as the optimal fitness function, and it is decomposed into multiple modal components IMF. The sample entropy value of each IMF is calculated, and the number of modal components K and the penalty factor alpha are iteratively optimized by the RIME algorithm, and the center frequency, bandwidth and mode of each mode are updated. When the convergence condition is met, the iterative process stops, the optimal parameters are obtained, and the optimal parameters are used to decompose VMD. In particular, The iterative optimization of the parameters in VMD using the RIME algorithm includes: first, taking the load value as the original signal, specifying the parameters to be optimized as the number of modal components K and the penalty factor alpha, then selecting the sample entropy as the fitness function to quantitatively evaluate the quality of the parameter combination, and before starting the optimization, first presetting an individual extreme value for each parameter that represents the optimal solution of each parameter in the current iteration process, and then setting the population size and the maximum number of iterations as the termination conditions for the optimization, calling the RIME algorithm, and iteratively updating the speed information and position information of each parameter. In each iteration, according to the fitness function The calculation results are used to update the individual extreme values ​​of each parameter, find the global optimal parameters, and then use the global optimal parameters to perform VMD decomposition on the load value to obtain the subsequence component with the minimum complexity. The RIME optimization algorithm is called, and the sample entropy is selected as the optimal fitness function. The sample entropy value of each IMF is calculated, the number of iterations and the population size are set, and the optimization range of the modal component number K and the penalty factor alpha is set. The RIME algorithm is used to find the optimal solution within the range, and the optimized modal component number K and penalty factor alpha are automatically obtained. The optimal parameters are used to decompose VMD to obtain multiple optimized IMF values; According to the decomposed subsequence components, combined with the first three selected weather features as model input, an energy consumption prediction model integrating spatiotemporal features with attention mechanism is constructed. The BiTCN model and BiGRU model are used to operate in parallel and receive the input sequence at the same time. After the BiTCN model and BiGRU model operate in parallel and receive the input sequence at the same time, the forward and reverse sequence information of the fusion are obtained respectively. The sequence information of the BiTCN model is used as the query, and the sequence information of the BiGRU model is used as the key and value to be passed to the cross attention layer for feature enhancement. The output is output through the fully connected layer to realize the prediction. Among them, Constructing an energy consumption prediction model that integrates spatiotemporal features with an attention mechanism includes: in the BiTCN model, firstly, passing the sequence data X=(x1, x2, ..., xT) as input to the forward TCN to obtain a forward output result R; in order to capture the reverse semantic information of the sequence, the original sequence X is reversed to obtain X'=(xT, xT-1, ..., x1), and input it into the reverse TCN to obtain a reverse output result R'; then, the forward output R and the reverse output R' are integrated, and the integrated result is normalized to obtain a temporal feature that integrates the forward and reverse semantic information; at the same time, the sequence data is passed to the BiGRU model to capture the temporal features of the sequence, and multi-feature fusion is performed; then, the attention weights between different positions are calculated through the cross attention mechanism, the input sequence is weighted summed, and the key features are input into the fully connected layer for prediction to obtain the prediction result.

2. The load forecasting method of integrating spatiotemporal features with the attention mechanism according to claim 1 is characterized in that: The weather data set contains 11 dimensions of weather features. The first 10 weather features are selected as input variables, and the load value at the current moment is selected as the target variable. The data set is divided into a training set and a test set. The weather features include date, temperature, air pressure, relative humidity, water vapor content per kilogram of air, light radiation, air density, wind speed, wind direction, rainfall, and load value.

3. The load forecasting method of integrating spatiotemporal features with attention mechanism according to claim 1 is characterized in that: The power load data is subjected to Z-Score standardization. The Z-Score function is as follows: (1) In the formula, is the current i-th data point, is the mean value of the entire data set, σ is the standard deviation of the entire data set, is the standardized score of the ith data point.

4. The load forecasting method of integrating spatiotemporal features with attention mechanism according to claim 1 is characterized in that: During VMD decomposition, the values ​​of K and alpha are preset to obtain multiple IMF values. The sample entropy of formula (2) is selected as the fitness function to minimize the entropy value. The sample entropy calculation formula is as follows: (2) In the formula, N represents the total number of data points, r represents the similarity threshold, They represent the probability that the distance between vectors is less than r when the embedding dimension is m and m+1, respectively. Represents the sample entropy of a time subsequence of length m.

5. The load forecasting method of integrating spatiotemporal features with attention mechanism according to claim 1 is characterized in that: The energy consumption prediction model that integrates the attention mechanism with spatiotemporal features includes a bidirectional temporal convolutional network module, a cross-attention mechanism module, and a bidirectional gated recurrent unit. The bidirectional temporal convolutional network module adds a weight normalization layer, an activation function, a Dropout layer, and a bidirectional time block to the TCN convolutional layer. After the input data is processed by two layers of dilated causal convolutional layers, a direct mapping of the input is added for residual connection.

6. The load forecasting method of attention mechanism fusion spatiotemporal features according to claim 5 is characterized in that: The bidirectional time block contains two TCN modules. The input x passes through the forward time convolution block to obtain the result of forward propagation. At the same time, the input x is reversed in the time dimension, and then passes through the backward time convolution block to obtain the result of backward propagation. The results of forward and backward propagation are spliced ​​in the feature dimension, and the Dropout layer is applied to prevent overfitting to obtain the output result.

Images