Long time series power load forecasting method based on improved Informer model

Abstract

The long-term power load forecasting method based on the improved Informer model provided in this application specifically relates to the technical field of power load forecasting. This method uses a feature embedding layer and an SSI model to encode and embed meteorological features, scalar information, location information, time information, and trend information, respectively, to provide the model with a richer input feature representation. In addition, a method for evaluating the sparsity of the key-value matrix is used to perform secondary sparsity on the probabilistic sparse attention, thereby reducing the time complexity of the attention mechanism from O(LlnL) to O((lnL) 2 ), and finally output the final prediction value through the decoder and encoder; this method solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module in the original Informer model when applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term power load forecasting.

Description

Technical Field

[0001] The present invention relates to the technical field of power load forecasting, and in particular to a long-time series power load forecasting method based on an improved Informer model. Background Art

[0002] With the rapid development of society and the continuous advancement of science and technology, the demand for electricity is growing in areas such as residential life, industrial production, commercial activities, information and communications, and traffic management. In particular, the widespread adoption of various smart technologies and products has further increased society's demand for electricity. However, electricity is an immediate energy source that is difficult to store in large quantities for long periods of time. Therefore, it is necessary to find a balance between electricity production and use. When power generation exceeds actual demand, unnecessary waste results. When power generation falls short of demand, power shortages, widespread power outages, emergency failures of power equipment, and a series of safety issues can occur. Accurate power load forecasting can help power plants and power supply departments find the balance between power supply and demand, thereby avoiding these problems. Therefore, research on power load forecasting methods can help power operators effectively plan power supply, reduce power waste, lower costs, ensure the smooth operation of the power system, and ultimately provide a safe and stable power supply, which is of great significance to social production and life.

[0003] Current deep learning models, such as RNNs and LSTMs, have strong representational learning capabilities and improve load forecasting accuracy. However, when processing extremely long sequences, these models face challenges such as high computational complexity, long training times, and difficulty capturing long-term dependencies. Furthermore, these models often ignore the temporal granularity of time series data, resulting in inaccurate forecasts.

[0004] In summary, how to improve the accuracy and efficiency of long-term power load forecasting is the problem that this application needs to solve. Summary of the Invention

[0005] The main purpose of the present invention is to provide a long-time series power load forecasting method based on an improved Informer model, which solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0006] In order to achieve the above objectives, the present invention provides a long-time series power load forecasting method based on an improved Informer model, so as to improve the accuracy and efficiency of long-time series power load forecasting.

[0007] The present invention provides a long-time series power load forecasting method based on an improved Informer model, comprising:

[0008] Obtaining time series data and an SSI model, extracting feature information of the time series data through a feature embedding module, and calculating the feature information to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features, and location features; the SSI model is used to indicate a model based on an improved informer model, and the SSI model includes: the feature embedding module, an encoder module, a decoder module, and an output layer;

[0009] The encoder module filters the first output information, performs secondary sparse processing, and performs attention distillation processing to obtain the second output information;

[0010] Processing the second output information through a decoder module to obtain a high-dimensional feature vector, wherein the decoder includes: a probabilistic sparse self-attention module and a multi-head attention module;

[0011] The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

[0012] Optionally, before acquiring the time series data and the SSI model, the SSI model can be constructed. The process of constructing the SSI model is as follows: first, data preprocessing is involved, including missing value filling and data normalization, and then the data is divided into a training set and a test set. The time series data is input into the feature embedding layer, and the XGBoost model is used to select features of the meteorological information, and the position encoder, time encoder, trend encoder and scalar encoder are used to extract features of each dimension to enrich the feature information of the model. These features are input into the encoder part of the model. The encoder consists of a multi-headed quadratic sparse ProbSparse Self-Attention mechanism and a distillation mechanism. It screens out important query matrices and key matrices, and reduces the dimension through self-attention distillation to capture the long-term correlation of the load series. The output of the encoder is then input into the decoder. The decoder contains hidden multi-headed probabilistic sparse attention and multi-headed attention. Finally, the high-dimensional output of the decoder is input into the output layer to output the final predicted value.

[0013] Optionally, the time series data and the SSI model are acquired, feature information of the time series data is extracted through a feature embedding module, and the feature information is calculated to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features and location features, and the SSI model is used to indicate a model based on an improved Informer model, and the SSI model includes: the feature embedding module, an encoder module, a decoder module and an output layer, including:

[0014] Analyzing the meteorological information in the time series data by using an XGBoost model to obtain the meteorological characteristics;

[0015] Processing the time series data in the time series data by a global time encoder to obtain the global time feature;

[0016] Acquire the trend feature in the time series data by a trend encoder;

[0017] Acquire the scalar feature and the position feature in the time series data by direct observation and position encoder;

[0018] The meteorological features, global time features, trend features, scalar features and location features are linearly added to obtain first output information.

[0019] Optionally, analyzing the meteorological information in the time series data by using an XGBoost model to obtain the meteorological characteristics includes:

[0020] The meteorological information is selected using the XGBoost model, and the feature with the highest score is selected as the meteorological feature to be used in model calculation.

[0021] Optionally, the processing of the time series data in the time series data by a global time encoder to obtain a global time feature includes:

[0022] The global time feature is obtained by splitting and encoding the time series data into six time granularities: year, month, day, hour, minute, and holiday.

[0023] Optionally, obtaining the trend feature in the time series data by using a trend encoder includes:

[0024] Construct a matrix D using the time series data at time t over the past h days;

[0025] Constructing a matrix P using the time series data of the past h weeks at time t and constructing a matrix M using the time series data of the past h months at time t;

[0026] Then D, P and M are fused to form a trend matrix, and finally convolution is used to extract the trend features.

[0027] Optionally, the acquiring of scalar features and position features in the time series data by direct observation and position encoder includes:

[0028] Extracting the scalar features from the time series data by direct observation;

[0029] The position feature in the time series data is extracted by a position encoder.

[0030] Optionally, the obtaining of the second output information after filtering, performing secondary sparsification and attention distillation processing on the first output information by the encoder module includes:

[0031] Filtering the first output information to obtain dominant query data by calculating the similarity between the self-attention distribution and the uniform distribution;

[0032] Performing secondary sparse processing on the dominant query data by a secondary sparse probabilistic sparse attention method to obtain secondary sparse data;

[0033] A distillation operation is performed on the secondary sparse data to obtain the second output information.

[0034] The long-term power load forecasting method based on the improved Informer model provided in this application is characterized by encoding and embedding meteorological features, scalar information, location information, time information and trend information respectively through the feature embedding layer, providing a richer input feature representation for the model, and adopting the method of sparsity evaluation of the key-value matrix to perform secondary sparseness on the probabilistic sparse attention, thereby reducing the time complexity of the attention mechanism from O(LlnL) to O((lnL) 2 ), and finally output the final prediction value through the decoder and encoder; this method solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module in the original Informer model when applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term power load forecasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0036] Figure 1 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 1 ;

[0037] Figure 2 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 2 ;

[0038] Figure 3 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 3 .

[0039] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] The terms "first," "second," "third," "fourth," and so forth (if any) in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0042] In the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] In recent years, with the rapid development of society and the continuous advancement of science and technology, the demand for electricity in areas such as residential life, industrial production, commercial activities, information and communications, and traffic management has been growing. In particular, the widespread adoption of various smart technologies and products has further increased society's demand for electricity. However, electricity is an immediate energy source that is difficult to store in large quantities for long periods of time. Therefore, it is necessary to find a balance between electricity production and use. When power generation exceeds actual demand, unnecessary waste occurs. When power generation falls short of actual demand, power shortages, large-scale power outages, emergency failures of power equipment, and a series of safety issues can occur. Accurate power load forecasting can help power plants and power supply departments find the balance between power supply and demand, thereby avoiding these problems. Therefore, power load forecasting is a key component of power system operation and planning, and is of great significance for ensuring stable operation and optimizing resource allocation. Research on power load forecasting methods can help power operators effectively plan power supply, reduce power waste, lower costs, ensure stable operation of the power system, and ultimately provide a safe and stable power supply, which is of great significance to social production and life.

[0044] Current deep learning models, such as RNNs and LSTMs, or more complex neural network models with larger parameters for training and inference, offer strong representational learning capabilities and improve load forecasting accuracy. However, these models face challenges when processing extremely long sequences, including high computational complexity, lengthy training times, and difficulty capturing long-term dependencies. Furthermore, these models often ignore the temporal granularity of time series data, resulting in inaccurate forecasts.

[0045] In summary, how to improve the accuracy and efficiency of long-term power load forecasting is the problem that this application needs to solve.

[0046] The long-term power load forecasting method based on the improved Informer model provided in this application is characterized by encoding and embedding meteorological features, scalar information, location information, time information and trend information respectively through the feature embedding layer, providing a richer input feature representation for the model, and adopting the method of sparsity evaluation of the key-value matrix to perform secondary sparseness on the probabilistic sparse attention, thereby reducing the time complexity of the attention mechanism from O(LlnL) to O((lnL) 2 ), and finally output the final prediction value through the decoder and encoder; this method solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module in the original Informer model when applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term power load forecasting.

[0047] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 1 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 1 , the execution subject of this embodiment is the power load forecasting system, such as Figure 1 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0049] S101: Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information.

[0050] The feature information refers to meteorological features, global time features, trend features, scalar features and location features. The SSI model is used to indicate a model based on the improved Informer model. The SSI model includes: the feature embedding module, the encoder module, the decoder module and the output layer.

[0051] Meteorological features are obtained by using the XGBoost model to select meteorological information and choosing the feature with the highest score as the meteorological feature to be used in model calculation.

[0052] The global time feature is obtained by: t (X) provides global time information for the model. For data X, this paper splits and encodes it from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data with the current time of 2023-03-0315:10, the extracted global time feature is represented by a numerical vector [23, 3, 3, 15, 1, 0], which represents 23 years, March, 3rd, 15 hours, 10 minutes (60 minutes is divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1). The vector is input into the fully connected network to obtain the global time feature, and its formula is defined as:

[0053] x time =E t (X)

[0054] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0055] The trend feature is obtained by designing a trend encoder Es(X). The trend feature proposed in this paper mainly consists of three parts: using the load data of the past h days at time t to construct the matrix D, using the load data of the past h weeks at time t to construct the matrix P, and using the load data of the past h months at time t to construct the matrix M. Then D, P and M are fused to form the trend matrix T∈R 3×h Finally, convolution is used for feature extraction, and its trend coding is defined as:

[0056] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0057] The scalar feature and position feature are obtained by directly observing the scalar feature x sca Extraction, for the position feature x pos The extraction can be done by using the position encoder calculation method.

[0058] The calculation method for obtaining the first output information by calculating the characteristic information is:

[0059] The first output information is obtained by linearly adding the above-mentioned meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics.

[0060] The purpose of this step is to extract complete time features, periodicity and trend features to enhance the use of meteorological features when the model is applied to load forecasting tasks. These features play an important role in mining nonlinear long-distance dependencies between data and improving forecasting results.

[0061] Optionally, before acquiring the time series data and the SSI model, the SSI model can be constructed. The process of constructing the SSI model is as follows: first, data preprocessing is involved, including missing value filling and data normalization, and then the data is divided into a training set and a test set. The time series data is input into the feature embedding layer, and the XGBoost model is used to select features of the meteorological information, and the position encoder, time encoder, trend encoder and scalar encoder are used to extract features of each dimension to enrich the feature information of the model. These features are input into the encoder part of the model. The encoder consists of a multi-headed quadratic sparse ProbSparse Self-Attention mechanism and a distillation mechanism. It screens out important query matrices and key matrices, and reduces the dimension through self-attention distillation to capture the long-term correlation of the load series. The output of the encoder is then input into the decoder. The decoder contains hidden multi-headed probabilistic sparse attention and multi-headed attention. Finally, the high-dimensional output of the decoder is input into the output layer to output the final predicted value.

[0062] S102: The encoder module filters the first output information, performs secondary sparsification and attention distillation processing on the first output information, and obtains the second output information.

[0063] The second output information is obtained by filtering the first output information through the encoder module, performing secondary sparse processing and attention distillation processing, and then performing the second output information is completed by the following steps:

[0064] Step 1: Filter the first output information through the encoder module:

[0065] The dominant query in the first output information is screened by calculating the similarity between the self-attention distribution and the uniform distribution. The Kullback-Leibler divergence can be used to quantitatively calculate it, which is defined as:

[0066]

[0067] Where q i For the i-th query, k j is the jth key, N k is the number of keys. α represents the query q i The attention distribution of , which is expressed as follows:

[0068]

[0069] By removing the constant in the definition of KL divergence, a sparsity metric is designed to measure the significance of the query, which is defined as follows:

[0070]

[0071] The first term of this formula is q i The second is the arithmetic mean of the division results. i ,K) value is higher, indicating that q i The higher the probability of attention in the head field of the long-tail self-attention distribution, the higher the probability of including the dominant point pair. However, the complexity of the sparsity measure is still quadratic because it requires calculating the dot product of each query and each key. For this reason, the Informer model designs an alternative measurement method, which is defined as:

[0072]

[0073] Under the long-tail distribution, we only need to randomly select L k Ln k Dot product is calculated That is, fill the other pairs with zero. Finally, only select The self-attention calculation for each key is performed using the top s leading queries, defined as:

[0074]

[0075] Among them, Q is the query matrix of the dominant query, which consists of the data points of the dominant query, K is the key matrix, which provides the characteristics of the information required by the query, and V is the value matrix, which represents the actual value information. Contains s dominant queries, By assigning a specific value to s, the ProbSparse self-attention mechanism is used to reduce the time complexity and space complexity of self-attention from N 2 Reduced to N×s.

[0076] Step 2: The ProbSpare Self-Attention mechanism proposed in the present invention performs secondary sparseness on the data obtained in step 1 using the secondary sparse self-attention mechanism. The specific process is as follows:

[0077] First, the Kullback-Leibler divergence is used to evaluate the sparsity of K. The sparsity evaluation of the i-th key is defined as:

[0078]

[0079] Where Q is the query matrix of the dominant query, q j For the jth query, k j is the jth key, Q Q The first term of the formula is a Log-Sum-Exp for all keys, and the second term is their arithmetic mean. The higher the value, the better the k i The probability of attention in the head field of the long-tail self-attention distribution contains the probability of advantage point pairs. The more M(k i ,Q)high.

[0080] Based on the above evaluation method, the present invention performs sparsity evaluation on K, and then selects Top-u keys (u=lnL / c, c is a constant) from the results of the sparsity evaluation. These keys are the u keys with the highest activity in the sparsity evaluation, and are composed of these keys. Replace K in the original formula.

[0081] It should be noted that in the process of sparsity evaluation, if the sparsity score is calculated for each key, it will bring additional calculations. Therefore, we can use the assumption that the dot product result follows a long-tail distribution to calculate the sparsity score of each key. Randomly sample some qurey and key to calculate. This reduces the time loss caused by the sparsity evaluation of the key. Therefore, under the long-tail distribution, only U=lnQlnK dot product pairs need to be randomly sampled to calculate M(k i,Q), and fill the other dot product pairs with zeros. The reason for filling with zero values is that M(k i ,Q) is not very sensitive to zero values and is numerically stable.

[0082] Based on the above method, the input of the attention mechanism dot product operation is changed from K is converted into This makes the ProbSparse Self-Attention after quadratic sparseness only need O((lnL) 2 ) can be operated on the dot product of and The dot product operation is calculated It can perform matrix multiplication operation with V matrix. In addition, the present invention uses the median filling method to smooth S, and the filled S is recorded as Based on the above method, this paper proposes a new measurement method based on the attention mechanism proposed by the Informer model, which is defined as:

[0083]

[0084] Where: is a query matrix that only contains a few selected queries that contribute a large proportion to the attention value. is the key matrix with the highest activity ranking selected by the quadratic sparse ProbSpare Self-Attention, V is the value matrix, and d is the dimension of Q, K, and V.

[0085] Step 3: The specific process of processing the output of step 2 through the attention distillation mechanism is as follows:

[0086] After applying the quadratic sparse self-attention mechanism to the Q, K, and V matrices, an attention distillation operation is performed to reduce the feature dimension while maintaining the attention mechanism's high sensitivity to sequential relationships. The attention distillation operation is performed by a 1-D convolutional layer with an ELU activation function and passed through a max pooling layer with a stride of 2. The distillation operation is defined as:

[0087]

[0088] Where [.]AB represents an attention block. It contains a multi-head quadratic sparse ProbSparse Self-Attention block and basic operations. Conv1d(.) uses the ELU(.) activation function. By using distillation extraction, the dimensionality of the feature map of each head's probable sparse self-attention is reduced to half of its original dimensionality, and all output information is concatenated as the hidden representation of the model encoder.

[0089] As you can understand, the first output information after the feature embedding module is used as the input to the quadratic sparse ProbSpare Self-Attention encoder. The ProbSpare Self-Attention mechanism considers the sparsity of the self-attention matrix as manifested by different queries showing different levels of interest in the key (i.e., different activity levels, with higher self-attention scores). A dominant query is one whose self-attention distribution is more consistent with the long-tail distribution, and thus has more dominant dot products. The present invention uses a quadratic sparse probabilistic sparse attention method. In the present invention, experiments show that while the dot product pairs under the ProbSpare Self-Attention mechanism sparse the query matrix, some keys maintain a small correlation with almost all queries. When these keys are calculated with the query, the time complexity and space complexity of the self-attention module will increase, that is, the attention matrix after the sparse self-attention mechanism is still sparse. Therefore, the present invention performs a quadratic sparse processing on the attention score matrix, re-evaluates the self-attention mechanism under the ProbSpare Self-Attention mechanism, and realizes the quadratic sparseness of ProbSpare Self-Attention by sparsely processing the key-value matrix K. After the quadratic sparseness, the attention scores of the query and key are maintained at a relatively high score as a whole, which shows that the quadratic sparse method can indeed eliminate keys with low activity.

[0090] S103: Process the second output information through a decoder module to obtain a high-dimensional feature vector.

[0091] The method of processing the second output information by the decoder module to obtain a high-dimensional feature vector is:

[0092] Next, the second output information obtained by the encoder is used as part of the input to the decoder. The decoder consists of a probabilistic sparse self-attention layer and a multi-head self-attention layer. Here, the masked dot product is directly set to -∞, and the output of the multi-head self-attention layer is concatenated with the result of the encoder. The obtained high-dimensional vector is then represented as follows:

[0093] X decoder =Concat(MultiHeadAttention(Q de ,K de ,V de ),X encoder )

[0094] Among them, Qde , K de 、V de are respectively the query, key and value vectors of the decoder, X ender is the output of the encoder.

[0095] S104: Using the high-dimensional feature vector as input of an output layer, and converting the high-dimensional feature vector into a final prediction value through the output layer.

[0096] The method of converting the high-dimensional feature vector into the final prediction value through the output layer is as follows: the high-dimensional feature vector obtained in step S103 is input to the fully connected layer, i.e., the output layer, whose output is fully connected with the load value to obtain the final result. The input of the output layer at time t is the concatenation of the two parts, and the final prediction value is defined as:

[0097]

[0098] in, is the length L token The start tag of , as shown in this definition:

[0099]

[0100] The target sequence length is L y The placeholder is set to 0. Therefore, the length of the decoder input sequence is x token The sum of the lengths of x0, L de =L token +L y .

[0101] The embodiment of the present application provides a long-time series power load forecasting method based on an improved Informer model. The method extracts feature information of time series data through a feature embedding module, obtains first output information after linear addition processing of the feature information, obtains second output information after screening, secondary sparseness and attention distillation mechanism processing of the first output information through an encoder module, obtains a high-dimensional feature vector after processing the second output information through a decoder module, uses the high-dimensional feature vector as the input of the output layer, and converts the high-dimensional feature vector into the final prediction value through the output layer. Through this method, the method solves the problem of insufficient feature richness when the original Informer model is applied to power load forecasting and the problem of insufficient sparsity of the probabilistic sparse attention module, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0102] Figure 2 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 2 ,like Figure 2 As shown, this embodiment Figure 1 Based on the embodiment, the long time series power load forecasting method based on the improved Informer model is described in detail. Figure 2 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0103] S201: Analyze meteorological information in time series data through the XGBoost model to obtain meteorological characteristics.

[0104] Among them, the meteorological characteristics have an important impact on load forecasting, but some meteorological information has a weak correlation with load forecasting or has information redundancy, which will reduce the performance of the prediction model. Therefore, in the meteorological feature extraction part, the present invention uses XGBoost to extract meteorological information X. wth Perform feature selection. These meteorological information include weather, temperature, humidity, wind speed, etc. According to the feature frequency, select the top_f features with the highest feature importance score and add them to the feature subset T wth In the example, the fully connected layer is used as the meteorological encoder E w (X), and finally T wth ∈R Top_f×t Enter E w (X) The meteorological characteristics are obtained, and the meteorological characteristics are expressed as follows:

[0105] x wth =E w (T wth )

[0106] S202: Processing the time series data in the time series data through a global time encoder to obtain global time features.

[0107] Among them, the present invention provides global time information for the model by designing a global time encoder. For time series data, this paper splits and encodes from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data whose current time is 2023-03-03 15:10, the extracted global time feature is represented by a numerical vector as [23, 3, 3, 15, 1, 0], which represents 23 years, March, 3rd, 15 hours, 10 minutes, (60 minutes are divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1), and the vector is input into the fully connected network to obtain the global time feature. Its formula is defined as:

[0108] x time =E t (X)

[0109] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0110] It is understandable that load forecasting is a classic time series forecasting task, and its global time characteristics play an important role in the model learning the dependency of the time dimension of data. Therefore, a global time encoder is designed to provide global time information for the model.

[0111] S203: Obtain the trend features in the time series data through a trend encoder.

[0112] The specific implementation method of obtaining the trend features in the time series data through the trend encoder is as follows: the trend features are obtained by the trend encoder Es(X) designed by the present invention. The trend features proposed in this application mainly include three parts: using the load data of the past h days at time t to construct the matrix D, using the load data of the past h weeks at time t to construct the matrix P, and using the load data of the past h months at time t to construct the matrix M. Then D, P and M are fused to form the trend matrix T∈R 3×h Finally, convolution is used for feature extraction, and its trend coding is defined as:

[0113] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0114] It can be understood that the purpose of this step is to better reflect the periodicity and trend of the load data.

[0115] S204: Acquire scalar features and position features in the time series data through direct observation and position encoder.

[0116] Among them, the scalar feature x sca It can be obtained from time series data through direct observation.

[0117] Position feature x pos The specific method of obtaining it is as follows:

[0118] The input representation is an embedding of three independent parts and a scalar projection, as well as local and global timestamps.

[0119] For time t The value of the input sequence indicates that the method first passes d modle The convolution filter projects it onto a d modle -dim vector In the example, the kernel size is 3 and the stride is 1, where d modle is the dimensionality of the input representation.

[0120] Then, the fixed position embedding at time t is used to retain the local context and obtain the position information. Its position feature is defined as:

[0121]

[0122] Where pos=1,2,…,L x and j = 1, 2, ..., d modle . L x is the length of the input sequence, L x =L en or L de denote the input of the encoder or decoder of the proposed model, respectively.

[0123] The purpose of this step is to obtain global hierarchical timestamps (such as weeks, months, and years) and agnostic timestamps (such as holidays and others), which are necessary to obtain long-term independence for long-series time series forecasts.

[0124] S205: Perform linear addition calculation on the meteorological features, the global time features, the trend features, the scalar features and the location features to obtain first output information.

[0125] The specific implementation of this step is as follows: the meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics obtained in steps S201-205 are linearly added to obtain the input of the model. The model input is defined as:

[0126] x input =Add(x wth ,x time ,x tre, x pos ,x sca )

[0127] S206: Filter the first output information to obtain dominant query data by calculating the similarity between the self-attention distribution and the uniform distribution.

[0128] The specific implementation of filtering the dominant query in the first output information by calculating the similarity between the self-attention distribution and the uniform distribution can be quantitatively calculated using the Kullback-Leibler divergence, which is defined as:

[0129]

[0130] Where q i For the i-th query, k j is the jth key, N k is the number of keys. α represents the query q i The attention distribution of , which is expressed as follows:

[0131]

[0132] By removing the constant in the definition of KL divergence, a sparsity metric is designed to measure the significance of the query, which is defined as follows:

[0133]

[0134] The first term of this formula is q i The second is the arithmetic mean of the division results. i ,K) value is higher, indicating that q i The higher the probability of attention in the head field of the long-tail self-attention distribution, the higher the probability of including the dominant point pair. However, the complexity of the sparsity measure is still quadratic because it requires calculating the dot product of each query and each key. For this reason, the Informer model designs an alternative measurement method, which is defined as:

[0135]

[0136] Under the long-tail distribution, we only need to randomly select L k Ln k Dot product is calculated That is, fill the other pairs with zero. Finally, only select The self-attention calculation for each key is performed using the top s leading queries, defined as:

[0137]

[0138] Among them, Q is the query matrix of the dominant query, which consists of the data points of the dominant query, K is the key matrix, which provides the characteristics of the information required by the query, and V is the value matrix, which represents the actual value information. Contains s dominant queries, By assigning a specific value to s, the ProbSparse self-attention mechanism is used to reduce the time complexity and space complexity of self-attention from N 2 Reduced to N×s.

[0139] S207: Perform secondary sparse processing on the dominant query data using a secondary sparse probabilistic sparse attention method to obtain secondary sparse data.

[0140] The specific process of performing secondary sparse processing on the dominant query data by the secondary sparse probabilistic sparse attention method to obtain secondary sparse data is as follows:

[0141] First, the Kullback-Leibler divergence is used to evaluate the sparsity of K. The sparsity evaluation of the i-th key is defined as:

[0142]

[0143] Where Q is the query matrix of the dominant query, q j For the jth query, k j is the jth key, Q Q The first term of the formula is a Log-Sum-Exp for all keys, and the second term is their arithmetic mean. The higher the value, the better the k i The probability of attention in the head field of the long-tail self-attention distribution contains the probability of advantage point pairs. The more M(k i ,Q)high.

[0144] Based on the above evaluation method, the present invention performs sparsity evaluation on K, and then selects Top-u keys (u=lnL / c, c is a constant) from the results of the sparsity evaluation. These keys are the u keys with the highest activity in the sparsity evaluation, and are composed of these keys. Replace K in the original formula.

[0145] It should be noted that in the process of sparsity evaluation, if the sparsity score is calculated for each key, it will bring additional calculations. Therefore, we can use the assumption that the dot product result follows a long-tail distribution to calculate the sparsity score of each key. Randomly sample some qurey and key to calculate. This reduces the time loss caused by the sparsity evaluation of the key. Therefore, under the long-tail distribution, only U=lnQlnK dot product pairs need to be randomly sampled to calculate M(k i ,Q), and fill the other dot product pairs with zeros. The reason for filling with zero values is that M(k i ,Q) is not very sensitive to zero values and is numerically stable.

[0146] Based on the above method, the input of the attention mechanism dot product operation is changed from K is converted into This makes the ProbSparse Self-Attention after quadratic sparseness only need O((lnL) 2 ) can be operated on the dot product of and The dot product operation is calculated It can perform matrix multiplication operation with V matrix. In addition, the present invention uses the median filling method to smooth S, and the filled S is recorded as Based on the above method, this paper proposes a new measurement method based on the attention mechanism proposed by the Informer model, which is defined as:

[0147]

[0148] Where: is a query matrix that only contains a few selected queries that contribute a large proportion to the attention value. is the key matrix with the highest activity ranking selected by the quadratic sparse ProbSpare Self-Attention, V is the value matrix, and d is the dimension of Q, K, and V.

[0149] S208: Perform a distillation operation on the secondary sparse data to obtain second output information.

[0150] The specific process of performing the distillation operation on the secondary sparse data to obtain the second output information is as follows:

[0151] After applying the quadratic sparse self-attention mechanism to the Q, K, and V matrices, an attention distillation operation is performed to reduce the feature dimension while maintaining the attention mechanism's high sensitivity to sequential relationships. The attention distillation operation is performed by a 1-D convolutional layer with an ELU activation function and passed through a max pooling layer with a stride of 2. The distillation operation is defined as:

[0152]

[0153] Where [.]AB represents an attention block. It contains a multi-head quadratic sparse ProbSparse Self-Attention block and basic operations. Conv1d(.) uses the ELU(.) activation function. By using distillation extraction, the dimensionality of the feature map of each head's probable sparse self-attention is reduced to half of its original dimensionality, and all output information is concatenated as the hidden representation of the model encoder.

[0154] S209: Process the second output information through a decoder module to obtain a high-dimensional feature vector.

[0155] S210: The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

[0156] Among them, steps S209-S210 are similar to steps S103-S104 and will not be repeated here.

[0157] The embodiment of the present application provides a long-term power load forecasting method based on an improved Informer model. The method analyzes meteorological information in time series data through an XGBoost model to obtain meteorological features, processes the time series data in the time series data through a global time encoder to obtain global time features, obtains the trend features in the time series data through a trend encoder, obtains scalar features and position features in the time series data through direct observation and position encoders, obtains first output information after linear addition processing on the above feature information, obtains second output information after filtering, secondary sparseness and attention distillation mechanism processing on the first output information through an encoder module, processes the second output information through a decoder module to obtain a high-dimensional feature vector, uses the high-dimensional feature vector as the input of the output layer, and converts the high-dimensional feature vector into the final prediction value through the output layer. Through this method, the method solves the problem of insufficient feature richness and insufficient sparsity of the probability sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term power load forecasting.

[0158] Figure 3 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 3 , this embodiment is in Figure 2 Based on the embodiment, “analyzing meteorological information in time series data by XGBoost model to obtain meteorological features”, “processing time series data in time series data by global time encoder to obtain global time features”, “obtaining the trend features in time series data by trend encoder”, and “obtaining scalar features and position features in time series data by direct observation and position encoder” are described in detail, such as Figure 3 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0159] S301: Use the XGBoost model to select meteorological information and select the feature with the highest score as the meteorological feature to be used in model calculation.

[0160] Among them, in the meteorological feature extraction part, the present invention uses XGBoost to extract meteorological information X wth Perform feature selection. These meteorological information include weather, temperature, humidity, wind speed, etc. According to the feature frequency, select the top_f features with the highest feature importance score and add them to the feature subset T wth In the example, the fully connected layer is used as the meteorological encoder E w (X), and finally T wth ∈R Top_f×tEnter E w (X) The meteorological characteristics are obtained, and the meteorological characteristics are expressed as follows:

[0161] x wth =E w (T wth )

[0162] It is understandable that the meteorological characteristics have an important impact on load forecasting, but some meteorological information has a weak correlation with load forecasting or has information redundancy, which will reduce the performance of the prediction model. Therefore, the present invention uses XGBoost to optimize the meteorological information XGBoost. wth Perform feature selection.

[0163] S302: Obtain global time features by splitting and encoding the time series data into six time granularities: year, month, day, hour, minute, and holiday.

[0164] Among them, the present invention provides global time information for the model by designing a global time encoder. For time series data, this paper splits and encodes from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data whose current time is 2023-03-03 15:10, the extracted global time feature is represented by a numerical vector as [23, 3, 3, 15, 1, 0], which represents 23 years, March, 3rd, 15 hours, 10 minutes, (60 minutes are divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1), and the vector is input into the fully connected network to obtain the global time feature. Its formula is defined as:

[0165] x time =E t (X)

[0166] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0167] It is understandable that load forecasting is a classic time series forecasting task, and its global time characteristics play an important role in the model learning the dependency of the time dimension of data. Therefore, a global time encoder is designed to provide global time information for the model.

[0168] S303: Construct a matrix D using the time series data of the past h days at time t.

[0169] S304: Build a matrix P using the time series data of the past h weeks at time t and build a matrix M using the time series data of the past h months at time t.

[0170] S305: D, P, M and are fused to form a trend matrix, and finally convolution is used to extract trend features.

[0171] Among them, steps S303 and S304 are routine operations for those skilled in the art and can be implemented using conventional matrix construction techniques, so they will not be described in detail here.

[0172] The specific implementation method of fusing D, P, M and π to form a trend matrix and finally using convolution to extract trend features is as follows:

[0173] The specific implementation method of obtaining the trend features in time series data through the trend encoder is as follows: the trend features are obtained by the trend encoder Es(X) designed by the present invention. The trend features proposed in this application mainly include three parts: using the load data of the past h days at time t to construct the matrix D, using the load data of the past h weeks at time t to construct the matrix P, and using the load data of the past h months at time t to construct the matrix M. Then D, P and M are fused to form the trend matrix T∈R 3 ×h Finally, convolution is used for feature extraction, and its trend coding is defined as:

[0174] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0175] It can be understood that the purpose of this step is to better reflect the periodicity and trend of the load data.

[0176] S306: Extracting scalar features from time series data through direct observation.

[0177] Among them, for the scalar feature x sca , those skilled in the art can obtain it directly from time series data.

[0178] S307: Extracting position features from the time series data through a position encoder.

[0179] Among them, the position feature x pos The specific method of obtaining it is as follows:

[0180] The input representation is an embedding of three independent parts and a scalar projection, as well as local and global timestamps.

[0181] For time t The value of the input sequence indicates that the method first passes d modle The convolution filter projects it onto a d modle -dim vector In the example, the kernel size is 3 and the stride is 1, where d modle is the dimensionality of the input representation.

[0182] Then, the fixed position embedding at time t is used to retain the local context and obtain the position information. Its position feature is defined as:

[0183]

[0184] Where pos=1,2,…,L x and j = 1, 2, ..., d modle . L x is the length of the input sequence, L x =L en or L de denote the input of the encoder or decoder of the proposed model, respectively.

[0185] The purpose of this step is to obtain global hierarchical timestamps (such as weeks, months, and years) and agnostic timestamps (such as holidays and others), which are necessary to obtain long-term independence for long-series time series forecasts.

[0186] The embodiment of the present application provides a long-term power load forecasting method based on an improved Informer model. The method analyzes the meteorological information in the time series data through the XGBoost model to obtain meteorological features, processes the time series data in the time series data through a global time encoder, obtains global time features, so as to obtain global hierarchical timestamps (such as weeks, months and years) and unknowable timestamps (such as holidays and others), obtains the trend features in the time series data through a trend encoder, obtains scalar features and position features in the time series data through direct observation and position encoders, obtains the first output information after linear addition processing of the above feature information, obtains the second output information after filtering, secondary sparseness and attention distillation mechanism processing of the first output information through the encoder module, obtains the second output information through the decoder module, obtains a high-dimensional feature vector, uses the high-dimensional feature vector as the input of the output layer, and converts the high-dimensional feature vector into the final predicted value through the output layer. Through this method, the method solves the problem of insufficient feature richness and insufficient sparsity of the probability sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term power load forecasting.

[0187] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0188] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0189] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0190] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A long-time series power load forecasting method based on an improved Informer model, characterized in that: Applied to a power load forecasting system, the method includes: Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features, and position features. The SSI model is used to indicate a model improved based on an Informer model. The SSI model includes: the feature embedding module, an encoder module, a decoder module, and an output layer. Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information, specifically including: analyzing meteorological information in the time series data through an XGBoost model to obtain the meteorological features; processing the time series data in the time series data through a global time encoder to obtain the global time features; obtaining the trend features in the time series data through a trend encoder; obtaining the scalar features and the position features in the time series data through direct observation and a position encoder; and performing linear addition calculation on the meteorological features, global time features, trend features, scalar features, and position features to obtain the first output information. The encoder module filters the first output information, performs secondary sparsification and performs attention distillation processing on the first output information, and obtains the second output information. The encoder module filters the first output information, performs secondary sparsification and performs attention distillation processing on the first output information, and obtains the second output information. The method further comprises: filtering the first output information by calculating the similarity between the self-attention distribution and the uniform distribution to obtain the dominant query data; performing secondary sparsification processing on the dominant query data by a secondary sparse probabilistic sparse attention method to obtain secondary sparse data; and performing a distillation operation on the secondary sparse data to obtain the second output information. Processing the second output information by the decoder module to obtain a high-dimensional feature vector; The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

2. The method according to claim 1, characterized in that The analyzing the meteorological information in the time series data by the XGBoost model to obtain the meteorological characteristics includes: The meteorological information is selected using the XGBoost model, and the feature with the highest score is selected as the meteorological feature to be used in model calculation.

3. The method according to claim 1, characterized in that The processing of the time series data in the time series data by a global time encoder to obtain a global time feature includes: The global time feature is obtained by splitting and encoding the time series data into six time granularities: year, month, day, hour, minute, and holiday.

4. The method according to claim 1, wherein The step of obtaining the trend feature in the time series data by using a trend encoder includes: Construct a matrix using the time series data from the past h days at time t ; Construct a matrix using the time series data from the past h weeks at time t And use the time series data of the past h months at time t to construct a matrix ; Will 、 and The trend matrix is formed by fusion, and finally the trend features are extracted by convolution.

5. The method according to claim 1, wherein The obtaining of the scalar feature and the position feature in the time series data by direct observation and a position encoder comprises: Extracting the scalar features from the time series data by direct observation; The position feature in the time series data is extracted by a position encoder.

Images