Long-time sequence power load prediction method based on improved Informer model

Abstract

The invention provides a long-time sequence power load prediction method based on an improved Informer model, and particularly relates to the technical field of power load prediction, and the method comprises the steps: carrying out the feature coding and embedding of meteorological features, scalar information, position information, time information and trend information through a feature embedding layer and an SSI model, and carrying out the feature coding and embedding of the meteorological features, scalar information, position information, time information and trend information; the method comprises the following steps of: providing richer input feature representation for a model, carrying out secondary sparsity on probability sparse attention by adopting a method for carrying out sparsity evaluation on a key value matrix, reducing the time complexity of an attention mechanism from O (LlnL) to O ((lnL) 2), and finally outputting a final predicted value through a decoder and an encoder; according to the method, the problems that when an original Informer model is applied to power load prediction, feature richness is insufficient and a probability sparse attention module still has insufficient sparsity are solved, so that the effect of improving the precision and efficiency of long-time sequence power load prediction is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of power load forecasting, and in particular to a long-time series power load forecasting method based on an improved Informer model. Background Art

[0002] With the rapid development of social economy and the continuous advancement of science and technology, the demand for electricity in areas such as residents' lives, industrial production, commercial activities, information communication, and traffic management is growing. In particular, the widespread popularization of various intelligent technologies and products has further expanded the demand for electricity in society. However, electric energy is an instant energy that is difficult to store in large quantities for a long time. It is necessary to find a balance between the production and use of electric energy. When the power generation is greater than the actual power demand, it will cause unnecessary waste. When the power generation is less than the actual power demand, there will be power shortages, large-scale power outages, emergency failures of power equipment, and a series of safety problems. Accurate power load forecasting can help power plants and power supply departments find the balance between power supply and demand, thereby avoiding the above problems. Therefore, the research on power load forecasting methods can help power operation departments to effectively plan power supply, reduce power waste, reduce costs, ensure the smooth operation of the power system, and then provide safe and stable power supply, which is of great significance to social production and life.

[0003] Current deep learning models, such as RNN and LSTM, have strong representation learning capabilities and improve the accuracy of load forecasting. However, when processing very long sequences, they face problems such as large amount of calculation, long training time, and difficulty in capturing long-term dependencies. In addition, when processing time series data, these models usually ignore the time granularity information of the data, resulting in inaccurate prediction results.

[0004] In summary, how to improve the accuracy and efficiency of long-term series power load forecasting is the problem that this application needs to solve. Summary of the invention

[0005] The main purpose of the present invention is to provide a long-time series power load forecasting method based on an improved Informer model, which solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0006] In order to achieve the above objectives, the present invention provides a long-time series power load forecasting method based on an improved Informer model, so as to improve the accuracy and efficiency of long-time series power load forecasting.

[0007] The present invention provides a long-time series power load forecasting method based on an improved Informer model, comprising:

[0008] Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features, and location features, and the SSI model is used to indicate a model based on an improved Informer model, and the SSI model includes: the feature embedding module, an encoder module, a decoder module, and an output layer;

[0009] The encoder module filters the first output information, performs secondary sparse processing and attention distillation processing on the first output information to obtain the second output information;

[0010] Processing the second output information through a decoder module to obtain a high-dimensional feature vector, wherein the decoder includes: a probabilistic sparse self-attention module and a multi-head attention module;

[0011] The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

[0012] Optionally, before acquiring the time series data and the SSI model, the SSI model can be constructed. The process of constructing the SSI model is as follows: first, data preprocessing is involved, including missing value filling and data normalization, and then the data is divided into a training set and a test set. The time series data is input into the feature embedding layer, and the XGBoost model is used to select features of the meteorological information, and the features of each dimension are extracted through the position encoder, time encoder, trend encoder and scalar encoder to enrich the feature information of the model. These features are input into the encoder part of the model. The encoder consists of a multi-headed secondary sparse ProbSparse Self-Attention mechanism and a distillation mechanism. It screens out important query matrices and key matrices, and reduces the dimension through self-attention distillation to capture the long-term correlation of the load sequence. The output of the encoder is then input into the decoder, which contains hidden multi-headed probabilistic sparse attention and multi-headed attention. Finally, the high-dimensional output of the decoder is input into the output layer to output the final predicted value.

[0013] Optionally, the acquiring of time series data and the SSI model, extracting feature information of the time series data through a feature embedding module, calculating the feature information to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features and location features, the SSI model is used to indicate a model based on an improved Informer model, and the SSI model includes: the feature embedding module, an encoder module, a decoder module and an output layer, including:

[0014] Analyze the meteorological information in the time series data by using the XGBoost model to obtain the meteorological characteristics;

[0015] Processing the time series data in the time series data by a global time encoder to obtain the global time feature;

[0016] Acquire the trend feature in the time series data by a trend encoder;

[0017] Acquire the scalar feature and the position feature in the time series data by direct observation and position encoder;

[0018] The meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics are linearly added to obtain first output information.

[0019] Optionally, analyzing the meteorological information in the time series data by using an XGBoost model to obtain the meteorological characteristics includes:

[0020] The meteorological information is selected using the XGBoost model, and the feature with the highest score is selected as the meteorological feature to be used in model calculation.

[0021] Optionally, the processing of the time series data in the time series data by a global time encoder to obtain a global time feature includes:

[0022] The global time feature is obtained by splitting and encoding the time series data into six time granularities: year, month, day, hour, minute, and holiday.

[0023] Optionally, acquiring the trend feature in the time series data by using a trend encoder includes:

[0024] Construct a matrix D using the time series data at time t over the past h days;

[0025] Constructing a matrix P by using the time series data of the past h weeks at time t and constructing a matrix M by using the time series data of the past h months at time t;

[0026] Then D, P and M are fused to form a trend matrix, and finally convolution is used to extract the trend features.

[0027] Optionally, the acquiring of scalar features and position features in the time series data by direct observation and position encoder includes:

[0028] Extracting the scalar features in the time series data by direct observation;

[0029] The position feature in the time series data is extracted by a position encoder.

[0030] Optionally, the obtaining the second output information after filtering, performing secondary sparse processing and attention distillation processing on the first output information by the encoder module includes:

[0031] The first output information is filtered to obtain dominant query data by calculating the similarity between the self-attention distribution and the uniform distribution;

[0032] Performing secondary sparse processing on the dominant query data by a secondary sparse probabilistic sparse attention method to obtain secondary sparse data;

[0033] A distillation operation is performed on the secondary sparse data to obtain the second output information.

[0034] The long-term power load forecasting method based on the improved Informer model provided in the present application performs feature encoding and embedding of meteorological features, scalar information, location information, time information and trend information through a feature embedding layer, thereby providing a richer input feature representation for the model. The method also uses a method for evaluating the sparsity of the key-value matrix to perform secondary sparsity on the probabilistic sparse attention, thereby reducing the time complexity of the attention mechanism from O(LlnL) to O((lnL) 2 ), and finally output the final prediction value through the decoder and encoder; this method solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term series power load forecasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 1 ;

[0037] Figure 2 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 2 ;

[0038] Figure 3 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 3 .

[0039] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0042] In the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0043] In recent years, with the rapid development of social economy and the continuous advancement of science and technology, the demand for electricity in areas such as residents' lives, industrial production, commercial activities, information communication, and traffic management has been growing. In particular, the widespread popularization of various intelligent technologies and products has further expanded the demand for electricity in society. However, electric energy is an instant energy that is difficult to store in large quantities for a long time. It is necessary to find a balance between the production and use of electric energy. When the power generation is greater than the actual power demand, it will cause unnecessary waste. When the power generation is less than the actual power demand, there will be power shortages, large-scale power outages, emergency failures of power equipment, and a series of safety problems. Accurate power load forecasting can help power plants and power supply departments find the balance between power supply and demand, thereby avoiding the above problems. Therefore, power load forecasting is an important part of power system operation and planning, and is of great significance to ensuring the stable operation of the power system and optimizing resource allocation. The research on power load forecasting methods can help power operation departments to effectively plan power supply, reduce power waste, reduce costs, ensure the smooth operation of the power system, and then provide safe and stable power supply, which is of great significance to social production and life.

[0044] Current deep learning models, such as RNN and LSTM, or more complex neural network models with larger parameters for training and reasoning, have strong representation learning capabilities and improve the accuracy of load forecasting. However, when processing very long sequences, these models face problems such as large amount of calculation, long training time, and difficulty in capturing long-term dependencies. In addition, when processing time series data, these models usually ignore the time granularity information of the data, resulting in inaccurate prediction results.

[0045] In summary, how to improve the accuracy and efficiency of long-term series power load forecasting is the problem that this application needs to solve.

[0046] The long-term power load forecasting method based on the improved Informer model provided in the present application performs feature encoding and embedding of meteorological features, scalar information, location information, time information and trend information through a feature embedding layer, thereby providing a richer input feature representation for the model. The method also uses a method for evaluating the sparsity of the key-value matrix to perform secondary sparsity on the probabilistic sparse attention, thereby reducing the time complexity of the attention mechanism from O(LlnL) to O((lnL) 2 ), and finally output the final prediction value through the decoder and encoder; this method solves the problems of insufficient feature richness and insufficient sparsity of the probabilistic sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-term series power load forecasting.

[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 1 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 1 , the execution subject of this embodiment is the power load forecasting system, such as Figure 1 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0049] S101: Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information.

[0050] The feature information refers to meteorological features, global time features, trend features, scalar features and location features. The SSI model is used to indicate a model based on the improved Informer model. The SSI model includes: the feature embedding module, the encoder module, the decoder module and the output layer

[0051] The meteorological features are obtained by using the XGBoost model to select meteorological information and select the features with the highest scores as the meteorological features to be used in model calculation.

[0052] The global time feature is obtained by: t (X) to provide global time information for the model. For data X, this paper splits and encodes it from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data whose current time is 2023-03-0315:10, the extracted global time feature is represented by a numerical vector [23, 3, 3, 15, 1, 0], which represents 23 years, March, 3rd, 15 hours, 10 minutes, (60 minutes are divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1), and the vector is input into the fully connected network to obtain the global time feature. The formula is defined as:

[0053] x time =E t (X)

[0054] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0055] The trend feature is obtained by using the designed trend encoder Es(X). The trend feature proposed in this paper mainly includes three parts: constructing the matrix D using the load data of the past h days at time t, constructing the matrix P using the load data of the past h weeks at time t, and constructing the matrix M using the load data of the past h months at time t. Then D, P and M are fused to form the trend matrix T∈R 3×h , and finally use convolution to extract features, and its trend coding is defined as:

[0056] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0057] The scalar feature and position feature are obtained by directly observing the scalar feature x sca Extraction, for the position feature x pos To extract, the position encoder calculation method can be used.

[0058] The calculation method of calculating the first output information by calculating the characteristic information is:

[0059] The first output information is obtained by linearly adding the above-mentioned meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics.

[0060] The purpose of this step is to extract complete time features, extract periodicity and trend features to enhance the use of meteorological features when the model is applied to load forecasting tasks. These features play an important role in mining nonlinear long-distance dependencies between data and improving forecasting results.

[0061] Optionally, before acquiring the time series data and the SSI model, the SSI model can be constructed. The process of constructing the SSI model is as follows: first, data preprocessing is involved, including missing value filling and data normalization, and then the data is divided into a training set and a test set. The time series data is input into the feature embedding layer, and the XGBoost model is used to select features of the meteorological information, and the features of each dimension are extracted through the position encoder, time encoder, trend encoder and scalar encoder to enrich the feature information of the model. These features are input into the encoder part of the model. The encoder consists of a multi-headed secondary sparse ProbSparse Self-Attention mechanism and a distillation mechanism. It screens out important query matrices and key matrices, and reduces the dimension through self-attention distillation to capture the long-term correlation of the load sequence. The output of the encoder is then input into the decoder, which contains hidden multi-headed probabilistic sparse attention and multi-headed attention. Finally, the high-dimensional output of the decoder is input into the output layer to output the final predicted value.

[0062] S102: The encoder module filters the first output information, performs secondary sparse processing and performs attention distillation processing on the first output information to obtain the second output information.

[0063] The filtering of the first output information by the encoder module, the secondary sparseness and the attention distillation mechanism, and the acquisition of the second output information are completed by the following steps:

[0064] Step 1: Filter the first output information through the encoder module:

[0065] The dominant query in the first output information is screened by calculating the similarity between the self-attention distribution and the uniform distribution, which can be quantitatively calculated using the Kullback-Leibler divergence, which is defined as:

[0066]

[0067] In the formula, q i is the i-th query, k j is the jth key, N k is the number of keys. α represents the query q i The attention distribution of is expressed as follows:

[0068]

[0069] By removing the constant in the definition of KL divergence, a sparsity metric is designed to measure the significance of a query, which is defined as follows:

[0070]

[0071] The first term of this formula is q i and the Log-Sum-Exp of all keys, the second one is the arithmetic mean of the division results. i ,K) value is higher, indicating that q i The higher the probability of attention, the higher the probability of including the dominant point pair in the head field of the long-tail self-attention distribution. However, the complexity of the sparsity measure is still quadratic because it needs to calculate the dot product of each query and each key. For this reason, the Informer model designs an alternative measurement method, which is defined as:

[0072]

[0073] Under the long-tail distribution, we only need to randomly select L k LqCy k The dot product is calculated That is, fill the other pairs with zeros. Finally, only select The s top-valued dominant queries are used to perform self-attention calculation for each key, defined as:

[0074]

[0075] Among them, Q is the query matrix of the dominant query, which consists of the data points of the dominant query, K is the key matrix, which provides the characteristics of the information required by the query, and V is the value matrix, which represents the actual value information. Contains s dominant queries, By assigning a specific value to s, the ProbSparse self-attention mechanism is used to reduce the time complexity and space complexity of self-attention from N 2 Reduced to N×s.

[0076] Step 2: The secondary sparse ProbSpare Self-Attention mechanism proposed in the present invention uses the secondary sparse self-attention mechanism to perform secondary sparse on the data obtained in step 1. The specific process is as follows:

[0077] First, the Kullback-Leibler divergence is used to evaluate the sparsity of K. The sparsity evaluation of the i-th key is defined as:

[0078]

[0079] Where Q is the query matrix of the dominant query, q j is the jth query, k j is the jth key, Q Q is the number of keys. The first term of the formula is a Log-Sum-Exp for all keys, and the second term is their arithmetic mean. The higher the value, the higher the k i The probability of attention of the head field of the long-tail self-attention distribution containing the advantage point pair is greater than M(k i ,Q)high.

[0080] Based on the above evaluation method, the present invention performs sparsity evaluation on K, and then selects Top-u keys (u=lnL / c, c is a constant) from the results of the sparsity evaluation. These keys are the u keys with the highest activity in the sparsity evaluation, and these keys are composed of Replace K in the original formula.

[0081] It should be noted that in the process of sparsity evaluation, if the sparsity score is calculated for each key, it will bring additional calculations. Therefore, we can use the assumption that the dot product result follows a long-tail distribution. When calculating the sparsity score of each key, we only need to calculate the sparsity score of the key after performing a sparsity evaluation. Randomly sample some qurey and key to calculate. This reduces the time loss caused by the sparsity evaluation of the key. Therefore, under the long-tail distribution, only U = lnQlnK dot product pairs need to be randomly sampled to calculate M(k i,Q), and fill the other dot product pairs with zeros. The reason for filling with zero values ​​is that M(k i ,Q) is not very sensitive to zero values ​​and is numerically stable.

[0082] Based on the above method, the input of the attention mechanism dot product operation is changed from K is converted to This makes the ProbSparse Self-Attention after quadratic sparseness only need O((lnL) 2 ) can be operated on by the dot product of and The dot product operation is calculated It can perform matrix multiplication operation with the V matrix. In addition, the present invention uses the median filling method to smooth S, and the filled S is recorded as Based on the above method, the present invention proposes a new measurement method based on the attention mechanism proposed by the Informer model, which is defined as:

[0083]

[0084] Where: is a query matrix that contains only a few selected queries that contribute a large proportion to the attention value. is the key matrix with the highest activity ranking selected by the quadratic sparse ProbSpare Self-Attention, V is the value matrix, and d is the dimension of Q, K, and V.

[0085] Step 3: The specific process of processing the output of step 2 through the attention distillation mechanism is as follows:

[0086] After applying the quadratic sparse self-attention mechanism to the Q, K, V matrices, an attention distillation operation is required to reduce the feature dimension while maintaining the high sensitivity of the attention mechanism to the sequence relationship. The attention distillation operation is performed by a 1-D convolutional layer with an ELU activation function and passed through a maximum pooling layer with a stride of 2. The distillation operation is defined as:

[0087]

[0088] Where [.]AB represents the attention block. It contains multi-head quadratic sparse ProbSparse Self-Attention and basic operations, Conv1d(.) uses the ELU(.) activation function. By using the distillation extraction operation, the dimension of the feature map of each head probabilistic sparse self-attention will be reduced to half of its original dimension, and all output information will be connected as the hidden representation of the model encoder.

[0089] It can be understood that the first output information of the feature embedding module is used as the input of the encoder of the secondary sparse ProbSpare Self-Attention. The ProbSpare Self-Attention mechanism believes that the sparsity of the self-attention matrix is ​​manifested as: different queries show different levels of concern for the key (that is, different activity levels, the higher the activity level, the higher the self-attention score). A dominant query means that if the self-attention distribution of a query is more in line with the long-tail distribution, the query has more dominant dot products, which is called a dominant query. The present invention uses a quadratic sparse probabilistic sparse attention method. In the present invention, experiments show that while the dot product pairs under the ProbSpare Self-Attention mechanism sparse the query matrix, some keys maintain a small correlation with almost all queries. When these keys are calculated with the query, the time complexity and space complexity of the self-attention module will increase, that is, the attention matrix after the sparse self-attention mechanism is still sparse. Therefore, the present invention performs a quadratic sparse processing on the attention score matrix, re-evaluates the self-attention mechanism under the ProbSpare Self-Attention mechanism qualitatively, and realizes the secondary sparseness of ProbSpare Self-Attention by sparsely processing the key value matrix K. After the secondary sparse attention score matrix, the attention scores of the query and the key are maintained at a relatively high score as a whole, which shows that the quadratic sparse method can indeed eliminate keys with low activity.

[0090] S103: Process the second output information through a decoder module to obtain a high-dimensional feature vector.

[0091] The method of obtaining a high-dimensional feature vector by processing the second output information through the decoder module is:

[0092] Next, the output of the second output information obtained by the encoder is used as part of the input of the decoder, which consists of a probabilistic sparse self-attention layer and a multi-head self-attention layer. Here, the masked dot product is directly set to -∞, and the output of the multi-head self-attention layer is concatenated with the result of the encoder, and then the obtained high-dimensional vector is represented, which is defined as follows:

[0093] X decoder =Concat(MultiHeadAttention(Q de ,K de ,V de ),X encoder )

[0094] Among them, Qde , K de 、V de are respectively the query, key and value vectors of the decoder, X ender is the output of the encoder.

[0095] S104: Using the high-dimensional feature vector as an input of an output layer, and converting the high-dimensional feature vector into a final prediction value through the output layer.

[0096] The method of converting the high-dimensional feature vector into the final prediction value through the output layer is as follows: the high-dimensional feature vector obtained in step S103 is input into the fully connected layer, i.e., the output layer, and its output is fully connected with the load value to obtain the final result. The input of the output layer at time t is the concatenation of the two parts, and the final prediction value is defined as:

[0097]

[0098] in, The length L token The start tag of , as shown in this definition:

[0099]

[0100] The target sequence length is L y The length of the decoder input sequence is x. token The sum of the lengths of x0, L de =L token +L y .

[0101] The embodiment of the present application provides a long-time series power load forecasting method based on an improved Informer model. The method extracts feature information of time series data through a feature embedding module, obtains first output information after linear addition processing of the feature information, obtains second output information after screening, secondary sparseness and attention distillation mechanism processing of the first output information through an encoder module, obtains a high-dimensional feature vector through a decoder module, uses the high-dimensional feature vector as input of the output layer, and converts the high-dimensional feature vector into the final prediction value through the output layer. Through this method, the method solves the problem of insufficient feature richness when the original Informer model is applied to power load forecasting and the problem of insufficient sparsity of the probability sparse attention module, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0102] Figure 2 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, the long time series power load forecasting method based on the improved Informer model is described in detail. Figure 2 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0103] S201: Analyze the meteorological information in the time series data through the XGBoost model to obtain meteorological characteristics.

[0104] Among them, the meteorological characteristics have an important impact on load forecasting, but some meteorological information has a weak correlation with load forecasting or has information redundancy, which will reduce the performance of the prediction model. Therefore, in the meteorological feature extraction part, the present invention uses XGBoost to extract meteorological information X wth Perform feature selection. These meteorological information include weather, temperature, humidity, wind speed, etc. According to the feature frequency, select the top_f features with the highest feature importance score and add them to the feature subset T wth In the example, the fully connected layer is used as the meteorological encoder E w (X), and finally T wth ∈R Top_f×t Input to E w (X) to obtain the meteorological characteristics, which are formally expressed as:

[0105] x wth =E w (T wth )

[0106] S202: Processing the time series data in the time series data through a global time encoder to obtain global time features.

[0107] Among them, the present invention provides global time information for the model by designing a global time encoder. For time series data, this paper splits and encodes from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data whose current time is 2023-03-03 15:10, the extracted global time features are represented by a numerical vector [23, 3, 3, 15, 1, 0], which respectively represent 23 years, March, 3rd, 15 hours, 10 minutes, (60 minutes are divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1), and the vector is input into the fully connected network to obtain the global time features, and its formula is defined as:

[0108] x time =E t (X)

[0109] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0110] It is understandable that load forecasting is a classic time series forecasting task, and its global time characteristics play an important role in the model learning the dependency of the time dimension of data. Therefore, a global time encoder is designed to provide the model with global time information.

[0111] S203: Acquire the trend features in the time series data through a trend encoder.

[0112] The specific implementation method of obtaining the trend features in the time series data through the trend encoder is: obtaining the trend features through the trend encoder Es(X) designed by the present invention. The trend features proposed in this application mainly include three parts: constructing a matrix D using the load data of the past h days at time t, constructing a matrix P using the load data of the past h weeks at time t, and constructing a matrix M using the load data of the past h months at time t. Then D, P and M are fused to form a trend matrix T∈R 3×h , and finally use convolution to extract features, and its trend coding is defined as:

[0113] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0114] It can be understood that the purpose of this step is to better reflect the periodicity and trend of the load data.

[0115] S204: Acquire scalar features and position features in time series data through direct observation and position encoder.

[0116] Among them, the scalar feature x sca It can be obtained from time series data through direct observation.

[0117] Position feature x pos The acquisition method in the specific implementation is:

[0118] The input representation is an embedding of three independent parts and a scalar projection, as well as local and global timestamps.

[0119] For time t The value of the input sequence indicates that the method first passes through d modle The convolution filter projects it onto a d modle -dim vector In the example, the kernel size is 3 and the step size is 1, where d modle is the dimensionality of the input representation.

[0120] Then, the fixed position embedding at time t is used to retain the local context and obtain the position information, and its position feature is defined as:

[0121]

[0122] Where pos=1,2,…,L x and j = 1, 2, ..., d modle .L x is the length of the input sequence, L x =L en or L de denote the input of the encoder or decoder of the proposed model, respectively.

[0123] The purpose of this step is to obtain global hierarchical timestamps (such as weeks, months, and years) and agnostic timestamps (such as holidays and others), which are necessary to obtain long-term independence for long-series time series forecasts.

[0124] S205: Perform linear addition calculation on meteorological features, global time features, trend features, scalar features and location features to obtain first output information.

[0125] The specific implementation method of this step is: linearly adding the meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics obtained in steps S201-205 to obtain the input of the model. The model input is defined as:

[0126] x input =Add(x wth ,x time ,x tre, x pos ,x sca )

[0127] S206: Filter the first output information to obtain dominant query data by calculating the similarity between the self-attention distribution and the uniform distribution.

[0128] The specific implementation method of filtering the dominant query in the first output information by calculating the similarity between the self-attention distribution and the uniform distribution can be quantitatively calculated using the Kullback-Leibler divergence, which is defined as:

[0129]

[0130] In the formula, q i is the i-th query, k j is the jth key, N k is the number of keys. α represents the query q i The attention distribution of is expressed as follows:

[0131]

[0132] By removing the constant in the definition of KL divergence, a sparsity metric is designed to measure the significance of a query, which is defined as follows:

[0133]

[0134] The first term of this formula is q i and the Log-Sum-Exp of all keys, the second one is the arithmetic mean of the division results. i ,K) value is higher, indicating that q i The higher the probability of attention, the higher the probability of including the dominant point pair in the head field of the long-tail self-attention distribution. However, the complexity of the sparsity measure is still quadratic because it needs to calculate the dot product of each query and each key. For this reason, the Informer model designs an alternative measurement method, which is defined as:

[0135]

[0136] Under the long-tail distribution, we only need to randomly select L k LqCy k The dot product is calculated That is, fill the other pairs with zeros. Finally, only select The s top-valued dominant queries are used to perform self-attention calculation for each key, defined as:

[0137]

[0138] Among them, Q is the query matrix of the dominant query, which consists of the data points of the dominant query, K is the key matrix, which provides the characteristics of the information required by the query, and V is the value matrix, which represents the actual value information. Contains s dominant queries, By assigning a specific value to s, the ProbSparse self-attention mechanism is used to reduce the time complexity and space complexity of self-attention from N 2 Reduced to N×s.

[0139] S207: Perform secondary sparse processing on the dominant query data through a secondary sparse probabilistic sparse attention method to obtain secondary sparse data.

[0140] The specific process of performing secondary sparse processing on the dominant query data by the secondary sparse probabilistic sparse attention method to obtain secondary sparse data is as follows:

[0141] First, the Kullback-Leibler divergence is used to evaluate the sparsity of K. The sparsity evaluation of the i-th key is defined as:

[0142]

[0143] Where Q is the query matrix of the dominant query, q j is the jth query, k j is the jth key, Q Q is the number of keys. The first term of the formula is a Log-Sum-Exp for all keys, and the second term is their arithmetic mean. The higher the value, the higher the k i The probability of attention of the head field of the long-tail self-attention distribution containing the advantage point pair is greater than M(k i ,Q)high.

[0144] Based on the above evaluation method, the present invention performs sparsity evaluation on K, and then selects Top-u keys (u=lnL / c, c is a constant) from the results of the sparsity evaluation. These keys are the u keys with the highest activity in the sparsity evaluation, and these keys are composed of Replace K in the original formula.

[0145] It should be noted that in the process of sparsity evaluation, if the sparsity score is calculated for each key, it will bring additional calculations. Therefore, we can use the assumption that the dot product result follows a long-tail distribution. When calculating the sparsity score of each key, we only need to calculate the sparsity score of the key after performing a sparsity evaluation. Randomly sample some qurey and key to calculate. This reduces the time loss caused by the sparsity evaluation of the key. Therefore, under the long-tail distribution, only U = lnQlnK dot product pairs need to be randomly sampled to calculate M(k i ,Q), and fill the other dot product pairs with zeros. The reason for filling with zero values ​​is that M(k i ,Q) is not very sensitive to zero values ​​and is numerically stable.

[0146] Based on the above method, the input of the attention mechanism dot product operation is changed from K is converted to This makes the ProbSparse Self-Attention after quadratic sparseness only need O((lnL) 2 ) can be operated on by the dot product of and The dot product operation is calculated It can perform matrix multiplication operation with the V matrix. In addition, the present invention uses the median filling method to smooth S, and the filled S is recorded as Based on the above method, the present invention proposes a new measurement method based on the attention mechanism proposed by the Informer model, which is defined as:

[0147]

[0148] Where: is a query matrix that contains only a few selected queries that contribute a large proportion to the attention value. is the key matrix with the highest activity ranking selected by the quadratic sparse ProbSpare Self-Attention, V is the value matrix, and d is the dimension of Q, K, and V.

[0149] S208: Perform a distillation operation on the secondary sparse data to obtain second output information.

[0150] The specific process of performing the distillation operation on the secondary sparse data to obtain the second output information is as follows:

[0151] After applying the quadratic sparse self-attention mechanism to the Q, K, V matrices, an attention distillation operation is required to reduce the feature dimension while maintaining the high sensitivity of the attention mechanism to the sequence relationship. The attention distillation operation is performed by a 1-D convolutional layer with an ELU activation function and passed through a maximum pooling layer with a stride of 2. The distillation operation is defined as:

[0152]

[0153] Where [.]AB represents the attention block. It contains multi-head quadratic sparse ProbSparse Self-Attention and basic operations, Conv1d(.) uses the ELU(.) activation function. By using the distillation extraction operation, the dimension of the feature map of each head probabilistic sparse self-attention will be reduced to half of its original dimension, and all output information will be connected as the hidden representation of the model encoder.

[0154] S209: Process the second output information through a decoder module to obtain a high-dimensional feature vector.

[0155] S210: The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

[0156] Among them, steps S209-S210 are similar to steps S103-S104 and will not be repeated here.

[0157] The embodiment of the present application provides a long-time series power load forecasting method based on an improved Informer model. The method analyzes meteorological information in time series data through an XGBoost model to obtain meteorological features, processes time series data in time series data through a global time encoder to obtain global time features, obtains the trend features in time series data through a trend encoder, obtains scalar features and position features in time series data through direct observation and position encoders, obtains the first output information after linear addition processing of the above feature information, obtains the second output information after screening the first output information through an encoder module, processes the second output information through a secondary sparse and attention distillation mechanism, obtains the high-dimensional feature vector through a decoder module, uses the high-dimensional feature vector as the input of the output layer, and converts the high-dimensional feature vector into the final prediction value through the output layer. Through this method, the method solves the problem of insufficient feature richness and insufficient sparsity in the probability sparse attention module when the original Informer model is applied to power load forecasting, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0158] Figure 3 Schematic diagram of the process of long-term power load forecasting method based on improved Informer model provided in this application Figure 3 , this embodiment is in Figure 2 Based on the embodiments, “analyzing meteorological information in time series data through the XGBoost model to obtain meteorological features”, “processing time series data in time series data through the global time encoder to obtain global time features”, “obtaining the trend features in time series data through the trend encoder”, and “obtaining scalar features and position features in time series data through direct observation and position encoder” are described in detail, such as Figure 3 As shown, the long-time series power load forecasting method based on the improved Informer model provided in this embodiment includes:

[0159] S301: Use the XGBoost model to select meteorological information and select the feature with the highest score as the meteorological feature to be used in model calculation.

[0160] Among them, in the meteorological feature extraction part, the present invention uses XGBoost to extract meteorological information X wth Perform feature selection. These meteorological information include weather, temperature, humidity, wind speed, etc. According to the feature frequency, select the top_f features with the highest feature importance score and add them to the feature subset T wth In the example, the fully connected layer is used as the meteorological encoder E w (X), and finally T wth ∈R Top_f×tInput to E w (X) to obtain the meteorological characteristics, which are formally expressed as:

[0161] x wth =E w (T wth )

[0162] It can be understood that the meteorological characteristics have an important impact on load forecasting, but some meteorological information has a weak correlation with load forecasting or has information redundancy, which will reduce the performance of the prediction model. Therefore, the present invention uses XGBoost to optimize the meteorological information XGBoost. wth Perform feature selection.

[0163] S302: Obtain global time features by splitting and encoding the time series data at six time granularities: year, month, day, hour, minute, and holiday.

[0164] Among them, the present invention provides global time information for the model by designing a global time encoder. For time series data, this paper splits and encodes from six time granularities: year, month, day, hour, minute, and holiday. For example, for the data whose current time is 2023-03-03 15:10, the extracted global time features are represented by a numerical vector [23, 3, 3, 15, 1, 0], which respectively represent 23 years, March, 3rd, 15 hours, 10 minutes, (60 minutes are divided into 6 segments, so the value of 10 minutes is 1), non-holidays (because holidays are 1), and the vector is input into the fully connected network to obtain the global time features, and its formula is defined as:

[0165] x time =E t (X)

[0166] Where X is the extracted global time feature numerical vector representation, E t (.) is the fully connected layer.

[0167] It is understandable that load forecasting is a classic time series forecasting task, and its global time characteristics play an important role in the model learning the dependency of the time dimension of data. Therefore, a global time encoder is designed to provide the model with global time information.

[0168] S303: Construct a matrix D using the time series data of the past h days at time t.

[0169] S304: constructing a matrix P using the time series data of the past h weeks at time t and constructing a matrix M using the time series data of the past h months at time t.

[0170] S305: D, P, M and are combined to form a trend matrix, and finally convolution is used to extract trend features.

[0171] Among them, steps S303 and S304 are routine operations of those skilled in the art, and can be implemented by using conventional matrix construction technology, so they will not be described in detail here.

[0172] The specific implementation method of fusing D, P, M and to form a trend matrix and finally using convolution to extract trend features is as follows:

[0173] The specific implementation method of obtaining the trend features in time series data through the trend encoder is as follows: the trend features are obtained through the trend encoder Es(X) designed by the present invention. The trend features proposed in this application mainly include three parts: constructing a matrix D using the load data of the past h days at time t, constructing a matrix P using the load data of the past h weeks at time t, and constructing a matrix M using the load data of the past h months at time t. Then D, P and M are fused to form a trend matrix T∈R 3 ×h , and finally use convolution to extract features, and its trend coding is defined as:

[0174] x tre =E s (X)=Conv1d(T)=concat(D,P,M)

[0175] It can be understood that the purpose of this step is to better reflect the periodicity and trend of the load data.

[0176] S306: Extract scalar features from time series data through direct observation.

[0177] Among them, for the scalar feature x sca , technicians in this field can directly observe and obtain it from time series data.

[0178] S307: Extracting position features in the time series data through a position encoder.

[0179] Among them, the position feature x pos The acquisition method in the specific implementation is:

[0180] The input representation is an embedding of three independent parts and a scalar projection, as well as local and global timestamps.

[0181] For time t The value of the input sequence indicates that the method first passes through d modle The convolution filter projects it onto a d modle -dim vector In the example, the kernel size is 3 and the step size is 1, where d modle is the dimensionality of the input representation.

[0182] Then, the fixed position embedding at time t is used to retain the local context and obtain the position information, and its position feature is defined as:

[0183]

[0184] Where pos=1,2,…,L x and j = 1, 2, ..., d modle .L x is the length of the input sequence, L x =L en or L de denote the input of the encoder or decoder of the proposed model, respectively.

[0185] The purpose of this step is to obtain global hierarchical timestamps (such as weeks, months, and years) and agnostic timestamps (such as holidays and others), which are necessary to obtain long-term independence for long-series time series forecasts.

[0186] The long-time series power load forecasting method based on the improved Informer model provided in the embodiment of the present application, the method analyzes the meteorological information in the time series data through the XGBoost model to obtain meteorological features, processes the time series data in the time series data through the global time encoder, obtains the global time features, so as to obtain the global hierarchical timestamps (such as weeks, months and years) and unknown timestamps (such as holidays and others), obtains the trend features in the time series data through the trend encoder, obtains the scalar features and position features in the time series data through direct observation and position encoder, obtains the first output information after linear addition processing of the above feature information, obtains the second output information after filtering the first output information through the encoder module, processes the second output information through the secondary sparse and attention distillation mechanism, obtains the high-dimensional feature vector through the decoder module, takes the high-dimensional feature vector as the input of the output layer, and converts the high-dimensional feature vector into the final prediction value through the output layer. Through this method, the method solves the problem that the original Informer model is insufficient in feature richness when applied to power load forecasting and the probability sparse attention module still has insufficient sparsity, thereby achieving the effect of improving the accuracy and efficiency of long-time series power load forecasting.

[0187] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0188] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0189] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0190] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A long-time series power load forecasting method based on an improved Informer model, characterized in that: Applied to a power load forecasting system, the method comprises: Acquire time series data and an SSI model, extract feature information of the time series data through a feature embedding module, and calculate the feature information to obtain first output information, wherein the feature information includes: meteorological features, global time features, trend features, scalar features, and location features, and the SSI model is used to indicate a model improved based on an Informer model, and the SSI model includes: the feature embedding module, an encoder module, a decoder module, and an output layer; The encoder module filters the first output information, performs secondary sparse processing and performs attention distillation processing on the first output information to obtain the second output information; Processing the second output information by the decoder module to obtain a high-dimensional feature vector; The high-dimensional feature vector is used as the input of the output layer, and the high-dimensional feature vector is converted into a final prediction value through the output layer.

2. The method according to claim 1, characterized in that The acquiring of time series data and the SSI model, extracting feature information of the time series data through a feature embedding module, and calculating the feature information to obtain first output information include: Analyze the meteorological information in the time series data by using the XGBoost model to obtain the meteorological characteristics; Processing the time series data in the time series data by a global time encoder to obtain the global time feature; Acquire the trend feature in the time series data by a trend encoder; Acquire the scalar feature and the position feature in the time series data by direct observation and position encoder; The meteorological characteristics, global time characteristics, trend characteristics, scalar characteristics and location characteristics are linearly added to obtain first output information.

3. The method according to claim 2, characterized in that The step of analyzing the meteorological information in the time series data by using the XGBoost model to obtain the meteorological characteristics includes: The meteorological information is selected using the XGBoost model, and the feature with the highest score is selected as the meteorological feature to be used in model calculation.

4. The method according to claim 2, characterized in that: The step of processing the time series data in the time series data by a global time encoder to obtain a global time feature includes: The global time feature is obtained by splitting and encoding the time series data into six time granularities: year, month, day, hour, minute, and holiday.

5. The method according to claim 2, characterized in that: The step of obtaining the trend feature in the time series data by using a trend encoder includes: Construct a matrix D using the time series data at time t over the past h days; Constructing a matrix P by using the time series data of the past h weeks at time t and constructing a matrix M by using the time series data of the past h months at time t; D, P and M are fused to form a trend matrix, and finally convolution is used to extract the trend features.

6. The method according to claim 2, characterized in that The step of obtaining the scalar feature and the position feature in the time series data by direct observation and position encoder includes: Extracting the scalar features in the time series data by direct observation; The position feature in the time series data is extracted by a position encoder.

7. The method according to claim 1, characterized in that The obtaining of the second output information by filtering, performing secondary sparse processing and attention distillation processing on the first output information through the encoder module includes: The first output information is filtered to obtain dominant query data by calculating the similarity between the self-attention distribution and the uniform distribution; Performing secondary sparse processing on the dominant query data by a secondary sparse probabilistic sparse attention method to obtain secondary sparse data; A distillation operation is performed on the secondary sparse data to obtain the second output information.

Images